Virtual and physical training system for semi-autonomous teleoperated manipulators

Problem statement

There is presently a lack of integrated online learning capabilities for robotic systems that can seamlessly bridge virtual simulation and physical operation with human inputs to enable efficient adaptive manipulation in unstructured, contact-rich environments. Typical applications that can benefit from these systems include inspection, maintenance and repair of utility infrastructures (e.g. power lines, dams, wind turbine blades) search and rescue, medical interventions and Explosive Ordnance Disposal (EOD) operations.

Using a combination of pre-programmed behaviors and human inputs via a force-enabled input device, a virtual robot is to interact dynamically with an environment consisting of a scene and objects using a gripper or specialized tools, while providing feedback to the operator using visual and kinesthetic/haptic feedback via the input device. Virtual scene and objects can be created using a CAD environment or from digitized real-world elements. The system is to allow the development and implementation of online machine learning algorithms via feedback from virtual sensors (visual, robot encoders, force sensors, input device, etc.). Once tasks have been prototyped in the virtual environment, further training and validation of the learned behaviors and trajectories is to be carried out using the same operator interface and previously used virtual environment coupled to a hardware manipulator in a physical environment, thereby providing force information and sensor feedback resulting from physical interaction. This approach aims at providing realistic feedback related to systems dynamics and contact, as well as providing an environment where system training can be carried out in an online, ongoing manner during human interaction.

The system is expected to have a minimum Technical Readiness Level of 5, more specifically, a system capable of prototyping and testing in a laboratory environment. While the resulting components are not expected to be fully field deployable, elements related to successful field deployment such as compacity, ruggedness, reliability, stiffness vs weight, power, should be considered in designing the solution.

Desired outcomes and considerations

Essential (mandatory) outcomes

The proposed solution must:

• Provide an integrated hardware and 3-D graphical virtual environment with dynamic capabilities for programming paths and behaviors of a remotely operated robot in an unstructured environment in a shared autonomy manner concurrent with human inputs.
• Provide the ability for users to model robotic manipulators operating in highly realistic environments (urban, rural...) and dynamic scene objects (e.g. chairs, doors, bags, pegs, screws wires) allowing the implementation of remote manipulation scenarios involving physical interaction with the environment.
• Provide the means of implementing online machine learning algorithms using data from operator inputs and feedback from both virtual and physical robots as they interact with their respective virtual and physical environments.
• Include a hardware robot with anthropomorphic dynamic capabilities that can interface with the virtual scene and provide real-time feedback based on real-world motion and force sensing capabilities.
• Include a low noise (mechanical and electronic) and high-fidelity input device capable of transmitting motion and reliable force commands to both the remote virtual environment and its physical counterpart, while providing force feedback to the user based on motion tracking and interaction forces experienced at the remote virtual and/or physical sites to enable accurate identification of human input dynamics.
• Provide a control and actuation chain that is near field-ready for both the input device and robot, such that transitioning from a laboratory setup to a field-deployed system requires no changes to the control methodology, actuator dynamics, torque density, or maintenance approach.
• Include the necessary real-time software environment for deploying the paths and learned behaviors prototyped in the virtual environment, as well as performing further online task development and learning.

Additional outcomes

The proposed solution should:

• Be modular and robot-agnostic in its implementation, such that although it features a specific robot and its digital twin, it can accommodate different robot models with minimal modifications and effort, with the same flexibility applying to the input device across system configurations.
• Include a high-level graphical scene modeling environment that is user-friendly and supports real-time dynamic capabilities.
• Incorporate a realistic, high-performance physics engine that, while not required to exactly replicate real-world interaction forces, provides contact-realistic modeling with real-time capabilities such as MujoCo^{Footnote 1} or Drake^{Footnote 2}.
• Provide a seamless interface between software modules that minimizes the need for additional programming, ensuring that system integration requires minimal effort and allowing users to focus on robotics, control and learning development.
• Include a lightweight, self-contained hardware robot with sufficient dynamic bandwidth for contact applications, designed to maximize transportability and reconfigurability without compromising dynamic force performance, and ideally incorporating features such as a quick-connect base, integrated power and motor drives, and a portable controller.

Background and context

Over the recent years remotely operated robotic systems, whether aerially or ground deployed, have been used increasingly in critical civil or military applications such as inspection, maintenance and repair of utility infrastructures (e.g. power lines, dams, wind turbines) search and rescue, medical interventions and Explosive Ordnance Disposal (EOD) operations. For these types of operations, a high level of intelligence and autonomy on behalf of the remote robot is desirable to minimize task load and increase situation awareness for the operator. In addition, elements related to telepresence, intuitive operation, and limiting information overload through multimodal feedback quickly become determining factors in the successful execution of the task.

Many of these applications require the manipulator to be in contact with their environment, where both force and motion as well as their dynamic relationship must be controlled in a stable and efficient manner. Contact-based tasks are notoriously difficult for robot manipulators to execute in a fully autonomous manner, especially in unstructured environments such as the ones involved in the use cases mentioned above. Thus, the presence of humans-in-the-loop to assist the system is often the most viable solution. Recent advances in Artificial Intelligence and Machine Learning have greatly contributed to increasing the level of autonomy of robotic systems. Indeed, a study^{Footnote 3} performed by the Toyota Research Institute in collaboration with Columbia University and the MIT Computer Science and Artificial Intelligence Lab has clearly demonstrated the advantages of using Humans-in-the-Loop and haptic feedback in teaching new robot behaviors. An environment combining online learning capabilities based on human inputs would potentially create more versatile, adaptable remotely operated robots for applications in unstructured environments involving physical interaction.

Multiple grants could result from this Challenge.

Phase 2

• Maximum funding: $1,500,000.00 CAD
• Project duration: Up to 18 months
• Estimated number of grants: 1

Note: Selected companies are eligible to receive one grant per phase per challenge. This disclosure is made in good faith and does not commit Canada to award any grant for the total approximate funding. Final decisions on the number of Phase 2 awards will be made by Canada based on factors such as evaluation results, departmental priorities, and availability of funds. The Government of Canada reserves the right to make partial awards and to negotiate project scope changes.

Travel

The successful applicant will be expected to travel to NRC's Aerospace Manufacturing Technology Center, Montreal to demonstrate the operation of the system and provide training to enable NRC researchers to independently test and validate its performance.

Kick-off meeting

All communication will take place by telephone or videoconference.

Progress review meeting(s)

Any progress review meetings will be conducted by telephone or videoconference.

Final review meeting

All communication will take place by telephone or videoconference.

Solution proposals can be submitted by a business that meets all of the following criteria:

• for profit
• incorporated in Canada (federally or provincially)
• small and medium sized business with 499 or fewer full-time equivalent (FTE) employees^**
• research and development activities that take place in Canada
• 50% or more of its annual wages, salaries and fees are currently paid to employees and contractors who spend the majority of their time working in Canada^{Footnote **}
• 50% or more of its FTE employees have Canada as their ordinary place of work^{Footnote **}
• 50% or more of its senior executives (Vice President and above) have Canada as their principal residence^{Footnote **}

All incoming questions regarding this specific challenge should be addressed to solutions@ised-isde.gc.ca.

All enquiries must be submitted in writing no later than ten calendar days before the Challenge Notice closing date. Enquiries received after that time may not be answered.

A glossary is also available.