Visual AI for space robotics obstacle detection

The Canadian Space Agency (CSA) is seeking solutions that will improve the autonomy of future space robotic systems through the use of a vision system based on Artificial Intelligence (AI) techniques to detect in real-time potentially hazardous obstacles and verify clearance margins in dynamic and uncertain environments.

Challenge sponsor:
Canadian Space Agency (CSA)

Funding mechanism:
Contract

Opening date:
August 25, 2022

Closing date:
October 20, 2022 14:00 Eastern Time

Please refer to the tender notice for this challenge on CanadaBuys.

 

Challenge
 

Problem statement

Collision-free autonomous path execution in a dynamic and uncertain environment is an important issue which CSA seeks to resolve in order to increase robotic system autonomy for future space missions. Robotic systems operate on space infrastructure or in other environments where collisions can potentially be catastrophic. Future space systems will need to have greater autonomy in order to be less reliant on communications with the ground and to alleviate crew workload; consequently they will need to rely on multiple layers of intelligent software for supervision and safety assurance.

Cameras both built into the robot and mounted to the infrastructure provide a relatively low mass, power, and volume solution for sensing the robotic workspace. Canadarm2 for example is equipped with on-board cameras in its end-effectors and along its booms in order to allow human operators to guide and monitor its operations. Future missions such as the cislunar Gateway call for autonomous robotic operations during periods when video cannot be downlinked to the ground for oversight.

This challenge seeks to explore the feasibility of an AI-based vision system to assist or offload the need for human ground controllers to monitor clearances. For CSA, an efficient robot supervisory system translates into improved safety and operational efficiency of the mission. CSA believes that a computer vision system based on machine learning will be able to detect obstacles and monitor proximity to structure and obstacles while handling payloads. CSA can provide Mobile Servicing System and International Space Station video and telemetry data during phase 1 and 2.

Desired outcomes and considerations

Essential (mandatory) outcomes

The solution must:

  • Operate in real-time, in a space environment, under all solar lighting conditions.
  • Use 2D camera data (still images and video streams) from existing cameras built into a space manipulator or mounted to its infrastructure and not assume any additional sensor functionality.
  • Have sufficient resolution and accuracy to detect the presence of slender objects (~2 cm wide) such as cable harnesses and EVA tethers.
  • Have a false positive rate of 2 % or lower and a false negative rate of 1% or lower.
  • Have to assume that the cameras could be mounted on the manipulator's booms or end-effector (e.g. may not be stationary during operation).
  • Work with a data bandwidth (video stream from the camera to processor) limited to 512 kbps and an update rate of at least 2 Hz.
  • Provide a 3D virtual world display of the manipulator and any obstacles detected.
  • Provide obstacle detection along the entire length of a space robotic arm and about any attached or manipulated payloads.
  • Be able to operate on a typical space platform such as a 2-4 core, ARM-based FPGA System on a chip, running VxWorks or Linux, a few GB of RAM, a solid-state drive, a power budget of approximately 10 Watts, Ethernet connectivity capable of transfer rates of at least 100Mbps.

Additional outcomes

The solution should:

  • Have robust performance with respect to distorted and blurred images streamed from the onboard cameras due to the oscillations caused by movements of the robotic arm.
  • Incur minimum budget increase to power, mass, volume, and computation resources (CPU and memory) allocated to a space robot's avionics concept.
  • Investigate the feasibility of adding path-to-flight computing hardware to the on-board avionics if the computation power of the space robot's avionics is found to be insufficient.
  • Be able to detect features inside of a given clearance volume or a keep out zone, for example a cylinder with depth of 2.4 m and diameter 2 m.
  • Investigate multiple cameras to cover the entire volume of the keep out zone.
  • Use a vision algorithm requiring a minimum number of independent camera views (expected to be part of study).
  • Be applicable to a 7 degree of freedom, offset joint robot similar to the Special Purpose Dexterous Manipulator on the International Space Station.

Background and context

It is a critical safety consideration for a space servicing robotic manipulator to avoid unintended contact during operations. For example, collision prevention in Canadarm2 begins at the planning stage. Robotic trajectories are designed using a kinematic simulator that uses high-fidelity graphical models of the external International Space Station (ISS) structure, the Space Station Remote Manipulator System (SSRMS), and the Special Purpose Dexterous Manipulator (SPDM). Robotics mission designers use this tool to define a series of poses and trajectories for the manipulator. A graphical planning tool can be used to verify that the trajectory meets the needs of the mission while avoiding singularities, self-collisions, and joint limits, and respecting clearances to structure and any defined keep-out zones (e.g., a volume around a radiating antenna).

While these planned paths are a good starting point, they cannot be executed blindly for the following reasons:

  • Disturbances in the station geometry due to pressure effects at docking ports and thermal expansion/contraction cannot be accounted for in the Computer-Aided Design (CAD) model.
  • Differences may exist between the as-designed model and the as-built modules.
  • Space vehicles have appendages such as solar arrays and antennas which can deploy or articulate to maintain their required pointing. If their articulation states are unknown, the robot must avoid their entire possible swept volume.
  • The exterior configuration of the vehicle may be physically altered by Extra-Vehicular Activity (EVA) crew members, left-behind debris, or installation of logistics and science payloads which were not represented in the model at planning time.
  • Faults in the attachment of external components may occur, such as cable harnesses coming loose or floating vehicle tether, and these may interfere with the operational workspace of the robotic system.
  • EVA crew members may not be aware of or misjudge upcoming manipulator motion, so may accidentally be in the planned path of the end-effector or booms.

Again in the example of Canadarm2, at execution time, when the manipulator's end effector approaches proximity to the station, a human operator must observe all motions to ensure that clearance is maintained between the station structure, arm, and payload. The SSRMS elbow pan-tilt-zoom cameras, the station infrastructure cameras, or both can provide the necessary video views for this task. For SPDM operations, ideal camera views are often not available. It is important to note that the clearance monitoring hinges on the human's perceptual system to filter out noise, infer depth, and accommodate the harsh and variable on-orbit lighting conditions.

For future space missions, such as the cislunar Gateway station, the majority of robotic operations are expected to be performed autonomously with no crew onboard and no communication link to the ground. As such, the hazard control function that was previously provided by the operator watching the downlinked video must be performed by some aspect of the onboard system. The AI-based vision system is one possible solution for providing this hazard control.

It may be possible for the robot to build or correct its world model based on some high-density exteroceptive sensors (e.g., stereo camera or lidar). This approach would require accurate and reliable 3D reconstruction, registration, and segmentation, and these problems have not been solved to the point where they can be fully trusted with the safety of a crewed space vehicle without human supervision. As such, a reactive behavior-based system that uses real-time sensing to gracefully override commands which would result in collisions seems to be a more appropriate solution.

The solution being sought takes the converse approach by using existing or easily accommodated sensors (i.e., cameras), and a more advanced decision-making algorithm. AI-based approaches such as machine learning, pattern recognition, or spatiotemporal reasoning can be used to detect obstacles and monitor proximity to structure. The operation is then halted (or potentially adjusted autonomously) if any unexpected object is found to be in proximity to the arm.

An example use case of the vision system is as follows:

  • A dexterous manipulator is autonomously maneuvering to insert a payload into a berthing location.
  • Mission control, and in particular the robotics personnel who planned the operation, are unaware that a slender antenna from an adjacent site has come loose, and is now floating and partially blocking the approach corridor.
  • As the payload is being moved into position, the vision system estimates that there is insufficient clearance between the payload and the antenna.
  • The autonomous agent halts the motion of the arm, and signals to the ground that the operation cannot continue as planned.

Links (for information only):

Maximum grant value and travel

Multiple contracts could result from this challenge.

Phase 1:

  • The maximum funding available for any Phase 1 contract resulting from this challenge is : $150,000.00 CAD excluding applicable taxes, shipping, travel and living expenses, as required.
  • The maximum duration for any Phase 1 contract resulting from this challenge is up to 6 months (excluding submission of the final report).
  • Estimated number of Phase 1 contracts: 2

Phase 2:

Note: Only eligible businesses that have successfully completed Phase 1 will be invited to submit a proposal for Phase 2.

  • The maximum funding available for any Phase 2 contract resulting from this challenge is : $1,000,000 CAD excluding applicable taxes, shipping, travel and living expenses, as required.
  • The maximum duration for any Phase 2 contract resulting from this challenge is up to 24 months (excluding submission of the final report).
  • Estimated number of Phase 2 contracts: 1

This disclosure is made in good faith and does not commit Canada to award any contract for the total approximate funding. Final decisions on the number of Phase 1 and Phase 2 awards will be made by Canada on the basis of factors such as evaluation results, departmental priorities and availability of funds. Canada reserves the right to make partial awards and to negotiate project scope changes.

Note: Selected companies are eligible to receive one contract per phase per challenge.

Travel

For Phase 1, a successful bidder may need to travel to the Canadian Space Agency (Longueuil, QC) for the Final Review Meeting.

Kick-off meeting

Teleconference/videoconference

Progress review meeting(s)

Teleconference/videoconference

Final review meeting

Longueuil, Quebec (anticipated), or teleconference/videoconference. All other communication can take place by telephone, or videoconference.

Eligibility
 

Solution proposals can only be submitted by a small business that meets all of the following criteria:

  • for profit
  • incorporated in Canada (federally or provincially)
  • 499 or fewer full-time equivalent (FTE) employeesFootnote **
  • research and development activities that take place in Canada
  • 50% or more of its annual wages, salaries and fees are currently paid to employees and contractors who spend the majority of their time working in CanadaFootnote **
  • 50% or more of its FTE employees have Canada as their ordinary place of workFootnote **
  • 50% or more of its senior executives (Vice President and above) have Canada as their principal residenceFootnote **

Evaluation criteria

The official source of the Evaluation Criteria for this challenge is the Government Electronic Tendering System (CanadaBuys) (https://canadabuys.canada.ca/en/tender-opportunities/tender-notice/pw-22-01003385)

In the event of a discrepancy between the information below and the information published on CanadaBuys, CanadaBuys will take precedence.

The Bidder must complete the Challenge Stream Electronic Submission Form with a degree of information sufficient to enable Canada's assessment of the proposal against the criteria and the Evaluation Schema. The information must demonstrate how the proposal meets the criterion.

Part 1: Mandatory Criteria

Proposals must meet all mandatory criteria identified by achieving a "Pass" in order to proceed to Part 2. Proposals that do not meet all mandatory criteria will be deemed non-responsive and given no further consideration.

Mandatory Criteria

(Bidder's proposal must address)

Question 1 a: Scope

Describe the proposed solution and demonstrate how it responds to the challenge. Include in your description the scientific and technological basis upon which the solution is proposed and clearly demonstrate how the solution meets all of the Essential Outcomes (if identified) in the Desired Outcomes section in the Challenge Notice.

Evaluation Schema (Mandatory — Pass/Fail)

Pass

The Bidder's proposed solution is clearly articulated, within the scope for the challenge and addresses all Essential Outcomes (if identified) in the Challenge Notice.

Fail

The proposed solution is articulated as out of scope for the challenge.
OR
The proposal does not clearly demonstrate how the proposed solution addresses all Essential Outcomes listed in the challenge.
OR
The proposed solution is poorly described and does not permit concrete analysis.
OR
There is little to no scientific and/or technological evidence that the proposed solution is likely to meet the challenge.

Question 2: Current Technology Readiness Level (TRL)
  1. Indicate the current TRL of the proposed solution. (Drop Down Menu of the Challenge Stream Electronic Submission Form)
  2. Describe the research and development activities that have taken place to bring the proposed solution to the stated TRL.
Evaluation Schema (Mandatory — Pass/Fail)

Pass

The Bidder has demonstrated that the proposed solution is currently between TRLs 1 and 6 (inclusive), and provided justification by explaining the research and development (R&D) that has taken place to bring the solution to the stated TRL.

Fail

The Bidder has not provided sufficient evidence to demonstrate that the current TRL is between 1 to 6 (inclusive) including:

  1. There is insufficient/no evidence provided for TRL judgment.
  2. The solution involves the development of basic or fundamental research.
  3. The solution is demonstrated at TRL 7 or higher.
  4. Insufficient/unclear/no justification explaining the R&D that took place to bring the solution to the stated TRL.
  5. The explanation simply paraphrases the description of a given TRL level.
Question 3a: Innovation

Demonstrate how the proposed solution meets one or more of the ISC definitions of innovation below:

  1. An invention*, new technology or new process that is not currently available in the marketplace.
  2. Significant modifications to the application of existing technologies/components/processes that are applied in a setting or condition for which current applications are not possible or feasible.
  3. An improvement in functionality, cost or performance over an existing technology/process that is considered state-of-the-art or the current industry best practice.

* An "invention" is defined for the purposes of ISC as: "A manufacturing design or any other new and useful improvement that is new or novel, that is, not commonly known or not an obvious derivative of an existing way of doing things."

Evaluation Schema (Mandatory — Pass/Fail)

Pass

The Bidder has demonstrated that the proposed solution meets one or more of the ISC definitions of innovation.

Fail

The Bidder has not provided sufficient evidence to demonstrate that the current TRL is between 1 to 6 (inclusive) including:

  • Bidder has not provided sufficient evidence to demonstrate that the proposed solution meets any of the ISC definitions of innovation; OR
  • Bidder has demonstrated that the proposed solution is an incremental improvement, "good engineering", or a technology that would go ahead in the normal course of product development (i.e. the next version or release).
Question 3b: Advance on State of the Art

Describe in detail the competitive advantages and level of advancement over existing technologies. Where appropriate, name existing technologies as well as potential substitutes or competitors.

To demonstrate this, proposals should include the following information:

  • Improvements (minor or major) over existing technologies or substitutes. Use direct comparison.
  • How the proposed innovation will create competitive advantages in existing market niches or market spaces.
Evaluation Schema (Mandatory Criteria — Pass/Fail + Points)

0 points/Fail:

  • The Bidder has not demonstrated that the proposed solution advances the state-of-the-art over existing technologies, including available competing solutions; OR
  • The proposed solution improves minimally upon the current state of the art, though not sufficiently enough to create competitive advantages in existing market niches; OR
  • The stated advancements are described in general terms but are not substantiated with specific, measurable evidence.

5 points/Pass:

  • The Bidder has demonstrated that the proposed solution offers one or two minor improvements to existing technologies, including available competing solutions that have potential to create competitive advantages in existing market niches.

12 points/Pass

  • The Bidder has demonstrated that the proposed solution offers three or more minor improvements to existing technologies, including available competing solutions, that together are likely to create competitive advantages in existing market niches; OR
  • The Bidder has demonstrated that the proposed solution offers one significant improvement to existing technologies that is likely to create competitive advantages in existing market niches

20 points/Pass:

  • The Bidder has demonstrated that the proposed solution offers two or more significant improvements to existing technologies, including available competing solutions that are likely to create competitive advantages in existing market niches and could define new market spaces; OR
  • The Bidder has demonstrated that the proposed solution can be considered a new benchmark of state of the art that is clearly ahead of competitors and that is likely to define new market spaces

Part 2: Point-Rated Criteria

Proposals must meet the overall minimum pass mark of 50% to be deemed responsive. Proposals that do not achieve the minimum pass mark will be declared non-responsive and given no further consideration.

Point-Rated Criteria

(Bidder's proposal to address)

Question 1b: Scope

Demonstrate the scientific and technological basis of how the proposed solution addresses the Additional Outcomes(if identified) in the Desired Outcomes section in the Challenge Notice. If no Additional Outcomes are identified in the Challenge Notice, text entered in this section will not be considered.

If no Additional Outcomes are identified in the Challenge Notice, Bidders will receive 10 points.

Evaluation Schema (Point-Rated)
  1. Insufficient or no information provided to demonstrate that the solution will address any of the Additional Outcomes. 0 points
  2. Information provided clearly demonstrates that the solution will address some (<50%) of the Additional Outcomes. 3 points
  3. Information provided clearly demonstrates that the solution will address most (50% or more) of the Additional Outcomes. 6 points
  4. Information provided clearly demonstrates that the solution will address all (100%) of the Additional Outcomes. 10 points
Question 4: Phase 1 Science and Technology (S&T) Risks

Describe potential scientific and/or technological risks to the successful development of the proof of feasibility and how they will be mitigated in Phase 1.

Evaluation Schema (Point-Rated)
  1. Insufficient or no information provided to demonstrate that the Bidder has considered potential risks and mitigation strategies and/or information provided contains significant gaps. 0 points
  2. Information provided demonstrates that the Bidder has considered some potential risks and associated mitigation strategies but there are minor gaps in risks and/or associated mitigation strategies. 5 points
  3. Information provided clearly demonstrates that the Bidder has sufficiently considered the risks and defined associated mitigation strategies. 10 points
Question 5: Phase 1 Project Plan

Demonstrate a feasible Phase 1 project plan by completing the table.

  • Indicate if any milestones and activities will be completed concurrently
  • Indicate the estimated exit TRL at the completion of Phase 1. (Drop Down Menu of the Challenge Stream Electronic Submission Form)
Evaluation Schema (Point-Rated)
  1. Insufficient or no information provided to demonstrate a feasible project plan for Phase 1 and/or the project plan exceeds the maximum duration indicated in the Challenge Notice. 0 points
  2. Project plan for Phase 1 is conceivably feasible but not clearly demonstrated and/or includes gaps. 10 points
  3. Information provided clearly demonstrates a feasible project plan for Phase 1. 20 points
Question 6: Phase 1 Project Risks

Describe potential project risks to the successful development of the proof of feasibility and how they will be mitigated in Phase 1.

Bidders should address the following risks, as applicable:

  • Human Resources
  • Financial
  • Project Management
  • Intellectual Property
  • Other project-related risks

Note to Bidders: S&T risks should not be included in this section. Question 4 addresses S&T risks.

Evaluation Schema (Point-Rated)
  1. Insufficient or no information provided to demonstrate that the Bidder has considered potential risks and mitigation strategies and/or information provided contains significant gaps. 0 points
  2. Information provided demonstrates that the Bidder has considered some potential risks and associated mitigation strategies but there are minor gaps in risks and/or associated mitigation strategies. 5 points
  3. Information provided clearly demonstrates that the Bidder has sufficiently considered the risks and defined associated mitigation strategies. 10 points
Question 7: Phase 1 Implementation Team

Demonstrate how the project implementation team has the required management and technological skill sets and experience to deliver the project plan for Phase 1 by completing the table. A member of the implementation team can have more than one role.

Evaluation Schema (Point-Rated)
  1. Insufficient or no information provided to demonstrate that the project team has the required management and technological skill sets and experience to deliver the Phase 1 project plan. 0 points
  2. Information is provided but there are minor gaps in required management and/or technological skill sets and/or experience to deliver the Phase 1 project plan. 10 points
  3. Information provided clearly demonstrates that the project team has the required management and technological skill sets and experience to deliver the Phase 1 project plan. 20 points
Question 8: Inclusivity

If your business were to receive funding from Innovative Solutions Canada, describe what actions (e.g., recruitment strategy, internships, co-op placements, etc.) might be taken in Phase 1 to support the participation of under-represented groups (e.g., women, youth, persons with disabilities, Indigenous people, visible minorities) in the research and development of the proposed solution. Each Bidder in their response to this question must focus only on describing relevant programs, policies, or initiatives that it currently has in place or would put in place to support the R&D effort in Phase 1.

Note: Do not provide any personal information of individuals employed by your company or that of your subcontractors in the response.

Evaluation Schema (Point-Rated)
  1. No description and/or concrete examples of actions provided that would be taken to encourage greater participation of under-represented groups. 0 points
  2. A description and concrete examples of actions to encourage greater participation of under-represented groups provided. 5 points
Question 9: Phase 1 Financial Proposal

Demonstrate a realistic financial proposal for the Phase 1 project plan by completing the table.

Evaluation Schema (Point-Rated)
  1. Insufficient information provided and/or information provided significantly lack credibility. Does not demonstrate a realistic financial proposal for the Phase 1 project plan. 0 points
  2. Information is provided but some costs lack credibility and/or are unclear for the Phase 1 project plan. 7.5 points
  3. Information provided contains credible elements to clearly demonstrate a realistic financial proposal for the Phase 1 project plan. 15 points
Question 10: Phase 1 Financial Controls, Tracking and Oversight

Describe the financial controls, tracking and oversight that will be used to manage the public funds throughout Phase 1. Bidders should indicate if an individual or firm will be managing the public funds and provide their credentials and/or relevant experience.

Evaluation Schema (Point-Rated)
  1. Insufficient or no information provided to demonstrate the Bidder's ability to manage public funds in Phase 1. 0 points
  2. Information provided is vague and/or contains gaps. The Bidder has some controls, tracking and/or oversight in place to manage the public funds in Phase 1. 5 points
  3. Information provided clearly demonstrates that the Bidder has strong financial controls, tracking and oversight to manage public funds in Phase 1. 10 points
Question 11: Phase 2 Overview

Demonstrate a realistic overview for the prototype development plan if selected to participate in Phase 2.

Responses should include:

  • key tasks
  • estimated cost for materials
  • human resources
  • project risks and mitigation strategies

Note: A more detailed proposal will be requested if selected to participate in Phase 2.

Evaluation Schema (Point-Rated)
  1. Insufficient or no information provided to demonstrate that the Bidder has contemplated a realistic overview for the Phase 2 prototype development. 0 points
  2. Information provided demonstrates a conceivably realistic overview for Phase 2 prototype development, however there are gaps and/or the strategy is vague. 6 points
  3. Information provided demonstrates that the Bidder has a clear and realistic overview. 12 points
Question 12: Commercialization Approach

Demonstrate a realistic overall commercialization approach/business model that can successfully take the technology/service to market, and how the technology/service will help you develop and sell other products.

Responses should include:

  • Target markets (excluding Government of Canada)
  • Non-ISC funding sources
  • Transition to a commercially-ready product or service
  • Any other indicators of commercial potential and commercial feasibility

Note: A more detailed proposal will be requested if selected to participate in Phase 2 or the Testing Stream.

Evaluation Schema (Point-Rated)
  1. Insufficient or no information provided to demonstrate that the proposed solution has commercial potential. 0 points
  2. Some information provided to demonstrate that the proposed solution has commercial potential, however there are gaps in the commercialization approach. 6 points
  3. A realistic commercialization approach is provided that demonstrates that the proposed solution has commercial potential. 12 points
Question 13: Resulting Benefits to Canada

Describe the benefits that could result from the commercialization of the proposed solution. Bidders should consider the potential benefits using the following three categories and provide justification for each claim:

  1. Innovation Benefits: Expected contribution towards the enhancement or development of new industrial or technological innovations within your firm. Responses could include: potential spillover benefits, creation of intellectual property, impact on productivity of the new technology, etc.
  2. Economic Benefits: Forecasted impact on the growth of Canadian firms, clusters and supply chains, as well as its expected benefits for Canada's workforce. Responses could include: number of jobs created, number of high-paying jobs, investment in Canada's economy, etc.
  3. Public Benefits: Expected contribution to the broader public to the degree that the solution is expected to generate social, environmental, health, security or other benefits to Canada. Responses could include: solution-related environmental benefits, solution-related accessibility benefits, and solution-related impact on Indigenous communities.
Evaluation Schema (Point-Rated)
  1. Innovation Benefits
    Benefit not identified or insufficient claim of benefit. 0 points
    Benefit has marginal increment or limited justification. 1 point
    Benefit is significant and well justified. 2 points
  2. Economic Benefits
    Benefit not identified or insufficient claim of benefit. 0 points
    Benefit has marginal increment or limited justification. 1 point
    Benefit is significant and well justified. 2 points
  3. Public Benefits
    Benefit not identified or insufficient claim of benefit. 0 points
    Benefit has marginal increment or limited justification. 1 point
    Benefit is significant and well justified. 2 points

Questions and answers

Please refer to the tender notice for this challenge on CanadaBuys.

All incoming questions regarding this specific challenge should be addressed to SIC-ISC@pwgsc.gc.ca

You can also consult the Frequently asked questions about the Innovative Solutions Canada Program.

A glossary is also available.