Evaluation of the Strategic Aerospace and Defence Initiative: Final report

January 2012

Recommended for approval to the Deputy Minister by the Departmental Evaluation Committee on January 25, 2012

Approved by the Deputy Minister on February 7, 2012

Appendices (Separate document)
Appendices are available via an Access to Information

Acronyms, Abbreviations and Definitions used in this Report

List of Tables

List of Figures

Executive Summary

Program Overview

The Strategic Aerospace and Defence Initiative (SADI) provides repayable contributions for industrial research and pre-competitive development to aerospace, defence, space and industrial security (A&D) firms. The objectives of the program are to encourage strategic research and development that will result in innovation and excellence in new products and services; enhance the competitiveness of Canadian A&D companies; and foster collaboration between post-secondary institutions and the private sector. Since April 2007, the program has approved over $824 million in repayable contributions to 22 recipients for 24 projects.

SADI is managed and delivered by the Industrial Technologies Office (ITO) which is housed in Industry Canada's Science and Innovation Sector.

The 2011 Budget announced the launch of a comprehensive review of all policies and programs related to the aerospace/space industry. During this 12-18 month consultative review, the Government of Canada committed to ensuring that stable funding is provided to SADI.

Evaluation Purpose and Methodology

In accordance with the Policy on Evaluation and the Directive on the Evaluation Function, the purpose of this evaluation was to assess the core issues of relevance and performance of SADI.

The evaluation findings and conclusions are based on the analysis of multiple lines of evidence. The methodology included a review of documents, an environmental scan and literature review, interviews, case studies, a focus group of small and medium-sized enterprises (SMEs), and a survey.

Findings

Relevance

The evaluation found that SADI addresses a demonstrable need for research and development (R&D) financing for aerospace firms and for high-tech SMEs. Within the aerospace sector, R&D investments are considered high risk because of the long product development cycles. For high-tech SMEs, many firms are considered high risk from a financial perspective and lack viable funding options for their R&D projects.

To date, SADI has reached a relatively small proportion of the aerospace and defence sector. Part of the limited uptake can be attributed to the global economic recession. The business aircraft and general aviation subsectors were particularly hard hit. In addition, various elements of the program design were noted as some of the reasons why firms did not apply. For SMEs, limited awareness of the program was identified as a key factor.

Overall, SADI is aligned with the priorities of the federal government as outlined in Speeches from the Throne, Budgets, and the Science and Technology Strategy. SADI is also consistent with Industry Canada's strategic outcomes. In terms of alignment with the federal government's role and responsibilities, SADI is consistent with the overall federal responsibility of increasing competitiveness and encouraging the development of science and technology. Responsibility for economic development is shared with the provinces. In this context, SADI plays a complementary role with other programs because of its size and national scope.

Performance

In terms of its impact on R&D spending, some recipients increased their total R&D investments because of SADI. However, the most common benefit from SADI was its role in accelerating the timing of their R&D projects. In addition, some recipients reported that SADI repayable contributions allowed them to broaden the scope of their projects, take on riskier projects, and develop new areas of focus.

There has been a renewed focus within the program on fostering collaboration. Close to half of the recipients interviewed reported an increase in collaboration with post-secondary institutions that could be attributed to SADI. Most of these collaborations involved hiring more students, but there were also some research collaborations.

SADI's intermediate outcomes include the development and commercialization of innovative products, processes and services. These outcomes are expected to begin occurring five years after the launch of each project. Given that the program was in its fourth year at the time of the evaluation, the full results will be clearer when SADI projects reach the repayment phase. Nevertheless, more than half of the recipients have already developed products, processes and services, and some are beginning to reach the point of commercialization.

Overall, there were no significant unintended negative impacts or outcomes identified for SADI recipients. However, individual recipients experienced positive benefits from their SADI projects that they had not anticipated. These benefits included new areas of growth, cultural changes, and enhanced reputations as leaders in the marketplace.

In terms of the efficiency of the program, the evaluation determined that while SADI is largely meeting its service standard for most applications, the end-to-end application process itself was perceived as onerous and lengthy by A&D firms. The overall efficiency of ITO in delivering the program improved consistently year over year.

Recommendations

1. ITO should consider identifying additional ways to streamline the end-to-end application process;
2. ITO should consider examining whether changes in program design are required to improve uptake, particularly to attract SMEs and enhance collaboration; and
3. ITO should consider building greater awareness of the program, especially among SMEs.

1.0 Introduction

This evaluation was undertaken to assess the relevance and performance of the Strategic Aerospace and Defence Initiative (SADI) in accordance with the Policy on Evaluation. This section of the report describes the program context and objectives. The remainder of this report is organized as follows:

• Section 2 presents the evaluation methodology;
• Section 3 provides the key findings related to the evaluation issues of relevance and performance; and
• Section 4 summarizes the study's conclusions and provides recommendations.

1.1 Program Context

By investing in research and development (R&D) projects, firms can achieve technological breakthroughs and increase their competitiveness, which, in turn, contributes to economic, technological and other benefits for Canadians. The level of R&D investment in the private sector is a concern because Canadian business expenditures on research and development (BERD) as a percentage of Gross Domestic Product are lower than what is found in roughly two-thirds of other developed countries (including the United States and other G7 countries).^{Footnote 1},^{Footnote 2},^{Footnote 3} This is important because BERD is believed to affect the long-run rate of productivity growth in the economy. Consequently, the Government of Canada has a number of programs and initiatives that seek to encourage the private sector to invest more in research and development.

In this context, the aerospace, defence, space and industrial security (A&D) industries play a key role in driving innovation. A&D firms are often early users of innovative technology including composite materials, information and communications technologies, nanotechnology, and advanced manufacturing techniques. In addition, Canadian aerospace firms are currently leaders in several global markets, including those for simulators, landing gears and aircraft environmental systems.

1.2 Program Overview

Over the years, the Government of Canada has played an active role in providing support to the A&D industries for research and development. Programming has evolved from the Defence Industrial Productivity Program to Technology Partnerships Canada (TPC) to the current Strategic Aerospace and Defence Initiative.

SADI is the largest contribution program at Industry Canada. The program provides support for industrial research and pre-competitive development.^{Footnote 4}

The program was launched in April 2007 and has three objectives:

• Encouraging strategic research and development that will result in innovation and excellence in new products and services;
• Enhancing the competitiveness of Canadian A&D companies; and
• Fostering collaboration between research institutes, universities, colleges, and the private sector.

Initially, proposals from applicants were required to identify the benefits to Canada in two of the following three areas: commercialization, technology spin-off or diffusion, and collaborative partnerships. Over time, the program has placed more emphasis on collaboration with post-secondary institutions. Appendix B outlines the current definition of collaboration and presents examples of collaborative activities that would be considered favourably by the program.

1.2.1 Program Resources and Repayment Terms

SADI is managed and delivered by Industry Canada's Industrial Technologies Office (ITO). ITO is led by an Executive Director and is made up of five directorates that manage various aspects of ITO's two main programs, SADI and TPC. SADI and TPC funding are tied because SADI is funded from amounts previously allocated to the TPC program, including repayments. TPC was closed to new applications in December 2006, but previously approved projects are long-term and funding is required over several years for eligible expenses.

Annually ITO is allocated resources from the department and from appropriations made available through the Estimates process. This allocation includes operating resources for the management of SADI and TPC, and resources to fund the repayable contributions made under both programs. In the first years of SADI, the allocated resources were mostly dedicated to the ongoing management of the legacy TPC program and disbursements of TPC commitments. Over time and with the declining number of TPC commitments and increased demand for SADI, resources have shifted to SADI. In addition to the base appropriation for repayable contributions, SADI also has authority to access repayments made by TPC and SADI projects.

SADI can also seek authority to re-profile funds in any given year up to 20% of the total program annual funding level and to a maximum of $60 million, with appropriate justification and subject to affordability constraints. R&D projects typically last up to five years and are sensitive to changing economic conditions. Re-profiling provides SADI with the flexibility to meet fluctuations in the demand for R&D financing; fund large, multiyear projects with uneven cash flow requirements; and, respond to cash flow fluctuations which can be significant given project sizes.

In June 2009, SADI received an additional $200 million over four years (2010-11 to 2013-14) for repayable contributions. The following table provides the estimated operating expenses and SADI repayable contributions since the outset of the program.^{Footnote 5}

There are two different types of repayment terms for SADI projects. If repayment is unconditional, then there is a defined payment schedule. If repayment is conditional, then repayment is based on the recipient's gross business revenues. In both cases, repayment starts after the end of the R&D phase and covers 15 years.

1.2.2 SADI Recipients

Since April 2007, the program has approved over $824 million in repayable contributions to 22 firms for 24 projects. Table 2 identifies the recipients, the authorized amounts of repayable contributions, and the date when the Contribution Agreement was finalized.

As of December 2011, small firms made up 41% of the recipients and medium-sized firms made up another 27%.^{Footnote 6} Large firms accounted for the remaining 32%. With respect to contribution amounts, repayable contributions to small, medium and large firms represent 5%, 10%, and 85% of the total respectively.

On average, SADI has authorized repayable contributions equal to 32.3% of total eligible project costs as a percentage of the SADI portfolio. In terms of actual disbursements of funds, as of March 2011, SADI has provided $223 million in repayable contributions since the inception of the program. These SADI repayable contributions have been matched by $448 million in funding by recipients. Thus, $2.01 of R&D investment had been leveraged per SADI dollar invested.

1.2.3 The Proposal Process

Proposals to the program are accepted on an ongoing basis and projects are assessed based on whether they involve strategic R&D activities. These involve one or more technologies that:

• Support the development of next generation A&D related products and services;
• Build on Canadian strengths in A&D technology development;
• Enable Canadian companies to participate in major platforms and supply chains; or
• Assist the sector in achieving Canada's international obligations such as the Joint Strike Fighter program (JSF).

Under the current process, there are four distinct phases:^{Footnote 7}

1. Pre-Application: An applicant typically performs a self-assessment of its eligibility against the program criteria. The applicant can submit a draft proposal for review and feedback by the program office. This phase may proceed rapidly or may take several months before an applicant submits a formal proposal.
2. Suitability Assessment: When a proposal is received, it is initially checked to ensure that the applicant is eligible (e.g. incorporated in Canada) and that the proposal meets the minimum eligibility criteria for approval. This phase includes reviewing whether the applicant has the necessary financial resources, management expertise, technical expertise, and a solid business plan to achieve the forecasted benefits. The proposal must show that significant technological benefits will result from the work and that social and economic benefits will arise.
3. Due Diligence: Program officers assess the information presented in the proposal. Specifically, they verify the financial, resource, managerial capability, business plan, and technical information provided in the proposal. They also examine the degree of technological innovation, the feasibility of the research and development, and the social and economic benefits and any other information provided in the proposal. SADI officers will conduct a site visit and may seek the advice of experts within Industry Canada, other departments, and external experts. The program office also negotiates repayment terms and finalizes the terms of the Contribution Agreement.
4. Approval and Contracting: This phase begins with the recommendation for approval by the program office. Projects in excess of $2 million must also be reviewed by the Programs and Services Board.^{Footnote 8} Once approved, the Contribution Agreement is finalized and sent to the applicant. Treasury Board approval is required for repayable contributions in excess of $10 million and both Treasury Board and Cabinet approvals are required for repayable contributions in excess of $20 million.

1.2.4 Expected Results

According to the program design, it was anticipated that repayable contributions provided through SADI would encourage recipients to undertake strategic R&D activities and help leverage other private sector and government investments. The launch of SADI projects would in turn create R&D opportunities for recipients to engage research institutions, universities, and colleges in collaborative relationships. These SADI projects would lead to the transformation of ideas, technology and expertise into marketable products, processes and services. It was anticipated that SADI projects would result in commercialization and this would contribute to the increased competitiveness of Canadian A&D firms. In this way, support to recipients in the A&D industry would ultimately contribute to the achievement of broader technological, economic, environmental and social benefits for Canadians.

The following logic model for SADI was developed in 2007 and updated as part of the program's ongoing performance measurement strategy (April 2011). The logic model outlines the program's inputs, activities, and outputs, as well as the intended short-term, intermediate, and longer-term outcomes. In terms of timeframes, immediate outcomes are expected to occur during the R&D Phase of a SADI project, which lasts five years. Intermediate outcomes are expected to occur when SADI projects are in the Repayments Phase. This period lasts for up to 15 years and begins after the completion of the R&D Phase.

Figure 1: Logic Model of SADI

[View full size version]

2.0 Evaluation Methodology

This section describes the overall approach of the evaluation, the objective and scope, the specific evaluation issues and questions that were addressed, the data collection methods, and data limitations.

2.1 Evaluation Approach

The evaluation study was conducted by Industry Canada's Audit and Evaluation Branch (AEB). Two components were completed through third party contracts. Specifically, Science-Metrix completed a literature review and an environmental scan, and EKOS Research Associates conducted a survey of A&D firms.

2.2 Objective and Scope

In accordance with the Policy on Evaluation and the Directive on the Evaluation Function, the purpose of this evaluation was to assess the core issues of relevance and performance of SADI.

Given that SADI-funded projects are still in their early stages and that the full extent of the program's success in achieving its intermediate outcomes will only be known in a few years, the evaluation focused on the achievement of immediate outcomes and intermediate outcomes where possible.

2.3 Evaluation Issues and Questions

Based on the SADI logic model, the SADI performance measurement strategy of April 2011, and consultations with the program, the evaluation addressed the following questions:

Relevance

• To what extent does SADI address a demonstrable need?
• To what extent is SADI reaching its target groups?
• To what extent do the objectives of SADI align with priorities of the federal government and strategic outcomes of Industry Canada?
• To what extent does SADI align with the federal government's role and responsibilities in supporting research and development?

Performance

• To what extent have SADI recipients increased their investments in strategic R&D as a result of SADI repayable contributions?
• To what extent has collaboration between recipients and post-secondary institutions (universities, colleges and research institutes) increased as a result of SADI?
• Recognizing that results may take years to occur given the nature of the program, to what extent have SADI-supported R&D activities increased (or will be likely to increase) the development and commercialization of innovative products, processes and services?
• Has SADI resulted in any unintended impacts or outcomes?
• How efficient is ITO in the delivery of the SADI program?

Appendix C contains the SADI Evaluation Matrix, which includes the indicators, data sources and data collection methods for each of the evaluation questions.

2.4 Data Collection Methods

Multiple lines of evidence, along with the triangulation of data, were used where possible to address all of the evaluation questions. The data collection methods included a review of documents, an environmental scan and literature review, interviews, case studies, a focus group, and a survey.

Document Review

A large number of documents were reviewed by the evaluation team. Policy documents included the Government of Canada's Science and Technology Strategy, Federal Budgets, and Speeches from the Throne. Departmental documents included Industry Canada's Program Activity Architecture and the Business Plan. Program documentation included program files and data, financial and operational data, proposals, Contribution Agreements, the SADI Annual Program Benefits Report, ITO report cards, Annual Project Benefit Reports, and briefing notes. Finally, industry documents were reviewed to situate SADI within the larger A&D context. They included industry reports, white papers, evaluations, and submissions to the Review of Federal Support to Research and Development by industry associations and by individual firms.

Environmental Scan and Literature Review

The environmental scan identified government programs around the world that provide direct support to firms for research and development in the aeronautics and defence sectors. This included a comparative analysis of the strengths and weaknesses of these programs, and the identification of leading practices. The scope of the environmental scan included Europe, Japan and the United States.

The literature review examined best practices for performance metrics for R&D collaboration between universities and private sector firms. The literature review also covered the drivers for collaboration, the types of collaboration, and ways to measure the success of collaborations.

Interviews

A total of 48 interviews were conducted with ITO executives, managers and staff (7), Industry Sector executives and staff (5), SADI recipients (19), applicants (2), other A&D firms (11), industry associations (3), and the National Research Council's Industrial Research Assistance Program (1).

The list of individuals interviewed and the interview guides are presented in Appendix D.

Case Studies

ITO initiated a series of case studies in early 2009, as part of the program's overall performance measurement strategy. The purpose of the case studies was to describe in detail the extent to which selected SADI projects had contributed to the three main objectives of innovation, competitiveness and collaboration. The initial case studies were the first projects contracted by the program. The remaining recipients were chosen to reflect a broad coverage of recipients (large versus small, location, etc). Updates to the case studies were conducted by AEB as part of the evaluation for eight active SADI projects.

Focus Group

Given that most of the interviews with non-SADI firms included large firms and there was a heavy representation by aerospace firms, a focus group was conducted to identify some of the barriers to uptake for SMEs in the defence sector. The participants represented seven defence SMEs from across southern Ontario.

The list of participants and the discussion questions are presented in Appendix E.

Survey

A telephone survey was conducted in order to validate findings from interviews and develop a deeper understanding of non-recipients and the most significant barriers to increasing investment in research and development. A list of A&D firms was generated by sector experts and provided to AEB by the program. This list included firms that conduct research and development and would have been eligible to apply to the program.

In total, 99 firms were contacted. There were 36 completed responses to the survey, which included 19 small firms (i.e., less than 100 employees), 12 medium-sized firms (i.e., between 100 and 499 employees), and five large firms (i.e., more than 500 employees).

The survey frame and the survey instrument are presented in Appendix F.

2.5 Data Limitations

The following were the main data limitations:

1. The Lack of Information on the Scope of the Target Market for the Program

An initial document review and literature scan revealed that there is no common and accepted definition of A&D firms in Canada. The absence of a common and accepted definition of the relevant A&D firms made it more difficult to determine the relevance of the program in addressing the R&D needs of A&D firms, as well as the extent to which the program has reached its target group. To address this challenge, the evaluation used a list of A&D firms that had been generated by sector experts within Industry Canada. It is believed that all of the firms on this list conduct research and development and would be eligible to apply to the program.

2. The Timing of the Evaluation

SADI's intermediate outcomes are the development and commercialization of innovative products, processes and services. These outcomes are expected to occur five years after the launch of the program. Given that the program was in its fourth year at the time of the evaluation, there was not enough data available to assess the value-for-money for each dollar invested in SADI programming in terms of the intermediate outcomes. In addition, the global recession slowed down progress towards the intermediate outcomes. Given that programs are evaluated on a five year cycle, there should be sufficient data to examine the cost-effectiveness of the program at the time of the next evaluation.

3. The Confidential Nature of R&D Information

The evaluation sought to determine the extent to which there had been a net increase in R&D investment by SADI recipients. However, R&D investment information is considered confidential by A&D firms for competitive reasons. This meant that the evaluation relied on self-reported information to determine the net increase in R&D expenditures attributable to SADI and the extent to which recipients developed and commercialized products, processes and services from their SADI projects. Similarly, the confidential nature of R&D investment information meant that it was not possible to compare the effects of SADI on recipients with the performance of equivalent A&D firms that had not received repayable contributions from the program.

3.0 Constatations

3.1 Relevance

3.1.1 To what extent does SADI address a demonstrable need?

Within the aerospace sector, many R&D investments are considered high risk because it can take 10-15 years for a new aircraft to move from conception to test flight to sales.^{Footnote 9} During this period, Canadian aerospace suppliers have to make R&D investments up front and over a relatively short period of time (e.g. 2-3 years). If firms are successful and are selected by the original equipment manufacturers (OEMs) like Boeing or Airbus, then their products will be locked into the development of the new aircraft program and it will be very difficult for a competitor to dislodge them because of the need to meet airworthiness certification requirements.

The challenge is that the revenues from these R&D investments will only occur when the OEM begins booking orders for the aircraft and will only generate cash flows once the aircraft is in production. This means that the aerospace supplier may not see any returns from its R&D investments for a decade or even longer. However, once the aircraft is in production, these R&D investments can prove to be very profitable because an aircraft part may be sold in the marketplace for thirty years.

The long product development cycles in aerospace make R&D projects extremely risky and particularly sensitive to economic downturns that affect aircraft sales. Consequently, private sector sources of financing are not very receptive to accepting these kinds of risks. As a result, the level of R&D expenditures would be lower in the absence of government support.

The general challenge of obtaining funding is accentuated by the fact that the aerospace sector is in the midst of a tremendous change in use of materials in aircraft.^{Footnote 10} Canadian aerospace firms have to adjust to new technology demands, which present both opportunities and threats. In particular, the use of steel in aircraft is being replaced by composite materials for aircraft frames. While these new composites offer the promise of lighter aircraft, greater fuel efficiency, and greater resistance to corrosion, they also require substantial new R&D investments to develop knowledge of the best practices for their manufacture and maintenance.

Within A&D industries, high-tech SMEs face additional challenges in obtaining funding to support their R&D investments. According to an Industry Canada white paper, SMEs with high R&D expenditures have greater external financing needs than other SMEs.^{Footnote 11} These SMEs are also more likely to be turned down by financial institutions because they represent a higher risk than their peers who have lower levels of expenditures on research and development. Within the A&D industries, interviews and document reviews also found evidence that OEMs and their key suppliers have been more hesitant to help SMEs with financing than they have been in the past. This situation was confirmed in the survey. About 81% of SMEs in the survey identified access to funding as a significant barrier to increasing their investments in research and development.

The long product development cycles and the limited funding alternatives for SMEs may be reasons why so many foreign governments support their aerospace and defence sectors. Beside national security concerns, foreign governments support their A&D industries because these sectors produce high paying, high technology jobs, and create the potential for spin-offs for their economies.

There is a pervasive presence of international government support for research and development in the A&D industries.^{Footnote 12} There are dozens of aerospace programs and defence programs in Europe, Japan and the United States that provide direct support for research and development.^{Footnote 13} Many of these programs extend beyond basic research and into industrial research and pre-competitive development. They include:

• At the European Union level, there is the Clean Sky Joint Technology Initiative.
• At the national level in Europe, there are programs like Germany's Federal Aeronautical Research Program that provides grants. The United Kingdom has the National Aerospace Technology Strategy and the Netherlands has the Civil Aircraft Development Program. Within Europe, Austria, France, Italy, Spain and Sweden provide various combinations of grants, loans, and contracts to support their national aerospace industries.
• In the United States, the government supports research and development through defence programs and government contracts.

Despite the many differences among A&D firms, all firms that were interviewed reported that because of the extensive programming in other countries, SADI was needed to remain competitive.

3.1.2 To what extent is SADI reaching its target groups?

The potential reach for SADI could be quite large. In 2007, the Aerospace, Defence and Marine Branch of Industry Canada identified 4,129 firms for a survey of defence, aerospace, and industrial security firms. The survey found that about 54% of the A&D firms conducted at least some research and development. While not all of these firms would have R&D projects that would meet the requirements for SADI, the large size of these industries suggests that hundreds of firms could, in theory, apply to the program. A narrower measure of the potential reach of the program would be the list of A&D firms that were contacted for the survey conducted for the evaluation.

It is clear that the actual reach of the program to date has been considerably smaller than its potential reach. As of December 2011, there were 24 SADI projects with 22 firms. In terms of the breakdown by industry, six recipients are in the aerospace sector; four are in the defence sector; and eight recipients operate in both sectors. Two recipients are in the industrial security sector and two recipients operate exclusively within the space component of the aerospace sector.

The relatively low level of uptake among A&D firms for SADI compared to the potential reach of the program is reflected by the fact that there has been a surplus of SADI repayable contributions in every year of the program since inception. The annual surplus has ranged from $38 million to $123 million.

In part, this can be explained by the global economic recession. According to a report by the Standing Committee on Industry, Science and Technology, the global recession hit the Canadian aerospace industry hard as its customers were predominantly foreigner companies that faced the full force of the financial crisis and global recession. The aerospace industry was forced to cut employment levels and manage costs more thoroughly.^{Footnote 14}

These observations are consistent with available economic data. According to the Conference Board of Canada, the aerospace product manufacturing industry experienced a growth in revenues of 15.6% in 2008 followed by a drop of 14.6% in 2009 and another drop of 12.3% in 2010.^{Footnote 15} Real aerospace GDP fell by 7.7% in 2009 and 6.29% in 2010. Compared to all manufacturing industries, GDP fell by 12.4% in 2009 and rose by 5% in 2010. Aerospace employment fell by 5.2% in 2010, while employment in all manufacturing industries fell by 1.4% over the same period. Two subsectors of the industry were particularly hard hit. The business aircraft and general aviation subsectors both experienced a 53% decline in shipments between 2007 and 2010.^{Footnote 16}

The evaluation team also interviewed A&D firms and included questions in the survey of A&D firms on the reasons why firms had not applied to the program. While there was often more than one factor that contributed to the decision, they emphasized the following:

• Other Funding Sources: Firms noted that there were more attractive funding sources available (8 of 20 survey responses and 6 of 11 interviews with key non-recipients). Key sources identified included the use of Scientific Research and Experimental Development (SR&ED) tax credits, cash flows from internal operations, and regional development programs (e.g. Atlantic Innovation Fund).
  
  SADI's value proposition has been an area of study by the program. Overall, the findings drawn from these internal studies conclude that SADI's value proposition may be lowest for small, profitable, Canadian-controlled private corporations with relatively small SADI project proposals.
• The Level of Required Documentation: According to some A&D firms, the work required to complete an application (5 of 20 survey responses) and comply with reporting requirements (5 of 20 survey responses) seemed onerous for a contribution that is repayable. In interviews with recipients, SMEs believed that SADI was designed more for larger firms than for smaller firms. Specifically, they believed that most SMEs did not have sufficient internal resources to manage the application process; handle the reporting requirements; and deal with contract amendments.
• Application Processing Times: According to some firms, the time to complete the application process was too long (4 of 20 survey responses). All of the large firms (i.e. with 500 or more than employees) that responded to this part of the survey identified this as a contributing factor in their decision not to apply.
• Intellectual Property Requirements: As set out in the Contribution Agreement template, recipients must agree: "… not to transfer title to, nor grant any right to, the project intellectual property, except in end-user licence in conjunction with the sale of resulting products, without the prior written consent of the Minister."
  
  In the survey, intellectual property restrictions did not appear as a significant barrier to uptake (1 of 20 survey responses). However, according to the interviews with non-recipients, four large Canadian subsidiaries of multinationals chose not to apply to SADI in part because they believed the treatment of intellectual property was too restrictive. These A&D firms said that they need to transfer rights to any intellectual property developed in Canada throughout their global organization and throughout their global supply chains.
• Manufacturing Requirements: The requirement to manufacture in Canada was an issue for some firms (3 of 20 survey responses). In the interviews, two recipients said that the high cost of the Canadian dollar had made manufacturing in Canada difficult. Three recipients reported that they reduced their costs by sourcing components from abroad but they continued to produce the final product in Canada. One applicant was very concerned about the manufacturing requirements because its product requires ultra high strength steel and there are no Canadian suppliers at present.
  
  In practice, firms have obtained intellectual property and manufacturing exceptions from the department in cases where less restrictive requirements are expected to result in greater net benefits to Canada.
• Accounting Considerations: In the survey, some non-recipients (6 of 20 survey responses) identified the accounting treatment of repayable contributions as a factor in their decision not to apply to the program.

While these factors were identified as impediments to uptake, primarily by non-recipients, it should be noted that the ultimate outcome for the program is to contribute to the achievement of broader technological, economic, environmental and social benefits for Canadians. As such, a balance needs to be achieved between producing benefits for individual A&D firms and achieving broader program objectives.

In order to learn more about the experiences of the smaller firms, AEB also conducted a focus group with seven SMEs in the defence sector in southern Ontario. According to the focus group, the main reason for not applying to SADI was a lack of awareness of the program. Focus group participants were also concerned about the amount of paperwork and the length of time to obtain approval for repayable contributions.

A review of the detailed survey data confirmed that there was a lack of awareness regarding SADI among SMEs. About 26% (8 of 31) of the SMEs who were contacted for the survey were not aware of the program.

3.1.3 To what extent do the objectives of SADI align with priorities of the federal government and strategic outcomes of Industry Canada?

The program's objectives are consistent with Speeches from the Throne, Budgets, and other key documents. The following table contains some excerpts from key documents that demonstrate the importance that the Government of Canada places on research and innovation as well as the government's continued commitment to economic growth.

In Budget 2011, SADI was specifically mentioned as part of the larger review of aerospace policies and programs. The review will examine policies and programs related to the aerospace industry in order to develop a federal policy framework to maximize the competitiveness of this sector.

SADI's objectives are also consistent with the Government of Canada's Science and Technology (S&T) Strategy. According to the S&T Strategy, "The Government of Canada will ensure that its policies and programs inspire and assist Canadians to perform at world-class levels of scientific and technological excellence."^{Footnote 17} Specifically, SADI is consistent with the S&T Strategy's focus on fostering an entrepreneurial advantage and a people advantage. The entrepreneurial advantage aims at fostering a competitive and dynamic business environment, pursuing public-private research and commercialization partnerships, and increasing the impact of federal business R&D assistance programs. The people advantage includes increasing the supply of highly qualified and globally connected science and technology graduates.

Finally, SADI's objectives are consistent with the strategic outcomes of Industry Canada. In particular, SADI is directly aligned with Industry Canada's strategic outcome that "advancements in science and technology, knowledge, and innovation strengthen the Canadian economy".^{Footnote 18} In addition, SADI contributes to the department's strategic outcome that "Canadian businesses are competitive". In this sense, SADI also supports the department's mission of fostering a knowledge-based competitive economy.

3.1.4 To what extent does SADI align with the federal government's role and responsibilities in supporting R&D?

Within the federal domain, the objectives of the program fall under the Department of Industry Act of 1995. According to this legislation, the powers, duties and functions of the Minister of Industry extend to matters relating to "industry and technology in Canada".^{Footnote 19} Specifically, the objectives are to "… increase the international competitiveness of Canadian industry, goods and services…" and "encourage the fullest and most efficient and effective development and use of science and technology".^{Footnote 20}

To achieve these objectives, there are a number of programs within the Government of Canada that provide support to the A&D industries. They tend to be directed towards earlier stages of research and development than SADI. The Canada Foundation for Innovation, Natural Sciences and Engineering Research Council of Council, and the Canadian Space Agency generally support basic and applied research. Similarly, the Institute for Aerospace Research at the National Research Council has traditionally focused on technology development and demonstration. Like SADI, the National Research Council's Industrial Research Assistance Program (IRAP) and Defence Research and Development Canada's Defence Industrial Research Program (DIRP) can focus on technology development rather than research, but they typically focus on technology that is less ready for product development than SADI.

In theory, some IRAP and DIRP projects may unintentionally result in some overlap with SADI projects because their funded research and development may achieve a higher level of readiness for commercialization than originally anticipated. However, SADI differs profoundly from IRAP and DIRP in terms of its scale. There is no maximum for the total amount of repayable contributions that can be authorized for a SADI project. The time frame also differs from IRAP and DIRP in that SADI's repayment period is fifteen years. In contrast, IRAP provides non-repayable contributions but there is a limit of $1 million per project per year. The largest IRAP project over the past five years disbursed $2 million over two years. Similarly, the maximum R&D contribution under DIRP is $500,000.

In addition to these programs, the Industrial and Regional Benefits Policy (IRB) helps ensure that Canadian industries benefit from government defence and security procurement. The Policy requires that international companies that win defence and security contracts with the Government of Canada place business activities in Canada at the same value of the contract. IRB is almost always applied on procurements over $100 million and remains discretionary below that threshold.

SADI also supports two agreements with the United States: The Joint Strike Fighter (JSF) program and the Defence Development Sharing Agreement (DDSA). The JSF program is a US-led multinational effort to build multi-role, fifth generation stealth fighter aircraft. In February 2002, the Government of Canada signed a Memorandum of Understanding with the US Department of Defense to participate in the JSF program. As a partner nation, Canada is in a position to secure high-value work on the JSF. SADI supports JSF-related projects by awarding repayable contributions for JSF-related research and development, provided the requested work falls within the terms and conditions of the program. Special terms are given to JSF-related projects such as a 40% sharing ratio, nominal repayments, and a 20-year repayment period.

The DDSA is a bilateral Canada-United States Memorandum of Understanding signed in 1963, which enables Canadian firms to participate in R&D projects to meet the requirements of the US armed forces. DDSA project funding is normally split between the US Department of Defense (50%), SADI repayable contributions (40%), and the company (10%).

Overall, SADI plays a complementary role in providing support to the A&D industries. SADI differs from other federal programs primarily in terms of its focus on later stage research and development. It also differs profoundly in terms of its scale. SADI can support much larger projects through repayable contributions. Finally, the time frame also differs in that SADI supports a repayment period that is fifteen years long.

In addition to Government of Canada support programs, the provinces play an active role in promoting their own industrial development.^{Footnote 21} In terms of potential overlaps with provincial programming, provincial support for economic development can include support that could be considered industrial research and pre-competitive development. For example, the Government of Ontario's Ministry of Economic Development and Trade operated its Advanced Manufacturing and Investment Strategy (AMIS) until February 2010. This program provided interest free loans to companies that invested in advanced manufacturing projects. Specifically, AMIS covered up to 30% of projects costs, which included R&D costs. However, it should be noted that participation by A&D firms in provincial programs would be restricted to their local (rather than national) operations.

In terms of provincial support that is targeted towards the aerospace sector, the Government of Quebec plays the most active role among provinces in supporting its aerospace cluster. Investissement Québec provides loans, financial guarantees, and provincial tax credits to aerospace firms.

In accordance with the Directive on Transfer Payments, SADI manages potential overlaps in government support across jurisdictional boundaries by ensuring that the Government of Canada's stacking rules are applied when it assesses proposals. The stacking limit is the total of all government assistance that is allowed on a common project as a percentage of eligible costs. Total assistance includes all federal, provincial and municipal funding, and is limited to 75% of eligible costs.

3.2 Performance

3.2.1 To what extent have SADI recipients increased their investments in strategic R&D as a result of SADI repayable contributions?

SADI has led to an increase in the total amount of R&D investments by A&D firms and SADI repayable contributions investments have leveraged private sector investments. Since the creation of SADI in 2007, the program has contracted 24 projects totalling $824 million of repayable contributions, leveraging an additional $1.8 billion in R&D investments for a total investment of $2.6 billion on innovation. This represents a leverage ratio of more than $2 for every dollar of SADI repayable contributions.

The question of whether there was a net increase in R&D investments that can be attributed to SADI is more complex. In the interviews, some recipients were unequivocal about the positive impact that SADI had in increasing their total R&D spending. Of the 19 firms that were interviewed, eight SADI recipients attributed an increase in their R&D activities directly to SADI.

Most firms reported a beneficial impact from SADI, but not necessarily an increase in their R&D spending as a result of the program. For these firms (11 of 19), the main impact of SADI repayable contributions was to accelerate their planned R&D spending. Given the necessity to invest in technology and to position themselves to take advantage of potential opportunities, these recipients reported that they would have eventually generated sufficient revenues from internal operations or found other sources to fund the necessary strategic R&D investments. That said, SADI played a critical role. The acceleration of planned R&D spending was beneficial because it helped reduce their time to market. For some firms where there was a competitive advantage in being first-to-market, this impact was significant.

SADI repayable contributions provided other related benefits to recipients. The program reduced the perceived level of risk of the R&D investment. It allowed recipients to be in a more cash positive position during commercial development and this made it easier for the firm to make the initial investments. The existence of SADI repayable contributions also allowed recipients to maintain their R&D capacity despite the fall in sales during the recent economic downturn. One firm stressed that it would have cancelled its R&D project because of the recession if it had not been for SADI.

Overall, case study participants firms were generally positive about the impact of SADI. They indicated that SADI repayable contributions helped them take on more R&D projects; broaden the scope of their projects; and accelerate the timelines of their projects. SADI repayable contributions also allowed firms to develop new areas of focus, and in some cases take on riskier projects. The case study participants were the earliest of the SADI projects so it is possible that the full impacts of SADI repayable contributions on later recipients will become more evident over time.

3.2.2 To what extent has collaboration between recipients and post-secondary institutions (universities, colleges and research institutes) increased as a result of SADI?

In the summer of 2011, AEB conducted an independent stocktaking of the status of collaboration activities between recipients and post-secondary institutions. As part of the review, the program identified nine SADI projects that included a new or enhanced collaboration between a recipient and a post-secondary institution (See Appendix G). The remaining 15 SADI projects were characterized as continuations of existing collaboration activities between recipients and post-secondary institutions. These existing collaborations had been established prior to SADI.

In interviews with researchers and other university stakeholders, the review found that the overall perception of post-secondary institutions was that they had not benefitted from new and enhanced collaborations under SADI. In most cases, the impetus for current collaborations was attributed to the activities and funding available through the Natural Sciences and Engineering Research Council and the Consortium for Research and Innovation in Aerospace in Quebec, and to a lesser extent smaller programs such as the Ontario Centres of Excellence or the Ontario Research Fund. This perception was compounded by the fact that many university collaboration projects have several partners and their timelines often preceded SADI, making it difficult to attribute increased collaboration to a specific program.

For the evaluation, AEB conducted interviews with recipients. In these interviews, almost half of the recipients (9 of 19 interviews) reported an increase in collaboration with post-secondary institutions that could be directly attributed to SADI. Specifically, five recipients reported that SADI repayable contributions had helped them hire more students and another two firms reported that SADI repayable contributions had led to increased research collaboration. For two recipients, SADI had led to both enhanced research collaboration and the hiring of more students.

An innovative example of collaboration came from the CAE case study. In addition to helping foster a number of research collaborations and the hiring of students, SADI played a direct role in enhancing CAE's collaboration with École Polytechnique. In 2011, CAE began collaborating with École Polytechnique to create a laboratory environment at the university for CAE's Augmented Engineering Environment (AEE). The AEE is a simulation product that enables Original Equipment Manufacturers to evaluate, test, and validate a range of aircraft models and systems during the development phase. In early 2012, the contribution of early versions of AEE licenses to the lab will help CAE because the contribution will serve as an end-user test platform for the product and help identify refinements.

Overall, the renewed focus on collaboration within SADI also raised awareness among recipients about the benefits and the potential for collaborations with post-secondary institutions. According to a senior manager at one recipient, the firm had struggled in the past to find effective ways of collaborating with universities and did not have a history of looking for ways to collaborate with universities. As a result, the recipient's SADI project originally did not include a collaboration element. Given the greater emphasis on post-secondary collaboration placed by the program, the recipient decided to try using students to test the quality of a recent software release. This project gave the students valuable work experience and provided the recipient with an opportunity to identify potential hires. According to the senior manager, the firm may never have come up with this idea for collaboration if SADI had not communicated the value that it placed on collaboration.

3.2.3 Recognizing that results may take years to occur given the nature of the program, to what extent have SADI-supported R&D activities increased (or will be likely to increase) the development and commercialization of innovative products, processes and services?

One of the challenges facing firms in the A&D industries is that they need to invest in research and the development of new technologies, but they will only see the benefits of these investments in the far future. Given that the program is only in its fourth year, all of the SADI projects are still in the R&D Phase. Among the 24 projects, the earliest year that any of the recipients will enter the Repayment Phase is 2013-14. The Repayment Phase is when the commercialization of products, processes and services is expected to take place.

When a firm submits its proposal to SADI, program officers look at the degree of strategic innovation and excellence as well as the feasibility of the research and development in terms of its likelihood to lead to product applications or service applications. The result is that the program likely screens out weak proposals that are unlikely to result in development and commercialization.

In this context, recipients reported that many projects have already resulted in the development of new products, services or processes. According to the SADI Annual Program Benefits Report of July 2011, five of the 24 SADI projects (21%) have already resulted in the development of new products, services, or processes. Interviews with recipients revealed a slightly more positive picture than the program's data. While almost half of the recipients (8 of 19) believed that it was too early to tell whether their SADI projects would result in full commercialization, the majority of recipients (11 of 19) reported progress towards the development of products, services or processes.

The case studies provided the richest source of detailed information on the success of recipients in developing and commercializing new products and services. There are three prominent examples of how SADI has already contributed to successful commercialization.

1. EMS: This recipient supplies high-speed data communications equipment for commercial and military aircraft. Supported by SADI funding, EMS has completed development on several products including the AMT-700 high gain antenna, an ARINC 781 transceiver set, and the Aspire 200 LG airborne communications system. In July 2011, EMS announced sales of the Aspire 200 LG to Anderson Air. This will deliver feature-rich voice and data connectivity for small and medium-sized business aircraft and allow passengers to make telephone calls and have Internet access while in-flight.
2. Norsat: This recipient produces communication solutions for remote and austere locations including satellite ground equipment which enables the high-speed transmission of data, audio and video signals. With the help of its SADI project, Norsat has developed a number of new microwave and satellite products. The firm also added X-band and improved frequency capabilities to its product families in order to enable their use by military organizations.
   
   Norsat has successfully sold products associated with its SADI-funded R&D project to several military customers. In August 2011, the firm announced that it had won a $1.3 million contract with the NATO Consultation, Command and Control Agency. The firm's equipment and services will play a critical role in NATO's communications in remote regions in Afghanistan.
3. SkyWave Mobile Communications: This recipient designs and manufactures satellite terminals and operates satellite network systems to assist customers in tracking, monitoring and controlling their fixed and mobile assets. In August 2011, the firm launched a new satellite service for machine-to-machine communications that allowed tracking and messaging for commercial transport and government fleets, the transmission of telemetry information from oil and gas distribution equipment; and the remote management and control of fixed assets.

3.2.4 Has SADI resulted in any unintended impacts or outcomes?

There were no significant unintended negative impacts arising from SADI. However, individual recipients experienced benefits and outcomes from SADI that had not been foreseen at the time that their applications were approved. These emerging benefits included cultural changes, improved reputation, and the potential for growth in entirely new product areas.

1. New Areas for Growth: Some case study firms reported that the products that had been developed through their SADI projects were being used for markets that had not been anticipated at the time of their application to the program. These recipients are entering new geographic markets, new adjacent product markets and establishing partnerships in new countries.
2. Cultural Changes: For one recipient, SADI's reporting requirements and the underlying need to develop products and commercialize its research led to a cultural change within the firm's engineering group. In the past, the engineering group would conduct research purely for the sake of research. It investigated particular technologies and features without having an end customer in mind. Because of the firm's participation in SADI, there is now a much stronger emphasis on defining concrete deliverables and developing project schedules with clear milestones and associated project risks. All of the firm's R&D activities now identify its intended customers.
   
   According to another recipient, the firm has become more philanthropic as a result of its SADI project. The technology that it developed has been used to support disaster relief efforts in Japan, Chile, and Haiti. In these circumstances, communicating is often as important as providing the basics for survival (i.e., food, water, and shelter). The recipient also helped an isolated First Nations community use this technology to set up distance learning and establish communication links with other First Nations communities. As a result of these sales and visits to support equipment installations, the firm reported that it has a greater appreciation of communities in need and is making more charitable contributions.
3. Reputational Benefits: Two recipients reported that participation in SADI has been used as a tool to market their firms. In particular, one recipient stressed that "the Government of Canada investment" in the firm through SADI has helped it present itself as a reputable, high quality supplier in the United States and this has contributed to greater sales.

3.2.5 How efficient is ITO in the delivery of the SADI program?

The key measures for assessing the efficiency of the program were the time required to process applications and the efficiency of the organization in terms of operating expenses as a percentage of total program expenses.

SADI Application Processing Times

Since the inception of the program, the average time required from the start of the due diligence phase to obtaining departmental sign-off on Contribution Agreements was 8.9 months for all projects. To better manage its performance, ITO introduced a service standard in July 2009 to complete these activities within six months for SADI projects that were less than $10 million.

Since July 2009, the average time required to complete these activities across all SADI projects under $10 million was 6.3 months, slightly exceeding the service standard. The range was between 4.6 and 8.2 months. These average processing times are a noticeable improvement from the average time that it took for these activities prior to the introduction of the service standard. It took an average time of 9.6 months for equivalent projects in 2008-09 and an average of 11.3 months in 2007-08.

According to the program, the main challenges in meeting the service standard included delays by the applicants in providing necessary information, challenges in scheduling site visits (especially during holiday and vacation periods), and the availability of external experts to conduct due diligence.

The full process that applicants actually experience remains longer than what the service standard measures. In particular, there are several steps that occur in the application process prior the start of the due diligence phase. Specifically, SADI officers will provide guidance and feedback on draft applications and conduct a suitability assessment on a submitted proposal. According to SADI staff, the time from initial discussions of eligibility with an applicant to the due diligence phase can take from 6 to 12 months.

The challenges that ITO has faced in processing SADI applications since the inception of the program were reflected in the interviews with recipients. Recipients recognized the need for due diligence and accountability when disbursing taxpayer funds. They also found ITO officers to be diligent and client-focused. However, recipients found the application process to be onerous. Half of the recipients (10 of 19) believed that a weakness of the program was the application process. It was seen as too complicated, too slow, and involving too much paperwork. Some recipients experienced many requests for additional details or the same information in a different format. The length of the application process was cited by two A&D firms as a contributing factor in their decision not to apply to SADI.

The delays in the application process also had negative impacts on recipients. Because they did not have clarity on whether their planned R&D expenses would be approved, some firms reported that they slowed down their R&D activities and held off on hiring decisions, and this affected their time-to-market. Some recipients also reduced the scope of their planned SADI project.

Actual versus Planned Efficiency

When the program was established, ITO was allocated funding to develop an internal operating capacity to meet the eventual expected demand for SADI repayable contributions. During the initial years, some additional overhead would be required to design, promote and manage SADI as the program gradually ramped up. Over the longer term, the operating budget was expected to represent 5.4% of total program funding.

The evaluation used this 5.4% figure to assess the efficiency of the program. It should be noted that this measure for operational efficiency is an indicative, rather than conclusive, measure of the desired efficiency. Further, the figure is based on operating costs allocated to both SADI and TPC.

In order for SADI to demonstrate efficiency, the use of resources would be broadly in line with the anticipated design. If actual operating expenses were greater than 5.4% of total program funding, then SADI would be operating less efficiently than anticipated. If actual operating expenses were lower than 5.4% of total program funding, then SADI would be more efficient than anticipated.

Table 4 presents operating costs as a percentage of total program funding after the initial year of SADI operations. Operating costs included salary expenses and operating and maintenance expenses. The total costs of the program included these expenses plus SADI repayable contributions.

From 2008-09 to 2009-10, the operating efficiency percentages were higher than 5.4%, which indicates that SADI operated at a lower level of efficiency than anticipated. This can be attributed to the fact that the expected demand for SADI repayable contributions did not materialize. As a result, the program's capacity was greater than required for actual demand. Although the lower-than-expected operating efficiency percentage was not desirable, the program leveraged its extra capacity and improved other aspects of SADI service delivery. In particular, SADI undertook stakeholder engagement activities, developed program guidelines and tools, and improved its internal processes. The program also expedited amendments to Contribution Agreements and claims processing.

Over time, ITO decreased operating expenditures. Specifically, resources dedicated to managing TPC and SADI were reduced and the total number of full-time-equivalents fell from 99 at the inception of the program to 84. In total, SADI operating costs were reduced by 25% from 2007-08 to 2011-12.

These reductions also coincided with an additional flow of disbursements of repayable contributions. Disbursements of SADI repayable contributions increased from $62.0 million in 2009-10 to $114.6 million in 2010-11. The increased efficiencies led SADI operating expenses to fall below 5.4% of total program funding. The net result was that the program operated in 2010-11 and beyond at a greater level of efficiency than what had been anticipated at the time of SADI's inception.

4.0 Conclusions and Recommendations

4.1 Relevance

Regarding relevance of the program, the evaluation determined that:

• SADI addresses a continued need for R&D financing for aerospace firms and high-tech SMEs.
• SADI reached a small proportion of A&D firms. This is because of external factors (i.e., the economic recession), various elements of the program design, and limited awareness of the program;
• SADI is aligned with the priorities of the federal government and the strategic outcomes of Industry Canada; and
• SADI is aligned with the federal government's role and responsibilities.

4.2 Performance

Regarding the effectiveness of the program, the evaluation determined that:

• SADI helped increase or accelerate R&D spending among recipients;
• There has been some increase in collaboration with post-secondary institutions that can be attributed to SADI; and
• While the full results will not be known for a few years, SADI projects have already resulted in development and commercialization of products, services and processes.

Regarding the efficiency of the program, the evaluation determined that:

• While meeting the service standard, the application process was perceived as onerous and lengthy; and
• The overall efficiency of SADI improved consistently year over year.

4.3 Recommendations

The conclusions of the evaluation lead to the following recommendations.

1. ITO should consider identifying additional ways to streamline the end-to-end application process;
2. ITO should consider examining whether changes in program design are required to improve uptake, particularly to attract SMEs and enhance collaboration; and
3. ITO should consider building greater awareness of the program, especially among SMEs.

Management Response and Action Plan