Evaluation of the Bombardier CSeries Program

PDF version

252 KB, 30 pages

Table of contents

• Executive Summary
• 1.0 Introduction
• 2.0 Profile
  • 2.1 Program Context
  • 2.2 Program Objectives
  • 2.3 Program Description
  • 2.4 Stakeholders
  • 2.5 Program Resources
  • 2.6 Logic Model
• 3.0 Methodology
  • 3.1 Evaluation Scope and Objectives
  • 3.2 Evaluation Questions
  • 3.3 Evaluation Approach
  • 3.4 Data Collection Methods
  • 3.5 Limitations
• 4.0 Findings
  • 4.1 Relevance
  • 4.2 Performance
• 5.0 Conclusions and Lesson Learned
  • 5.1 Relevance
  • 5.2 Performance
  • 5.3 Lessons learned

List of acronyms used in report

List of tables

List of figures

Executive Summary

Program Overview

The Bombardier CSeries program advances R&D in the aerospace industry and maintains and enhances the technology base and technological capabilities of Canadian aerospace firms. Specifically, the program funds the development of generic technologies applicable to a variety of aircraft platforms and funds the development of technologies for a new fixed-wing commercial aircraft, the Bombardier CSeries aircraft.

The Government of Canada, in 2008, committed to provide $350 million in repayable contributions to Bombardier Aerospace through the Bombardier CSeries program. The funds come from existing departmental allocations and will be repaid through royalties from deliveries of CSeries aircraft and derivative aircraft that result from the CSeries aircraft development. Of the total $350 million committed $278.9 million was disbursed as of March 31, 2013 .

It is expected that the repayable contributions for R&D activities associated with the Bombardier CSeries program will result in the development of new technologies, increase the competitiveness of Canada's aerospace sector and foster the growth of a competitive, knowledge-based Canadian economy.

Evaluation Purpose and Methodology

In accordance with the Policy on Evaluation and the Directive on the Evaluation Function, the purpose of this evaluation was to assess the core issues of relevance and performance of the Bombardier CSeries program. The evaluation covered the period from September 2008 to March 2013.

The evaluation findings and conclusions are based on the analysis of multiple lines of evidence. The methodology included a review of documents, a review of administrative data, an environmental scan, and interviews with stakeholders.

Findings

Relevance

The evaluation found that the Bombardier CSeries program addresses a demonstrable need for aerospace sector funding due to the sector's financial risks and the importance of the economic benefits that the sector offers for Canadians. While the sector requires long product development cycles that limit funding alternatives the resulting high paying, high technology jobs, and the creation of spin-off benefits provide value to the Canadian economy.

The Bombardier CSeries program is aligned with government priorities as outlined in Speeches from the Throne, Budgets and the Science and Technology Strategy. The program is also consistent with Industry Canada's strategic outcomes and with the federal government's role to increase international competitiveness and encourage the development of science and technology. While this role is shared with the provinces, the Bombardier CSeries program contributes to it and the federal government's involvement is an important part.

Performance

The evaluation found that the Bombardier CSeries program contributed to the creation and maintenance of direct R&D jobs at both the recipient, Bombardier Aerospace, as well as at supply chain companies involved in the development of the CSeries aircraft. The number of jobs directly linked to Bombardier's R&D activities associated with generic aerospace technologies and with CSeries aircraft development is higher than originally forecast.

The evaluation found that Bombardier increased its R&D both in terms of its R&D investment and in terms of R&D activities undertaken as a result of the Bombardier CSeries program. This led to numerous projects to identify, develop and integrate leading edge technologies into this "first-of-kind" CSeries aircraft. A large proportion of the R&D activities concentrated on new technology integration and testing processes required to ensure the seamless manufacturing of the aircraft.

The Bombardier CSeries program R&D has resulted in the development of improved products, manufacturing processes and services for the CSeries aircraft and for future aircraft platforms. Improved developments include technologies and products integrated into the CSeries aircraft as well as the development of technologies that Bombardier will apply to future aircraft platforms. In addition, supply chain companies reported that their development of new products, processes and services will have application beyond the CSeries aircraft and are driving new business opportunities.

The evaluation noted that the Bombardier CSeries program has contributed to numerous new and enhanced collaborative activities between Bombardier, supply chain companies, research institutions and universities. However, documentation of these activities and their outcomes was not a requirement of the Bombardier CSeries program and, as a result, information on expected results and recipient performance was lacking. This made attribution of Bombardier's collaborative activities to the Bombardier CSeries program's repayable contributions difficult.

With respect to the operational efficiency of the Bombardier CSeries program, the evaluation determined that operating expenses as a percentage of total program spending were less than planned and that claims processing times were consistently faster than the Industry Canada (IC) standard for claims processing. The recipient indicated that the claims process was efficient and timely and that any administrative problems were resolved quickly.

Lesson Learned

It is important that future IC grants and contribution programs, that include results with respect to collaboration, clearly define expected collaborative activities, implement processes to monitor and document results on a regular basis and ensure that collaborative outcomes are regularly reported.

1.0 Introduction

This report presents the results of an evaluation of the Bombardier CSeries program.

The purpose of the evaluation was to assess the relevance and performance of the Bombardier CSeries program in accordance with the Policy on Evaluation requirements.

The report is organized into four sections:

• Section 2 presents the program profile;
• Section 3 presents the evaluation methodology;
• Section 4 provides the key findings related to the evaluation issues of relevance and performance; and
• Section 5 summarizes the study's conclusions and provides lessons learned.

Throughout this report the following terminology has been used:

• Bombardier - refers to the company Bombardier Aerospace that is the recipient of repayable contributions from Industry Canada
• Bombardier CSeries program - refers to the Industry Canada program that provides repayable contributions to Bombardier and which is managed by Industry Canada
• CSeries aircraft - refers to the aircraft platform developed by Bombardier that the Bombardier CSeries program repayable contributions fund
• CSeries project - refers to the R&D, assembly and test activities associated with production of the CSeries aircraft

2.0 Profile

This section provides information on the context, the objectives, the description of the program, stakeholders, resources and the logic model.

2.1 Program Context

By investing in research and development (R&D) projects, firms can achieve technological breakthroughs and increase their competitiveness, which, in turn, contributes to economic, technological and other benefits for Canadians. The level of R&D investment in the private sector is a concern because Canadian business expenditures on research and development (BERD) as a percentage of Gross Domestic Product are lower than what is found in roughly two‑thirds of other developed countries (including the United States and other G7 countries).^{Footnote 1},^{Footnote 2},^{Footnote 3} This is important because BERD is believed to affect the long-run rate of productivity growth in the economy. Consequently, the Government of Canada has a number of programs and initiatives that seek to encourage the private sector to invest more in research and development.

The aerospace industry is one of Canada's largest employers with 73,000 direct jobs in total and contributions of $12 billion to Canada's gross domestic product.^{Footnote 4} It plays a key role in driving innovation as aerospace firms are often early users of innovative technology including composite materials, information and communications technologies, nanotechnology, and advanced manufacturing techniques. As Canada's only fixed-wing commercial aircraft manufacturer, Bombardier Aerospace is an important member of the sector. Bombardier is a world-leading manufacturer of regional and business aircraft and has 35,500 aerospace employees worldwide including 20,000 in Canada.

In the past, Bombardier focused on building smaller regional aircraft with less than 100 seats. With the maturing of the smaller regional aircraft market, Bombardier made the strategic business decision to develop a new family of larger regional aircraft with a seating capacity of between 100 and 149 seats and a transcontinental capability. These new CSeries aircraft will respond to market demands for quieter aircraft with lower fuel consumption and lower operating costs per passenger mile. Specifically, the aircraft are expected to have 20% lower fuel consumption and 15% lower operating costs.

The CSeries aircraft development is reaching its final stages. In March 2013, Bombardier presented to the industry a completed aircraft that will take part in the flight test program. The CSeries aircraft is scheduled to enter into service in 2014.

2.2 Program Objectives

In 2008, the Government of Canada committed to provide $350 million in repayable contributions through the Bombardier CSeries program to fund Bombardier R&D associated with new generic aerospace technologies and the CSeries family of aircraft.

The program's repayable contributions provided funding for the development of new commercial aircraft technologies. Specifically, the objective was to encourage R&D that will result in:

• the development of generic technologies applicable to a variety of aircraft platforms including advanced materials, technologies and manufacturing processes; and,
• the development of technologies for a new fixed-wing commercial aircraft, the Bombardier CSeries aircraft.

2.3 Program Description

The Bombardier CSeries program provides repayable contributions through two contribution agreements. The federal government's contribution will be repaid through royalties from deliveries of CSeries aircraft and derivatives that result from the CSeries aircraft. The two projects that make up the CSeries program are:

• The Generic Technologies Project: This project involves the development of a range of aircraft technologies (e.g., composite materials, fly-by-wire technologies, noise abatement technologies etc.) that will be applied to the CSeries aircraft and to other aircraft platforms.
• The CSeries Project: This project includes the development of technologies that will contribute to the unique design and specifications of the CSeries aircraft family. This includes systems integration activities, systems testing, and specific process and production engineering efforts that apply only to the CSeries aircraft family.

The Bombardier CSeries program is managed and delivered by Industry Canada's Aerospace, Defence and Marine Branch (ADMB).

2.4 Stakeholders

In addition to the recipient, there are a number of stakeholders in the Bombardier CSeries program including:

• Supply Chain Companies: The development of the CSeries aircraft includes a number of Canadian suppliers and other firms who are engaged in technical collaboration projects with Bombardier. These private sector collaboration projects include composite manufacturing technologies, new engine technologies and simulation technologies for both manufacturing and operation of the aircraft.
• Industry Associations: The Aerospace Industry Association of Canada (AIAC) is the national industry association organization for the aerospace sector and plays a key advocacy role on aerospace policy issues. Aéro Montréal is a strategic think tank which seeks to increase the cohesion and competitiveness of the Québec aerospace cluster. Finally, the Consortium for Research and Innovation in Aerospace in Québec (CRIAQ) promotes industry-led research collaboration projects involving universities and research centres in Quebec's aerospace cluster.
• Universities and Research Institutions: As part of the development of the CSeries and other aircraft, Bombardier is working with university researchers. Some of these research projects include work in the areas of computational fluid dynamics, drag prediction, wing icing modelling, in-flight and systems simulation and acoustics systems.

2.5 Program Resources

The funds for the Bombardier CSeries program come from existing departmental allocations. Of the total $350 million committed to the program $278.9 million was disbursed as of March 31, 2013 .

2.6 Logic Model

The Logic Model in Figure 1 describes the program design and delivery strategy and intended outcomes. It is expected that the contributions for R&D activities associated with the Bombardier CSeries program will result in the development of new technologies; increase the competitiveness of Canada's aerospace sector; and foster the growth of a competitive, knowledge-based Canadian economy.

Figure 1: Bombardier CSeries Program Logic Model

3.0 Methodology

This section provides information on the evaluation approach, objective and scope, the specific evaluation issues and questions that were addressed, the data collection methods, and data limitations for the evaluation.

3.1 Evaluation Scope and Objectives

The objectives of the evaluation are to address the core issues of relevance and performance in accordance with the Directive on the Evaluation Function. The evaluation covers the period from September 2008 to March 2013.

3.2 Evaluation Questions

The evaluation focussed on the immediate outcomes of the program and to the extent possible the intermediate outcomes. The evaluation addressed the following questions of relevance and performance:

Relevance

1. Does the Bombardier CSeries program address a demonstrable need?
2. Does the Bombardier CSeries program align with the priorities of government and the strategic outcomes of Industry Canada?
3. Does the Bombardier CSeries program align with the roles and responsibilities of the federal government?

Performance

4. To what extent has the Bombardier CSeries program contributed to the creation and maintenance of highly qualified R&D employment?
5. To what extent has the recipient increased its R&D activities as a result of the Bombardier CSeries program?
6. To what extent have the R&D activities funded by repayable contributions from the Bombardier CSeries program contributed to the development of new and improved product, process and service technologies?
7. To what extent has collaboration increased as a result of the Bombardier CSeries program?
   • Research institutions and universities
   • Private sector firms
8. To what extent has Industry Canada been efficient in the administration of the Bombardier CSeries program contribution agreements?

3.3 Evaluation Approach

Like many evaluations of government programs, this evaluation was goal-based and focussed on the goals and expected outcomes of the program as stated in the Bombardier CSeries program's foundational documents and logic model. The evaluators looked to measure the outcome variables using a variety of research methods.

The evaluation was conducted by the IC Audit and Evaluation Branch (AEB). AEB solicited feedback from ADMB on the draft report prior to the presentation of the final report to the Departmental Evaluation Committee (DEC).

3.4 Data Collection Methods

Qualitative and quantitative data was collected by different means and sources to provide multiple lines of evidence to support the conclusions and recommendations.

The data collection methods included:

• Document review
• Administrative data review
• Environmental scan
• Interviews

Document Review

A document review was conducted to gain a thorough understanding of the program and to provide insights into relevance and performance. Policy documents included Speeches from the Throne, Government of Canada Budgets and the Bombardier CSeries program 2008 Treasury Board Submission. Program documentation included Project Summary Forms, Contribution Agreements and Bombardier updates from claims reports, project review meetings and other monitoring documentation. Other industry documents reviewed included the Strategic Aerospace and Defence Initiative (SADI) Collaboration Review report and SADI Evaluation report, the Aerospace Review final report "Beyond the Horizon: Canada's Interests and Future in Aerospace" and other industry investment reports and analyses.

Administrative Data Review

Financial, operational and other administrative data were reviewed to analyze program efficiency and inform the analysis of performance questions.

Environmental Scan

The environmental scan examined the objectives, activities, funding levels, collaboration requirements, and other funding conditions for comparable programs with a focus on the European Union, the United States, and Japan.

Interviews

The objective of the interviews was to gather in-depth information related to the relevance and performance questions. Interviews were conducted with the following groups of stakeholders:

3.5 Limitations

The following were limitations to the methodology:

• The evaluation sought to determine the extent to which there had been an increase in R&D investment by the recipient. However, R&D investment information is considered confidential by Aerospace and Defence (A&D) firms for competitive reasons. This meant that the evaluation relied on information reported by interviewees to determine the net increase in R&D activities attributable to the program.
• Innovative R&D always incorporates a certain level of technological risk. As a result, the assessment of the development of new and improved product, process and service technologies was based on findings to date and reasonable expectations about the future. The full nature of the achievement of this outcome will become apparent after the evaluation is completed and once an aircraft goes into production.
• The timing of the evaluation limited availability of information on intermediate and final outcomes. The program's intermediate and final outcomes include the commercialization of new and improved product, process and service technologies, as well as increased revenues to aerospace firms and other suppliers. These outcomes are expected to occur after the entry into service of the new aircraft.

4.0 Findings

4.1 Relevance

4.1.1 Does the Bombardier CSeries program address a demonstrable need?

By their nature aircraft development programs are subject to large cost, technological, and scheduling risks^{Footnote 5}. For instance, significant R&D investments are required well in advance (up to 10 years) of revenue being generated from aircraft sales. Even after entry into service, new programs can take more than a decade to "break‐even". Therefore the market interest and capacity to provide funding for these projects is low as no return is generated for many years. These long product development cycles are also particularly sensitive to economic downturns that affect aircraft sales. This further limits the interest of private sector sources of finance as they are adverse to these kinds of risks.

In addition, the aerospace sector is in the midst of tremendous change in terms of technologies and the use of materials in aircraft^{Footnote 6}. For example, the use of metal in aircraft is being replaced by composite materials for aircraft frames in many cases. While these new composites offer the promise of lighter aircraft, greater fuel efficiency, and greater resistance to corrosion, they also require substantial new R&D investments to develop knowledge of the best practices for their integration into new aircraft, their manufacture and their maintenance. Other innovative technologies utilized in the CSeries aircraft included new engine technologies, avionics, fly-by-wire technologies and new manufacturing and systems integration processes - all created unique challenges that increased the risk associated with the aircraft development.

The long product development cycles, limited funding alternatives and rapidly changing technologies are reasons why so many foreign governments support their aerospace sectors. Governments also support their aerospace sectors because the economic benefits of these sectors are significant. They directly create high quality employment and produce additional jobs in spin off companies and industries. The sector drives innovation and technology progress which leads to increased prosperity and well being in countries with strong aerospace sectors.

Examples of the broad range of international government support for research and development in the aerospace industry have been well documented^{Footnote 7 Footnote 8}. In Europe, Japan and the United States dozens of programs provide direct funding for R&D. In addition, new competitors in China and Russia are being funded by government programs that help address requirements for the development of their products.

The federal government's repayable contributions to the CSeries project through the Bombardier CSeries program was part of an integrated web of financial support involving Bombardier itself, numerous private sector companies and the governments of the Province of Quebec and the United Kingdom. In the interviews, the federal government's contribution was viewed by aerospace industry associations and Bombardier as a key commitment to help mitigate the risks and challenges associated with this large technology development project. According to interviewees, without repayable contributions from Industry Canada, the aircraft development would not have proceeded on the same schedule and the achievement of advanced technology integration would have been limited.

4.1.2 Does the Bombardier CSeries program align with the priorities of government and the strategic outcomes of Industry Canada?

The Bombardier CSeries program's objectives are consistent with Speeches from the Throne and Budgets. Table 1 contains some excerpts from key documents that demonstrate the importance that the Government of Canada places on research and innovation as well as the government's continued commitment to the competitiveness of the aerospace sector.

In Budget 2011 the Government recognized the importance of the contribution of private sector aerospace investment to economic growth and identified the aerospace sector as one of the leading industrial sectors in the economy. Further it announced that it would initiate a review of all policies and programs related to the aerospace/space industry to develop a federal policy framework to maximize competitiveness of the sector and the resulting benefits to Canadians. The Aerospace Review^{Footnote 9} report, published in November 2012, found that "well-implemented public policies and programs can play a critical role" in facilitating the success of the sector. The report made 17 recommendations that were intended to help create conditions for the sector's success and deliver the economic, technology and security benefits that arise from a competitive aerospace sector.

In Budget 2012 the Government reiterated its commitment to supporting innovation and the funding of R&D projects in key sectors including the aerospace sector. Budget 2013 further reiterated this support and more specifically made commitments to implement recommendations from the Aerospace Review.

The Bombardier CSeries program's objectives are also consistent with the Government of Canada's Science and Technology (S&T) Strategy. According to the S&T Strategy, "The Government of Canada will ensure that its policies and programs inspire and assist Canadians to perform at world-class levels of scientific and technological excellence^{Footnote 10}." The Bombardier CSeries program supports this strategy by contributing strategically to R&D in the aerospace sector and maintaining and growing the technology base and technological capabilities of Canadian aerospace firms. Funding the development of the CSeries family of aircraft contributes to the continued success of Canada's aerospace sector.

The Bombardier CSeries program is also aligned with Industry Canada's Strategic Outcome 2: "Advancements in S&T, Knowledge and Innovation Strengthen the Canadian Economy." The department works to encourage investment in S&T that enables Canadian companies to compete and prosper in the global knowledge-based economy and to ensure that discoveries and breakthroughs happen in Canada and that Canadians can realize the social and economic benefits. The Bombardier CSeries program contributions to the CSeries aircraft strengthens the ability of Bombardier, its supply chain and the aerospace sector to compete in the global economy and it ensures that resulting jobs remain in Canada so that social and economic benefits can be achieved by Canadians.

4.1.3 To what extent does the Bombardier CSeries program align with the federal government's role and responsibilities?

Within the federal domain, the objectives of the Bombardier CSeries program fall under the Department of Industry Act of 1995. According to this legislation, the powers, duties and functions of the Minister of Industry extend to matters relating to "… industry and technology in Canada"^{Footnote 11}. Specifically, the objectives are to "… increase the international competitiveness of Canadian industry, goods and services…" and "encourage the fullest and most efficient and effective development and use of science and technology"^{Footnote 12}. Also, as part of his powers, duties and functions the Minister may "… develop and implement programs…" and "… make grants and contributions^{Footnote 13}."

The Bombardier CSeries program is one of a number of programs within the Government of Canada that provide funding to the aerospace industry. For example, the Strategic Aerospace and Defence Initiative (SADI) also provides repayable contributions for industrial research and pre-competitive development to aerospace, defence, space and industrial security firms.

The provinces also play an active role in promoting their own industrial development including the aerospace sector^{Footnote 14}. For instance, the Province of Quebec plays an active role in supporting its aerospace cluster through Investissement Québec loans, financial guarantees, and provincial R&D tax credits. In the case of the CSeries aircraft, the Province of Quebec made a $117 million contribution.

Federal government programs play a complementary role and are an important part of ensuring that projects like the Bombardier CSeries aircraft, the largest R&D aerospace project in Canadian history, have the necessary funding to achieve their objectives. These programs help fulfill the government's responsibilities to increase the competitiveness of Canadian industry.

4.2 Performance

4.2.1 To what extent has the Bombardier CSeries program contributed to the creation and maintenance of highly qualified R&D employment?

Bombardier reported^{Footnote 15} that as of December 31, 2013 the CSeries program will have created and maintained 9,076 person years of knowledge based employment (science, engineering and technical jobs) at Bombardier since 2008. This number includes a forecast of 2,066 person years of work for the 2013 calendar year. In addition, Bombardier reported that 532 person years of management and administration work were maintained or created over the same time. In total, the program is forecast to create and maintain 9,608 person years of work by December 31, 2013 compared to the original 2008 forecast of 7,771 person years of work.

This employment represents jobs directly linked to the Bombardier CSeries program and represents jobs that would not have been created without the federal government's funding. Had government funding not been available the timing of development of the CSeries aircraft would have been delayed and design compromises would have had to be made to reduce costs. According to the recipient, this would have reduced the number of jobs, impacted the ability of Bombardier to deliver a technically competitive product and limited Bombardier's ability to meet the market window for the aircraft. This would have jeopardized the viability of the development of the aircraft. Bombardier further indicated that a contributing factor to its creation of these jobs in Canada was the government's decision to fund the CSeries project.

In addition to direct employment at Bombardier, CSeries suppliers reported increased employment as a result of their involvement in the CSeries project. One SME reported that it had grown from 40 employees in 2007 to 115 today with all the growth directly attributable to their involvement in the CSeries project. The company developed a capability to provide integrated systems for the CSeries cockpit which included metallic, composite and plastic components. The employee growth included a doubling of the number of R&D resources in the company (from 15 to 30) due to the CSeries aircraft development and certification requirements.

Another SME indicated that it had grown to 150 employees from 35 in the last two years through internal growth and through the acquisition of another small complementary company. The growth was attributable to work on CSeries aircraft ground support maintenance tools. For this company, the CSeries experience also led to business opportunities associated with other aircraft platforms.

The evaluation found that creation and maintenance of jobs linked to the CSeries program occurred at both Bombardier and its supply chain collaborators.

4.2.2 To what extent has Bombardier increased its R&D activities as a result of the Bombardier CSeries program?

Bombardier's increased activity as a result of the Bombardier CSeries program is reflected in both the R&D investment that Bombardier made in the CSeries project as well as in the range of R&D initiatives that were undertaken to develop the CSeries aircraft. Bombardier R&D was focussed on identifying, developing and integrating advanced technologies in order to create the new aircraft platform.

The total project cost was forecasted to be $3.5B and was funded by Bombardier and multiple private sector and government organizations. These costs translate directly into incremental R&D activities in support of the CSeries project.

A Conference Board of Canada analysis^{Footnote 16} substantiates the fact that the Bombardier CSeries program increased Bombardier R&D activity. It reports that Canada's aerospace sector's R&D Intensity (R&D as a percentage of sales) increased in both 2009 and 2010 partly due to Bombardier's spending on its CSeries aircraft.

The project costs went to fund many R&D activities associated with the integration of advanced technologies into the aircraft and the use of testing processes and techniques to optimize the assembly of the aircraft. Key initiatives included;

• the development of simulation tools and devices to test the design of the aircraft, analyze the integration of different systems and to assess impacts of the different components working together before the parts were physically assembled.
• research to understand load distribution impacts and aerodynamic implications of under-wing mounted engines. Under-wing mounted engines offer greater accessibility for servicing and improved flexibility for cabin layout.
• research on the high-lift system of the aircraft (flaps and slats) to understand the take-off and landing performance of the aircraft. To minimize costs of traditional wind tunnel approaches Bombardier developed computer simulations (e.g., using computational fluid dynamics methods) to evaluate performance and reduce costs.
• development of thermo dynamic simulation methods to minimize the amount of energy needed to remove ice that forms on the wings of the aircraft.
• development of digital simulation models to reduce cost and time associated with physical prototypes. For example, a virtual landing gear simulation was developed to support the design and sizing of the landing gear system and manual landing release mechanism.
• research that addresses both internal and external noise of the aircraft to improve passenger comfort and reduce environmental impacts. This involved development of tools to predict in-flight noise levels based on ground tests of the engine and to model the interior cabin vibration absorption of new materials.
• research into alternative component joining and machining technologies to reduce costly and time consuming manufacturing cycles. Examples of technologies investigated included laser welding, bonding and thermoplastic assembling.
• development of a simulated manufacturing facility to test and validate various sequencing scenarios for optimizing and balancing of the assembly line, as well as for determining the impact of an assembly change, part change or process change. This reduced risks, costs and time associated with changes to the production routine.

The evaluation found that Bombardier increased its R&D activities as a result of the CSeries project. Incremental R&D investment by Bombardier led to numerous R&D activities to identify, develop and integrate leading edge technologies that contributed to the development of a "first-of-kind" new aircraft platform.

4.2.3 To what extent have the R&D activities funded by repayable contributions from the Bombardier CSeries program contributed to the development of new and improved product, process and service technologies?

The Bombardier CSeries program funded two key projects – the Generic Technologies project and the CSeries Technologies project.

The Generic Technologies project was focussed on the development and proving of advanced aircraft technologies essential to existing and future aircraft programs. The project involved the development of a range of generic aircraft technologies that would be applied to the development of the CSeries family of aircraft and would also be applicable to a variety of other aircraft platforms.

The CSeries Technologies project was intended specifically to fund the development of the CSeries aircraft. The project introduced several technical firsts for Canada's aerospace industry, including the first application of a fly-by-wire flight control system in a Canadian manufactured aircraft, extensive use of composites and light weight metal alloys for structural parts, and the latest in aircraft design and manufacturing processes. The CSeries aircraft is expected to minimize the overall environmental footprint of the aircraft by achieving 20% less carbon dioxide emissions and more than 50% less nitrous oxide emissions. This new family of aircraft is also targeted to be four times quieter (a significantly reduced noise signature that meets the latest International Civil Aviation Organization (ICAO) requirements) and to deliver better fuel consumption (i.e. 20% less fuel burn).

Bombardier reported that the Bombardier CSeries program has contributed to the development of the CSeries aircraft and the development of technologies for future aircraft platforms. The project has resulted in the integration of advanced technologies into the CSeries aircraft and has contributed to the development of an aircraft that combines advanced materials, leading edge technology, and new manufacturing and testing processes to meet commercial airline requirements of the future.

Examples of improved product, process and service technologies incorporated into the CSeries aircraft include;

• the turbo fan engine design that uses advanced combustion technology to deliver reduced emissions, noise and fuel burn. Each CSeries aircraft is powered by two Pratt & Whitney Pure Power PW1500G series engines. The Pure Power PW1500G engine uses an advanced gear system, allowing the engine's fan to operate at a different speed than the low-pressure compressor and turbine. The combination of the gear system and the advanced core delivers double-digit improvements in fuel efficiency, environmental emissions and noise. The engine has been subject to over 4,000 hours of rigorous testing since September 2010.
• advanced structural materials that achieve more than 2,000 pounds in weight savings and contribute to reduced fuel burn. For instance, the wings manufactured from composite materials at Bombardier's new facility in Belfast, Northern Ireland use new resin impregnation technology that enables the manufacturing of larger and more complex composite structures. Bombardier's unique Resin Transfer Infusion (RTI) process was used in manufacturing of the primary structural components of the wings. In comparison to competing materials additional benefits include the need for fewer inspections and lower maintenance costs as a result of the corrosion resistance properties and the fatigue strength of the material.
• composite materials were also used in the forward and aft fuselage sections, as well as the cockpit. Supplied by Bombardier's St-Laurent facility, Bombardier and its partners capitalized on their expertise in composites technology in the development of these carbon-fibre sections. Using advanced graphite and resin materials and Automated Fibre Placement (AFP) technology they developed manufacturing processes that ensured the required thickness of multiple layers of material was correctly placed and cured. AFP utilizes robotic technology to increase the preciseness and repeatability of the fibre placement and to meet the engineering specifications required for the composite components. Bombardier's unique approach to manufacturing of composite components is currently being patented by the company.
• advanced aluminum-lithium alloys used in the main fuselage to provide a high-strength, light, metallic fuselage. By combining alloy sheets and extrusions to manufacture fuselage skins, stringers, frames and floor beams significant improvements in aircraft operating costs, fuel consumption and emissions reduction were realized.
• avionics including a fly-by-wire flight control system that uses electronic means, rather than traditional mechanical means, to control the aircraft during flight. This allows for the introduction of a range of computer controlled enhancements to benefit the pilot, passenger and operator. The system provides an enabling platform to execute complex aircraft manoeuvres, optimize aircraft fuel efficiency, improve passenger safety, and enhance overall passenger ride comfort all of which contribute to an easier and more responsive aircraft to fly. For example, these enhancements ease the burden on pilots adjusting for adverse flying conditions.
• electro-mechanically controlled brakes that are designed to provide expanded capabilities when compared to hydraulic braking systems. For example, stopping performance meets or exceeds equivalent requirements of traditional hydraulic brakes. Electric controls also provide high reliability and ease of maintenance as brake wear and system health is reported automatically through on-board systems.
• Bombardier developed a comprehensive ground testing facility - Complete Integrated Aircraft System Test Area (CIASTA) - to ensure early integrated testing of all key CSeries aircraft systems. The test area incorporates a re-configurable engineering flight simulator, an avionics systems integration test rig, a flight controls integration lab to test the fly-by-wire flight control system and a full scale aircraft test rig to test interior systems. The development of the integrated test facility is a first in terms of creating an environment where all aircraft systems can be fully tested early in the development cycle.

In addition the program has contributed to the development of new products, processes and services throughout the CSeries aircraft supply chain. The project involves many suppliers – there are over 40 suppliers that work directly with Bombardier and many others that are part of the supply chain. Several companies have developed new capabilities as a result of their involvement in the CSeries aircraft project. The evaluation team interviewed eight supply chain firms and found several examples of companies that had developed new products or processes associated with CSeries aircraft components and manufacturing. Many companies have leveraged their new products and expertise to broaden their market opportunities.

For example, one SME invested significantly in new technology and equipment for manufacturing of the cockpit composite material to replace sheet metal. Its participation in the project resulted in the development of new products to meet Bombardier's design requirements and also created a manufacturing capability that is now being applied to the manufacturing of components for other aircraft platforms

Another supplier developed new "bootstrap maintenance equipment" that is used to remove the under-wing engines. The new solution addressed problems with the previous design and contributed to the improvement of processes for engine installation and maintenance. The same supplier also designed and manufactured a new water tank ballast system for CSeries flight testing. The water tank ballast replicates the presence of passengers during the test flight.

Pratt & Whitney Canada indicated that their new facility in Mirabel where the CSeries engine is assembled and tested has been developed using the latest in production technologies adopted from automobile assembly processes used in Germany. The engines are assembled using an automated flow assembly line – the engine is suspended from a track in the ceiling – a first for an aerospace engine manufacturer. The facility is believed by Pratt & Whitney Canada to be the most modern engine and test facility in the world.

CAE indicated that it has developed a suite of engineering services and simulation based technology tools to support the design, testing and certification of the CSeries aircraft as part of Bombardier's CIASTA program. The Augmented Engineering Environment (AEE) allows integrators to evaluate, test and validate a range of aircraft models and systems during the development phase to help reduce design risk. The selection of CAE as a supplier of the AEE for other Bombardier aircraft platforms is directly attributable to the successful development of the AEE for the CSeries aircraft.

The Bombardier CSeries program has contributed to a wide range of new and improved products, processes and services both for Bombardier itself and also for the chain of suppliers that have participated in the project. This array of new technology development has been integrated through the CSeries manufacturing process into the new CSeries family of aircraft.

4.2.4 To what extent has collaboration increased between the recipient and research institutions and universities and between the recipient and private sector firms as a result of the Bombardier CSeries program?

According to the RMAF-RBAF, an expected outcome of the Bombardier CSeries program was that the recipient would undertake R&D collaborations with research institutes, universities and other firms. In this context, the evaluation sought to determine the extent to which Bombardier collaborated with its supply chain and research institutions and universities as part of the Bombardier CSeries program. The evaluation team interviewed 13 organizations to discuss collaborative aspects of the project. While strong working relationships between Bombardier and these organizations were acknowledged and several examples of collaborative activities cited, it was difficult to attribute some of these activities directly to the Bombardier CSeries program and to the Government of Canada's contribution to the CSeries project.

Eight supply chain companies reported that there were strong working relationships between themselves and Bombardier and that this collaboration was an important contributor to the success of the CSeries project. Suppliers identified different types and levels of collaboration. Examples included;

• Pratt & Whitney East Hartford indicated that the two partners worked closely together on development of the PW1500G engine. While no Canadian government funds were involved in the development of the engine, the project involved detailed ongoing meetings between engineers from both companies to discuss design requirements, to conduct option analyses and to determine modifications required to tailor the engine to CSeries aircraft needs. Large amounts of collaborative work were done to integrate the aircraft and engine systems. Pratt & Whitney Canada also reported that Bombardier's commitment to being a lead customer helped establish market credibility for the new engine and helped open new markets.
• A SME supplier indicated that collaboration with Bombardier on requirements for system testing of passenger loading led to development of collaborative solutions with a third company. The relationship between these two companies ultimately led to their merger, the hiring of new employees and the creation of new market opportunities beyond the CSeries aircraft. The company reported that if not for the Bombardier CSeries program these opportunities would not have materialized.
• Another SME reported a strong collaboration with Bombardier that led to it becoming an integrated supplier for large system packages associated with CSeries cockpit. The company indicated that Bombardier worked with them to develop their internal procedures and engineering capability to meet CSeries aircraft technology and cost targets. The collaboration provided benefits for both companies. Bombardier was able to offload technology design and financial risk and the supplier developed new capabilities and products that addressed CSeries aircraft needs and created new business opportunities. The company expects to triple its number of employees over the next 3-4 years as a result of the CSeries aircraft collaboration.
• Development of the AFP process involved collaboration between Bombardier and a supply chain firm to enhance software controls for the robotic placement of fiber in component molds. This required complex developments to achieve the rigorous engineering specifications with respect to gaps between fiber layers, overlap of layers and other characteristics.
• An existing supplier indicated that their relationship with Bombardier, which began in 1994, has expanded as a result of the Bombardier CSeries program. Although headquartered in Europe the company established a facility in Laval. The company credits their joint pursuit with Bombardier for performance, efficiency and productivity, as well as the proximity of their teams as a key factor in it being awarded the landing gear contract for the CSeries aircraft. The Laval plant has been expanded to accommodate the assembly and testing of the landing gear system. The company plans on hiring about 35 employees to meet the assembly and test requirements of the CSeries aircraft.

In terms of research collaborations, the five universities that were interviewed all reported that they collaborated with Bombardier in a number of different ways. Collaborative activities included hiring of students for internship programs, funding of research chairs, hands-on flight testing programs for engineering students, funding of basic research and science projects to explore new technologies and funding of product development projects to meet short term product and manufacturing requirements. Examples of collaborative activities that could be directly linked to the Bombardier CSeries program included;

• growth in the number of students involved in Bombardier internships – one university reported that the number of interns hired by Bombardier had doubled from 50 to 100 between 2011 and 2012 as a result of the demand from Bombardier to support the CSeries project. Internship programs were cited by multiple universities as an excellent way to expose students to industrial experiences.
• funding of a research project, involving the used of computational fluid dynamics, to minimize draft and optimize the wing design

Universities also identified that a number of successful industry-university collaborative research projects had been initiated by CRIAQ. They indicated that the number of projects has increased significantly recently and that the research is resulting in new product developments. CRIAQ's mission is to increase the competitiveness of the aerospace industry, and enhance the collective knowledge base in aerospace through improved education and training of students. Part of its focus is to lead industry driven collaborative research projects. CRIAQ projects involve collaborative R&D efforts on projects that involve at least two industry partners and two research institutions or universities. On average each project involves 3-4 companies and 3-4 research institutes or universities. CRIAQ reported that it has launched more than 100 research projects over the last ten years and that Bombardier has been a very active member of the consortium. Since 2005, CRIAQ reports that Bombardier has significantly increased its collaboration efforts with CRIAQ and Bombardier is now involved in approximately 38 CRIAQ projects with about half of them being linked to the CSeries project. Examples of these projects include:

• Sound field rendering in aircraft cabins – a project including Bombardier and the University of Sherbrooke to build an "acoustic simulator" to emulate the environment of aircraft cabins. The simulator has the capacity to reproduce various noise sources. This allows the ambiance of the cabin to be "heard" before the aircraft flies and to "hear" the impact of planned modifications before they are actually implemented.
• Impact modeling of composite aircraft structure – a project including Bombardier and Université Laval to enhance understanding of impacts on CSeries aircraft wings and the resulting failure mechanisms. The project also determined how to improve composite resins to increase wing strength.

Aéro Montréal whose mission is to increase the cohesion and optimize competitiveness of Quebec's aerospace cluster also promotes consolation and collaboration of its members on specific projects aimed at the sector. Aéro Montréal reported that Bombardier is an active member of several working groups setup by Aéro Montréal that are dedicated to issues that the aerospace sector wants to address. Bombardier's collaboration with the sector, while not directed specifically at CSeries aircraft, has increased and knowledge gained has been applied across Bombardier's family of aircraft platforms.

Several interviewees commented on the challenges associated with fostering collaboration between industry and universities. They noted that the nature of repayable contributions incents recipients to focus on late stage development projects that are likely to generate a financial return. As a result early stage research activities that are key to driving future developments receive less support. Their view was that funding for early stage research should not be repayable.

The evaluation noted that increased collaboration between the recipient and supply chain companies, research institutions and universities was an outcome expected at the time of the creation of the Bombardier CSeries program. However, few details on how collaboration was to be defined and measured and what the expected outcomes, monitoring and reporting activities were to be were included in program documents. Annual Information Updates and Progress reports provided by Bombardier contained little information on collaborative activities and their results. This created a challenge for the evaluation to determine whether the degree of collaboration achieved was as expected at the start of the program.

4.2.5 To what extent has Industry Canada been efficient in the administration of the Bombardier CSeries program contribution agreements?

The evaluation sought to assess the efficiency of the Bombardier CSeries program based on metrics associated with the program's activities and outputs as well as on immediate outcomes. Operational efficiency metrics, related to activities and outputs, were program management costs in relation to overall program contributions and claims processing times vs. standards.

The first metric assessed Bombardier CSeries program operating costs. When the program was established, ADMB was allocated funding of $3.9 million to develop an internal operating capacity to support the Bombardier CSeries program over six years. The operating budget was expected to represent 1.1% of the total program funding.

The evaluation used this 1.1% target to assess the efficiency of the program. If actual operating expenses were lower than 1.1% then the Bombardier CSeries program would be considered to be more efficient than anticipated. If greater than 1.1% the program would be considered to be less efficient than anticipated.

ADMB estimated that actual total salary, operations and maintenance costs to manage the program over its six year period were $0.6 million. Program support involved claim reviews and approvals, monitoring activities (e.g., site visits, recipient meetings, review of progress reports), tracking of sector trends and reporting of program activity. The resulting cost as a percentage of program funding is 0.2% which is significantly better than originally planned.

The improvement in the operating cost relative to the initial plan is mainly a result of less personnel being required to manage the program than originally planned. The original plan forecast a requirement for four full time employees. The actual requirement was estimated to be the equivalent of 0.7 full time employees.

The second metric assessed claims processing times. The IC service standard of 45 days for payment of claims was used to assess the operational efficiency of the Bombardier CSeries program claims process. ADMB advised that, in addition to this external service standard, an internal target for payment of claims was set at 30 days. The evaluation assessed the program's efficiency in meeting this internal target as well.

Each claim is comprised of two parts - one for each of the two Contribution Agreements. The claims process involves an initial review of the claim progress report by ADMB, a detailed verification of the costs and receipts by the Industrial Technology Office (ITO) claims verification officer, final review and sign-off by ADMB executive management and financial processing by Corporate and Administrative Services (CAS). The process was efficient in that it utilized existing resources and processes that were already established in ITO and CAS for processing of program claims.

In total 23 claims have been submitted and paid by Industry Canada since the inception of the Bombardier CSeries program. The average time from receipt to payment of these claims was 23.3 days - all met the 45 day service standard. In addition, the average time of 23.3 days was faster than ADMB's internal 30 day target for claims processing.

Bombardier indicated that it has been getting claims processed efficiently and that payments have been received in a timely manner. Bombardier also reported that they were getting good support from IC with respect to claims payment and that when there were administrative issues occurred they were quickly resolved.

5.0 Conclusions and Lesson Learned

5.1 Relevance

Regarding relevance of the program, the evaluation determined that:

• The Bombardier CSeries program addressed demonstrated needs related to the sector's R&D financial risks and to the achievement of economic benefits for Canada.
• The Bombardier CSeries program is consistent with government priorities as outlined in Speeches from the Throne, Budgets and the Science and Technology Strategy.
• The Bombardier CSeries program is consistent with the federal government's role to increase international competitiveness and encourage the development of science and technology.

5.2 Performance

Regarding the effectiveness of the program, the evaluation determined that:

• The Bombardier CSeries program contributed to the creation and maintenance of more R&D jobs directly linked to generic aerospace technology and CSeries aircraft development than were originally forecast.
• Bombardier increased its R&D activity both in terms of its R&D investment and the range of R&D initiatives undertaken as a result of the Bombardier CSeries program.
• The Bombardier CSeries program has funded the development of improved products, manufacturing processes and services for the CSeries aircraft and for future aircraft platforms.
• The Bombardier CSeries program has contributed to numerous new and enhanced collaborative activities between Bombardier, supply chain companies, research institutions and universities. However, documentation of these activities was not a requirement of the Bombardier CSeries program and, as a result, information on results and performance was lacking.

Regarding the efficiency of the program, the evaluation determined that:

• The Bombardier CSeries program was operating efficiently as indicated by the fact that expenses as a percentage of total program spending were less than planned.
• The Bombardier CSeries program claims processing times were consistently less than the IC standard for claims processing.

5.3 Lesson Learned

The evaluation led to the following lesson learned.

It is important that future IC grants and contribution programs, that include results with respect to collaboration, clearly define expected collaborative activities, implement processes to monitor and document results on a regular basis and ensure that collaborative outcomes are regularly reported.