Evaluation of Innovation, Science and Economic Development (ISED) Canada funding to CIFAR

From: Innovation, Science and Economic Development Canada

May 2022

Key Definitions

Fundamental research, also commonly known as 'basic research', 'blue sky research' or 'curiosity-driven research', is a form of research that is undertaken to acquire new knowledge or a more comprehensive understanding of a subject without specific applications in mind^{Footnote 1}.

Applied research, is original investigation undertaken in order to acquire new knowledge. It is, however, directed primarily towards a specific practical aim or objective^{Footnote 2}.

Interdisciplinary research is a form of research that utilizes the tools and perspectives of more than one discipline to generate knowledge^{Footnote 3}. When conducting interdisciplinary research, it is common to see linkages formed between disciplines while some aspects of each discipline are distinctly recognizable or left intact^{Footnote 4}.

Artificial intelligence (AI) is a fast emerging technology that has made considerable advancement from the 1990s onward^{Footnote 5}. AI can be understood as systems or applications that are designed to mimic or augment human intelligence^{Footnote 6}. AI applications vary in complexity, ranging from simple tasks or purposes, such as video games, to advanced decision making agents which adapt to their environment. It is predicted that AI will have profound impacts on each industry – from financial services, mobility, and manufacturing to healthcare, education, and the public sector^{Footnote 7}.

Machine learning refers to the ability of a machine to learn how to perform a task without instructions but rather through experience acquired during a practice process^{Footnote 8}.

Deep learning refers to machine learning with a specific and particularly powerful technique, which uses neural networks connected on several layers. These artificial neural networks, inspired by the functioning of neurons in the brain, consist of mathematical processing of incoming data^{Footnote 9}.

Reinforcement learning is a branch of machine learning that enables AI systems to learn through experience. Instead of being told what actions to take to achieve a goal, the system must learn which actions yield the most reward by trying them. Over time, the system develops a policy (or way of acting) that lets it select the action that will best achieve the goal in a given scenario^{Footnote 10}.

About the Evaluation

Evaluation Context

The evaluation was performed to provide a strategic view of ISED's funding to CIFAR. An evaluation of ISED's funding to CIFAR is also required every five years to meet the requirements of the Financial Administration Act.

The evaluation covers the two contribution agreements with CIFAR during the evaluation period. These include ISED's funding to CIFAR towards their core activities (Core) and in support of the Pan-Canada Artificial Intelligence Strategy (PCAIS). As both contributions share the same Performance Measurement Strategy, it enabled an opportunity to examine both contributions through joint reporting.

Evaluation Objectives, Scope and Approach

The objectives of the evaluation are to examine the relevance, performance, and efficiency of ISED funding to the CIFAR (Core and PCAIS) in accordance with the Treasury Board Secretariat Policy on Results.

The scope of the evaluation includes two contributions to CIFAR (Core and PCAIS). It covers ISED funding to CIFAR during the period from April 1, 2016 to March 31, 2021.

The evaluation was conducted in-house by ISED's Audit and Evaluation Branch using a cluster approach.

The evaluation examined the achievement of expected outcomes, as identified in the CIFAR logic model in Appendix A.

All evaluation findings and recommendations were supported by multiple lines of evidence.

Background

• History and Evolution of CIFAR
• Overview of CIFAR Programming

History and Evolution of the CIFAR Program

CIFAR (formerly the Canadian Institute for Advanced Research) is a Canadian-based, global non-profit organization established in 1982 as an independent institute of advanced study, focused on delivering research excellence and impact. CIFAR's core activities include research, knowledge mobilization and developing future research leaders (Core). To do so, CIFAR supports long-term interdisciplinary collaboration and knowledge sharing to inspire new directions of inquiry, accelerate discovery and yield breakthroughs across borders and academic disciplines. It has an annual budget of approx. $21M per year for its core activities over the evaluation period.

Beyond its core activities, CIFAR also administers talent and research initiatives under the federal government's Pan-Canadian Artificial Intelligence Strategy (PCAIS), an initiative announced in Budget 2017 and supported through a distinct contribution agreement with ISED. The strategy brings together and enhances Canada's capacity in artificial intelligence (AI) research and training by building on and enhancing existing capacity. The PCAIS has an annual budget of about $16M per year over the evaluation period.

1983:

CIFAR launches one of Canada's first AI research programs: AI, Robotics & Society.

1987-2021:

Since 1987 ISED has invested total funding of $154M in CIFAR.

2017:

CIFAR was selected to develop and lead the federal government's PCAIS. ISED provided CIFAR with $125M to support the PCAIS.

2017:

Canada was the first country to announce a national AI strategy.

2017:

PCAIS funding supports research and training at Amii in Edmonton, Mila in Montreal and Vector Institute in Toronto.

2018:

CIFAR announces the first 29 Canada CIFAR AI Chairs.

2021-2031:

Renewal of funding to CIFAR: $48M for Core and $160M for the PCAIS.

Overview of Core Activities

CIFAR's Core activities aim to strengthen the Canadian research environment through the promotion of excellence and engagement with the international research community to ensure that Canadian researchers take part in world-leading research networks. CIFAR's Core activities are built on four interconnected pillars:

Research programs (Knowledge Creation)

CIFAR's research programs bring together international, interdisciplinary researchers who work together for five-year terms focusing on advanced, fundamental research aimed at addressing important challenges.

CIFAR arranges and supports meetings of program members with national and international guests. It also provides the researchers with funds designed to facilitate their research and involvement in the research network, as well as increase collaboration in high-risk research opportunities related to the program's overarching research goals.

Knowledge Mobilization

CIFAR connects the knowledge and outcomes of its research programs with leaders in key sectors, both nationally and internationally, to bring about new ways of thinking about important societal challenges, to inform the research process, and to help drive innovations in policy and practice. Individual knowledge users or communities engage in meetings with researchers via roundtables, lectures, workshops, events, etc.

Next Generation Initiatives

CIFAR supports researchers in the early stages of their careers through mentorship, networking, and skills training.

Public Engagement (Communication)

CIFAR's public engagement activities serve two important functions: to fuel scientific curiosity among the public by sharing the latest research from top scholars and scientists from around the world, and to share advances within its programs back to stakeholders who provide support.

Overview of PCAIS Activities

CIFAR's PCAIS activities aim to solidify and enhance Canada's position as a world leader in AI. The strategy is designed to foster a national approach in recruiting, retaining, and training top-tier talent that will advance AI-driven innovation in diverse areas of the Canadian economy. The CIFAR's PCAIS activities include the following pillars:

National AI institutes

The national AI Institutes: Amii in Edmonton, Mila in Montreal, and the Vector Institute in Toronto are based in regions with recognized research strengths and concentrations of deep learning, reinforcement learning and machine learning. The national AI institutes operate in close partnership with their local universities. While each institute has its own research priorities and strengths, all three share a common mandate of advancing Canada's research excellence in AI, training the next generation of students and working closely with industry and other partners to implement AI applications for positive social impact.

AI & Society program

The AI & Society Program, develops global thought leadership on the economic, ethical, political, and legal implications of advances in AI. These dialogues deliver new ways of thinking about issues, and drive positive change in the development and deployment of responsible AI.

Canada CIFAR AI Chairs

The Canada CIFAR AI (CCAI) Chairs program aims to recruit the world's leading AI researchers to Canada, while retaining existing top talent. The program provides long-term, dedicated research funding to support research programs and help train the next generation of AI leaders.

The CCAI Chairs are affiliated with Canada's three national AI Institutes.

National AI Program

The National AI Program includes training programs, task forces, grant programs, conferences, and symposiums. These events bring AI researchers and trainees together from across the country and around the world to foster collaboration and advance AI.

Program data show that the majority of funding (93%) has flowed directly to the national AI Institutes, with CIFAR retaining 7% for administration, coordination and the AI & Society Program.

CIFAR Revenue and Expenses

From 2016 to 2021, CIFAR's annual revenues averaged $31.7 million. ISED funding towards CIFAR Core activities represented $38.8 million over 6 years, and averaged $6.5 million per year. ISED funding towards CIFAR PCAIS activities represented $66.1 million over 4 years, and averaged $16.5 million per year.

CIFAR Core activities are also supported by the governments of Alberta, Ontario, British Columbia and Quebec as well as foundations, individuals, corporations, and international partner organizations.

From 2016 to 2021, program expenses related to Knowledge Creation accounted for the largest share of expenses (41.9%), followed by the PCAIS (35.9%) and Knowledge Mobilization and Communications (9.7%). Appendix B provides a breakdown of Knowledge Creation expenses by Research Program.

Methodology

• Evaluation Areas and Questions
• Data Collection Methods
• Challenges for the Evaluation

Evaluation Areas and Questions

The evaluation examined the following areas and questions.

Relevance:

To what extent is CIFAR addressing a unique and continued need in Canada's research environment?

Performance:

To what extent is CIFAR contributing to:

• Advancing interdisciplinary knowledge creation?
• Supporting Long-Term Growth of the Next Generation of Researchers?
• Driving Societal Impact through Knowledge Mobilization?

Efficiency:

To what extent is the CIFAR funding model an efficient approach in strengthening the Canadian research environment?

Data Collection Methods

Five data collection methods were used to support the evaluation.

Literature and Document review

The literature review was comprised of pertinent literature to gain a thorough understanding of the of the current and continued need for the CIFAR Core and PCAIS programs/activities. The document review comprised of key program and reporting documents to support the assessment of relevance, performance and efficiency.

Online Survey

An online survey was used to gather detailed information from past and present participants in CIFAR research programs from 2016-17 to 2020-21, as well as the Canada CIFAR AI Chairs and training participants (Azrieli Global Scholars). A total of 219 surveys were completed for a response rate of 30%.

Performance, Administrative and Financial Data Review

The CIFAR Core and PCAIS performance data, collected as part of CIFAR's Performance Measurement Strategy, was reviewed in order to assess the extent to which progress has been made towards achieving the expected outcomes outlined in the CIFAR logic model. An analysis of the administrative and financial data for both contributions was also performed to assess efficiency.

Virtual Interviews

A total of 31 virtual interviews were conducted using MS Teams across the following stakeholder groups to gather diverse perspectives on the relevance, performance and efficiency of both CIFAR Core and PCAIS:

• CIFAR Management and Board members;
• ISED program management;
• CIFAR Research Council;
• International Scientific Advisory Council (PCAIS);
• National Program Committee (PCAIS)
• Canadian universities, research institutes;
• National AI institutes (Amii, Mila, Vector); and
• Other government representatives.

Case Studies

Five case studies were conducted to support the overall evaluation of CIFAR. The case studies examined include two CIFAR Core research programs (e.g., Genetic Networks and the Learning in Machines & Brains program) as well as the three national AI Institutes (i.e., Amii, Mila and Vector). The case studies relied on document review, administrative data, and a total of 3 interviews with management staff of the three national AI Institutes.

Challenges for the Evaluation

The evaluation encountered four limitations and evaluators applied related mitigation strategies.

Attribution

The presence of other funding partners, and other complimentary initiatives could make isolating and measuring the direct impact of the federal government's contribution challenging. To alleviate this challenge, interview questions were designed and articulated in a way that respondents could answer, to the extent possible, the incremental impact of ISED funding.

Respondent Bias

Many interview participants are either involved in program delivery or are direct beneficiaries and as a result, responses may have been positively biased. To mitigate this, the purpose of the interview and its strict confidentiality was communicated to participants. Responses were validated across stakeholder groups as well as other lines of evidence.

Data quality

Self-reported data collected on an annual basis by CIFAR from research program members is the base of the CIFAR Performance Measurement Strategy. During the evaluation, data quality issues were encountered, such as unusually high values, reporting beyond the reference period and double-counting. As a result, external data was used to help validate the findings for certain indicators.

Limited HQP data

A survey of Highly Qualified Personnel (HQPs) who participated in CIFAR activities was planned as part of the evaluation. Contact information was only available for the Global Scholars so it was not possible to proceed with a full survey of HQPs. In order to mitigate this challenge, the evaluators used other available information provided by CIFAR (e.g., EDI data, post events surveys).

Findings: Relevance

Findings (Core)

Fundamental research can ultimately benefit Canada in terms of acquiring new knowledge, driving innovation and promoting the application of new technologies across the economy and society.

Documents and literature indicate that fundamental or basic research is critical as it lays the foundation for scientific breakthrough to occur^{Footnote 11} and further develops research capacity to strengthen Canada's research environment. Publicly-funded fundamental research has been identified by the OECD as a crucial determinant of competitiveness and national progress. The document review also suggests that fundamental research leads to discoveries that are the basis for solutions to real world problems. There are examples like AI in the history of science and innovation that demonstrate the best way to fuel innovation is to support fundamental research^{Footnote 12}. The 2017 Fundamental Science Review panel noted that over time, basic research will be shown to be "the upstream source of the foundational building blocks for innovations of transformative importance.^{Footnote 13} "

Literature indicates that Canada lags with respect to the overall OECD and G7 averages for R&D investments. Canada's gross domestic expenditures on R&D (GERD) intensity has been declining slowly over the last 15 years, in contrast with its G7 peers and key east Asian nations (i.e., Japan, Korea, China)^{Footnote 14}. For 2015-19 specifically, the average GERD intensity for OECD countries was 2.4% versus 2.1% for Canada.

However, according to interviews, Canada is currently punching above its weight when looking at its global standing in fundamental research, given the country's size and GDP. In terms of research, stakeholders agreed that Canada excels in a number of fields, particularly in areas of strength such as AI, Quantum and Genetics. Many interviewees noted that even though Canada does reasonably well on the fundamental research side, continued funding is necessary to keep researchers in Canada, generate new knowledge and accelerate innovation.

Primary benefits of conducting fundamental research: Acquisition of new knowledge, Attraction and retention of highly qualified professionals, Economic gains

Interdisciplinary and collaborative approaches have become increasingly important to solve complex questions that require input from more than one discipline.

Interdisciplinary research has gained popularity in scientific and academic communities as it is perceived as critical for innovation and addressing the scientific problems that exist in today's world^{Footnote 15}. It also complements discipline-specific research and can help address complex problems that require consideration of multiple perspectives^{Footnote 16}.

Collaborative research is conducted with the goal of generating new knowledge that cannot be achieved by one researcher alone^{Footnote 17}. International research collaborations have become increasingly visible with the advancement of knowledge, higher education, and increased professionalization and specialization in the sciences^{Footnote 18}.

CIFAR Core supports interdisciplinary and collaborative approaches, through its research programs that lay the foundations for future fundamental and applied research.

For example, CIFAR's Core activities supports interdisciplinary and collaborative approaches through their Global Calls for ideas – where CIFAR invites the global research community to submit proposals for new programs that address complex, fundamental questions of importance to the world. Ideas must be novel, bold, and potentially transformative to warrant the creation of a sustained, interdisciplinary, and collaborative research program. The proposals should confront challenging problems that necessitate a diverse team incorporating multiple disciplines (e.g., humanities, social sciences, arts, physical sciences, and life sciences) to engage in a sustained dialogue to develop new insights.

Further, CIFAR Core's collaborative approach and support of research is unique, focusing on challenges that transcend disciplinary boundaries and geographic borders. They support sustained collaborations between Canadian and international researchers to address global problems, open new fields of inquiry, and accelerate the creation and mobilization of knowledge that address these challenges^{Footnote 19}.

According to interviews, CIFAR is the only organization in Canada that plays the role of convener and that brings together international partners, thought leaders and the world's research community to address fundamental questions of importance to people everywhere.

Findings (PCAIS)

AI is a rapidly growing technology that is expected to impact various sectors of the economy.

AI has high potential to realize significant social and economic benefits, with estimates of AI contributing up to US$15.7 trillion per year to the global economy by 2030^{Footnote 20}. Emerging and innovative technologies such as AI are crucial to growing a competitive, knowledge-based future economy, and are expected to impact a broad range of industries, including: manufacturing; healthcare; digital industries; agriculture; transportation and clean technology. According to literature and interviews, machine learning, a subfield of computer science and AI research, has seen dramatic improvements in performance and availability in the last 10 to 15 years, which have positioned it as a transformative technology.

Documents and interviews indicate that maintaining leadership in AI is a priority for Canada as it will lead to a more innovative economy, increased economic growth, and an improvement in the quality of life for Canadians.

While Canada has been recognized as a leader in AI and deep learning research and use, talent and investment needs to remain in Canada in order to reap the benefits produced. AI has become increasingly more competitive on a global scale in considering that by the end of January 2020, 27 countries and the EU have developed national AI strategies, while an additional 18 countries were in the process of developing their national strategies at the time^{Footnote 21}. With other countries investing heavily in AI , it is believed that Canada needs to continue its efforts to build on its strengths and maintain the high-skilled jobs that AI is creating as a result of increasing global competition^{Footnote 22}.

Supported by the PCAIS and growing private sector investment, in 2021 Canada ranked fifth on the Stanford Institute for Human-Centered Artificial Intelligence (HAI) Index^{Footnote 23}, which compares countries on their AI capabilities based on key indicators including research and development, economy and inclusion (see table 1).

Table 1: Stanford Institute for Human-Centered Artificial Intelligence (HAI)Index

Rank	Country
1	United States
2	China
3	India
4	United Kingdom
5	Canada
6	South Korea
7	Germany
8	Australia
9	Israel
10	Singapore

Source: Stanford University

The PCAIS has helped Canada to maintain a globally competitive position in AI research.

Interviews and case studies indicate that the Pan-Canadian AI Strategy (PCAIS) plays a critical role in supporting research and stimulating investment in AI, growth in the AI ecosystem and the number of tech jobs, and enabling the research and innovation opportunities that help to keep highly-talented Canadians at home.

The PCAIS, through the national AI institutes, supports collaboration and brings researchers with different expertise, skills, approaches and application of AI together. Further, the national AI institutes work interdisciplinarily within different domains through partnerships with companies and different industry sectors. They also organize activities and events in order to stimulate interdisciplinary and collaborative approaches in AI.

Working with partner universities and organizations across the country and internationally, the PCAIS has helped advanced research in the societal implications of AI through CIFAR's AI & Society Program. The national AI institutes have cemented themselves as leaders in the responsible development and use of AI, including through participation in international initiatives such as the Global Partnership on Artificial Intelligence (GPAI).

The PCAIS complements other activities within the AI landscape.

For example, the Innovation Superclusters Initiative, which is focused on strengthening Canada's most promising clusters and accelerating economic growth in highly innovative industries across Canada. Both the Scale AI and Digital Technologies superclusters are well aligned with the PCAIS. Whereas the PCAIS supports the development of AI talent and research, these two superclusters support the commercialization of AI solutions in Canada and support domestic technology developers' efforts to grow their businesses, thereby also helping retain top AI talent and associated intellectual property (IP).

Collaboration among the national AI Institutes

In November 2020, Roche Canada launched the Roche AI Centre of Excellence (CoE), the first such collaborative centre to combine the expertise of all three national AI institutes under the CIFAR Pan-Canadian AI Strategy. The CoE focuses on advancing digital transformation in health, marrying Roche's longstanding history of leadership in health and life sciences, and the three institutes' established track record of excellence in AI research and enabling industry application. The CoE will work to deliver quality AI based digital solutions that optimize and reduce the cost of healthcare delivery, improve health outcomes, and enable Canada to learn and nimbly respond to opportunities and potential challenges in the healthcare system^{Footnote 24}.

Findings (Core and PCAIS)

Support for fundamental research and AI activities are aligned with the Government's objectives.

Since the recommendations of the Fundamental Science Review were released in 2017, the federal government has emphasized the importance of fundamental research and investing in the future of Canadian research. For example, Budget 2018 provided the single largest investment in fundamental research in Canadian history with nearly $1.2 billion over 5 years for Canada's granting councils^{Footnote 25}. It further noted that Canada has communities of research, homegrown talent, and a diverse ecosystem of start-ups and scale-ups, but these Canadian innovators need investment in order to ensure Canada's economy takes advantage of the enormous growth opportunities ahead in this sector^{Footnote 26}.

ISED's funding to CIFAR for both Core and PCAIS align with ISED's mission to foster a growing, competitive and knowledge-based economy. It contributes to the department's mandate to support science research, and to position Canada as a global centre for innovation. These investments are also aligned with Canada's Innovation and Skills Plan (ISP), which supports global science through world-class research partnerships. The ISP also identified AI as a key platform technology that will drive economic growth across Canada's industries, and an area where the country can be a global leader.

Further, Budget 2021 announced $443.8 million over 10 years, starting in 2021-22, in support of the next phase of the PCAIS. This enhanced second phase of the PCAIS will utilize new and existing programs, including:

• $162.2 million to help retain and attract top academic talent across Canada (renewed CIFAR PCAIS)
• $48 million for CIFAR Core to renew and enhance its research, training, and knowledge mobilization programs
• $60 million to Amii, Mila and Vector to support engagements to develop, use or commercialise AI technologies
• $40 million to provide dedicated computing capacity for researchers at the national artificial intelligence institutes
• $125 million for the Innovation Superclusters Initiative for the commercialization of AI innovations and research
• $8.6 million to advance the development and adoption of standards related to AI

CIFAR's objectives are consistent with the federal government's priorities to invest in fundamental research and innovation.

Building on the success of the first phase of the PCAIS, Budget 2021 committed an additional $443.8 million over 10 years to expand the strategy with a focus on commercialization and talent development and attraction.

CIFAR plays a niche and complementary role in the fundamental research ecosystem.

In Canada, the tri-council funding agencies are the main federal funding bodies supporting fundamental research and training at post-secondary institutions, with a combined annual budget of $3.6 billion in 2020-21 – about 100 times greater than CIFAR. These include the Canadian Institutes of Health Research ($1.3 billion), the Natural Sciences and Engineering Research Council ($1.4 billion) and the Social Sciences and Humanities Research Council ($1.0 billion). The tri-council agencies were the organizations most commonly identified as complementary to CIFAR and the PCAIS by stakeholders. Interviews and documents indicate that CIFAR funding primarily supports convening of researchers whereas the tri-council funding largely supports the research itself (salaries, equipment, etc.), with a tendency to be discipline specific. The evaluation survey showed that 51 of the 58 CIFAR Core respondents based at Canadian Institutions (88%) and 43 of 51 PCAIS respondents (84%) also reported receiving funding from the tri-council agencies.

One interviewee indicated that the CIFAR model is effective in that it focuses on important, cutting edge or new emerging areas, where other funding bodies are not able to draw to the same extent on expertise or identify experts who can do the peer review and selection of successful applicants. It was noted that CIFAR fills that niche of funding emerging fundamental research areas.

The document review identified that among the programs administered by the tri-council funding agencies, the New Frontiers Research Fund (NFRF) shares the most similarities with the CIFAR objectives. This is a new program announced in Budget 2018 which confirms the federal government's recognition of the growing importance of supporting interdisciplinary, international and collaborative approaches to fundamental research. The NFRF provides funding through three streams (i.e., exploration, transformation, and international) to support ground breaking research in Canada. Additionally, there are grants available for special calls which support the exploration of new and innovative research methodologies. The fund has a budget of $275 million over five years (2018–19 to 2022–23) and will grow to have an annual budget of $124 million in 2023–24, which represents a very small portion of the overall funding budget for the tri-councils.

The Canada Excellence Research Chairs (CERC) is another complementary initiative led by the tri-council agencies that supports Canadian universities in research and innovation (an award of $10 million over seven years), facilitating the attraction of world-class researchers to become CERC chairholders in areas of strategic importance to Canada^{Footnote 27}. As of 2020-21, the program had an annual envelop of about $43.1 million and is expected to reach $45.6 million in 2021-22^{Footnote 28}.

CIFAR plays a unique role as a convenor that complements the capabilities of the tri-council.

The COVID-19 pandemic highlighted the importance of a strong and collaborative research ecosystem.

The literature review found that the research ecosystem has responded well during the COVID-19 pandemic. The research funding system as well as research infrastructures were able to focus on crisis-related subjects and streamline procedures. However, as significant resources have been reallocated towards research that is relevant to the pandemic, there have been concerns regarding long-term funding for research once the immediate emergency has passed^{Footnote 29}. As a result, the OECD recommended that governments and research funding bodies quickly define and communicate which research they will be able to support in the coming years, in addition to their strategic priorities. This is necessary in order to foster cooperation and collaboration, prevent duplication, and identify areas where research is needed but not being performed. This would allow organizations that perform research to create long-term strategic plans which are realistic in scope, and it will also enable a coordinated global approach^{Footnote 30}.

The literature review also found that the COVID-19 crisis has demonstrated a variety of positive and desirable characteristics that many science systems possess which have enabled an effective response:^{Footnote 31}

• Flexibility with research funding and the ability to quickly allocate or reallocate resources as necessary.
• The ability to rapidly share data, which supports open science.
• Some capacity to co-ordinate internationally on certain objectives, often with the help of philanthropic organisations.
• An important role for research infrastructures from many different domains in supporting the performance of emergency research.

Documents and interviews showed that when the COVID-19 pandemic was declared, CIFAR acted quickly in order to mobilize its resources to help align with the federal government's response to the pandemic. They provided advice, funding and helped convene global experts. For example, projects supported through the CIFAR AI & COVID-19 Catalyst Grants program include machine learning research and applications to identify potential treatments, to support public health measures such as social distancing, and to better understand the viral transmission of COVID-19.

CIFAR Response to COVID-19

The pandemic has thrust science onto centre stage, and researchers from across every discipline — epidemiologists, social scientists, ethicists, virologists, clinical trialists, and drug developers — are addressing the questions and providing the tools needed to shape our successful response to the COVID-19 pandemic. CIFAR took immediate action during the COVID-19 pandemic, leveraging our experience as a global convenor of multilateral discussions to address the most urgent questions facing science, society, and governments.

Findings: Performance

Findings (Core)

The CIFAR approach to program selection has been successful in identifying areas with greater potential.

CIFAR Core's programs are tackling some of the most fundamental questions in science and society. Its model of bringing together fellows who would not normally interact, due to disciplinary or geographic boundaries, provides a unique approach that opens up new ways of thinking about important research challenges^{Footnote 32}. According to interviews, CIFAR Core program researchers are continuing to advance research across disciplines and contribute to scientific breakthroughs. Further, interviews noted that CIFAR programs have led to numerous landmark advances, including catalyzing the development of today's AI revolution, quantum computing, the recognition of the importance of a child's early years, and the socio-economic determinants of health. Documents also showed that CIFAR invested early in some areas which have become established research strengths. For example, CIFAR support for the Quantum Materials program dates back to 1987, while Canada is now recognised as a world leader in this field.

Case studies showed how CIFAR Core research programs have advanced interdisciplinary knowledge creation. For example, the Genetic Networks program published a study that examined the genetic interactions of 6,000 yeast genes. The research discovered 1 million genetic interactions. The mapping work was described as a "landmark accomplishment that will benefit the entire international community"^{Footnote 33}. This research was in part made possible by interdisciplinary collaboration between geneticists and computer scientists. Appendix C provides additional examples.

CIFAR funding was instrumental in producing research outcomes and securing additional funding.

The evaluation survey showed that CIFAR Core funding was instrumental, with 24% (38 of 160) indicating the research would not have occurred without CIFAR funding, while 63% (100 of 163) indicated the research may have occurred without CIFAR funding but it would have been affected negatively in terms of scope, quality, timing. CIFAR participation was also beneficial for securing research funding from other sources, with 114 of 165 CIFAR respondents (69%) indicating their involvement with CIFAR had a moderate or major influence on securing additional research funding.

75% or 124 of 167 CIFAR program researchers surveyed by ISED indicated that CIFAR's interdisciplinary research environment was the main factor that attracted them to CIFAR.

CIFAR convenes global thought leaders to enrich its programs and ongoing research, however current data suggests representation of women is low.

CIFAR plays a key role in connecting researchers to strengthen research capacity in Canada. Documents indicate that during the evaluation period, CIFAR Core supported 18 research programs spanning four broad themes: Life & Health; Individuals and Society; Information & Matter; and Earth and Space. Interviews and the document review found that CIFAR has established a diverse network of researchers through its research programs. As a program member, researchers have access to opportunities to participate in research activities, networking, program meetings, and conferences. Data analysis found that from 2016-17 to 2020-21, the research programs supported a total of 600 program members, which included fellows, advisors and scholars from 25 countries that are based at 211 universities and research centres around the world, with 233 researchers representing Canadian institutions and 367 from international institutions. The majority were based in North America (78%), followed by Europe (16%).

Data on gender for researchers was only available for 2019-20. Looking at the overall gender breakdown, women accounted for 100 of 343 researchers (29%). At the research theme level, the representation of women varied, with the highest concentration in the Individuals & Society area (36%), and the lowest being in Information & Matter area (23%) (see figure 4). While some of the differences are attributable to the lower concentration of women in certain fields of study (e.g., physical sciences and computer sciences), the literature review found that underrepresented or marginalized researchers continue to face biases and structural barriers.

Considering EDI from a regional perspective, some interviewees noted that CIFAR Core is not well known across Canadian Institutions, and the program data showed a high concentration of researchers at two of the largest post-secondary institutions in Canada. This is consistent with the literature review which found that smaller institutions and communities are often at a disadvantage in funding competitions^{Footnote 34}.

Additional data on EDI was available from the evaluation survey, which showed that 23 of 167 respondents (14%) identified with a visible minority community, while 5 of 166 respondents (3%) were persons with a disability and 4 of 166 respondents (2%) identified as Indigenous.

Accomplishments and scientific achievements of CIFAR researchers have been recognized worldwide.

CIFAR's success in supporting high quality research is evidenced through the number of awards and prizes received by their research members. For example, since CIFAR's inception, there have been 20 Nobel prizes awarded to CIFAR Core-affiliated researchers. The most recent, the 2019 Nobel Prize in Physics was awarded to CIFAR's Cosmology & Gravity program (now Gravity & the Extreme Universe) founding fellow. In addition, during the evaluation period, CIFAR Fellows, Advisors, CIFAR Azrieli Global Scholars and Canada CIFAR AI Chairs received 434 major awards and honours, including the 2020 Willet G. Miller and Flavelle Medals of the Royal Society of Canada.

The document review also found that CIFAR Core is having a notable impact on scientific advancement, with 179 fellows, advisors and CIFAR Azrieli Global Scholars contributing to the top 1% of most-cited papers worldwide from 2014-2018^{Footnote 35}. Also, a bibliometric analysis study conducted by CIFAR for its Quantum Material (QM) program revealed that publications by CIFAR researchers in QM have a high scientific impact. Of these papers, 26% were among the top 10% most cited worldwide (2.6 times higher than the world average), and about 4% were in the top 1% (4 times higher than the world average)^{Footnote 36}.

The A.M. Turing Award often referred as the "Nobel Prize of Computing" was awarded jointly to CIFAR Fellows Yoshua Bengio, Geoffrey Hinton and Yann LeCun in 2019 for their major contributions of lasting importance to computing.

CIFAR Core has been effective in improving coordination in Canada's established research strengths.

Interviews found that CIFAR Core has been effective in helping to create a national coordinated approach in important scientific fields. For example, CIFAR published a report about global policies for Quantum technology, which provided a snapshot of the rapidly evolving global policy landscape and gave policymakers a toolkit of potential measures to shape the development of the field in their own countries. The report supported NSERC and the National Research Council with the development of a national quantum strategy announced in the federal budget in April 2021. Interviewees also highlighted that the national quantum strategy is a direct outcome of having a national discussion and collaboration that was facilitated by CIFAR. Moreover, CIFAR has released two editions of a report on National and Regional AI Strategies. This report showcases an overview of the national and regional AI strategies landscape, and provides a snapshot of the rapidly evolving area of technological policy, highlighting the variety of approaches taken by different countries.

Findings (Core and PCAIS)

CIFAR facilitates collaborations among researchers by creating an environment that provides individuals with the opportunity to identify new partnerships.

The evaluation survey results indicate that Core program members have a high level of satisfaction with CIFAR's support in facilitating research collaboration. Respondents rated their satisfaction on an average of 8 or higher from a scale from 1 to 10. Further, respondents indicated that joint research projects (82% or 135 of 164) and peer-reviewed publications (65% or 106 of 164) were the primary ways in which CIFAR researchers collaborate with other researchers. Additionally, survey respondents reported frequent collaboration mostly with researchers in Canada (87% or 145 of 166) and the United States (83% or 138 of 166).

According to documents and interviews, CIFAR Core has helped increase networking, partnerships and collaborations among researchers, government and industry (domestically and internationally), particularly through research program meetings, workshops and symposia events. Data shows that during the evaluation period, CIFAR partnered with 205 organizations including governments, research organizations, industry, and foundations. For example, in 2019, the Learning in Machines & Brains program (LMB) partnered with the European Laboratory for Learning and Intelligent Systems (ELLIS), a grassroots organization of leading European AI scientists, to advance cutting-edge machine learning research. In June 2020, they hosted their first joint virtual workshop to discuss approaches to using contact-tracing apps during the COVID-19 pandemic.

Interviews and document review also indicate that CIFAR catalyst grants often support multi-year collaborations. This unique time-limited incentive of up to $50,000 provided by CIFAR supports high-risk, high-reward projects between program participants. Catalyst grants can also be used to foster synergies between programs, spur new ideas and insights and the exchange of research results, approaches, and techniques, which facilitates the evolution of the research direction for both individual members and for the programs^{Footnote 37}. During the evaluation period, a total of 161 Catalyst funds for a total of $7.7 million were awarded to projects within and across CIFAR's portfolio of research programs.

In 2019-20, through a catalyst fund, fellows from two CIFAR programs collaborated on a project about detecting the roots of consciousness in infants and were awarded a $2.5 million partnership Grant from SSHRC^{Footnote 38}.

The PCAIS supports collaborations and partnerships with the private sector, academia and other partners.

Case studies and interviews revealed that CIFAR's PCAIS activities have been very effective at fostering a collaborative AI ecosystem though the collaboration between the three national AI institutes. Interviewees highlighted that while the PCAIS has a series of primary affiliated universities with each of the three national AI Institutes, Amii, Vector, and Mila act as centres of excellence, providing opportunities to bring in leading researchers outside of those regions to partner and work collaboratively.

Stakeholder views indicate that the PCAIS has also been effective in enticing companies to approach universities to set up labs and collaborate – the national AI Institutes are benefitting from an unprecedented industry appetite for open-source basic research collaborations, resulting in strategic partnerships with major players such as Microsoft, Google, Samsung and Facebook. Furthermore, the institutes have been effective in co-locating academic researchers and SMEs.

Moreover, documents and case study findings show that the PCAIS has done essential work in advancing national AI initiatives, including the CIFAR-led AI for Health (AI4H) Task Force in September 2019, where CIFAR brought together AI and health innovation leaders from across Canada to ensure an integrated, coordinated, and collaborative approach to AI4H in Canada^{Footnote 39}. Key recommendations included: improving the effectiveness, efficiency and safety of health service delivery; providing insights to inform both disease prevention and policies; and underpinning the discovery and development of new diagnostic tools and treatments.

Over the evaluation period, the PCAIS increased the total number of partnerships from 52 in 2017-18 to 202 in 2020-21. This greatly exceeds the target of 46 by March 31, 2020.

Canada CIFAR AI Chairs and PCAIS Institute researchers have contributed to enhance visibility in AI research.

CIFAR's efforts to enhance visibility of AI research are extensive, according to the document review and data analysis. From 2017-18 to 2020-21, Canada CIFAR AI Chairs conducted 118 international policy consultations, including contributions to developing legislation, participation to advisory bodies, working groups, panels, or providing advice on AI policy. This greatly exceeds the 2020 CIFAR Performance Measurement Strategy's target of 5 consultations by March 31, 2023. Moreover, CCAI Chairs reported 465 interviews with media outlets since 2017-18, exceeding the target of 50 interviews cumulative total by March 31, 2021.

Findings (Core) – Next Generation

The CIFAR Azrieli Global Scholars program has helped accelerate the career of the next generation of research leaders.

According to program documents and interviews, the CIFAR Azrieli Global Scholars program provides a research fellowship and leadership training for early career researchers from around the world. Participants must hold a PhD (or equivalent), be within the first five years of starting a full-time research position at an institution of higher education or research, and must be engaged in research that complements a CIFAR research program. All Global Scholars receive:

1. Funding - $100,000 in undesignated research support, and eligibility to apply for CIFAR Catalyst funds.
2. Training - leadership development workshops are offered on topics such as mentoring, managing teams, networking and communication.
3. Mentorship - by renowned investigators through a two-year appointment to one of CIFAR's research programs.
4. Networking - both through research programs, and regular peer meetings of CIFAR Azrieli Global Scholars.
5. Experience - Opportunities are provided to put their expertise, leadership and communication skills into action through CIFAR' activities.

Launched in April 2016, the program aims to support 20 to 25 active Global Scholars per year. Based on performance data, the number of applications received during the evaluation period totalled 1,554. The total number of individual Global Scholars supported during the evaluation period was 72, with the number of active Global Scholars ranging from 18 to 32 per year (see figure 5).

The majority were based in the United States (43%), followed by Canada (32%) and Israel (7%). The remaining 13 Global Scholars were located in distinct countries. The largest share of Global Scholars were associated with research programs in the Information and Matter theme (38%), followed by Individuals and Society (28%) and Life and Health (25%) and Earth and Space (10%).

Participation in the Global Scholars Program mostly resulted in increased soft skills and academic benefits.

The evaluation survey asked current and past Global Scholar members about the program impact on their skills, as well as the benefits of participating in the Global Scholars program. Soft skills were the most common, with 94% reporting either to a 'great extent' or 'very great extent'. This was followed by: research and analytical skills (34%); increased skills related to the ethical, economic, legal, social and other issues related to AI (25%); and technical and computer skills (11%) (see figure 6). Results of post-training satisfaction surveys from Global Scholars provided by CIFAR also showed high levels of satisfaction with training provided.

The main benefits identified were increased networking opportunities with other academics (97%), followed by a competitive advantage to pursue further studies (77%), and increased opportunities to share and/or promote research to the broader scientific community (74%) (see figure 7).

Fewer respondents reported networking opportunities with non-academics (e.g., government and industry) (40%) and increased opportunities to share and/or promote research to the business community (46%).

Findings (Core) – Training

Summer/Winter Schools and integration of students into program meetings were other mechanisms supporting the growth of HQP.

Program documents show that to foster the next generation of researchers, CIFAR Core research programs also engage PhD students and postdoctoral fellows (trainees) through participation in CIFAR summer/winter schools. These are intensive training programs where trainees can learn from distinguished researchers about cutting-edge topics not yet covered in university curriculum. Interviewees noted that Global Scholar and other trainees often lead planning and organization efforts for the summer/winter schools, which provides them with valuable experience.

During the evaluation period, a total of 1,868 trainees attended CIFAR summer/winter schools, with the number of attendees ranging from a low of 300 in 2019-20 to a high of 437 in 2017-18 (see figure 8). Attendance at the summer/winter schools is normally reserved for trainees of program members, or done via open global competition.

The summer/winter schools are normally organized in conjunction with the research program meetings, in order to facilitate trainee participation. Based on interviews and case studies, attendance at these meetings provided trainees with opportunities to present their research through talks and posters, and engage in discussions with research leaders in the field. Interviewees noted participation at these meetings was an excellent opportunity for networking, which was reported as one of the main benefits during the evaluation survey for the Global Scholars.

Interviews noted that the summer/winter schools are beneficial especially in the early years of a research career, for instance enabling researchers to make connections to pursue post-doctoral positions. Interviewees also noted that the summer/winter schools are excellent recruitment tools for Canadian Institutions as they bring together several hundred students (in the past physically, but in the last two years virtually) to Canada to learn more about what the country has to offer.

Case study highlight

The Genetic Network Case Study demonstrated the success of the Global Scholars Program in accelerating the careers of two promising scholars. Both individuals started as Global Scholars and went on to become independent leaders within the Genetic Network Program. Global Scholar Profiles produced by CIFAR highlighted some of their subsequent achievements. This includes one scholar using algorithms and cutting-edge software tools to discover how tumours develop at the molecular level. And for the second individual, developing a deep mutational scanning method, which is a method widely used by laboratories across the world. Both credit their CIFAR participation as greatly impacting their work.

Findings (PCAIS) – AI Talent

The CCAI Chairs Program is an attractive recruitment tool that supports the growth of the AI ecosystem.

The majority of the PCAIS funding ($86.5 million or 69%) is intended to support the CCAI Chairs Program associated with the three national AI Institutes. The CCAI Chairs are awarded on a selection basis for a duration of 5 years, with candidates being nominated by the national AI Institutes and approved by CIFAR. Interviewees noted that access to this funding was crucial to attracting and recruiting AI researchers to Canada as it allows the CCAI Chairs the freedom to carry out research, train and recruit students, and interact with industry.

The funding agreements between CIFAR and the national AI Institutes outline four broad categories (e.g., Graduate Students, Salary supplement to chairholder, Administrative costs, and Minor hardware and equipment) towards which CCAI Chairs can use the funding. Based on a review of expenditures (for one AI Institute) from January 2019 to March 2021, the largest share of funding supported salary and training for graduate students (42%), followed by salary supplement to the chairholder (40%). The funding breakdown confirms what interviewees noted, which was that much of the CCAI Chair funding supports graduate students.

Overall, interviewees felt the PCAIS has been very successful at recruiting and attracting AI researchers, which has created a critical mass that is greatly increasing the attractiveness of Canada for companies and new industrial research groups. Interviewees noted that the environment that has been created at the three national AI Institutes is very attractive, and it has had a reverse brain drain effect, with expatriates returning to Canada as a result of the opportunities that are being created.

Net Talent Migration to Canada has Increased

Canada currently ranks fourth on the AI Skills Migration Index which measures the net migration flows of LinkedIn members with AI skills. Canada advanced 20 spots from 2015 to 2019.

Source: Accenture (2020) PCAIS Impact Assessment Report

The PCAIS has helped recruit and retain top leaders in AI, which is in turn is helping to attract students in AI.

Overall, 39 of 52 PCAIS survey respondents (75%) either agreed or strongly agreed that the PCAIS contributes to the attraction of AI HQPs to Canada, while 38 of 52 respondents (73%) either agreed or strongly agreed that the PCAIS contributes to the retention of AI HQPs. As of March 31, 2021, program data shows there were 105 active CCAI Chairs, with 55 in the recruitment category and 50 under retention (the split being close to 50/50 for each year). Looking at the data by year, the results were 3 below target for 2018-19 (43 vs 46), while the target was met in 2019-20 (80) and exceeded in 2020-21 by 20 (105 vs 85). (See figure 9)

Interviewees noted that the influx of researchers who have come to Canada for CCAI chairs positions is helping to attract and retain talent, and would not have happened without the PCAIS. The Strategy has also provided Canada with improved visibility to attract HQPs. Factors noted by interviewees that are driving attraction/retention were:

• Financial support and the flexibility for how funding is used
• Presence of AI pioneers in Canada like Yoshua Bengio (CIFAR program co-director and current CCAI Chair), Geoffrey Hinton (former CIFAR fellow and advisor) and Rich Sutton (CIFAR associate fellow and current CCAI Chair)
• Critical mass of top quality AI researchers
• Canada being a great place to live
• Opportunity for Chairs to hold joint positions with industry

The evaluation survey showed that the CCAI program has been effective in attracting and retaining AI HQPs, as 22 of 52 CCAI Chair respondents (42%) indicated they considered a similar position abroad, while 17 of 52 (33%) considered positions both in Canada and abroad. In total, only 8 of 52 respondents (15%) only considered a position at CIFAR, and 5 of 52 (10%) considered a similar position within Canada. Along with the growth in CCAI Chairs, the program data and case studies also showed growth in the number of AI students. Looking at the number of students and post- doctoral candidates supervised by AI Institute members, the number has nearly tripled from 504 in 2017-18 to 1,431 in 2020-21, with requests for admission far exceeding the available spots (see figure 10). Student EDI data was not available.

The national AI Institutes and the various training programs are helping to shape the next generation of AI researchers.

Program documents and interviews demonstrated that there were two main mechanisms by which the PCAIS is helping to train the next generation of AI leaders. The first was by enabling students at the national AI Institutes to develop leading-edge expertise in AI by interacting with, and being mentored by, world-renowned experts recruited through the CCAI Chairs program. Interviews also noted that the CCAI funding enables the Chairs to establish research teams, which provides practical experience for students.

The second mechanism was through participation in AI training programs – which include both those delivered by the national AI Institutes, as well as the broader initiatives delivered as part of the CIFAR AI4Good National Training program.

Performance data shows that the number of workshops, conferences and training sessions delivered by the national AI Institutes during the evaluation period totalled 1,529, ranging from a low of 106 in 2017-18 to a high of 680 in 2019-20 (see figure 11). Data regarding the type of events and the number of participants is not available, however case studies provided some insights. A few key examples include:

• Symposium IA Montreal: Annual event held at Mila bringing together 800 experts and academic and industrial professionals interested in fundamental advances and applications of AI.
• Vector Institute – Computer Vision Symposium: Vector has launched a multi-phase industrial-academic collaborative project focusing on recent advances in Computer Vision. This included a 2-day symposium featuring presentations and demonstrations from both industry and academia.
• Ammi – AI Seminar (University of Alberta): The Amii AI Seminar is a weekly meeting where researchers interested in AI can share their research. Presenters include both local speakers from the University of Alberta and visitors from other institutions.
• AI Can: Annual symposium hosted by CIFAR bringing together over 600 participants. The event is open to trainees, researchers, and thought leaders across all disciplines, sectors and industries.

Findings (PCAIS) – Training

The AI4Good National Training Program is inspiring students to consider a career in AI while advancing EDI.

The CIFAR AI4Good National Training Program is aimed at inspiring students at all levels to consider a career in AI. Under this program of activities, CIFAR and its partners engage more than 800 students and trainees every year, from high school students to postdoctoral fellows in Canada and from around the world – including those from underrepresented groups (e.g., two activities specifically for women – AI4All and AI4Good Lab) and participants from low-income countries^{Footnote 40}. These activities also help advance EDI through the topics addressed. For instance, the IVADO-Mila Summer School includes a focus on bias and discrimination.

Program documents identified six distinct annual training events which are part of the National Training Program (see figure 12). In general, there was little performance data available regarding the impact of the events, and contact information was unavailable resulting in an inability to survey participants. Summary data was provided for the AI4Good Lab, which showed that the training program is helping to foster an interest in AI among undergraduate women, with 73% of participants reporting they were more interested in pursuing a career in AI at the end of the event^{Footnote 41}. Program documents for the AI4Good Lab also highlighted a number of women alumni who went on to have careers in AI^{Footnote 42}.

Demand and growth can also be used as proxies to measure success. Program documents showed that there were 1,200 applicants from 75 countries for the 300 spaces at the 2020 Deep Learning and Reinforcement Learning (DLRL) Summer School^{Footnote 43}. Interviewees also noted that the AI4Good lab was expanded in 2020, with Amii hosting a program in Edmonton^{Footnote 44}.

Findings (Core and PCAIS) – HQP & EDI

Training and opportunities provided through CIFAR (Core and PCAIS) are effective at supporting the development of HQP.

The evaluation survey found that 120 of 133 Core respondents (90%)) either agreed or strongly agreed that CIFAR contributes to skill and knowledge development of HQPs, while 42 of 52 PCAIS respondents (80%) either Agreed or Strongly agreed that the PCAIS support enabled the creation of training opportunities for AI HQPs. In general, interviewees also reported that both CIFAR and the PCAIS training activities were effective in contributing to HQP's skills and knowledge development. The PCAIS was noted as effective at getting universities to establish AI graduate programs and organizing seminars and lectures. Other specific aspects identified by interviewees that contributed the most to the development of skills and knowledge of HQP were:

• Training activities, such as Summer Schools and professional development
• Catalyst grants
• Trainee participation at program meetings
• Networking opportunities
• The student competition at AI Can, and
• The flexibility of the CCAI Chair funding which enables professors to bring on graduate students without having to worry about how they will be funded.

CIFAR efforts to support skills and knowledge development of HQP include six key aspects: Networking, Student Competition, Catalyst Grants, Training, Flexibility, Program meetings.

Equity, diversity and inclusion have been identified as a priority for CIFAR.

At the time of the evaluation, gender data was available for both the Global Scholars and the CCAI Chairs for the year 2019-20, with a commitment that data would be updated as new members are onboarded. The available data shows that the more than half of Global Scholars identified as female (58%), while roughly one-in-five CCAI Chairs (19%) identified as female (see figures 13-14)

The evaluation survey showed that 108 of 167 Core respondents (65%) either agreed or strongly agreed that CIFAR supports an equitable, diverse and inclusive groups of HQP, while 40 of 52 PCAIS respondents (77%) agreed or strongly agreed with the statement.

There is strong competition for a small pool of equity-deserving candidates, particularly in AI suggesting a need to diversify the pipeline of students.

Interviews and the data for the Global Scholars suggests that efforts are being made to ensure women have equal access to training programs. The higher representation of females is likely due to the broad pool of candidates which spans many disciplines. With respect to the CCAI Chair data, the share of females is consistent with that found in the literature^{Footnote 45}. One of the challenges identified around EDI for the PCAIS during interviews was the small pool of candidates and high competition for those from equity-deserving groups. For example, while university enrollments in Math, Computer and Information Science increased by 102% between 2008-09 and 2018-19, the gender breakdown has remained fairly constant, with women accounting for 27.0% in 2008-09 and 27.8% in 2018-19.^{Footnote 46} With respect to the CIFAR Summer/Winter Schools and the PCAIS national training program, there is limited data available on the EDI of training participants, although some specific activities under the PCAIS are targeted to women (e.g., AI4All and AI4Good Lab). Data from the 2019-20 DLRL Summer School showed that 31% of participants identified as women, while 12% identified as visible minorities.^{Footnote 47} Data for other years was not available.

With respect to opportunities for improvement, some interviewees noted that more efforts need to target the development of a strong pipeline of diverse students by focussing more on engaging students at the secondary and undergraduate level, in particular for AI. With respect to CIFAR, it was also suggested that more could be done to support funding for post-doctoral opportunities.

Findings (Core and PCAIS) – COVID-19

CIFAR transitioned to a virtual model and contributed to COVID-19 response efforts.

According to interviews, CIFAR was perceived as effective in delivering its programs and activities, including the PCAIS, during the COVID-19 pandemic. The inability to meet in person meant they moved to virtual forums for meetings and events, with some noting environmental benefits (e.g., reduced carbon footprint), and making it easier to attend for people with children. Some also reported that meetings in person may not always be necessary, although interviewees noted it may be challenging for some of the newer research programs, where the relationship between members have yet to be established. The virtual setting also enabled increased participation, for instance the AICan Symposium had over 600 virtual participants in 2021, compared to 371 participants in 2019. Recruitment at the national AI Institutes was also impacted by COVID-19 due to the inability to bring in students or faculty from outside the country.

Document and interviews showed that CIFAR adapted their activities and launched two rapid-response funds and issued a call for proposals to support research collaborations that address COVID-19 (AI and COVID-19 Catalyst Grants and the Manulife-CIFAR Population Health & Well-being Grant Program). The flexibility for research to evolve in both the CIFAR Core and PCAIS models also enabled researchers to shift their focus. Interviews provided many examples of how CIFAR mobilized in response to COVID-19, and the document review found that nearly one-quarter of the CIFAR community have focused their research to address the challenges of COVID-19, with projects spanning COVID-19 Response Leadership, Biology of SARS-CoV-2, Epidemiology & Transmission, Medical Countermeasures, Social & Economic Impact, and Mental Health & Well-being.^{Footnote 48}

For AI, the pandemic provided an opportunity to highlight some of the ways in which AI could be applied to health. Interviews and document review showed the national AI institutes also pivoted their research and applications to address the impacts of the pandemic, tackling important issues such as tracking mental health, detecting and monitoring pneumonia in patients, and contributing expertise to the development of provincial data health platforms.

Supporting the COVID-19 Response

CIFAR's AI and COVID-19 Catalyst Grants are supporting researchers to use machine learning to find drugs that affect the same proteins targeted by COVID-19, in a similar way to how social networks recommend friends. "In social networks, for example, you want to recommend friends based on existing links". "In this case, it's the same intuition. Based on the existing links between the drugs and proteins and the proteins and disease, we try to predict those new links." ^{Footnote 49}

Findings (Core and PCAIS) – Mobilization

Knowledge mobilization events were held in Canada, Internationally and virtually with high impact on knowledge users.

Program documents and interviews showed that through knowledge mobilization activities such as roundtables, lectures, workshops, and events, CIFAR connects the knowledge and outcomes of its Core research programs with thought leaders, both nationally and internationally. Program data shows that between April 2016 and March 2021, CIFAR held a total of 45 knowledge mobilization events. The COVID-19 pandemic impeded CIFAR's ability to hold in-person events, which resulted in a significant shift to fewer and mostly virtual events post 2018-19. During the evaluation period, 20 events were held in Canada, 16 Internationally (US, UK and Netherlands) and 9 virtually (see figure 15). This represents 64% of the established target of 70 events by March 2023 (60 in Canada and 10 internationally), with the target for international events having already been exceeded.

Program data collected from the knowledge mobilization events showed that they are impacting attendees and influencing their work. The percentage of knowledge users who plan to integrate the new ideas gained through CIFAR into their work was high and exceeded the 80% target for four of the five years covered in the evaluation period. The percentage ranged from a low of 78% in 2020-21 to a high of 94% in 2019-20 (see figure 16).

Knowledge promotion and dissemination activities undertaken by CIFAR were perceived as effective.

All interviewees reported that CIFAR is extremely effective in promoting and disseminating research knowledge, however some interviewees noted that within academia there may be greater awareness at the researcher level than the institutional level, while some also noted that CIFAR may have greater recognition internationally than within Canada. With respect to promoting its research outside of academia to the broader public, this was an area where CIFAR was noted as being more active than the tri-councils, however some interviewees noted that CIFAR could do more in this area. There are no formal metrics collected as part of the performance measurement strategy around public engagement, however the CIFAR website does host a number of articles, videos and events which are available to the public. The evaluation survey however showed that blogs (17%) and CIFAR virtual talks (38%) were the activities that were less frequently used to promote and share research, although the general public (78%) was the third most common intended audience of knowledge dissemination activities, following the scientific community (93%) and universities and post-secondary institutions (87%).

70% or 100 of 142 of respondents surveyed by ISED rated the effectiveness of CIFAR's knowledge promotion and dissemination activities as either 'Extremely effective' or 'Very effective'.

Knowledge mobilization activities were supplemented through various publications, which help increase the reach of the research programs, however data quality issues were noted with self-reported data.

Program data showed that during the evaluation period, Core program members reported a total of 11,741 major publications (peer-reviewed journal articles and conference papers, books and book chapters, and publicly released working papers, white papers or policy reports), with the number ranging from 1,440 in 2020-21 to 3,183 in 2016-17. The data however showed some unusually high values that suggested respondents were reporting activity beyond the year in question. The number of research products co-authored by two or more members totalled 2,155, and ranged from a low of 396 in 2020-21 to a high of 457 in 2018-19. Both of these indicators should however be interpreted carefully as they are self-reported and also subject to double-counting given the collaborative nature of the research. A number of interviewees noted though that while CIFAR plays an active role in knowledge mobilization by connecting individual and organizing events, disseminating knowledge is a task for the academic and not CIFAR.

A significant number of peer-reviewed publications have been produced, with many being highly cited.

Program data showed that peer-reviewed articles were the most common type of publication reported, accounting for 65% of all publications. This is consistent with the evaluation survey where Core respondents indicated that the primary dissemination activities to promote their research are peer-reviewed publications (98%), conferences (92%) and lectures (92%) at the international level. Looking at peer-reviewed articles by research program, Quantum Materials and Gravity and the Extreme Universe (the two longest standing programs founded in 1986 and 1987 respectively) accounted for the largest share of articles, followed closely by Bio-inspired Solar Energy (founded in 2014) (see figure 17).

The self-reported publication data obtained from CIFAR was supplemented by an analysis conducted through SciVal, an online bibliometric tool based on the Scopus database which covers over 25,100 titles from more than 5,000 international publishers^{Footnote 51}. Between 2015 and 2020, SciVal showed an output of 1,802 across 370 authors associated with CIFAR. The output for CIFAR-authors was highly collaborative, with 78% involving international collaboration, compared to 52% for Canada overall. The publications by CIFAR-researchers were also highly cited with an average of 27.8 citations per publication compared to 11.2 for Canada overall, and more than one-quarter of the publications (491 or 27%) were in the top 10% most cited publications worldwide, and 5% being in the top 1% most cited. Publications by subject area showed Physics and Astronomy accounted for the largest share (29%), followed by Material Science (14%) and Biochemistry, Genetics and Molecular Biology (10%).

Recommendation 1: ISED's SRS and CIFAR should explore opportunities to improve the rigour of performance measurement data for publications. This could include adding quality assurance reviews to address any abnormal values, and also exploring the use of external data to avoid double-counting joint publications.

Findings (PCAIS) – Mobilization

The PCAIS AI & Society Program is helping mobilize knowledge to advance the responsible development of AI.

One of the objectives of the PCAIS is to understand the societal implications of AI by developing global thought leadership on the economic, ethical, policy and legal implications of advances in AI^{Footnote 52}. Program documents showed this is achieved through the AI & Society Program, which helps with the responsible development of AI (e.g., considering ethical principles and accountability) through four components:

AI & Society Workshops: AI & Society workshops bring together diverse experts from across disciplines, sectors, and borders to address some of the most fundamental challenges posed by AI, including topics related to ethical medicine in AI, the use of AI to address the climate crisis, and the role of AI as an amplifier of inequality for vulnerable populations.

AI Futures Policy Labs: AI Futures Policy Labs engage the public policy community in conversations with experts from other sectors to help them understand, navigate and, respond to the policy opportunities and challenges emerging from the rapid deployment of AI technologies.

Solutions Network: In partnership with the International Development Research Centre (IDRC), CIFAR launched a call for Solution Networks in September 2020. A Solution Network is a global team of cross-sectoral, interdisciplinary experts who design, develop, and implement responsible AI governance solutions.

Building an AI world: Canada was the first country in the world to announce a federally-funded national AI strategy, prompting many other nations to follow suit. CIFAR published two reports detailing the global landscape of AI strategies.

Source: CIFAR (2021) AI & Society. https://cifar.ca/ai/ai-society/

AI & Society events are opening dialogue around responsible AI and exposing participants to new ideas.

The literature review identified that AI poses challenges with respect to ethics and fairness. Some of the more prominent concerns are related to respect for human rights and democratic values, and the transfer of biases from the analogue into the digital world.^{Footnote 53} As the application and use of AI progresses, it is critical to have conversations pertaining to big data, privacy, bias and inclusiveness, and access to AI.^{Footnote 54} Initiatives like CIFAR's AI & Society program help to facilitate these conversations by focusing on building dialogues about individual values, community values, and commercial values and the intersections between them.^{Footnote 55} Program data showed that during the evaluation period a total of 22 AI & Society events were held, with 12 in Canada, 8 internationally (US, UK and France) and 2 virtually in 2020-21 as a result of the COVID-19 pandemic (see figure 18).

In general, interviewees agreed that the PCAIS has been effective in promoting and disseminating research knowledge (e.g., through the AI & Society Program). Data also shows that the majority of participants at AI & Society events indicate exposure to new ideas as a result of their participation, ranging from 84% to 89% per year (see figure 19). Interviews and documents showed that there is still a high amount of public distrust in AI, and CIFAR has actively worked with partners to open the public discourse – for example working with Algora Lab and Canada's Advisory Council on AI, CIFAR hosted a series of public workshops for Canadians to share their views on AI developments. The AI & Society Solution Networks is also partnering with NGOs to pilot initiatives to design and develop responsible and beneficial AI solutions, with the first Solution Network announced in 2021 to explore whether AI data platforms can increase economic opportunities for women agricultural workers in India.^{Footnote 56}

The literature review found that overall, it is important that AI systems are designed with the intent of being transparent about the use of AI and there is accountability for their outcomes. AI systems must function properly and in a secure and safe manner. Continued public funding helps address and mitigate risks associated with the use of AI and balance private interests.

Knowledge mobilization events were supplemented by publications and show collaboration with industry.

CIFAR also produced a number of AI-related publications during the evaluation period in order to complement the knowledge mobilization activities and increase reach. A total of 21 reports were published by CIFAR during the evaluation period largely focused on AI's impact on Society. Examples include Establishing Global Standards for Children and AI, AI-Powered Information Ecosystems & Democracy, Regulation of Defense and Security AI Technologies and Sustainability in the Digital Age.

Program documents showed that the PCAIS knowledge mobilization activities are also supplemented through various publications across the national AI Institutes, which help increase the reach of the research knowledge and outcomes. Program data showed that during the evaluation period, the CCAI chairs reported a total of 2,397 peer-reviewed publications across the national AI Institutes over four years, with the number increasing from 332 in 2017-18 to 880 in 2020-21 (See figure 20). The publication data also shows growing collaboration with industry, with the number of joint publications involving authors from the national AI Institutes and business sectors totalling 479, and increasing from 16 in 2017-18 to 276 in 2020-21. The evaluation survey showed that 36 of 52 PCAIS respondents (69%) were either Satisfied or Very satisfied with CIFAR's overall support for disseminating research.

The PCAIS has been effective at conveying the value of AI technologies to industry and the public sector.

This was noted in interviews and is further evidenced by the growing number of organizations and firms reaching out to the national AI Institutes to find out how AI can support their operations. The growing interest has led to the development of programs at the national AI Institutes such as the Reducing Emissions through Machine Intelligence (REMI) program at Amii which helps SMEs improve their operations^{Footnote 57} , and the Face-to-Face Program at Vector which provides AI guidance for an organization from Vector researchers^{Footnote 58}. Ultimately this has led to the inclusion of new programs at Amii, Mila and Vector as part of the PCAIS renewal to support projects that increase the capacity of Canadian organizations to develop, adopt, use or commercialize AI-based technologies.

The Innovation Superclusters Initiative was also noted as another knowledge mobilization example, with the Scale AI Supercluster aiming to increase the integration of AI across business sectors. This is also another program which is being enhanced through the PCAIS renewal with dedicated resources to support the commercialization and adoption of AI solutions in Canadian ecosystems.

Findings (Core and PCAIS) – Outcomes

CIFAR research has contributed to social and economic benefits for Canadians.

The Literature Review found that as fundamental research accelerates innovation, there is an increased likelihood of economic returns. However, it is often challenging to determine the exact economic gains from fundamental research as it is difficult to separate funding levels between fundamental and applied research. ^{Footnote 59} With that being said, new technology depends on advances in fundamental research, and there is an interdependence between national strengths in industry and strengths in fundamental research. ^{Footnote 60}

Given CIFAR Core's focus on fundamental research, economic and social outcomes are hard to predict. The evaluation survey nevertheless confirmed that CIFAR has helped to foster innovation and provide social and economic benefits to society. With respect to how, the majority of Core respondents (87%) indicated that their research enabled new questions to be asked or ideas to be explored, while 72% indicated that their research made significant advances in fundamental research, with 25% specifically indicating their research influenced/changed university curriculum (e.g., new courses or programs). Respondents indicated that the General public (56%) followed by Private-sector enterprises/industries (39%) and Federal government (39%) were the main beneficiaries of these results, with the nature of the benefits being mostly educational/training (44%), public health (39%) or economic (25%) (See figure 21).

The program data also showed similar results, with the majority of outcomes beyond research being related to influencing university curriculum or public policy, along with some start-ups and commercialisation outcomes. Interviewees also confirmed that the impact has tended to be on how future research is conducted and then indirectly in terms of policies.

Case Study Highlight

The Genetic Networks Case Study provided examples of social and public policy outcomes, including helping establish Bill S-201 (An Act to prohibit and prevent genetic discrimination) and establishing an international genome sequencing consortium.

The PCAIS has strengthened Canada's position in AI and increased the use of AI for social good.

The literature review found that the investments made to date by the federal government in PCAIS have already produced returns, in addition to reinforcing Canada's global standing in AI. There are a number of AI companies which have either emerged or opened offices in Canada, including Integrate AI, ROSS Intelligence, and Google's DeepMind research lab, among many others.^{Footnote 61} Interviewees also noted that on the public policy side, that Canada has been a leader in developing AI policy internationally.^{Footnote 62} For example, CIFAR hosted a roundtable and panel discussion in January 2020 in partnership with ISED and the U.S. Department of State, to consider innovative public policy to develop competitive AI strategies for economic growth, while mitigating potential societal risks for both countries. Other examples included CIFAR helping the Government of Canada Advisory Council on AI with consultations, to reach stakeholders groups and to host round table discussions on the development of national AI policies.

Interviews and case studies also showed increased application of AI to the health care system – with clear examples of how AI has been deployed to help with COVID, for instance to support contact tracing, develop case prediction models, and to help identify drug candidates.

The AI ecosystem has experienced growth since the introduction of the PCAIS.

Interviews, documents and case studies also showed growth in the AI ecosystem since the establishment of the PCAIS. For instance, the commercialization of AI has driven rapid growth in the employment of skilled labour across technology occupations, with employment in technology occupations growing by 3.6% between 2018 and 2019.^{Footnote 63} Further, the document review showed that since the launch of the PCAIS, Canada saw 50 percent growth in foreign direct investment in information and communication technologies (ICT) from 2017 to 2019. Since 2017, over 45 companies have invested in AI research labs in Canada. The AI startup ecosystem also grew significantly and benefited from the presence of large technology companies. In 2019, total AI startup funding exceeded $600 million, up 2.3 times since 2017. ^{Footnote 64}

Both CIFAR Core activities and the PCAIS have contributed to commercial outcomes through intellectual property and spin-off companies.

Regarding commercial outcomes for CIFAR, the evaluation survey showed that during the evaluation period the most common forms of IP applications made by Core members were for provisional patents (92 applications across 15 respondents), followed by patents (72 applications across 22 respondents) and copyright (35 applications across 2 respondents). At the time of the survey, many of the IP applications had been successful, with 49 patents having been granted, as well as 46 provisional patents, and 35 for copyright. Additionally, 10% of respondents indicated they had entered into a licensing agreement based on IP related to their research, and 10% of respondents indicated having created a spin-off companies as a result of their research.

The case studies also highlighted some specific commercial outcomes resulting from the Core research programs. For example, products and technologies developed and commercialized by the Genetic Networks research members included a new algorithm, a genome library and new methods and procedures to discover new therapeutic targets. A number of spin-off companies were also reported, with five companies named as examples – one of which is Deep Genomics, a University of Toronto spin-off company launched in 2015 and led by a CIFAR senior fellow. Media reports show that the company has been very successful, and by 2021, they had raised $180USD million from domestic and foreign investors.^{Footnote 65}

18% or 30 of 167 CIFAR respondents reported at least one IP application was made or granted as a result of their research.

For the PCAIS, the evaluation survey also showed strong results for IP. The most common applications were for patents (29 applications across 13 respondents), followed by provisional patents (6 applications across 4 respondents). At the time of the survey, many of the IP applications had been successful with 16 patents having been granted, as well as 4 provisional patents. Additionally, 9 of 52 respondents (18%) indicated having created a spin-off companies as a result of their research, and 6 of 52 respondents (12%) indicated they had entered into a licensing agreement based on IP related to their research.

The case studies for the national AI Institutes also showed strong outcomes related to the creation of start-up companies. For instance, in 2020-21, 46 start-ups were members of Mila, with 16 having emerged at Mila.^{Footnote 66}

29% or 15 of 52 PCAIS respondents reported at least one IP application was made or granted as a result of their research.

Findings: Efficiency

CIFAR funding provides a flexible approach to research.

The majority of interviewees noted that the CIFAR model is efficient, with no alternative delivery models suggested. Some of the characteristics that contribute to the efficiency were the small size of the organization, and the agility with which it is able to respond and adjust, for instance with the deployment of the COVID Action Fund. Other aspects that were reported as efficient were the open inquiry, which lets the researchers be driven by intellectual stimulation, challenge ideas and advance thinking that comes from the group of researchers. Interviewees also reported that ISED is managing the contribution agreement efficiently, the oversight is appropriate and there is a good working relationship with ISED staff.

The evaluation survey results also substantiated the efficiency of the model. Core respondents who had received funding from the Government of Canada were asked to rate the efficiency of CIFAR processes compared to those where they received funding directly from the government of Canada. The majority of Core respondents indicated that CIFAR's processes are very efficient compared to direct support, most notably with the flexibility available for how funding is used, the flexibility for research to evolve, as well as the application and approval processes.

The majority of CIFAR respondents surveyed by ISED indicated that CIFAR's processes are very efficient compared to direct support, with flexibility on how funding is used (81% or 47/58) and flexibility for research to evolve (79% or 46/58) being the most efficient aspects.

The PCAIS activities have been efficient at facilitating coordination across the PCAIS Institutes.

With respect to the PCAIS, interviewees generally reported the model was efficient, with the small CIFAR team managing the PCAIS able to coordinate effectively across the PCAIS Institutes. The PCAIS Institutes also reported good working relationships with CIFAR.

Overall, CIFAR has demonstrated efficiency in their use of resources to produce outcomes.

The administrative costs for CIFAR Core were calculated as the percentage of expenses for non-program costs relative to total expenses (minus the PCAIS). The annual ratio ranged from a high of 14.3% in 2021 to a low of 9.7% in 2017 with the overall ratio for the period at 11.8%. The administrative costs were generally lower than observed during the last program evaluation (range from 13%-16%, average of 14%) (see figure 22).

The Performance Measurement Strategy for CIFAR identifies a detailed operational cost comparison with other similar Canadian and US organizations as a performance indicator for economy and efficiency, and lists five organizations with similar mandates and activities for possible comparison. Data for four of the organizations was publicly available, with some limitations (i.e., data for the last year was only available for one organization, and only two years for one institute). All available data was included in the analysis, which shows that CIFAR administration costs (minus the PCAIS) are generally lower than the comparators. The organization with the ratio of operational costs closest to CIFAR's (9.1%), had annual revenues that were approximately three times higher than CIFAR. Direct comparisons are however difficult to draw given some differences in operational models. For instance, the three comparators with administrative costs above 20% all maintain physical locations where research can be conducted. One element that explains CIFAR's greater efficiency is likely its virtual model – with researchers conducting work at their home institutions – compared to the greater reliance of physical locations by the other organizations.

Both CIFAR and the PCAIS Institutes have been successful in leveraging funding from other sources.

Program data shows that CIFAR Core has been successful in leveraging funding from other sources. They consistently exceeded their leverage target (1:1) for each year of the evaluation period, with the result ranging from a high of 3.0 in 2017, to a low of 1.3 in 2021 (which only covered a 9 month period due to changes to their fiscal year). The overall leverage ratio for the period was 2.1 (see figure 23).

Interviewees from CIFAR noted that ISED's funding significantly contributes to their ability to secure funding from other sources – and this was also noted as an efficient aspect of the funding model. They indicated that ISED funding is a vote of confidence and that it would be difficult to obtain the same level of support without it. ISED's funding also gives CIFAR credibility when engaging philanthropic

funders or international partners. One interviewee also noted that knowing CIFAR has support from the government enables the organization to support high-risk, high-reward research that in the end are going to allow for big breakthroughs like in AI.

Looking at the total funding by source for the evaluation period, the federal support for CIFAR's Core activities accounted for the largest share of funding (32.0%), followed by Private Sector (26.8%) and Provincial funding (24.2%). Partnerships (research organization and universities) also accounted for 10.7% during the period, however the level of funding on an annual basis has been declining, due primarily to a large agreement coming to an end. That agreement, however, may be renewed. Provincial funding also declined significantly during the period, with the largest contributing factor being the cancellation of a 5-year provincial agreement. Private Sector support has remained fairly constant, while the federal support has increased. The largest growth was observed in investment income, however this source represents the smallest share of funding (6.2%)

Although the PCAIS does not have a leverage target, many interviews also felt that ISED's funding was critical for securing financial support from the provinces, companies and other players. Many interviewees noted it would have been more difficult to secure this funding without ISED support, and that the private sector has invested because of the federal government's financial support for the PCAIS.

CIFAR has made EDI a priority, but the share of underrepresented groups in the research programs is low.

Program documents and interviews showed that CIFAR has taken a number of actions during the evaluation period in order to include EDI considerations, into its organizational structure and the delivery of its programs. Key actions taken include:

• Approval of an EDI Action Plan in April 2020, and hiring an EDI lead to oversee the implementation
• Creating an anti-racism committee and offering training for staff
• Renewing its board membership, with a focus on EDI
• Participating in the ISED 50 – 30 challenge (see box)
• Embedding EDI considerations into the CIFAR 2021 global call for ideas
• Setting internal targets for equity-deserving groups for each research program

The 50 – 30 Challenge at CIFAR

At the organization's level, as of 2021, 60% of CIFAR's Senior Management and Board of Directors identify as woman or gender non-binary and 25% identify as members of equity deserving underrepresented groups, demonstrating positive progress relative to the ISED 50-30 challenge.

In terms of programs and activities, many interviewees noted that CIFAR is doing a good job, and the Global Scholars program was often cited as an example. It was also noted that CIFAR has been more explicit on EDI requirements for its programs and activities – for example, with the recruitment for the global scholars and with the CIFAR 2021 global call for ideas. While program data presented earlier showed strong results on EDI for the Global Scholars and to some extent the CCAI Chairs, the data for the research program members indicates that efforts should continue to ensure that opportunities are provided for researchers from underrepresented groups. The consistent collection of data on EDI of the HQP participating in CIFAR programs and activities would also be beneficial to help assess the extent to which the trainees supported reflect an equitable, diverse and inclusive group of students.

Recommendation 2: ISED's SRS should work withCIFAR to continue its efforts to integrate EDI within the organization and the delivery of its programs. Particular emphasis should be placed on providing more opportunities for underrepresented groups within the research programs and ensuring consistent collection of EDI data for researchers and HQP.

Conclusions

Summary of the Evaluation

Three findings presented the relevance and need for supporting fundamental research and the complementarity of organizations, such as CIFAR in Canada's research environment.

Finding 1 (Core): There is a continued need for the government of Canada to support fundamental research to advance innovation and provide social and economic benefits to Canada. CIFAR addresses this need by convening top researchers from across disciplines and around the world, and creating the conditions that result in successful collaborations.

Finding 2 (PCAIS): The PCAIS addresses the need to support AI research in Canada by enhancing Canada's position as a world leader in AI. There is a continued need for the government of Canada to support AI research for Canada to maintain its leadership and ability to compete globally.

Finding 3 (Core and PCAIS): Support for CIFAR contributes to the government's efforts to invest in science and research, and complements other investments in the research ecosystem. The pandemic further highlighted the importance of a strong, collaborative research ecosystem.

Nine findings demonstrated CIFAR's effectiveness in advancing interdisciplinary knowledge creation, supporting long-term growth of the next generation of researchers and driving societal impact through knowledge mobilization.

Finding 4 (Core): CIFAR has contributed to advancing interdisciplinary knowledge creation through the identification and selection of research programs. CIFAR funding was instrumental for research and helped secure additional funding from other sources. CIFAR supported distinguished researchers and helped improve coordination in Canada's established research strengths.

Finding 5 (Core and PCAIS): CIFAR has helped increase collaboration among researchers domestically and internationally, through its Core research programs and associated activities. Through the PCAIS, CIFAR has also helped advance interdisciplinary and national collaboration, with the PCAIS Institutes (Amii, Vector, Mila) being key to bringing together researchers from academia, industry, and other sectors.

Finding 6 (Core): CIFAR has contributed to supporting the development of the next generation of researchers through the Global Scholars program, the participation of students in various summer/winter schools, and the integration of students into research program meetings.

Finding 7 (PCAIS): The PCAIS has supported the growth of the next generation of researchers through the attraction and retention of Canada-CIFAR AI Chairs, who are in turn attracting students, and teaching the next generation of AI researchers at AI Institutes and through AI training programs.

Finding 8 (Core and PCAIS – linked to recommendation 2): Interview and survey respondents generally agreed that CIFAR is effective at supporting skill and knowledge development of HQP, and support an equitable, diverse and inclusive group of HQP. Opportunities for improvement included more support at the post-doctoral level, further developing the EDI of the talent pipeline by engaging more students at the secondary and undergraduate level, and to ensure EDI data is consistently collected.

Finding 9 (Core and PCAIS): CIFAR's flexibility and agility allowed them to transition some of their activities to help support the pandemic response. The shift to a virtual setting also provided opportunities to increase participation at events. Additionally, the pandemic highlighted opportunities for AI applications to support public health.

Finding 10 (Core and PCAIS – linked to recommendation 1): Through its knowledge mobilization activities and publications, CIFAR has been successful at connecting the outcomes of its research programs with leaders, academics and to some extent, the general public. Data quality issues were however noted with the self-reported input, including high values and double-counting.

Finding 11 (PCAIS): The PCAIS has been successful at mobilizing knowledge to support the development of responsible AI. Knowledge dissemination activities have also been effective in sharing the outcomes of AI research more broadly, resulting in an increase in programs and funding to support the deployment of AI technologies across various sectors and industries.

Finding 12 (Core and PCAIS): Evidence shows that CIFAR has contributed to public policy changes, influenced university curriculum and created start-up companies and commercial products.

Three findings highlighted the extent to which the CIFAR funding model is an efficient approach in strengthening the Canadian research environment.

Finding 13 (Core and PCAIS): The CIFAR funding models is efficient. CIFAR was noted as responsive, provided flexibility for how funding is utilized and allowed research to evolve. The PCAIS has also helped advance national coordination among the PCAIS Institutes.

Finding 14 (Core): CIFAR administrative costs have decreased compared to the five years prior. Administrative costs for CIFAR were generally lower relative to the comparators identified. Funding from ISED has helped leverage additional investments. For CIFAR, financing from other sources fluctuated during the evaluation, with some declines in provincial support.

Finding 15 (Core and PCAIS – linked to recommendation 2): While CIFAR has taken some concrete actions in order to integrate equity, diversity and inclusion into its organizational structure and the delivery of its programs, the need remains to provide more opportunities for under-represented groups within the research programs. Opportunities to improve collection of EDI data should also be explored.

Two recommendations were produced in the evaluation, stemming from the assessment of performance and efficiency, and supported by multiple lines of evidence.

Performance

Recommendation 1:ISED's SRS and CIFAR should explore opportunities to improve the rigour of performance measurement data for publications. This could include adding quality assurance review to address any abnormal values, and also exploring the use of external data to avoid double-counting joint publications.

Efficiency

Recommendation 2:ISED's SRS should work withCIFAR to continue its efforts to integrate EDI within the organization and the delivery of its programs. Particular emphasis should be placed on providing more opportunities for underrepresented groups within the research programs and ensuring consistent collection of EDI data for researchers and HQP.

Appendix A: Logic Model

The evaluation of ISED funding to CIFAR (Core and PCAIS) was based on the outcomes in the logic model below.

Appendix B: Expenses by Research Program

-	2016	2017	2018	2019	2020	2021	TOTAL	(%)
Life & Health	1,829	1,963	2,221	2,292	1,994	1,004	11,303	14.3
Fungal Kingdom: Threats & Opportunities	0	0	0	0	641	474	1,115	-
Genetic Networks	611	664	678	656	(2)	34	2,641	-
Humans and the Microbiome	602	720	886	840	814	496	4,358	-
Molecular Architecture of Life	616	579	657	796	541	0	3,189	-
Individuals & Society	3,743	4,444	4,276	4,297	3,055	2,227	22,042	27.9
Brain, Mind and Counsciousness	679	881	1,634	1,897	1,239	752	7,082	-
Boundaries, Membership and Belonging	0	0	0	0	528	420	948	-
Child and Brain Development	751	1,119	1,002	894	681	594	5,041	-
Innovation, Equity & the Future of Prosperity	0	0	0	0	482	460	942	-
Institutions, Organizations and Growth	745	851	833	816	68	0	3,313	-
Social Interactions, Identity and Well-Being	832	777	47	(2)	0	0	1,654	-
Successful Societies	736	816	760	692	57	1	3,062	-
Earth & Space	1,584	1,500	977	824	1,230	889	7,004	8.9
Earth 4D: Subsurface Science & Exploration	0	0	0	0	477	378	855	-
Gravity & the Extreme Universe	799	826	809	814	753	511	4,512	-
Integrated Microbial Biodiversity	785	674	168	10	0	0	1,637	-
Information & Matter	3,357	3,879	3,982	3,978	2,480	1,734	19,410	24.6
Bio-inspired Solar Energy	776	885	808	745	589	403	4,206	-
Learning in Machines and Brains	796	1,035	1,211	1,250	743	440	5,475	-
Quantum Information Science	788	846	829	849	502	344	4,158	-
Quantum Materials	997	1,113	1,134	1,134	646	547	5,571	-
Program Support	3,254	4,287	3,705	3,195	2,650	2,167	19,258	24.4
Total Knowledge Creation Expenses	13,767	16,073	15,161	14,586	11,409	8,021	79,017	100.0

Appendix C: CIFAR Research Highlights

Since its inception in 1982, CIFAR has had a significant impact on the way the world understands such critical issues as the Earth's evolution, early child and brain development, artificial intelligence, building stronger societies, and many others. The examples below highlight some of the major contributions of CIFAR researchers.

Appendix D: End Notes