Evaluation of CANARIE

This publication is available online at www.insertwebaddresshere.ca.

To obtain a copy of this publication or an alternate format (Braille, large print, etc.) please fill out the Publication Request Form at www.ic.gc.ca/Publication-Request or contact:

Web Services Centre
Innovation, Science and Economic Development Canada
C.D. Howe Building
235 Queen Street
Ottawa, ON K1A 0H5
Canada

Telephone (toll-free in Canada):
1-800-328-6189
Telephone (Ottawa):
613-954-5031
TTY (for hearing-impaired):
1-866-694-8389
Business hours:
8:30 a.m. to 5:00 p.m. (Eastern Time)
Email:
info@ic.gc.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Innovation, Science and Economic Development Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Innovation, Science and Economic Development Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Innovation, Science and Economic Development Canada.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the Web Services Centre mentioned above.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Innovation, Science and Economic Development Canada, 2018.

Cat. No. Iu4-263/2019E-PDF
ISBN 978-0-660

Aussi offert en français sous le titre Évaluation du CANARIE.

Audit and evaluation branch

March 2019

Presented to the Performance Measurment and Evaluation Committee on March 20, 2019


List of abbreviations and acronyms

CFI
Canada Foundation for Innovation
DAIR
Digital Accelerator for Innovation and Research
DRI
Digital Research Infrastructure
ICT
Information and Communication Technology
ISED
Innovation, Science and Economic Development
OECD
Organization for Economic Co-operation and Development
R&E
Research and Education

List of figures


List of tables


Executive summary

Program overview

CANARIE is a not-for-profit organization established in 1993 to support Canadian-based research, discovery and innovation by providing advanced high-speed networking capability that enables researchers to manage and exchange large volumes of data. It is a key player in Canada's digital research infrastructure (DRI) landscape.

ISED Funding to CANARIE was renewed in Budget 2015 for a total of $105 million over five years and is used to support the delivery of the network and CANARIE's other programs: 1) the National Research and Education Network, 2) Canadian Access Federation, 3) Research Software, 4) Research Data Management, and 5) Digital Accelerator for Innovation and Research. Funding is also used to support Research Data Canada, which operates as a separate organization with its own governance structure.

Evaluation objectives, scope and approach

The objectives of this evaluation were to examine CANARIE in accordance with the Financial Administration Act and the Treasury Board Policy on Results to inform funding renewal. The evaluation examined the relevance, performance, and efficiency and economy of ISED's contribution to CANARIE. It was conducted by ISED's Audit and Evaluation Branch and covered the period from April 1, 2014 to March 31, 2018.

Findings

There is a demonstrated need for the federal government to continue supporting CANARIE in the delivery of its core service, a national high-speed network, which facilitates data sharing and collaboration among the Research and Education (R&E) community and which is essential for research and innovation.

CANARIE has been effective at expanding the access to and utilization of a world-class R&E network across Canada, including the facilitation of knowledge creation and collaboration via the Research Software program. CANARIE has also helped facilitate the development of information and communication technology (ICT) products and services through its Digital Accelerator for Innovation and Research (DAIR) program. However, there is limited evidence on the extent to which CANARIE has helped accelerate ICT commercialization in Canada.

CANARIE has efficiently delivered the network using a federated model involving provincial and territorial delivery partners, resulting in cost savings, cooperation, and coordination.

CANARIE's governance structure is clear and effective for CANARIE's core service, which is to deliver the national network. However, in regards to research data management the role of CANARIE's governing body is unclear.

Recommendations

The evaluation findings led to the recommendations noted below.

Recommendation 1: Clarity in Mandate

ISED should consider reviewing CANARIE's eligible activities, particularly those pertaining to research software, research data management, and DAIR, to ensure that their objectives are clear and unique relative to those of other organizations in order to simplify the DRI landscape for the R&E community.

Recommendation 2: Measuring Impact

Consideration should be given to a review of expected outcomes in order to ensure that CANARIE's programs are aligned with its objectives. ISED should review CANARIE's logic model to determine whether additional indicators and an associated data strategy should be developed to monitor CANARIE's impact on the commercialization of information and communication technology products and services.


1.0 Introduction

This report presents the results of an evaluation of Innovation, Science and Economic Development Canada's (ISED) contribution to CANARIE. The evaluation assessed the relevance, performance, and efficiency and economy of the CANARIE contribution program.

1.1 Context

Program description

CANARIE is a not-for-profit organization established in 1993 to support Canadian-based research, discovery and innovation by providing advanced high-speed networking capability that enables researchers to collaborate and exchange large volumes of data. CANARIE uses over 31,000 kilometres of fibre-optic cable to connect over one million users at over 715 Canadian institutions. This capability is critical, as the commercial Internet cannot technically or cost-effectively handle this high volume of research traffic.

Through the CANARIE network, researchers also have access to over 100 international peer networks in 80 countries to facilitate global research collaboration. In addition to the network, CANARIE supports the development of collaborative research platforms and components that accelerate discovery and enable broad use of digital infrastructure.

Expected Results: Through the activities undertaken with ISED's funding, CANARIE is expected to, among other things:

  • Maintain and expand a world-class research and education (R&E) network that will accelerate research and development in Canada;
  • Facilitate the adoption and use of advanced research network and digital technologies; and
  • Assist Canadian-based firms by providing a test bed for their innovative products and services.

ISED has been funding CANARIE's operations since the organization's inception in 1993Footnote 1. As of 2018, funding is used to support CANARIE's network and five programs across three program areas that deliver benefits to the scientific research community, private sector, higher education institutions, and Canadians:

Network Operations
National Research and Education Network:
provides interprovincial and international connections through a national backbone network made up of ten provincial and two territorial Regional Advanced Networks to connect Canadians to national and global data, tools, and collaboration opportunities. CANARIE also supports cybersecurity initiatives by working with partners to strengthen the security level of the network.
Canadian Access Federation:
offers identity and access management solutions for Canadian research and education institutions by providing users with Wi-Fi connectivity and content access using the log-in credentials of their home institution.
Technology Innovation
Research Software Program:
supports and promotes the development of software tools that simplify researchers' access to big dataFootnote 2 and digital infrastructure, such as high performance computing and storage resources, to accelerate discovery.
Research Data Management Program:
responds to a community-identified need to fund the development of software components and tools to enable Canadian researchers to adopt best practices in research data management. The CANARIE Board approved this program in October 2017.
Private Sector Innovation
Digital Accelerator for Innovation and Research (DAIR) Program:
supports cloud technology by providing Canadian entrepreneurs and small businesses with free cloud-based computing and storage resources that help speed time to market by enabling rapid and scalable product design, prototyping, validation and demonstration. DAIR does not provide funding support. CANARIE also contributes to the ecosystem that is working to accelerate the growth of Canada's information and communication technology (ICT) sector by supporting existing and emerging testbed initiatives.

Through CANARIE, ISED funding also supports the activities of Research Data Canada, which operates as a separate organization with its own governance structure.

Key player in Canada's digital research infrastructure

Digital research infrastructure (DRI) is the collection of connectivity, computing power, research software, and storage services needed to support data-intensive and computationally-intensive research. CANARIE, the Canada Foundation for Innovation (CFI) and Compute Canada are key players in Canada's DRI system. CANARIE's core service in the DRI system is to operate a national ultra-high-speed backbone network and work closely with provincial and territorial network partners to enable data-intensive research.

1.2 Program resources

Since 1993, the Government of Canada has committed a total of $634.5 million to support CANARIE through a series of three- to five-year contributions and grants. Most recently, funding to CANARIE was renewed in Budget 2015 for a total of $105 million over five years. Table 1 provides a detailed breakdown of ISED funding allocated to each fiscal yearFootnote 3.

Table 1: ISED Funding of CANARIE from 2015-16 to 2019-20

Fiscal Year Funding Amount

2015-16

$15.0 million

2016-17

$20.0 million

2017-18

$22.3 million

2018-19

$20.4 million

2019-20

$27.3 million

Total

$105.0 million

There are three program areas under CANARIE, as well as administration, which receive funding from ISED. The portion of the total contribution to be allocated to each program area and administration is outlined in Table 2.

Table 2: Program Areas and Programs under CANARIE

Program Area and Program Total Allocation Requirement from 2015-16 to 2019-20

Network Operations (Network, National Research and Education Network Program, Canadian Access Federation)

At least $60 million

Technology Innovation (Network,
Research Software Program, Research Data Management Program)

Up to $22 million
(including up to $1.5M to Research Data Canada)

Private Sector Innovation (Digital Accelerator for Innovation and Research)

At least $4 million

Administration (e.g. salaries, professional services, occupancy costs, marketing, etc.)

Up to $15.75 million

ISED's operating costs related to the monitoring of CANARIE are approximately $60,000 per year.

1.3 Target population and stakeholders

CANARIE's primary target population is the research and education (R&E) community in Canada, including researchers and scientists working at universities, colleges, research hospitals, and non-profit research institutions.

CANARIE also has a number of stakeholders, which includes other players that either participate in Canada's DRI landscape alongside CANARIE or are direct/indirect beneficiaries. These key stakeholders are presented in Figure 1 below.

Figure 1: Key Stakeholders in Canada's DRI LandscapeFootnote 4

Text version

Figure 1 is a diagram illustrating the seven key stakeholders in Canada's digital research infrastructure landscape:

  1. DRI Service Providers. This group includes:
    • Advanced Research Computing (Compute Canada);
    • Research Software (Canada Foundation for Innovation);
    • Research Data Management (Canadian Association of Research Libraries, Research Data Canada);
    • Information Technology Management (Canadian University Council of Chief Information Officers);
    • Platform for Development, Testing and Commercialization of Network and Communication Technologies (Centre of Excellence in Next-Generation Networks);
    • and Cloud Technology Resources (commercial providers).
  2. DRI Users. This group includes:
    • international R&E networks;
    • research and post-secondary institutions;
    • small- and medium-enterprises;
    • and science-based federal departments and agencies.
  3. Strategic Direction in DRI. This group includes:
    • ISED and Leadership Council for Digital Research Infrastructure.
  4. Funding Support for R&E. This group includes:
    • ISED;
    • Tri-Agencies (Natural Sciences and Engineering Council, Canadian Institutes for Health Research, and Social Science and Humanities Research Council);
    • provincial and territorial governments;
    • and Canada Foundation for Innovation.
  5. CANARIE. This group includes:
    • stakeholders in Research and Education Networking; Research Software;
    • Research Data Management;
    • Information Technology Management;
    • platforms for commercialization;
    • and cloud technology resources.
  6. Regional Advanced Networks. This group includes:
    • the ten provincial and two territorial networks.
  7. Private sector stakeholders.

1.4 Logic model

The logic model in Figure 2 depicts the activities to be pursued, the outputs to be produced and the expected outcomes resulting from the delivery of the program and how program objectives are achieved.

Figure 2: CANARIE Logic Model

Text version

Figure 2 depicts the logic model of CANARIE. In this logic model, there are three activities leading to their respective outputs (of which there are seven). These seven outputs lead to three levels of outcomes – five immediate outcomes, three intermediate outcomes, and one ultimate outcome.

The three activities are:

  1. Network Operations: this includes the Network Program, NREN Program, and Canadian Access Federation Program;
  2. Technology Innovation: this includes the Network Program, Research Software Program, and Research Data Program; and
  3. Private Sector Innovation: this includes the DAIR Program and Centre of Excellence in Next Generation Networks Program.

The first activity (Network Operations) leads to three outputs:

  1. "Effective and efficient National Research and Education Network", which leads to the immediate outcome of "Canada's R&E community has increased access to a high-capacity network in support of research, education and innovation".
  2. "Projects supporting network capacity, capability, and security", which leads to the three immediate outcomes of "Canada's R&E community has increased access to a high-capacity network in support of research, education and innovation", "Canada's R&E community has increased access to reusable, leading-edge software tools, services and platforms", and "Greater recognition and adoption of CANARIE-supported tools in Canada and internationally".
  3. "International network connections and collaboration", which leads to the immediate outcome of "Greater recognition and adoption of CANARIE-supported tools in Canada and internationally".

The second activity (Technology Innovation) leads to three outputs:

  1. "Next generation network technologies", which leads to the immediate outcome of "Greater recognition and adoption of CANARIE-supported tools in Canada and internationally".
  2. "Software interfaces, applications and tools", which leads to the immediate outcomes of "Canada's R&E community has increased access to reusable, leading-edge software tools, services and platforms", and "Greater recognition and adoption of CANARIE-supported tools in Canada and internationally".
  3. "Projects and forums to develop data standards and services", which leads to the immediate outcome of "Greater recognition and adoption of standards and best practices that support effective data management".

The third activity (Project Management and Monitoring) leads to one output:

  1. "Infrastructure for cloud technology and network testbed", which leads to the immediate outcome of "Canadian firms have improved access to and increasingly utilize infrastructure and services to support innovation".

The immediate outcomes "Canada's R&E community has increased access to a high-capacity network in support of research, education and innovation" and "Canada's R&E community has increased access to reusable, leading-edge software tools, services and platforms" lead to the intermediate outcome "Expanded access to and utilization of a world-class research and education network by Canadian R&E communities".

The immediate outcomes "Greater recognition and adoption of CANARIE-supported tools in Canada and internationally" and "Greater recognition and adoption of standards and best practices that support effective data management" lead to the intermediate outcome "Enhanced opportunities for collaborative knowledge creation and innovation through increase adoption and use of advanced digital technologies".

The immediate outcome "Canadian firms have improved access to and increasingly utilize infrastructure and services to support innovation" leads to the intermediate outcome "Growth of innovative ICT products and services and accelerated ICT commercialization in Canada".

The three intermediate outcomes lead to the ultimate outcome "Enhanced research, knowledge creation, and innovation strengthens Canada's economy".


2.0 Methodology

This section provides information on the evaluation objectives, scope and approach, the evaluation issues and questions addressed, the data collection methods, and limitations.

2.1 Evaluation objectives, scope and approach

Objectives

An evaluation of CANARIE is required under the Financial Administration Act. The objectives of this evaluation were to examine CANARIE in accordance with the Treasury Board Policy on Results and address issues identified by program management in order to inform funding renewal of the program.

Scope and approach

The evaluation was conducted in-house and covered the last four fiscal years, from April 2014 to March 2018. It examined the relevance, performance, and efficiency and economy of ISED's contribution to CANARIE. In the assessment of performance, the evaluation focused on the intermediate outcomes identified in CANARIE's 2017 logic model, as the last evaluation in 2015 had already examined the immediate outcomes.

2.2 Evaluation issues and questions

The evaluation addressed the following questions.

Relevance

  1. To what extent is there a continued need for CANARIE in Canada's digital research infrastructure (DRI) landscape?

Performance

  1. To what extent has CANARIE expanded access to and utilization of a world-class research and education network (R&E) by the Canadian R&E community?
  2. To what extent has CANARIE enhanced opportunities for collaborative knowledge creation and innovation through increased adoption and use of advanced digital technologies?
  3. To what extent has CANARIE supported growth of innovative information and communication technology (ICT) products and services and accelerated ICT commercialization in Canada?

Efficiency and Economy

  1. How efficiently and economically is CANARIE being delivered?
  2. Does CANARIE's governance structure support efficient program delivery?

2.3 Data collection methods

Multiple lines of evidence were used to address the evaluation questions. The data collection methods included a literature review, document review, data review, interviews, and case studies.

Literature review

A literature review was conducted to support other lines of evidence in the assessment of relevance, and efficiency/economy of program delivery. It included academic literature and studies related to the need for CANARIE-like programs, as well as the models and approaches used for network delivery in other countries.

Document review

A document review was performed to gain a thorough understanding of the program and to provide insights into relevance and performance. The review included ISED program foundational documents; government priority-setting documents; and CANARIE annual reports, corporate plans and supporting documents from the CANARIE website.

Data review

CANARIE's performance and survey data were reviewed and analyzed to assess the achievement of expected intermediate outcomes. Financial, administrative and operational data were also reviewed and analyzed to support the assessment of the efficiency and economy of program delivery.

Interviews

Interviews were conducted to gather in-depth information related to the relevance, performance, and efficiency and economy of the program. The interviews were semi-structured in nature and included 45 participants (38 individual interviews and three group interviews) across a range of stakeholder categories in DRI, specifically:

  • Program management and delivery partners;
  • International R&E networks;
  • Innovation partners and DRI players;
  • Ultimate recipients of CANARIE funding; and
  • CANARIE network users.

Case studies

Four case studies were conducted to provide a more detailed perspective on CANARIE's achievement of expected intermediate outcomesFootnote 5. The case studies included eight interviews with project stakeholders as well as a review of CANARIE's project-related documents. One project was selected from each of the following CANARIE programs: Canadian Access Federation, Digital Accelerator for Innovation and Research, National Research and Education Network and Research SoftwareFootnote 6.

2.4 Limitations and mitigation strategies

The evaluation noted the following limitations. The mitigation strategies used to address each of the challenges in the evaluation are also described below.

Program Data Availability:
Data on funded projects was limited for CANARIE's Research Data Management program given that the CANARIE Board only approved this program in October 2017 and proposals to begin funding projects only began in 2018-19. To mitigate this, the evaluation relied on qualitative information gathered from interviews when examining this program and focused only on assessing its relevance as well as design and delivery (i.e. the performance of the Research Data Management program was not evaluated).
Respondent Bias:
Many interviewees were involved in program design and delivery or are direct beneficiaries. As such, the findings were at times biased towards more favorable program outcomes. The evaluation mitigated this by interviewing participants across five different stakeholder categories. These categories included members from the R&E community as well as representatives from organizations that exist in similar areas of the DRI landscape.

3.0 Findings

3.1 Relevance

3.1.1 To what extent is there a continued need for CANARIE in Canada's digital research infrastructure (DRI) landscape?

Key Finding: CANARIE addresses the unique need for a national high-speed network to support research collaboration and facilitate knowledge sharing in order to stimulate innovation. The need for CANARIE is also demonstrated through increased traffic volume on the network. CANARIE uses its network to deliver programs in multiple areas in Canada's Digitial Research Infrastructure landscape, but there are other organizations that support these areas as well.

There is a demonstrated need for the federal government to continue supporting CANARIE in its delivery of a national high-speed network to the R&E community in Canada. Interviews and the literature review, including international literature, have emphasized that high-speed networks are in increasing demand to facilitate data sharing and collaboration among the R&E community and are essential for research and innovationFootnote 7, Footnote 8, Footnote 9.

Further, Budget 2018 noted that improved technologies, such as faster networking, allow for new opportunities to address scientific challenges. Improved access to these technologies will strengthen Canada's reputation as a global leader in science, research and innovation. According to an OECD report, the ICT sectorFootnote 10 remains a key driver of innovation, accounting for over one-third of total patent applications worldwideFootnote 11.

An analysis of CANARIE's traffic data indicated that traffic volume grew 149% between 2013-14 and 2017-18, from 92,000 terabytes to 229,107 terabytes, as a result of the increased demand for CANARIE's high-speed network by the research community. An analysis of CANARIE's performance data also suggested the need for CANARIE, given that from 2014-15 to 2017-18 the number of users increased across all of its programsFootnote 12.

The analysis of performance data found that the National Research and Education Network program, which supports network infrastructure projects across Canada, observed the smallest total growth in the number of connected institutions during this time, estimated at 14.0%. However, this can be attributed to the expansion of the network and the fact that, according to interviews, a majority of Canadian institutions now have established connections as a result of the completed network infrastructure projects.

Interview findings further suggested that stakeholders view CANARIE as addressing a national need through its National Research and Education Network program which is delivered in collaboration with provincial and territorial network delivery partners. It is perceived by stakeholders that Canada's network would be fragmented without CANARIE serving as a national coordinating body. These interviews, along with the document review, revealed several examples of the need for CANARIE in supporting network infrastructure projects. For example, CANARIE has helped establish connectivity to the national research network in Churchill to facilitate research in the North. It has connected data centres to ocean cables to support real-time data collection. CANARIE has also facilitated Canada's participation in global science initiatives such as the Large Hadron Collider project in Geneva.

According to the document review, CANARIE's mandate has broadened in order to deliver programs through its network that support multiple areas in DRI. However, the majority of stakeholders view the network as being CANARIE's core function. Although CANARIE's programs are viewed favourably by stakeholders, certain programs are not perceived as being unique to CANARIE or as critical to the R&E community given that other players can provide support in these areasFootnote 13.

The literature review, document review and interviews confirmed that many players exist in Canada's DRI landscape. In areas such as research software, cloud infrastructure (via the Digital Accelerator for Innovation and Research (DAIR)), and research data management, interviewees perceived that there was some duplication in objectives between CANARIE and other organizations in these spaces, which has created confusion within the R&E community when seeking support (e.g. Canada Foundation for Innovation funding for research software).

CANARIE programs in specific areas of DRI:
Digital Accelerator for Innovation and Research:
provides access to cloud resources for start-up companies and small-and-medium enterprises to develop and test their products in a low-risk environment.
Research Software:
provides funding and support to the R&E community in the development of research software.
Research Data Management:
launched in 2018-19, it provides funding for the development of software and tools that promote research data management.

Recommendation 1: ISED should consider reviewing CANARIE's eligible activities, particularly those pertaining to research software, research data management, and DAIR, to ensure that their objectives are clear and unique relative to those of other organizations in order to simplify the DRI landscape for the R&E community.

3.2 Performance

3.2.1 To what extent has CANARIE expanded access to and utilization of a world-class research and education (R&E) network by the Canadian R&E community?

Key Finding: CANARIE has expanded its network access, speed and data capacity across Canada to support the research and education community. Its network is comparable to other OECD countries in terms of the availability and use of current technologies.

Access to and utilization of a world-class R&E network
Improved Network Speed

CANARIE has been effective at expanding the access to and utilization of a world-class R&E network across Canada.  An analysis of CANARIE's performance data found that the network's data transfer speed per kilometer of fibre optic cable has been consistently increasing for the last three years to support the traffic from the R&E community. There was 47.0% total growth in the capacity of the network between 2014-15 and 2017-18.  Stakeholders indicated that CANARIE's network has grown large enough to support data transfer of 100 Gigabits (Gbps) per secondFootnote 14. Evidence from interviews also suggested that CANARIE's network speed has enabled teleconferencing to support course offerings through remote learning in northern universities in Canada.

Increased Network Access

Network access through CANARIE's 'eduroam', a service offered under the Canadian Access Federation to provide students and researchers with access to Wi-Fi at participating institutions without having to obtain special credentials, has also increased. Log-ins to the network using the 'eduroam' service grew from 71 million in 2014-15 to 224 million in 2017-18. Aside from the service being provided through the Canadian Access Federation, evidence from the case studies confirmed that the City of Mississauga has also adopted and successfully implemented 'eduroam' through a service agreement with CANARIE. In its first year of implementation, between May 2017 and May 2018, Mississauga observed over 1 million Wi-Fi connections. It is expected that the annual usage of 'eduroam' will increase given that it has been broadcasted to over 60 public-facing facilities across the city.

Expanded Network Capacity

Evidence from the document review, data review and interviews suggest that CANARIE has supported its provincial and territorial delivery partners in infrastructure projects to build and expand their networks. It has supported 28 projects through the National Research and Education Network program (e.g. the program has helped build connections for two research institutions in Nova Scotia and helped address gaps in the network in northern New Brunswick). It has also established connections to institutions in Northern Ontario and connected to the regional network in Manitoba that connects all post-secondary and K-12 schools in Manitoba. Recently, it connected institutions in Alberta that did not previously have connection to the network.

Achieved Global Reputation

Interviews with both Canadian and international stakeholders revealed that Canada is viewed as being comparable to other countries such as the UK, Germany and the Netherlands in terms of the technology used to support and deliver CANARIE's network. For example, it was noted that Canada is pursuing a pre-commercial 5G network which can be built on the national network. Findings from interviews with international stakeholders further revealed that Canada is recognized as having a world-class R&E network, as evidenced by the fact that CANARIE is asked to play a leadership role in supporting other countries in the development of their R&E networks.  CANARIE has provided guidance to the R&E network in Brazil on strategic planning and it also participates in an international working group of R&E networks that is examining more efficient and less costly ways to better serve the R&E community.  However, it was also noted in interviews that the challenge for Canada's R&E network in comparison to others, such as the US or in Europe, appears to be in regards to the country's vast geography and relatively small population size. Both the literature review and interview findings noted that this has an impact on the cost structure of Canada's network, flexibility in growth, and capacity utilization per square kilometerFootnote 15.

3.2.2 To what extent has CANARIE enhanced opportunities for collaborative knowledge creation and innovation through increased adoption and use of advanced digital technologies?

Key Finding: CANARIE's network has facilitated collaboration in the research and education community to support knowledge creation and innovation.

Collaborative knowledge creation and innovation
Knowledge Sharing on the Network

The use of CANARIE's network, along with the services available through it, has supported collaboration across a wide range of scientific disciplines within the R&E community in Canada, including bioinformatics, astronomy, physics, and the digital humanities. For example, the network and its ability to connect to scientific instruments that are used to collect data has been essential in enabling Ocean Networks Canada to stream real-time data collected through its underwater cable observatory in British Columbia. The network has also enabled the institution to collaborate and share ocean data with marine institutions on the east coast of Canada. It was noted by stakeholders that CANARIE's 'eduroam' service also facilitates knowledge creation since it provides the R&E community with the opportunity to conduct research and access the network at institutions from coast-to-coast.

Interviews found that the 'peering'Footnote 16 service offered through the network, which enables access to services and tools from third-parties that have agreements with CANARIE to exchange traffic, has also been important for the R&E community. These offerings would otherwise be accessible only through commercial providers which would come at a cost to the user.

Collaborative Development of Research Software

CANARIE's Research Software program has facilitated knowledge creation and supported the collaborative development of platforms which, once developed, are made available to the R&E community through the network. According to interviews, the sharing of research software enables the R&E community to leverage pre-existing software when developing similar software across disciplines. The data review found that CANARIE has funded 11 projects, through the Research Software program, between 2015-16 and 2017-18 to support the development and/or maintenance of services, platforms, and tools created or enhancedFootnote 17. For example, one researcher reported in an interview that CANARIE supported the optimization of their platform. The platform involves common metadata standards to support data-interoperability and re-use, with a range of functionalities for supporting collaboration.

Although there are other players in DRI that support research software development, the case studies found evidence suggesting that CANARIE's support has facilitated collaboration. In 2016-17, the Research Software program supported the development of 'Adnotare', a web-based annotation platform that is unique in its use of current technology and flexibility, with bilingual features and information sharing capabilities between researchers. The platform also provides the R&E community with a free alternative for annotation in comparison to similar software, which was reported in interviews as costing up to $200 per user.  Its development was a collaborative effort between research groups at various post-secondary institutions in Quebec. The platform has been adopted by three post-secondary institutions for projects in criminology, Alzheimer's disease, and the digital humanities. It is also being used in Australia for research in environmental disaster recovery as well as for research in Germany.

3.2.3 To what extent has CANARIE supported growth of innovative information and communication technology (ICT) products and services and accelerated ICT commercialization in Canada?

Key Finding: CANARIE has supported the development of innovative information and communication technology (ICT) products and services by facilitating collaboration via its network across the research and education community. CANARIE has also helped facilitate the development of ICT products and services by providing free cloud resources through its Digital Accelerator for Innovation and Research (DAIR) program. While stakeholders view DAIR positively, there is limited evidence on the extent to which CANARIE has helped accelerate ICT commercialization in Canada.

Commercialization of ict products and services
Low-Risk Development and Testing using Cloud Resources

CANARIE has facilitated the development of ICT products and services by providing free cloud resources to small-and-medium enterprises and start-ups through its Digital Accelerator for Innovation and Research (DAIR) program and via the Centre of Excellence in Next Generation Networks programFootnote 18. DAIR is part of CANARIE's Private Sector Innovation program area, which accounted for 3.5% of CANARIE's total program expenses between 2014-15 and 2017-18. As participants in the DAIR program, small-and-medium enterprises are able to develop and test products in a low-risk environment before pursuing commercialization.

For example, Rebel TechnologiesFootnote 19 was developed as a software solution to modernize data analysis in the automotive sector.  The findings from the case study indicated that the DAIR program provided a platform for the start-up company to test the server, cloud-end applications and product development.  The case study also found that access to cloud resources through the DAIR program enabled cost savings for the start-up company as it allowed them to reallocate their venture capital funding towards the development of human resources and expertise. It also facilitated moving towards a production cloud environment.  It was noted in interviews that the advantage of CANARIE's cloud resources is that it allows for flexible movement of data which helps simplify product development.  It also offers the ability to test multiple scenarios to refine product design whereas free trials of commercial cloud resources only offer the ability to test one scenario.  Commercialization of Rebel Technologies is expected in early 2019. The target reach for Rebel Technologies post-commercialization is to be in 10,000 service shops within three years across North America, Western Europe, and Australia.

A review of performance data found that there has been an increase in the number of unique small-and-medium enterprise users of CANARIE's infrastructure and tools available through the DAIR program, rising from 380 in 2014-15 to 1,062 in 2017-18. Although this provides evidence of the use of the DAIR program, its impact on the growth in the commercialization of ICT products and services that were developed and/or tested using its cloud resources is limited. The review of performance data did not find evidence on the number, type and quality of products and services developed and/or commercialized.

According to interviews, CANARIE has supported the development of ICT products.  For example, its support has enabled the development of a prototype earthquake warning system which has secured further investment from the provincial government to operationalize and implement it in British Columbia. Interviewees also noted that a survey is administered by CANARIE to DAIR participants, but its completion is voluntary and there are currently no reporting requirements for the program. This has made it difficult for CANARIE to monitor the success of ICT products and services supported by DAIR. Although stakeholders view the program positively, evidence to support the assessment of the effectiveness of the DAIR program was limited, as was evidence to confirm the extent to which CANARIE has helped accelerate ICT commercialization in Canada.

Indirect Impacts on the ICT Sector

Aside from the DAIR program, evidence from interviews suggested that CANARIE has contributed to the ICT sector by facilitating collaboration through its network across the R&E community. For example, one stakeholder reported a growth in the number of intellectual property patents and innovative commercialization opportunities at their institution as a result of collaboration. Another stakeholder reported that collaboration has opened doors for their institution and led to private sector partnerships in artificial intelligence and autonomous vehicles. By facilitating access to collaboration opportunities with other stakeholders in the DRI landscape, for example, through its annual National Summits, members of the R&E community stated that CANARIE has also indirectly supported the scaling of their projects.

Recommendation 2: Consideration should be given to a review of expected outcomes in order to ensure that CANARIE's programs are aligned with its objectives.  ISED should review CANARIE's logic model to determine whether additional indicators and an associated data strategy should be developed to monitor CANARIE's impact on the commercialization of information and communication technology products and services.

3.3 Efficiency and economy

3.3.1 How efficiently and economically is CANARIE being delivered?

Key Finding: CANARIE's model has been efficient in delivering a national network and has involved provincial and territorial delivery partners. A federated model has resulted in increased cost savings, cooperation and coordination.

Provincial and territorial delivery partners

CANARIE has demonstrated its ability to deliver the network efficiently by using a national model which involves provincial and territorial delivery partners. Interviews found that having an arms-length organization manage the relationships with provincial and territorial network delivery partners has allowed for effective communication, collaboration and cooperation.

International literature also suggests that the 'federation' model is an efficient approach to delivering national networks. For example, existing research data infrastructure is fragmented across disciplines and Member States in Europe, and work is underway to develop a pan-European federation of data infrastructure built around a federating coreFootnote 20. The Australian Government also recognizes that nationally coordinated eResearch infrastructure would strengthen the country's position in the research environment and ensure that Australian research can accelearate innovation and foster engagement between researchersFootnote 21.

It was also noted that the increased cooperation over time across provincial and territorial network delivery partners has resulted in CANARIE establishing the Directed Funding program at the beginning of the 2015-2020 mandate under the National Research and Education Network program for collaborative projects. As of 2017-18, one collaborative project (involving the provincial and territorial network delivery partners) has been successfully completed – the PerfSONAR upgrade project.

Aside from the provincial and territorial network delivery model delivered by CANARIE, alternative models for network delivery exist. Interviewees suggested possible alternative network delivery models including: a model delivered by the federal government instead of a third-party organization, a model delivered by a commercial provider, and a regional model delivered by each province and territory independently. However, although alternative models were proposed, stakeholders indicated that none would be as efficient as CANARIE's 'federation' model and the alternative models proposed would result in a loss of cost savings, cooperation, and coordination.

Collaboration and coordination across provincial and territorial network delivery partners:

The PerfSONAR upgrade project was CANARIE's first collaborative project across 13 National Research and Education Network delivery partners.

PerfSONAR is an open-source software tool used to monitor connectivity on the CANARIE network from point-to-point across the country. It signals which parts of Canada are connected to the network and enables standardized and efficient reporting on network performance.

PerfSONAR provides the R&E community with a high-level dashboard of the health of the national network as well as those for specific locations of interest. This helps researchers identify viable network locations to perform their work and avoid those that are experiencing issues.

CANARIE operations

In assessing the efficiency of CANARIE's operations, the evaluation examined its funding sources, direct and indirect expenditures, cost-recovery mechanism, and spending by program area.

Funding Sources, Expenditures, and Cost-Recovery

CANARIE allocates its funding from ISED across three program areas: Network Operations, Technology Innovation, and Private Sector Innovation. The majority of CANARIE's operating expenses (97.5%) were covered by funding from ISED between 2014-15 and 2017-18. Direct program spending as a proportion of total expenses has been relatively stable during this same time period, representing 85% of the ISED contribution. CANARIE has demonstrated its ability to diversify its program offerings while keeping its overheadFootnote 22 costs fairly constant.

CANARIE spending is in line with its allocations for administration (15% of the ISED contribution) as well as in all three program areas (Network Operations, Technology Innovation, and Private Sector Innovation).

CANARIE has implemented cost-recovery to further support its operations and remaining expensesFootnote 23. A review of financial data found that it implements user fees as part of its cost recovery for the Network, Canadian Access Federation and DAIR programs. In-kind funding is also provided through Research Software and Research Data Management program participants, while the National Research and Education Network, Research Software and DAIR programs receives matching funds. The total average cost recovery per year was approximately $3.2 million between 2014-15 and 2017-18.

Spending by Program Area

A review of CANARIE's financial data revealed that although overall spending is aligned with ISED funding allocations, the proportion of CANARIE's spending on its program areas can vary from one year to the next. For example, in 2015-16, $11.3 million (or 88.2% of $12.8 million in total program spending) was for network operations, while $943,000 (or 7.4%) was for technology innovation (i.e. Research Software, Research Data Management, Research Data Canada, and the Joint Security Program). In 2017-18, even though spending on network operations increased to $14.9 million, as a percentage of total program spending it decreased (to 78.1%). For technology innovation, spending increased to $3.4 million (or 17.6% of total program spending) in 2017-18.

The Research Software program, which was launched in 2007, is CANARIE's key program under the technology innovation spending area. Under this program, CANARIE committed $4.8 million to fund a total of 11 projects between 2015-16 and 2017-18 to support the development and/or maintenance of services, platforms and tools created or enhanced (an average of over $436,000 per project).

CANARIE is not the only provider of funding support in the area of research software, however, which has caused some confusion among stakeholders in terms of the perceived focus of CANARIE versus that of other players. An analysis of CANARIE's 2017 research software survey data found that only 15% of respondents reported that they used CANARIE funding to support their current research software development work. Further, only 18% of these respondents relied solely on CANARIE funding to support their projects, with 51% of respondents reporting that they also received funding from at least one of the tri-AgenciesFootnote 24 and 17% receiving funding from the Canada Foundation for Innovation (CFI).

However, only two federally-funded organizations (CANARIE and the CFI) explicitly support the development of research software through software development calls, and the focus of the software development programs of these organizations are different. CANARIE's strategic objective in its Research Software program is to support the development and reuse of high-quality research software, to reduce the amount of research grant funding that is used to develop software, and to support the development of a Canadian community of research software developers. The CFI's Cyberinfrastructure initiative challenges the community to submit projects that bring together a community of researchers from across the country who share similar challenges linked to the availability of research data. Projects should address an existing or emerging challenge for this community through the development of new tools and applications or novel ways of organizing and using research data that would enhance the community's capacity to conduct leadingedge researchFootnote 25.

While there is some limited overlap between CANARIE and the CFI in the projects submitted, and those that were eventually funded by the CFI, these initiatives are complementary, as the CFI funding (at approximately $2 million per project), supports the development of large platforms that address the needs of a large community of researchers in a given domain. By contrast, CANARIE's Research Software program focuses on the development of high-quality research software that can be used and reused in a wide range of research project workflows across multiple research disciplines. The CFI and CANARIE have committed to, and continue to, collaborate on research software projects to ensure exactly this kind of complementarity.

Interviewees noted that CANARIE investments allowed targeted research software development efforts to support and enhance significant CFI investments made in the early 2000s, such as McGill University's neuroimaging software, cBrainFootnote 26. In effect, the CFI provided the foundational infrastructure and CANARIE extended their software capabilities.

3.3.2 Does CANARIE's governance structure support efficient program delivery?

Key Finding: CANARIE has a clear governance structure to support the efficient delivery of a national network. It has made recent improvements to its National Research and Education Network Governance Committee to ensure efficient and economic use of resources. However, the role of CANARIE's governing body in the area of research data management is unclear.

The Board of Directors is CANARIE's governing body and is currently comprised of both private sector representatives and members of the R&E community. Although it does not hold official observer status on the Board, ISED is regularly invited by CANARIE to observe the Board meetings. The Board is responsible for overseeing the delivery of CANARIE's programs as well as supporting the organization's strategic direction. According to interviews, the governance structure is clear and effective for CANARIE's core service which is to deliver the national network.

CANARIE uses stakeholder committees to provide advice on improving the design and delivery of its programs. In recent years, it made improvements to the committee for the National Research and Education Network program by assigning a full-time staff resource to coordinate and manage the Directed Funding program to improve efficiency in project decision-making for network infrastructure projects; ensure that decisions are consensus-driven, selected based on priority and needs across provinces and territories; and that decisions are put forward to the Board in a timely manner for approval.

In regards to research data management, stakeholders are unclear as to what role the Board plays in comparison to other players in this areaFootnote 27. It was noted in interviews that the CANARIE Board is viewed as providing advice but not direction in research data management and that setting the direction in this area requires extensive coordination, collaboration and consultation with other playersFootnote 28. Further, stakeholders indicated that the role of CANARIE's governing body in regards to Research Data CanadaFootnote 29 is also unclear given that it has a separate governing body to set its strategic direction but receives funding for its operations and activities from CANARIE.


4.0 Conclusions and recommendations

4.1 Conclusions

Relevance

  • CANARIE addresses the unique need for a national high-speed network to support research collaboration and facilitate knowledge sharing in order to stimulate innovation. The need for CANARIE is also demonstrated through increased traffic volume on the network. CANARIE uses its network to deliver programs in multiple areas in Canada's Digitial Research Infrastructure (DRI) landscape, but there are other organizations that support these areas as well.

Performance

  • CANARIE has expanded its network access, speed and data capacity across Canada to support the research and education community. Its network is comparable to other OECD countries in terms of the availability and use of current technologies.
  • CANARIE's network has facilitated collaboration in the research and education (R&E) community to support knowledge creation and innovation.
  • CANARIE has supported the development of innovative information and communication technology (ICT) products and services by facilitating collaboration via its network across the research and education community. CANARIE has also helped facilitate the development of ICT products and services by providing free cloud resources through its Digital Accelerator for Innovation and Research (DAIR) program. While stakeholders view DAIR positively, there is limited evidence on the extent to which CANARIE has helped accelerate ICT commercialization in Canada.

Efficiency and economy

  • CANARIE's model has been efficient in delivering a national network and has involved provincial and territorial delivery partners. A federated model has resulted in increased cost savings, cooperation and coordination.
  • CANARIE has a clear governance structure to support the efficient delivery of a national network. It has made recent improvements to its National Research and Education Network Governance Committee to ensure efficient and economic use of resources. However, the role of CANARIE's governing body in the area of research data management is unclear.

4.2 Recommendations

The evaluation findings led to the recommendations noted below.

Recommendation 1: Clarity in Mandate

ISED should consider reviewing CANARIE's eligible activities, particularly those pertaining to research software, research data management, and DAIR, to ensure that their objectives are clear and unique relative to those of other organizations in order to simplify the DRI landscape for the R&E community.

Recommendation 2: Measuring Impact

Consideration should be given to a review of expected outcomes in order to ensure that CANARIE's programs are aligned with its objectives. ISED should review CANARIE's logic model to determine whether additional indicators and an associated data strategy should be developed to monitor CANARIE's impact on the commercialization of information and communication technology products and services.