Overview
In 2018, the Digital Research Infrastructure (DRI) Strategy was created to consolidate responsibility for four DRI pillars to two organizations – CANARIE and the Digital Research Alliance of CANADA (DRAC) – to streamline access and deliver more open and equitable access to advanced computing and big data resources; and improve availability of more digital tools and services and increase optimum use of DRI resources.
Under the DRI Strategy, CANARIE and DRAC support four pillars:
- Digital network that allows researchers to send data between institutions and around the world (CANARIE).
- Advanced Research Computing (ARC) or supercomputers used for big data analysis and simulation (DRAC).
- Research Software (RS) that allows researchers to access, share and utilize data (DRAC).
- Research Data Management (RDM) ensures the management, publication and preservation of datasets to promote data sharing, security, accessibility and reuse (DRAC).
Budget 2018 committed $572.5M to the DRI Strategy to ensure that Canadian researchers have the digital tools they need to support scientific excellence, which included $375M towards DRAC, $50M to Host Site institutions towards the ARC Expansion Program, and $145M towards CANARIE (from 2019-20 to 2023-24). An additional $37M was allocated to CANARIE via a Contribution Agreement amendment for 2024-25.
The evaluation was conducted using multiple methodologies and sought to assess program relevance, performance and efficiency, as well as DRAC implementation. The evaluation covered the April 1, 2018, to March 31, 2023, period.
Findings
Relevance
The evaluation found that the need for DRI is increasing. The DRI Strategy, delivered in collaboration by DRAC and CANARIE, is helping address this demand, as many institutions do not have the resources to address it on their own. However, further work is required by DRAC in terms of addressing the DRI needs of the ecosystem. CANARIE and DRAC established activities to address a need for enhanced cybersecurity capabilities and alignment, but increased collaboration is required to minimize overlap, ensure continued alignment, and address emerging cybersecurity risks.
Centralized DRI services are supported by:
- CANARIE's National Research & Education Network (NREN), which includes 13 NREN partners
- DRAC's ARC Platform, which includes 5 Regional Organizations and 5 National Host Sites
Performance
DRAC Implementation
60% of DRAC's eligible activities were either completed (27%) or on target to be completed (33%), as of December 31, 2022.
DRAC implementation took longer than planned. Challenges were experienced in implementation, transitioning DRI responsibilities to DRAC, navigating the delivery model, and tracking performance to effectively measure success. These challenges were exacerbated by the COVID-19 pandemic.
While DRAC has consulted stakeholders and improved the effectiveness of its governance, engagement, and relationships over time, more adjustments and meaningful consultations are needed to better respond to stakeholder needs and increase awareness of roles, responsibilities, relationships, and funding mechanisms.
There have been some improvements to computing capacity and in 2023 DRAC began work on the refresh of aging infrastructure across national host sites. However, the pace of renewal does not meet the growing needs of the research community. Renewal of ARC infrastructure has been slow, partly due to difficulty obtaining match funds in a timely and predictable manner.
Network
CANARIE's network is the backbone of the NREN. Over the evaluation period, CANARIE enhanced the speed of the network by 72% (surpassing target) and increased the number of institutions with access to the network by 7.6% (slightly under target), including adding Nunavut Arctic College as an NREN partner.
DAIR
CANARIE's DAIR cloud program supports private sector innovation by providing SMEs with free access to cloud services.
DAIR led to cost savings and allowed SMEs to develop products quicker/more effectively.
- 100% increase in the number of CANARIE supported software services/tools created/enhanced per year over the evaluation period.
- 57% increase in the number of SMEs using DAIR services over the evaluation period.
CANARIE launched the Cybersecurity Initiatives Program (CIP) in 2020. By 2022-23, there were 213 Canadian institutions participating to increase their security capabilities (target of 205). DRAC created a cybersecurity framework to mature the ecosystem's cybersecurity capabilities and reduce risks.
There is a large variety of DRI training dispersed unevenly across institutions. DRAC conducted a needs assessment and determined highly qualified personnel (HQP) training to be the greatest need for researchers. DRAC is developing supports and platforms to address training gaps (e.g., cybersecurity training program). CANARIE training and support through the NREN is effective and accessible.
Top 5 areas identified in the needs assessment where DRI training is most required:
- Data Analysis
- Best practices on software development/coding
- Using High Performance Computing systems
- Machine Learning
- Archiving and preserving data and other digital objects
Efficiency
CANARIE spending and its allocation among activities was less than planned during the evaluation period.
CANARIE administration costs as % of expenditures ranged from 14% to 17%, but is expected to be below 15% by the end of 2023-24.
Due to implementation delays, DRAC was unable to use the allocated funding within approved timeframes.
- $96M: Unspent funding
DRAC administration costs as % of expenditures averaged 19% between 2019-20 to 2022-23, but are now trending to be below 10%.
CANARIE and DRAC formed a joint EDI working group and developed a Joint EDI Strategy to embed EDI measures within processes, communications, and recruitment.
Lessons Learned
The establishment of a centralized, national program for DRI infrastructure within a decentralized environment was a highly complex undertaking, which required a significant amount of time and resources to implement. These challenges were further exacerbated by the disruptions stemming from the COVID-19 pandemic, as well as supply chain issues, which further impacted implementation timelines. For future initiatives involving similarly complex environments, further planning and assessment of the ecosystem and organizational landscape could help to better understand and effectively mitigate potential implementation challenges and risks before they arise.
Recommendations
SRS will work with the third-party organization(s) to monitor the implementation of the following recommendations:
- CANARIE and DRAC should explore opportunities to enhance CANARIE's information sharing and collaboration with DRAC on cybersecurity-related activities and further clarify roles to minimize overlap in the development of cybersecurity initiatives.
- There is a need for DRAC to enhance funding recipients' level of understanding by further clarifying roles and responsibilities and improving communication. DRAC should explore approaches to facilitate further meaningful engagement of funding recipients and other stakeholders (e.g., via a comprehensive stakeholder engagement strategy).
- DRAC should develop a consistent and comprehensive approach to reporting on the outcomes and indicators of the Performance, Evaluation, Risk and Audit Framework.
- DRAC should explore how to increase the timeliness and predictability of match funding, including examining lessons learned and approaches used by other organizations.
- DRAC should explore additional tools or measures to minimize negative impacts on the research software community stemming from the funding transition period for research software funding and support.
- DRAC should develop a strategy or approach to improve the ARC resource allocation process to better meet the needs of researchers.