Evaluation of Business + Higher Education Roundtable

From: Innovation, Science and Economic Development Canada

May 2024

Table of contents

• Program Context
• Evaluation Context
• Finding
• Summary
• Recommendations
• Appendices

1.0 Program context

1.1 Program description

The Business + Higher Education Roundtable (BHER) is a non-partisan, not-for-profit organization that brings together the industry and post-secondary education (PSE) sectors across Canada to create opportunities for young Canadians who have undertaken post-secondary education, as well as boost innovation, and collaboration between Canadian industries and academia. It was founded in 2015 by the Business Council of Canada (BCC) to create "a thriving Canada where everyone has an opportunity to reach their potential".^{Footnote 1} BHER has received a non-repayable contribution from the Government of Canada since 2019.

In Budget 2019, the Government of Canada made a commitment that within 10 years, every young Canadian who wants a work-integrated learning (WIL) experience should be able to have one. By providing $798.2M into the WIL ecosystem, it enabled the expansion and creation of existing and new WIL programming across Canada. As a funded recipient, BHER is supporting this initiative through three priorities:

Connect and convene with purpose : BHER promotes the value of WIL to educators, businesses and students, by serving as a leader in the conversation on skills, talent and innovation in Canada through engagement with stakeholders.

Skills and talent : BHER builds and maintains a network of partners for the development and expansion of WIL experiences that support student capacity-building and talent development in Canada, as well as build expertise in the future of work.

Innovation and research partnerships : BHER strengthens the WIL ecosystem through the development of innovative tools and approaches to WIL as well as connecting members to existing initiatives. (Research activities are not funded by the Government of Canada contribution.)

What is WIL?

Work-integrated learning (WIL), is an experiential learning opportunity for students that allows them to have real-world work experiences related to their field of study. There are many approaches to the delivery of WIL as well as its typology. For this evaluation, it will be classified into traditional WIL and untraditional WIL.

Traditional WIL examples:

• Co-operative (co-op) Education
• Field placement
• Internships, Apprenticeships
• Entrepreneurships
• Institutional Partnerships
• Mandatory Professional Practice/Clinical Placement

Untraditional WIL examples:

• Innovative WIL
  • (e.g., industry projects, hackathons, micro-placements, etc.)

Detailed descriptions on different types of BHER WIL can be found in Appendix F.

1.2 Program governance

Innovation, Science and Economic Development Canada (ISED) is responsible for the Contribution Agreement (CA) with BHER. The Industry Sector provides oversight to BHER and ensures that they meet their quarterly and annual reporting requirements.

BHER is governed by a Board of Directors composed of 8 business and PSE sector leaders that represent the organization's pan-Canadian and cross-sectoral membership.

BHER's WIL programming is supported by ISED's contribution which is redistributed to ultimate funding recipients who deliver WIL experiences.

1.3 Program funding

Budget 2019 committed $17 million in funding over three years to create 44,000 WIL experiences for students. The program was renewed in Budget 2022, which allocated an additional $17.1 million over three years. Of the $17.1M in renewed funding, $5.3M was allocated in 2022-23 for a total of $21.4M over the four-year evaluation period.

As per the Contribution Agreement, ISED's BHER funding supports two pillars of activities:

• Create WIL Placements
  • Generating new partnerships: planning call cycles and requests for proposals (RFPs), negotiating master funding agreements (MFAs), and managing all onboarding activities.
  • Managing partnerships: monitoring operations and reporting, assisting partners with WIL program delivery, ecosystem-building in collaboration with R&D team.
• Strengthen the WIL Ecosystem
  • Developing new tools and resources for employers, as well as other stakeholders like practitioners, educators, and professors.
  • Launching 5 virtual regional hubs to create opportunities for businesses and PSE institutions to collaborate on WIL-based solutions to emerging talent challenges, share best practices, and access professional learning opportunities.*
    • *This component was added to the 2022-23 CA and there was insufficient data to be scoped into this evaluation.

1.4 Program delivery model

Primary funding recipient: BHER receives funding from ISED through the CA. BHER announces a Request for Proposal (RFP) on projects for the delivery of WIL.

Ultimate funding recipients: Employers^{Footnote 1*}, PSE institutions, as well as local business organizations and not-for-profit organizations, (e.g. Chambers of Commerce) who prepare proposals in response to the RFP and negotiate a Master Funding Agreement (MFA) to receive BHER funding to focus on the development of WIL experiences. BHER funding does not pay for WIL experience compensation.

The employers, PSE institutions and local business organizations form relationships with each other in the development of WIL experiences.

Beneficiaries: Students who attend PSE institutions gain access to WIL experiences and employers^{Footnote 2*} who host students through WIL experiences as a result of WIL partnerships established with PSE institutions and local business organizations.

Program funding for uncompensated and compensated WIL

BHER funding to ultimate funding recipients supports a variety of activities for the creation of WIL experiences. Typically, BHER funding supports staff (i.e., direct labour) who create WIL programs with unsubsidized WIL experiences.

When WIL experiences are created, they may be compensated or uncompensated, depending on the type of WIL experience. For example, traditional co-op is often a compensated hourly wage full-time on-the-job experience, whereas an industry project is often an uncompensated academic assignment meant to fulfill a real-world industry tasking.

BHER funding cannot directly fund WIL experiences, so staff, who develop compensated WIL experiences, must draw funding from other external funding sources outside of BHER, e.g.:

• Federal, provincial, or municipal grants;
• Non-profit fundraising; and,
• Employer self-funding.

2.0 Evaluation context

An evaluation of ISED's funding to BHER is required every five years under the Financial Administration Act.

The objective of this evaluation was to assess the relevance, performance and efficiency of ISED funding to BHER.

The scope of the evaluation included only activities found in the CAs. The evaluation covered all ISED funding from April 1, 2019, to March 31, 2023.

The evaluation was conducted by the Audit and Evaluation Branch at ISED. A results-based approach was used to examine the achievement of expected outcomes for BHER (Appendix B).

As a newer program, BHER underwent significant changes throughout the evaluation period and during conduct, which limited data availability and the ability to assess longer-term outcomes and impact. Where relevant, the evaluation noted changes that are underway.

2.1 Evaluation questions

Relevance

To what extent is there a continued need for BHER?

Are the objectives of BHER in alignment with existing and emerging ISED strategic objectives and federal government priorities?

Performance

To what extent has BHER contributed to the strengthening of the WIL ecosystem?

To what extent has BHER contributed to post-secondary students having access to WIL experiences?

To what extent has BHER contributed to employers providing WIL experiences to students?

To what extent has BHER contributed to students developing in-demand skills?

To what extent has BHER contributed to employers having access to new talent and recruitment opportunities?

To what extent has BHER contributed to participating organizations being able to innovate and grow?

Efficiency

To what extent is the BHER delivery model an efficient and effective approach to achieving its objectives?

2.2 Evaluation methodology

The following lines of evidence were used (details in Appendix C):

1. Literature and document review
2. Performance, Administrative and Financial Data Review
3. Case studies
4. Online surveys
5. Interviews
6. Questionnaire

3.0 Findings

The needs of employers, PSE institutions, and students

Literature and interviews highlighted the three key stakeholders of quality WIL experiences and their needs.^{Footnote 2Footnote 3}

Employers:

• Access to populations with identified in-demand skills and relevant workforce experience.
• Increased understanding and awareness on the delivery and administration of WIL.
• Talent pipelines directly linked to post-secondary

PSE institutions:

• Develop and maintain relationships with employers to ensure that curricula reflect and adapt to in-demand skills.
• Capacity building for internal WIL programming connected to courses to meet the growing demand of WIL.
• Support international students with WIL experiences.

Students:

• Gain a practical application of learning through real-world on the job experiences.
• Improved job confidence and job prospects due to valuable work experience acquired during studies.
• Improved transition from academia to the professional workspace.
• Development of technical, emotional and professional skills.

Impacts of BHER and WIL experiences

Evidence noted how BHER specifically fulfilled these needs through programming and services directly, (MFA partnerships and WIL Hub Tools), or indirectly through the WIL experiences that were delivered to students, as well the improved job outcomes that students receive as a benefit of having done a WIL experience.^{Footnote 4}

BHER's unique WIL contribution

BHER is unique in providing unsubsidized WIL placements, meaning that ultimate funding recipients could not use BHER funding for WIL experience compensation. Instead, BHER funding was used for capacity building and developing WIL, as discussed in Finding 7. Stakeholders noted that BHER's flexible funding allowed them to more directly support their organization's WIL programming needs rather than broadly funding specific student placements.

BHER's CAs allowed organizations to create WIL experiences for international students. This is unique compared to other organizations such as Student Work Placement Program and the Innovative WIL Initiative (I-WIL), for which students must be a Canadian citizen, permanent resident, refugee, and legally able to work in Canada according to the laws of the province or territory of the student.^{Footnote 5} BHER addresses this gap in WIL programming which also incentivizes international students to gain Canadian work experience and contribute to Canada's economy and productivity, as discussed in Finding 5.

The WIL ecosystem is complex

In budget 2019, the Government of Canada invested $798.2M over 5 years to ensure that every young Canadian who wanted a WIL experience should be able to get one within 10 years.^{Footnote 6} This contribution has supported the creation and expansion of WIL programming across the country.

WIL programs noted by the evaluation in the WIL ecosystem^{Footnote 3*}

• SWPP
• BHER
• Mitacs

Funded by I-WIL: ^{Footnote 7}

• Actua: Future Skills Program
• Canadian Institute for Exponential Growth: Wavemakers
• Canadian Mobility and Aerospace Institute: Mini-WILs
• Co-operative Education and Work Integrated Learning (CEWIL)
• Colleges & Institutes Canada: Virtu-WIL
• Riipen: Level UP
• UCalgary: Experience Ventures
• Venture for Canada

Stakeholders noted that WIL programming was complementary of each other and emphasized the importance of their presence to help support the commitment to 100% WIL for post-secondary students in Canada. However, many acknowledged that the funding injection in the ecosystem increased competition between programs as they sought to find their place.

The evaluation found that many of the programs noted had similar mandates or objectives, (i.e., creating WIL by bringing together academia and employers). Stakeholders indicated that this may be duplicative and confusing for those seeking grants to fund initiatives due to different rules, requirements and terminology regarding WIL according to different organizations.

A Potential Convener

WIL ecosystem stakeholders highlighted BHER as a potential key national convener of employers, academia and other WIL programs. Interviewees stated that BHER had the benefit of being closely tied to major Canadian employers through its partnerships (e.g. the BCC) as well as being less tied to bureaucracy and tradition typically found in academia.

It was noted that their unique position as a primary recipient of federal government funding allowed them to be directly plugged into topics concerning in-demand employer skills and highlight potential skill gaps at a national level. BHER indicated that this was of interest to them and that they were planning on holding convening events in 2024. It is too soon to conclude whether BHER could effectively act in this role.

Recommendation 1

Contribution Agreements (CA): Timeline

Over the evaluation period there were two CAs with ISED; the first CA was signed in 2019 and amended in 2020 to launch a virtual WIL challenge in response to the COVID-19 pandemic (due to the cancellation of many WIL experiences across Canada), while the second CA was signed in 2022 when the program was renewed. The nature of the eligible activities in both agreements was broad to enable flexibility in their approach to the delivery of WIL experiences and the development of partnerships.

Although BHER consistently remained in scope over the period, the evaluation noted that BHER's activities evolved over this timeframe, partially in response to these agreements and changes in the WIL ecosystem identified through BHER research and engagement, (i.e., new WIL programs and gaps noted by employers).

• In 2019, BHER focused on developing partnerships with larger employer organizations based on the premise that they were able to pay for the development of their own WIL placements with BHER's unsubsidized WIL support.
• In 2020, BHER's amended agreement led to the Canada Comeback Challenge (C3), an online skill-building student competition hosted by BHER during the COVID-19 pandemic.
• BHER's focus changed from larger employers to Small and Medium Enterprises (SMEs), which typically had more difficulty establishing WIL experiences according to interviewees and the 95% of employers that responded to the BHER survey.
• In 2022, BHER pivoted to more partnerships with PSE institutions where WIL-specialized staff lead the development of opportunities with employers, as discussed in Finding 7.

Alignment with ISED priorities

The shift in BHER partnership focus resulted in a decrease of partnerships with private sector organizations.

ISED officials indicated that, while these shift in activities were within the scope of the CAs, BHER could improve its alignment with departmental and industry sector priorities by increasing the percentage of private sector employers engaged directly in the delivery of WIL as host organizations.

Contribution Agreements: Impacts

The evaluation identified several impacts associated with BHER's CAs:

Performance measurement

• The definitions of indicators and data presented in annual reports changed between funding agreements which limited consistent long-term trend analysis of the program. (e.g., geographic delineations, size of employer organizations, employer engagement, etc.)
• In the first CA, indicators did not have enough of an employer focus to demonstrate program impacts. While improvements were made in the second CA employer data is only reported to ISED every two years instead of a yearly basis.
• There was a lack of capacity-building focused indicators to demonstrate medium and long-term impacts of BHER to the growth and sustainment of the WIL ecosystem.

Data collection

• A misunderstanding of reporting requirements in the CA of 100% response rate for student surveys was identified as a significant challenge by BHER staff and the majority of ultimate funding recipients.
• BHER staff noted that the response rate data collection requirements put undue burden on WIL delivery partners and resulted in smaller organizations to withdraw from MFA negotiations due to their lack of capacity to fulfill these obligations.
• ISED officials acknowledged the high response rate and are working with BHER to have a more balanced reporting response rate.

Timeframes

• BHER indicated that the duration of the three-year CA limited their ability to conduct relationship building and to support the stability of WIL programming since planning for renegotiation of a CA may take a year. This results in a "start-stop" cycle of WIL programming as they focus on CA negotiations and interrupts program activities, such as capacity building, which requires considerable effort and time to establish.
• Stakeholders stated that the timeframe contributed to funding reprofiling, since BHER could not always spend their money within the fiscal year. The evaluation noted three reprofiles over the evaluation period totalling approximately $7.7M.
• The misalignment of the federal fiscal calendar and the academic calendar impacts the ability of BHER to launch their RFP, negotiate MFAs, and develop WIL programming for the academic year within a short time frame before the required reporting timeline on student participation. As a result, BHER is reporting figures to ISED, however there may be additional results to observe following the close of the fiscal year.
• BHER indicated an MFA may last for 18 months and that developing and launching a WIL program can take up to a year to establish. This would leave approximately six months to collect data, which may be drawn from one to two academic terms. This limits the ability to observe capacity building impacts to the WIL ecosystem, students, and employers, over a longer period of time.

BHER'S WIL approach demonstrates value for money

The average cost per WIL experience is below $500.

First CA	Second CA
$280.87	$417.21

The average cost per unsubsidized WIL experience by BHER demonstrated considerable value for money for federal funding within the WIL ecosystem. Other WIL programming that exists in Canada that provide wage subsidies for all supported students demonstrated higher costs of up to $7,000 per WIL experience.^{Footnote 8}

Discussions with ISED officials highlighted the complementary nature of the different costs of WIL programs across the federal government as well as their mandates and noted that the costs vary due to the types of activities undertaken. Specifically, BHER supports unsubsidized WIL placements that may leverage wage subsidies to create traditional WIL experiences, which indicated an effective degree of uptake of this model, while other federal programs often provide subsidized WIL experiences for every student.

Some stakeholders indicated that the variety of WIL experiences they provided were unfunded due to them being embedded curricular work, or voluntary work. Instead, funding primarily supported the development and administration of the WIL program, as discussed in Finding 7.

The funding model supports WIL ecosystem sustainability

As per the CA, BHER funding supports the development of unsubsidized WIL placements. However, the evaluation noted that some WIL placements funded by BHER had compensation for students in the form of hourly wages, stipends, and salaries.

In 2023-23, 63% of students who responded to the BHER survey indicated that their WIL experience had financial compensation. Program officials noted that the representation of a majority compensated WIL is due to the first cohort of funding under the current CA with ultimate recipients who were more focused on traditional WIL types. This may change as future untraditional WILs are completed, as discussed in Finding 5.

Interviews with BHER funding ultimate recipients stated that having funded WIL placements was a best practice as it mitigated financial impact on students, and that they often tried to develop WIL experiences with compensation. WIL delivery partners indicated that they found resources from other sources, e.g.:

• Federal, provincial, or municipal grants
• Non-profit fundraising
• Employer self-funding

The evaluation raised the potential for WIL programs "double-counting" reported student placements at the federal level in the first CA. In BHER's second CA, due diligence was taken to ensure that numbers reported by ultimate recipients do not count numbers where WIL funding from multiple federal programs may have taken place.

Stakeholders noted that although funding could not support student compensation, it allowed funding to more directly support capacity building and the sustainment of the WIL ecosystem. This allowed WIL practitioners to fund things such as industry liaisons or WIL project coordinators, who can be responsive to industry skill changes or the variety of WIL types. BHER's approach indicated its long-term sustainable contribution to the WIL ecosystem.

A lack of BHER awareness and engagement

The majority of stakeholders indicated that they were unaware of or had not participated in BHER events (e.g. information sessions, etc.), and the majority of interviewed employers specifically indicated that they were unaware of BHER. This aligns with BHER's internally conducted needs assessment which found that "many employers, especially small and medium sized enterprises, are unfamiliar with existing WIL experiences, resources, and supports".^{Footnote 9}

The percentage of visitors to the BHER website who stay on the site after viewing one page has been below 50% and has been on a decreasing trend.^{Footnote 10}

The downward trend of 3% on average indicates that online visitors became less engaged with the BHER website over the evaluation period.

BHER WIL hub usage

59% of surveyed employes indicated that they did not use BHER WIL hub tools.

The BHER WIL Hub is a resource created by BHER providing WIL delivery partners with tools needed to create or expand access to WIL. The low level of WIL Hub Usage indicated by the majority of surveyed employers who have not used BHER services aligns with the lack of awareness of BHER.

As a result of this challenge, BHER's effectiveness in its ability to improve the expansion of awareness, knowledge, and variety of WIL experiences to strengthen the WIL ecosystem is limited.

In 2023, 111 out of 270 employers surveyed indicated that they specifically used the following WIL Hub services.

Useful BHER WIL hub tools

41% of BHER employer survey respondents indicated that they did use BHER WIL hub services.

The Top 4 most helpful and most used BHER WIL hub services:

1. Coaching support and advice:

   96 % of the 46 employer survey respondents who used this service found it helpful.

2. Performance assessment guide

   96 % of the 33 employer survey respondents who used this service found it helpful.

3. EDI strategies in WIL

   95 % of the 22 employer survey respondents who used this service found it helpful.

4. WIL mentorship

   94% of the 30 employer survey respondents who used this service found it helpful.

Signs of improvement

BHER acknowledged the lack of awareness of events and tools and noted that they needed to do a better job of promoting themselves. They indicated that they are currently working on making BHER's role in the ecosystem more known. Some initiatives noted by BHER included:

• Launching the Your Skills Now campaign, a toolkit that ultimate funding recipients will be given with creative assets, e.g., logos and promotional templates to help advertise BHER to students and employers
• Building a new BHER website to improve brand awareness
• Adding a section in the MFAs that required ultimate funding recipient partners to commit to promoting the BHER WIL Hub and promoting it among their networks

As these initiatives were quite recent, the evaluation could not assess their effectiveness or impact.

A variety of WIL experiences

Over the period of the evaluation, a total of 42, 391 WIL placements were completed through BHER partnerships with associated PSE institutions, local business organizations, and employers during the evaluation period.

FY	2020-21	2021-22	2022-23
Number of completed WILs	5,089	18,681	3,753

Stakeholders noted that BHER funding required them to be more creative in their approaches to the development of WIL experiences beyond traditional WIL placements, which may explain why the majority of planned WIL placements were untraditional WIL, which are often unsubsidized.

The number of planned untraditional WIL indicates that the ratio between traditional and untraditional WIL may change in the future. The benefits of these types of WIL experiences were primarily associated with skill development. Benefits and limitations of traditional and untraditional WIL are discussed in Appendix F.

The variety of WIL types and compensation strategies used by partners reflect the flexible nature of BHER funding support in the development of a broad variety of both traditional and untraditional WIL, as discussed in Finding 1.

57.3% of WILs committed (i.e. planned) in MFAs are untraditional WIL experiences.

Untraditional WIL increased accessibility of WIL experiences to students

Subject matter experts noted that traditional WIL experiences were a barrier to some groups of students in accessing WIL, for example:

• In 2023, the average annual undergraduate tuition fees for Canadian citizens was $7,076, which is the highest it has been since 2019.^{Footnote 11}
• The Consumer Price Index, which represents changes in prices as experienced by consumers, increased on average by 3% in Canada since 2019.^{Footnote 12}
• Traditional WIL co-op placements are often at least 4-month full-time job placements.

Considering these factors, access to a traditional WIL experience is often not feasible due to the amount of time and/or finances that a student has to commit to access university co-op programs and job boards, according to interviewees.

In comparison, untraditional WIL experiences were more flexible, (i.e. part-time, conducted during course hours, and/or free), which allowed more vulnerable students to maintain a job, support their family, be involved in extra-curricular activities, etc.^{Footnote 13} This allowed for more students to participate in WIL experiences due to the lower impact of the time and cost barriers, according to interviewees.

International students

WIL stakeholders highlighted the importance of international students to businesses and Canada's labour market.

In 2023, Canada ranked 5th out of the G7 for labour productivity with -1.12%.^{Footnote 14}

Labour productivity, calculated as the growth in gross domestic product per hours worked, has decreased in Canada, which may reflect a lower use of capital, an increase in the employment of low-productivity workers or less efficiency in the economy.^{Footnote 15} 56.1% of all industries reported skills gaps in their workforce according to Statistics Canada.^{Footnote 16} Evidence highlighted that WIL experiences gave international students an incentive to settle in Canada to contribute to productivity and the economy by enabling them to find high-productivity jobs in industries related to their post-secondary education after their studies.^{Footnote 17}

Most traditional WIL programming is ineligible to international students, (e.g., SWPP); however, due to the eligibility guidelines of BHER's funding, international students had access to more WIL placements. A key component of BHER's CA spoke to the prioritization of WIL experiences developed for this population.

Over the evaluation period, BHER supported the development of a variety of placements that increased the opportunity for international students to have a WIL experience.

Although BHER reported that it completed all activities reported to partnership development for its inclusive WIL for international students RFP, data was not yet available to assess effectiveness.

34% of BHER 2022-23 pre-survey students identified as an international student

COVID-19 impacts to WIL placements

Evidence indicated that the immediate impact to WIL was a mass cancellation of work placements during the May to August 2020 work term.^{Footnote 18} Stakeholders noted that both employers and students were less likely to engage in WIL experiences because they did not have the capacity (e.g., emotional, financial, and/or physical availability), to deliver or undertake a WIL experience. Reporting on the COVID-19 impacts to post-secondary students, Statistics Canada found^{Footnote 19}:

• 48% Of post-secondary students had lost their jobs or been temporarily laid off.
• 26% Of post-secondary students had reduced work hours.
• 49% Of post-secondary students had lost their job prospects.

Stakeholders noted that the COVID-19 pandemic pushed WIL-practitioners to develop more virtual placements to align with the hybrid work requirements of employers.

The majority of stakeholders considered the circumstances of the COVID-19 pandemic to be a major motivation for the evolution of the WIL ecosystem and one of the reasons why there was more support for untraditional WIL. It was during this time that the C3 was initiated by BHER, as discussed in Finding 2. Stakeholders indicated that this contributed to the increase of untraditional WIL experiences in the WIL ecosystem.

Limitations of virtual WIL

Although increased virtual WIL placements were created, interviewees noted that there were challenges that impacted the quality of the WIL experience due to the virtual environment, such as:

• Mentorship: students did not have immediate access to colleagues for networking, knowledge transfer and advice.
• Social/emotional skill development: students were not in an office environment where many of these skills would be informally acquired.
• Peer-to-peer exchange: students could not as easily share experiences that they learned on the job or in the classroom with each other.
• 56% Of employers have adjusted how they approach leadership, management, or mentoring practices at their organizations as a result of the pandemic.^{Footnote 20}

PSE institutions: Capacity building

The flexibility of BHER's MFAs allowed PSE institutions to develop innovative WIL experiences and expand existing programs:

• BHER funding was used in some cases as a "start-up" for the development of WIL experiences in PSE institutions where they may not have existed before.
• Experiential learning offices that already had established networks with employers prior to BHER were able to expand or deepen their network of relationships due to their partnership with BHER.
• BHER funding allowed PSE institutions to pilot WIL projects to demonstrate their effectiveness and potentially access internal resources or other sources of future funding. This would enable the sustainability of the WIL project offered beyond BHER funding.

However, the evaluation noted that some of the PSE institutions indicated that they would not be able to offer their WIL programming without BHER funding, which presents a risk to long-term sustainability and capacity building. They noted that BHER funding supported the wages of those running WIL programming.

71% Of PSE MFAs reviewed indicated that more than 70% of BHER funding went to salaries and administration.

BHER indicated that they were aware of this risk and had taken measures to mitigate its impact.

• Most BHER funding agreements were not renewed after the original agreement. BHER noted that they intended to create new partnerships through every CA cycle to ensure that they were finding new programs to fund. In the second CA, only the top performing partners were re-signed.
• BHER required RFP proposals to include a section on programming sustainability. In their most recent round of MFA negotiations, ultimate recipients had to demonstrate the continued existence of their WIL program in the future. Those that could not demonstrate sustainability were not able to get funding.
• BHER leveraged university consortia and partnerships to encourage an exchange of best practices. BHER noted that they took a hands-on approach to share best practices in the WIL ecosystem to foster learning from other successful programs. The MFA agreement included a clause on success stories to support this initiative.

PSE institutions: Capacity building

PSE institution partnerships have increased

Over the evaluation period, the number of BHER partnerships (including secondary partnerships) with PSE institutions increased by 81%: 26 PSE institutions in the 1st CA, 15 additional partnerships only for C3 and 47 PSE institutions in the 2nd CA.

Approximately 24,300 WIL placements have been committed to and/or completed through PSE institution-led BHER partnerships over the two CAs thus far.

The increased partnerships with PSE institutions allowed BHER to work with established WIL departments in these organizations, (e.g. co-op offices and experiential learning offices), who had embedded WIL-specialized staff available to develop and administer WIL programming. According to the reviewed MFAs, PSE institutions used part of their BHER funding to support direct labour and overhead, (i.e. salaries to hire individuals to support and administer the university's WIL programming). However, the evaluation could not confirm whether all of these staffed positions were created strictly administer BHER activities and collaborations, or to administer WIL experiences more broadly for their institution.

Many PSE institutions highlighted had knowledge of WIL prior to partnering with BHER. Therefore, they did not rely on BHER or BHER tools to guide them on the development of WIL placements to the same degree as employers or local business organizations, which could indicate that they learned more about WIL through BHER, however direct attribution to BHER's effectiveness in this space is difficult.

Opportunity: Capacity building with PSE institutions and employers is a key activity for BHER. The evaluation noted that in some cases, capacity building in programs would not continue without ongoing funding from BHER. Further, there was insufficient data to determine the impact of BHER's capacity building results among employers. ISED could explore how its program design could best support capacity building efforts in the WIL ecosystem that directly benefit SMEs and ultimately better align with ISED priorities.

A profile of employers

An analysis of program data illustrated the types of employers engaged with BHER over the two Cas :

• The number of SME employers decreased 90% in the 1st CA to 75% in the 2nd CA. The decrease could not be fully assessed because the 1st CA indicates a cumulative result over the first agreement, while the 2nd CA only included FY 2022-23.
• 45% Of BHER employer survey respondents reported that they worked in the private sector.

BHER reported employer representation across 17 industrial sectors, with the top four being:

• Education services
• Accommodation and food services
• Professional, Scientific and Technical Services
• Healthcare and Social Assistance

Accommodation and food services, as well as education services were identified as one of the top ten industries with the highest percentage of businesses that reported skill gaps in their workforce, (78.8% and 65.4% respectively).^{Footnote 21}

Employer engagement:

1,570 employers were engaged to deliver WIL in FY 2022-23.

126,392 employers were outreached in FY 2022-23.

In FY 2022-23, BHER was unable to meet its year 1 target of 2000 employers. BHER noted this was due to the scheduling of engagement opportunities in April and indicated that they met the engagement target of 2,191 by Q1 of FY 2023-24.

Uneven regional representation

There is an uneven distribution of post-secondary BHER students reported in 2022-23 compared to the total post-secondary student population representation in Canada last reported by Statistics Canada in 2021-22.^{Footnote 22}

	Canada	BHER Students	BHER Employers
Ontario	41.9%	65.8%	42%
Quebec	24.5%	10.6%	4%
Prairies	14.9%	13.4%	11%
BC	13%	4.5%	31%
Atlantic	5.5%	5.3%	12%
Territories	0.1%	0.4%	0.1%

The evaluation could not align reported BHER 2021-22 numbers with the same year as those reported by Statistics Canada. Where data was unavailable, other lines of evidence were used to confirm or support evaluation findings.

BHER students in Ontario, the territories, were overrepresented, while Quebec, the prairie provinces and BC were underrepresented. This misalignment was also noted between students and employers. This indicates that supply and demand of BHER placements across the country are not equitable to their resident populations.

the evaluation noted that there were challenges concerning access to Francophone partnerships and opportunities, as demonstrated with the 4% of Quebec BHER employers and 10.6% BHER students. BHER indicated that working in this space was a challenge; however, they indicated that they were building partnerships with PSE institutions to hopefully improve their regional presence.

The evaluation noted a disproportionate distribution of employer representation across the country in comparison to Canada's population distribution.

More employers are delivering WIL

• 81% Of employer survey respondents indicated that they were had improved collaborations with PSE institutions.
• 29% Of employers indicated that they had created their first WIL experience.
• 31% Of employer survey respondents indicated that they were offering more WIL experiences than previously.

Employer interviewees agreed that increased collaboration with the PSE institutions and students were mutually beneficial, and noted how it allowed them to expand the depth of their relationship with academia in the delivery of WIL experiences (e.g. innovative experiences beyond a professor's curriculum). However, the evaluation could not directly speak to whether this was specifically due to BHER activities, since interviewees indicated that they were already seeing an increased relationship before BHER, or they did not work directly with BHER to facilitate the increased collaboration. This speaks to employer lack of awareness, as discussed in Finding 4.

Although there was an expansion of WIL delivery among employers, one employer stated that economic uncertainty may make it difficult for them to grow their organization, which may limit the expansion or sustainment of WIL ekeholders, which is highlighted by literaturligns with literature which speaks to economic uncertainty negatively impacting SMEs.^{Footnote 24}

Employer WIL benefits

The top four performing organization outcomes reported greater than 94% positive outcomes in the employer survey.

78% to 97% of BHER employer survey respondents indicated that their organization had experienced positive outcomes due to hosting a WIL experience, which aligned with employer interviewees who stated that WIL experiences allowed them to have a direct talent pipeline from PSE institutions with valuable skills and knowledge that they could bring to their organizations through the WIL placements.

Employers are hiring WIL students

453 students, (i.e., 15% of the surveyed population), indicated that they were working part-time or full-time with the organization that they worked with during their WIL experience.

117 employers, (i.e., 48% of the survey population), indicated that they were hiring students. They indicated hiring from a range of 1 to 200 students.

Coaching and mentorship on WIL delivery

Evidence noted that strong management capacity is required for quality WIL delivery, which was echoed by interviewees who highlighted that employers of SMEs did not often have the necessary knowledge and/or Human Resources (HR) management capacity to effectively deliver WIL.23 For example, ensuring that students have appropriate onboarding material or relevant tasks that would contribute to the development of skills. Employers delivering WIL indicated that this was a key priority and believed they would benefit from additional support in this regard.

BHER noted that they were aware of this opportunity for improvement and have partnered with an HR group in Canada that provides a micro-credential on delivering quality WIL. This may potentially help employers who noted that challenges, such as mentoring students during a WIL experience, were a particular priority according to stakeholders, which is highlighted by literature as a critical component of a quality WIL experience.^{Footnote 24}

Technical in-demand skill development

In the 2022-23 BHER student survey, critical thinking, problem solving, communication skills and creativity were identified as the top four employer in-demand skills that were developed during their WIL experience.

• Critical thinking : 86%
• Problem solving : 86%
• Communication : 85%
• Creativity : 84%

84% of BHER student survey respondents indicated that they improved their technical skills

All in-demand skills identified in the BHER student survey demonstrated a rating of greater than 80% who indicated skill development. Literature and interviewees highlighted the importance of quality WIL experiences that allow students to develop in-demand skills.^{Footnote 25} Interviewees noted that quality WIL experiences supported the development of skills through the application of what students learn in the classroom to real-world experiences.

Professional skill development :

86% to 90% Of BHER students indicated that they improved skill articulation.

86% of students indicated that they had improved their ability to understand employer expectations, which echoed commentary from the student case study.

The students in the case study who participated in the traditional WIL experience indicated that they initially felt stressed by the new WIL experience, but as they spent more time at the job, they had a better understanding of their manager's needs, and how to respond to and prioritize tasks.

Social-emotional skill development :

Of those who indicated that they did not have prior social-emotional skills, the majority of BHER student survey respondents indicated that they improved during their WIL experience.

Excluding those who indicated that they already had social-emotional skills, students indicated a range of 89% to 95% improvement. The student case study and questionnaire highlighted how their WIL experiences helped them to work better on teams and be more adaptable in a variety of situations that they faced in the workplace. Interviewees highlighted the importance in the development of these skills, as employers are looking for those who can integrate into their work environment.

86% Of BHER student survey respondents indicated that the WIL experience made them feel more prepared for the workforce.

The benefits of WIL for EDI

Literature noted that there is a higher level of underemployment and unemployment among equity-deserving populations, including women.^{Footnote 26} Further, low self-reporting rates, privacy concerns, and long approval processes with PSE institutions, made it difficult for BHER to collect extensive Gender Based Analysis Plus data, which limited the assessment of EDI impacts. However, the evaluation found that a WIL experience may contribute to decreasing the typical barriers that equity-deserving populations traditionally face in getting hired.

A majority of BHER employer survey respondents indicated that their WIL experience had improved organizational policies and practices related to EDI and team diversity.

The survey results speak to the employer perception of benefits received through hosting WIL experiences and the importance of EDI to these organizations. Stakeholders noted that employers are seeking skilled and diverse employees and that WIL placements provided them with direct access to these talent pools.

Case study finding

It was noted during a Case Study interview, that WIL programming at university allowed the student to have the opportunity to receive coaching and support to enable them to successfully obtain a WIL experience.

Acknowledging their barriers, (i.e., being an international student, not having a local network, not able to speak French, having a disability, and a lack of relevant job experience), the student directly attributed their WIL experience to being able to successfully obtain interviews in comparison to before. This aligns with literature highlighting that WIL might contribute to reducing wage and employment gaps faced by equity-deserving populations.^{Footnote 27}

4.0 Summary

4.1 Conclusion

BHER supports the Government of Canada's commitment to 100% WIL for Canadian post-secondary students. Its unsubsidized WIL experience approach fosters capacity-building of Canada's WIL ecosystem and contributed to the development of relationships between employers and PSE institutions to deliver a variety of quality traditional and untraditional WIL experiences to Canadian and international students. As a newer program, BHER is still finding its space in the WIL ecosystem and must work to ensure that its activities align with ISED private sector priorities and support students across Canada.

4.2 Opportunity

ISED could explore how its program design could best support capacity building efforts in the WIL ecosystem that directly benefit SMEs and ultimately better align with ISED priorities.

5.0 Recommendations

Recommendation 1: ISED Industry Sector should conduct an environmental scan of the WIL ecosystem in Canada to better understand existing WIL offerings, services and tools, including potential gaps, and areas for alignment of BHER activities within the ecosystem.

Recommendation 2: ISED Industry Sector should explore mechanisms to improve performance measurement and monitoring to enable consistent medium and long-term analysis.

Recommendation 3: ISED Industry Sector should explore approaches with BHER to enhance the organization's regional presence across Canada.

Appendices

Appendix A : Acronyms

BCC Business Council of Canada

BHER Business + Higher Education Roundtable

C3 Canada Comeback Challenge

CA Contribution Agreement

FY Fiscal Year

ISED Innovation, Science and Economic Development

PSE Post-Secondary Education

RBC Royal Bank of Canada

ROI Return On Investment

SME Small and Medium Enterprise

WIL Work-Integrated Learning

Appendix B : Logic Model

Through the eligible activities funded under the ISED funding agreement, BHER is expected to achieve the results in the logic model below, along with related short-term, medium-term and ultimate outcomes. The logic model was derived from program documentation.

Appendix C: Methodology

The evaluation was based on five data collection methods, including qualitative and quantitative sources.

In order to answer the evaluation questions, information from multiple lines of evidence was collected and triangulated.

Literature and document review

The literature review was comprised of 75+ documents of pertinent literature exploring the state of WIL in Canada and internationally, as well as the benefits and challenges to WIL. The document review included key government priority setting documents, program reporting documents and materials to support the assessment of performance and efficiency.

Performance, administrative and financial data review

Performance data was reviewed to assess the extent to which progress has been made towards achieving the immediate intermediate and ultimate outcomes outlined in BHER's logic model. An analysis of administrative and financial data was conducted to inform program effectiveness and efficiency.

Case studies

A case study was conducted regarding the impact of WIL to three students to demonstrate the different experiences across structured and unstructured WIL to student skill development outcomes as well as the unique experiences of an international student.

Online surveys

BHER conducts surveys of students before and after their WIL experiences as well as a survey of employers engaged in the delivery of WIL experiences which is completed biannually. The evaluation team reviewed all three surveys to confirm alignment with evaluation questions and mitigate survey fatigue. The evaluation had access to 2023 survey results with a total of responses to the pre-survey: 3,411, to the post-survey: 1,436, and to the employer survey: 245.

Online questionnaire

An online short-form questionnaire was conducted targeting the perspectives of students and their WIL experiences. There was a total of four student responses out of a total of seven students contacted. This was done in place of interviews with students.

Interviews

A total of 31 interviews were conducted using MS-teams and in-person across the following stakeholder groups to gather diverse perspectives on the relevance, performance and efficiency of BHER.

• ISED Management and staff
• BHER board of directors, management and staff
• BHER strategic partners
• BHER funding recipients (employers, local business organizations, PSE institutions)
• BHER beneficiaries (employers)
• Subject matter experts
• External WIL ecosystem stakeholders

Appendix D: Challenges and mitigations

The evaluation encountered 3 limitations and evaluators applied related mitigation strategies.

• Attribution
• Challenge: The existence of other WIL organizations made isolating and measuring the direct impact of the federal government's contribution challenging.
• Mitigation: Interviews and survey questions were designed in a way that respondents could answer, to the extent possible, the impact of BHER's funding to WIL activities and impacts.
• Respondent Bias
• Challenge: Interview participants may have been directly involved with program delivery or direct beneficiaries, and as a result, responses may have been positively biased.
• Mitigation: Strict confidentiality was communicated to participants to encourage open and transparent communication. Responses were validated across stakeholder audiences and other lines of evidence.
• Data Quality :
• Challenge: During the evaluation, data quality issues were encountered, such as inconsistent indicators and unavailable data. Additionally, data was incomplete due to the evaluation being conducted in the beginning of a new contribution agreement with only one year of available data.
• Mitigation: Where data was unavailable, other lines of evidence were used to confirm or support evaluation findings.

Appendix F : Work Integrated Learning

Traditional Models of WIL

These are definitions as described by BHER for some commonly used traditional WIL examples:

• Co-operative Education: Co-op consists of alternating academic terms and paid work terms. Co-op internship consists of several co-op work terms back-to-back. In both models, work terms provide experience in a workplace setting related to the student's field of study. The number of required work terms varies by program; however, the time spent in work terms must be at least 30% of the time spent in academic study for programs over 2 years in length and 25% of time for programs 2 years and shorter in length.
• Field Placement: Provides students with a part-time/ short term intensive hands-on practical experience in a setting relevant to their academic discipline. Field placements may not require supervision of a registered or licensed professional and the completed work experience hours are not required for professional certification. Field placements account for work-integrated educational experiences not encompassed by other forms, such as co-op, clinic, practicum, and internship.
• Internship: Offers usually one discipline-specific, supervised, structured paid or unpaid, and for academic credit work experience or practice placement. Internships may occur in the middle of an academic program or after all academic coursework has been completed and prior to graduation. Internships can be of any length but are typically 12 to 16 months long.
• Mandatory Professional Practice/Clinical Placement: Involves work experience under the supervision of an experienced registered or licensed professional in any discipline that requires practice-based work experience for professional licensure or certification. The practicum is generally unpaid and, as the work is done in a supervised setting, typically students do not have their own workload/caseload.
• Apprenticeships: (i.e., systematic workplace training, typically 80% on the job and 20% in the classroom).
• Entrepreneurships: (i.e., supporting students to develop ideas to solve real-world problems and/or start their own business, often integrating mentorship, workshops, space, work with start-ups, etc.
• Institutional Partnerships: (i.e., learning activities with a focus on industry or community goals, such as Applied Research Projects, Service Learning)

Emerging Models of WIL

Emerging WIL models, also known as untraditional or unstructured WIL include innovations or variations upon traditional models (e.g., shared apprenticeship models). They also include novel forms of WIL that fall outside the traditional categories and WIL types, but that increasingly create opportunities for students who have been traditionally under-represented in WIL, owing to their location, equity group, or diploma/degree program. Some examples include:

• Micro-placements: Students work individually or in teams for short periods between two and ten days.
• Hackathons/competitions/events: Students are engaged in short, industry-partnered sessions that allow them to apply their learning to solve challenges and/or learn more about industry needs and build connections.

Traditional vs Untraditional WIL

Based on interviews, case studies and literature, the evaluation noted key benefits and limitations of traditional and untraditional WIL within the ecosystem.^{Footnote 25}, ^{Footnote 26}

Traditional WIL

Untraditional WIL

Benefits: Longer-duration allows for increased student exposure to day-to-day experiences and the realities of the job. Increased student interaction results in more opportunities to develop technical, social, and professional skills relevant to their field. A direct pipeline to potential hiring if completed near the end of the student's studies. Mentorship opportunities and development of a professional network through increased interaction with work colleagues. More likely to receive financial compensation. Limitations: Are often hybrid or in-person, restricting placements to those in the same region or willing to move. May be emotionally overwhelming for students who are new to their field of work. Time consuming and cost-prohibitive, as discussed in Finding 4. Academic requirements may conflict with employer needs.

Benefits: Shorter-duration allows it to quickly react to changing employer in-demand skills between cycles of delivery. Part-time WILs allow students to more easily integrate the experience into their schedules. Light exposure to potential careers without overinvestment of personal resources (i.e., time and finances one might use for a co-op placement) The development of academic relationships with faculty members who are running untraditional WIL experiences. Limitations: Non-placement-based innovative WIL does not have the same degree of exposure of day-to-day experiences on the job, which limits the sense of future job readiness. Hybrid or virtual untraditional WIL experiences are often not available to rural and remote regions with limited internet access. May not have the same degree of professional mentorship exposure due to limited interaction with employers. Are often unpaid experiences. Academic requirements may conflict with employer needs. Perceived less value to students that have already completed a traditional WIL experience.

The benefits and limitations of both traditional and untraditional WIL indicate that there is value to both approaches to WIL in the ecosystem.

Observed untraditional WIL experiences seem to be beneficial in the early years of a student's development and exposes them in a controlled, often academic, approach to their future career field. traditional WIL experiences seem to better placed to leverage future job opportunities and real-world on-the-job skill development.