2019 BAI performance measurement framework: Desired outcomes

PART I. Mandate and Progress to Date (continued)

1.6 Desired Outcomes from the BAI PMF 2.0

The Logic Model, shown in Figure 1, identifies the linkages between the activities of the PMF Pilot and the achievement of its results/ outcomes, including the next phases of implementation. In addition to a number of immediate and longer-term outcomes, the Logic Model highlights four overarching objectives for the BAI PMF 2.0. These include:

  • An updated performance measurement framework, informed by the lessons learned from the pilot, with consistent metrics for measuring the impact of BAI programs on firm performance, along with clear definitions for each performance indicator;
  • An updated process for managing the collection, aggregation and analysis of consistent data from multiple BAIs across Canada using a data sharing platform that will streamline the data collection, analysis and reporting process;
  • Shared commitment across the BAI community to using the lessons learned from the pilot to establish a sustainable institutional framework for rolling out the PMF on a national basis;
  • And, over the longer term, enhanced, data-driven insights into the role of BAI programs in firm growth, and ultimately, superior economic growth enabled by the accelerated growth of high-potential firms working with BAIs across Canada.

Figure 1: Logic model for the PMF 2.0.

Graphic representing Logic model for the PMF 2.0.
Description of Figure 1
Logic model for the PMF 2.0.
INPUT & PARTICIPANTS PHASE I
ACTIVITES
PHASE I
OUTCOMES
PHASE II
ACTIVITIES
PHASE II
OUTCOMES
ULTIMATE OUTCOME
  • Representative group of business accelerators and incubators (BAIs) from across Canada
  • Federal government partners
  • Data sharing platform
  • BAI client firms.
  • Define performance measurement framework (PMF) with common metrics and definitions
  • Select data sharing platform for aggregating and reporting data
  • Enlist BAI participants for pilot of the PMF and data sharing platform
  • Obtain consent from client firms. Collect and upload data for common performance indicators
  • Export data to Statistics Canada and ISED for a trial run of econometric analysis and preparation of descriptive statistics.
  • Pilot outcomes, lessons learned and data analysis are documented in a progress report
  • Standardized metrics and definitions are calibrated based on BAI/government feedback
  • Processes for collecting, storing, analyzing and reporting data are refined
  • Opportunities, challenges and tips for managing the process with an expanded number of BAI participants are identified.
  • Recruit a larger group of BAIs and gov't partners to participate in phase II of the pilot
  • Initiate Phase II data collection, analysis and reporting cycle
  • Convene BAIs and gov't partners for an assessment of lessons learned in phase II
  • Initiate discussions and plans for a national rollout of the PMF.
  • A national baseline for longitudinal data collection is established
  • Econometric analysis produces data driven insights into the role of BAIs in firm growth
  • Data insights help BAIs refine program offerings to better support firms
  • Ecosystem data highlights opportunities for collaboration and synergy between BAIs and other partners
  • Performance measurement processes are streamlined across multiple levels of government.

Superior economic performance is enabled by the enhanced growth of accelerated firms as a result of systematic improvements across the ecosystem:

  • BAIs can benchmark their performance and drive improvement
  • Companies can choose their best options for support
  • Governments as all levels can increase the effectiveness of public investments in BAIs.

As the Logic Model indicates, the inputs and participants for the PMF include a representative group of BAIs and the programs and services they currently offer to client firms, federal government partners (particularly ISED) and the data sharing platform supplier. The activities in phase I centred mainly on establishing the PMF (i.e., identifying the right metrics and survey questions), identifying a platform for the collection and aggregation of client data, enlisting BAIs to participate in the pilot, obtaining consent from client firms to use their data for the pilot and performing an analysis on the data collected. The inputs and activities represent the foundational steps taken by the project partners and ISED to create a PMF and conduct a test run of the data collection process.

For phase II, the activities include refining the measurement framework and data collection process, expanding the group of BAIs participating in the pilot, doing a second round of data collection and performing more sophisticated statistical analysis using a larger dataset linked to administrative datasets at Statistics Canada, and continuing to engage government partners across Canada.

The logic model indicates that the outputs are linked to the pilot process itself. It is expected that participating BAIs will work with their client firms to collect all of the data specified by the performance measurement framework (or request their client firms to upload the data directly) over two data collection and reporting cycles: one for 2017 BAI program cohort/entrant data and one for 2018 BAI program cohort/entrant data. In both instances, the data sharing platform provider will aggregate and collate the data and prepare it for export. ISED and Statistics Canada will download the data according to the protocols defined in part II and produce a report. Following these steps, all parties will participate in an evaluation of the pilot outcomes and document the lessons learned.

As depicted in the logic model, the pilot phase is expected to produce a number of short and long-term outcomes as the BAI community works towards the realization of a national performance measurement solution for Canada. Immediate outcomes will be observable at the conclusion of the second year of the pilot in Winter 2019–20. Medium to long-term outcomes for the pilot are expected to take a year or more to manifest, with some outcomes – such as the use of data-driven insights to refine BAI programming and government investment priorities – taking several years to come to fruition.

The immediate outcomes for phase I of the pilot include:

  • Updated agreement by stakeholders on what metrics should be collected. While the BAI Steering Committee and feasibility study ensured that a representative group of BAIs contributed to the process of identifying a common set of performance metrics for phase I, the experience of collecting 2017 data provided an opportunity to identify areas for improvement and implement changes to the performance measurement framework and survey instrument.
  • Updated agreement on standardized definitions for each metric in the PMF. BAIs participating in the pilot also took the opportunity to provide useful input on the clarity and usefulness of the definitions provided in the BAI and company surveys.
  • An improved understanding of how the data will be collected, stored, and analyzed. Processes for data collection, analysis and reporting have been designed, and partially tested during the feasibility study. Phase I of the pilot provided an opportunity identify potential pain points and alleviate any concerns about data security and confidentiality, among other things, for phase II.
  • An improved understanding of what the frequency of data collection should be. The default assumption is that company performance data points will be collected annually. BAI program profiles could also be updated once annually. However, the pilot led to further discussion about analytical or administrative reasons for collecting certain metrics more frequently.
  • A demonstration of success and lessons learned. The successful collection and aggregation of consistent data from multiple BAIs across Canada demonstrated that performance measurement can work within a diverse ecosystem of BAIs. The subsequent Mini-Summit in Waterloo provided an opportunity to identify valuable lessons and insights for managing the national performance measurement process with an expanded group of BAI participants.
  • Shared commitment to building a sustainable institutional framework. The successful run of the phase I pilot also set the stage for discussions between BAIs and their partners in government about how best to establish a sustainable institutional framework for the ongoing management of a national performance measurement process.

As the PMF moves into phase II, a number of additional outcomes will come to fruition. These include:

  • A consistent baseline for longitudinal data. By using standardized metrics, Phase I helped establish a consistent baseline for collecting longitudinal data over a longer period of time. Phase II will further extend and enrich this baseline.
  • Linkages to other datasets for econometric analysis. Phase I enabled researchers at Statistics Canada to successfully link the resulting database with other datasets held by Statistics Canada and to provide a demonstration of the types of descriptive analyses that are possible using the linked data. Phase II will provide an opportunity to test more sophisticated forms of analysis and to link the resulting data to additional datasets (for example, databases held by federal programs such as NRC-IRAP and industry associations such as the National Venture Capital and Private Equity Association of Canada).
  • Streamlined performance reporting. Phase I of the pilot demonstrated to federal, provincial and municipal funding bodies how multiple government departments can use the national PMF to collect the same data through a single process and platform, thereby lessening the reporting burden on BAIs. Phase II presents an opportunity for these entities to engage more fully in the pilot process.
  • Enhanced insight into business growth and Canada's broader economic performance. Phase I of the pilot set the stage for developing a data driven understanding of the role of BAIs programs in Canadian firm growth. Phase II will set the stage for an enriched analysis with the possibility to establish industry benchmarks, highlight opportunities for collaboration and synergy between BAIs, and provide a window into the performance of high-potential sectors and regional tech clusters across the country.
  • Accelerated growth of supported firms. Over time, insights on how to most effectively support innovative growth-oriented firms in Canada will help BAIs to refine their program offerings. Improved options for support, in turn, will ensure that firms get the targeted, high quality services they need to build world class ventures in a wide variety of high-potential sectors.

In summary, a national performance measurement framework will provide BAIs with reliable and comparable data on which to make sound decisions, as well as timely information on the relevance, success and cost-effectiveness of their programs and activities. A common evaluation framework and reporting process will also ease the administrative burden on BAIs, while providing governments and other funders with a rigorous and objective evidence base with which to assess the performance of BAIs and make informed resource allocation decisions. Entrepreneurs across the country will be better equipped to identify the best options for support at various stages of their entrepreneurial journey.

The work Canadian BAIs are doing together at the national level will result in many other tangible benefits for BAIs, their clients and the broader economy. These benefits include the ability to share best practices across institutions and jurisdictions, establish relevant performance benchmarks for different regions and sectors, enhance input into public policy, and position and promote Canada as a destination for start-up activity. Ultimately, this work will inform BAIs and the Government on how to most effectively support innovative growth-oriented firms in Canada and these insights, in turn, will help accelerate the growth of world class companies in a variety of high value sectors. In other words, the collaborative efforts to build a common system for measuring the performance of BAIs could herald the beginning of an exciting new chapter in the growth and evolution of Canada's start-up ecosystem.