2018 BAI performance measurement framework: Mandate and progress

Table of contents

1. Mandate and Progress to Date
   • 1.1 Rationale for a National Performance Measurement Framework
   • 1.2 Designing The Performance Measurement Framework
   • 1.3 Key Outputs and Lessons Learned from the Feasibility Study
   • 1.4 Desired Outcomes from the Pilot.
2. Performance Measurement Framework
   • 2.1 Key Design Principles and Considerations
   • 2.2 BAI and Company Profiles
   • 2.3 Performance Metrics and Definitions
   • 2.4 Data Collection
   • 2.5 Data Disclosure, Analysis and Reporting
     • Data Disclosure
     • Linkages to External Datasets
     • Data Analysis: Assessing the Impact of BAIs
     • Reporting on BAI Performance
3. Performance Measurement Platform
   • 3.1 Platform Description
   • 3.2 Data Security
   • 3.3 Consent to Share Information
4. Conclusions and Next Steps
5. Annexes
   • Appendix A: BAI Questionnaire
   • Appendix B: Company Questionnaire
   • Appendix C: Platform Requirements
   • Appendix D: BAI Steering Committee and Working Group

1. Mandate and Progress to Date

Over the past number of years, public sector organizations have increasingly recognized the need to develop and deploy performance measurement systems to ensure that they have timely, strategically focused, objective and evidence-based information on their performance, in order to produce better results and remain high-performance organizations. With this goal in mind, the Department of Innovation, Science and Economic Development (ISED) has been working a representative group of policy leaders and stakeholders to develop a performance measurement framework for business accelerators and incubators (BAIs) in Canada. This framework will enable these organizations to benchmark their performance and drive improvement, help companies to choose their best options for support, and assist governments at all levels in increasing the effectiveness of public investments in this area.

The work of the BAI community and its partners in government will result in a standardized reporting framework that establishes consistent definitions for job creation, revenue generation, investment and other outcome-related metrics; a common performance measurement platform that will streamline the collection, analysis and reporting of data; and a pilot process to be launched in early 2018 that will provide a small, representative group of BAIs with an opportunity to test and refine the framework before rolling it out on a national basis. Other deliverables include a set of legal agreements to govern the collection and reporting of client data and an agreed process and methodology for analyzing the economic impact of BAIs.

This report is intended to guide the BAI community and its partners in government as they proceed with the next phase of building a national performance measurement solution. It presents the Performance Measurement Framework (PMF) which will form the basis of the 2018 pilot, and documents the progress achieved to date towards developing the analytics and reporting platform.

More specifically, the report:

• Outlines the rationale for establishing a national PMF, describes the process used to develop it and provides a simple logic model that guides the design of the PMF.  
• Provides clear definitions for the metrics that the PMF will draw on, identifies the sources for those metrics (companies, BAIs, Statistics Canada, etc.), and describes the approach for collecting, analyzing and reporting the data, including the methodology that will be used by approved researchers to produce descriptive statistics and the econometric analysis.
• Provides a preliminary view of the PMF dashboard design and details the operations and administration of the performance measurement platform, including processes for obtaining consent to share information and protecting the confidentiality of data.

1.1 Rationale for a National Performance Measurement Framework

The efforts to forge agreement on a national performance measurement solution are premised on a growing appreciation of the value that will be created for Canada's start-up ecosystem, including BAIs, their clients, partners and funders. Indeed, the majority of business accelerators and incubators (BAIs) participating in the feasibility study already collect performance data and issue annual reports to their funders, partners and stakeholders. However, both a national dialogue among BAIs in February 2017 and a subsequent feasibility study revealed that BAIs are currently measuring their performance using a diverse and (often) inconsistent range of metrics and with widely varying levels of success in obtaining data from their clients. A broad consensus emerged that a national framework for performance reporting would generate several key benefits for the BAIs, their clients and the start-up ecosystem as a whole.

The benefits envisioned by the group include:

• Improving economic impact analysis: A consistent, national performance measurement approach would allow BAIs to better evaluate their impact on client performance and on the broader economy, both locally and nationally. As an industry-led effort, a national framework would also ensure that the metrics used to evaluate performance are measuring the right outcomes (i.e., the outcomes that create the most value for BAI clients and stakeholders) and are appropriately calibrated to the services BAIs deliver and the context in which they deliver them.
• Increasing transparency: Improved data collection and reporting of economic impacts, in turn, would better inform firms in their search for support, provide the transparency that public and private funders require to allocate resource efficiently, and allow BAIs themselves to benchmark their own performance against their peers.
• Enabling collaboration: BAIs would also be able to showcase their areas of comparative strength—be it by sector, growth stage or connections—which could help facilitate collaboration among BAIs and other ecosystem participants, including investors and government service providers.
• Facilitating learning and continuous improvement: Simultaneously, public reporting on outcomes will allow BAIs to share best practices and benchmark their performance against organizational leaders (domestically and internationally), thereby facilitating a process of learning and continuous improvement.
• Positioning and marketing of Canada as a destination for start-up activity: A national dashboard highlighting the activities and achievements of Canada's BAIs could help tell a powerful story to the world and thereby attract international participants to the ecosystem, including founders, investors and corporate partners.
• Reducing administrative burdens: Finally, the Canadian ecosystem is complicated by its multiplicity of funders, and a subsequent proliferation of different demands for different data. This reporting burden puts a significant strain on the resources of Canadian BAIs. A national framework holds the potential to simplify and streamline performance reporting for BAIs by creating a consistent set of metrics against which to report and (in some circumstances) a unified reporting solution that would deliver one set of annual results to all relevant stakeholders.

With these benefits in mind it is worth clarifying how the PMF will be used to inform policy and funding decisions. The purpose of a standardized national measurement framework is to generate consistent and reliable data about the economic impact of BAIs, for the benefit of BAIs, companies seeking BAI support, and governments that fund BAIs. Analysis performed using the data collected from BAI clients during the pilot period – including the production of descriptive statistics and econometric modelling using linked datasets by ISED, Statistics Canada and/or approved researchers – will not be used to evaluate the performance of individual BAIs. It will, however, be used to inform robust conclusions about the role BAI programs play in firm growth and how to most effectively support innovative growth-oriented firms in Canada. For policymakers in particular, the objective is to use the PMF to evaluate the overall effectiveness of national funding programs, identify policy gaps and frame responses that boost the performance of Canada's business support ecosystem.

With respect to future funding applications for individual BAIs, it is expected that BAIs will present their performance data in a manner consistent with the PMF and, when applicable, use the framework to report their performance against specific program funding they receive. In doing so, BAIs and their partners in government can achieve greater alignment on reporting requirements and eventually reach a point where BAIs can enter data points once for multiple audiences and purposes.

In the interest of enabling fair and effective funding and policy decisions, BAIs will need to work closely with governments and other funding partners to interpret the data collected through the PMF. What constitutes high performance for BAIs will always be subject to variations across regions (e.g., population densities, funding models, and proximity to complementary business support services) sectors and level of ecosystem maturity, among other things. It is incumbent upon all stakeholders to recognize that while performance benchmarks and comparisons across ecosystems are useful, caution should be exercised to ensure that data is interpreted using a sophisticated and nuanced approach thattakes context into account. For this reason, the pilot process is designed to enable a representative group of BAIs and policymakers to incrementally test, evaluate and refine the processes for data collection, aggregation, analysis and reporting to ensure that the PMF informs fair and effective decision-making by all relevant stakeholders.

1.2 Designing The Performance Measurement Framework

The process for developing a performance measurement framework for BAIs in Canada has unfolded over two stages to date, with stage three (the pilot) set to begin in the Spring of 2018. An evaluation of the pilot will take place in Q2/3 of 2019, following two data collection and reporting cycles. However, data collection and reporting activities will continue in parallel, with a national rollout planned for September 2019.

Figure 1: Developing a national performance measurement framework for BAIs

1. BAI Engagement (Autumn 2016). During the Autumn 2016, the Department of Innovation, Science and Economic Development (ISED) met with representatives from Canada's entrepreneur support ecosystem at three informal roundtables to discuss how government, entrepreneurs, investors and industry could work together to establish a national performance measurement framework for Business Accelerators and Incubators (BAIs). During the consultations, stakeholders expressed a willingness to collaborate on a national scale and, more specifically, to advance a national discussionon best practices in performance measurement.

   National Dialogue (February 2017). With the support of ISED, an industry-led group of BAIs took the next step toward framing a national solution for data collection and performance reporting by hosting a national discussion on February 10th in Toronto. Representatives from 18 organizations spent the day discussing BAI activities, opportunities and challenges, and exploring the benefits and challenges of creating a national performance measurement framework. The session provided a baseline understanding of what metrics BAIs are currently collecting and for what reasons, and helped to develop a shared understanding of how a national performance measurement framework could provide value to the community and its funders, as well as the challenges and obstacles that would need to be overcome to make industry collaboration successful.

2. Steering Committee and Feasibility Study (April–December 2017). In April 2017, a BAI Steering Committee consisting of a representative group of BAI leaders and policy makers formed to continue an inclusive national discussion and provide leadership in crafting a national performance measurement solution that works for the BAI community and its partners in government (see Appendix D for a list of Steering Committee and various Working Group members). The Committee's overarching mandate is to work alongside the Government of Canada in partnership to a) establish a performance measurement framework, and b) pilot a performance measurement platform for BAIs. It's first step was to initiate a feasibility study, during which the Committee drafted a list of common performance metrics and selected a platform for data collection and reporting. With the platform selected and metrics defined, the Committee proceeded to upload a set of test data to the platform with the objective of assessing the appropriateness of this metrics list for the pilot, and refining processes for data sharing and analysis (see section 1.3 below for a brief summary of the key findings from the feasibility study).
3. Pilot Project (March 2018–April 2019). The primary output of the Committee is a Performance Measurement Framework (PMF) and platform that will be in operation as a pilot by March 2018. To date, a provider has been selected for the platform that will be used for the pilot, the measurement framework has been refined, and the Steering Committee is in the process of inviting BAIs to participate in the pilot with the understanding that the pilot group should reflect the diversity of programming models and services offered by BAIs, along with the key economic sectors and regions of Canada. Beginning in March 2018, participating BAIs will complete their organizational profiles and upload their company benchmark and 2017 program data to the data sharing platform (see sections 2 and 3 for details. Depending on their internal data collection processes, participating BAIs will proceed to upload 2018 program data, either in quarterly batches throughout 2018 or for the entire annum during the first quarter of 2019.
4. Pilot Evaluation and Reporting (April–Sept 2019). In the April to September 2019 period, ISED will produce a report analyzing the data collected from participating BAIs. This report will detail, and where possible demonstrate, the types of descriptive and econometric analyses are possible using the data and other linked datasets from Statistics Canada and other federal partners. In parallel, participating BAIs and government partners will work together to evaluate the pilot process and identify valuable lessons and insights for managing a national performance measurement process with an expanded group of BAI participants. As part of the evaluation, BAIs will be consulted on how best to govern and manage the performance measurement framework and reporting process on an ongoing basis.
5. National Rollout (Sept 2019 and beyond). Following the pilot and evaluation phase, the Steering Committee and its partners in government will prepare for a national rollout of the PMF. Key activities will include making necessary adjustments to the PMF and platform and further encouraging government programs (federally and provincially) to adopt the metrics defined in the PMF to assess BAI programs. With these foundational steps in place, the BAI Steering Committee and its partners government can begin to invite BAIs from across the country to join in contributing their data to the performance measurement platform. As always, the Steering Committee will continually monitor and make adjustments to the PMF to ensure it remains useful to its stakeholders.

1.3 Key Outputs and Lessons Learned from the Feasibility Study

The feasibility study conducted between April and December 2017 provided the members of the BAI Steering Committee with an opportunity to draft a list of common performance metrics, select a platform for data collection and reporting, and upload a set of test data to the platform with the objective of assessing the appropriateness of this metrics list for the pilot, and refining the processes for data sharing. Along the way, the Committee reflected on the broader purpose and objectives of a national performance measurement framework and problem-solved a variety of technical and operational issues that were deemed likely to arise during the pilot. What follows is a very brief overview of some of the key discussion points and lessons learned during the feasibility study.

Defining a measurement framework. Among the first tasks for the Committee was defining a common set of performance metrics and complementary survey instruments for data collection. The results of this exercise are outlined in section 2 (performance metrics) and in appendixes A and B (the questionnaires for data collection). A consensus was reached that the performance metrics would focus on a core set of financial indicators linked to the annual revenues, employment, capital raised and intellectual property portfolio of client companies. A key challenge was arriving at common definitions for these indicators. While BAIs generally track the same outcomes (e.g., client revenues, employment and investment), they do so using different methods and differing indicators. Considerable time and effort went into crafting acceptable definitions for each metric. For example, how to define a job and which number of hours constitutes full-time employment; how to parse differing types of investment capital; and whether to track indicators using a calendar year or fiscal year. Other challenges included defining an approach to collecting information about founder demographics and establishing a common industry/sector list to ensure that both BAI programs and client firms could be categorized the same way. The effort to reach consensus on these issues had ensured that BAIs participating in the pilot will be tracking the same indicators using the same definitions and methods over the same time period.

Selecting a data sharing platform. With respect data collection and storage, the Committee determined that the pilot data will be aggregated into a secure central platform that complements established platforms/processes used by mature BAIs. For BAIs, the data platform solution had to be low-cost, secure, convenient and useful. For BAIs with existing CRM solutions and data management systems, it was also important that participation in the pilot would not require BAIs to transition to a new platform. For the federal government partners, it was important that the platform provide a secure and trusted environment for hosting and visualizing data on servers located in Canada, as well as a secure data export function to provide approved analysts with convenient access to data. With these criteria in mind, the Committee considered several available data platforms and, for a variety of reasons discussed further in section 3, chose Hockeystick as the data sharing platform for the pilot.

Designing an approach to data collection and analysis. Having defined metrics and selected a platform, the Committee proceeded to work on designing the data collection process and determining how data collected during the pilot will be analyzed. The details of the data collection process are outlined in section 2, however, it was determined that data would be collected from client companies upon entry to a BAI program to establish a benchmark, and for two full reporting cycles covering 2017 and 2018 BAI program cohort/entrant data. With regard to data analysis, the Committee agreed on three fundamental points. First, that the purpose of the data analysis is to draw robust conclusions about the economic impact of BAIs, for the benefit of BAIs, companies seeking BAI support, and governments that fund BAIs. Second, in the interest of enabling longitudinal analysis, strict protocols to protect client confidentiality will be followed to link the data collected to Statistics Canada and other Government of Canada sources using client names and business numbers. Third, that data analysis should be performed by a reliable, committed party capable of consistent interpretation of data. For the purpose of the pilot, it was determined that ISED will manage the data analysis and reporting process and work in partnership with Statistics Canada and approved researchers.

Performing a test run and planning for the pilot. With the platform selected and metrics defined, the Committee proceeded to upload a set of test data to the platform with the objective of assessing the appropriateness of the metrics list for the pilot, and refining the processes for data collection. Following a successful test run, key challenges at this stage included tweaking the metric definitions and survey instruments based on feedback from the participants; working with Hockeystick to clearly define the process for collecting, aggregating and exporting client data with a larger group of BAIs; deciding precisely how and with whom client data will be shared; determining how and when to obtain client consent to aggregate data using Hockeystick and to share data with the federal government for the purpose of research; and communicating the rationale and objectives of the pilot to the broader BAI community, while enlisting a larger group of organizations to participate. The results of these various activities and discussions are detailed in the remainder of the document.

1.4 Desired Outcomes from the Pilot

The Logic Model, shown in Figure 2, identifies the linkages between the activities of the PMF Pilot and the achievement of its results/outcomes. In addition to a number of immediate and longer-term outcomes, the Logic Model highlights four overarching objectives for the PMF Pilot. These include:

• A performance measurement framework with consistent metrics for measuring the impact of BAI programs on firm performance, along with clear definitions for each performance indicator;
• A pilot run of the PMF that demonstrates the successful collection and aggregation of consistent data from multiple BAIs across Canada using a data sharing platform that will streamline the data collection, analysis and reporting process;
• Shared commitment across the BAI community to using the lessons learned from the pilot to establish a sustainable institutional framework for rolling out the PMF on a national basis;
• And, over the longer term, enhanced, data-driven insights into the role of BAI programs in firm growth, and ultimately, superior economic growth enabled by the accelerated growth of high-potential firms working with BAIs across Canada.

Figure 2: Logic model for the PMF pilot

As the Logic Model indicates, the inputs and participants for the PMF clients include a representative group of BAIs and the programs and services they currently offer to client firms, federal government partners (particularly ISED) and the data sharing platform supplier. The activities are centred mainly on establishing the PMF, identifying a platform for the collection and aggregation of client data, enlisting BAIs to participate in the pilot, and obtaining consent from client firms to use their data for the pilot. The inputs and activities represent the foundational steps taken by BAI Steering Committee and ISED to create a PMF and prepare for a test run using the data sharing platform.

The logic model indicates that the outputs are linked to the pilot process itself. It is expected that participating BAIs will work with their client firms to collect all of the data specified by the performance measurement framework (or request their client firms to upload the data directly) over two data collection and reporting cycles: one for 2017 BAI program cohort/entrant data and one for 2018 BAI program cohort/entrant data. In both instances, the data sharing platform provider will aggregate and collate the data and prepare it for export. ISED and Statistics Canada will download the data according to the protocols defined in Section 2 and produce a report. Following these steps, all parties will participate in an evaluation of the pilot outcomes and document the lessons learned.

As depicted in the logic model, the pilot phase is expected produce a number of short and long-term outcomes as the BAI community works towards the realization of a national performance measurement solution for Canada. Immediate outcomes will be observable at the conclusion of the pilot in Spring 2019. Medium to long-term outcomes for the pilot are expected to take a year or more to manifest, with some outcomes – such as the use of data-driven insights to refine BAI programming and government investment priorities – taking several years to come to fruition.

The immediate outcomes for the pilot include:

• Agreement by stakeholders on what metrics should be collected. While the BAI Steering Committee and feasibility study ensured that a representative group of BAIs contributed to the process of identifying a common set of performance metrics, the pilot will open the door for a larger group of BAIs to provide feedback on the performance measurement framework.
• Agreement on standardized definitions for each metric in the PMF. BAIs participating in the pilot will likewise have an opportunity to provide useful input on the clarity and usefulness of the definitions provided in the BAI and company surveys.
• An understanding of how the data will be collected, stored, and analyzed. Processes for data collection, analysis and reporting have been designed, and partially tested during the feasibility study. The pilot will provide an opportunity identify potential pain points and alleviate any concerns about data security and confidentiality, among other things.
• An understanding of what the frequency of data collection should be. The default assumption is that company performance data points will be collected annually. BAI program profiles could also be updated once annually. However, the pilot will shed light on whether there are analytical or administrative reason for collecting certain metrics more or less frequently.
• A demonstration of success and lessons learned. The successful collection and aggregation of consistent data from multiple BAIs across Canada will demonstrate that performance measurement can work within a diverse ecosystem of BAIs. At the same time, the pilot will surface valuable lessons and insights for managing a national performance measurement process with an expanded group of BAI participants.
• Shared commitment to building a sustainable institutional framework. A successful pilot will also set the stage for discussions between BAIs and their partners in government about how best to establish a sustainable institutional framework for the ongoing management of a national performance measurement process.

Medium to long-term outcomes for the pilot include:

• A consistent baseline for longitudinal data. By using standardized metrics, the pilot will establish a consistent baseline for collecting longitudinal data over a longer period of time.
• Linkages to other datasets for econometric analysis. The pilot will enable researchers at Statistics Canada to determine the degree to which the resulting database can be linked with other datasets (for example, databases held by Statistics Canada, federal programs such as NRC-IRAP and industry associations such as the Canadian Venture Capital and Private Equity Association of Canada) and provide a demonstration of the types of descriptive and econometric analyses are possible using the linked data.
• Streamlined performance reporting. The pilot will demonstrate to federal, provincial and municipal funding bodies how multiple government departments can use the national PMF to collect the same data through a single process and platform, thereby lessening the reporting burden on BAIs.
• Enhanced insight into business growth and Canada's broader economic performance. The pilot will set the stage for developing a data driven understanding of the role of BAIs programs in Canadian firm growth. It will also highlight opportunities for collaboration and synergy between BAIs and provide a window into the performance of high-potential sectors and regional tech clusters across the country.
• Accelerated growth of supported firms. Over time, insights on how to most effectively support innovative growth-oriented firms in Canada will help BAIs to refine their program offerings. Improved options for support, in turn, will ensure that firms get the targeted, high quality services they need to build world class ventures in a wide variety of high-potential sectors.

In summary, a national performance measurement framework will provide BAIs with reliable and comparable data on which to make sound decisions, as well as timely information on the relevance, success and cost-effectiveness of their programs and activities. A common evaluation framework and reporting process will also ease the administrative burden on BAIs, while providing governments and other funders with a rigorous and objective evidence base with which to assess the performance of BAIs and make informed resource allocation decisions. Entrepreneurs across the country will be better equipped to identify the best options for support at various stages of their entrepreneurial journey.

The work Canadian BAIs are doing together at the national level will result in many other tangible benefits for BAIs, their clients and the broader economy. These benefits include the ability to share best practices across institutions and jurisdictions, establish relevant performance benchmarks for different regions and sectors, enhance input into public policy, and position and promote Canada as a destination for start-up activity. Ultimately, this work will inform BAIs and the Government on how to most effectively support innovative growth-oriented firms in Canada and these insights, in turn, will help accelerate the growth of world class companies in a variety of high value sectors. In other words, the collaborative efforts to build a common system for measuring the performance of BAIs could herald the beginning of an exciting new chapter in the growth and evolution of Canada's start-up ecosystem.