2019 BAI performance measurement framework: Partipant feedback

PART I. Mandate and Progress to Date (continued)

1.5 Participant Feedback and Lessons Learned from the Pilot

In parallel with the data analysis and reporting activities, participating BAIs and government partners worked together to reflect on the pilot process and identify valuable lessons and insights for managing a national performance measurement process with an expanded group of BAI participants. This section details the findings from a participant survey, a feedback session held at the BAI mini-summit on February 20, 2019 and a series of one-to-one interviews. It also outlines the improvements that have been made to the PMF and data collection process in response to the feedback received from the community of participants.

Findings from the Participant Survey

Following the conclusion to the data collection phase, TEC Edmonton administered a short survey to better understand how BAIs participated in the pilot, to identify challenges encountered by participants, and to solicit input from BAIs on accommodations that would enhance phase II of the pilot. A total of 20 BAIs responded to the survey.

Data collection methods for phase I

The first section of the survey asked respondents to provide further details on how they went about collecting and submitting 2017 data to the PMF pilot. Among other things, this included the number of data sources consulted to complete the questionnaire, the method for submitting data to Hockeystick, and whether this method would be scalable to a larger number of firms. For the purpose of this report, the main finding of interest is that only some of respondents (4/20) were confident that the method used to collect the 2017 data would be scalable to a larger number of firms. Four BAIs indicated that they thought process was not scalable, and a further 6 said 'maybe' or they 'weren't sure'.

Data collection methods and challenges for phase II

The next batch of questions asked respondents to provide further details on how they intend to collect and submit data for phase II of the pilot. 10 of 20 of the BAI respondents indicated that they intend to use an annual survey to collect data from client companies, while 8 plan to use a mixture of methods, and 2 collect data on a quarterly basis.

Participants were asked whether there are any specific questions or issues that may prevent their organizations from collecting and submitting 2018 data for the PMF Pilot. The majority of BAIs (15 of 20) indicated there were no questions or issues that would inhibit their participant. For the remainder, BAIs cited service level agreements that prohibited the sharing of client data and low response rates from client companies as key challenges. Participants were also asked what kind of supports would help them adopt the performance measurement framework for 2018. 11 of 20 BAIs indicated that a simple consent statement and communication tools to use with client firms would be helpful. Other requests included a reduction in the length of the survey instrument (7/20 BAIs) and greater clarity for some of the questions (7/20 BAIs).

Data confidentiality

BAI participants were asked whether they think the involvement of Statistics Canada in the pilot process will give the majority of their clients' confidence that their data will be stored and used in a secure and appropriate manner. Seven BAIs answered affirmatively, 11 were unsure and 2 said that the involvement of Statistics Canada in the pilot process would not be a major factor.

Data utility and opportunities for national analysis

BAI participants were asked whether and how they planned to use the data collected for the PMF pilot. The majority of BAI respondents indicated that they are using the data in their reporting activities to government partners and other sponsors and many are also using the data to inform their program marketing materials and funding/grant applications. A smaller number of respondents indicated that they are beginning to use the data to inform strategic decision-making, shape future programming, identify service gaps, and implement quality control processes.

BAI participants were also asked whether there were specific questions they would like to be able to answer using the data collected at the national level. Participant responses included:

  • Better understanding which programs/organizations are providing the most efficient and effective support for startups.
  • Determining the degree to which a venture's growth trajectory is positively affected by participation in a BAI, and specifically how those trajectories are typically affected.
  • Using the data to feed benchmarking initiatives like Startup Genome and to enable BAIs to benchmark their impact on company success and growth against other BAIs across Canada.
  • Building a national picture of how many tech startups exist in Canada, how many people they employ, and their growth rates over time.
  • Comparing the growth performance and valuations of BAI-supported companies with similar cohorts of non-BAI companies in different industry segments.

Feedback from the BAI Mini-Summit and Participant Interviews

The February 20, 2019 Mini-Summit in Waterloo provided an additional opportunity for BAI participants to provide their feedback on the pilot experience, including the survey instrument, the process for collecting and analyzing performance data, the implementation support provided by ISED and Hockeystick, and potential future direction of the performance measurement framework. BAIs were also consulted on how best to govern and manage the performance measurement framework and reporting process on an ongoing basis. A series of one-to-one interviews were also held with officials representing the regional development agencies (RDAs) participating in the BAI PMF Mini-Summit. The feedback and insights received from BAI participants and government partners has been synthesized into six categories.

Utility of the PMF for BAIs

There was broad agreement among the BAIs that participation in the pilot created value for them that went well beyond their pre-existing data collection and reporting activities. That value manifested in several different ways.

  • Enabling program analysis. Rather than evaluating the impact of their BAI as a whole, several organizations noted that the PMF got them doing a program-by-program analysis for the first time. "Getting feedback on the individual programs is valuable," said one BAI leader. "We need to use data to understand what is working so that we can refine our programs over time. We want to establish KPIs to evaluate whether we should we be dropping some programs or adding to others."
  • Introducing new indicators and milestones. For BAIs that were already collecting and analyzing performance data, several that noted that the PMF introduced new indicators that they were not yet tracking. For example, many BAIs started tracking founder demographics for the first time using the PMF. "The demographic insights are very interesting," said one BAI leaders. "It leads to a better understanding to who we are supporting and how. We found quite a bit of attrition with women entrepreneurs and a big disparity generally between male and female founders." Some BAIs are now actively thinking about how new data collected through the PMF can be used to develop internal milestones. Such milestones could be used to track things like how long it takes for supported companies in different sectors and different programs to go from incorporation to first sale. "We want to understand the hard outcomes and growth stage milestones that companies need to achieve to grow," said one BAI leader. "If we can articulate and measure what the hard milestones are, we can design our programs to help them achieve those milestones."
  • Fostering a performance management and customer service ethos. Several BAIs noted that participation in the PMF was helping to instill a more rigorous culture of performance management in their organizations. "We want to track how companies are performing," said one BAI leader. "This ties into our KPIs. We are a for-profit organization, so we need to track revenue and investment and the PMF provides an instrument to help us do that." Another BAI leader reported how collecting the net promoter score encouraged program managers to seek more detailed feedback on entrepreneurs from each group, including promoters, passives and detractors.
  • Supporting internal training and program marketing. Finally, one BAI noted that the PMF was supporting their internal employee training process by supplying the data they needed to train their staff on how to better support companies. Several BAIs also reported that they are now using the data collected through the PMF for promotional purposes.
The measurement framework and survey instrument

With its focus on revenue growth, capital investment and employment growth, there is broad agreement among the BAI participants that the measurement framework is focused on the right high-level metrics. However, there are some areas that BAIs pinpointed for improvement or further consideration.

  • Clarity. Clear questions and good definitions are critical to getting reliable data back from BAI companies and to making the data collection experience as seamless and friction-free as possible. There was a sense among participants that some survey questions may have been too complicated and/or lacked clear definitions.
  • Parsimony. BAIs would like to see a shorter and more streamlined survey instrument. Many feel that a shorter survey and a more intuitive structure would lessen the burden on respondents, generate better response rates and lead to a lower drop-off rate.
  • Early stage metrics. At the risk of complicating or elongating the survey, a number of BAIs have noted that the measurement framework lacks metrics that work well for organizations and programs that principally work with early stage, pre-revenue companies. As one participant put it, "We report back a lot of zeros because we have pre-revenue companies that have yet to raise capital." It was suggested that the PMF should try to account for how early-stage incubation and acceleration activities converts to greater economic impact later on in the company maturity cycle.
  • Leading versus lagging indicators. On a related note, several BAIs noted that the majority of the metrics are lagging indicators. Measurable and sustained growth in revenues, capital investment and employment, for example, may not manifest until months or years after a client firm has left the BAI program. In the short term, these indicators are not very useful in determining whether a BAI program is working or not. Measures of client satisfaction such as the Net Promoter score and BAI impact question were cited as indicators that provide more immediate feedback on program success.
Data collection process

The data collection process for the PMF pilot generated the most discussion amongst participants, many of whom shared both concerns about the pilot experience and suggestions for how to improve the data collection process going forward. The key recommendations for improving version 2.0 of the PMF included the following:

  • Lessen the data collection burden as much as possible. A key rationale and objective for the BAI PMF is to reduce the reporting burden on BAIs by aligning reporting requirements across funding partners in government. BAIs are eager for this reality to come to fruition and voiced their support for continued efforts to bring other government partners into year two of the pilot process. "We need a single process, with one annual request for data," said one BAI leader, who also noted that operational costs are very high for managing the data collection process. However, it is not only BAIs that are feeling the data collection burden. Firms get inundated with requests too because they are getting requests from the multiple institutions that they have interacted with. "We need to make it as frictionless as possible," said one BAI leader. "Don't collect too much data and collect data that is meaningful."
  • Provide tools to deal with concerns about data privacy and security. BAIs noted that it was sometimes hard to convince clients that the confidential data they report will be safe and secure. Said one BAI leader: "For a lot of companies, they are concerned about big brother and the security of their data. They don't want their data shared. They really worry about their competitors seeing their revenue numbers." While BAIs understand the importance of managing client expectations and assuaging their data privacy fears, they don't feel fully equipped to do so. As one BAI leader put it, "We had to do 6 months of work to get client companies to better understand how the data will be handled. We need a more succinct description of how the data flows from Hockeystick to ISED to Statistics Canada."
  • Create a value proposition for the client firms. While there was quite a bit of discussion about how to make the PMF valuable to BAIs, there has been less focus on how to make the data reporting process more valuable to client firms. BAI participants suggested that a greater focus creating direct value for the end client would not only boost the response rates, but also lead to better outcomes for the ecosystem. "How do we make it easy? How do we ensure they see it as a valuable exercise?" asked one BAI participant. Some noted that NRC-IRAP receives good data from client companies because the entrepreneurs don't get the last cheque until they have supplied the data. As they put it: "The people with money have the leverage to get the data from client companies." While the PMF process can't deliver cheques to client companies, it can deliver benchmarking opportunities and valuable information about BAI services, ecosystem trends and other things that founders will find useful. The sooner these opportunities are realized, the easier it will be to get client companies across Canada engaged in the process.

Several BAIs shared lessons and strategies for increasing their survey response rates.

  • Include data reporting in client service agreements. The BAIs that were most successful in their data collection efforts had already baked data sharing rights into their service agreements with client firms. Other BAIs were less fortunate, with some cases where existing client service agreements specifically forbade the sharing of non-anonymized performance data. As the pilot moves into phase II, BAIs without data collection and sharing rights embedded in their client service agreements are now having to update their agreements to accommodate this.
  • Use data collection as a way to build the financial literacy of client companies. One BAI leader argued that it helps to get client companies accustomed to sharing their data early since they will face similar requests from VCs and Angels. "We argue with founders that they will need to share data with Angels and VCs, so they better get used to it. It's an education process. We request data on a quarterly basis which makes it very similar to a board meeting cadence." Others argued that regular data reporting instills good management discipline. "It is really important for companies to measure what they are doing and to hold themselves accountable," said one BAI leader. "It's not just about our organization telling our story and claiming successes. This is for the entrepreneurs. It's building their financial literacy."
  • Timing is important. The advantages and disadvantages of annual and quarterly reporting were discussed, with arguments made on both sides. Some argued that quarterly was better, pointing out that annual collection creates a long lag for metrics that change quickly for fast moving companies. "The longer the lag in reporting, the less likely they are to answer correctly," said one BAI leader. Some argued quarterly reporting generates a better response rate because data request comes while the BAI is still providing support or very shortly after the relationship has ended. "We get higher response rates when we have engaged with them recently," said another BAI leader. Others argued that fewer touch points are better and that issuing surveys once a year is optimal, especially if BAIs can use the PMF to report to both levels of government at once.
  • Ensure that requests for data come from recognizable entities. Finally, there was a recommendation to send surveys requests from recognizable entities and individuals. One BAI sends requests from the mentors who work directly with the client companies. "If they don't know the entity or the individual," said one BAI leader, "the response rate will be really low."
Implementation support

A key piece of feedback from BAIs is that implementation support will be vital to a successful rollout of the PMF on a pan-Canadian basis. Many BAIs reported that the data collection process was time consuming, especially tracking survey completions, sending out reminders to client companies, and cleaning up data contributed by client companies.  With the largest BAIs being an exception, many organizations do not have full-time staff members dedicated to data collection and reporting. Staff turnover and a lack of residual knowledge pertaining to the PMF pilot also turned out to be challenge. For example, a number of BAIs required multiple onboarding sessions as new recruits were brought in partway through the pilot process.

BAIs suggested that they need better communications materials to educate internal staff (including management and legal executives) and to support their outreach to client companies. BAIs also noted that they would like better visibility into the timing of key pilot activities so that they can allocate resources accordingly.

Data analysis and reporting

The February 20th meeting in Waterloo provided the first opportunity for BAIs to see ISED present an analysis of the data aggregated from phase I pilot. This was clearly a beneficial exercise for all involved as it became clear, even at this early stage, that the data would generate powerful insights regarding the role that BAIs are playing in supporting the growth of innovative, technology firms in Canada. Subsequent discussion with BAIs led to a couple of additional suggestions and issues for consideration. 

  • BAI Benchmarking. There is a clear appetite among BAIs for the PMF to enable benchmarking with their peers. For example, a taxonomy to distinguish BAI types could be used to delineate different company growth benchmarks for early stage incubators and later-stage business accelerators. It was also noted, however, that benchmarking on a program-by-program and sector-by-sector basis will only be possible once the PMF has collected a larger number of observations from a larger number of institutions in order to enhance the quality of the analysis and protect the anonymity of the participants.
  • Attribution. It is a well-known fact that a lot of company founders bounce around a lot and touch many parts of the ecosystem. This creates a challenge in attributing the success of a given company to an individual BAI. As one BAI leader put it, "The engagement of StatsCan will help with the authoritative nature of the data. However, there will be companies that will have worked with multiple different institutions and we will all try to claim credit for their successes." Others called for less focus on claiming successes and more focus on building local ecosystems. "We hope that by collecting data we get a better sense of whether we are adding value," said one BAI leader. "We have no other way to know whether we are doing a good job. I don't need to broadcast the success. We should just focus on building the ecosystem."
  • Customized BAI dashboards. Several BAIs, especially those that have not invested in sophisticated CRM systems, are hoping that Hockeystick will develop a dashboard for analyzing and interacting with the data they collect through the PMF. Some suggested that creating dashboard capabilities for BAIs was one way to demonstrate value for the investment of time and resources BAIs put into participating in the pilot. It was noted that one challenge in building BAI-specific dashboards relates to the fact that doing so would require a non-trivial investment of time to validate and patch up the data contributed by client firms.
  • Potential sample bias. Finally, it was suggested that there could be some bias evident in the results due to the underreporting of data by companies that are struggling and those that are super successful. Although based on conjecture, the assertion was that silent entrepreneurs (i.e., those that elect not to respond to survey requests) are usually not successful. On the other hand, it was suggested that super successful companies may also be reticent about contributing data for fear that confidential details could leak out to competitors.
Future directions for the BAI PMF

BAI participants in the pilot also provided suggestions on future directions for the PMF, beyond those already documented in the categories above. Ideas and suggestions from participants included the following:

  • Mapping Canada's startup ecosystem. Several participants would like the PMF to shed light on the overall size and shape of Canada's startup ecosystem. They noted that it would be good to document the size of the technology startup population is in Canada and the role of BAIs in shaping that population. For example: How many companies are entering BAIs? How many are exiting? How much entrepreneurial capacity are BAIs generating? What is the distribution of those companies across sectors and geography?
  • Understanding the startup support landscape. On a similar note, several BAIs argued that it would be useful to have a way to better understand the pipeline or landscape of support organizations and programs in Canada. As one BAI leader put it, "We are doing discovery on programs the serve companies before they come to us, as well as where companies go for support after they leave our program. That will help us understand what companies already know and what they still need to learn. What fundamentals still need to be covered during our program? Right now, it is very hard for us to know who is doing what."
  • Documenting service trends. One BAI leader called for better insights into service trends across the startup support landscape. This particular BAI noted that they saw a huge influx of startups in 2013/14, a number which has been slowly tracking downwards since then. As a result, they are spending less time on high volume, light touch interactions and while the amount of time they spend with quality startups has gone up. There was a shared interest in knowing whether other BAIs across Canada were seeing similar or different service trends in the sectors and geographies they cater to.
  • Managing scale with automation. For the larger BAIs, data collection and reporting are challenging to do at scale. "We are dealing with 1,000 companies a year," said one BAI leader. "It's hard to do some of the follow-up for survey completions that one can do with a smaller number of companies." It was suggested that potential solutions could involve finding ways to automate as much of the data collection process as possible.
  • Understanding exits and company deaths. Finally, there was a call for better understanding the causes and circumstances of company deaths. While BAIs are understandably keen to track the success cases, very little attention is given to the companies that exit BAIs and fail to survive. "We don't often follow-up with the companies that died," said one BAI leader. "it's hard to know what happened and why, but there could be some value in better understanding the challenges or issues that prevented the company from succeeding."

Key Improvements for the PMF 2.0

A number of improvements for phase II of the pilot are either in the works or have already been implemented in response to the feedback from the community of participants. These measures address the core concerns raised by the participant community, including the following:

  • Onboarding materials and implementation support. ISED has developed a package of on-boarding and communications materials to educate BAIs about the PMF and to lessen the need for time-intensive direct outreach and support. A draft version of these materials was presented to BAI participants at the Mini-Summit. The team at ISED is also always available to provide support on the PMF itself, and Hockeystick is available to provide technical support pertaining to the data collection platform.
  • Addressing concerns regarding data security and confidentiality. The aforementioned communications materials contain a clear explanation of the Statistics Canada's approach to data protection and data management that should help alleviate concerns about data security and confidentiality. A similar description is available in Section 2.3 of this report. Additionally, Statistics Canada has prepared a presentation overview of its approach, which can be obtained from ISED upon request.
  • Obtaining consent to share data. ISED continues to collaborate with the community on consent language and has developed suggested language that organizations can use. There are also a number of organizations that have implemented their own systems to ease the process of obtaining consent. BAIs are encouraged to communicate with one another other with respect to best practices. ISED can facilitate these conversations as appropriate.
  • Lessening the data collection and reporting burden. Several steps have been taken to make the data collection process as efficient and streamlined as possible. First, the length of the survey for the PMF 2.0 has been reduced by eliminating low-response rate questions. ISED has clarified a number of questions where issues with interpretation have been identified. Second, BAIs that have gone through the phase I process will have worked out some of the implementation kinks, created some institutional knowledge, and established better alignment between their internal data collection and CRM processes and the PMF. This should make year two of the pilot considerably easier. Third, the involvement of other government partners in the PMF will eventually streamline reporting requirements and further lessen the data collection burden on BAIs and the implementation support burden on ISED.