Evaluation of the Strategic Activities Program

October 2012

Recommended for approval to the Deputy Minister by the Departmental Evaluation Committee on

Approved by the Deputy Minister on


Table of Contents


List of acronyms used in this report

List of acronyms used in this report
Acronym Meaning
AEB Audit and Evaluation Branch
CA Contribution Agreement
CAS Comptrollership and Administration Sector
IC Industry Canada
O&M Operating & Maintenance
PSB Programs and Services Board
PSD Programs and Services Directorate
SAP Strategic Activities Program

List of tables

List of tables
Table # Table title
Table 1 List of SAP Funded Projects
Table 2 Project alignment with federal government priorities and IC strategic outcomes
Table 3 Comparison between a Treasury Board submission process and the SAP

List of figures

List of figures
Figure # Figure title
Figure 1 SAP Logic Model

Executive Summary

Program Overview

In 2008-2009, the Strategic Activities Program (SAP) was introduced to enable timely initiatives that support the department's strategic outcomes. Specifically, the program seeks to inform IC's policy development, program development and management practices. The potential recipients of SAP contributions can include research institutions and organizations, legal entities including incorporated entities, partnerships, cooperatives, trusts, associations and not-for-profit corporations. Since inception, five projects have been funded through the program, totalling $255,000.

Evaluation Purpose and Methodology

In accordance with the Treasury Board Policy on Evaluation and the Directive on the Evaluation Function, the purpose of this evaluation was to assess the core issues of relevance and performance of SAP. The evaluation findings and conclusions are based on the analysis of multiple lines of evidence. The methodology included a document review, project file review, and interviews.

Findings

Relevance

There is a continued need for an instrument such as SAP to provide Industry Canada with the means to support small, ad-hoc initiatives. These projects can provide the department with timely information that can help meet its mandate.

The projects funded through SAP are consistent with federal government priorities and Industry Canada's strategic outcomes. SAP's program design, including its selection criteria and oversight controls, ensure alignment with federal roles and responsibilities.

Performance

The uptake of the program has been lower than initially expected. Contributing factors include the unfunded nature of the program and the lack of awareness across the department about SAP.

For the projects that did receive funding, both the recipient organizations and the sponsoring IC sectors believed that SAP contributions played a key role in gaining and transferring knowledge. Recipients directly attributed the success of their projects to SAP contributions. In terms of IC impact, although difficult to link tangible impacts directly to the funded projects, departmental sectors provided evidence to suggest that SAP projects did inform policy development, program development and management practices.

As SAP was introduced without an increase to the department's reference levels, the program is in effect cost neutral. When compared to a Treasury Board submission process, SAP was significantly more efficient, which is of particular importance when considering the materiality of the projects. In that regard, SAP sufficiently demonstrated efficiency and economy.

Recommendations

The findings of the evaluation lead to the following recommendation:

  • CAS should increase the awareness of the program by re-establishing its presence and describing its benefits on the department's intranet.

1.0 Introduction

This report presents the results of an evaluation of the relevance and performance of the Strategic Activities Program. The report is organized into four sections:

  • Section 1 provides the program context and profile of SAP;
  • Section 2 sets out the evaluation methodology along with a discussion of data limitations;
  • Section 3 presents the findings pertaining to the evaluation issues of relevance and performance; and
  • Section 4 summarizes the study's conclusions and provides recommendations.

1.1 Program Context

In the past, Industry Canada (IC) was occasionally requested by external parties to provide contributions towards small ad-hoc initiatives that align with departmental priorities. Without a program in place to fund such initiatives, IC would either forego the project or pursue a Treasury Board submission process. In this context, SAP was introduced in 2008-2009 as a mechanism to provide the necessary contribution authority for departmental sectors to act quickly on opportunities that meet the department's objectives and priorities.

1.2 Program Overview

The purpose of SAP is to enable IC to support timely initiatives that align with the department's strategic outcomes. Specifically, the program seeks to integrate the economic, environmental and social interests of Canadians and influence strategic policy development. SAP initiatives can cover a broad range of areas, such as policy research, studies, conferences, symposiums, discussion papers, educational events, workshops, and the promotion and identification of best practices.

The potential recipients of SAP contributions are organizations that can advance initiatives that are aligned with Industry Canada's objectives and priorities. These organizations can include research institutions and universities, legal entities including incorporated entities, partnerships, cooperatives, trusts, associations and not-for-profit corporations.

The program is administered by the Programs and Services Directorate (PSD) within the Comptrollership and Administration Sector (CAS). The Directorate's primary responsibility is to guide the sponsoring sectors through the processes, guidelines and procedures of the program. Other roles and responsibilities include:

  • Sponsoring sectors: The groups within Industry Canada that provide the funding and support the initiatives. Their responsibilities include developing and managing the contribution agreement (CA) and monitoring performance.
  • Recipients: Thesegroups receive funding from the sponsoring sectors to launch their initiatives. Their responsibilities include implementing the initiative as outlined in the contribution agreement and reporting to sponsoring sectors on the progress of the projects and the results achieved.
  • The Programs and Services Board (PSB): The PSB is a committee of senior departmental officials who provide advice and make recommendations to program managers relating to grants, contributions and procurement activities. For SAP, PSB reviews all proposed initiatives to ensure consistency in its utilization and to prioritize competing proposals. Through this review, PSB provides assurance of departmental oversight of the SAP.

1.3 Program Resources

The SAP's funding authority is up to of $750,000 per year over five years (2008–2009 to 2012–2013) for a total of $3,750,000. Contributions under the program are funded from existing reference levels through budget transfers from sponsoring sectors to SAP. The incremental costs of administering the program are absorbed by the departmental budget.

To date, three projects were funded in 2008–09, totalling $100,000. Two additional projects were approved in 2012–13, totalling $155,000, for a cumulative total of $255,000 (see table 1).

Table 1: SAP Funded Projects
Recipient Project Description Contribution Amount
Couchiching Institute on Public Affairs
2008–09
The contribution went towards organizing the 77th Annual Couchiching Conference, the Power of Knowledge: The New Global Currency. The topic of the conference was on the "knowledge explosion" and how Canada could position itself to compete in a knowledge economy. $25,000
Carleton University, The Centre for Trade Policy and Law
2008–09
The contribution provided funding towards five research papers focusing on the new U.S. administration in 2009. The themes included economics and trade, resources and environment and, defence and security. $25,000
Impact Entrepreneurship Group
2008–09
The contribution funded a workshop to promote and support the development of entrepreneurship, business leadership and social responsibility among second and post-secondary students in Canada. $50,000
Institut de l'Énergie et de l'environnement pour la Francophonie
2012–13
The funding went towards a conference entitled "Le Forum mondial de la langue française". The topic of the event explored how Canada can position francophone content in the digital economy. $150,000
Centre for the Study of Living Standards
2012–13
The contribution provided support towards the publication of the volume "Towards Intelligent Government". The publication discussed the role of government in addressing social and economic policy issues. $5,000
Total: $255,000

1.4 Program Design and Delivery

In order to receive funding under the SAP, the applicant and the proposed project must meet the eligibility criteria. The SAP process has various assessment criteria that are used to evaluate whether the project can be regarded as strategic to the sponsoring sector. Once the sponsoring sector has had preliminary discussions with the prospective applicant, the sponsoring sector completes the SAP application form under the guidance of PSD.

When the application form is finalized, the sponsoring sector must receive support from the PSB. Discussions are based on the appropriateness of Industry Canada's involvement and how the project is aligned with the objectives and priorities of the sponsoring sector.

Once PSB has provided support, the sponsoring sector transfers the budget to the SAP fund centre and prepares the Contribution Agreement (CA) for the project. The CA outlines the obligations and responsibilities of both parties and the conditions under which payment(s) will be made. The project begins once the CA is finalized and signed. When the project has been completed, the recipient submits a report outlining the results achieved by the project.

1.5 Logic Model

The following logic model for the SAP was developed in 2008 and was updated in 2012 prior to the evaluation. The logic model outlines the program's activities and outputs, as well as the intended immediate, intermediate, and ultimate outcomes.

Figure 1: SAP Logic Model

Diagram of the SAP Logic Model (the long description is located below the image)
Description of Figure 1

This figure depicts a logic model for SAP. A logic model shows how program inputs (financial and non-financial resources) enable activities that produce outputs and in turn, how these outputs are expected to lead to different levels of outcome, culminating in the ultimate outcome for the program.

SAP has four principle inputs that enable the program's activities. The SAP inputs include:

  • Funding from sponsoring sectors,
  • Human resources,
  • Information Management and Information Technology, and
  • Accommodations.

With these inputs, the principle program activity is to administer SAP's processes, guidelines and procedures.

This activity results in the program's output. Outputs are the direct products stemming from the day-to-day activities of a program. For SAP, its output is the Contribution Agreement resulting from the SAP process that enables the eligible activities.

In turn this output leads to the program's immediate outcome. Immediate outcomes are generally considered directly attributable to the program's activities and are expected to occur in the short-term. For SAP, its immediate outcome is that the recipient of the funding engages in the eligible activities defined in the Contribution Agreement.

The immediate outcome results in two intermediate outcomes, one for Industry Canada and one for the recipient. An intermediate outcome is expected to occur once one or more immediate outcomes have been achieved. For SAP, the two intermediate outcomes are the following:

  • For Industry Canada, the new knowledge gained from the funded activities informs IC sectors' policy and program development and management practices, and
  • For SAP recipients, new knowledge is applied in the organization and other beneficiaries / parties.

The two intermediate outcomes lead to ultimate outcomes. The ultimate outcome is the highest level outcome that can be reasonably attributed to a program and is the consequence of the intermediate outcomes having been achieved. SAP has two ultimate outcomes, one for Industry Canada and the other for the SAP recipients:

  • For Industry Canada, SAP projects contribute to robust IC policy and program frameworks, and
  • For SAP recipients, SAP projects enable innovative and competitive industry sectors and a robust and innovative economy.

2.0 Methodology

This section provides information on the evaluation approach, objective and scope, the specific evaluation issues and questions that were addressed, evaluation methodologies and data limitations for the evaluation.

2.1 Evaluation Approach

The evaluation study was conducted by Industry Canada's Audit and Evaluation Branch.

2.2 Objective and Scope

The evaluation of this program was required under Section 42.1 of the Financial Accountability Act. This was the first evaluation of the program. In accordance with the Treasury Board Policy on Evaluation and Directive on the Evaluation Function, the purpose was to assess the core issues of relevance and performance.

Given that the program has only been in existence for four years with limited projects funded to date, the evaluation focused on the achievement of immediate outcomes and examined intermediate outcomes to the extent possible.

2.3 Evaluation Issues and Questions

Based on the SAP's Results-Based Management and Accountability Framework and consultations with PSD, the evaluation addressed the following questions:

Relevance

  • Is there a continued need for SAP?
  • Does SAP align with federal roles and responsibilities?
  • Does SAP align with federal government priorities and Industry Canada's departmental strategic outcomes?

Performance

  • To what extent are eligible recipients and departmental sectors aware of SAP and use it to fund eligible projects?
  • To what extent have recipient organizations (and other beneficiaries) applied the knowledge gained or transferred through SAP?
  • To what extent has SAP informed IC sectors' policy development, program development and management practices?
  • To what extent has SAP demonstrated efficiency and economy?

2.4 Evaluation Methodologies

The methodology for this evaluation was calibrated to take into consideration the low materiality of the program. SAP accounts for less than 0.1% of IC's forecasted spending for the 2012-2013 fiscal year. Nevertheless, multiple lines of evidence were used to address the evaluation questions. The data collection methods included a document review, project file review and interviews.

2.4.1 Document Review

A number of documents were reviewed for the purpose of this evaluation. These documents included Federal Budgets and Speeches from the Throne, IC reports, the SAP Treasury Board Submission (and Terms and Conditions), and the Results-Based Management and Accountability Framework.

2.4.2 Project File Review

SAP project files and sponsoring sector project files provided a comprehensive perspective of the outputs, activities and impacts of the program. These documents included Project Summary Forms, Statements of Work (with milestones and requirements), and Final Project Performance Reports. Some SAP projects resulted in published reports and other end-products, which were also reviewed to inform the achievement of the program's outcomes.

2.4.3 Interviews

Interviews provided in-depth information, including views, objectives and factual information that address the evaluation questions. The interviews were semi-structured in nature and were designed to obtain qualitative feedback from a range of respondents. In total, 13 interviews were conducted among the following stakeholder groups:

  • Representatives of the organizations who received SAP funding (4);
  • Industry Canada Sectors who sponsored projects (5);
  • PSD staff involved in SAP delivery (2); and
  • Representatives from Industry Canada Sectors who have not used the program (2).

2.5 Limitations to the Data Collection Methods

The primary limitation to the evaluation methodology was the availability of interviewees. As it has been four years since the completion of three of the SAP projects, some of the key Industry Canada participants involved in the projects, as well as the program itself, were no longer with the department. With regard to the recipient organizations, one group was not interviewed as the organization was no longer operating in the same capacity.

For the absent Industry Canada stakeholders, AEB was able to conduct proxy interviews with representatives who were not directly involved in the projects, but possessed the appropriate corporate memory to provide insight into the projects. These interviews still provided important information, and the limitations from their lack of direct involvement were minimal. As for the non-active recipient organization, sufficient information on this initiative was gathered from the other lines of evidence, including interviews with Industry Canada representatives and the file review.


3.0 Findings

3.1 Relevance

3.1.1 Is there a continued need for SAP?

Key Findings: Prior to SAP, Industry Canada did not have an instrument to provide the necessary contribution authority to support small, ad-hoc initiatives. There is an ongoing need for a mechanism to support ad-hoc initiatives, as they provide the department with timely information that can help meet its mandate.

According to program management interviews, prior to SAP's introduction CAS would be approached from time to time by IC sectors seeking advice on how to fund certain initiatives. These proposed projects were such that regular means of procurement (e.g. contracts) were not the ideal funding mechanism. CAS would inform IC sectors that there were no funding instruments available to support such initiatives and Sectors would either forego the initiative or pursue a Treasury Board submission process. In exceptional cases, for initiatives that had more than one federal department contributing funds, IC would transfer money to another participating department and that department would provide contributions on IC's behalf. To address this need, the SAP was introduced in 2008–09 as a mechanism to provide contribution authority to departmental sectors to act on such ad-hoc initiatives.

IC representatives who have used the program reported that there is a continued need for a mechanism like SAP, which provides an option to help them meet their mandate. The greatest application of the program was for groups within policy development, as four of the five IC sponsors were directly involved in policy work, which is fitting as transfer payments are a key instrument in furthering policy objectives and priorities.Footnote 1

As policy groups do not typically have the financial authority to fund projects through contributions, SAP largely addresses this need. In particular, SAP is designed to support, among others, research institutions, associations and not-for-profits. These types of organizations are typically composed of a wide range of stakeholders and provide multiple perspectives for policy development.

Specifically, one SAP project supported a research publication that focused on the role of government in addressing economic and social issues through policy influence. The recipient organization was composed of academia (economists), retired government officials and private sector representatives. Supporting such an organization would not have been possible under another type of funding instrument despite the benefits such a project would provide.

All five sponsoring groups stated that without the program, they would not have been able to provide contributions to their respective projects. Besides an inter-departmental transfer or a Treasury Board submission, there was no funding instrument that could have enabled the projects funded under the SAP. Sponsoring groups were clear that due to the low monetary value of the contributions, a Treasury Board submission process would have been an inappropriate use of the department's resources.

3.1.2 Does SAP align with federal roles and responsibilities?

Key Findings: The program aligns with federal roles and responsibilities as set out in the Department of Industry Act.

Due to the broad range of projects eligible under the SAP criteria, the program is an effective means for Industry Canada to deliver on its powers, duties and functions as legislated under the Department of Industry Act of 1995. Consistent with the function of SAP, the Act states that the Minister shall:Footnote 2

  • (b) collect, gather, by survey or otherwise, compile, analyse, coordinate and disseminate information in respect of matters under the Minister's administration, as well as in relation to trends and developments, both within and outside Canada, in respect of those matters; and
  • (d) promote cooperation with the governments of provinces and their agencies and non-governmental entities in Canada and participate, as appropriate, in the promotion of cooperation with agencies of other nations and international agencies.

The approved projects were all closely aligned with the roles of the department. For instance, the five projects supported under SAP fell into the following jurisdictional areas as defined by the Act:Footnote 3

  • (a) industry and technology in Canada;
  • (b) trade and commerce in Canada; and
  • (n) small businesses

Specifically, three of the projects aligned with IC's role of supporting industry and technology, by either aiming to advance Canada's knowledge economy, spur private sector innovation or commercialize Canadian digital content. Another project aligned with trade and commerce as it sought to improve trade relations between Canada and the U.S. Finally, the last project aligned with small business, as its objective was to raise awareness and to create networking opportunities for young entrepreneurs.

3.1.3 Does SAP align with federal government priorities and Industry Canada's departmental strategic outcomes?

Key Findings: The projects funded through SAP are consistent with federal government priorities as outlined in Speeches from the Throne and Budgets. SAP projects are also consistent with Industry Canada's strategic outcomes.

Projects funded through SAP are consistent with federal priorities and IC strategic outcomes. The high level of alignment with government priorities and strategic outcomes was built into SAP's program design. In order for a project to be approved, SAP proposals must clearly demonstrate alignment, and PSB reviews recommended projects to ensure they are consistent with federal priorities and strategic outcomes. The following table demonstrates this alignment:

Table 2: Project Alignment with Federal Government Priorities and IC Strategic Outcomes
SAP Project Federal Government Priorities Industry Canada Strategic Objective

Carleton University – The Centre for Trade Policy and Law

"Blueprint for Canada-US Engagement under a New Administration"

Both investment and trade matter to Canada's prosperity. Our Government is committed to seeking out new opportunities for Canadians and to promoting global prosperity through free trade. (Speech 2008)

This project fulfills federal priorities through its advancement of knowledge in trade policy with the goal of strengthening the national economy as well as the integration of economic, environmental and social interests of Canadians through strategic policy development.

  • Advancements in science and technology, knowledge, and innovation strengthen the Canadian economy
  • The Canadian marketplace is efficient and competitive

IC's strategic objectives are met as the papers produced through this project will lead to increased understanding for designing economic and trade policy, thus strengthening the Canadian marketplace.

Couchiching Institute on Public Affairs

"The Power of Knowledge: The New Global Currency"

Knowledge Advantage: In the modern global economy, the most successful nations are those that best combine people, skills, new ideas and advanced technologies to create a competitive edge. Canada must be well positioned to succeed in this new environment. (Budget 2007)

This project feeds into the government's commitment to fostering a knowledge-based economy.

  • Advancements in science and technology, knowledge, and innovation strengthen the Canadian economy
  • The Canadian marketplace is efficient and competitive

This project supports IC's objective of advancing these policy areas to strengthen the Canadian economy.

The Centre for the Study of Living Standards (CSLS)

"Towards Intelligent Government"

Part of a solid economic and fiscal foundation is the sound management of government. To make Canada's national government more effective, our Government is committed to reform and streamline the way it does business. (Speech 2008)

Project supports the government's commitment to reform its role and streamline its policies to better serve the Canadian population.

  • The Canadian marketplace is efficient and competitive
  • Canadian businesses and communities are competitive

This project is consistent with IC's strategic objectives as it provides information on ways to improve the Canadian economy and ensure that markets are efficient.

Institut de l'énergie et de l'environnement pour la Francophonie (IEPF)

« Forum mondiale de la langue française »

Leadership in the creation, adoption and use of digital technologies and content will help Canada to increase its world-class standard of living. (Budget 2011)

The Forum will feed into the government's priority of fostering a stronger digital economy as well as supporting bilingualism in Canada.

  • The Canadian marketplace is efficient and competitive

This project is consistent with Industry Canada's mandate and objectives to ensure that new digital markets are efficient and competitive.

Impact Entrepreneurship Group

"IEG Leadership Summit"

Budget 2010 invests in measures that will directly protect jobs. This includes extensions to work-sharing and investments in training and skills development for youth. (Budget 2010)

This summit supports federal priorities to protect national employment levels and increase entrepreneurship in youth.

  • Canadian businesses and communities are competitive
  • The Canadian marketplace is efficient and competitive

The summit is in line with IC's strategic objectives as it will support a competitive marketplace.


3.2 Performance

3.2.1 To what extent are eligible recipients and Departmental sectors aware of SAP and use it to fund eligible projects?

Key Finding: The uptake of SAP was lower than initial expectations. Contributing factors include the unfunded nature of the program and the lack of awareness across the department of SAP.

Uptake for the program has been lower than expected, with three projects funded in 2008–09 and an additional two in 2012–13. During the development phase of the program, PSD held consultations with various sectors of the department. Based on these consultations, PSD anticipated funding 2 to 5 projects for the first year and up to 10 projects in subsequent years. Besides the projects that were funded, other sectors have approached the program with potential initiatives, but these discussions did not advance past the preliminary consultation stage since either the projects did not align with the objectives of SAP, or the program was not the appropriate mechanism.

The availability of funding within sectors was a limiting factor to SAP uptake, given the program is unfunded (i.e. sponsoring sectors must have the available funds to transfer to SAP). As sectors plan their spending early in the fiscal year, having available funding to provide unplanned contributions can be a barrier, depending on the amount of contributions sought. One interviewee mentioned that they would need to know of the upcoming project at least by March, in order to work the contributions into their budget for the next fiscal year. If a directorate did not have the available funding, they would seek the funding from other parties within the sector, creating a new set of complications. Recently, this theme was particularly relevant, as the department has been undergoing expenditure reviews, both in the forms of a Strategic Review followed by the Deficit Reduction Action Plan. In interviews, IC sectors that were yet to use the program recognized its benefits, but were concerned with their ability to find funding during times of fiscal restraint.

Interviews with IC sectors suggest that another contributing factor to the program's low usage was lack of awareness. A consistent theme from managers who used the program was that they had no prior knowledge of SAP until it was required. When interviewed, sectors that have yet to use SAP had no prior knowledge of the program.

The lack of awareness is mostly the result of the program's communication strategy, its unfunded status and its low materiality. After the initial launch, CAS did not actively promote the program and took a more practical strategy and played a reactive role. Only when a sector approached their group with an initiative did PSD present SAP as a potential funding solution. The primary promotion strategy CAS employed was word-of-mouth.

As the target audience of the program was the managers across Industry Canada, their lack of awareness may have limited opportunities that could benefit the department. In interviews, it was suggested that some managers may have dismissed potential initiatives because they assumed that a mechanism was not available to enable the initiative. Thus, the lack of awareness may have limited the number of proposed initiatives presented to PSB.

Further compounding the issue, SAP does not currently have a presence on the department's intranet. At one point, it had a dedicated webpage, which was removed when the website was taken down for maintenance and was subsequently not reinserted. In addition, the loss of corporate memory across the department may also have had an effect on awareness.

3.2.2 To what extent have recipient organizations (and other beneficiaries) applied the knowledge gained or transferred through SAP?

Key Finding: According to program recipients, SAP played a key role in gaining and transferring knowledge. Recipients directly attributed the success of their projects to SAP contributions as the funding allowed greater scale, broader scope, and increased timeliness of initiatives.

All recipient organizations interviewed for this evaluation associated the success of their respective projects to contributions provided by SAP. For the conferences and network events, recipient organizations were of the opinion that the contributions did have a considerable impact, especially with the content delivered. For these events, the funds were used to allow distinguished guest speakers to attend the events. As guest speakers are the primary draws to these types of events, their presence can attract attendants, media bodies and even other speakers. According to recipients, SAP allowed their projects to attract the right speakers and was critical to the event scale and scope. In that regard, having SAP funding played an important role in the transfer of knowledge.

Other projects under the SAP umbrella went towards the publication of policy research papers. These funds were disbursed as payment to the authors to conduct their research. Once these papers were completed, they were distributed to federal departments, provincial governments and private sector sponsors. In interviews, recipients stressed the importance of the SAP funds, and were doubtful that the papers could have retained the same scale or scope without the funding. For example, for the project with The Centre for Trade Policy and Law, there was a pressing need to secure funding quickly, as findings from the reports needed to be released in a timely matter in order to influence decision-making.

3.2.3 To what extent has SAP informed IC sectors' policy development, program development and management practices?

Key Finding: Although difficult to associate tangible impacts directly to funded projects, there is evidence that links SAP projects with policy development, program development and management practices.

SAP is one of many tools used by the department to inform policy development, program development and management practices. This makes it difficult to attribute the one-off  events of SAP initiatives to tangible impacts since policy, program and management changes often arise from the integration of multiple sources. Despite these limitations, sponsoring sectors were of the opinion that the projects did have an impact on their sector and on the department as a whole. Although sectors could not provide a direct causal impact, there is evidence to suggest that the information gained through the SAP initiatives was one of many sources that led to new policy development, program development and management practices.

For instance, the two projects sponsored by the Spectrum, Information Technologies and Telecommunications Sector fed directly into the policy thinking for the department's Digital Economy Strategy. Interviewees noted that the 2008 project fed into the preliminary thinking about the department's need to develop a strategy, while the 2012 project addressed specific strategy issues. The sponsoring sector made note of the importance of the projects and stated that both forums brought together key stakeholders across government and industry to discuss policy issues. Specifically for the 2012 forum, the discussions brought together over fifteen hundred delegates to discuss themes such as commercializing French-Canadian digital content.

An example of SAP's impact on program development and management practices was the initiative sponsored by the Strategic Policy Sector, which provided contributions towards research papers concerning trade between Canada and the U.S. These papers played an important role in the formation of a new directorate within the department, the Security and Prosperity Partnership Secretariat (which was later split into the Policy Integration and the International and Trade Policy Directorates). Later, the papers provided some groundwork for the Beyond the Border Working Group program, which began as an Economic Action Plan initiative with the objective of bringing further trade coordination between the two countries.

3.2.4 To what extent has SAP demonstrated efficiency and economy?

Key Finding: SAP was introduced without an increase in the department's reference levels, making it effectively cost neutral. When compared to a Treasury Board submission process, SAP is significantly more efficient, particularly for relatively low amounts of contributions.

SAP is distinct in the sense that it is not a traditional contributions program with allocated O&M, but a funding mechanism for IC managers to use if required. The program's contributions are funded from existing reference levels through budget transfers from sponsoring sectors and any O&M costs to administer the program are absorbed by the departmental budget.

Given that the program was introduced as an alternative to pursing separate contribution authorities through Treasury Board submissions for each initiative, it is appropriate to assess the efficiencies achieved with the SAP when compared with the alternative. The main source of the efficiencies achieved by SAP is that it streamlines the approval process while maintaining proper accountabilities. Table 4 provides a brief overview of the differences between the two processes.

Table 3: Comparison between a Treasury Board submission process and SAP
Indicators Treasury Board Submission Strategic Activities Program
Average time from project initiation to CA 4 to 7 months 2 weeks to 1 month
Number of sign-offs 4 to 7 sign-offs 1 sign-off
Number of distinct "steps" in processTable note 4 14 8

When comparing with Treasury Board approval, the level of time and effort is significantly decreased when funding a project through SAP. Most significant is the average time from project initiation to Contribution Agreement. As SAP projects are ad-hoc, there is typically a need to get an agreement finalized quickly, as a project presented in the spring may need funding by summer. Depending on the level of commitment from the sponsoring sector, a SAP project could be finalized within a matter of weeks.

Another significant source of efficiency derives from the number of sign-offs. Every Treasury Board submission requires sign-off from various sectors and branches in the department and depending on the submission, can vary from 4 to 7 sign-offs, each with their own review process. This is in contrast with SAP, where the only sign-off required is that from the participating sector itself, following PSB support.

Lastly, the number of distinct steps in the two processes was indicative as well. In order to get a submission through, many different parties, both within and outside the department, must be involved in the review process. For instance, among other steps there are consultations with corporate sectors, consultation with the Treasury Board Secretariat, document development as well as Ministerial approval. Avoiding this front end process with SAP was regarded as a major benefit by all participants.

The incremental level of effort between SAP and a Treasury Board submission was considered material to Industry Canada managers. Interviews indicated that recipients would never have pursued their projects in the absence of SAP. For these kinds of initiatives, the Treasury Board submission process was considered time consuming and involved too many participants for contribution amounts of relative low materiality. Managers believed that the level of effort to distribute contributions through the Treasury Board submission process would have been inappropriate and an inefficient use of both internal and external resources.


4.0 Conclusions and Recommendations

4.1 Conclusions

Regarding the relevance and performance of the program, the following conclusions can be reached.

Relevance:

  • There is a continued need for Industry Canada to support ad-hoc initiatives and SAP is an appropriate mechanism to do so.
  • The program aligns with the Department of Industry Act of 1995.
  • The projects funded under SAP are consistent with federal government priorities and the strategic objectives of Industry Canada.

Performance:

  • The uptake of the program was lower than initially anticipated which can be partially attributed to its unfunded nature and a lack of awareness of the SAP across the department.
  • Recipients indicated that SAP played a key role in the gain and transfer of knowledge. They stated that the contributions impacted the scale, scope and timeliness of the initiatives.
  • There is evidence to suggest that SAP projects were one among other sources that influenced policy development, program development and management practices.
  • SAP is effectively cost neutral as funding is sourced from existing reference levels. The SAP process was regarded as efficient and appropriate when compared to a Treasury Board submission process.

4.2 Recommendations

The conclusions of the evaluation lead to the following recommendation:

  • CAS should increase the awareness of the program by re-establishing its presence and describing its benefits on the department's intranet.

PDF version

Management Response and Action Plan

Management Response and Action Plan - SAP
Recommendation Management Response and Action Plan Management Accountability Action Completion Date
  1. CAS should increase the awareness of the program by reestablishing its presence and describing its benefits on the department's intranet.

Agreed.

Program information, Terms & Conditions and operational procedures, will be posted on the intranet.

Action Plan

  • Revise and update procedures
  • Page set-up, coding
  • Posting on intranet

Director, PSD