Mid-Term Evaluation of the Northern Ontario Development Program (NODP)

Final Report

May, 2008

Tabled and approved at DEC on June 9, 2008

Table of Contents

• Executive summary
• 1.0 background
  • 1.1 Understanding the NODP
  • 1.2 2008 Mid-Term Evaluation
• 2.0 Methodology/approach
  • 2.1 Evaluation Issues
  • 2.2 Evaluation Matrix
  • 2.3 Limitations of Methodology
• 3.0 Findings & analysis
  • 3.1 Program Relevance
  • 3.2 Design and Delivery
  • 3.3 Lessons Learned
• 4.0 Conclusion
  • 4.1 Program Relevance
  • 4.2 Design and Delivery
  • 4.3 Lessons Learned
• Management Response or PDF version (10 KB, 2 pages)

Annexes (Separate document)

(Note: Annexes are available via an Access to Information)

• Annex A: List of Steering Committee Members
• Annex B: Program Related Documents Reviewed
• Annex C: Literature Review
• Annex D: Key Informant Interview Guides
• Annex E: Observations of Performance Measurement System

This publication is available upon request in accessible formats.

Contact:
Multimedia Services Section
Communications and Marketing Branch

Telephone: 613-948-1554
Fax: 613-947-7155
Email: ic.cmb-creative.ic@canada.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Industry Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Industry Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Industry Canada.

For permission to reproduce the information in this publication for commercial redistribution, please email: copyright.droitdauteur@pwgsc.gc.ca

Aussi offert en français sous le titre de l'évaluation de mi-année du Programme de développement de l'Est de l'Ontario.

Executive Summary

Introduction

Industry Canada developed terms of reference for a mid-term evaluation of the Northern Ontario Development Program (NODP) of the Federal Economic Development Initiative in Northern Ontario (FedNor). The evaluation was managed by the Audit and Evaluation Branch, Industry Canada and conducted by TNS Canadian Facts. The evaluation covers three issues—program relevance; design and delivery; and lessons learned—with an emphasis on performance measurement and reporting. Evaluation methods include an extensive review of program documentation and past evaluations, a more limited literature review, a performance measurement system review, and nineteen key informant interviews. Evidence across these various sources were compared and contrasted by evaluation issue before an overall assessment was undertaken.

The following report presents the findings from the mid-term evaluation. This section provides a brief profile of the NODP, summarizes the findings of the mid-term evaluation, and presents recommendations for consideration by program management.

Northern Ontario Development Program

FedNor promotes economic growth, diversification, job creation and sustainable self-reliant communities in Northern Ontario through a range of initiatives aimed at improving small business access to capital, information, and markets. One such initiative, NODP, provides support primarily to the not-for-profit sector. Assistance to the private sector (approximately 10 percent) is limited to small- and medium-sized enterprises (SMEs) with fewer than 500 employees.

NODP targets six areas for support: Community Economic Development (CED); Trade and Tourism; Innovation; Information and Communications Technology (ICT); Business Financing Support; and Human Capital.

Findings

Program Relevance

Program relevance looks at the possibility of overlap/duplication with other government programs and what, if anything, FedNor is doing in this regard.

We found that, although NODP and other programs overlap, there are mechanisms in place to ensure collaboration occurs and to avoid duplication across funding sources. FedNor typically acts as a lead agency to co-ordinate the involvement of the proponent and other funding organizations to maximize the value from the partnerships and ensure all involved are more likely to achieve the objectives set for their contribution. FedNor has mechanisms in place to ensure no excessive support and no unaccounted funding is provided.

Because of the very real possibility of duplication in the environment where the NODP operates, coordination is key to ensuring taxpayers' resources are not being wasted. FedNor now plays a useful role as co-ordinator and facilitator for Northern Ontario development. FedNor should continue its leadership role to coordinate activities to ensure that duplication does not occur in support of projects and that multiple supports are used to achieve desired outcomes for Northern Ontario.

Design and Delivery

Three design and delivery issues were investigated: the extent to which FedNor has addressed monitoring, data capture, performance reporting and previous related issues identified in past evaluation studies; the appropriateness of the current performance measurement system; and the extent to which data quality is being assured.

Based on this evaluation we found that FedNor has addressed monitoring, data capture, performance reporting, and previous related issues identified in past evaluation studies. FedNor has made significant changes to their performance measurement system in keeping with the issues raised in earlier evaluations.

The program now collects, captures, and safeguards performance measurement information in keeping with the program's RMAF requirements. Collected indicators focus on immediate outcomes from the program's logic model as called for in the RMAF. Outcomes in the program's logic model appear consistent with the nature of the program based on our assessment. Tracking outcomes as part of the performance measurement system is appropriate.

Other data requirements of the RMAF involve data collection activities around administrative data (the Grants and Contributions Reporting System), secondary data (Statistics Canada), case studies and the data requirements for the final evaluation. The performance monitoring system could do more, by collecting additional intermediate outcome data, and reporting periodically on project outcomes.

The current performance measurement system provides a snapshot of project performance of a group of projects that have ended in the year. In the following year, another snapshot of another group of projects occurs and so on. When the final evaluation occurs, there will be an aggregation of results of different projects, but all at the same stage.

A gap in performance measurement data will exist related to the outcomes of some projects after their end date and before the final evaluation. To fill it, periodic follow-up of projects would be required. This might be conducted following a fixed schedule—end of project, first, and third year following the end—for all or a stratified random sample of those projects that could be reasonably expected to have longer term outcomes, and for a limited list of indicators (with more emphasis on intermediate outcomes) beyond the one-year anniversary of the project. Such a revised system would eliminate the current practice of placing some projects in a pending file for later follow-up. It might also eliminate the tension between immediate versus longer-term outcomes and early versus later indicators of performance.

Program officers helped to develop the performance measurement system and, in the process, have come to embrace it. This involvement and support contributes to data quality as do monthly meetings to iron-out data integrity/consistency issues. Mechanisms are in place to help assure data quality. Using the data to produce a report (to FedNor Management Committee—April 8, 2008) has provided an opportunity to identify and correct data problems.

This Mid-Term Evaluation has also identified a few data issues that have led to corrections by Policy, Planning and Coordination staff.

Edit checks of performance data occur, but are not formalized. More formal mechanisms would ensure data quality such as a comparison of intended and actual data for each project for consistency.

Involving FedNor staff in the development of the performance measurement indicators has been instrumental in achieving initial buy-in, future support related to efforts to gather and maintain consistency of performance data, and promotion of the indictors with project proponents.

FedNor should continue efforts to maintain the integrity and consistency of the performance measurement system in the future, especially as new staff is introduced. Monthly meetings and the Performance Management Working Group should be continued to identify and deal with performance measurement issues.

Lessons Learned

Factors that facilitated/impeded performance measurement and the management of these results and lessons learned through the process of dealing with them were reviewed. The experience suggests the following lessons.

Involving project officers in the development of the performance measurement system resulted in greater buy-in and improved consistency around performance measurement. Dedicating staff to the development task was seen as valuable. In the future, there is a concern related to on-going staffing needs as uses for the system ramp up. Also, staff turnover creates an on-going demand for training on the system requirements and data consistency issues leading to a further need for additional staff.

Although extensive discussions have occurred in developing the measures, there is a continuing debate about whether longer-term measures should be included. At issue is the appropriate balance between early evidence on immediate outcomes and later evidence related to intermediate or ultimate outcomes.

Project officers have available to them, and use, intended outcomes when they gather data related to actual outcomes for the FPE. An intended value, whether correct or incorrect, may have more fixity than it deserves in the data collection activity related to actual outcomes. Those asked to provide an actual number may choose the pre-existing intended number as it has been accepted before. This avoids the extra effort involved in generating a new number and might prevent questions justifying the reason for any change. Using intended data in the actual data collection process has the potential to bias the actual data.

If FedNor moves to periodic collection of project outcomes from proponents through a survey, it should not provide previous intended results at the time of the actual data collection, so as to encourage independence in the data collection events and avoid a potential source of bias.

Recommendations

NODP and the predecessor program NOD Fund has been in existence for a number of years. Consecutive evaluations had identified deficiencies in the performance measurement approach. Over the past few months there has been a significant amount of work and effort expended by FedNor to address these concerns in the area of performance measurement and monitoring.

Based on this work and the findings of the current mid-term evaluation, recommendations pertaining to strengthening the Performance Measurement System and Data Quality are clustered under two recommendations.

It is recommended that:

1. FedNor determine the benefit and cost of collecting performance measurement information on project outcomes beyond the project end date of projects. If appropriate, FedNor should extend its performance measurement system to capture these longer-term results.
2. FedNor strengthen mechanisms and formalize ongoing edit checks to ensure data quality, including:
   • consistent comparison of intended and actual outcome data for each project;
   • both, program officers and proponents report on final results independent of intended outcomes captured at the project launch stage;
   • analyze minimum, maximum and most frequent values for indicators;
   • create separate sub-indicators based on project grouping with unique outcome characteristics; and
   • address concerns, anomalies with data and indicators as required.

1.0 Background

1.1 Understanding the NODP

Program Description

The Federal Economic Development Initiative in Northern Ontario (FedNor) was established in 1987 to promote business development and economic diversification in Northern Ontario. FedNor is responsible for the delivery of the Northern Ontario Development Program (previously called the Northern Ontario Economic Development Fund),  ¹ the Community Futures Program,  ² the Eastern Ontario Development Program, and the Government of Canada Action Plan for Official Languages/Francophone Tele-Learning Pilot Project Initiative.

Initially, FedNor was introduced as a five-year program. In March 1992, it received its first program extension for a second five-year term. In March 1996, the Minister of Industry announced a re-engineered FedNor  ³ with a renewed three-year, $60 million mandate developed in consultation with various stakeholders, interest groups, regional organizations, and the Northern Ontario caucus. In the 1998 Budget Plan, the Minister announced continuing support for Northern Ontario communities by confirming that the financial commitment provided to FedNor would be maintained at $20 million per year after 1998–99. In June 1999, the Minister announced that the FedNor budget would be increased by a further $60 million over the next three years. As a follow-up to this announcement, new program priorities, arising from a series of 18 community dialogues held in 1998 were discussed by Northern Ontario MPs in September 1999. A Treasury Board submission for the additional 60 million was approved in October 1999 and the Secretary of State announced new program details shortly thereafter. ⁴

In 2005, the Government approved a new strategy for NODF, an integrated regional development strategy, designed to maximize the potential of regions to succeed in the new economy. The new strategy focuses on sub-regions, communities and special populations by building their assets and strengths; invests in innovation, human capital and modern infrastructure; and improves the coordination and relevance of policies and programs. The key to this new strategy was to lever the actions of others.  ⁵

As reported in the 2005 summative evaluation, program resources for grants and contributions and operations in FY 2002–03 totalled $60.4 million, compared to $53.4 million in 2003–04, $49.0 million in 2004–05, and $57.9 million in 2005–06. ⁶ Program resources for grants and contributions and operations for FY 2006–07 to 2009–10 are $59.1 million annually. ⁷ Included in the funding envelope for FYs 2005–06 to 2009–10 is approximately four (two FTEs) Policy, Planning Coordination staff to work jointly on both ongoing performance measurement and evaluation. ^{8  9}

Program Mission

FedNor's mission for Northern Ontario is to "promote economic growth, diversification, job creation and sustainable self-reliant communities in Northern Ontario through a range of initiatives aimed at improving small business access to capital, information, and markets." ^{10  11}

NODP promotes economic growth over a large and diverse geographic area that stretches from the Muskoka Lakes to James Bay and from the Manitoba border to western Quebec. ¹² While the program's designated area is "Northern Ontario," in instances where applicants outside of Northern Ontario can demonstrate a "clear and significant benefit to the Northern Ontario economy," the request may be considered for assistance on an exceptional basis. Most support provided (about 90 percent) is to the not-for-profit sector with private sector assistance limited to small- and medium-sized enterprises (SMEs) with fewer than 500 employees. ^{13  14}

To achieve its mission of accelerating Northern Ontario's movement to a knowledge-based economy and success in building more globally competitive businesses, NODP has targeted six areas for support in the RMAF dated January 29, 2007: Community Economic Development (CED); ¹⁵ Trade and Tourism; ¹⁶ Innovation; ¹⁷ Information and Communications Technology (ICT); ¹⁸ Business Financing Support; ¹⁹ and Human Capital. ²⁰

Overall responsibility for the achievement of FedNor results is vested with the FedNor Director General, assisted by the Management Team, and reporting to the Assistant Deputy Minister, Regional Operations Sector of Industry Canada.

Program Clients

Key clients, under the current NODP program, include commercial operations (particularly SMEs that are unlikely to obtain financing through a Loan/Investment Fund because the risk is too high); non-profit organizations engaged in research and development; Aboriginal capital corporations, Aboriginal non-profit organizations, Aboriginal-controlled financial organizations and similar groups able to increase the economic activity of Aboriginal people; First Nations and legal entities including incorporated not-for-profit organizations such a municipalities and municipal organizations, community development organizations, post-secondary institutions, hospitals and regional health care centres and other not-for-profit consortia of partnerships, sole proprietorships, corporations or association; businesses, including corporations, partnerships, cooperatives, proprietorships, trusts or consortia; and Community-Futures organizations. ^{21  22}

Program Delivery

i) Delivery model

To better serve its clients, and allow FedNor officers to stay in close contact with the community, NODP has adopted a community-based approach to development that focuses on local capacity-building initiatives targeted at strategic priorities and key economic sectors. It dedicates over 80 percent of its resources to supporting a decentralized field network. FedNor has three main offices in Sudbury (head office), Sault Ste. Marie and Thunder Bay, and satellite offices in several other geographically strategic locations (Timmins, North Bay, Parry Sound, and Kenora). ²³

As described in the 2001 RMAF, the revised organizational structure was implemented to provide for continuity of relationships at a community level. Clients served in a geographical area would be able to readily identify with specific FedNor officers as their point of contact. In addition, the structure would allow program officers to develop a stronger sense of client needs and opportunities, as well as identify strengths, and weaknesses, not only from an economic development perspective, but in terms of a client's capacity to manage projects, funds, and submit properly completed claims. ²⁴

Each of the three main offices has been allocated Monitoring and Payments Officers; and included in the restructuring of the service delivery model in 2000 was the addition of a Policy and Planning function with responsibility which includes program and policy development, functional guidance to program officers, and program evaluation. ²⁵

Program Support Clients of the NODP program receive two types of support from program officers: ²⁶

• Advice—The advice provided by the FedNor field officers takes a variety of forms, such as advice on project and applicant eligibility; assistance brokering partnerships, guidance in preparing proposals and helping proponents understand the Terms and Conditions governing project and applicant eligibility; the provision of economic data; marketing/promotional events coordination; and claims processing and project approval processes.
• Funding—Funding includes contributions in support of a wide range of community and regional development projects. Projects usually involve leveraged funding from other external sources, including community and regional partners, other levels of government, the private sector and non-governmental organizations.

Application Process and Procedure to Monitor Projects

As part of its annual business planning process, FedNor Management use input from delivery staff, from the Sectoral Strategies Unit (SSU) and from the Policy Planning and Coordination (PPC) Unit to establish funding priorities for NODP's six business units. These priorities are communicated to delivery staff who then share these priorities with clients interested in receiving funding support.

Typically, clients collaborate with FedNor staff on the development of proposals within the context of FedNor priorities, although, occasionally, unsolicited proposals are received. This collaborative approach allows FedNor to manage the NODP budget on an ongoing basis, while also managing client expectations, and ensures maximum flexibility in responding to community and client needs. Project ideas are introduced and circulated to regional colleagues with the NODP New Proposal form. This form includes the immediate outcome indicators required for measuring outcomes identified in the performance measurement matrix in the 2007 Results-Based Management and Accountability Framework (RMAF) for the NODP.

Once a funding proposal has been received, the FedNor program delivery officer prepares a detailed project assessment and summary of the proposal on a Project Summary (PSF) form. The form documents the objectives and merits of the project, as well as the rationale for approving the project. The assessment serves as a point of comparison against which the officer can assess and monitor the progress of the project, and demonstrates that FedNor has carried out due diligence in reviewing and approving proposals. The officer completes the PSF as part of the application review process/project work-up phase, and provides his/her recommendation via this and accompanying documents to the appropriate delegated authority for approval or rejection and issuing of a Letter of Offer (LOF) or notice of rejection.

Prior to submission to the appropriate delegated authority, proposals are peer reviewed. The draft PSF is circulated by e-mail to all Program Delivery Officers throughout Northern Ontario for general review and feedback. The final version of the PSF, which should address comments, concerns, and issues raised in this initial review is also vetted through peer review for purposes of quality assurance. A review is also conducted by an assigned Monitoring and Payment Officer with a view to addressing possible claims-related issues, identifying possible risks related to project delivery and claims submission, and agreeing to a risk-based monitoring plan.

After the Project Summary and New Proposal forms have been completed, they are moved from "Under Development" to "Sent for Signing" and then to either "Project Approved" or "Project Rejected-Withdrawn." Each sub-directory has its own set of user rights to grant or limit access. Once in the "Projects Approved" or "Projects Rejected-Withdrawn" directories, edit access is limited to a very small number of users. Reader access is provided to all program delivery staff for reference purposes only.

Recipients are required to submit regular Progress Reports and a Final Project Summary Report. The letter of offer between NODP and the recipient specifies that the recipient must supply, on request, such data and results, as the government deems necessary for purposes of monitoring the Contribution and for evaluation purposes. The data, which includes immediate outcome indicators required for measuring outcomes identified in NODP RMAF, are to be provided in a timely fashion.

The Project Delivery Officer is responsible for administering the Final Project Evaluation (FPE) that collects information on final immediate outcomes of the project. The intended project outcomes, as entered in the PSF, are also entered into the FPE as "intended results." At this point, officers must transcribe the intended project outcomes from the PSF (a Word Perfect document) into the FPE (a PDF document) because the two document types (WP and PDF) are not compatible. FedNor Policy, Planning and Coordination (PPC) is responsible for administering the FPE database (MS Access), maintaining and developing all forms related to NODP (Letters of Offer, New Proposal, Project Summary and Final Project Evaluation forms).

All original documents (hard copy), including the New Proposal, Project Summary and Final Project Evaluation forms, are placed in a master file. Managerial approval is now required on a completed FPE prior to submitting the FPE for storage in the FPE database as a mechanism to ensure data integrity. Managerial approval of the completed FPE is a new requirement as of January 28, 2008. Since this change had just been implemented before data collection for this evaluation, no assessment of the effectiveness of this new procedure is possible.

1.2 2008 Mid-Term Evaluation

Industry Canada has developed terms of reference for a mid-term evaluation of the Northern Ontario Development Program (NODP). The terms of reference specify both the parameters of the study and the methodologies that are to be employed in capturing data for the study (i.e., a review of files and documentation; key informant interviews; and a literature review) and the elements of the NODP to be addressed. The evaluation is limited in scope to the issues and research questions of interest to senior management as specified in Annex A which accompanied the terms of reference, with performance measurement, monitoring of performance outcome data, and reporting the primary focus of the mid-term evaluation.

2.0 Methodology/Approach

2.1 Evaluation Issues

The mid-term evaluation was to address three key areas identified in the RFPs Annex A: relevance, design and delivery, and lessons learned; specifically:

Program Relevance:

• Are there overlaps/duplication with other government programs? (EQ1.2)
• Is FedNor collaborating with other departments and governments to reduce overlaps? (EQ1.2)

Design and Delivery:

• What Factors have Facilitated/Impended the FedNor Strategy for NODP Performance Measurement and Management of these Results?
• What Lessons have been Learned from the FedNor Strategy for NODP Performance Measure and Management of These Results?

Lessons Learned:

• What factors have facilitated or impeded the implementation of the program?
• What lessons have been learned that can be applied to future NODP programming? (5.1)
• What lessons have been learned that can be applied to performance measurement, performance monitoring and data collection? (5.2)

Following review and approval of the Methodology Report, FedNor Management questioned the relevance of the Lessons Learned questions specified in Annex A and agreed to in the Methodology Report. FedNor Management held discussions with the Audit and Evaluation Branch (AEB) and it was agreed that the "lessons learned" questions specified in the Annex would be replaced with questions which focussed more specifically on the issue of performance measurement and the management of these results. The replacement questions are as follows:

• What factors have facilitated/impeded the FedNor strategy for NODP performance measurement and management of these results?
• What lessons have been learned from the FedNor strategy for NODP performance measurement and management of these results?

The interview guides provided with the Methodology Report were modified accordingly. The revised interview guides are appended to this report.

Given the short timelines of the evaluation, individual and group interviews were limited to key informants, mainly program people and academic experts. Savings through avoided travel were transferred to additional resources devoted to the literature review. Surveys of clients were not conducted at this mid-term evaluation stage due to the nature of the issues covered. Client surveys will be more appropriately part of a summative evaluation where the focus will be on client impacts.

2.2 Evaluation Matrix

The methodology involves the following components:

• Review of program-related documents.
• Review of literature.
• Review of performance monitoring and reporting.
• Interviews with 20 key informants.

The first task under the contract was to develop a detailed methodology report for the evaluation. Exhibit I-1 summarizes the research plan for the evaluation, including the methods and sources that were used to answer the evaluation questions. The exhibit has been modified from that provided in the methodology report to reflect the agreed-upon change in focus with respect to the lessons learned questions.

Review of Program-Related Documents

We reviewed key program documents for the evaluation, paying particular attention to aspects related to the current evaluation; namely, ensuring the availability of data for the 2010–11 summative evaluation, identifying any data gaps, and offering suggested strategies for redressing these gaps. The documents reviewed included the Results-based Management and Accountability Framework, previous audits and evaluations of the predecessor program and other relevant documents. For the list of documents reviewed, see Appendix B—Program Related Documents Reviewed.

The documents assisted in the development of the methodology approach and in formulating the questions for the key informant interviews. The documents have also been used in this report to provide a context for the mid-term findings

Review of Literature

We conducted a very limited literature review for the mid-term evaluation, due to resource constraints and the fact that significant resources had been invested in the literature review for the 2005 summative evaluation. A more comprehensive review will be undertaken for the 2010–11 summative evaluation.

FedNor management identified a core group of documents/reports/studies for the review, including work published by the OECD. The documents/reports/studies include the following:

• The New Rural Paradigm, OECD (2006).
• Territorial Review: Canada, OECD (2002).
• Mobilizing Science and Technology to Canada's Advantage, Government of Canada, (2007).
• Understanding Freefall: The Challenge of the Rural Poor, Senate Committee on Agriculture and Forestry (2006).
• Youth Intern Program: Comparative Report, FedNor, (2006).
• A Review of Rural and Regional Development Policies and Programs, Canadian Policy Research Networks, (2008).

The literature review can be found in Appendix B.

The primary focus of the mid-term literature review was to review literature related to community economic development, regional development, and the science and technology strategy. Based on this review, we conducted two interviews with academic experts to solicit their views on the approach adopted by FedNor for the NODP. The interview guide used in these interviews is appended. (See Appendix D)

Review of Performance Monitoring and Reporting

The review of performance monitoring and reporting focussed on three evaluation questions:

• Are appropriate performance indicators being collected, captured, safeguarded and used to measure the outcomes of NODP projects?
• To what extent is data quality being assured?
• To what extent has FedNor addressed the data capture, monitoring of performance data and reporting issues identified in past evaluation studies?

The Performance Measurement Matrix in the 2007 RMAF provided the starting point for our review and assessment of performance measurement, monitoring of performance outcome data, and reporting. ²⁷ The first task was to assess whether the information was being collected as specified in the Matrix. We then addressed issues of completeness (number of missing observations) and accuracy (implausible or questionable values) in the NODP Final Project Evaluation Data Capture spreadsheet provided to us and covering the period July 1, 2006 to March 31, 2007. Beyond simple compliance to the RMAF requirements, we looked at how appropriate the measures are to their intended use. Both the review of the database and key informant interviews assessed:

• Are indicators at a level appropriate to measure the expected program impact?
• Is the timing of the data collection related to indicators appropriate to capture expected program impacts? Should there be different intervals for different potential program impacts?
• What uses for the monitoring data were intended? Have data been used for these or other purposes? Why or why not?

The findings from this review are reported in the relevant sections of Chapter II.

Key Informant Interviews

We conducted key informant interviews with six groups of key informants as follows, talking with a total of 19 respondents ²⁸:

• Program Delivery Managers (PPM) (n=2)
• Payment & Monitoring Officers (n=4)
• Program Officers (n=3) ²⁹ Originally 4 had been scheduled.
• Policy, Planning & Coordination (PPC), including Director (n=5)
• Sectoral Strategies Unit (SSU), including Manager (n=3)
• Academic Experts (n=2)

FedNor officials and steering committee members assisted the research team in identifying key informants for each respondent group, and in providing contact information for candidate respondents. FedNor management was also involved in structuring the joint interviews to maximize participation. Candidate respondents were very accommodating, making themselves available on short notice to participate in the interviews. Without their assistance we would not have been able to meet the timelines set for the evaluation. Interviews were conducted over a two-week period. We did not interview clients. Clients will be consulted as part of the summative evaluation, which is scheduled for 2010–11.

All of the interviews were conducted by telephone and, with the exception of the interviews with the Program Delivery Officers and Payment and Monitoring Officers, which were longer. Interviews ranged in length from 30 minutes to 2 hours. On average, interviews were about 75 minutes in length.

In collaboration with AEB and FedNor staff, we developed separate interview guides for the interviews, with some groups sharing an interview guide. Separate interview guides were developed to ensure that FedNor respondents were only asked questions relevant to their involvement in NODP. Respondents were given permission to opt out of any question they did not feel was relevant for them. While we were seeking similar types of information from the various groups (i.e., the appropriateness of the indicators, quality of the data, etc.), the officers delivering the program were asked more specific questions than the others. For example, while FedNor management was asked general question about the extent to which FedNor had addressed the data capture, monitoring of performance outcome data, and reporting issues identified in past evaluations, the officers were asked about specific changes to data collection forms/processes/procedures to improve the quality of the data that is being collected. Both were asked to provide an assessment of the changes that had been implemented and what, if anything, further had to be done. The interview guides for the key informant interviews are appended. (See Appendix D)

2.3 Limits of Methodology

Review of Program-Related Documents

We conducted an extensive review of the key program documents provided for the mid-term evaluation, including the documentation related to performance measurement. These documents have been critical in developing the methodology approach for the mid-term evaluation and in providing a context for the findings. Review of Literature

The primary focus of the literature review is on community economic development, regional development, and the science and technology strategy. The findings from the literature review are intended to inform the assessment of the factors that have facilitated or impeded the implementation of NODP. The review also provides an overview of the lessons that can be applied to future NODP programming and reviews the appropriateness of the performance measures, monitoring, and data collection used in NODP. The literature review is limited to core documents, reports and studies that have been published since the earlier reviews of NODP.

Review of Performance Monitoring and Reporting

We reviewed data from the NODP Final Project Evaluation Data Capture spreadsheet covering the period July 1, 2006 to March 31, 2007. This provided information on a limited number (n=273) of projects from all six program areas supported by the NODP. The data coverage period was not extensive due to the relative newness of the RMAF and associated performance indicators. However, the period was likely sufficient for initial problems to be identified and rectified so that the data set was at a reasonable state of maturity for the review. The review looked for coverage with respect to the RMAF requirements and data anomalies (values that were missing, over-used, or outliers—i.e., values beyond the expected range.)

Key Informant Interviews

The interviews were conducted within a very short time frame, with the result that we were able to conduct only a small number of interviews, and few of the interviews were conducted on a one-to-one basis. Interviews with two or three people were the norm. In the case of the Program Delivery Officers and Payment and Monitoring Officers, the groups were structured to include four people (two from the northwest and two from the northeast). We recruited officers at different stages in their career, including relatively new officers as well as people who had been with the program for a number of years.

To ensure that respondents would feel comfortable expressing their views in a small group setting, we conducted separate interviews with staff and managers. There was one exception where the people involved opted to be interviewed together.

Given the small number of interviews conducted, particularly with the front-line staff (three out of 40 program delivery officers), the findings from the key informant interviews should be considered to be indicative but not definitive. They are the views of the few respondents consulted. They may not be the views of the group as a whole. We attempted, in the interviews, to establish whether other officers shared the views expressed on issues, but this is not a totally satisfactory way of querying the views of the population of officers. Other respondents may have offered different insights on some of the questions had they been interviewed. This should be kept in mind when reviewing the findings.

Clients were not part of the mid-term evaluation as they were expected to have little to contribute to the evaluation issues of this study. They are expected to have a more significant role in the summative evaluation given its likely focus on client impacts.

For training and development purposes an AEB staff member listened in to four of the seven interviews; covering 12 of the 19 respondents interviewed. Although it is impossible to judge the impact of having a third party observer listening to the interviews given the form of the research conducted by this study, any resulting bias is likely positive toward the program.

^*1 The Evaluation Matrix is based on Annex A, NODP Formative Evaluation Issues Matrix provided with the Terms of Reference for the Mid-Term Evaluation of NODP. (Return to Text)

^*2 Analysis Plan. (Return to Text)

^*3 Reference to the comparable evaluation question in the Performance Measurement Matrix in the RMAF dated January 29, 2007. (Return to Text)

^*4 Revised following the submission of the methodology report as agreed-to the NODP management and AEB. (Return to Text)

3.0 Research Findings

Program Relevance

The terms of reference asks the 2008 mid-term evaluation to address two questions related to Relevance:

• Are there overlaps/duplications with other government programs? (EQ1.2)
• Is FedNor collaborating with other departments and governments to reduce overlaps? (EQ1.2)

3.1.1 Are there Overlaps/Duplication with Other Government Programs?

Conclusion

Feedback provided during interviews with FedNor employees suggests that clients are encouraged to seek multiple funding sources. Consequently a project may be funded by both NODP and other program funds. Funding agencies determine the aspects of the project that they will support through their own assessment criteria. NODP respectively does the same. NODP and other programs overlap in the sense that they may provide funding to the same project. There are however, mechanisms in place to ensure collaboration occurs and to avoid duplication across funding sources.

Context

We define program overlap as "multiple coverage" for the same project or activity. Overlap can be a problem but it need not be a problem. Only if overlap is also accompanied by duplication, will society's resources be used inappropriately. For example, a worthwhile project might not proceed without $10,000 in government support. The project meets the eligibility requirements of two overlapping programs. Society's resources are not wasted if one, or the other, or both programs in some combination provide the $10,000. However, if the programs jointly provide more than $10,000, duplication results and the amount of the contribution exceeding $10,000 is a waste of resources. As the example illustrates coordination is required to avoid duplication in situations of overlap. This is more fully explored through the next evaluation question.

Findings

The NODP is not the only program that can be used to support the Northern Ontario economy. Other significant program and funders include:

• Northern Ontario Heritage Fund (through Ministry of Northern Development and Mines).
• Ontario Trillium Foundation (an agency of the Government of Ontario).
• Rural Economic Development (RED) Program (through Ontario Ministry of Agriculture, Food and Rural Affairs).
• Ontario Job Creation Partnerships (through Ontario Ministry of Training, Colleges and Universities).
• Various programs of Human Resources and Skills Development Canada, National Research Council (particularly Industry Research Assistance Program), Indian and Northern Affairs Canada (particularly supports to Aboriginal businesses through the Aboriginal Business Canada programs), and contributions through municipalities.

This suggests there are multiple sources potentially providing coverage to a project that is eligible for support by NODP. As a result, overlap exists.

3.1.2 Is FedNor Collaborating with Other Departments and Governments to Reduce Overlaps?

Conclusion

FedNor is collaborating with other departments an cases where there is more than one government funding source, a stacking analysis is done to ensure they abide by the stacking guidelines. FedNor gains an intimate knowledge of the funding arrangement through close involvement with other funding agencies and the client at the onset of the project. The letter of offer also outlines to the client that they must declare all funding sources. Closely monitoring the project throughout its life cycle also serves to ensure the client abides by the funding agreement and FedNor officers can challenge payment if anomalies occur.

Context

Previous evaluations concluded that NODF funding complements, rather than duplicates, the contribution of other government programs/funding sources.

Findings

NODP is a broad program that gets involved in all kinds of activities related to regional development. Respondents indicated that there are many programs recipients can access. Some of these programs overlap. However, the programs have different mandates, eligibility criteria, and funding restrictions, with the result that even where overlapping programs are involved, the funding partners support different components of the project or, if they fund similar components, they abide by the stacking guidelines. Proponents are expected to seek (multiple) partners for their projects. FedNor officers are involved in negotiating these arrangements with the client and the other funding agencies. In so doing, FedNor may act to improve the effectiveness of partnerships in keeping with the OECD approach in The New Rural Paradigm. In the view of the OECD report, it is necessary to consider needed factors for effective partnerships and to be aware of potential obstacles to effective partnering.

Clients must certify all sources of funding, and must notify NODP of any changes in funding. Projects are closely monitored to ensure that the funding arrangements that were agreed to at the start of the project are not breeched. If there are anomalies in invoices, payment can be deferred until the issue is resolved.

The key informants for the 2008 mid-term evaluation were unanimous in their agreement that safeguards are in place to reduce overlaps/duplication, and that no further improvements are required.

Review of Documents

Previous evaluations addressed the issue of overlap, less directly (with respect to other departments and governments) in the 2002 Formative Evaluation, and more directly in the 2005 Summative Evaluation. Below is a summary of findings from these evaluations to provide a context for the 2008 findings.

The 2002 Formative Evaluation ³⁰ found "almost no evidence" that FedNor programs and services were competing with those of private or non-governmental organizations; notwithstanding the fact that FedNor was working with banks and other lending institutions to make high-risk loans available to firms. FedNor was considered by most of the respondents interviewed for the 2002 evaluation to be unique within Northern Ontario," including in comparison to the Northern Ontario Heritage Fund Corporation (NOHFC). ^31 32 Outside the Northern Ontario region, FedNor was found to be similar in mandate, program design, and delivery strategy, to the three regional development agencies that have a similar role to FedNor in other regions of Canada (the Atlantic Canada Opportunities Agency (ACOA), the Canadian Economic Development for Quebec (CED-Q), and the Western Economic Diversification (WD)). Staff talked about the complementarity between FedNor and the NOHFC, and the role of FedNor in helping communities to access funds from other sources.

The 2005 Summative Evaluation ³³ asked respondents about the availability of other programs or services from the federal or provincial government that were comparable to NODF, and available in northern Ontario. Less than half of the respondents interviewed were able to identify comparable programs and services. Those who were able to identify other programs and services mentioned: Northern Ontario Heritage Fund, HRSDC, the Trillium Foundation, and Northern Development and Mines. When asked about the importance of FedNor funding, close to half indicated that their project would not have occurred without FedNor involvement. In cases where the project would have proceeded without FedNor funding, respondents indicated that the absence of FedNor funding would have had an impact on their project by either delaying the start of the project; taking more time to complete the project; reducing the scope of the project; or affecting the quality of the project. The 2005 evaluation found that FedNor was cooperating extensively with other organizations; that NODF funding complemented, rather than duplicated, the contribution of other sources of funding; and that there was no other alternative to NODF that would have produced the same results at lower costs.

Key Informant Interviews for the 2008 Mid-term Evaluation

We asked the Program Delivery Managers; Policy, Planning & Coordination respondents, Sectoral Strategies Unit respondents (PDM/PPC/SSU) and Program Delivery Officers (PDOs) and academics a series of questions that looked at the issue of overlap, and what has been done by FedNor to reduce overlap with other departments and governments. ³⁴ Respondents who were not involved with this aspect of the program opted out of answering these questions.

Other programs/funding sources available

FedNor respondents indicated that there are multiple government programs/sources of funding available to NODP recipients, in addition to that provided by NODP, and that some of these programs overlap. NODP is a broad program that gets involved in all kinds of activities related to regional development. The critical factor is not that a program overlaps with that of another, but that the programs have different mandates, eligibility criteria, and different funding restrictions, with the result that even where overlapping programs are involved in the same project, their involvement is confined to different components of the project or, if they fund similar components, they abide by the stacking guidelines. The word that came up most often in discussions was that of "complementarity." NODP funding complements that from other sources; there is no duplication in the view of key informants. FedNor respondents were then asked about what was being done to reduce overlaps/duplications (where "duplicate" means the same cost is claimed to two parties); and whether these safeguards were adequate, or whether improvements were needed.

Procedure to reduce overlaps/duplication

The safeguards begin at the program level when new initiatives or programs are being considered. An environmental scan is conducted to establish whether there are similar programs to that being considered, who is providing the program, and the program's focus or objective. The purpose of the scan is to ensure that new programming does not duplicate support that is already available; specifically, to identify gaps in programming that the new initiative might address.

The safeguards continue at the funding level. Programs only fund certain types of costs, and have limits on how much any one program can contribute. Programs are not allowed to fund 100 percent of the cost of a project. This includes the NODP, notwithstanding its predominant not-for-profit orientation. There are different thresholds for funding, depending on the project; for example, R&D projects may qualify for 75 percent of the cost; other projects may qualify for 50 percent. If they fund similar components, FedNor officers do a stacking analysis of funding from the various government programs (federal/provincial/local) and try not to exceed 90 percent.³⁵ Depending on the project, NODP may fund from one-third to one-half of the cost of a project. The rest of the funding comes from the proponent and other funding partners. Building partnerships around funding is an accepted practice. Officers communicate with each other across programs/funding sources; they exchange information on who is going to fund what; and are often in discussion with other agencies before the project application has even been submitted. Proponents are expected to seek other sources of funding for their projects. ³⁶ There are also restrictions with some programs (such a youth internships) that preclude funding from both the federal and provincial governments.

The agreed-to funding arrangements are specified in the Letter of Offer. Proponents must "certify" all sources of funding as part of the application process, and must notify NODP if any other sources of funding become available to the project, to address any potential for duplication after the offer has been issued. There is a clause in the Letter of Offer to this effect. The discussions and schedules that can be put into the Letter of Offer can be very specific. In addition, there is a news tracking service that could identify any changes in funding arrangements in the event the client failed to advise the program.

If NODP cannot fund a particular project, the officer will refer the proponent to other programs/funding sources, and may work with the other programs/funding sources to obtain alternate funding for the project. The proponent's ability to successfully access alternate sources of funding will depend on the project/proponent meeting the requirements/criteria of the other funders. Since NODP is not an entitlement program, NODP has declined to participate in some projects that were technically eligible (eligible proponent, eligible specific costs). The reasons for such turndowns vary in specifics, but generally revolve around questions related to the economic impact of the project, identified through the due diligence and project evaluation process. In the absence of NODP funding, a project could be postponed or scaled back or might not proceed. In some instances a proponent might rethink their plan and decide not to proceed in light of NODP feedback.

Projects are closely monitored by the PDOs and Monitoring & Payment Officers (M&POs) through the progress reports to ensure that they are on target. This is directly tied to activities and the payment of invoices. If there are anomalies in invoices, payment may be deferred until the issue has been discussed with the client and/or invoices may be reviewed on a line-by-line basis to ensure that they comply with the agreement. Recipients of NODP funding are often repeat clients. They are familiar with the process with no desire to jeopardize their relationship with NODP. The Sectoral Strategies Unit (SSU) is also available to provide additional information/insight to the program officers as required.

What would happen without NODP funding?

Given the community-based approach NODP has adopted and staff's awareness of issues facing Northern Ontario communities, NODP is often the first to respond to needs. Its presence and front-end work reportedly sets the stage for the province and/or private sector to follow with their investments and, in this sense, NODP can be seen as a catalyst for change. As reported in the 2005 summative evaluation and the comments of respondents, NODP's role is "quite unique." The impact of the support NODP provides for projects involved in economic development (funding strategic planning or capacity building or demonstrating the feasibility of a capital project that might later be funded by, for example, the Heritage Fund) can be "very significant." NODP support for building the business case for telecom investment in rural areas, for example, has had "immense implications" for things like education and health care, doubly so in isolated Aboriginal communities where, in at least one instance in the northwest, the entire community was served by one pay phone. Without NODP support for the up-front planning, the infrastructure investments (supported by other programs/funding sources) would not have been made. As one FedNor respondent described the situation, finding a replacement for NODP funding might be possible, depending on the flexibility of another program's criteria and the priority that a particular project might have for them, but it would take work and is not usual.

Adequacy of safeguards

FedNor respondents were unanimous in their agreement that there is collaboration with other departments and governments to reduce overlap. The safeguards that have been developed (working closely with other funding agencies, specifying the funding arrangements in the Letter of Offer, challenging anomalies in invoices, etc.) are perceived to be adequate. Nothing further is required in the view of key informants.

Literature Review

In March 1996, the Minister of Industry announced a re-engineered FedNor, developed in consultation with various stakeholders, interest groups, regional organizations, and the Northern Ontario caucus. At this time, and consistent with new approaches to regional development introduced in other economically depressed regions of Canada, FedNor initiated an alternative service delivery method. It moved away from the direct assistance to business mechanism first employed by the agency to a fully repayable, commercially based lending approach backed by loan loss reserves delivered by third parties. FedNor also shifted its programming emphasis towards the needs of a knowledge-based economy.

As a result of this new program approach, FedNor focused on the following five strategic areas:

• Trade.
• Innovation.
• Investment.
• Connectedness.
• Community Partnerships.

These five priority areas were established to specifically address key issues facing people who lived and worked in Northern Ontario. New program components were also introduced in 1999 to facilitate a shift to a knowledge-based economy with an emphasis on connectedness, innovation, and community partnerships. They included: youth internships, community economic infrastructure, telecommunications infrastructure, applied research and development, community recovery, innovation capacity building, and information and communications technology (ICT) applications.

In 2005, the Government approved a new strategy for NODP, an integrated regional development strategy, designed to maximize the potential of regions to succeed in the new economy. The new strategy focuses on sub-regions, communities and special populations by building their assets and strengths; invests in innovation, human capital and modern infrastructure; and improves the coordination and relevance of policies and programs. The key to this new strategy was to lever the actions of others.

3.2 Design and Delivery

The terms of reference asks the 2008 mid-term evaluation to address three questions related to Design and Delivery:

• Is appropriate performance information being collected, captured, safeguarded and used? (EQ2.1)
• To what extent is data quality being assured? (EQ2.1)
• To what extent has FedNor addressed monitoring, data capture, performance reporting and previous related issues identified in past evaluation studies? (EQ2.4)

3.2.1 Is Appropriate Performance Information Being Collected, Captured, Safeguarded & Used?

Conclusion

The program now collects, captures, and safeguards performance measurement information in keeping with the program's RMAF. Collected indicators focus on immediate outcomes from the program's logic model as called for in the RMAF. As such the program is in compliance with the requirements set for it. Other data requirements of the RMAF involve data collection activities around administrative data (the Grants and Contributions Reporting System), secondary data (Statistics Canada), case studies, and the summative evaluation.

To date the collected indicators have been used to produce an annual report. Other uses anticipated in the future include renewing and/or making revisions to the program, new submissions to Treasury Board or Cabinet, strategic planning, and the development of their Business Plan.

Context

Earlier evaluations indicated that FedNor was in a relatively early stage in performance measurement management and reporting. At the time of the 2005 Summative Evaluation, there was no link between project objectives, as stated in the application, and the project outcomes report completed by the program officers, and there were no tools available to collect baseline data to measure the impact of the projects.

Findings

Considerable progress has been made in the interim. With the exception of the tourism indicators, all of the respondents agree that FedNor has identified the "right indicators." The process of soliciting staff input was ongoing at the time of the interviews; however, the focus was shifting from identifying indicators, something almost everyone agreed was pretty much finished, to clarifying and refining the understanding and use of the indicators. Officers are using the indicators to screen projects to ensure that projects meet program priorities, and they are using the indicators to help proponents define results. In short, they are "thinking in terms of outcome data." A number of limitations were identified with respect to the short term (immediate outcome) nature of the indicators which key informants felt, for the most part, could be addressed through data collection in a future summative evaluations focussed on longer-term outcomes.

Most FedNor respondents indicated that it is too early in the process to be using the data to make improvements to the program. However, projected future uses of the data include renewing and/or making revisions to the program, new submissions to Treasury Board or Cabinet, strategic planning, and the development of their Business Plan.

Review of Documents

The 2002 Formative Evaluation indicates that, at the time of the evaluation, FedNor was at a relatively early stage in the development of performance indicators and targets and, more generally, of a performance measurement management and reporting culture. In response to the findings, FedNor management developed a Results-Based Management and Accountability Framework (RMAF) which was approved by Treasury Board Secretariat in March 2005.

The 2005 Summative Evaluation reported that, at the time of this evaluation, FedNor had enhanced its capacity in the area of performance-based planning, monitoring, and reporting. It had invested more resources into performance monitoring, and undertaken special studies to review specific aspects of the program. Program officers were meeting their requirements in terms of assessing project risk and completing the project outcome score sheets; and there was evidence of a lot of analysis and reporting of the data. The report went on to say, that while FedNor management agreed that a lot of progress had been made, ³⁷ there were still issues that needed to be addressed. There was no link between the project objectives, as stated in the application, and the project outcomes report completed by the program officers; there were difficulties working with the Industry Canada departmental system for capturing and analyzing information on projects; at a project level, the tools were not available to collect baseline data to measure the impacts of the projects. ³⁸ The evaluation also notes that most FedNor staff seemed unaware of how project specific results were integrated or interpreted on a more macro level. Further, the evaluation indicated that it was unclear whether the information collected from the program outcome score sheets was valid and reliable, or whether the program officers understood how to complete the score sheets.

Key Informant Interviews for the 2008 Mid-term Evaluation

The PDM/PPC/SSU respondents and the PDOs and M&POs were asked about the appropriateness of the performance measures. With the exception of the PDOs and M&POs, respondents were asked to think about the performance measurement plan outlined in the 2007 RMAF, and assess the appropriateness of the performance indicators that are being collected, captured, safeguarded, and used to measure the outcomes of NODP projects. Since we did not expect the officers to be knowledgeable about the 2007 RMAF, they were asked about their level of confidence in the performance indicators that are being used to measure outcomes, as part of a set of questi ons examining the issue of data quality. As a supplementary question, they were asked if anything further had to be done to ensure the quality of the performance outcome data. ³⁹ We also asked respondents about the extent to which NODP has utilized the performance-based information that is being collected to report on outcomes and make improvements to the program, and to provide examples of data use and decisions that have been taken based on the data. ⁴⁰

The following sections discuss the main findings from the 2008 mid-term evaluation key informant interviews with respect to the appropriateness of the performance measures.

Process to develop the indicators

As part of the response to the 2005 summative evaluation, FedNor engaged in what all FedNor respondents describe as an extensive consultation process with staff. This process began in January 2006 at the program delivery level with the identification of activities, and has almost been completed through development of the performance measurement indicators identified in the 2007 RMAF (published in January 2007). ⁴¹ To assist with the process, FedNor hired the evaluators who conducted the 2005 summative evaluation of NODF. They were tasked with implementing their recommendations with respect to performance measurement. This included assisting with the development of the logic model. The actual consultation with staff was conducted internally by PPC, with training on how to conduct the consultations provided by the evaluators.

As part of the process, FedNor established a Performance Measurement Working Group (PMWG) to provide feedback and advice on program outcomes and appropriate indicators. The group continues as a standing committee. Members of the working group include a mix of stakeholders including representatives from program delivery, policy, and management.

The officers on the working group have expertise/experience in one (or more) of the six program components. The working group has recently been expanded to include representatives from Monitoring & Payment as well.

A number of other avenues are available to staff (in addition to the PMWG) to discuss issues related to the development, refinement, and/or clarification of the indicators, including regular staff meetings; one-on-one correspondence between the program officers and PPC; presentations; ⁴² and the most recent option, adding someone from PPC to the monthly meetings with field staff to talk about issues, and address questions and concerns related to the indicators.

This process of soliciting input was ongoing at the time of the interviews, although the focus is reported to be shifting from identifying the indicators (something almost all respondents indicated is pretty much finished) to clarifying and refining (including improving instructions and definitions) the use of the indicators in specific situations what they describe as "grey" areas. These monthly meetings are seen as a valuable contributor to ensuring uniform interpretation of indicators and integrity of the data captured.

Response to the process and indicators developed

The approach adopted to develop the indicators has had two major advantages:

• Built on the expertise of front-line staff. The indicators have evolved in collaboration with the officers who deliver the program. They are based on the expertise of those who could be expected to be the most knowledgeable about which indicators are most likely to best measure success in the six program areas. They are "reality based" indicators and draw their strength from the willingness of staff to collaborate in their development and refinement.
• Had a positive impact on staff buy-in. The approach adopted has facilitated acceptance and buy-in on the part of staff; and contributed to what is clearly a culture change around performance measurement. Respondents talked about a heightened acceptance of the importance of, and need for, the indicators. As one respondent phrased it, "There are less people who just want to deliver the program … do stuff … Now they are thinking in terms of outcome data." The indicators are structuring both the external discussions with clients and internal discussions with colleagues. Officers are using the indicators to screen projects to ensure they meet program priorities, and they are using the indicators to help proponents define results. The indicators are reported to be helping the officers to do their job. They can explain to proponents the decision to fund or reject the project, and they become the rationale to justify funding a project to others in FedNor. As one respondent phrased it. "Officers appreciate having a basis for recommending projects to their colleagues and dealing with the client."

FedNor respondents acknowledge that there were difficulties initially, and there was some resistance, mainly around whether or not they had identified the right indicators. As one respondent phrased it, "There are no perfect indicators or groups of indicators….But we are at a point now where everyone pretty much agrees that these are a 'reasonable collection of indicators." This view was shared by almost all of the respondents who were asked about the appropriateness of the indicators. As another phrased it, "We're so much further ahead in capturing the right data." However, a minority expressed strong reservations about the appropriateness of the indicators that have been developed. Tourism was singled out, although management suggests this sentiment extends beyond tourism indicators. The concern, as inferred from materials supplied by those expressing this view, is that the indicators should be measuring longer-term effectiveness and efficiency measures.

Clearly the current focus on immediate outcomes by the performance measures is in keeping with the program's RMAF. All other indicators are the responsibility of other data collection methods including those around the summative evaluation.

The mix of performance measures across immediate, intermediate and ultimate outcomes of the program may be something FedNor might want to consider before finalizing the discussion on the indicators.

Limitations of the indicators

A number of other observations/reservations were made over the course of the interviews:

• Indicators Capture Outcomes at a Point in Time. The Final Project Evaluation (FPE) form captures data at a point in time—the end of the project. There is some concern that outcomes may not be fully evident at that time, and that the "true value" of a project may not be captured in all instances. This appears mainly to be in instances where the outcomes will not have been realized when the FPE is completed. To address this concern, officers are allowed to put projects into a "618" status, deferring completion of the final evaluation form for up to a maximum of 12 months. It is not clear that this has totally resolved the issue, in that the extra 12-month period may not be sufficient in some cases. Most acknowledge, however, that this is part of the reality of what they do. The benefits of projects may not be evident until some time into the future. Most of these FedNor respondents believe it will be possible to capture these outcomes in the program evaluations which are conducted on a regular basis. A few talked about doing this in-house, conditional upon a reassessment of resources. There is concern that resources are being "pushed to the limit" and that, were FedNor to lengthen the monitoring and reporting term on projects, this would require a reassessment of resources and structure to better accommodate any increased monitoring requirements.
• Indicators are not Capturing the Indirect Impact of NODP-Funded Projects. FedNor respondents acknowledged that some PDOs are frustrated that the indicators are mainly limited to immediate outcomes. NODP may, for example, fund a feasibility study for a capital project. On completion of the feasibility study, the proponent will seek funding from other sources, such as the Northern Ontario Heritage Fund, to implement the project. Without NODP support for the up-front planning, the capital investment would not have been made. In infrastructure projects, for example, the impact on the community could be significant; however, these indirect and longer-term outcomes of the program are not being captured with the current tools. This is another place where key informants feel the summative evaluation could seek information from clients and/or conduct case studies to get a better handle on these outcomes.
• Recognize the Importance of Qualitative Data. Another issue raised by some FedNor respondents, including the academic experts, was the focus on quantitative data to the exclusion of qualitative data. While there is a narrative section where officers can provide commentary in the FPE, ⁴³ this does not appear to totally resolve the issue for these FedNor respondents. They recommend that the program not lose sight of the qualitative side: "It's important and we shouldn't forget about it." Some saw this "narrowness of measurement" as a challenge when evaluating some projects; in that some sectors are "easier to quantify than others." While the qualitative outcomes are being captured in the narrative, FedNor respondents are not sure how, or if, these data are being utilized.

Use of the performance-based information being collected

A first NODP Performance Measurement Report has recently been released based on existing outcome data.

Most of the FedNor respondents indicated that it is too early in the process to be using the data to make improvements to the program. As several FedNor respondents phrased it, "we're not there yet," but they believe that they will use the information when they are seeking to renew or make changes to the program; that it will be reflected in new submissions to Treasury Board or Cabinet; and that it will play a role in their strategic planning and the development of their Business Plan. There was a minority view, however, which cautioned against becoming too reliant on these data to the exclusion of everything else. While the data is perceived to have value, these FedNor respondents believe it is only one source and should be supplemented with other information. Recall that key informants stressed the value of qualitative data available from the FPE as well as the expected contribution from the summative evaluation.

A couple of additional suggestions were raised in the interviews related to other uses for the data that is being collected:

• Provide feedback to the community. FedNor identifies and communicates successes to the community. Some thought these data might add an additional dimension to the feedback that is already being provided to the community.
• Allow PDOs to query the database. A few FedNor respondents raised the possibility of allowing the officers to query the database themselves to help in doing their work. Examples provided of possible uses for the data include assessing the level of risk with a particular client, by getting historical data on the client and other projects undertaken by the client; ⁴⁴ identifying the relative merits of different strategies that have been employed on similar types of projects; or being able to get comparative data (by whatever variables chosen) to examine performance over periods of time; for example, targeted versus actual outcomes or all projects by sector in one period compared to another period. ⁴⁵

The comment by one FedNor respondent provides a succinct summary of the views of almost everyone with respect to the use of the indicators:

The other side of this, of course, raised by a minority of FedNor respondents, is that NODP should be cautious about rejecting projects simply because they do not "fit the indicators." A couple of FedNor respondents cautioned about the danger of allowing the indicators to drive the program. Their major concern is that if there is a change in the business environment, would the focus of the program change to accommodate this, or would the decision on how to proceed be driven by the indicators.

Literature Review

In The New Rural Paradigm, the OECD (2006) outlines an approach to rural development that is based on assessments of practices in a wide variety of member countries. In this study, the central focus of the new approach is on investments instead of subsidies and on policies that focus on rural places rather than on supporting specific sectors.

The OECD report provides perspectives on successful delivery approaches from many member countries, including a detailed description of Canada's Rural Lens. This is a policy framework to assess proposed rural policy initiatives by the Rural Secretariat (Agriculture and Agri-food Canada). The Rural Lens is intended to ensure that federal regional initiatives support the priority areas that have been identified by rural Canadians such as: accessing federal government programs and services, access to financing, opportunities for rural Aboriginal youth, infrastructure, skills and technology for new economy, economic diversification, access to reasonable cost education and health care, strategic partnerships and promoting rural Canada as a place to live and raise a family.

The Rural Lens approach is linked to governance strategies. The OECD (2006) notes that governance issues cross levels of government. The objective (supported in Canada's case by the Rural Lens) is to establish those governance structures most likely to achieve the goals of regional policy. The OECD points out, that to do this effectively; central level financing should be tied to coordination of the policy with lower levels of government. It is also important to consider necessary factors for effective partnerships and to be aware of potential obstacles to effective partnering. Intergovernmental grants should be used with a clear incentive framework to produce the most effective initiatives. This requires accountability that should be built on targets and performance indicators, supported by ongoing monitoring and evaluation. The OECD suggests that many governance challenges can be met with best practice approaches.

Review of Performance Monitoring and Reporting

As part of the document review, we reviewed the three data collection forms that are being used to capture data on the NODP: the New Proposal form, developed to circulate the project idea to regional colleagues at an early stage in the process; the Project Summary Form (PSF) which documents the objectives and merits of the project, as well as the rationale for approving the project; and the Final Project Evaluation (FPE) Form that collects information on indicators to measure program outcomes. These forms were assessed against the Performance Measurement Matrix in the 2007 Results-Based Management and Accountability Framework for the Northern Ontario Development Program and were found to align. Data are being captured and recorded in the NODP Final Project Evaluation Database.

Exhibit D-1 of Appendix D provides information related to data collected at the proposal stage. Exhibit D-2 in the same appendix provides comparable data typically collected at the end of the project. These exhibits identify, for each indicator, summary statistics (Count=number of observations, Min=minimum value, Max=maximum value, Mean=average value, Mode=most frequent value ⁴⁶, and Median=middle value of distribution of values). They also identify if indicators represent a "New" indicator (since the 2007 RMAF), or an indicator requiring access to another database ("Link needed") for example for indicators related to youth internships (Indicator 7.7). Entries in this exhibit will be discussed in the following text.

Further development of the performance monitoring and reporting system has resulted in:

• Additional indicators (15 identified as "New" in the Exhibits) providing greater coverage related to outcomes in the 2007 RMAF.
• More detail on labour market impacts (assessing jobs as being temporary or permanent, maintained or created, and measuring impacts in terms of numbers of jobs and of full-time equivalence (FTE). (These are collapsed and presented in the Exhibit as counts of data on the total employment impacts by indicator.)
• Assessing impacts by target group. (See for example, jobs (Indicators 5.4–5.7) or business impacts (Indicator 7.3) for Aboriginals, Francophones, youth, and women; or impacts for Aboriginal communities or organizations (See Indicator 1.7). (Detail by target group is not presented in the Exhibits but is available in the database.)

Most indicators represent immediate outcomes related to the NODP projects. However, a minority represent intermediate outcomes that first require achievement of immediate outcomes of the project before they can be achieved. (See Indicator 2.4: As a result of the increased access to markets, the number of businesses that become new exporters). A general observation is that such intermediate indicators are less likely to be populated in the database. The typical cut-off related to the end of the project may limit the opportunity for intermediate outcomes to be achieved, as may be the case for Indicator 2.4. (Also see the intermediate Indicator 3.7 relative to the related immediate Indicator 3.6.)  ⁴⁷

3.2.2 To What Extent is Data Quality being Assured?

Conclusion

• Steps are being taken to address previously identified data quality issues. Issues identified previously such as: data collection, consistency, and client reporting have been addressed through the re-design of data collection forms, program delivery officer training and involvement and using the letter of offer to emphasize the indicators for which the client must report data. Program officers have contributed to the development of the performance measurement system and in the process have embraced it. This is likely to aid in achieving data quality as are monthly meetings to iron-out data integrity issues. Producing the annual report (NODP Performance Measurement Report for the FedNor Management Committee, Summary Results from the NODP Final Project Evaluation Report dated February 27, 2008) has been instrumental in identifying data quality problems and in correcting problems found.

Context

The earlier evaluations identified key gaps in FedNor's performance management system related to the data collection tools, consistency in using these tools, and the overall reliability of the data that was being collected.

Findings

As part of the response to the earlier concerns, FedNor has designed a number of new forms to collect outcome data, and has revised others. There are new requirements of clients to track and report on outcomes specified in the Letter of Offer; and they have developed a new database to capture and store the date. PPC continues to consult with staff but is shifting the emphasis of the consultations from identifying the appropriate indicators to ensuring consistency in understanding and use of the indicators. Clients are, for the most part, reported to be meeting their reporting requirements, and are being "educated" by the program officers on the importance of the indicators.

Producing the annual NODP Performance Measurement Report has allowed FedNor to identify outliers and missing data and to consult with the PDOs to resolve problems. FedNor respondents talked about the need for technical support beyond entry level, and worry that they are reaching a point where there is too much work for the resources available. There are concerns that this will worsen with an increased focus on evaluation and increased requests to produce technical reports from the data. FedNor is in the process of hiring another data analyst (SI-02).

FedNor has created a performance measurement system that fully meets the requirements set for it in the RMAF. In keeping with most performance measurement systems and the RMAF requirement its focus is on immediate outcomes. Longer-term outcomes, as a consequence, become the responsibility of a future evaluation. This too is in keeping with the specifications found in the RMAF.

An issue is whether it would be more efficient and cost-effective to extend the performance measurement system to collect longer-term outcome information and thereby free up some resources that might otherwise be spent on the future evaluation. This issue is beyond the mandate of the current study and as such is left as a topic for consideration by management.

Review of Documents

The 2002 Formative Evaluation identified a key gap in FedNor's performance management system related to the quality of the data; namely, the existing tools (the Project Outcome and Evaluation Score Sheets) were not being used consistently. The information in the Contribution Management Information System (CMIS) was not reliable because all performance-related fields for the CMIS were not being consistently captured. FedNor management responded by indicating that it would ensure that all performance-related fields were consistently filled out; that the Program Officers would consistently complete the Project Outcome and Evaluation Score Sheets; that FedNor would develop a mechanism to obtain results from project clients; and that the project client data would be incorporated into the CMIS database. ⁴⁸ Further, FedNor indicated that it would use the findings of the evaluation's benchmarking study ⁴⁹ to share knowledge and best practices in areas of performance measurement with the three Regional Development Agencies, and would continue its Service Improvement Initiative, and ensure that any performance-related information was captured and utilized. Performance information was to be reviewed regularly, and incorporated into decision-making on an ongoing basis throughout FedNor.⁵⁰

The 2005 Summative Evaluation found that the Project Officers were completing the required Project Outcomes and Evaluation Score Sheets for most of the completed projects reviewed; however, the evaluation also found that the score sheets were completed "inconsistently and incorrectly." FedNor staff interviewed for the evaluation indicated that they were "somewhat satisfied" with the timeliness, reliability and completeness of reporting from clients (though there were variations between clients). Greater effort was being made with new clients (at the outset) to ensure that clients understood their reporting requirements. Monitoring and Payment Officers were sometimes involved in these discussions (to discuss procedures and answer client questions). For the most part, clients interviewed for the evaluation reported that they fully understood when they were to submit reports, and what information they were required to include. The Program Officers reported that electronic reminders of when client reports were due had been "useful." Interviews with other stakeholders, such as CFDCs, revealed mixed responses with respect to how easy or difficult it was to meet FedNor requirements as they related to NODF projects. Some were concerned that reporting requirements had increased exponentially.

Key Informant Interviews for the 2008 Mid-term Evaluation

All of the FedNor respondents were asked questions about the quality of the data that is being collected, captured and stored to measure outcomes. The PDOs and M&POs were asked more detailed questions on specific changes to the data collection instruments and/or procedures ⁵¹ and to the way data is being captured and stored ⁵² than the other respondents. The other FedNor respondents were asked more general questions, ⁵³ but had the option of addressing the more detailed questions if inclined due to their limited involvement in the end analysis of the data quality. Changes to the tools/forms/instruments/procedures ⁵⁴

FedNor respondents indicated that the "tools are an ongoing work;" and that they have been in a constant state of change with the addition of indicators and/or refinement of definitions. Respondents raised the suggestion to change the tourism and ICT indicators, to more clearly capture the outcomes and refining the wording so definitions are clear; the FPE as being a "living document" that will change as they continue to refine the indicators in the interest of clarity and consistency in use/understanding. However, caution is required in that continuous change to indicators could have a negative effect on the establishment of a baseline for NODP data.

A number of changes have been made to the tools and procedures since the 2005 summative evaluation: the addition of the New Proposal form and the replacement of the earlier Score Sheets by the Final Project Evaluation form; the inclusion of indicators in the Letter of Offer which informs clients, at the outset, how success will be measured; and a requirement for manager sign-off on the FPE. The other change has been to allow the PDOs to defer completion of the FPE for up to 12 months in instances where data are not available at the close of the project.

The biggest change to client reporting is the requirement to track and report on the indicators specified in the Letter of Offer. This is new. PDOs are also reported to be using the logic model with clients to educate clients on the purpose of the indicators and the importance of data collection. The final reports provided by clients should reflect this new emphasis. Specifying which indicators to include in the Letter of Offer was an issue at the time of the interviews. Those most closely involved talked about the list of indicators that comes up with the PPG. There has been a tendency to "paste in" all of the indicators in the Letters of Offer. This was confusing for clients, in cases where the indicator(s) were not relevant. Discussions to restrict the inclusion of the indictors in the Letter of Offer to those that are pertinent to the project were later resolved in staff meetings. ⁵⁵ The other major change was the development of a new database to capture the data from the Final Project Evaluation form, the FPE database.

Compliance with reporting requirements

The program has a lot of repeat clients who understand and are used to conforming to program requirements. Getting client reports is not a problem. Some FedNor respondents talked about the challenge of managing the reporting requirement to meet everyone's needs. Ensuring that reporting is not too onerous was a concern mentioned by a number of FedNor respondents, particularly as it pertains to clients. Many clients do not have much in the way of resources for data collection. The breakdown of jobs by target groups was raised by a number of FedNor respondents who questioned the value of this level of detail. "Are these data going to be used? Are they really necessary? Are they going to be reported?" They see this requirement as being particularly problematic for those clients who subcontract to a third party, a trade advisor, for example, who might have to approach 50 different businesses for this information. While it affects a minority of projects, it is deemed to be too onerous for these clients. As well, there are concerns about the reliability of these data given that they rely on self-reporting.

There is a process in place to track the status of the progress reports, the final client report, and the FPE to ensure they are completed in a timely fashion. This is tracked by both the PDO and the M&PO. The M&PO takes the lead in identifying instances where the reports are not received in a timely fashion, or where they are incomplete. This is tied to the processing of claims. Reports are, for the most part, submitted in a timely fashion; however, if there are difficulties it tends to be with the small, not-for-profit organizations that lack the staff and resources to do reporting. This happens with a minority of projects—about 10 percent. These projects may require PDO follow up.

Consistency in data collected

Most of the FedNor respondents concur that they have "the right indicators" and that they have gone a long way to ensuring consistency in the use of the indicators. The ongoing process of consultation is directed at refining and clarifying the use of the indicators. These discussions will continue, as necessary, in an effort to ensure everyone is using the indicators in the same way, thereby improving the consistency of the data across the program.

New and revised tools described in the previous footnote have been available to collect project data since April 19, 2007, the date on which officers were requested to go back and retroactively report on projects that ended on or after July 1, 2006. It took about six months to complete the exercise. FedNor respondents indicate that about 80 percent of the indicators are available for these projects. The biggest issue with the retroactive data is that they do not have the intended results (the targets officers set at the application stage, reflected in the New Proposal form and the Letter of Offer) for the retroactive projects. There will be no indicators in the Letter of Offer for these projects, and no requirement to report on them in the final report. The PDOs went back to the clients affected to request data on the (new) indicators with mixed results. Some of these projects will not close for months. There will be gaps in the data for the "before" finalization of indicators projects.

The NODP Performance Measurement Report is also credited with helping to improve data quality. One of the by-products of the report is that it has allowed the PMWG to identify missing data and outliers in the data not caught in earlier analyses and to go back to the PDOs for clarification.

Another factor that will have an impact on the quality of the data available is the buy-in they have been able to achieve at the program delivery level and are currently establishing with clients. The officers are working with the clients to impress upon them the importance of the indicators. The quality (and completeness) of the data provided by clients can be expected to improve as clients come to better understand the importance of the indicators and self reporting becomes more reliable.

Data capture and storage

For the most part, FedNor respondents had little to say about the capture and storage of data. Officers submit a hard copy to the Master file and an electronic copy which is automatically downloaded to the FPE database. Since the FPE's go directly to the database, they believe there is no potential for human error in capturing and storing data. Some indicated that it would be advantageous if all of the forms were on line.

Little has been done to examine the quality of data that is captured and stored beyond identifying reports with no data. The main reason seems to be the lack of resources. FedNor respondents talked about the need for technical support, particularly beyond an entry level, and worry that they are reaching a point where there is too much work for the resources available. There are concerns that this will only get worse, with an increased focus on evaluation, and increased requests to produce technical reports from the data.

Review of Performance Monitoring and Reporting

Our observations from the performance measurement system review suggest:

• Initially zeros were entered in the database to record no data: "If none, write "0"". As part of the Mid-Term Evaluation, we observed that this made it difficult to distinguish a true impact of zero from no information about the impact. Program staff implemented a fix whereby an unintended impact of zero meant "no intended impact" and was recoded as a "blank". In such cases, a zero recorded as an actual outcome was replaced by a "blank" as no outcome was intended and none occurred, while a non-zero actual value indicated an unintended impact. This fix provided a correction to the problem and will be used in the future. This problem of the "overuse" of zeros led to inappropriate counts in indicators. This problem was eliminated once the "fix" had been made.
• Results are highly similar across the intended and actual data for the same project. This may suggest that intended data are used to reflect actual data unless there is evidence to the contrary to change the data. We understand that some back filling of actual data was conducted on the earliest projects in the database. Potentially actual data was used for this back-filling. Our review found one example of a negative number in both the intended and actual data for a project (See Indicator 3.1b). This project is from the period where back filling occurred resulting, we assume, in the negative actual result being applied as an intended result. This indicator is: "If applicable, what is the increase in attendance over the previous festivals/meetings/conventions or events?" (In the future the description of this indicator should be changed if negative values are to be accepted as it suggests only positive values are to be accepted.) But if this number had been first entered as an intended value it further illustrates the potential harm in using intended results as a mechanism to establish actual results. A given value, whether correct or incorrect may have more fixity than it deserves. Those asked to provide an actual number may choose to take the pre-existing intended number as it has been accepted before, might prevent questions justifying the reason for any change, or would avoid the extra effort involved in generating a new number. In the future, we recommend data collection related to the FPE occur as an independent event, without the aid of prior existing (intended) data, to ensure that collected data reflect a valid point in time estimate of impacts. Using intended data in the actual data collection process has the potential to bias the actual data.
• Some projects are highly uniform in terms of outcomes—for example, Indicator 3.2a where all 22 projects have one regional/destination tourism marketing initiative implemented. Other indicators capture disparate projects (For example, Indicator 7.1 captures the total number of participants at training or professional development events. This captures a large number of Youth Internship Projects (YIP) with typically a single intern and events with up to 750 participants.) This variety limited our ability to identify values that might suggest obvious data input errors. In the future, we recommend groupings of comparable project types at least for analysis purposes (i.e. separate YIP from others) might ease error checking.

Recommendations

NODP and the predecessor program NOD Fund has been in existence for a number of years. Consecutive evaluations had identified deficiencies in the performance measurement approach. Over the past few months there has been a significant amount of work and effort expended by FedNor to address these concerns in the area of performance measurement and monitoring.

Based on this work and the findings of the current mid-term evaluation, recommendations pertaining to strengthening the Performance Measurement System and Data Quality are clustered under two recommendations.

It is recommended that:

1. FedNor determine the benefit and cost of collecting performance measurement information on project outcomes beyond the project end date of projects. If appropriate, FedNor should extend its performance measurement system to capture these longer-term results.
2. FedNor strengthen mechanisms and formalize ongoing edit checks to ensure data quality, including:
   • consistent comparison of intended and actual outcome data for each project;
   • both, program officers and proponents report on final results independent of intended outcomes captured at the project launch stage;
   • analyze minimum, maximum and most frequent values for indicators;
   • create separate sub-indicators based on project grouping with unique outcome characteristics;
   • address concerns, anomalies with data and indicators as required;

3.0 Research Findings (continued)

3.2 Design and Delivery (continued)

3.2.3 To What Extent has FedNor Addressed Monitoring, Data Capture, Performance Reporting and Related Issues Identified in Past Evaluation Studies?

Conclusion

FedNor has addressed issues with regard to monitoring, data capture, and performance reporting identified in past evaluation studies. Significant time and resources have been dedicated to undertaking the improvements identified in previous evaluation studies. Efforts expended by FedNor to develop a new RMAF, tools and databases, and specification of client requirements for data collection, are expected to contribute to improved outcome data reporting. Reporting is still in its preliminary stages. The first performance report was recently produced and while it did reveal that some past challenges still exist for the most part data was collected against each indicator. Potentially risk variables for projects could be added to the performance monitoring system. This would allow FedNor to explore the risk-reward trade-off from projects and potentially guide decisions related to project risks NODP should be willing to undertake in the future.

Context

Earlier evaluations identified a number of concerns related to monitoring and evaluation. The 2005 summative evaluation talked about the need to refine the 2005 RMAF to more accurately reflect program priorities, and the need to develop a performance and reporting system and tools to meet the requirements outlined in the revised RMAF. The 2002 evaluation talked about the need to develop a performance measurement management and reporting culture.

Findings

FedNor has invested considerable time and resources redressing the problems identified in the earlier evaluations. Among the changes introduced are:

• The development of a new RMAF that more clearly reflects the design and delivery of the NODP.
• The design of new tools and a new database for the data collected.
• New requirements for clients with respect to outcome data.

While staff are in the preliminary stages of analyzing and reporting data, they have produced a first report based on the outcome data, and have plans for the future use of the outcome data.

Review of Documents

Exhibit II-1 summarizes the review of the three RMAFs produced for NODF/NODP under a number of key variables: context, stated mission, programming focus, key clients, activities supported, intended outcomes, and monitoring and performance measurement plans. More detail is provided in the following text.

As indicated earlier, the 2002 Formative Evaluation found that FedNor was in a relatively early stage in the development of performance indicators and targets and, more generally, of a performance measurement management and reporting culture. The report indicated that FedNor needed to develop an organizational-wide culture that ensured that performance information was not only gathered on an on-going basis, but was analyzed and used for decision-making purposes throughout FedNor. The report recommended that FedNor continue to develop its performance measure system and ensure that the system drew upon information from a wide range of sources. The two key gaps in FedNor's performance management system identified in the 2001–02 Formative Evaluation were that the existing tools (for data collection) were not used consistently, ⁵⁶ and there was no consistent, ongoing information collected from clients on the outcomes of projects. ⁵⁷ In response to the 2002 evaluation, FedNor management developed the 2005 RMAF and instituted changes to its data collections procedure, as described previously; i.e., would continue to use CMIS for information on inputs, outputs and reach; ensure that all performance-related fields were consistently filled out; would continue to have program officers complete the Project Outcome and Evaluation Score Sheet; would develop a mechanism to obtain results from project clients; and would share knowledge and best practices in the area of performance measurement and report with the three Regional Departmental Agencies. ⁵⁸

The 2005 Summative Evaluation reports that while the 2005 RMAF provided management with a better tool for measuring performance, and while the planning process (at the time of the 2005 evaluation) was more closely linked to the NODF's performance management requirements, there were still problems. Key amongst these was the finding that the 2005 RMAF did not fully reflect the design and delivery of the NODF and that the existing tools and systems had not been updated to reflect the performance measurement requirements outlined in the 2005 RMAF. It was also unclear at that time if the information being collected from the project outcome score sheet was valid and reliable, or if the program officers clearly understood how to complete the score sheets. The departmental systems for capturing and analyzing project information also came in for criticism. The Grants and Contributions Reporting System (GCRS) was perceived to be inadequate with respect to meeting the monitoring and accountability requirements of FedNor with respect to NODF. It was deemed to be poorly aligned with NODF requirements, and not at all aligned with the project outcomes sheets, making it difficult to determine the extent to which projects were achieving their intended outcomes. The fields of the database could not be adjusted as the program evolved to better reflect the changing needs of Northern Ontario communities.

The Summative Evaluation offered two suggestions to redress the problem: 1) refine the NODF RMAF to more accurately reflect program priorities and refine the performance report systems and tools to meet the requirements outlined in the RMAF; and 2) consult with Audit and Evaluation Branch and Information Management Branch to ensure that the department system provided FedNor management with the flexibility it needed to appropriately capture the performance information required to meet its monitoring and accountability requirements. In response to the evaluation, FedNor management indicated that FedNor would revise the NODF RMAF by Fall 2006, in accordance with the evaluation's recommendations; and FedNor would proceed with the development of new data collection tools and an accompanying guide which would align with, and reflect, the expected results of the current program priorities. ⁵⁹ FedNor also indicated that it would also continue to consult with the AEB and IMB regarding the design of the departmental information systems. ⁶⁰

Key Informant Interviews for the 2008 Mid-term Evaluation

As described above, earlier evaluations identified a number of concerns related to monitoring and evaluation. Key informants were asked a series of questions about the extent to which FedNor had responded to issues identified in these past evaluations. The questions were limited to PDM/PPC/SSU respondents, since it was anticipated that the officers might be less familiar with the RMAF. ⁶¹

FedNor clearly addressed the criticisms in developing the 2007 RMAF and the reporting system and tools currently employed. Earlier sections of this report indicate that FedNor has invested considerable time and resources addressing the problems identified in the earlier evaluations; specifically, they have:

• Replaced the 2005 RMAF with the 2007 RMAF which, FedNor respondents overwhelmingly agree, more clearly reflects the design and delivery of the NODP.
• Held extensive consultations with staff, throughout, to develop the appropriate performance indicators of immediate outcomes for each program component.
• Continue to consult as they refine and clarify the use of the indicators to ensure consistency in understanding and use across the program.
• Reached a point where, with the exception of additional intermediate or longer-term indicators as identified related to tourism indicators, FedNor respondents accept the indicators that have been developed.
• Officers have "bought into" the process, using the indicators to structure their discussions with clients and colleagues when proposing or rejecting projects. Projects are being screened on the basis of their fit with program priorities. This has culminated in a culture change towards performance measurement, and is extending to the clients themselves.
• Developed new requirements for clients (with respect to outcome data), clearly specified in the discussions with program officers and the Letter of Offer.
• Revised, and in some instances, replaced the tools to collect the data; and developed a new database to capture and store information from the Final Project Evaluation form. ⁶²
• While they are in preliminary stages of analyzing and reporting the data, they have produced a first report (NODP Performance Measurement Report) that is expected to receive wide distribution. Projected future use of the outcome data includes: input when seeking renewals or changes to the program; submissions to Treasury Board or Cabinet; strategic planning; and development of their Business Plan. Other suggested uses of the data include feedback to the community and input for the officers to assist them in doing their work.

Almost without exception, FedNor respondents indicated that FedNor has addressed the earlier issues, and is well positioned to have data for the 2010–11 summative evaluation.

Review of Performance Monitoring and Reporting

Based on our review of the Performance Monitoring system, the new system responds to earlier criticisms around performance measurement. It collects, stores and provides access to data that can be used to assess all immediate and some intermediate outcomes of the program logic model. This more than meets the requirements specified for the Final Project Evaluation database in Table 3 of the RMAF.

One area not explored by the RMAF's program logic model or by Table 3, and consequently not included in the Final Project Evaluation database, is the role FedNor plays as a catalyst for development in Northern Ontario. Information is not recorded in the database related to instances when FedNor does not fund a project yet is instrumental in the project's development. Such information would be valuable to a summative evaluation but risks being lost unless it is tracked on an on-going basis. Potentially this might be handled as a qualitative indicator. Changes should also be made to reflect this activity and outcome in the RMAF.

3.3 Lessons Learned

As discussed earlier, the focus of the questions related to "Lessons Learned" was modified after submission and approval of the Methodology Report. The revised questions (that replaced those in Annex A) are as follows:

• What factors have facilitated/impeded the FedNor strategy for NODP performance measurement and management of these results?
• What lessons have been learned from the FedNor strategy for NODP performance measurement and management of these results?

3.3.1 What Factors have Facilitated/Impeded the FedNor Strategy for NODP Performance Measurement and Management of these Results?

Conclusion

Involving project officers in the development of the performance measurement system resulted in greater buy-in and improved consistency around performance measurement. Having dedicated staff applied to the development task was seen as valuable, but raises a concern related to on-going staffing needs as uses for the system ramp up. Also, staff turnover creates an on-going demand for training on the system requirements and data consistency issues.

Context

FedNor expended considerable efforts on improvements to its performance measurement system and data management procedures. Factors that facilitated or impeded the process would be valuable to others engaged in similar activities.

Findings

Key Informant Interviews

i) Things that worked well

All of the FedNor respondents were asked about the factors that facilitated/impeded the FedNor strategy for NODP performance measurement and management of these results. A small number of themes evolved from the discussions.

When identifying things that worked well, FedNor respondents most frequently talked about the collaborative process adopted to identify and clarify the indicators. In the view of one respondent, who reflects the view of almost all, FedNor has "spent a lot of time getting the right indicators and getting people to understand them." As a result, there is "greater consistency" across the program. One of the consequences of the approach adopted has been the level of buy-in on the part of staff. Some talked about a "culture change" with respect to performance measurement; others talked about how it has changed the way we do business, interact with clients, and manage results. The PPC unit came in for a lot of praise for the approach adopted.

The other major theme, in terms of what worked well, was the decision to provide dedicated resources to the task. Without these resources, FedNor would not have advanced as far into the process as it has. However, there are concerns, expressed by a number of FedNor respondents, about the need for additional resources. The staff who have been involved in this activity have responsibility for other FedNor programs, in addition to NODP. Some of these other programs have made major demands on the time commitment of staff responsible for NODP performance measurement.

Challenges

Two factors were identified as having challenged the work. The first was staff turnover, which saw some FedNor respondents talking about whether greater attention needs to be paid to human resources; specifically, the need for more staff and, perhaps, in the case of support/administrative staff, staff with higher levels of technical training than entry level.

The second factor that was seen to impact negatively on the FedNor strategy was simply the complexity of what they do. There was a long learning curve to develop the 2007 RMAF; and most would agree that the refinement of definitions and instructions around data capture is not finished yet.

Literature Review

In terms of evaluation and monitoring, the OECD emphasizes the importance of these activities for promoting policy effectiveness and as a forum for dialogue among the various partners in regional policies. The report notes that competitive grant programs may have built-in reporting mechanisms to support monitoring and evaluation.

Review of Performance Monitoring and Reporting

From our review of the Performance Monitoring System:

• The system focuses primarily on immediate outcomes of projects but includes some intermediate outcomes. This may be an attempt to balance the need for early indications of potential program impact with the desire for longer-term measures of potential success.
• To the extent that the Performance Monitoring System is used to track intermediate outcomes it may result in more projects being included in a pending file (status 618). This would limit the value of the system as a tool to provide early indications of potential impact.
• FedNor staff may instead consider closing all files after one year and instituting periodic tracking of select indicators (intermediate indicators and updates of key immediate indicators). This might better balance the dual goals (early indications and longer-term success) while providing evidence on how impacts extend over time.

3.3.2 What Lessons have been Learned from the FedNor Strategy for NODP Performance Measure and Management of These Results?

Conclusion

Although extensive discussions have occurred on the correct measures there remains debate about whether longer-term measures should be included. At issue is the appropriate balance between early evidence of immediate outcomes and later evidence related to intermediate or ultimate outcomes. Also important is the benefit to be derived through monitoring such data by management and the savings in resources devoted to a future summative evaluation.

Context

FedNor expended considerable efforts on improvements to its performance measurement system and data management procedures. Lessons learned through the process would be valuable to others engaged in similar activities.

Findings

Key Informant Interviews

The interviews asked all of the FedNor respondents about the key strengths of the NODP performance evaluation measures design; major limitations or challenges with the NODP performance evaluation measures; and any improvements needed to the performance evaluation measures.

The questions did not generate a lot of response, but a number of themes did emerge. One theme was the challenge and frustration associated with performance measurement. As one respondent phrased it, "it's been a learning curve." The PPC unit was credited for what has been accomplished; in particular, the persistence of PPC staff in moving this forward. The other major theme related to the problem of attribution. As one respondent phrased it, "we're talking about indicators early in the process." "It's a small program in terms of the amount of money injected into the economy. How much can it be expected to influence macroeconomic factors such as GDP, employment, etc.?"

Literature Review

The New Rural Paradigm (OECD 2006) notes that there may be important challenges in developing appropriate indicators because of the uniqueness of different rural locations and the absence of robust models for assessing "exactly how and why regional development occurs". The two related problems identified by the OECD are the "incomplete information framework" in which policy implementers hold much of the necessary evaluation knowledge and the causality issue that makes it difficult to link policies and results.

In addressing the challenge of developing good indicators, the OECD indicates that these indicators must be unambiguous and must be broadly accepted as valid ("uncontroversial and reliable"—there must be "trust by all actors").

Review of Performance Monitoring and Reporting

There is a continuum of information that is useful to program management. At one end of the continuum are operational and accounting details on activities. These are normally provided through management systems and databases on grants and contributions. At the other end of the continuum is evaluative information typically coming from formative and summative evaluations. Performance measurement is situated between these two end points. The exact location depends on whether management performance information needs are for immediate outcomes, longer-term outcomes or some combination of the two.

The current NODP Performance Measurement System is positioned on this continuum closer to the immediate outcomes end. This is typical for most performance measurement systems and is what was called for in the RMAF.

Our key informant interviews identified minority views suggesting that performance measurement might be better positioned farther along this continuum. We think this is an issue that can only be addressed by management based on their information needs.

To the extent that there is value to management by extending Performance Measurement to more intermediate (and perhaps ultimate outcomes) the system could be expanded to cover other outcomes or longer timeframes or periodic measurement. One of the benefits that should be considered in extending the "reach" of the performance measurement system is the savings in resources needed by the summative evaluation.

4.0 Conclusion

Introduction

This chapter provides our assessment of the results of the mid-term evaluation.

4.1 Program Relevance

FedNor's NODP operates in an environment where there are other programs offering support to projects that meet its eligibility requirements. Although programs overlap, staff of the various programs co-ordinate their activities in an attempt to avoid duplication that would waste society's resources.

Given NODP's community-based approach, and expertise in the issues facing Northern Ontario communities, it is often the first to respond and often fulfils a coordination role with regard to the other available funders. As the OECD suggests there is a need for coordination in situations of overlap and FedNor is well positioned to ensure this is achieved.

Support across the various funders may reflect different parts of the project whole. When the same parts are funded by different sources, a "stacking analysis" is conducted to ensure support does not exceed a preset level. Proponents are required to inform staff of all funding sources and a media tracking service is used to alert staff of the potential of support by some other unknown funding source.

Because of the very real possibility of duplication in the environment where the NODP operates, coordination is key to ensuring taxpayers' resources are not being wasted. FedNor plays a useful role as co-ordinator and facilitator for Northern Ontario development.

In conclusion:

• NODP and other programs overlap. However there are mechanisms in place to ensure collaboration occurs and to avoid duplication across funding sources.
• FedNor is collaborating with other departments and government agencies to reduce duplication—a potential negative consequence of overlap.

4.2 Design and Delivery

There is general agreement that the current performance measures address the concerns of the previous evaluations and that they meet the requirements as set out in the 2007 RMAF. Performance measures focus on immediate outcomes of the program as called for in the RMAF. There were some suggestions that performance measurement might be better focused on more intermediate or longer-term outcomes or also consider qualitative indicators. There were minor concerns that the indicators, and not the inherent worth of the project, might influence project selection.

Our review of the performance measurement system suggests that it exceeds the design established by the 2007 RMAF. Although it includes some intermediate outcomes, the focus is on immediate outcomes in keeping with most other performance measurement systems and presumably the needs of program management given the effort that has been devoted to developing the system internally. We note that there is a trade-off in terms of the timeliness of information if the focus of the performance measurement system is shifted to longer-term outcomes.

Data on most projects are captured at the end of the project. Some projects can be put into a pending file for follow-up for up to an additional year, if it is judged too early to assess the outcomes of the project. Projects are closed when the Final Project Evaluation is completed and no further project follow-up is scheduled, in keeping with the RMAF. As a result, the performance measurement system will provide a series of snapshots of project performance typically at the end of the project. Longer-term follow-up is reserved for the summative evaluation and possible case study activities, also in keeping with the RMAF.

This result is an information gap. To fill it, periodic follow-up of projects would be required. This might be conducted following a fixed schedule—end of project, first, and third year following the end—for all or a stratified random sample of those projects that could be reasonably expected to have longer term outcomes, and for a limited list of indicators (with more emphasis on intermediate outcomes) beyond the one-year anniversary of the project. Such a revised system would eliminate the current practice of placing some projects in a pending file for later follow-up.

To reduce cost, FedNor might consider mailing the data collection request to proponents (along with clear instructions on how to complete the form) and follow up only on those not returned. FedNor should not provide previous intended results to proponents at the time of the actual data collection so as to encourage independence in the data collection events and avoid a potential source of bias.

In conclusion:

• The program now collects, captures, and safeguards performance measurement information in keeping with the program's RMAF. Collected indicators focus on immediate outcomes from the program's logic model as called for in the RMAF. As such the program is in compliance with the requirements set for it.
• To date the collected indicators have been used to produce a not-yet-released annual report. Other uses anticipated in the future include renewing and/or making revisions to the program, new submissions to Treasury Board or Cabinet, strategic planning, and the development of their Business Plan.
• Program officers helped develop the performance measurement system and in the process have embraced it. This is likely to aid in achieving data quality as are monthly meetings to iron-out data integrity issues. Producing the annual report has been instrumental in identifying data quality problems and in correcting problems found.
• FedNor has addressed monitoring, data capture, performance reporting and previous related issues identified in past evaluation studies. Potentially, risk variables for projects could be added to the performance monitoring system. This would allow FedNor to explore the risk-reward trade-off from projects and potentially guide decisions related to project risks NODP should be willing to undertake in the future.

4.3 Lessons Learned

Involving FedNor staff in the development of the performance measurement indicators has been instrumental related to buy-in, future support related to efforts to gather and maintain consistency of performance data, and promotion of the indictors with project proponents.

We conclude:

• Involving project officers in the development of the performance measurement system resulted in greater buy-in and improved consistency around performance measurement. Having dedicated staff applied to the development task was seen as valuable, but raises a concern related to on-going staffing needs as uses for the system ramp up. Also, staff turnover creates an on-going demand for training on the system requirements and data consistency issues.
• Although extensive discussions have occurred on the correct measures there remains debate about whether longer-term measures should be included. At issue is the appropriate balance between early evidence of immediate outcomes and later evidence related to intermediate or ultimate outcomes. Also important is the benefit to be derived through monitoring such data by management.

Management Response to Recommendations in the 2007 Mid-term Evaluation of the Northern Ontario Development Program

Footnotes

¹ The program name was changed in July 2006. (back to footnote reference 1)

² In October 1995, FedNor was given responsibility for the direct delivery of the Community Futures Program in Northern and Southern Ontario. Community Futures Development Corporations (CFDCs) are strategically located in rural communities throughout Ontario and help FedNor and Industry Canada to fulfill its mission by providing three service lines: Community Economic Planning and Development, Investment in Local Business, and Business Development and Counselling Services. (Formative Evaluation of FedNor Final Report dated May 9, 2002) (back to footnote reference 2)

³ Consistent with new approaches to regional development introduced in other economically depressed regions of Canada, in March 1996, FedNor initiated an alternative service delivery method which moved away from the direct assistance to business mechanism first employed by the agency to fully repayable commercially-based lending backed by loan loss reserves delivered by third parties. FedNor also shifted its programming emphasis towards the needs of a knowledge-based economy. (Results-Based Management and Accountability Framework (RMAF) for the Federal Economic Development Initiative in Northern Ontario (FedNor) dated September 24, 2001) This change is credited in the 2005 RMAF with championing Northern Ontario's drive towards economic diversification and laying a solid foundation in Northern communities and businesses for meeting the challenges of the new economy. (back to footnote reference 3)

⁴ In 1999, FedNor reorganized its programming into five strategic areas: Trade, Innovation, Investment, Connectedness, and Community Partnerships. These priorities were established to specifically address key issues facing people who lived and worked in Northern Ontario. New program components were also introduced in 1999 to facilitate a shift to a knowledge-based economy with an emphasis on connectedness, innovation, and community partnerships. They included: youth internships, community economic infrastructure, telecommunications infrastructure, applied research and development, community recovery, innovation capacity building, and information and communications technology (ICT) applications. (Evaluation of FedNor Final Report dated May 9, 2002) For a summary of program evolution see Exhibit II-1 in the Approach and Work Plan Chapter of the Methodology Report, dated February 2008. (back to footnote reference 4)

⁵ Results-Based Management and Accountability Framework for the FedNor Program dated March 31, 2005. (back to footnote reference 5)

⁶ Summative Evaluation of the Northern Ontario Economic Development Fund (NODF) Final Report dated January 2006. (back to footnote reference 6)

⁷ Results-Based Management and Accountability Framework (RMAF) dated January 29, 2007. (back to footnote reference 7)

⁸ ibid. (back to footnote reference 8)

⁹ The Results-Based management and Accountability Framework for the Eastern Ontario Development Program dated October 2006, makes reference to the same resources dedicated to performance and evaluation as that in the 2007 RMAF, suggesting that these resources are shared across programs. (back to footnote reference 9)

¹⁰ Summative Evaluation of the Northern Ontario Economic Development Fund (NODF), Executive Summary January 2006 and the Results-Based Management and Accountability Framework for the Northern Ontario Development Program dated January 29, 2007. (back to footnote reference 10)

¹¹ NODF/NODP's objective has changed over the three RMAFs. This mission statement is different from that specified in the 2001 RMAF, where the objective of the mission at that point in the program was "to promote economic development in Northern Ontario leading to a strengthening of the national economy and contribute to the successful development of business and job opportunities." (RMAF dated September 24, 2001) (back to footnote reference 11)

¹² The designated area of coverage includes the following Census Divisions: 44 Muskoka District Municipality, 48 Nipissing District, 49 Parry Sound District, 51 Manitoulin District, 52 Sudbury District, 53 Sudbury Regional Municipality, 54 Timiskaming District, 56 Cochrane District, 57 Algoma District, 58 Thunder Bay District, 59 Rainy River District, and 60 Kenora District. (RMAF dated January 29, 2007) (back to footnote reference 12)

¹³ ibid. (back to footnote reference 13)

¹⁴ Initially the program was restricted to SMEs with fewer than 250 employees and less than $20 M in annual sales. (RMAF dated September 24, 2001) (back to footnote reference 14)

¹⁵ Engage and empower communities to take responsibility for managing their own futures and seizing opportunities that will result in long-term economic benefits, including job creation, economic diversification and enhanced business competitiveness, supporting sustainable development. The RMAF indicates that FedNor was to refocus its CED efforts in 2006–07 to be more proactive and strategic, focusing on three priority areas: community strategic planning, implementation of community plans and community economic infrastructure. (back to footnote reference 15)

¹⁶ The RMAF indicates that with the creation of FedNor's International Business Centre (IBC), the Team Northern Ontario (TNO) trade network of community-based trade organizations and advisors, and support for trade missions well in place, FedNor will refocus the work of the IBC in 2006–07 to concentrate more on direct assistance to business, such as business counselling, youth internships and skills development with a view to building export and business management competencies in Northern Ontario firms. FedNor support for tourism was to be strategically focused toward projects that stimulated higher levels of tourism activity and tourism revenues in the North. (back to footnote reference 16)

¹⁷ Six key areas of support are identified to strengthen the innovation capacity and performance of the Northern Ontario economy: innovation infrastructure and education/training programs; knowledge-based activities and new productivity-enhancing technology/new forms of work organization; export development and international competitiveness; help in developing strategic clusters and linkages among research/post secondary education institutions, business, communities and government; competition in ideas, technologies and solutions and partnerships that connect new ideas/solutions to societal challenges and market opportunities; and access to advanced telecommunications services. (back to footnote reference 17)

¹⁸ Developing and enhancing telecommunications infrastructure, networks and ICT applications. (back to footnote reference 18)

¹⁹ The CFDCs receive their core funding from FedNor through the Community Futures program. It will also provide repayable direct assistance to SMEs in areas such a applied research and development and explore was to facilitate access to patient and venture capital. (back to footnote reference 19)

²⁰ Support initiatives that help to attract, develop and retain highly skilled people and provide opportunities for life-long learning. (back to footnote reference 20)

²¹ RMAF dated January 29, 2007. (back to footnote reference 21)

²² The CFDCs are a vehicle to help FedNor managers and staff accomplish their mandate. They are incorporated non-profit organizations governed by a local volunteer board of directors that represent various community interests. Local CFDCs pursue their own priorities and strategies for development by creating and implementing a strategic community plan in cooperation with their partners. CFDCs provide advice, information and referral service to local businesses and entrepreneurs and access to capital for small business financing by operating locally governed investment funds that can provide loans, loan guarantees or equity investments for business start-up, expansion or stabilization. (Evaluation of FedNor Final Report dated May 9, 2002) (back to footnote reference 22)

²³ Evaluation of Fed Nor Final Report dated May 9, 2002. (back to footnote reference 23)

²⁴ RMAF dated September 24, 2001. (back to footnote reference 24)

²⁵ RMAF dated September 24, 2001. (back to footnote reference 25)

²⁶ RMAF dated January 29, 2007. (back to footnote reference 26)

²⁷ The Performance Measurement Matrix identifies the data (indicators) to be collected, the source (typically where the data is stored but sometimes an external source of information), and frequency (annual, quarterly, ongoing) for the collection. (back to footnote reference 27)

²⁸ Originally 20 were scheduled. This was reduced by one when one of the officers failed to participate in a scheduled group interview. (back to footnote reference 28)

²⁹ Interviews were planned in advance to accommodate respondent schedules. One officer did not call in to participate in the scheduled interview. (back to footnote reference 29)

³⁰ Evaluation of FedNor, Final Report dated May 9, 2002. (back to footnote reference 30)

³¹ A comparative review of FedNor to the NOHFC that was carried out as part of the evaluation found that their goals were similar (to support and provide economic growth and development with the communities of Northern Ontario), as were their processes and components. However, the report finds that other programs of the Ontario government were more restrictive and less effective than FedNor in practice. Another benefit of FedNor was that it provided project development and project management support, in addition to funding. (ibid.) (back to footnote reference 31)

³² FedNor's uniqueness was seen to lay in the degree to which it had moved (since 1996) from firm-specific financing to community-based programming; and while other regional development agencies were doing this as well, the report concluded that they were not doing it to the same extent as FedNor. (back to footnote reference 32)

³³ Summative Evaluation of the Northern Ontario Economic Development Fund (NODF), Final Report dated January 2006. (back to footnote reference 33)

³⁴ Questions 2 a) through 4) in both Interview Guides. (back to footnote reference 34)

³⁵ There are exceptions where funding can exceed 90 percent in the case of Aboriginal projects or not-for-profit projects of less than $100,000. (back to footnote reference 35)

³⁶ As described earlier, officers solicit proposals as opposed to waiting for proposal to arrive. They work with the proponent to develop the project, to ensure that it meets with program objectives. Part of the complete evaluation that is done on every project involves discussing how the project will be funded, who else will be involved, where the other funding sources will be directing their contribution, and how to balance this to ensure sufficient funding for the project, while reducing the potential for overlap or duplication. If FedNor is the initial point of contact for the proponent, and the project meets NODP priorities, the NODP officer will (as necessary) suggest potential partners to the proponent and/or bring other programs/funding sources "to the table" to work out the role each will play in the contribution formula/agreement that is worked out for the project. The partners who fund tourism projects have tried to be more strategic in their approach through a Partnership Agreement that has been in place for the last five years. Partners to the Agreement have formalized what they are prepared to fund and the way in which the various partners will participate in tourism-related projects. This is reported to keep investment directed at the strategy. In sectors such as Innovation, it was suggested that the program may not be accessing the funding available from the granting councils to the extent it could. This is under review. The procedure is also slightly different with these projects. Proponents are directed to the other programs, first, and come back to FedNor with the results. NODP has the option to top-up or take on the project itself. (back to footnote reference 36)

³⁷ Things identified included: changes to the FedNor RMAF, changes to the departmental system; and the addition of new staff/changes in the staff involved in the development of performance measurement tools. (back to footnote reference 37)

³⁸ Some of the program officers thought the score sheets needed to be revamped and that more work need to be done with clients (at the outset) to impress upon them the importance of performance data. (Summative Evaluation of the Northern Ontario Economic Development Fund (NODF), Final Report dated January 2006). Instead of revamping score sheets they were replaced by the FPE. Also some project officers have begun using the program logic model with proponents to identify intended outcomes and stress the importance of providing the requested performance data. (back to footnote reference 38)

³⁹ Question 4 in the PDM/PP&C/SSU Interview Guide; Question 3 f) and h) in the PDO Interview Guide; and Question 2 g) and h) in the M&PO Interview Guide. (back to footnote reference 39)

⁴⁰ Question 9 in the PDM/PP&C/SSU Interview Guide, Question 6 in the PDO Interview Guide; and Question 5 in the M&PO Interview Guide. (back to footnote reference 40)

⁴¹ Up coming staff training sessions (in the Northeast on May 6 & 7, 2008 and in the Northwest on May 17, 2008 may further refine the measures (back to footnote reference 41)

⁴² The first presentation to staff happened in the spring of 2007 when the indicators were introduced at an annual meeting. However, respondents believe staff would have had some awareness prior to this from conversations with colleagues, including those on the PMWG. (back to footnote reference 42)

⁴³ The narrative is also automatically transferred to the FPE database. (back to footnote reference 43)

⁴⁴ Particularly in the case of new PDOs who would not have a history with the client. (back to footnote reference 44)

⁴⁵ At the moment there is very limited access to the raw data within the database. Allowing such access would require major changes to the technology. (back to footnote reference 45)

⁴⁶ Note that the Mode cannot be calculated in all circumstances. If there is no mode, the formula returns the characters N/A. (back to footnote reference 46)

⁴⁷ Deciding if an indicator is immediate or intermediate requires detailed knowledge of the particular project. However from the information available other probable intermediate indicators are: 1.7, 1.8, 2.5, 2.6, 2.7, 2.8, 2.9, 4.2, and 4.3. (back to footnote reference 47)

⁴⁸ The formative evaluation was concerned that the outcome data captured was limited to the perspective of the program officers. The evaluation recommended that program officers continue to capture these outcome data, but that they be supplemented with direct feedback on outcomes from the clients. (back to footnote reference 48)

⁴⁹ The benchmark study involved a review of three federal regional development agencies: Atlantic Canada Opportunities Agency (ACOA); Canada Economic Development for Quebec (CED-Q); and Western Economic Diversification (WD). Each of the other agencies within the Industry Portfolio are separate operating agencies whereas FedNor is a business unit within Industry Canada. As such, it relies on the expertise available from the Industry Canada Audit and Evaluation Branch to support internal performance measurement capabilities. FedNor annual business plans and performance reports are minimal compared to the other agencies which provide individual Reports on Plans and Priorities (RPP) and Departmental Performance Reports (DPR). FedNor plans and performance reports consist of a few pages within the overall RPP and DPR of Industry Canada. Most performance related targets and measurement are in the area of resources expended, activities completed and outcomes, such as number of projects completed. The consultants found that FedNor was responding to the increased requirements for performance measurement, management and reporting and had put more resources into this area over the previous two years. A performance measurement and reporting culture was gradually being introduced to FedNor program delivery areas as exemplified by the RMAF (developed and approved by Treasury Board in September 2001) and the introduction of the Project Outcomes and Evaluation Score Sheet to be completed by Program Officers. (Evaluation of FedNor final Report dated May 9, 2002) (back to footnote reference 49)

⁵⁰ Management Response to the Formative Evaluation of FedNor dated June 24, 2002. (back to footnote reference 50)

⁵¹ Question 3 a) to f) in the PDO Interview Guide; and Q2 a) to f) in the M&PO Interview Guide. (back to footnote reference 51)

⁵² Question 4 a) to d) and Question 5 in the PDO Interview Guide; and Question 3 a) to d) and Question 4 in the M&PO Interview Guide. (back to footnote reference 52)

⁵³ Questions 5, 6, 7 and 8 in the PDM/PP&C/SSU Interview Guide (back to footnote reference 53)

⁵⁴ We began the discussion by identifying the tools and procedures that are being used to collect outcome data for NODP projects. Below is a brief summary of each tool/instrument that is being used to capture data and monitor projects, and its intended purpose:

• Application form completed by the client with PDO input. It is the first step in obtaining project funding. As part of the application process, proponents are required to "certificate" all sources of funding for the project. At this stage in the process, proponents may be referred to other sources, for example, someone proposing a tourism project would be advised to go to the Ontario Ministry of Tourism website which posts a variety of tools they could draw upon to assist with the project.
• New Proposal form is a new form between the Application form and the PSF which introduces the project to other staff members and specifies the indicators which are the rationale for proposing the project. It is the first step in framing the project for the client and the peer review/feedback process. It is completed by the PDO.
• Project Summary form is distributed to everyone in FedNor, initially, as a draft working document. The revised PSF includes a full evaluation of the project, including a risk assessment, and the feedback provided by colleagues. All projects at the development stage receive a risk assessment that examines both client risk and project risk. This assessment is incorporated into the PSF and is written into the Letter of Offer, along with the indicators against which the project is to be monitored and assessed. The revised PSF goes forward for approval or rejection.
• Letter of Offer contains the indicators against which the project will be monitored and assessed and details on the funding arrangements, including a clause which stipulates that the proponent must keep the program apprised of any changes to the funding arrangements. The discussions and schedules that can be put into the Letter of Offer can be very specific.
• Progress Reports and Final Project Report are completed by the client. The progress reports are used by both the PDO and P&MO to monitor the project and process claims. Projects are monitored at an activity level (to ensure that they are on target) and financially, when invoices are processed. Clients are not required to report on indicators until the end of the project. There is no information on indicators in the progress reports.
• For some projects, the project file will contain impact analyses, survey results, and third party evaluations in addition to everything else. One of the spin-offs of the change in focus towards measurement is that the officers have begun to encourage applicants to build evaluation into their projects. In some cases, FedNor is funding this, along with the project funding.
• Informal documents created over the life of the project including PDO notes to the file stemming from site visits and/or discussions with clients, comments added by the M&PO to the claims documents, photographs, etc.
• A "shadow file" that most PDO reportedly keep which tracks the client's other activities/projects, etc., some of which may find its way into the project files.
• Final Project Evaluation (FPE) form completed by the PDO. There has to be a FPE for every completed project. In early January 2008, an additional requirement was added to the FPE in the form of manager sign-off on the FPE. If there are any differences between the Manager and PDO assessment, the Manager, PDO and Policy meet to discuss and resolve the issue. If the FPE needs to be changed or updated, this happens following the consultation.

(back to footnote reference 54)

⁵⁵ The ability to delete irrelevant indicators in the Letter of Offer was clarified in both the northwest and northeast staff meetings on February 19, 2008. (back to footnote reference 55)

⁵⁶ The review of the CMIS revealed that some important data elements were not consistently captured. (back to footnote reference 56)

⁵⁷ The outcome data that was being captured in the Project Outcome and Evaluation Score Sheet was criticized for being limited to the perspective of the Project Officer. The evaluation recommended that short-term outcome data be collected from clients. The report also noted that the Service Improvement Initiative was limited to the collection of process-related data. (back to footnote reference 57)

⁵⁸ Management Response to the Formative Evaluation of FedNor dated June 24, 2002. (back to footnote reference 58)

⁵⁹ Recommendation 9, Management Response to Conclusions and Recommended Areas for Follow-up (to the 2005 Summative Evaluation). (back to footnote reference 59)

⁶⁰ Recommendation 10, ibid. (back to footnote reference 60)

⁶¹ Question 3 a) to d) in the PDM/PP&C/SSU Interview Guide. (back to footnote reference 61)

⁶² The 2005 summative evaluation identified a problem with the CMIS. Respondents for the mid-term evaluation indicated that the earlier evaluation may have made too much of respondent comments with respect to the CMIS. At the time of the evaluation, they were working on replacing the database, and were consulting with users about potential changes to the database. FedNor's wanted to ensure that the CMIS was upgraded and revised so that it captured the data FedNor required. As of February 2008, the upgrade has been delayed for several years (expected completion now Winter of 2008), with the result that FedNor developed a new database for its own purposes, the FPE database. (back to footnote reference 62)