Archived

Table of Contents

• 1.0 Introduction
• 2.0 Program Profile
  • 2.1 Rationale
  • 2.2 Delivery Approach
  • 2.3 Primary Reach
  • 2.4 Results
  • 2.5 Governance Structure
• 3.0 Logic Model
• 4.0 Performance Measurement and Evaluation Strategies
  • 4.1 Ongoing Performance Measurement
  • 4.2 Evaluation
  • 4.3 Integrated Ongoing Performance Measurement and Evaluation Strategy
  • 4.4 Methodologies
• 5.0 Reporting Strategy

1.0 Introduction

This document sets out a Results-based Management and Accountability Framework (RMAF) for the h2 Early Adopters (h2EA) program managed by Technology Partnerships Canada (TPC). The RMAF includes the following components:

• Profile: an overall assessment of the program=s origin and rationale, target audience (reach), delivery approach, resources, governance, and commitment to achievable results;
• Logic Model: a comprehensive and logical model that clearly explains the sequence of outcomes that are expected to flow from the program=s activities;
• Performance Measurement and Evaluation Strategy: a strategy to measure the overall results of the program on an ongoing and periodic basis; and,
• Reporting Strategy: a strategy for the specific reporting requirements.

This RMAF will enable TPC to assess the results of the activities, adjust operations, and continue to improve the h2EA program.

Throughout this document, the term "hydrogen technologies" is used to indicate "hydrogen technologies and hydrogen-related technologies."

2.0 Program Profile

The h2EA program is aimed at addressing the urgent need to accelerate market adoption of hydrogen technologies and other hydrogen-compatible technologies that facilitate the transition to a hydrogen economy and attract world-class talent and investment to Canada. Through Technology Partnerships Canada (TPC), support will be given to the establishment of integrated hydrogen complexes such as "Hydrogen Villages", "Hydrogen Highways" and other similar partnerships in locations across Canada.

This support will target multiple public- and private-sector partners to demonstrate these technologies and showcase Canadian capabilities. These partnerships will involve integrating hydrogen-compatible technologies and hydrogen production, storage and distribution technologies with fuel cell and related portable, stationary and mobile applications in a microcosm of a hydrogen economy. Results of this initiative will include real-world experience and expertise; early market adoption of hydrogen technologies and infrastructure needed to support their wide-spread use; and increased consumer and investor awareness.

2.1 Rationale

The overall objective of the United Nations' Framework Convention on Climate Change aims to stabilize greenhouse gas concentrations in the atmosphere. This will require substantial investments in technologies to fundamentally change, over the long term, how we produce and consume goods and services. The development, demonstration and deployment of climate change technologies is a long-term and expensive endeavour. Climate change technologies compete with well-established and embedded infrastructures. Technological innovation will play a key role in meeting Canada's international obligations. This is particularly true with respect to the promise of hydrogen not only as an energy source, but as the foundation of a new economy: the hydrogen economy.

The transition to a hydrogen economy offers a unique opportunity for Canada. The use of hydrogen (produced from alternative energy sources), fuel cells and other hydrogen technologies can significantly reduce emissions of greenhouse gases and other pollutants. This transition offers significant growth and investment opportunities for other sectors, and for communities and regions across Canada, to develop clusters of expertise and capabilities, attract talent and investment, and offer rewarding careers for young Canadians. Canada is well positioned to provide international leadership in the hydrogen sector.

While Canada is currently a world leader in the development of hydrogen technologies, its first mover advantage is not secure since other countries are making significant new investments toward a hydrogen economy and are attempting to attract Canadian grown technological leadership. In addition to this international competition, Canadian firms facemany technology and marketplace hurdles which they cannot overcome alone. With respect to hydrogen technologies, significant headway must be achieved in reducing cost and improving their performance and safety. On the marketplace side, the most urgent need is to begin testing hydrogen technologies and infrastructure in real-world settings to assess and improve reliability and durability, to support efforts to further reduce costs, and to increase public, consumer and investor awareness and acceptance of hydrogen technologies. Implicit in a transition are necessary steps to an ultimate objective. Thus, emphasis is also required on technologies that are hydrogen-compatible.

Given the importance of social/behavioural change in the acceptance of new technologies, increased consumer and investor awareness and understanding of these new technologies and their uses are critical. There are significant technical, social, institutional and market barriers that cannot be overcome without government assistance.

Substantial progress has been and continues to be made through a variety of federal government initiatives in the areas of technology development and partnership activities, however greater collaboration and coordination is required to meet the next strategic objective - high profile, multiple partner market demonstration projects.

The h2EA program, managed by Technology Partnerships Canada, will support public and private-sector partnerships that showcase hydrogen complexes, for example "Hydrogen Villages" and "Hydrogen Highways" to demonstrate hydrogen technologies, illustrate their integration and accelerate market acceptance and take-up. Through demonstration projects, we will learn by doing: lessons learned from both successes and failures will advance the science and pace of technological development in this field. The historical experience has shown that these demonstration projects do not allow firms to rapidly realize main revenue streams, but rather help them to pursue further research and development on next generation technologies. Therefore, pursuant to the Treasury Board Policy on Transfer Payments, although contributions under the h2EA program are repayable, little or no repayments are expected.

Initiatives such as the h2EA program will allow Canada to stay at forefront of the transition to a hydrogen economy in order to maximize its benefits for Canadians.

2.2 Delivery Approach

The h2EA program will be implemented by Industry Canada through Technology Partnerships Canada (TPC) in conjunction with the regional development agencies and other federal departments and agencies and their respective programs. Memoranda of understanding may be developed between TPC and federal departments and agencies to support the implementation of this initiative.

During the period FY2003-04 to FY2007-08, funding will be made available by TPC to the Canadian Transportation Fuel Cell Alliance (CTFCA) to allow it to strengthen or extend its ability to meet the objectives of the h2EA program.

Regional development agencies will be invited to participate in the building of partnership coalitions, considering and monitoring the projects, and communicating the successes of regional initiatives.

Experts from appropriate Industry Canada sectors will be involved in the due diligence process.

Other federal departments and their respective programs will also be invited to participate, where their program objectives complement those of the h2EA program.

2.3 Primary Reach

In order for the h2EA program to be successful, it must "reach" several different types of organizations. These include organizations participating in hydrogen production, storage and distribution, the development of hydrogen and hydrogen-compatible technologies and users who will integrate hydrogen technologies into portable, stationary and mobile applications.

The primary clients and/or users of the h2EA program include:

• fuel cell and balance of plant technology developers;
• hydrogen producers, distributors and storage system developers;
• end users and original equipment manufacturers of hydrogen portable, mobile and stationary applications;
• material developers and developers of ancillary parts and other component providers;
• federal, provincial and municipal government organizations;
• NGOs, associations and institutes;
• hydrogen-compatible technology developers and other private sector users; and,
• public and private sector research organizations.

In addition to some of the above groups, co-deliverers and participants include:

• energy producers of renewable and non-renewable energy sources;
• Canadian and international technology and equipment developers, manufacturers and suppliers;
• Industry Canada Industry Sector, Regional Development Agencies and other Industry Portfolio partners; and,
• communities.

In addition to those listed above, there are a number of other stakeholders with interests in having a successful h2EA program operating in Canada. These include the Hydrogen and Fuel Cell Committee, National Research Council, NSERC, Canadian Transportation Fuel Cell Alliance, and other federal programs, which provide support and funding for research, development, and demonstration of hydrogen and hydrogen-compatible technologies, which relate to the results of the h2EA program.

2.4 Results

The Terms and Conditions for the h2EA program outline the following outcomes for the program:

• to foster the development and early introduction into the marketplace in Canada of technologies, such as fuel cells and those used to produce, store and distribute hydrogen, that support the transition to a hydrogen economy.

These are reflected in the ultimate outcomes included in the logic model in the next section. The ultimate outcomes are:

• accelerated market acceptance of hydrogen technologies and hydrogen-compatible technologies;
• world-class talent is attracted and retained; and
• domestic and foreign investment in Canada are attracted.

In addition, the ultimate outcome of the h2EA program is to make a contribution to the objectives of Canada=s Climate Change Agenda.

2.5 Governance Structure

TPC will be responsible for design, delivery and administration of the h2EA program. The chart in the following exhibit illustrates the relationship between the key groups related to TPC and the h2EA program.

Figure 1 - h2EA program Organizational Chart

The roles of key groups involved in the h2EA program are shown in the table below.

Table 1 - Roles and Responsibilities of Key Groups
Role	Responsibility
Program design	TPC
Outreach / communication	TPC
Program delivery	TPC in collaboration with Regional Development Agencies
Monitoring and reporting	TPC
Provision of advice	Hydrogen Fuel Cell Committee

3.0 Logic Model

The logic model for the h2EA program is provided in Figure 3. For the purposes of the model, the following definitions were used:

• Activities: An operation or work process internal to an organization intended to produce specific outputs (e.g., products or services). Activities are the primary link in the chain through which outcomes are achieved.
• Reach: The individuals and organizations targeted and directly affected by the program. This may include partners and intermediaries as well target clients communities.
• Outputs: Direct products or services stemming from the activities of a policy, program or initiative.
• Outcomes: An external consequence attributed to an organization, policy, program or initiative that is considered significant in relation to its commitments. Outcomes may be described as immediate, intermediate or ultimate; direct or indirect; intended or unintended.

As shown in Figure 2, TPC has varying degrees of control and influence over the success of the program.

Figure 2 - A Basic Results Logic

Note: Basic results logic as adapted from Exhibit 3.2of the Guide for the Development of Results-based Management andAccountability Frameworks, TBS.

The basic results logic for the h2EA program initiatives should flow as follows:

• TPC has direct control over the resources, activities and outputs of the program. These activities and outputs include program management, collaboration, monitoring and reporting.
• As TPC interacts with outside individuals and groups and the external environment, it ceases to have full control over the outcomes of the program, and begins to instead exert influence. TPC has a direct influence on the success of the program in the immediate outcomes. These outcomes are consistent across projects under the program.
• As the program logic moves further away from the actions of TPC, a greater number of external factors affect the success of the program. The immediate outcomes then lead to intermediate outcomes where TPC has an indirect influence.
• Later on in the program logic, external influences are greatest and TPC exerts only limited influence. At this stage, the program is contributing to the ultimate outcomes.

The logic model for the h2EA program appears in Figure 3.

Figure 3 - h2 Early Adopters Program Logic Model
Objective: To work in partnership with hydrogen industry stakeholders to foster the demonstration and early introduction into the marketplace in Canada of technologies, such as fuel cells and those used to produce, store and distribute hydrogen, that support the transition to the hydrogen economy

4.0 Performance Measurement and Evaluation Strategies

On-going performance measurement allows for regular assessment of the progress made towards desired outputs and outcomes. Evaluations provide periodic reviews of a program, policy or initiative and its associated projects on a more rigorous and in-depth scale. These two approaches should combine to provide the information needed to manage for results. This RMAF proposes that the h2EA program will collect performance information according to two strategies:

• Ongoing performance measurement: The h2EA program proposes to collect ongoing information on outputs, reach, immediate outcomes and intermediate outcomes. This information will be reliable because it will be based on a multitude of sources, both internal and external. These proposed performance measures are articulated in the ongoing performance measurement strategy (Section 4.1).
• Periodic evaluation: A periodic review or evaluation of the h2EA program can provide information about the degree to which outcomes have been achieved, as well as the degree to which the program contributed to these outcomes. In addition, the evaluation can consider issues related to operations and structure of the program and the effectiveness of strategies. The proposed evaluation issues are presented in Section 4.2.

As with the logic model, the relative control and influence of TPC are reflected here. For the activity and output measures, TPC has full control and is able to collect all of the associated measures. For example, TPC's fiscal management processes will provide the associated on-going financial measures. As the control of TPC turns into influence on partners/co-deliverers, the measurement strategy focuses on the collection of information which will describe the nature of these relationships. This information can then help maintain and improve these partnerships with other federal departments and funding recipients. Through a review of the content and frequency of communications with these key partner groups, the project leads can find the tone and tenor of the relationships, indicating the success of the h2EA program in developing these important partnerships.

When measuring the progress towards the intermediate outcomes of the program, where the influence of external factors begins to play a larger role, TPC will rely upon both their implementing departments / agencies and funding recipients to help gather and report on results.

The external influences are greatest on the ultimate outcomes and TPC exerts only limited influence. To reflect the level of external influence at this stage of the program, an evaluation will take place to report on the success of the program. This evaluation will summarize the

projects that have been completed according to expectations and standards. The program's metrics will be summarized, and will be contextualised within these external influences examining the broader communities' use of the infrastructure and the projects' contribution towards the final outcomes.The concept of learning by doing is fundamental to a demonstration program such as the h2EA program. This implies that the program, projects and other stakeholders learn from the successes and challenges encountered in implementing the demonstration projects. Courses corrections will need to be recognized and accepted in project monitoring and have been reflected in the ongoing performance measurement and evaluation strategies.

4.1 Ongoing Performance Measurement

Ongoing performance measurement provides a regular assessment of the performance of the program. The information provided by performance measurement helps to provide managers with the knowledge required in order to manage effectively. This strategy will identify the key pieces of information that need to be collected (in the form of performance indicators) in order to determine the progress of the initiative.In implementing and reviewing the measurement strategy, there are several factors that need to be taken into consideration. These include the amount of resources available, the amount and type of data and information that already exists and that which must be collected with new systems or sources, as well as the appropriate unit to collect the information. In developing the performance measurement strategy, every effort was made to use existing information sources, in order to reduce the costs and to minimize the collection burden within TPC. This performance measurement strategy is designed to be evergreen.

4.2 Evaluation

The purpose of this section is to outline the range of evaluation issues and questions to be used to guide the evaluation studies of the program. TBS guidelines for RMAFs note that evaluation issues should be identified. These guidelines, as well as the TBS evaluation policy and further TBS guidance, indicate that the evaluation issues should be categorized as follows:

• Relevance - Does the policy, program or initiative continue to be consistent with departmental and government-wide priorities and does it realistically address an actual need?
• Success - Is the policy, program or initiative effective in meeting its objectives, within budget and without unwanted outcomes?
• Cost-effectiveness - Are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches?
• Results-based management - Is results-based management being used effectively and is the appropriate information being collected?

The key issues that should be considered in evaluating the program are identified in the following pages according to these categories of issues. Given that this is a new program, a fifth category of issues focussed on implementation has been included as well.

Implementation

1) Is the design of the program appropriate given its objectives?

2) To what extent has the program been implemented as planned?

3) Have there been any unforeseen barriers / challenges to implementing the program as planned?

4) Are the communication and outreach activities reaching their intended audiences and having the desired impact?

Relevance

5) To what extent is the h2EA program fostering the development and early introduction into the marketplace in Canada of technologies that support the transition to the hydrogen economy?

6) To what extent has the national and internal environment changed since the program was created? Has the program evolved to meet the changing conditions? Are the program's objectives still relevant?

7) What role does the program play in the Technology and Innovation Initiative? Is there overlap or duplication?

Success / Progress

8) Are program objectives clearly understood by management and staff, co-deliverers stakeholders, clients and potential clients?

9) To what extent is the h2EA program achieving stated program objectives? Were there significant contributing factors and impediments to program success?

10) Who are the beneficiaries of the program? To what extent is the program reaching the intended target firms, including SMEs?

11) How have co-deliverers benefited from working with the h2EA program?

12) To what extent have funded projects achieved objectives identified in proposals and contribution agreements? How about technical and commercial success? What have been the environmental impacts of funded projects? What other impacts have occurred?

13) To what extent have the impacts of funded projects extended beyond the funded firm to the larger industrial community?

14) Were there any unintended outcomes?

Cost-Effectiveness

15) Are there alternative, more cost-effective ways of achieving the stated outcomes?

16) What changes could be made to improve the performance and likelihood of success for the h2EA program?

17) How effective are the governance structure and channels of communication for management and delivery?

18) Are the roles and responsibilities of the program and the co-deliverers clearly delineated and understood?

Results-Based Management

19) How effective is the on-going performance framework in providing a complete picture of the program's performance?

20) Has management made appropriate use of on-going performance measurement information?

21) How well is the on-going performance framework being used in planning and decision-making at the program and project level?

It is important to note that some of these issues may be excluded at the time of the evaluation and others may be added in order to better reflect the situation at that time.

The table below shows the proposed evaluation schedule for the h2EA program.

Table 2 - Proposed Evaluation Schedule
Evaluation	Timing	Focus
Six Month Review	6 months after start-up	Issues related to implementation particularly outreach and communication
Formative Evaluation	Q1 2005	Issues related to implementation, early success and results-based management
Summative Evaluation	Q4 2007	Issues related to rationale, success, cost-effectiveness and results-based management

4.3 Integrated Ongoing Performance Measurements and Evaluation Strategy

It is critical to understand the links between the performance measurement and evaluation strategies, especially as it relates to the h2EA program. Ongoing performance measurement collects data on early outcomes and some intermediate outcomes at specific points in time and over a number of years. Evaluation allows for more in-depth analysis of the achievement of program outcomes. It includes the outcome data collected through the ongoing performance measurement strategy. These more complex issues are examined in an evaluation study in which more comprehensive and complex approaches can be used to better understand the success of an initiative, and the underlying factors influencing success. The evaluation also addresses questions concerning the relevance and cost-effectiveness of an initiative.

In the case of TPC and the h2EA program, the small number of demonstration projects - their formation, planning, implementation, monitoring and progress reporting - will form the basis for most of the on-going performance measurement of this program. Evaluation will be used to complement the project-oriented reporting on a periodic basis - at the interim and final stage of the program cycle covered by the requested funding.

The proposed integrated performance measurement and evaluation strategy identifies the performance indicators and the information which is required to effectively monitor theperformance of the key results and sub-results at any particular time. The integrated performance measurement and evaluation strategy also outlines the indicators that will be collected or addressed through the periodic evaluation activities. This strategy also identifies how often the information needs to be gathered and by whom. The process and particulars of the strategy are provided in the following table.

Table 3 - Integrated Performance Measurement and Evaluation Strategy
Performance Area	Indicator	Data Source	Responsibility	Frequency
Immediate Outcomes
Quality, seamless service to clients and co-deliverers is delivered	Extent to which TPC consults with co-deliverers Evidence of TPC coordinating with appropriate OGDs	Records of meetings, MOU, e-mails	TPC	Ongoing
	Opinion of OGDs, co-delivers, stakeholders, clients	Interviews, surveys	TPC	Periodic (Evaluation)
Increased awareness / acceptance of program by target clients, stakeholders and the public	Level of awareness of program among target firms	Record of enquiries Survey of target firms	TPC	Periodic
	Level of acceptance of program among target firms	Survey of target firms	TPC	Periodic
		Correspondence with key stakeholders Records of media commentary	TPC	Ongoing
Appropriate uptake of the program	Interest in the program by target firms Interest in the program by key stakeholders	Record of enquiries Correspondence with key stakeholders	TPC	Ongoing
	Number of quality applications Range of interest (regional, public/private)	Internal program files Correspondence with key stakeholders	TPC	Ongoing
Partnerships / coalitions of companies working together are developed	Number of partnerships / coalitions developed Number of target firms participating in partnerships/coalitions	Contribution agreements Program and project files	TPC	Ongoing
Demonstration projects successfully implemented	Number of proposals developed Number of proposals accepted and funded	Contribution agreements Program and project files	TPC	Ongoing
Increased knowledge in demonstration participants	Knowledge of target firms	Periodic survey of participants	TPC	Periodic
	Nature of projected project deliverables from funded demonstration projects	Project progress reports Internal TPC project files	TPC	Periodic
Development of new firm capabilities	Analysis of new capabilities developed in firm through project	Application files, project files Final project reports	TPC Project participants	Ongoing / Annual
		Opinion of participants, stakeholders and co-delivers	TPC	Periodic (Evaluation)
Codes, regulations, standards and guidelines are identified and developed	Number of codes / standards that are identified Evidence of Canadian standards that become adopted by international agencies	Internal program files Industry press Decision records of standards bodies meetings Correspondence with stakeholders Survey	TPC	Periodic (Evaluation)
Strategic alliances are fostered and developed	Emergence of new firms Emergence of new joint ventures between firms	Project files Business media reports Correspondence with target firms	TPC	Ongoing
Identification of R&D gaps and new research needs and elements of governance structure	Number of new R&D projects related demonstrations	(R&D) TPC applications	TPC	Ongoing
		Survey of participants	TPC	Periodic
Intermediate Outcomes
Trades persons and companies build skill capacity	Development of curriculum in universities and colleges Development of qualified tradesmen (to service hydrogen applications)	Periodic literature search and/or survey	TPC	Periodic (Evaluation)
Companies, universities, and community colleges build technology capacity	Development of curriculum in universities and colleges	Periodic literature search and/or survey	TPC	Periodic (Evaluation)
	Analysis of extent of project success (exceeded, met, met most, met some of target)	Project files	TPC	Ongoing
Adoption of technology beyond funded firms (e.g., spin-off technologies)	Number of unfunded firms and projects that become involved in a demonstration	Business media reports, surveys, evaluation case studies	TPC	Ongoing and periodic (Evaluation)
Strengthened industrial capability in Canada and build community critical mass	Number of firms that join a cluster Growth or size and number of firms within a cluster	Business media reports, evaluation case studies	TPC	Ongoing and periodic (Evaluation)
Relative cost-effectiveness of different alternatives is improved	Cost / kWH for different energy alternatives Unit costs declining	Proponents through contribution agreements	TPC	Periodic (Evaluation)
An integrated supply of hydrogen technologies and hydrogen-compatible technologies is developed	Breadth of participation (from supplier to end-user) in demonstration projects Number of different types of participants in demonstration projects	Internal program files on projects Business media reports (re. New and unfunded participants)	TPC	Ongoing
Increased awareness and understanding, public confidence of hydrogen technology and hydrogen-compatible technologies	Number and nature of articles written in the media Number of questions that get raised with the Minister Number and nature of related Qs & As in Parliament	Files on media reports Qs & As Internal files on enquiries	TPC	Ongoing
	Opinion of general public	Surveys, focus groups	TPC	Periodic (Evaluation)
Development of new non-polluting sources of electricity is influenced	Announcements of new projects which do not use fossil fuels	Media reports	TPC	Periodic (Evaluation)
Ultimate Outcomes
Accelerated market acceptance of hydrogen technologies	Number of hydrogen services and products that are launched Sales of hydrogen products	Internal files Media and business press Surveys	TPC	Periodic (Evaluation)
World-class talent attracted and retained	Number of skilled labour in hydrogen cluster Number of PhDs in related fields Number of researchers	Surveys Literature search	TPC	Periodic (Evaluation)
Domestic and foreign investment in Canada attracted	Growth of clusters in terms of number of firms and size of firms	Media and business press Surveys	TPC	Periodic (Evaluation)
Contribution to the objectives of Canada's Climate Change Agenda	Linkage of the h2EA program outcomes and objectives to Canada's Climate Change application of SMART, particularly replication potential agenda	Analysis of program files, logic models	TPC	Periodic (Evaluation)
	Opinion of key stakeholders	Interviews	TPC	Periodic (Evaluation)
	Potential long-term GHG reductions by demonstration projects1	Project files	TPC	Periodic (Evaluation)

¹ Order of magnitude estimates would be used in this case (e.g., through the use of the SMART model). Detailed modelling would be beyond the scope of this project.

4.4 Methodologies

In accordance with best practices, the proposed approach for future evaluation studies involves the use of multiple lines of evidence and complementary research methods. The suggested methods make use of the ongoing performance measures as well as a number of additional methodologies. These methodological approaches are discussed in more detail below.

• Review of Correspondence Records - Day to day correspondence records (e.g., e-mails, phone calls, letters, etc.) may be reviewed to track the level of early engagement, reactions, awareness, understanding and actions being taken by project proponents and stakeholders. Given the nature of the projects, this should provide a rich, inexpensive sources of information on the progress of the initiative.
• Industry Canada Management Information Systems and Database Review - In addition to being a major source of information for ongoing performance measurement, information in the departmental management information system provides the foundation on which other elements of the evaluation study are built. It can provide basic resource utilization information as well as some of the information required to identify clients and beneficiaries.
  
  TPC is continually monitoring and adjusting its performance information system to meet management requirements. As TPC continues to modify its performance measurement and reporting system, the level of performance information will improve and support both ongoing performance reporting and periodic evaluation.
• Program and Project Document and File Review - This research method is closely connected to the first and is another foundation stone to the evaluation study. Policy and planning documents will be used to confirm program objectives and examine program design and delivery, roles and responsibilities, and performance management questions. Project files, client progress reports and final reports will support analysis of objectives achievement and beneficiaries.
• Personal and Telephone Interviews with Selected Clients - In-depth interviews by an experienced consultant with selected funded firms would complement the telephone survey and provide more extensive information on selected topics such as impacts of the project on firm technical capabilities and commercial success as well as relevance and alternatives.
• Case Studies - Case studies are a valuable method of probing more deeply into specific aspects of the program, in particular the details of how the project impacts on the firm and beyond it. While case studies cannot be considered representative, they are valuable to describe the types of results that can occur. It is therefore important to choose cases that demonstrate the various pathways by which impacts can occur, and that demonstrate the impacts beyond technical success and perhaps beyond the firm.
• Focus Groups / Expert Panel - Instead of interviews with individual key informants knowledgeable about government programs supporting Canadian industrial innovation, focus groups or expert panel is more likely to achieve useful input and discussion on program relevance and alternatives to current design and delivery.
• Socio-Economic Impact Analysis - As part of a major summative evaluation four or five years after the program is fully implemented, it may be appropriate to conduct an analysis of the economic effects of the program on the hydrogen industry as well as industrial areas benefiting from environmental and enabling technologies developed through the program.
• Literature Review and Media Monitoring - A review of related literature will provide information about the impacts of the program. In addition, monitoring of the media, including the business press, will be a source of information about program impacts and effects (e.g., non funded firms participating in projects, spin-off effects, etc.).

5.0 Reporting Strategy

The h2EA program will rely on the following reporting strategy:

• For each project, the funding recipient will provide an annual report on progress achieved during that fiscal year within one hundred and twenty days following the end of each fiscal year. This annual report shall contain the information listed in Schedule C of each contribution agreement.
• Performance information will also be provided through the Industry Canada reporting progresses for the Reports on Plans and Priorities (RPP) and the Departmental Planning Report (DPR).
• The program's annual progress will be included in TPC's Year In Review.
• The formative evaluation report, to be available in fiscal year 2005, will be submitted to the Department's Audit and Evaluation Committee, to TBS, and an executive summary will be posted on the Department's web site. The full evaluation report will be a public document and, as such, accessible to others upon request.
• The summative evaluation report, to be available in fiscal year 2008, will also be submitted to the Department's Audit and Evaluation Committee, to TBS, and an executive summary will be posted on the Department's web site. The full evaluation report will be a public document and, as such, accessible to others upon request.

The results of on-going performance measurement will be used to make the necessary adjustments to the initiative. In addition, the performance measurement strategy will be reviewed on an annual basis. Adjustments will be made as required to ensure that the performance information is appropriate and useful for on-going management requirements.