Six-Month Progress Report (00148)

This page has been archived on the Web

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Archived

Final Report

Submitted to:

Industry Canada / Technology Partnerships Canada

Submitted by:

Performance Management Network Inc.

November 23, 2004




Table of Contents




Executive Summary

This report summarizes the progress of the hydrogen Early Adopters(h2EA) program –launched in the Fall of 2003. The report focusses on the first six monthsof progress, with some reference to progress through August of 2004.

Implementation

The h2EA is recognizedas a program which is "pushing the envelope" in terms of attemptingto proactively foster hydrogen communities in Canada. For this reason itembodies several features which have caused some barriers to a smoothstart-up. These include:

  • A new and developing sector – leading to a newness in the collective understanding of issues and terms like 'infrastructure' as well as to ambiguities in terms of effected communities. This led to significant trial and error – both for applicants as well as for participants in the delivery network
  • Consortia applications – leading to communication complexity and specifically – to difficulties re: legal liability
  • An aggressive expenditure time frame – leading to a roll out process which was not able to iron out all of the key details beyond initial awareness building. This led to some confusion re: funding conditions as well as re: the role of various delivery partners in the process
  • A demonstration type of initiative (as opposed to R&D) – leading to some need for changes to TPC's typical contribution assistance approach. Given the demonstration nature of the initiative, and its focus on consortia within an emerging community of public and private enterprise – there has been an unprecedented need to adapt the contribution assistance mechanisms of TPC (e.g. less emphasis on technical risk assessment, less pay back potential, consortia liability, concerns re: treatment of capital)

Success / Progress

Given the implementation issues,h2EA has made reasonableprogress. Understanding of objectives has evolved over time, appropriateaudiences appear to have been eventually reached and co-deliverers areemerging in their program support roles.

Cost-effectiveness

The governance and role definition ofh2EA continues to evolve,as IC regional offices, regional development agencies and others begin to feelmore involved in the process. The program's designation of specific officersfor projects has helped the process. Despite what has generally been regardedas strong, professional and responsive support fromh2EA staff – there isa concern that the program is under resourced to handle the load for thissmall, but highly complex and innovative program.

Results-based Management

While the RMAF, and the early progress review (i.e. this study) were seenas positive features, there is concern re: the clarification of roles anddedication of resources to monitor further progress in the program. The mostpromising approach would seem to be one which engaged project stakeholders incase by case tracking of key desired outcomes such as technical milestones,collaborative achievements, target audiences reached, codes and standardsimpacts, sustained behavioural adaptations (if applicable) and where possible,quantitative benefits in terms of energy, wastage / pollution and / or costsavings.

Recommendations

In order to enhance the progress ofh2EA over the comingmonths, the following recommendations are offered.TPC managementshould:

  1. Clarify h2EA's niche vis a vis other alternative energy and hydrogen initiatives such as SDTC and CTFCA.
  2. With respect to item # 1, review the resource loading, roles, mandate and the implications of these on all delivery players – including h2EA, TPC, SDTC, CTFCA, Industry Canada regional offices and other partners.
  3. Pursue ways and means to improve channels of communication relating to program developments, application requirements, eligible activities and costs, and useful / important learning from the first year of operations.
  4. Enhance the performance planning and measurement framework with more explicit recognition of progress among co-delivery and intermediary communities, and the build-up of a network relating to h2EA adaption. (Examples are provided in this report – see Figure 1, Appendix D and Appendix E.) Elements of the measurement framework will need to be applied on a case by case basis – given the diversity of the projects at hand.
  5. In order to achieve the above, TPC should pursue ways and means to increase the resources devoted to h2EA program management.

1.0 Introduction

1.1 Objective and Scope

The objective of this report is to review the progress of the hydrogenEarly Adopters Program – launched in the fall of 2003. Thisreport addresses two goals:

  1. It provides a progress report focussed on the first six months of the initiative – addressing the intended results and issues established in the 2003 Results-based Management and Accountability Framework (RMAF); and,
  2. The report reviews the results logic and the appropriateness of the previously established framework and measurement strategy. It makes suggestions as to future adjustments.

The scope of this report involves an analysis of progress over the firstsix months of the h2EAinitiative. For this reason, the focus is on early activities and outputs, aswell as the engagement and support of key collaborators – and the reachand early influence of the h2EAtarget community.

1.2 The h2EA Initiative

A full program profile for h2EA is contained in the Results-based Management andAccountability Framework – filed in the summer of 2003. This sectionsummarizes the profile information.

The Program is aimed at addressing the urgent need to accelerate themarket adoption of H2 technologies and other H2 compatible technologies thatfacilitate the transition to a hydrogen economy and attracting world-classtalent and investment to Canada. Through Technology Partnerships Canada(TPC), support willbe given to the establishment of integrated hydrogen complexes such as"H2 Villages", "H2 Highways" and other similarpartnerships.

This support will be targeted to multiple public- and private-sectorpartners to demonstrate theses technologies and showcase Canadiancapabilities. These partnerships will involve integrating H2 compatibletechnologies and hydrogen production, storage and distribution technologieswith fuel cell and related portable, stationary and mobile applications in amicrocosm of the hydrogen economy. Results of this initiative will includereal-world experience and expertise; early market adoption of H2 technologiesand infrastructure needed to support their wide-spread use; and increasedconsumer and investor awareness. A total of $60M has been allocated fromBudget 2003 for this initiative, of which $10M has been made available byTPC to the CanadianTransportation Fuel Cell Alliance to allow it to strengthen or extend itsability to meet the objectives of the Early Adopters Program.

In terms of the intended delivery, h2EA was intended to be implemented by Industry Canadathrough Technology Partnerships Canada (TPC) in conjunction with the regionaldevelopment agencies and other federal departments and agencies and theirrespective programs. Memoranda of understanding may be developed betweenTPC and federaldepartments and agencies to support the implementation of this initiative.

During the period FY2003-04 to FY2007-08, a total of $10M was madeavailable by TPC to the CanadianTransportation Fuel Cell Alliance (CTFCA) to allow it to strengthen orextend its ability to meet the objectives of the Early AdoptersProgram.

Regional development agencies were invited to participate in thebuilding of partnerships and coalitions, considering and monitoringthe projects, and communicating the successes of regional initiatives.

Experts from appropriate Industry Canada sectors have been involved inthe due diligence process.

Other federal departments and their respective programs were alsoinvited to participate, where their program objectives complementthose of the Early Adopters Program.

As noted above, the reach of the h2EA initiative follows a progression. Over the earlystages, the program reaches Industry Canada groups (i.e., sector branch andregional office staff), followed by a broader group of agencies within thefederal government and other intermediaries. As these liaisons and networksare built, the initiative begins to reach key members of its primary targetcommunity – the groups representing important actors in the fledglinghydrogen community. From this group, the intent is to reach broadercommunities of interest – including the Canadian public. Figure 1represents a reach and results logic for the initiative. It is derived fromthe 2003 RMAF – embellished to include a more complete picture ofh2EA reach andresults-based on experience to date.

Figure 1: The Reach and Results of h2EA

2.0 Methodology

This progress report included the following methodologies:

  • document, database, file reviews;
  • in-depth interviews; and,
  • analysis and reporting.

Note that, after consulting TPC management, a survey of participants / applicants wasnot undertaken at this time.

Each methodology is described in the sections which follow.

2.1 Document, Database, File Reviews

In the context of this six month review, it was important to reviewinformation from existing sources such as program and project documents andfiles as well as the information in the program database. These included:

  • records of attendees at information / outreach meetings and sessions;
  • communication documents (e.g., press releases, presentations, invitations, etc.);
  • project proposals and other elements of the 'work-up' file for projects;
  • planning and other program reports produced to date;
  • information from existing program databases; and,
  • any other relevant documents.

This approach provided valuable information on program implementation aswell as program reach.

2.2 In-depth Interviews

A selection of in-depth telephone interviews were completed to collectinformation on program design (objectives, strengths, weaknesses,improvements), barriers or challenges, communication and outreach activitiesand early program success.

2.3 Analysis and Reporting

The above sources were analyzed with a view to addressing key questionsoutlined in the study planning report, as well as being used to review theresults story and measurement strategy for h2EA.

3.0 Findings

The findings of this report follow the issues established by the 2003 RMAFand are addressed in the following sections.

3.1 Implementation

The implementation of the h2EA initiative was conducted by TPC over the first six monthsas follows:

Pre-October 2003 √ Preparatory consultation, design development
October 2003 √ Announcement of program
√ Simultaneous release of program Terms and Conditions, explanatory material, "how to Apply" documentation – all released on the web expenditures in year 1.
October 2003 to March 2004 √ Liaison with partners and intermediaries
  • Hydrogen Fuel Cell Committee (meetings)
  • Canadian Transportation Fuel Cell Alliance (meetings)
  • Sustainable Development Transportation Committee (meetings)
  • Canadian Hydrogen Association
November 2003 to February 2004 √ Information sessions held in Montreal, Toronto, Saskatoon, Calgary, Edmonton, Vancouver and Charlottetown
√ 3 phased approach I) IC regional offices; ii) Government Partners; iii) Primary Target Community and public
November 2003 to February 2004 √ Application intake
√ Receipt of applications
√ Refinement

3.1.1 Is the design of the program appropriate given its objectives?

Early feedback suggests that the design of the program is generallyappropriate given its objectives, however there are items which requiredrefinement in the first 6 month period.

  1. Optimistic Community 'Readiness'

    The program was designed with significant funding to spend in a short time frame. Given the need to foster awareness, understanding and collaboration among a disparate set of players – there is some evidence that it would have been useful to have had a design which would have rolled out over a slightly longer 'sunrise' time frame. In other words – given the wide scope and cause-effect complexity, as well as the transformative effect desired (i.e., the need to foster a collaborative hydrogen community) it appears to have been optimistic to have laid out a design which anticipated the significant flow of contribution expenditures in year 1.

  2. Liabilities for Consortia

    Early applicants appear to have struggled to address questions of liability posed by TPC during the application process. The design of agreements requiring joint and several liability among consortium partners appears to form a barrier to progress. Liability issues have been found to surround other TPC programming (e.g., This has been an issue for the larger, mainstream TPC program as noted in the 2003 evaluation study.) and, if anything, it looms even larger for many members of the fledgling Canadian hydrogen sector.

  3. Clarity of Roles and Responsibilities

    There is some evidence that the design of the program may not have anticipated the full extent of the support roles and responsibilities to be played by all co-delivery partners –including other government departments, various 'alliances' (e.g. SDTC, CTFCA) and IC regional officers. (See also section 3.3.2)

3.1.2 To what extent has the program been implemented as planned?

The program released information on its website simultaneous to itsOctober announcement. This showed a strong harmony in terms of specificinformation supporting the general announcement.

The information sessions were generally conducted in the November-Decemberperiod – essentially as planned – with the exception of onesession extending into February 2004.

Applications for the first wave were slower than expected with only twoapplications submitted for the November 28th initial deadline.The process picked up steam with the second and third waves however, resultingin more than 40 applications by February 16, 2004.

In general, submissions have been refined and consolidated – indiscussions among h2EAprogram officers, regional officers and hydrogen sector proponents.

3.1.3 Have there been any unforeseen barriers / challenges to implementing the program as planned?

One area of unforseen barriers has been that of internal applicationprocessing and approvals. Questions and uncertainties emerging as parts of thestandard TPCapplication process – such as issues relating to liabilities and (seesection 3.1.1) revenues and assets arising from the demonstration projects,questions regarding the treatment of capital equipment and potentialrepayability appear to have slowed down the approval process.

3.1.4 Are the communication and outreach activities reaching their intended audiences and having the desired impact?

The strong eventual response of various groups, some of whom were notinitially 'known' to be interested in these areas – but who eventuallyassembled strong consortium proposals (names withheld due to pendingcontribution negotiations) suggests that program communications weresuccessful in widening the 'net' of potential actors in Canada's hydrogenclusters.

In terms of communications with partners and co-deliverers, there appearsto have been some gaps in clarity. It was noted that assistance in thedelivery of h2EA was notincluded in the HQ-Regional TPC Memorandum of Understanding.

3.2 Success / Progress

3.2.1 Are the program objectives clearly understood by management and staff, co-deliverers, stakeholders, clients and potential clients?

Program objectives appear to be clearly understood by most management andstaff – as well as among government co-deliverers. A detailed analysisof the understanding of objectives among clients and potential clients wasbeyond the scope of this review, however it appears that the need todemonstrate the workings of hydrogen initiatives has been reasonably wellunderstood by all parties.

3.2.2 Who are the early beneficiaries of the program? To what extent is the program reaching the intended target firms, including SMEs?

An analysis of the first three approved projects shows the following:

Project #1: Fuel Cell Technology

Lead recipients and partners
Fuel Cells Technology: Technology provider
Ontario Power Generation: System integrator
University of Toronto: End-user
Participants
Enbridge Gas Distribution: Energy supplier
Air Liquide Canada: Energy supplier

Project #2: Hydrogenics

Lead recipients and partners
Hydrogenics: Technology provider
John Deere e-Power Technologies: System integrator
Greenlight Power Technologies: Technology provider and system integrator
Marconi Corporation: System packaging and installation
Participants
City of Toronto: End-user
Toronto Renewable Energy Co-operative: Technology partner
Canadian Fuel Cell Alliance: Refuelling station support
Bell Canada: Technology adopter and system integrator
Purolator Courier: End-user

Project #3: Ballard

Lead recipients and partners
Ballard: Technology provider
MGE: Technology integrator and distributor
Bell Canada: End user, technology adopter
Marconi: System packaging and installation
Praxair: Hydrogen storage and distribution
IBM: End user, technology adopter and system integrator
UTM: End-user, technology adopter and system integrator

Other projects display a similarly wide and in some cases even widerdiversity in partners and participants.

3.2.3 How have co-deliverers benefited from working with the program?

It is early in the process to pronounce on co-deliverer benefits. The roleof co-delivery partners and communications with them have evolved extensivelyover time.

3.3 Cost-effectiveness

3.3.1 How effective are the governance structure and channels ofcommunication for management and delivery?

Collaboration among members of the Hydrogen and Fuel Cell Committeeappears to be progressing well.

Some early concerns have emerged in both governance and communications.

In terms of governance of the contributions, the approval process appearsto have emerged as a bit more lengthy and difficult than initiallyanticipated.

In terms of communications – while the program made a strong effortto release information at the time of the program announcement, there weresome aspects of program which appear to have remained unclear including:

  • consortia vs. individual company shared liabilities
  • treatment of revenues and assets arising from the projects
  • 'stacking' rules relating to contributions from other programs
  • crown corporation eligibility
  • the role of different co-delivery groups (Who does what?)

3.3.2 Are the roles and responsibilities of the program and the co-deliverers clearly delineated and understood?

The roles and responsibilities of all delivery partners and collaboratorsappears to have shown some ambiguity during the early months of the programroll-out. Regional IC officers in some cases appear to have been placed inroles which they may not have anticipated as interlocutors between projectproponents and TPCmanagement.

3.3.3 What changes could be made to improve the performance and likelihood of success for the h2EA program?

The resourcing for the delivery of this initiative – much like theresourcing for the delivery of TPC in general – appears to be somewhat light giventhe task at hand – to foster networks and to build hydrogen communities.The scope and cause-effect complexity of this effort combined with the diversenature of applicants – suggests that a good deal of patient hand-holdingis likely in order to get projects off of the ground. The three Ottawa-basedTPC staff dedicatedto delivery, while apparently capable and well intentioned – may not besufficient to acquire, disseminate and manage all of the information andknowledge required to address the needs of the program at this time.

3.4 Results-based Management

3.4.1 Based on the way in which the program was implemented, are there any changes required to the performance measurement and evaluation strategy identified in the h2EA program RMAF?

Analysis of h2EA'sfirst 6 months of progress suggests that a key element in the successfulimplementation of a fledgling initiative such as this is the establishment ofa supportive network – not just in the target communities – butalso among policy-rule (e.g. codes and standards) makers, delivery partnersand key intermediaries. The results logic and measurement scheme for h2EA needs to be adjusted toaccommodate important achievement in awareness, understanding and supportamong the co-delivery community. This has been done in Figure 1. Annex D suggests some key additions to the performancemeasurement strategy. Annex E goes further, andsuggests a social network analysis approach for future evaluation work.

4.0 Conclusions and Recommendations

This report summarizes the progress of the hydrogen Early Adopters Program(h2EA) – launched inthe Fall of 2003. The report focusses on the first six months of progress,with some reference to progress through August of 2004.

4.1 Conclusions

Implementation

The h2EA is recognizedas a program which is "pushing the envelope" in terms of attemptingto proactively foster hydrogen communities in Canada. For this reason itembodies several features which have caused some barriers to a smoothstart-up. These include:

  • A new and developing sector – leading to a newness in the collective understanding of issues and terms like 'infrastructure' as well as to ambiguities in terms of effected communities. This led to significant trial and error – both for applicants as well as for participants in the delivery network
  • Consortia applications – leading to communication complexity and specifically – to difficulties re: legal liability
  • An aggressive expenditure time frame – leading to a roll out process which was not able to iron out all of the key details beyond initial awareness building. This led to some confusion re: funding conditions as well as re: the role of various delivery partners in the process
  • A demonstration type of initiative (as opposed to R&D) – leading to some need for changes to TPC's typical contribution assistance approach. Given the demonstration nature of the initiative, and its focus on consortia within an emerging community of public and private enterprise – there has been an unprecedented need to adapt the contribution assistance mechanisms of TPC (e.g. less emphasis on technical risk assessment, less pay back potential, consortia liability, concerns re: treatment of capital)

Success / Progress

Given the implementation issues, h2EA has made reasonable progress. Understanding ofobjectives has evolved over time, appropriate audiences appear to have beeneventually reached and co-deliverers are emerging in their program supportroles.

Cost-effectiveness

The governance and role definition of h2EA continues to evolve, as IC regional offices, regionaldevelopment agencies and others begin to feel more involved in the process.The program's designation of specific officers for projects has helped theprocess. Despite what has generally been regarded as strong, professional andresponsive support from h2EA staff – there is a concern that the program isunder resourced to handle the load for this small, but highly complex andinnovative program.

Results-based Management

While the RMAF, and the early progress review (i.e. this study) were seenas positive features, there is concern re: the clarification of roles anddedication of resources to monitor further progress in the program. The mostpromising approach would seem to be one which engaged project stakeholders incase by case tracking of key desired outcomes such as technical milestones,collaborative achievements, target audiences reached, codes and standardsimpacts, sustained behavioural adaptions (if applicable) and where possible,quantitative benefits in terms of energy, wastage / pollution and / or costsavings.

4.2 Recommendations

In order to enhance the progress of h2EA over the coming months, the following recommendationsare offered. TPCmanagement should:

  1. Clarify h2EA's niche vis a vis other alternative energy and hydrogen initiatives such as SDTC and CTFCA.
  2. With respect to item # 1, review the resource loading, roles, mandate and the implications of these on all delivery players – including h2EA, TPC, SDTC, CTFCA, Industry Canada regional offices and other partners.
  3. Pursue ways and means to improve channels of communication relating to program developments, application requirements, eligible activities and costs, and useful / important learning from the first year of operations.
  4. Enhance the performance planning and measurement framework with more explicit recognition of progress among co-delivery and intermediary communities, and the build-up of a network relating to h2EA adaption. (Examples are provided in this report – see Figure 1, Appendix D and Appendix E.) Elements of the measurement framework will need to be applied on a case by case basis – given the diversity of the projects at hand.
  5. In order to achieve the above, TPC should pursue ways and means to increase the resources devoted to h2EA program management.

Annex A – List of Persons Consulted for this Report

The following people were consulted in the process of assembling thisreport. In some cases they deferred to others in submitting answers to ourquestions.

List of persons contacted for this report1:

Mark Romoff, Executive Director, Industry Canada, Ontario Region
Gerry Cooper, Special Consultant, Industry Canada, Ontario Region
DerekYue, Innovation Regional Agent, Industry Canada, Ontario Region

Robert Sirois, Investment Director, Industry Canada, Quebec Region
Claude Morasse, Regional Innovation Agent, Industry Canada, Quebec Region
François-Nicolas Pelletier, Regional Innovation Agent, IndustryCanada, Quebec Region

Genine McCurdy, Industry Director, Industry Canada, Vancouver
AnnieDesgagné, Senior Commerce Officer, Industry Canada, Vancouver
Gordon Giles, Regional Innovation Agent, Industry Canada, Vancouver

Glenn Fields, Executive Director, Industry Canada, Prairie and NorthernRegion
Dee Pannu, Principal Consultant, Innovations, Industry Canada,Alberta

Claire Lepage, Executive Director, Industry Canada, Atlantic Region
Patricia Hearn, Provincial Director, Industry Canada, Atlantic Region
Gailene Murphy, Regional Innovation Agent, Industry Canada, Atlantic Region

Kathryn Bruce, Director General, TPC, Industry Canada, Ottawa
Simon Brault, Director,Economic and Business Case Analysis Directorate, Industry Canada, Ottawa
Jacques Cloutier, Senior Investment Manager, Industry Canada, Ottawa
Sylvain Caron, Investment Officer, Industry Canada, Ottawa
Doris Dupuis,Assistant, Industry Canada, Ottawa

Rick Whittaker, Vice President, Sustainable Development Technology Canada,Ottawa
Richard Fry, Manager, Canadian Transportation Fuel Cell Alliance,Natural Resources Canada, Ottawa

Annex B – List of Documents Consulted for this Report

List of documents consulted for this report:

  1. Results Based Management and Accountability Framework (RMAF), Industry Canada – Technology Partnerships Canada Early Adopters Program
  2. Hydrogen Road Map for Canada, Vision and Mission, Meeting the Greenhouse Gas Challenges – Kyoto and beyond: Canadian Leadership in the Hydrogen Age, Natural Resources Canada
  3. Project Application files
  4. Canadian Fuel Cell Commercialization Roadmap
  5. NRC Hydrogen and Fuel Cell Program Presentation
  6. NSERC Research Partnerships Programs Presentation
  7. Website

Annex C – Original h2EA Logic Model from 2003 RMAF

Figure 3 - Early Adopters Program Logic Model Objective: To work inpartnership with hydrogen industry stakeholders to foster the demonstrationand early introduction into the marketplace in Canada of technologies, such asfuel cells and those used to produce, store and distribute hydrogen, thatsupport the transition to the hydrogen economy.

Annex D – Suggested Performance Measurement Framework Additions

Figure D-1: Performance Framework Additions
Performance Area Indicators Data Source / Methods Responsibility for Collection Timing / frequency of measurement
Coordination with OGDs type / nature of relationships with OGDs* and alliances (e.g.SDTC, CTFCA) Content analysis of meeting minutes / correspondence

Limited survey
TPC Ongoing
Awareness and Understanding Level of communication with potential partners*

Quality and timeliness of correspondence with potential and actual partners*
Content analysis of correspondence with (potential) partners

Limited survey
TPC Ongoing
Collaborative support of OGDs, as required # of OGDs providing information and support*
Ways in which OGDs have provided support
Contribution agreements, TPC files / content analysis

Qualitative assessment of support provided

Limited survey
TPC /supporting Department / Agency Ongoing
Resources leveraged from partners $s leveraged towards funded project (actual vs. proposed)*
Level of compliance
Content analysis of contribution agreements

Limited survey
TPC Periodic, at time of signing (proposed) and at time of evaluation (actual)

* See Annex E for suggested tracking ofthe social 'network' as a means of tracking h2EA progress.

As noted in Section 3.4, h2EA's early progress suggests that it will be importantto understand the level and extent of collaboration achieved with variouspartners and stakeholder communities – as a key to understanding programprogress.

The performance areas, indicators and sources / methods listed aboverepresent suggested measures which could be integrated into the existing RMAFperformance measurement strategy.

Annex E – Implications of Our Findings for h2EA Progress Reporting and Evaluation

During the course of the data collection and analysis involved inassembling this progress report, it has become clear that the unique nature ofthe h2EA initiativerequires a reconsideration of evaluation methods. Just as conventional projectapproval 'mindsets' and criteria are not appropriate for this program (seesection 4.1 – Conclusions on Implementation), a conventional evaluationapproach is likely not warranted.

The fact that h2EA isessentially in place to accelerate the adoption of hydrogen technologies byfostering networks and infrastructure, suggests that the evaluation approachapplied to understanding its success should adapt to this program theory. Theapproach should not focus on establishing 1st and 2ndorder impacts on firms per se, but rather should start byestablishing the degree to which a successful network of collaborations istaking place. In order to do this, a 'social interaction' or 'networkanalysis' view of h2EA isrequired.

J. T. Landry, in the November 2004 Harvard Business Reviewstated, "Innovation is a social process; the more successful theinnovation, the more social the process."

Recent work commissioned for the U.S. Department of Energy (DOE) suggeststhat it is important to understand the 'social' innovation system within whicha technology is to be deployed. Knowledge of the system is necessary forgauging the commercialization prospects in a given area (e.g. for hydrogen).Figure E-1 depicts an innovation system based on social interaction. The DOEhas concluded that "Mapping social relationships in an innovation systemcan be useful in facilitating technology development and in assessingimpacts."

In our view, this kind of mapping may be vital to understanding h2EA's progress. While a detailedanalysis is beyond the scope of this study, it would appear that one possibleapproach to mapping h2EA'ssocial networking and progress might go as follows:

  1. Categorize 2004 applicants for assistance by key (functional and geographic) communities (see Figure 1 for broad categories and section 3.2.2 for market players).
  2. Proceed to survey groups to establish their progress re: hydrogen initiatives, as well as their social networks – in addition to perceptions re: program assistance. Use a limited 'snowball' survey approach2 to identify the extent of social networks being developed.
  3. Map out the social network of h2EA, perhaps attempting to show a bit of a time series (i.e. the size and diversity should build over time), as well as various linkages (type and frequency) by groups and communities.

Figure E-1: A Social Interaction View ofInnovation

Source: Adopted by Przybylinski et al., TemporaryOrganizations for Collaborative R&D: Analyzing Deployment Prospects, 2000,from Havelock and Havelock.

Derived from U.S. DOE and U.S. National Institute forStandards and Technology (NIST), A Toolkit for Evaluating Public R&DInvestment Models, Methods and Findings from ATP's First Decade Ruegg andFeller, July 2003, p 132

The use of a social networking approach would nicely address certain keyquestions relating to the relative progress of h2EA, the patterns which emerge in different applications,key 'nodes', and the co-relation of different types of networks with variousmeasures of success.

We estimate that a network analysis, built upon a planned formativeevaluation survey methodology, could replace certain conventional monitoringapproaches relating to benefits and impacts, and would therefore add little,if any cost to anticipated data collection efforts. (i.e. Certain questionswould be added to an already anticipated survey and might replace certain'conventional' questions – this would likely not add to the surveyburden.)

Annex F – Review of Approved Project Reporting Requirements vs. RMAF Requirements

As part of the work for this study, the first three successful projectswere reviewed for their potential to serve as a data source to address theinformation needs outlined in the 2003 Results-based Management andAccountability Framework.

The findings are summarized by key performance area and indicator below(note: only those areas listing internal project files as a potential sourceare included):

Performance Area Indicator Finding
Appropriate uptake of the program Number of quality applications

Range of interest (regional, public / private)
  • Evidence in the file is ample. The use of independent review (i.e. the Deloitte review) provides a ready assessment of quality.
  • The various stakeholders are clearly identified.
Partnerships / coalitions of companies working together are developed Number of partnerships / coalitions developed

Number of target firms participating in partnerships / coalitions
  • This is evident in the application and review documents on file.
Demonstration projects successfully implemented Number of proposals developed

Number of proposals accepted and funded
  • Available from the project management system.
Increased knowledge in demonstration participants Knowledge of target firms

Nature of projected project deliverables from funded demonstration projects
  • The project reporting requirements suggest that this be covered, however it is open-ended, and will likely need to be complemented by another line of evidence (e.g. a survey).
Development of new firm capabilities Analysis of new capabilities developed in firm through project
  • The project reporting requirements suggest that this be covered, however it is open-ended, and will likely need to be complemented by another line of evidence (e.g. a survey).
Codes, regulations, standards and guidelines are identified and developed Number of codes / standards that are identified

Evidence of Canadian standards that become adopted by international agencies
  • The project reporting requirements suggest that this be covered, however it is open-ended, and will likely need to be complemented by another line of evidence (e.g. a survey).
Strategic alliances are fostered and built Emergence of new firms

Emergence of new joint ventures between firms
  • The emergence of new firms was not explicitly suggested as an outcome, however the emergence of new ventures (alliances etc.) was noted. This will likely need to be complemented by another line of evidence.
Identification of R&D gaps and new research needs andelements of governance structure Number of new R&D projects related demonstrations
  • Elements of this will likely be picked up in reporting, however other lines of enquiry will likely also be necessary.
Trades persons and companies build skill capacity Development of curriculum in universities and colleges

Development of qualified tradesmen (to service H2 applications)
  • This is indirectly addressed in the reporting requirements as part of a menu of potential outcomes. It would almost certainly need a more direct line of enquiry.
Companies, universities, and community colleges build technology capacity Development of curriculum in universities and colleges

Analysis of extent of project success (exceeded, met, met most, met some of target)
  • Not explicitly addressed.
  • Project success will be at least partially assessed by project reporting. It will definitely require complementary methods.
Adoption of technology beyond funded firms (first purchase,spin-off technologies) Number of unfunded firms and projects that become involved in a demonstration
  • No reporting is required of unfunded applicants. Follow-up will require a separate survey.
Strengthened industrial capability in Canada and build community critical mass Number of firms that join a cluster

Growth or size and number of firms within a cluster
  • The project files will serve as a base for follow-up (See Annex E for description of proposed social networking approach).
Relative cost-effectiveness of different alternatives is improved Cost / KWH for different energy alternatives

Unit costs declining
  • Project reporting requirements ask for this type of reporting. This may need other lines of enquiry for verification.
An integrated supply of H2 technologies and H2 compatible technologies is developed Breadth of participation (from supplier to end-user) in demonstration projects

Number of different types of participants in demonstration projects
  • The project files clearly show the participants and partners. These can be categorized (see section 3.2.2) and analyzed for their relative diversity.
  • The 'outreach' components of project reporting requirements are left a bit vague in terms of specific target groups and expected numbers. This will likely require some complementary follow-up.

In summary, TPCh2EA project files appearto establish a reasonable base for project tracking – as anticipated bythe 2003 RMAF. Some of the expected results are stated in terms which might bea little bit general. (e.g. A couple of descriptions of expected projectresults in form h2EA-1 makestatements like 'increased knowledge' and 'expanded awareness' withoutoperationally defining these terms to an extent where specific expectations ortargets could be set.

Management may wish to review the need to provide specific targets as partof future approvals. After all, if a project is essentially for demonstrationto increase awareness, the question of "how much awareness are webuying?" needs to be addressed.) Notwithstanding this potential gap, thefiles appear to set out reporting requirements which, if reasonably followed,would at minimum support and complement more detailed lines of enquiry interms of tracking h2EAresults.


1 While all of the above were contacted, not everyone on this list participated in an interview. In order to preserve sufficient anonymity, only the broader contact list is shown here.

2 The snowball survey approach would ask respondents for others they have had meaningful contact with in the development and implementation of their hydrogen project. This could apply to groups who did not receive assistance as well as those who did. The results of the first round of this questioning (i.e. the contacts) can also be contacted for their contacts, as well as for progress and perceptions, creating a 'snowball' effect. (See Figure E-1 for a graphic illustration.) For our purposes, we would likely limit the follow-up surveying so as to stay within reasonable resourcing limits.