Evaluation of the Policy on Title to Intellectual Property Arising Under Crown Procurement Contracts

Final Report

December 2010

Tabled and approved at the Departmental Evaluation Committee on March 31, 2011

The Evaluation of the Policy on Title to Intellectual Property Arising Under Crown Procurement Contracts is a joint Industry Canada—Treasury Board Secretariat Evaluation

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Principles and Scope
  • 1.2 Monitoring and Evaluation
  • 1.3 Reported Activity prior to the Revision of the Data Collection System
  • 1.4 Reported Activity under the Revised Data Collection System
  • 1.5 Assessment of Federal IP Management by the Office of the Auditor General
• 2.0 Methodology and Approach
  • 2.1 Objective
  • 2.2 Approach
  • 2.3 Evaluation Issues
  • 2.4 Data Collection Methods
  • 2.5 Data Quality and Limitations of the Methodology
• 3.0 Findings
  • 3.1 Relevance
  • 3.2 Performance (Effectiveness)
  • 3.3 Performance (Efficiency)
• 4.0 Conclusions and Recommendations
  • 4.1 Conclusions
• Management Response

Appendices (separate document)

(Note: Annexes are available via an Access to Information)

• Appendix A: Evaluation Framework
• Appendix B: Evaluation Advisory Committee Members
• Appendix C: Documents Reviewed
• Appendix D: Interview Participants
• Appendix E: Interview Guides
• Appendix F: Environmental Scan Summary
• Appendix G: Survey Summary
• Appendix H: Case Study Summary
• Appendix I: Detailed Comparison of the Policy with the Policy on Green Procurement
• Appendix J: Comparison of the Use of Exceptions among Departments
• Appendix K: Specific Areas Requiring Additional Support
• Appendix M: Estimates of Costs to Implement the Policy

Executive Summary

In 2010, a joint Industry Canada – Treasury Board of Canada Secretariat evaluation was performed on the relevance, effectiveness and efficiency of the Policy on Title to Intellectual Property Arising Under Crown Procurement Contracts (2000) ("the Policy").

Findings

Relevance

The evaluation determined that the Policy is consistent with Government of Canada principles, approaches, and priorities. The Policy is an appropriate means to meet its objectives and is not inconsistent with any other programs, policies, or legislation.

More specifically, procurement has been used as a lever to advance social and economic objectives in other policy initiatives such as the Industrial and Regional Benefits Policy, the Procurement Strategy for Aboriginal Business, and the Policy on Green Procurement. In this context, the Policy is an appropriate mechanism that provides a consistent approach across the federal government for IP ownership decisions. Canada's position of having ownership of IP default to the contractor is appropriate. Among international peers, the United States has a similar stance and other countries with opposite default positions (e.g. Australia) are moving toward the Canadian position.

Effectiveness

The Policy has been implemented in departments and incorporated into standard contracting processes. Procurement officers are aware of the Policy; data has been collected; and reports have been produced. However, processes in the four departments examined were insufficient in terms of documenting the justification for decisions to use exceptions and retain ownership of the IP. Overall, departmental processes lacked controls, documentation trails and validation steps.

In this context, IC and TBS have an opportunity to improve the monitoring of results and follow up on data abnormalities in order to ensure that the Policy has been implemented as intended across the whole of the Government of Canada. Despite the fact that the Policy's intent is for most contracts to vest IP ownership with contractors, a high percentage of the contracts have ended up with the Crown owning the IP.

The only source for information on IP ownership in contracts across government is the annual Government-Wide Procurement Reporting System IP Report (GWPRS) IP Report. According to the GWPRS IP Report, the Crown retained the IP in 59% of all 2009 contracts that were expected to generate IP. This was up from 53% in 2008. Closer examination revealed that some federal organizations had IP retention rates in 2009 that were greater than 90%. Unfortunately, it is not clear whether the high retention percentages reflected erroneous data, the nature of the work, or a failure to comply with the intent of the Policy. In addition, a review of contracts in selected departments found cases where the rationale for using a Crown IP ownership exception was not clear.

Overall, the Policy likely contributed to an increase in the commercialization of IP, but this cannot be measured from the data currently available. There is some evidence that the Policy increased the probability of contractors responding to Requests for Proposals (RFPs) when they could own the IP arising under the contract. A key benefit of the Policy has been to provide clarity to procurement officials and thus ensure that IP ownership issues are addressed early in the procurement process.

Efficiency

The Policy has likely produced economic benefits to Canadian society and operational benefits within government. While these benefits have not been measured to date and would be very difficult and costly to measure, they may have exceeded the operating costs which have been quite low.

Recommendations

1. To improve the monitoring and implementation of the Policy, IC and TBS should clarify roles and responsibilities with respect to the Policy within three months. Consistent with the Policy, monitoring should focus on cases where exceptions were invoked.
2. Once the roles and responsibilities have been clarified, the agreed upon department(s) should:
   1. Encourage departments to maintain appropriate documentation in contract files to justify IP ownership exceptions;
   2. Encourage departments to review the quality of data to be included in the GWPRS IP Report;
   3. Advise departments when they have reported high IP retention rates or the use of an exemption without Treasury Board approval;
   4. Share the results of the annual analysis of the GWPRS IP Report with departments through appropriate forums;
   5. Enhance the Implementation Guide to provide additional clarity; and
   6. Identify best practices used by departments and share these best practices through appropriate forums.
   7. Review IP retention rates one year after implementation of actions, and if a higher than expected use of exceptions still exists assess the appropriateness of their use and take appropriate action if required.

1.0 Introduction

This joint IC-TBS evaluation was performed to assess the relevance, effectiveness and efficiency of the Policy on Title to Intellectual Property Arising Under Crown Procurement Contracts (2000).

This section of the report describes the principles and scope of the Policy, monitoring and evaluation, activity to date, and the OAG's assessment of federal IP management.

The remainder of this report is organized as follows:

• Section 2 outlines the objectives of the evaluation; the approach, the methodology, data sources and limitations;
• Section 3 describes the key findings related to the evaluation issues of relevance and performance (effectiveness and efficiency); and
• Section 4 summarizes the study's conclusions and provides recommendations.

1.1 Principles and Scope

The Policy was approved in 2000 by the Treasury Board of Canada and sets out the following underlying principles:

• The Government of Canada's primary objective in entering into Crown Procurement contracts is to receive the deliverables contracted for, and to be able to use those deliverables along with any IP arising by the virtue of the contracts;
• The Government of Canada has an overall objective of promoting economic growth and job creation in Canada;
• The Government of Canada believes that commercial exploitation of IP contributes to economic growth and job creation;
• The Government of Canada believes that commercial exploitation of IP is best achieved by the private sector.¹

Accordingly, the Policy defaults ownership of IP to the contractor. However, specific exceptions can be invoked to allow the Crown to retain ownership of the IP in certain situations such as national security, the generation of knowledge for public dissemination of information, copyrightable material, etc. In addition, there may be other circumstances where Crown ownership of the IP is warranted. In these cases, departments are required to obtain prior approval from the Treasury Board of Canada.

The Policy applies to IP that arises by virtue of a Crown procurement contract and that is developed by the contractor. It does not apply to collaborative, contribution or partnership agreements. The Policy does not apply to trade-marks, trade-names or personal information as defined under the Privacy Act.

The Policy applies to all departments, unless specifically exempted by an act of Parliament or by the Treasury Board of Canada. According to the Implementation Guide, the Canadian Food Inspection Agency and the Canada Revenue Agency have been exempted.

1.2 Monitoring and Evaluation

IC and TBS are jointly charged with monitoring and evaluating the Policy. In 2002, an evaluation framework was developed for the Policy. In the course of developing the framework, it became clear that there would be difficulty in obtaining data to support an evaluation. Consequently, it was recommended that a baseline assessment of data availability be undertaken.

The Baseline Assessment Report was prepared in 2004 and it concluded that:

• There was insufficient evidence available to evaluate the commercialization impacts;
• Support varied for the implementation of the Policy; and
• There was a wide variance in the application of the Policy.

The data issues identified in this report led IC to establish an interdepartmental working group to prepare an action plan to address these limitations. IC submitted the Action Plan, and the findings of the Baseline report to the Treasury Board in 2007.

As a result of the Interdepartmental Action Plan, the data collection system was modified to add a field to specify those contracts under which "no IP" was expected to be created. Other modifications included the addition of a field to identify contracts by organizations that were not subject to the Policy.

1.3 Reported Activity prior to the Revision of the Data Collection System

The following table shows the allocation of IP ownership in procurement contracts according to GWPRS data for the period from 2001 to 2007.

1.4 Reported Activity under the Revised Data Collection System

The first set of contracting data collected under the revised system was compiled for 2008 and made available in September 2009. For the most recent year, 78 different departments and agencies reported contracts over $25K that were expected to generate IP. Overall, these contracts represented about 17% of all contracts, and contracts in which the Crown claimed an IP ownership exception represented an even smaller percent of the total number of contracts (10%).

According to the GWPRS IP Report, all of the exception codes in the Policy were used in 2008 and 2009. The following table shows the number of occurrences for each type of exception.

Despite the fact that Treasury Board exemptions (8.1) were reported in GWPRS, the Treasury Board of Canada did not actually grant any exemptions during this period.

1.5 Assessment of Federal IP Management by the Office of the Auditor General

In Spring 2009, the OAG released its report and included a chapter on the management of federal IP.² The OAG had a number of key findings:

• The federal government was not in a position to know whether the objective of the Policy was being met;
• The federal government did not know how much IP had been generated in the course of contracted work;
• The National Research Council Canada, Health Canada, and Fisheries and Oceans Canada had not adequately identified or reported whether work performed under Crown procurement contract was likely to generate IP. They had also not assessed whether the Policy had been implemented and applied within their respective organizations;
• Although the Policy stated that IP should rest with the Crown only in exceptional cases, IP ownership was retained by the Crown in over half the contracts that the OAG reviewed at Health Canada and Fisheries and Oceans Canada. Moreover, the IP was also often retained without clear justification; and
• IC and TBS had not adequately fulfilled their obligations to monitor the application of the Policy (with a focus on cases where exceptions were involved) and to evaluate the Policy.

In response to the chapter, departments presented their management action plans to the Standing Committee on Public Accounts in the fall of 2009. As part of the Joint IC-TBS Management Action Plan, IC moved the planned evaluation of the Policy forward from 2011 to 2010.

¹ 2000 Policy on Title to Intellectual Property Arising under Crown Procurement Contracts (Return to reference 1)

² Chapter 2 "Intellectual Property" of the Spring 2009 Report of the Auditor General of Canada (Return to reference 2)

2.0 Methodology and Approach

This section describes the objective of the evaluation, the overall approach, the specific questions that were addressed, the data collection methods used, and the limitations of the methodology.

2.1 Objective

The objective of this evaluation was to ascertain the extent to which the Policy's intent has been met, the factors that affected the process of implementation and management of the Policy, and the impacts, both intended and unintended, upon the contractors and the Crown.³

2.2 Approach

The Audit and Evaluation Branch engaged TDV Global to conduct the evaluation. An Evaluation Advisory Committee made up of IC representatives and key stakeholders provided input and guidance during the course of the evaluation. The list of members appears in Appendix A.

2.3 Evaluation Issues

Appendix B contains the Evaluation Framework and includes the indicators, data sources and methodology for each of the evaluation issues. The evaluation addressed the following questions:

Relevance

• Is the Policy consistent with the Government's principles and approach to intellectual property ownership in the context of Crown procurement?
• Is the Policy an appropriate means of achieving the objective?

Performance (Effectiveness)

• Is the Policy being implemented as intended (according to the Underlying Principles section of the Implementation Guide)?
• How have the departments selected for this evaluation implemented the Policy and what are the opportunities for improvement?
• What are the outcomes of the Policy?

Performance (Efficiency)

• Is the cost to administer the Policy commensurate with its benefits?

2.4 Data Collection Methods

The evaluation was conducted in accordance with the Policy on Evaluation using multiple lines of evidence. This included document review, a database review, interviews, an environmental scan, a survey and case studies.

2.4.1 Document Review

A large number of key documents were reviewed. They included policy documents, Treasury Board submissions, implementation guides, Speeches from the Throne, TBS circulars on the Policy, departmental materials related to the Policy, the outline for the planned e-learning course, Office of the Auditor-General reports, the IC-TBS Management Action Plan, the Baseline Assessment Report, IC's review of IP contracts, the Standing Committee on Public Accounts report, and the Government Response. A summary of the findings from each document was maintained. A review of key Intranet and Internet websites was also included in this task. The complete listing of documents is included in Appendix C.

2.4.2 Database Review

A review of the Government-Wide Procurement Reporting System (GWPRS) information was conducted to collect key statistics for the evaluation. A key source for summary information was the annual GWPRS IP Report. GWPRS data was also used to identify contracts that were examined in more detail in the Case Studies.

2.4.3 Interviews

Interviews were conducted with officials involved in the Policy, procurement officers and contracting authorities, and stakeholders from outside the government (including industry associations, contractors and universities). The list of individuals interviewed is outlined below and further detailed in Appendix D. Information gathered from these interviewees also provided evidence for the Case Studies and the Environmental Scan.

Interview guides were provided to each interviewee prior to the interview. The interview guides are included in Appendix E. Interviews were conducted in the interviewee's preferred official language and each interview was documented.

2.4.4 Environmental Scan

The evaluation team reviewed how IP arising from government contracts is handled in the United States, the United Kingdom, Australia, and in the following Canadian provinces: British Columbia, Alberta, Ontario, and Quebec. The Environmental Scan was used to compare the approach used by the Government of Canada to comparable jurisdictions. The results of the Environmental Scan are included as Appendix F.

2.4.5 Survey

A short electronic survey was sent to departmental contracting officers, project authorities and IP specialists. The objective of the survey was to gather information on the perceived effectiveness of the Policy, its tools, the process to identify the presence of IP in contracts, and their level of understanding of the Policy itself.

The survey was administered in September 2010 using an on-line tool. Overall, 441 individuals responded out of the 1,525 who received the survey for an overall response rate of 29%. The survey was anonymous in order to encourage participation. The profile of the participants completing the survey showed wide participation from many departments. The largest grouping of respondents was procurement officers and those that manage procurement units. The results were analyzed and incorporated into the findings. No responses were directly attributed to individuals or departments. A summary of the survey findings is included as Appendix G.

2.4.6 Case Studies

Two types of case studies were used:

• Department Case Studies (four); including broad review of 95 contracts, and
• Contract Case Studies (seven contracts).

Four departments were selected for a review of their implementation of the Policy. The departments were selected to include both science-based departments (with research and development contracts) and non-science-based departments.

The following table outlines the characteristics of the departments selected based on 2008 GWPRS data, which was the most recent data available at the time of selection:

Altogether the selected departments represented 18% of the total number of Crown IP ownership exceptions in 2008.

The Department Case studies examined how the selected departments implemented the Policy (e.g. what training and internal policies they developed and implemented). The Head of Procurement for each department was interviewed and documents such as training aids, policies and procedures were reviewed. A stratified sample of 20 to 30 contracts from each of four selected departments was examined to determine how IP was identified and whether the exceptions were used appropriately. This was based on a high-level examination of the contracted activities. Contracting authorities were also interviewed.

From the 95 contracts that were examined in the Department Case Studies, seven contracts were randomly selected for a more detailed examination to determine how IP ownership decisions were established and recorded. For each of the seven contracts examined in detail, the contracting authority and contractor were interviewed, and the contract file was reviewed.

The results of the Department and Contract Case Studies are included as Appendix H.

2.5 Data Quality and Limitations of the Methodology

This evaluation did not focus on actual commercialization of IP that may have resulted through the Policy, but instead on how the Policy has been applied and implemented in departments. In interviews and documents, it became clear there was little data available to allow attribution of economic benefits directly to the Policy. Commercialization of IP generated through Crown procurement contracts is not systematically tracked.

The evaluation was also affected by the following limitations for each line of evidence:

• Document Review: Financial costs that were not available for this evaluation included the costs to modify GWPRS to include data on IP ownership decisions and the costs to develop the online learning application (still in testing phase). These costs were considered to be relatively minor and so the evaluation was not significantly impacted by their unavailability.
• Database Review: Originally, GWPRS did not contain an indicator of whether or not a contract was expected to generate IP. Instead, contracts that were not expected to generate IP presumably were included in the "other" category. In response to the Baseline Assessment Report, GWPRS was modified to address this deficiency. Starting in 2008, data in the system included an additional code to identify if IP was expected to be created under a contract. As such, 2002–2007 data cannot be compared directly with 2008–2009 data, and this restricted the ability to identify trends.

  In addition, it is important to note that data in GWPRS captures expectations at the time of issuing of the contract. It does not contain information on whether or not IP was actually developed. Therefore, there is likely a gap between the reported data and the actual results.

• Environmental Scan: The Environmental Scan was conducted from literature review documents and Internet searches. It was not possible to validate the information through discussions with officials in the studied jurisdictions. This limitation was mitigated by deferring to official documents as much as possible. Consequently, the Environmental Scan should be seen as an indication of how Canada compares to some comparable jurisdictions, rather than a complete scholarly review of the global situation.
• Survey: The online survey likely suffered from some selection bias. Survey participants had to choose to respond. It is reasonable to assume that persons unfamiliar with the Policy would be less likely to complete the survey. To reduce selection bias and include a wider representation of the procurement community, the e-mail invitation for the survey did not require knowledge of the Policy to participate.

  While the anonymity of the survey likely increased response rates and prompted frank answers, it also meant that it was not possible to follow-up with individuals and pose supplementary questions based on their responses. In particular, a small percentage of respondents noted that they had experienced situations where their department had wanted to retain the IP in a contract but there were no exceptions in the Policy that permitted them to do so. In terms of evaluating appropriateness and completeness of the exceptions, it would have been useful to have been able to investigate these cases further.

• Case Studies: Typically, a contract would either contain a clause pertaining to IP ownership or not. However, no explanatory notes were found in the files explaining or supporting the decision to invoke an exception. This deficiency made it impossible to explore the appropriateness for the use of exceptions.

  The initial intent was to examine about ten contracts in detail (three or four per department). Unfortunately, GWPRS does not track contract numbers so the evaluation team experienced delays in all but one department in obtaining contracts without a reference contract number. In one department, these challenges made it impossible to review their contracts. As a result, only seven contracts were examined in detail.

  The Case Studies did not examine practices in either the Correctional Service of Canada or Indian and Northern Affairs Canada. According to the centrally reported data for 2009, these two departments invoked Crown ownership exceptions on a very high percentage of their IP generating contracts. They also constituted a large share of the total number of exceptions for the Government of Canada.

³ TB 2000 Policy on Title to IP Arising Under Crown Procurement Contracts, Evaluation Framework – Final Report, Performance Management Network, March 2001. (Return to reference 3)

3.0 Findings

3.1 Relevance

This section of the evaluation contains an assessment of the relevance of the Policy. It evaluates the extent to which the Policy is consistent with the Government of Canada's approach to IP and whether it is an appropriate means of achieving its objectives.

3.1.1: Is the Policy consistent with the government's principles and approach to intellectual property ownership in the context of Crown Procurement?

This evaluation question was addressed through three areas of examination:

• Is the Policy consistent with the principles of the Government of Canada regarding IP ownership?
• Are the principles in the Policy consistent with Government of Canada priorities?
• Is the Policy consistent with other Government of Canada approaches to Crown Procurement?

Consistency with Government of Canada Principles Regarding IP Ownership

The Policy and its Implementation Guide both present a clear position, which can be summarized by the stated principle: "The Government of Canada believes that Commercial Exploitation of Intellectual Property is best achieved by the private sector." The Policy further states that this is in support of "… an overall objective of promoting economic growth and job creation in Canada."

The TBS Contract Policy Notice of July 17, 2000 that advises departments of revisions to the Policy reiterates these objectives and underlying principles.⁴ The Policy itself also refers to its consistency with the Contracting Policy principles. The Contracting Policy states in Section 2c that "Government contracting shall be conducted in a manner that will support long-term industrial and regional development and other appropriate national objectives."⁵

The Policy contains another important principle. According to Section 5.2, the Crown "should have the unlimited rights to use the Foreground⁶ it licenses through Crown Procurement Contracts". This is consistent with the fundamental principle in the Contracting Policy that "Government contracting shall be conducted in a manner that will ensure the pre-eminence of operational requirements."

At a practical level, procurement and policy officials were asked if there were any tensions or inconsistencies between the Policy and other government principles or approaches. The interviews did not find inconsistencies with other government principles or approaches.

An analysis of the Patent Act, the Trade-marks Act, and the Public Servants Invention Act, and the Copyright Act revealed that the Policy is consistent with legislation.

Consistency with Government of Canada Priorities

Speeches from the Throne were examined and no inconsistencies were found with the principles outlined in the Policy. The following table includes quotes from key documents that demonstrate the importance that the Government of Canada places on research and innovation, and the federal government's commitment to economic growth:

A review of strategic and policy documents from IC and TBS confirmed the consistency of the Policy with government priorities.

Consistency with Other Approaches to Crown Procurement

Other Crown procurement policies are also used for the purposes of achieving specific benefits which align to government priorities. Examples include the Policy on Green Procurement, the Procurement Strategy for Aboriginal Businesses (PSAB), and the Industrial and Regional Benefits Policy.

• The Policy on Green Procurement states: "It is the objective of this policy to advance the protection of the environment and support sustainable development by integrating environmental performance considerations into the procurement decision-making process."
• The Contracting Policy Notice on the PSAB (1997-6) states: "The objective of the PSAB policy is to assist Aboriginal business development by increasing Aboriginal business participation in the awarding of contracts by federal departments and agencies. The PSAB is intended to make use of the government's purchasing system to help Aboriginal businesses develop and grow."
• The Industrial and Regional Benefits Policy states: "The IRB Policy provides the framework for using federal defence procurement to lever long-term industrial and regional development within Canada."

Thus, the leveraging of procurement practices to create additional social and economic benefits is consistent with similar approaches in the Government of Canada.

3.1.2: Is the Policy an appropriate means of achieving the objective?

Objectives Sought

The Policy has the objective to "increase the potential for the commercialization of IP created by contractors in the course of their work under Crown procurement contracts." However, it recognizes that the primary objective of Crown procurement contracts is to receive the deliverables and to be able to use these deliverables (and any IP that arises from these contracts) for Government of Canada activities. Therefore, the Policy must balance government, contractor and public interests. The Policy was also developed to provide greater clarity concerning exceptions and to improve reporting and monitoring.

Means Chosen and Possible Alternatives

The Policy was approved by the Treasury Board of Canada, and monitored jointly by IC and TBS. An Implementation Guide was developed by IC using input from PWGSC, IC, TBS, the Treasury Board Advisory Committee on Contracting and the Federal Partners in Technology Transfer. The Policy mandates Deputy Heads to implement the Policy in their departments and agencies but does not mandate performance reporting beyond the collection of the statistics that can be found in the GWPRS IP Report.

There were several alternatives available in 2000 to achieve the objectives. Instead of developing a policy, the Government of Canada could have pursued an Act of Parliament or introduced regulations. Alternatively, it could have issued guidance in various formats. Examples from other countries help to illustrate potential approaches that were available:

1. Laws and Regulations: IP ownership in the United States is addressed by a mixture of legislation and regulation. The Bayh-Dole Act was enacted in 1980 to address IP ownership arising from government-funded research for small-to-medium enterprises and academia. In addition, the Government Services Administration, Department of Defense, and National Aeronautics and Space Administration jointly administer the Federal Acquisition Regulations, which were issued under the authority of the Office of Federal Procurement Policy Act of 1974.
2. Guidelines or General Guidance: The United Kingdom provides guidelines such as the "Intellectual Property in Government Research Contracts Guidelines for Public Sector Purchasers of Research and Research Providers" issued by the Patent Office. These guidelines include a Policy Statement and "guidance" but they do not carry a strongly worded mandate.In Australia, certain provinces in Canada and most US state governments, there is no common government-funded approach to IP ownership arising from procurement contracts. Instead, general guidance is provided that advises against IP ownership by the government unless necessary, but it permits departments, ministries and agencies to develop their own policies.
3. Case-by-Case Approach: Ontario does not have a consistent policy that applies to all ministries. Major procurements are handled on a case-by-case basis. Similarly, IP is managed in British Columbia by a centralized IP Program which reviews requests to license Crown owned IP on a case-by-case basis.⁷

Appropriateness of Policy

The following indicators were used to assess whether the Policy was appropriate:

• Do internal stakeholders perceive the Policy as appropriate?
• How does the Policy compare to other government procurement approaches (e.g. Green Procurement)?
• How does the Policy compare to procurement and IP ownership approaches used in other countries or provinces?

Perceptions of Stakeholders

Virtually all interviewees felt that the Policy was a reasonable tool to use that did not produce an unnecessary burden on the implementing departments. Heads of Procurement found that the Policy was useful to their operations and did not disrupt the contracting process. They said that legislation would be too prescriptive and cumbersome, while a guideline would be too lax. The Policy did not place a noticeable financial or administrative burden on departments for implementing, managing or reporting. In each of the four departments studied in the Case Studies, no significant implementation expenses or administrative delays resulting from the Policy were noted by department officials.

Policy specialists in IC felt that the use of a Treasury Board approved policy provided support in influencing the behaviour of other departments, particularly in regard to monitoring of the use of Crown ownership exceptions. The Policy also leveraged the reporting capabilities of PWGSC.

Comparison to Other Government of Canada Approaches

When assessing policy options, it is important to note that TBS and Public Service Human Resource Agency of Canada launched a broad review of Treasury Board management policies and issued the Foundation Framework for Treasury Board Policies in 2005. The Framework defines the objectives of policies, frameworks, directives, standards and guidelines. According to this document, policies "… explain what Deputy Heads and their officials are expected to achieve" whereas directives "… explain how Deputy Heads and their officials must meet the policy objective." Directives are mandatory and are directed at functional specialists and managers.

The Policy is directed to Deputy Heads. It explains what is expected (that ownership vest with contractors as a default) but it also informs how it is to be obtained (e.g. it provides exceptions and information on licensing). As such, the Policy does not adhere to the new definition of a policy. Instead, it more closely resembles the new definition of a directive.

Comparison with the Policy on Green Procurement

Although both policies were approved by the Treasury Board of Canada, the Policy is co-administered by IC and TBS while the Policy on Green Procurement is administered by PWGSC with the support of other departments. Despite these differences, both policies:

• Use government procurement to further a policy objective while, at the same time, seek to ensure value-for-money;
• Establish core principles; and
• Place responsibilities on Deputy Heads.

A detailed analysis of the similarities and differences between the two policies appears in Appendix I.

While both policies seek to achieve similar objectives, their approaches are quite different. First, the Policy on Green Procurement focuses on an outcome but it is not prescriptive on how that outcome is achieved. In contrast, the Policy establishes very specific requirements (including detailing the specific exceptions that can be used) but it sets out a broad final outcome. In addition, the Policy also has more limited reporting requirements. The Policy on Green Procurement has formal reporting requirements that require departments to develop metrics and include these metrics in their Departmental Performance Reports. If the Policy were to follow this example, then there would be greater visibility into these kinds of procurement decisions and close attention would likely be paid to the quality of the data.

Comparison with Other Countries and Jurisdictions

The Environmental Scan examined IP ownership practices for government contracting in Australia, the United Kingdom, the United States, and Canadian provinces (see Appendix F). The Environmental Scan compared the default policy position for ownership and the mechanisms used for implementation.

Overall, the policy direction taken by Canada is different from the default position of many of the Provinces (i.e. Crown ownership) and two other countries (the United Kingdom and Australia), but it is similar to that of the United States. However, all countries and jurisdictions that default to government ownership for IP also encourage their contracting authorities to consider whether or not they really need to own the IP. Most governments also state that they believe getting IP into the hands of the private sector is beneficial to economic growth. Among the jurisdictions that follow a different approach, Australia is currently moving toward a "contractor owned IP" default position for information and communications technology contracts.⁸ Similarly, Ontario is in the process of revisiting its approach. Recently the province has entertained the possibility of developing a policy similar to that of the federal government.⁹ In this context, the Government of Canada position of granting default ownership of IP to contractors (except when an exception is invoked) appears to be reasonable. Similarly, the use of a whole-of-government policy (rather than a statute or regulation) appears in line with approaches taken by other jurisdictions.

⁴ Contract Policy Notice 2000-2, File No. 3800-007-001, paragraph 12. (Return to reference 4)

⁵ Contracting Policy Section 2c www.tbs-sct.gc.ca. (Return to reference 5)

⁶ Foreground is defined as all Intellectual Property first conceived, developed, produced or reduced to practice as part of the work under a Crown Procurement Contract. Policy on Title to Intellectual Property Arising Under Crown Procurement Contracts. Section 4. (Return to reference 6)

⁷ Core Policy Manual Sections 6.3.4(f), Government of British Columbia. (Return to reference 7)

⁸ IP Intellectual Property Principles for Australian Government Agencies Section 8.(a) revised October 2010. (Return to reference 8)

⁹ Presentation to ITAC, May 27, 2010, Marian Macdonald, ADM Supply Chain Management, Government of Ontario. (Return to reference 9)

3.2 Performance (Effectiveness)

Effectiveness is the extent to which a program (or policy implementation) is achieving expected outcomes. In examining the Policy's effectiveness, three evaluation questions were considered:

• Is the Policy being implemented as intended (according to the Underlying Principles section of the Implementation Guide)?
• How have the departments selected for this evaluation implemented the Policy and what are the opportunities for improvement?
• What are the outcomes of the Policy?

3.2.1: Is the Policy being implemented as intended?

The expectations for implementation of the Policy were compared with the day-to-day implementation in departments. Expectations concerning the monitoring of the Policy by IC and TBS were also reviewed.

3.2.1.1 Implementation by Departments

Intended Implementation

The intention of the Policy was that all departments would apply the Policy in their procurement activities and modify their reporting systems. The following specific requirements were found in the Policy:

• Departments were to include a reference to IP ownership in RFPs and sole source procurements;
• Contracting authorities were responsible for ensuring that their standard clauses reflected the requirements of the Policy;
• Deputy Heads were expected to ensure sufficient resources for training and the orientation of personnel and to ensure that reporting requirements were implemented; and
• Departments were expected to modify their reporting systems to capture IP information.

Observed Implementation

The following areas were examined when assessing whether the Policy was being implemented as intended in departments:

• General application of the Policy including reporting, the consideration of IP in the contracting process, and departmental guidance;
• Appropriateness of the use of exceptions; and
• Awareness of the Policy and training.

General Application

From the GWPRS IP Reports, it can be seen that all departments do report on their procurement contracts over $25,000. Their ability to report also indicates that departments have modified their reporting systems to capture IP information which they now submit to PWGSC.

The evidence suggests that IP was considered in all departments that were studied in detail in this evaluation. Two departments had specific procedures in place to provide a specialist review of IP ownership within the contract or RFP preparation activities. One of these procedures was used throughout the entire department, whereas the other was used locally in one organization within the department. In all other situations, the IP determination was prescribed as part of the routine discussions that occur between the project authority and the contracting authority. However, there is no guarantee that this happened in all cases or that these discussions were informed by the spirit and content of the Policy.

Departmental guidance mostly consisted of copying portions of the Implementation Guide or extracting content and developing checklists or components of departmental training materials. There was no indication that any departments suffered from any delays in the development of their departmental guidance although the Case Studies and interviews did not focus on the early years of the new Policy. The Implementation Guide was complete in that it addressed all clauses of the Policy, and included sample contract clauses and frequently asked questions.

Appropriateness of Use of Exceptions

All contractor and academia interviewees felt the exceptions in the Policy were comprehensive and appropriate. However, as the interviewees deal with multiple departments, they noted that not all departments applied the Policy in a consistent manner.

In addition, one contractor expressed the strong belief that procurement officials are not implementing the Policy in its intended spirit. He believed that because the assignment of IP rights to the contractor might present some risks, contracting officials and project authorities have often retained Crown ownership as the safest course of action. The contractor also believed that the rigor in challenging the use of exceptions has been insufficient. This view was supported by one high tech industry association and two interviewees in academia.

The survey of the procurement community confirmed that differences of opinion between contractors and departments have occurred. About 12% of the survey respondents (49 of 412) had experienced a situation where the contractor disagreed with the decision to use an exception and retain ownership of the IP.

Overall, procurement officials believed that the list of exceptions in the Policy was comprehensive. In interviews, departmental officials could not think of any instances where there had not been a suitable exception that would meet their needs.

The Use of Exceptions at the Aggregate Level

According to the data in the GWPRS IP Report, the majority of contracts across the Government of Canada that were expected to generate IP did not actually vest the ownership of the IP with the contractor. In 2008, the Crown retained IP ownership in 53% of these kinds of contracts. In 2009, the figure rose to 59%.

There are two reasons for considering the GWPRS data to be indicative of levels of compliance with the Policy rather than an authoritative source for assessing compliance.

1. Data Entry Errors. There is some evidence that the data being entered into GWPRS may not reflect the actual decisions made by procurement officers.

   In 2010, IC conducted a review of the 17 contracts over $25K in 2008 in which it had claimed a Crown ownership exception. In 18% of these contracts (3/17), the review found data entry errors. None of these errors posed a risk to the Crown.

   The Case Studies found that the figures reported in GWPRS did not match the information in the actual contracts in 22% of the cases. While the number of cases of over-counting of Crown ownership was about equal to the number of cases of under-counting, it is possible that GWPRS data could have overstated the number of Crown ownership exceptions in some departments.

2. Appropriate Use of Exceptions May Generate High IP Retention Rates. While the intent of the Policy is to have IP ownership vest in a majority of cases with the contractor, there is the possibility that a proper application of exceptions to protect Crown interests could, in theory, result in the Crown retaining the IP in a majority of cases.

   Unfortunately, the appropriateness of the use of exceptions in individual contracts could not be evaluated because detailed documentation justifying IP ownership decisions was absent in all contract files that were reviewed for the departmental Case Studies. The lack of supporting documentation meant that it was impossible to review or validate the reasoning behind IP ownership decisions.

The Use of Exceptions at the Departmental Level

A more detailed analysis of the use of exceptions revealed a wide range of IP retention rates across the federal government. Appendix J contains calculations for the use of exceptions among departments with more than ten IP-generating contracts in 2008 and 2009.

According to the 2009 GWPRS data, there were six departments that had retention rates higher than 90%. These findings are a concern in terms of adherence to the Policy. If these six departments are excluded from aggregate totals, then the aggregate IP retention rate for the Government of Canada drops from 59% to 45%. In other words, IP developed in the course of a Crown procurement contract would, in most cases, vest with the contractor in accordance with the intent of the Policy.

A similar pattern holds for the 2008 GWPRS data. There were three departments that invoked Crown ownership exceptions in more than 90% of their IP related contracts. If these three departments of concern are excluded from Government of Canada totals, then the aggregate IP retention rate for the federal government drops from 51% to 41%.

It is important to note that, in some cases, the use of an exception may have been appropriate. For example, about half of the exceptions reported by the Canada School of Public Service concerned copyright. Given the large volume of training materials that are developed, these exceptions may be justified. However, the high rates of Crown ownership are a source for concern. (Appendix J includes the detailed breakdown of which exceptions were used by the six departments in 2009.)

Awareness and Training

Awareness was high within the procurement community as demonstrated by the fact that 79% of survey respondents reported that they were aware of the Policy.

About 57% of respondents reported having taken training on procurement that included a section on the Policy. Their training was mostly from either the Canada School of Public Service or their department.

Despite the high self-reported levels of awareness, the Case Studies found data errors and misapplications of the exceptions. This suggested that the Policy was not always well understood. In particular, there were contracts which included information subject to privacy restrictions (and hence exempt from the Policy), cases where department exceptions were misapplied, and contracts where national security was invoked when it was difficult to believe that national security was an issue. Despite the fact that the Policy explicitly requires written confirmation from the contractor for the use of one particular exception, no such documentation was found in any files studied where this exception had been claimed. These errors and omissions suggest that although training has been available and has been given, issues remain.

The results of the survey support these findings. About 42% of the respondents reported that they had problems at some point in implementing the Policy. About 17% also said they had experienced problems understanding if the exception had been applied properly.

Finally, the interviews also suggested additional training and clarification may be required. Two procurement officers expressed a concern that defaulting ownership of IP to contractors could place departments in a position where they might be required to contract follow-on work to the original contractor because that contractor would own the associated IP. However, these concerns seem to indicate an incomplete understanding of the licenses that the Crown retains when granting IP to the contractor. Policy authorities noted that Exception 6.4.3 in the Policy protects the Crown from this occurring. In addition, they noted that the Standard Clauses provide usage rights to the Crown so that a department would not be restricted to the original contractor for future work.

3.2.1.2 Monitoring of the Policy by IC and TBS

Intended Implementation

IC and TBS were responsible for monitoring the application of the Policy and were expected to focus on cases where exceptions were invoked (Section 11.1). It can be inferred that this monitoring role should have examined why the exceptions were being invoked in order to ensure that the spirit of the Policy was respected.

Observed Implementation

Both IC and TBS have individuals whose responsibilities include the Policy. Both IC and TBS have received the GWPRS IP Report annually. However, from 2000 until recently, there was no evidence of proactive follow-up on these reports. There was no evidence of discussions with the Heads of Procurement regarding the reported use of Treasury Board exemptions or discussions with departments with high IP retention rates to determine whether the reported numbers were the result of erroneous data, the nature of the work, or a failure to comply with the Policy.

On the positive side, both IC and TBS have, at times, played an active role in managing the Policy. The Baseline Assessment identified data issues that occurred during the first four years of implementation and data collection processes were improved.¹⁰ In 2010, IC conducted a review of its own contracting files where a Crown IP ownership exception was used, and encouraged other departments to do the same. Similarly, IC shared the results of the 2009 GWPRS IP Report with the Assistant Deputy Minister Committee on Science and Technology, and TBS shared the findings with the Treasury Board Advisory Committee on Contracts in order to build awareness and encourage compliance.

Overall, the question of roles and responsibilities is poorly defined. The Policy made both IC and TBS responsible for monitoring and evaluation without identifying specific roles. In interviews, it was observed that both departments felt that the other needed to take a stronger role in monitoring the Policy.

Despite these challenges, the Implementation Guide and Contract Policy Notices were developed as required and e-mail records indicate that the effort has been collaborative with IC assuming the lead in content for training materials. As issues have emerged (e.g. the OAG Report), IC and TBS have worked together.

3.2.2: How have the departments selected for this evaluation implemented the Policy and what are the opportunities for improvement?

To identify opportunities for improvement, the evaluation team examined how the Policy has been implemented in selected departments. The evaluation team examined departmental policies and procedures, education and training materials, awareness activities, and contract files.

General Observations on the Implementation of the Policy in Departments

The Heads of Procurement in the four departments were well aware of the Policy. However, only one Head of Procurement identified the Policy as a high priority or concern. In one department, the Procurement Review Committee considered the IP assignment decision in its deliberations and there was an IP Office located within its contracting organization in order to ensure the consistent application of the Policy for all contracts over $25K. While one department had conducted an internal management review of its use of Crown IP exceptions, none of the departments had conducted formal audits or evaluations of its implementation of the Policy.

In terms of processes, departments ensured consistency through the use of standard IP clauses in contracts. Most departments used standard clauses that had been developed by PWGSC and departmental contracting templates that reflected the Policy. All four departments used checklists that included IP questions to be addressed with Project Authorities. These checklists were then used by the contracting authority to select the appropriate clause in the final contract. Of the four departments, only one department had a review of IP ownership decisions in its processes.

The Policy was mentioned in departmental procurement manuals and some documentation was available to support contracting authorities. In two departments, IP and the Policy occupied only two pages in a 160 page overview document on procurement. The Intranets for two of the departments studied had IP information available to assist project authorities. One department developed a full binder of material exclusively on the Policy and gave the workshop in 2001. This material was later used to prepare and periodically deliver a shorter training session on the Policy. In general, departments seemed to find the Implementation Guide and the Policy document self-explanatory so that a mention of its existence and links to the TBS webpage were deemed sufficient.

In general, the level of understanding of the Policy varied by department. From the interviews, it was determined that departments felt that additional clarification was required on certain types of contracts such as software IP and training materials. Differing opinions also existed on the limitations of Crown use when the contractor retained ownership. Some interviewees expressed the concern that exceptions were used too frequently and when not needed because departments did not fully understand the rights that licensing provides to the Crown.

In terms of future training, all departments will eventually have access to an online course on the Policy that is being developed by the Canada School of Public Service. Those that have seen the prototype of the course believe that it will be a useful tool once certain edits are made to the content. Two departments are trying to get additional training for staff.

Some interviewees also felt that their departments would benefit from a central IP resource for the Government of Canada to answer in-depth questions. This comment echoed comments received from survey participants when asked what could be done to improve implementation.

In the course of conducting the Case Studies, the evaluation team noted several specific aspects of the Policy that could lead to misunderstandings or inconsistent applications. They include the treatment of foreign contractors, personal information that falls under the Privacy Act, the definition of national security, suggested documentation for cases where a contractor is not interested in owning the IP, licensing and copyright, the treatment of IP that will be transferred in the future, copyright, and Treasury Board exemptions. These specific points and suggestions for clarification for the Implementation Guide can be found in Appendix K.

Challenges Concerning Discrepancies between Centrally Reported Data and Contract Files

The examination of how departments have implemented the Policy revealed noticeable differences between the data reported centrally in the GWPRS IP Report and the information in the contract files. The following table outlines the errors observed in a review of 65 contracts across three departments.¹¹

Overall, there were differences between the contract files and the information in GWPRS in about 22% of all contracts. In some cases, the exception that appeared in GWPRS differed from the exception in the contract file. In other cases, the difference was in the code that identified whether an exception had been invoked. Overall, the number of cases of over-counting of Crown ownership in GWPRS was about equal to the number of cases of under-counting.

It is important to recognize that there is not always a negative impact on either the department or the contractor if a contract does not match what is coded in GWPRS. This is particularly true when the wrong exception code is used. In these instances, the real impact is only in the accuracy of the GWPRS IP Reports. To be clear, it is the actual text of the contract that matters in terms of whether the Crown or a contractor owns the IP, not the coding in GWPRS.

Crown IP ownership exceptions were typically recorded before any contract tendering took place. In most departments, the decision was recorded by entering a code into a contracting information system (such as the Integrated Financial and Materiel System at IC). The contracting authority then took note of this in order to insert a Crown IP Ownership clause into the contract. This meant that the system for entering the IP ownership information that eventually flowed into GWPRS IP Report was separate from the actual contracting documentation.

This duplicate entry process created the potential for a clerk to enter the code for an exception when one was not intended (i.e., by mistake) or a different exception than intended (e.g. 6.3 rather than 6.2), or even fail to enter the exception at all. At IC, the failure to enter an exception code would generate a default contractor owned IP condition in the contract.

Challenges Concerning Potentially Inappropriate Applications of the Policy

A review of 95 contracts found that 13% of the contracts exhibited questionable uses of Crown IP ownership exceptions. An inappropriate retention of Crown ownership would constitute a barrier to commercialization. The following table outlines the potential errors observed across the four departments.

The 12 contracts of concern included:

• Two contracts that claimed a national security exception when there was no evidence of any national security issue in the contract;
• Two contracts that claimed an exception for generating knowledge for public dissemination but the nature of the reports was clearly intended for internal management purposes;
• Three contracts that claimed that the contractor had declined ownership of the IP. While the Policy clearly stipulates the contractor must declare this in writing, there were no records documenting that this had occurred; and
• Five contracts that did not identify which exception was being claimed.

The Case Studies also examined seven contracts in greater detail and interviewed key participants. In general, the use of exceptions in these contracts was appropriate and reasonable. However, there were decisions that were subject to interpretation:

• There was one example where similar work within the same department resulted in two different decisions being taken. In both instances, a contract was let for the use of computer programming support under very similar situations. In one, it was identified that there was no IP. In the other, IP was identified and an exception was invoked because the IP formed part of a larger computer system where the Crown already owned portions of the IP.
• In another instance, a contract identified that there was no IP created when the contract called for delivery of architectural drawings, documents and designs.

What was striking was the absence of supporting documentation for IP ownership decisions in all of the contracts reviewed. Among the four departments, not a single contract template asked for the rationale for why an exception was chosen. While one department included a review of the IP ownership decision as part of its contracting process, the other three departments simply noted the exception had been used. These practices are in sharp contrast to the procedures required for sole source contracts where a project authority is required to write a justification.

Having a justification on file would allow an independent reviewer to validate the appropriate use of the exception and encourage adherence to the Policy. It would allow managers to review staff work and identify root causes for the excessive use of an exception. As a result, departmental guidance could be improved to prevent reoccurrences and staff could be trained on the use of specific exceptions.

¹⁰ Baseline Assessment Report, 2004. (Return to reference 10)

¹¹ Once the sample of contracts was created for the Case Studies, the evaluation team initiated the process of matching these contracts with GWPRS data. Unfortunately, GWPRS does not retain contract numbers so significant effort was involved to complete this task. Given time constraints, contracts could not be matched to GWPRS records for one department. (Return to reference 11)

3.2 Performance (Effectiveness) (Continued)

3.2.3: What are the outcomes of the Policy?

The impact of the Policy on contractors and departments was assessed to determine the outcomes resulting from the Policy.

Impact on Contractors

The actual creation of IP that arose under Crown procurement contracts (as opposed to the expectation during the contracting process that IP would be created) was not tracked by departments. Industry commercialization of IP was also not tracked by departments because it was not required by the Policy. It was also seen as difficult and costly to measure. As a result, there were no sources of information that could guide the evaluation team on the extent of economic benefits generated as a result of the Policy.

The interviews found two cases of commercialization. One contractor asserted that his firm had developed a software package under a Crown Procurement contract and commercialized it along with other solution elements. This contractor also stated that there were other instances where his company was able to capitalize on IP. In addition, a non-government organization reported that it had produced a report for a department and posted this report on its web site.

It was suggested that the few instances where the IP arising from Crown Procurement contracts was actually used in commercial products and services could be explained by the fact that these kinds of contracts provide relatively few opportunities to produce exploitable IP. Industry and university interviewees confirmed that procurement was a minor source of IP compared to grants and contributions, and collaborative research.

Within government, most departments saw little economic impact from the Policy. None of the departments studied could identify an IP item that it had commercialized after invoking an exception.

A key benefit of the Policy was its positive impact on encouraging contractors to respond to Requests for Proposals. In the past, the Canadian Advanced Technology Alliance has asserted that allowing vendors to retain IP could result in an increased interest in bidding.¹² This sentiment was echoed in documents by the Australian Information Industry Association.¹³

Interviews with industry and academia suggested that if IP were not vested in contractors, then these stakeholders would be unwilling to supply their services to the Government of Canada. One academic interviewed even said it would be a "showstopper" and that there were definitely instances where IP ownership decisions influenced his institution's decision to bid. It should be noted that, in some of these cases, considerations other than commercialization were the main source of motivation for retaining the IP (e.g. the ability of academics to publish).

A complication for measuring the degree to which contractors would be more likely to respond was the absence of baselines and data. In addition, IP ownership was only one factor that contributed to the decision by a firm to bid on a contract. As a result, a definitive attribution of this impact to the Policy was not possible with the existing data.

No other jurisdiction had any metrics on the impact of their IP ownership policy on industry although all jurisdictions acknowledge that IP is an important enabler of industry development and that it should belong to those best enabled to exploit it for economic gain.

Impact on Departments

The biggest direct benefit of the Policy for departments was clarity for IP ownership decisions. Most interviewees felt that the Policy brought a certain level of rigor to decision-making. In particular, the Policy prompted discussions concerning the potential for generating IP and the ownership of the resulting IP early in the contracting process. The results from the survey supported this perception. The following figure shows that the category with the largest number of responses for perceived benefits was clarified IP ownership.

16. What benefits do you believe arise from the policy? (select all that apply)

[Description of Figure 1]

Of the 289 survey respondents for this question, 71% indicated that the Policy had clarified IP ownership. In contrast, 44% cited greater willingness of contractors to enter into contracts and 37% believed that the Policy contributed toward improved economic growth.

In terms of indirect benefits, the Policy has likely helped the Crown avoid legal costs. By ensuring that discussions on IP generation take place and by providing clear guidance on the assignment of IP owners, the Policy likely reduced the number of court cases that could have arisen from the contracting process. In interviews, several procurement officers and the representative from the Department of Justice expressed the opinion that, without the Policy, there would have been more legal challenges related to the ownership of IP. In the Case Studies, procurement officials provided examples of the legal ramifications of not properly addressing IP ownership at the time of contract.

The evaluation team also looked for other results from the Policy that may not have been articulated. No evidence was found of unintended negative outcomes.

Overall, the exceptions appeared to be comprehensive and appropriate. The interviewees felt that the Crown received the benefits expected, such as the protection of IP rights when a contract contributed to part of a larger project, or the protection of IP rights in the interests of national security. Most departmental officials could not think of any instances where there was not a suitable exception to meet their needs. However, there was a caveat. According to the survey, 18% of the respondents reported situations where their department had wanted to retain the IP but there was no exception that had allowed them to do so. Unfortunately, the nature of the survey and the absence of a unique identifier attached to individuals meant that it was not possible to investigate these responses further.

3.3 Performance (Efficiency)

An important aspect of assessing performance in an evaluation concerns the resources and costs used in relation to the development of the outputs and progress toward the expected outcomes.

3.3.1: Is the cost to administer the Policy commensurate with its benefits?

To determine the relative value of the Policy, the costs for the implementation of the Policy were identified and compared with the perceived benefits.

Policy Costs

The full costs of implementing the Policy have not been captured in a centralized fashion, which would have been costly to do. Since the Policy applies across the Government of Canada, all departments have spent resources but many of these costs were not available.

For a ten year period, it is estimated that the known costs for the Policy totalled about $600K (including salaries). These costs include the initial implementation of the Policy, monitoring, GWPRS systems modifications, the annual production of the GWPRS IP Report, a review of contracts in one department, the Baseline Assessment Report, and this evaluation. The operating costs of the Policy are expected to be about $31,500 per year going forward under the current monitoring arrangements.

A detailed breakdown of these cost estimates appears in Appendix M.

Relative Value

As outlined in Section 3.2.3, the benefits of the Policy included increased opportunities for commercialization, greater response rates for Requests for Proposals, increased clarity within the Government of Canada for IP ownership, and cost avoidance with respect to potential legal challenges. Unfortunately, most of these benefits have not been measured to date in quantitative terms because they would be very difficult and costly to measure.

Considering the potential costs of a single legal suit related to IP ownership and the overall low ongoing costs of implementing the Policy, it is believed that the economic and operational benefits outweigh the costs of the Policy.

¹² The Canadian Advanced Technology Alliance, Cal Fairbanks, Executive Director, letter to Alberta Government dated June 18, 2002. (Return to reference 12)

¹³ Australian Information Industry Association, Unleashing our IP Potential, 2006. (Return to reference 13)

4.0 Conclusions and Recommendations

Major conclusions reached during the evaluation are summarized. A set of recommendations are presented to improve the ability of the Policy to meet its objectives.

4.1 Conclusions

Regarding relevance, the evaluation determined that:

• The Policy is consistent with federal government principles and approaches for IP;
• The Policy is consistent with Government of Canada priorities and the use of Crown procurement to achieve social and economic objectives; and
• The Policy is an appropriate means to meet its objectives of increasing the potential for the commercialization of IP and meeting operational requirements.

Regarding performance (i.e. effectiveness), the evaluation determined that:

• The Policy has been implemented in applicable departments and incorporated into standard contracting processes; data have been collected and reports have been produced; and Department Heads have built awareness of the Policy among procurement staff;
• Processes in the departments studied in the Case Studies were insufficient in terms of documenting the justifications behind the decisions to invoke exceptions. There was a lack of controls, documentation trails and validation steps;
• The intent of the Policy was for most contracts to vest IP ownership with contractors, but a majority of contracts that were expected to generate IP actually ended up with the Crown owning the IP. Unfortunately, it is not possible to judge the appropriateness of the use of exceptions without more information on the rationale supporting the IP ownership decisions;
• There are opportunities for IC and TBS to improve the monitoring of results and follow up on data abnormalities;
• The Policy probably increased the likelihood of contractors responding to RFPs on contracts where an exception was not invoked;
• The Policy probably increased the likelihood of commercialization of IP, but this could not be measured from the data collected;
• The Policy has served to provide greater clarity for procurement officials for the treatment of IP; and
• The Policy may have helped the Crown avoid costs associated with potential legal challenges.

Regarding performance (i.e. efficiency), the evaluation determined that:

• The costs to administer the Policy were likely commensurate with its benefits.

4.2 Recommendations

The conclusions of the evaluation lead to the following recommendations.

1. To improve the monitoring and implementation of the Policy, IC and TBS should clarify roles and responsibilities with respect to the Policy within three months. Consistent with the Policy, monitoring should focus on cases where exceptions were invoked.
2. Once the roles and responsibilities have been clarified, the agreed upon department(s) should:
   1. Encourage departments to maintain appropriate documentation in contract files to justify IP ownership exceptions;
   2. Encourage departments to review the quality of data to be included in the GWPRS IP Report;
   3. Advise departments when they have reported high IP retention rates or the use of an exemption without Treasury Board approval;
   4. Share the results of the annual analysis of the GWPRS IP Report with departments through appropriate forums;
   5. Enhance the Implementation Guide to provide additional clarity; and
   6. Identify best practices used by departments and share these best practices through appropriate forums.
   7. Review IP retention rates one year after implementation of actions, and if a higher than expected use of exceptions still exists assess the appropriateness of their use and take appropriate action if required.

Description of Figure 1

The bar chart is titled: "16. What benefits do you believe arise from the Policy?"  The bar chart lists possible answers on the x-axis and the number of respondents who chose that answer on the y-axis.  There are seven possible options and the bar chart indicates that respondents could select all that applied, i.e. more than one. The seven options begin on the left and move to the right. They include "Clarified IP ownership", "Better willingness of contractors to enter into contracts", "Improved economic growth", "Improved industrial capacity", "Don't know", "There are no/few benefits", and "Other (please specify)". "Clarified IP ownership" was selected by about 205 respondents. "Better willingness of contractors to enter into contracts" was selected by about 125 respondents. "Improved economic growth" was selected by about 105 respondents. "Improved industrial capacity" was selected by about 90 respondents. "Don't know" was selected by about 15 respondents. "There are no/few benefits" was selected by about 10 respondents. Finally, "Other" was selected by about 10 respondents. The bar chart indicates that the total number of respondents (N) was equal to 289.

Return to Figure 1