Audit of the Spectrum Application Modernization – Commercial Software Implementation (SAM-CSI) Pilot Release

From: Innovation, Science and Economic Development Canada

Table of Contents

• 1.0 Executive Summary
  • 1.1 Introduction
  • 1.2 Audit Background
  • 1.3 Overview of Audit Results
  • 1.4 Audit Opinion and Conclusion
  • 1.5 Management Response
  • 1.6 Statement of Conformance
• 2.0 About the Audit
  • 2.1 Background
  • 2.2 Objective and Scope
  • 2.3 Methodology
• 3.0 Findings and Recommendations
  • 3.1 Introduction
  • 3.2 Change Management and Organizational Readiness
  • 3.3 Project Management
  • 3.4 Software Testing and Support
  • 3.5 Data Migration
  • 3.6 Implementation of Lessons Learned
  • 3.7 Management Response and Action Plan
• Appendix A: Audit Criteria
• Appendix B: Definitions

List of initialisms and acronyms used in report

AEB

Audit and Evaluation Branch

ADM

Assistant Deputy Minister

CIO

Chief Information Office

COTS

Commercial Off-the-Shelf

DG

Director General

ISED

Innovation, Science and Economic Development

IT

Information Technology

QA

Quality Assurance

SAM-CSI

Spectrum Application Modernization – Commercial Software Implementation

STS

Spectrum and Telecommunications Sector

SMS

Spectrum Management System

SSC

Shared Services Canada

TBS

Treasury Board Secretariat

1.0 Executive Summary

1.1 Introduction

The Spectrum Application Modernization Commercial Software Implementation Project (SAM-CSI) was established to address the impacts of aging Information Technology (IT) applications on the Spectrum Management Program (the Program). The Program manages the radio frequency spectrum, a unique resource that affects virtually all aspects of Canadian society, supporting wireless and mobile telecommunications. The objective of the Program is to support the Spectrum and Telecommunications (STS) Sector in achieving its mandate by maximizing the economic and social benefits that Canadians derive from the use of the radio frequency spectrum resource.

Spectrum officers rely on the in-house custom built Spectrum Management System (SMS) to collect radio licenses revenue and annualized spectrum auction revenue. The SMS is the fundamental suite of software tools used by Program staff to administer authorizations, to issue invoices for licensing fees associated with these authorizations, and to address signal coverage, potential interference and policy compliance.

The SMS is a mixture of older IT applications and supporting technologies that can no longer be adequately maintained. Many of the development platforms are now obsolete or reaching end-of-life. As such, the SMS has become difficult to maintain because the software is no longer supported and it is limited in its ability to address new Program requirements.

To address the end-of-life cycle issue, the Program decided to purchase a commercial-off-the-shelf (COTS) software solution to replace the outdated SMS, rather than develop another in-house custom system.

SAM-CSI project design and implementation work began in September 2012 and was slated to end in March 2016. The total cost of the project was estimated at $55 million, to be funded from existing departmental reference levels.

1.2 Audit Background

The objective of this audit was to assess the adequacy of the controls related to activities and processes supporting the implementation of the project, as informed by the experience of the Pilot Release (deployment date May 20, 2014) and Release 1 – Launch 1 (deployment date March 16, 2015).

The audit scope included project processes and activities in the areas of:

• Business requirements and changes to requirements;
• Change management and organizational readiness processes, activities and outcomes;
• Management of project budget, schedule and scope;
• Quality assurance processes and software support;
• Migration of data from legacy systems; and
• Implementation of lessons learned from the Pilot Release and Release 1 - Launch 1 to benefit subsequent releases.

1.3 Overview of Audit Results

Strengths

• SAM-CSI Project Management developed plans to address the change management requirements of the organization in support of SAM-CSI.
• A project management framework was developed and implemented, and oversight was established.
• A comprehensive testing strategy was established to support software development.
• Comprehensive data migration plans have been developed and implemented, and are regularly monitored.
• The SAM-CSI project team incorporated lessons learned from the Pilot Release to benefit subsequent release work.

Areas for Improvement

Some opportunities for improvement were identified by the audit:

• Change management plans were not revised to reflect challenges observed through project roll-out.
• A performance measurement system was not developed to track the intended improvements of the new business processes.
• Full payment was made despite performance issues noted in software development. Commitments to resolve the issues were not formalized.
• End-user testing was reduced from the plan, and did not reflect how the full software application would be used.

1.4 Audit Opinionand Conclusion

The results of the audit revealed that, with exceptions, the Project's control-related activities and processes support the implementation of the Project. Project governance is in place, and data migration is being managed effectively. There are opportunities to strengthen quality assurance and testing activities, as well as longer-term software support and maintenance planning.

1.5 Management Response

Management has agreed with the findings included in this report and will take action to address all recommendations by December 31, 2018.

1.6 Statement of Conformance

This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, as supported by the results of the Audit and Evaluation Branch's quality assurance and improvement program.

2.0 About the Audit

2.1 Background

The Minister of Innovation, Science and Economic Development Canada (ISED) is responsible for spectrum management in Canada: developing national policies and goals for spectrum resource use, and ensuring effective management of the radio frequency spectrum resource^{Footnote 1}. Within ISED, the radio frequency spectrum is managed by the Spectrum and Telecommunications Sector (STS) through the Spectrum Management Program (the Program). The objective of the Program is to maximize the economic and social benefits that Canadians derive from the use of the radio frequency spectrum resource.

The Program relies on an in-house custom-designed Spectrum Management System (SMS) to administer authorizations, to issue invoices for licensing fees associated with these authorizations, and to address signal coverage, potential interference, and policy compliance. The system is also used to coordinate and manage radio frequency assignments between Canada and the United States, and acts as an interface for information requests from clients and the general public. The SMS is a mixture of decades-old IT applications and supporting technologies that can no longer be adequately maintained or enhanced. Many of the development platforms are now obsolete or reaching end-of-life. As such, the SMS has reached its end-of-life stage, where components have been patched, and it has become complex, unstable and difficult to sustain.

The Spectrum Application Modernization Commercial Software Implementation Project (SAM-CSI) was established to replace most of the aging SMS applications. SAM-CSI is one of the largest IT system development projects that ISED has undertaken in the past ten years, at an estimated project cost of $55 million.

In consultation with Public Services and Procurement Canada, a contract was established to provide a commercial off-the-shelf software (COTS) solution, and to customize it where necessary. SAM-CSI Project implementation was also seen as an opportunity to streamline and standardize business processes.

The SAM-CSI project is administered by a dedicated project management team, which collaborates with a software Quality Assurance (QA) team, as well as business and functional leads and subject matter experts in various service lines. The project is being implemented in three phases: Pilot Release; Release 1 (separated into Launch 1 and Launch 2 stages); and Release 2.

2.2 Objective and Scope

Objective

The objective of this audit was to assess the adequacy of the controls related to activities and processes supporting the implementation of the project, as informed by the experience of the Pilot Release (deployment date May 20, 2014) and Release 1 – Launch 1 (deployment date March 16, 2015). The subsequent SAM-CSI Release 1 – Launch 2 and Release 2 fall outside of the scope of this audit.

Scope

The audit scope included project processes and activities in the areas of:

• Business requirements and changes to requirements;
• Change management and organizational readiness processes, activities and outcomes;
• Management of project budget, schedule and scope;
• Quality assurance processes and software support;
• Migration of data from legacy systems; and
• Implementation of lessons learned from the Pilot Release and Release 1 - Launch 1 to benefit subsequent releases.

2.3 Methodology

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada. Sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusion and opinion contained in this report. The opinion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were discussed with management.

The audit was performed in three phases: planning, conduct and reporting. A risk assessment was executed during the planning phase of this audit to confirm the audit objective and identify areas requiring more in-depth review during the conduct phase.

Based on the identified risks, the Audit and Evaluation Branch (AEB) developed audit criteria that linked with the overall audit objective (refer to Appendix A).

The methodology used to address the audit's objective included:

• Documentation examination and review; and
• Conduct of 25 interviews with personnel from the SAM-CSI project, STS, Chief Information Office (CIO), the Vendor and external clients.

All of the audit evidence gathered through the above noted processes was synthesized, analyzed and supports the audit findings presented throughout this report.

3.0 Findings and Recommendations

3.1 Introduction

This section presents detailed findings from the Audit of the SAM-CSI project. The findings are based on evidence and analysis from both the initial risk assessment and the detailed audit work. In addition to the findings below, AEB has communicated to management, either verbally or by management letter, findings for consideration that were non-systemic or of low materiality or risk.

3.2 Change Management and Organizational Readiness

The SAM-CSI project brings changes to both business processes and to the information technology being used. With this scope of organizational transformation, change management plans should be in place to define an approach that will transition the business organization to the proposed new environment, and to ensure the organization is ready to successfully conduct business using the new standardized processes and tools. Additionally, there should be a plan to decommission the legacy applications being replaced by the new solution.

Both a formal Change Management Strategy and Change Management Plan were developed during the project planning phase. The Strategy provided a high-level description of change management needs for the Program, and was complemented by a Change Management Plan that presented a detailed action plan for the SAM-CSI Pilot Release. It also described the communication and training strategy for the Program as it applied to SAM-CSI. Change management activities and processes supporting the Strategy and Plan included:

• the establishment of a change network throughout the organization that included a change management and communication lead;
• defining specific business process change requirements;
• communication and awareness activities;
• a change readiness assessment; and
• delivering training.

While a fulsome plan had been developed, not all activities were delivered as committed. Communication and training challenges were noted by external clients, stating they did not receive adequate information to allow them to properly prepare for the new system. Documentation and training materials were not completed at the time of release, and training was delivered on an interim, in-development software system that did not reflect the final system put into production.

The Program acknowledged these challenges and initiated enhancements for the subsequent releases with the aim of addressing the issues experienced with the Pilot Release. The changes included the development of new communication tools, targeting both internal users and external clients, in an effort to establish a more effective two-way dialogue, as well as the identification of personnel to maintain and update documentation.

In addition, the Program began familiarizing all regional staff with the existing Pilot Release to facilitate future Release 2 training, and continued assignment of a senior manager with Spectrum operational experience to work closely with the project team and regional operations management in support of organizational readiness. While management has taken steps to operationalize the transition for subsequent releases, these updates have not been documented in the Change Management Plan, and no plans address the entire project life cycle beyond the Pilot Release.

As a result, there are risks that change management activities will not properly prepare users to conduct business using the new SAM-CSI system and standardized processes. As well, external clients may not be familiar enough with the new IT tool to submit licence applications under the new process.

The ISED Support Model is not fully defined beyond the project end date, March 31, 2016. Without a well-defined Support Model, there is a risk that stakeholder responsibilities and accountabilities, as well as support requirements, will be unclear for the maintenance time period, as well as for the long-term operational and support state of the SAM-CSI and legacy applications.

The Decommissioning Plan for achieving the desired outcome of 80% legacy application decommissioning is in progress. However, there is no schedule and the plan has not been finalized or approved. There is therefore a risk that the target for decommissioning will not happen in a timely manner and additional costs to the organization will be incurred to maintain both the legacy and new system.

Recommendation 1

The DG, Spectrum Management Operations Branch should ensure that:

1. The Change Management Plan is an evergreen document; updated on a regular basis and include subsequent releases and the post implementation phase. The updated plan should include activities that need to be carried out for communication, documentation and training as well as post-implementation follow-ups; and
2. In consultation with SSC, CIO management, the Transition Plan, Support Model Plan and the Decommissioning Plan are finalized and approved prior to final project completion.

The focus of the Program has been to leverage the base functionality of the COTS system and, where necessary and feasible, to update or re-design business processes to support the base functionality. As well, the Program is using the SAM-CSI Project as an opportunity to standardize and consolidate business processes across the country. One of the benefits to be derived from the initiative, as described in the project business case, is that the new system will track and report on these standardized and consolidated processes.

While reports are being generated regularly to monitor project activities, key metrics have yet to be developed to track and report on the new and updated business processes, or performance against pre-defined outcomes. The Program has acknowledged this need and, at the time of the audit, was considering establishing key metrics and new performance measures.

Lack of definition and monitoring of performance measures may lead to inefficient business processes thereby impacting productivity and satisfaction of internal users and external clients. There is also the risk that management will not make fully informed decisions on whether and how to modify business processes.

Recommendation 2

The DG of Spectrum Management Operations should oversee the development and implementation of a performance measurement approach to track the timeliness and efficiency of key business processes across the country, to improve decision making and to allow for timely corrective action. This approach should include the development of key performance indicators that are aligned with operational goals.

3.3 Project Management

It is important to have processes in place to actively manage all facets of the project, including managing the budgets and schedules, the project scope and business requirements to ensure the delivered IT system addresses the needs of the Program and meets expected project outcomes.

The SAM-CSI Project Charter is a key document that sets up the framework for project management, defining the project governance and project team structure. It also provides an overview of various project management processes including those for budget, scope and schedule management. The charter also describes activities related to managing business requirements.

The Project Steering Committee endorsed the scope, schedule and budget of the Pilot Release in September 2012. At that time, the Project Director presented to the steering committee a breakdown of the expenditure maximums for the Vendor professional services. The total value for professional services aligned with the estimates provided in the effective project approval documentation as well as the existing contract.

Processes are in place to manage and control budgets, schedule and scope as well as business requirements. Multiple tools, including work break-down structures and critical paths, have been developed to track and monitor progress against the project schedule. A review of these documents demonstrated that they had been updated to reflect the schedule modifications of the Pilot Release and Release 1 – Launch 1 deployments.

The project scope was being managed to support the project outcome of decommissioning legacy applications. Furthermore, business requirements were defined, reviewed and approved with the participation of appropriate stakeholders. A formal process is in place for addressing change requests that affect the project scope, budget or timeline.

At a senior management level, oversight bodies, including the Project Steering Committee and Treasury Board Secretariat (TBS) Executive Project Oversight Committee, receive reports comparing budget versus actual expenditures, and a summary of the overall health and status of the project (e.g. scope schedule, issues).

Quality Management

The Contract Deliverables Requirement List (CDRL) defines the specific deliverables expected of the vendor for the project. While the software is providing a level of functionality that allows for the processing and issuance of licences and/or certificates, the audit determined that key software elements launched as part of the Pilot Release and Release 1 – Launch 1 did not meet all requirements as defined in the CDRL. In some instances, releases required workarounds or post-production fixes in order to meet operational needs.

After the Pilot Release, it was decided that, as part of the CDRL, outstanding defects that were not barriers to issuing licences and certificates were to be prioritized and fixed by the end of the project through subsequent releases. At the time of the audit, no prioritisation had yet occurred.

The deferral of some software deliverables as outlined in the CDRL has not been reflected in a contract amendment, nor has an amendment been developed to change project timelines. Without a formal contract amendment, there is a risk to the organization that the gaps will not be addressed as planned, or that avenues to seek resolution, if required, may be limited. The Program has indicated that these releases will not be considered fully accepted to ensure that the vendor will adhere to its commitments however full payment has been made to the Vendor for both releases.

Recommendation 3

The DG, Spectrum Management Operations Branch should ensure that all contract deferrals and commitments regarding the resolution of outstanding defects be formally documented and approved between ISED and the Vendor in a contract amendment prior to Project completion, scheduled for March 2016.

3.4 Software Testing and Support

When a project involves software configuration and customization, it is important to have resources, activities and tools in place supporting the QA testing process, in order to ensure that the software deliverables are functional and meet the needs of the project. Testing plans should build user feedback into the development cycle so that real-world requirements are factored into the testing, and the product meets ultimate business needs. Additionally, on-going software support to both project teams and end users is essential to ensuring that business operations continue while the software is in development.

A QA testing strategy for SAM-CSI was developed, and included a comprehensive testing process to ensure that software deliverables were formally tested using various test cases. These test cases were intended to validate and confirm whether the business requirements were being met by the new software.

A QA team was established and made responsible for carrying out the testing, with the support of other project team members including functional and business leads and subject matter experts. The QA team was also responsible for recording and managing the results of their testing.

In practice, the QA approach was modified from the plans and limited to prescribed scenarios; full testing of all uses of the software was not performed. The Program indicated this was due to resource re-prioritisation in addressing the above-mentioned CDRL. Further impacted by the resource constraints was enforcement of support resolution response times, which were largely suspended and limited to issues seen as higher urgency or risk.

By limiting testing activities, there is an increased risk that the software will not meet user requirements, and that its full functionality will not be realised. There is also a longer-term risk that on-going support requirements will increase after release and implementation, as new issues are discovered by users through operational usage.

Recommendation 4

SAM-CSI Project Management should ensure that comprehensive testing is conducted, in areas deemed to be of higher risk or of critical business importance for all future releases.

3.5 Data Migration

As legacy applications are replaced with a new system, it is important to have adequate plans and processes as well as resources in place to support data migration and clean-up activities.

Comprehensive data migration plans were developed by Project Management that included data mapping rules. A phased approach was used for data migration where each phase is based on a service line that aligns with the release strategy. For example, only data related to the microwave service line was to be transferred from the legacy system to the new system when the Pilot Release was deployed.

The project team developed data migration plans that provided the opportunity to test and improve the quality of the final migration process. Data migration logs demonstrated that validation activities took place to determine whether data was accurately translated. The data migration teams met regularly during the life of the Project to oversee and carry out data migration activities and continue to do so.

Prior to the migration process for the Pilot Release and Release 1 – Launch 1, data was cleaned to ensure that legacy data conformed to the requirements (i.e. validation rules) of the new system. To mitigate the risk of data migration errors, the Program has adopted a phased approach to data conversion and to ensure the data mitigation and clean-up team is well resourced.

3.6 Implementation of Lessons Learned

As per SAM-CSI's phased approach, the Pilot Release offered an opportunity to the SAM-CSI Project team to develop and adopt lessons learned. The Pilot Release focused on one service line with a relatively small user and client group to identify deployment issues requiring attention.

The audit team examined lessons learned recommendations derived by the SAM-CSI Project team from the Pilot Release. All of the recommendations were implemented where appropriate. In interviews with the Vendor, they indicated that they were also involved in the Pilot Release's lessons learned exercise. They indicated that benefits had been realized with design and data migration processes.

As of July 2015, the project team was still working on developing preliminary lessons learned from the Release 1 – Launch 1.

3.7 Management Response and Action Plan

Audit of the Spectrum Application Modernization – Commercial Software Implementation (SAM-CSI) Pilot Release – Management Response and Action PlanPDF - 64.5 KB

The findings and recommendations of the audit were presented to the Senior Assistant Deputy Minister of Spectrum and Telecommunications (STS), the Director General of the Spectrum Management Operations Branch, STS, and the Director of the SAM-CSI Project. Management has agreed with the findings included in the report and will take actions to address the recommendations by December 31, 2018. Notably STS management:

• has documented the Project Charter and Plan, and will document the future maintenance contract to reflect the agreement reached with the Vendor principals that all prioritized defects remaining after the final release will be fixed at no additional cost to ISED;
• will update and finalize the four plans that address the organization's needs in support of change management and its transition to the new IT and business process environment;
• is considering developing specific process performance metrics in the near future, and will continue to monitor and report on outcomes identified in the SAM Outcome Realization Plan; and
• will assign additional resources to the testing team and will implement more rigorous testing to address high risk areas in current and future code releases, which should improve the overall QA process.

Appendix A: Audit Criteria

Appendix B: Definitions