Audit of the Spectrum Compliance Report

From: Innovation, Science and Economic Development Canada

May 2019

Audit of the Spectrum Compliance [PDF - 404 KB]

1. 1.0 executive summary
   1. 1.1 Introduction
   2. 1.2 Audit background
   3. 1.3 Overview of audit results.
   4. 1.4 Audit opinion and conclusions
   5. 1.5 Management response
   6. 1.6 Statement of conformance
2. 2.0 Background
   1. 2.1 Spectrum overview
   2. 2.2 Spectrum management
   3. 2.3 Framework for spectrum compliance
   4. 2.4 Previous audit engagements
3. 3.0 About the audit
   1. 3.1 Audit objective, scope, and methodology
4. 4.0 Findings and recommendations
   1. 4.1 Introduction
   2. 4.2 Governance
   3. 4.3 Compliance program design
   4. 4.4 Strategic and operational planning
   5. 4.5 Compliance program delivery
   6. 4.6 IT application system
   7. 4.7 Equipment and fleet
   8. 4.8 Monitoring and reporting
   9. 4.9 Training and development
   10. 4.10 Continuous improvement
   11. 4.11 Management response and action plan
5. 5.0 Overall conclusion
6. Appendix A: Audit criteria
7. Appendix B: National compliance framework activities

List of acronyms used in report

AEB

Audit and Evaluation Branch

AOR

Atlantic and Ontario Region

DBCP

Broadcast, Coordination and Planning Directorate

DBIT

Business Intelligence and Transformation Directorate

DGEPS

Engineering, Planning and Standards Branch

DGSO

Spectrum Management and Operations Branch

DOS

Spectrum Management Operations Directorate

NCF

National Compliance Framework

NEWG

National Equipment Working Group

NCEWG

National Compliance and Enforcement Working Group

NHQ

National Headquarters

RF

Radio-Frequency

SC6

Health Canada Safety Code Guideline 6

SMO

Spectrum Management Officer

SOC

Spectrum Operations Committee

STTC

Spectrum Technical Training Committee

STS

Spectrum and Telecommunications Sector

TB

Treasury Board

TSO

Technical Support Officer

QNR

Quebec and Nunavut Region

WR

Western Region

1.0 Executive Summary

1.1 Introduction

Innovation, Science and Economic Development Canada (ISED) is responsible for spectrum management in Canada through the Department of Industry Act, the Radiocommunication Act, the Telecommunications Act, and related regulations.

The Spectrum and Telecommunications Sector (STS) within ISED is responsible for the management of Spectrum. The STS's general policy objective is to maximize the economic and social benefits that Canadians derive from the use of the radio-frequency Spectrum. The sector works with stakeholders across the Department, the government, and with the public to support a safe, competitive, and cyber-resilient wireless industry.

Within STS, the Spectrum Management and Operations Branch (DGSO) plays a primary role in supporting the broader sectoral mission and vision. This branch oversees Spectrum operations, including licensing and regulatory interpretation, and delivers the Spectrum compliance program.

In 2015-16, a National Compliance Framework (NCF) for Spectrum was developed to standardize compliance activities across the country. The framework was launched in 2016-17, using a phased five-year roll-out, focusing first on the development and implementation of higher-risk elements followed by the development and implementation of lower-risk elements in the later years. The program was in its third year at the time of the audit.  

The Spectrum Management and Operations Branch aims to adopt a common approach to compliance activities, increase efficiencies in the allocation of its resources, and achieve comparable results across regions.

In fiscal year 2017-18, the expenditure budget for the DGSO was approximately $30.9M, with 326 employees. Of these employees, 69 were in the National Capital Region, and 257 were spread across the Quebec and Nunavut Region (QNR), Atlantic-Ontario Region (AOR), and Western Region (WR), the last of which includes Yukon and the Northwest Territories.

1.2 Audit background

The objective of the audit was to assess the adequacy and operating effectiveness of the management control framework in place for Spectrum compliance.

The audit scope focused on the Spectrum compliance management control framework, including compliance activities designed and carried out in national headquarters and in the regional offices.

The audit scope focused on Spectrum compliance activities and processes between April 1, 2017 and August 31, 2018, including assessments of:

• Governance and oversight;
• Planning and reporting;
• Processes, procedures and tools;
• Training; and
• Continuous improvement.

1.3 Overview of audit results

Strengths

The Spectrum Management Operations (the "DGSO") has established governance and oversight bodies for Spectrum operations. The National Compliance Framework is being implemented to standardize compliance activities across regions. Compliance activities are nationally planned and directed, and formally include regional input to include sites of high-risk.

Equipment and fleet are generally planned for and maintained. Formal reporting occurs regularly at the national and regional level. A nationally standardized training and development program exists, and continuous improvement is embedded in the annual planning and reporting cycle.

Areas for Improvement

Some roles and responsibilities vary at the operational level across regions. The framework may not include all compliance activities within the Spectrum program, and expected compliance rates are still being established. The methodology used to support the planned volume of activities and their allocation to regions should be strengthened with a formal risk assessment.

The IT application system used by the program does not adequately support the management, monitoring, or reporting of compliance activities. The fleet is used beyond the recommended life span, and some equipment is not calibrated in accordance with the recommended schedule. Annual reporting contains limited information on compliance rates since the framework was in its third-year of implementation at the time of audit, and results do not formally inform year-over-year changes for most activities.

1.4 Audit opinion and conclusions

The Spectrum program is making progress in implementing its five-year National Compliance Framework. Future efforts will be required in order to ensure a nationally consistent approach to all compliance activities, including enhancing the methodology for planning compliance activities, further defining procedures, building quality controls over the capture of information, and producing more comprehensive annual reporting.

1.5 Management response

Management has agreed with the findings included in this report and will take action to address all recommendations by March 31, 2024.

1.6 Statement of conformance

This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, as supported by the results of the Audit and Evaluation Branch's quality assurance and improvement program.

Dawn Lumley-Myllari
Chief Audit Executive
Innovation, Science and Economic Development Canada

2.0 Background

2.1 Spectrum overview

Spectrum refers to the full range of electromagnetic energy in all its forms (radio, infrared, visible light, ultraviolet, x-ray, and gamma ray). This energy travels in waves, which may be harnessed for a vast array of different purposes depending on the wavelength and associated physical properties. Within the broader electromagnetic spectrum, the radio frequency (RF) portion is a finite natural resource that is integral to Canada's wireless communications infrastructure.

Wireless operators use transmitters equipped with antennas to produce invisible, intangible radio waves carrying information (audio, video, or digital signals) to one or more receivers through physical space. These communications serve many interests: private, commercial, public safety, consumer, defence, national security, and scientific. Efficient and effective use of radio spectrum is vital to Canada's economy, since mission-critical infrastructure in all sectors depends on it.

All wireless applications and devices rely on radio waves to send and receive information and must, therefore, contend with each other for access to the specific part of the limited radio spectrum that serves their specific communications needs.

As Figure 1 shows, Spectrum is used around the world every day by businesses, the public, and governments to access and connect a broad range of applications and services.

Figure 1: Examples of everyday use of the RF Spectrum

Canadians expect up-to-date wireless communications services to be ubiquitous, reliable, and safe. However, given that radio spectrum is scarce and highly in demand, meeting this expectation requires effective spectrum management; i.e., regulating access to radio frequencies to ensure efficient usage that is in the public interest. More than five billion devices are already using Spectrum, and in just a few years, that number is expected to reach 50 to 100 billion. With the rapid and ongoing pace of advancement in technologies, challenges exist to prepare for the future while continuing to support the services that Canadians depend on today.

With over $48 billion of telecommunications service revenue generated in 2016^{Footnote 1}, Canadian companies are continuing to invest in telecommunications infrastructure. It is projected that approximately $26 billion will be invested in 5G infrastructure between 2020 and 2026^{Footnote 2}. The economic impact of effective spectrum management is significant to the Canadian telecommunications industry, and through these investments and initiatives, Canadians will be better positioned to participate in the digital economy, now and in the future.

2.2 Spectrum management

Innovation, Science and Economic Development Canada (ISED) is responsible for spectrum management in Canada through the Department of Industry Act, the Radiocommunication Act, the Telecommunications Act, and related regulations.

ISED helps protect Canadians against any potential adverse health effects resulting from over-exposure to RF energy emitted from wireless transmitters, such as those in mobile phones and installed on cell towers. Since over-exposure to RF energy could potentially result in adverse effects to human health, the Government of Canada has established RF exposure limits that are among the most stringent in the world. These limits are set out in Health Canada's guidelines on safe human exposure to RF energy, which was revised in 2015 ("Safety Code 6").

The Spectrum and Telecommunications Sector (STS) within ISED manages the overall Spectrum program and its general policy objective is to maximize the economic and social benefits that Canadians derive from the use of the radio-frequency Spectrum.

The sector responsibilities include researching, auctioning, and regulating spectrum, regulating telecommunications equipment, and working with other organizations to ensure the safety and security of existing and future telecommunication infrastructure in Canada.

Spectrum Compliance Program Delivery

Within STS, the Spectrum Management and Operations Branch (DGSO) plays a primary role in supporting the broader Spectrum program's mission to manage Spectrum so that wireless works for Canadians and its vision to make Spectrum available to everyone and everything, anytime and anywhere. This branch oversees Spectrum operations, including licensing and regulatory interpretation, and administers the Spectrum compliance program.

The DGSO operationalizes Spectrum program policies across the country through its three directorates – Spectrum Management Operations Directorate; Broadcast, Coordination and Planning Directorate; and Business Intelligence and Transformation Directorate – and its three regional offices, including Atlantic and Ontario Region (AOR), Quebec and Nunavut Region (QNR) and Wester Region (WR). Yukon and Northwest Territories are part of WR. Each of these directorates contributes to the compliance program design and delivery.

In fiscal year 2017-18, the expenditure budget for the DGSO was approximately $30.9M, with 326 employees. Of these employees, 69 were in the National Capital Region, and 257 were spread across the three regions.

2.3 Framework for spectrum compliance

Spectrum must be actively managed so that it is used safely and interference to its effective use is prevented, detected and corrected in a timely manner. Compliance activities typically start with the principle of educating, advising, and influencing to promote voluntary compliance, and eventually enforcing proper practices among all Spectrum user groups.

In 2015-16, a National Compliance Framework (NCF) for Spectrum was developed to standardize compliance activities across the country. Fiscal year 2016-17 was the first year of the five-year NCF implementation plan, wherein the Spectrum Management and Operations Branch aims to adopt a common approach to compliance activities, increase efficiencies in the allocation of its resources, and achieve comparable results across regions.

Prior to the NCF, regions implemented their own compliance activities, and Spectrum licensees' operations were evaluated in isolation. The new approach establishes national consistency, but also reflects differences between the regions, such as geographic constraints and population concentrations, which can have significant impacts on radiocommunications and supporting industries. The goal is to deal with these differences equitably through regional autonomy, guided by unifying principles for the compliance activities outlined in the NCF.

The NCF framework groups compliance activities into four key themes, as shown in Figure 2^{Footnote 3}.

Figure 2: National Compliance Framework – Compliance Activities

Spectrum compliance program activities are carried out by delegated Spectrum Management Officers (SMOs) located in the regional offices. Some activities might include taking measurements in cellular towers and public spaces – such as parks or schools – to determine if Spectrum emission levels are within safe limits. SMOs might also inspect broadcasting stations to ensure they are constructed, operated and maintained within the terms and conditions of the broadcasting certificate.

In support of compliance with Health Canada's Safety Code 6 (SC6), ISED performs compliance activities. These are considered high-priority activities by the DGSO, as they help protect public health and safety.

The National Compliance Framework includes a process for annual planning of compliance activities, with a focus on SC6 activities, and including an expected volume of activities for each region. These activities must be factored into annual resource planning to ensure that SMOs are available to perform compliance activities, in addition to their core responsibilities relating to licensing, and resolution of harmful interference.

2.4 Previous audit engagements

While there were audit and advisory engagements completed on the Spectrum program in the past, none of them had the Spectrum compliance program in scope.

3.0 About the audit

3.1 Audit objective, scope, and methodology

In accordance with the approved Innovation, Science and Economic Development (ISED) 2018 to 2021 Multi-Year Risk-Based Internal Audit Plan, the Audit and Evaluation Branch (AEB) undertook an audit of the Spectrum compliance.

Audit Objective

The objective of the audit was to assess the adequacy and operating effectiveness of the management control framework in place for Spectrum compliance.

Audit Scope

The audit scope focused on the Spectrum compliance management control framework, including compliance activities designed and carried out in the national headquarters and in the regional offices between April 1, 2017 and August 31, 2018, including an assessment of:

• Governance and oversight;
• Planning and reporting;
• Processes, procedures and tools;
• Training; and
• Continuous improvement

Methodology

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada.

Based on the identified risks, AEB developed the audit criteria and sub-criteria linked to the overall audit objective (see Appendix A).

The methodology used for this audit included various procedures to address the engagement's objective. This included, but was not limited to, review of documentation, interviews, sample files review, site visits to regional offices, process and system walkthroughs.

Sample of compliance activity files, equipment, and fleet selected for review, included:

• 35 compliance activity files to assess whether processes and practices are comparable across the regions.
• 44 inspector card activities to determine that individuals had received the delegated or statutory authority before carrying out any inspections and testing in the field.
• 12 vehicles and 34 equipment items (such as Radio-frequency Receivers, Spectrum Explorers and Analyzers, Narda meters and probes) selected from five district offices.

A debrief meeting was held with DGSO on April 5th, 2019 to validate the findings that form the basis of this report. This meeting also provided the auditee an opportunity to offer any additional information and clarification regarding the findings.

4.0 Findings and recommendations

4.1 Introduction

This section presents detailed findings from the audit of the Spectrum compliance. The findings are based on evidence and analysis from both the initial risk assessment and the detailed audit work.

4.2 Governance

Program Governance & Oversight

The DGSO has established an overall Spectrum program governance framework with defined roles and responsibilities at the national level. Within the DGSO, the Spectrum Operations Committee (SOC) oversees the design and delivery of the Spectrum compliance program, and reports to DGSO's Management Committee. The SOC is chaired by a Senior Director at national headquarters (NHQ), with representatives from all three regions, as well as other branches such as the Engineering, Planning and Standards Branch (DGEPS). Additional sub-committees and working groups also report to the SOC, such as:

• Spectrum Technical Training Committee (STTC): discusses and resolves issues related to the National Training Program for Spectrum Management Officers within the Spectrum program.
• National Compliance and Enforcement Working Group (NCEWG):  provides direction and guidance to ensure consistency of compliance and enforcement policies, procedures and documentation.
• National Equipment Working Group (NEWG): provides direction on equipment needs and/or system requirements necessary to carry out technical measurements on RF signals, in support, of the Spectrum management operations.

The committees and working groups have documented terms of reference and have NHQ and regional representation, and they meet regularly to discuss the status of the compliance program, its plan and priorities. Other issues and challenges are also addressed through these committees and working groups. Additionally, each region has established governance committees such as the Regional Senior Management Team (RSMT) and the Regional Spectrum Operations Committee (RSOC) to discuss and address strategic and operational performance issues and challenges.

Roles and Responsibilities

In NHQ, roles, responsibilities, and authorities for Spectrum compliance program are defined for each of the three directorates within DGSO. The three regional offices work collaboratively with the other directorates in NHQ to offer full range of Spectrum Management activities to the public, including compliance activities pertaining to radiocommunication and broadcasting systems. Regions have established organizational structures, which are functionally comparable from region to region.

While all compliance activities related to safety such as SC6 are carried out by Spectrum Management Officers, there are inconsistencies between and within regions in how other compliance activities are carried out. These compliance activities are not associated with delegated authorities (Verifying Continued Operations, Broadcasting Inspection Requirements Analysis) and are performed by combinations of Spectrum Management Officers (SMOs), Technical Support Officers (TSOs), and summer students, depending on the availability of resources.

In accordance with the authorities conferred on the Minister pursuant to the Radiocommunication Act, a delegation of authority chart exists which identifies the levels of authority for tasks that are statutory in nature, as well as those that  support the exercise of statutory authority and the Spectrum compliance program generally. The issuance of inspector cards and badges to support these delegated authorities is decentralized to the regions, and, of the three regions, only the Atlantic Ontario Region (AOR) has formally documented their process and practices for the issuance, return, and replacement of inspector cards and badges. Through process walkthroughs and interviews, it was determined that regional processes vary, and that master lists of individuals with inspector cards are not kept up-to-date.

Without clearly documenting the roles and responsibilities at the operational level, the compliance program may not be uniformly delivered across every region. The absence of a documented and standardized process to issue inspector cards, could limit ISED's ability to ensure only delegated officers conduct activities that involve delegated authorities.

Recommendation 1 (Low Risk)

DGSO should document and standardize roles and responsibilities to support the delivery of all compliance activities to ensure efficiency and consistency across the country.

Recommendation 2 (Medium Risk)

DGSO should establish a standardized process for the issuance, management, and monitoring of inspector cards across all regions.

4.3 Compliance program design

The DGSO National Compliance Framework's purpose is to ensure national consistency in the application of verifying compliance of radio and Spectrum users with the Radiocommunication Act (the Act) and its regulations. The objective is to achieve compliance for the benefit, safety, and efficient use of all Canadians. The framework emphasizes the application of risk-based mitigation strategies, where the compliance activities are aligned with targeted areas of strategic priorities for the Spectrum program. The framework prioritizes the risk areas of public safety and the degradation and availability of spectrum use. A five-year cycle was adopted for the implementation of the framework, and through this cycle, the compliance activities are rotated to maintain a momentum of compliance as well as to inform any further changes and emphasis of the activities.

While the NCF incorporates a broad range of activities, additional Spectrum compliance activities take place outside of DGSO's framework, such as the monitoring of radio and broadcast equipment that are conducted by another branch, the Engineering, Planning, and Standard Branch (DGEPS). Other activities include the monitoring and oversight of the accredited organisations that administer programs such as training and examination for professional and amateur radio operator certificates on behalf of the Department.

The regions are also responsible for undertaking investigations outside of the NCF, both planned and reactive. These investigations are carried out to confirm whether the licensees and the certificate holders are operating within the terms and conditions of their radio authorizations . These investigations are also designed to detect whether any unauthorized use of Spectrum and equipment are causing harmful interference to other authorized users.

Without a compliance framework that includes all related activities and is focused on supporting the goals of the overall Spectrum program, senior management may not be in a position to uniformly conclude on the success of compliance activities, and efforts may lose efficiency through differently managed activities.

Recommendation 3 (Medium Risk)

The DGSO should reassess the National Compliance Framework (NCF) to determine whether the compliance activities are aligned to the needs of the overall Spectrum program.

4.4 Strategic and operational planning

Strategic Resource Planning

As a branch, the DGSO has established a formal process for strategic resource and operational planning. Each year, the designated planning officers in NHQ lead the exercise and solicit input from senior managers in the regions and in NHQ to develop the annual financial and operational plans. The plans are finalized through a series of discussion with the regions and formally through committee meetings, and subsequently approved by the DGSO senior management team.

In the regions, a similar approach is taken. The designated officers lead the planning exercise and coordinate with the regional management team for their inputs, which are then communicated to NHQ for consideration and inclusion in the yearly plans. To implement, the regions are given the flexibility in managing and utilising their resources to meet the expected performance targets while balancing other competing or emerging priorities. 

While the strategic resource and operational plans are in place, there are no detailed breakdowns for the Spectrum compliance program. For example, the overall financial plans in fiscal 2017-18 and 2018-19 included salary and non-salary operating expenditures for all authorization and licencing, interference investigation, and compliance and enforcement functions across the larger Spectrum program. Similarly, the human resourcing plan was for the entire DGSO branch, and did not specify the needed levels of effort to perform compliance activities.

Annual Targets for Compliance Activities

As part of the yearly operational planning exercise, NHQ sets the target numbers (i.e. volume) of compliance activities to be conducted by each region. The targets are set based on the strategic priority of health and safety first, followed by all other compliance activities. Throughout the planning, the regions are consulted for feedback and are given opportunity to provide inputs that incorporate their differences in geography, climate, and any unpredictable or major events that may affect their resource availability to carry out compliance activities and meet certain targets. For example, there are small windows of time to carry out compliance activities in specific weather conditions, namely hillside, mountain top and isolated fields that are inaccessible during winter months. Extreme cold and hot weather conditions can also affect the performance of certain equipment in taking accurate measurements.

While the regions are consulted in the development of operational plans and targets, it is unclear how the planned volume of compliance activities is established. It is not supported by historical data of non-compliance rates and the methodology is not formally documented to demonstrate how the volume targets are formulated and allocated to the regions. Although the program has identified all SC6 activities as high priority, 100% of SC6 street-level scans were compliant across all regions in 2018-19, yet the volume of regional targets was doubled.

While planning for compliance activities in this context is an operational rather than a safety issue, without a defined risk-based methodology used to develop annual compliance targets, Spectrum compliance activities may not be directed towards those areas most in need of attention. The risk-based methodology increases in importance if capacity constraints increase, if only periodically.

Recommendation 4 (High Risk)

The DGSO should formalize its risk-based methodology to support the annual planning for compliance activity targets, informed by resource requirements and constraints.

4.5 Compliance program delivery

Regions have been performing compliance activities for many years. The motivation for implementing the NCF was to standardize delivery of these activities, and to ensure consistency across regions.

At the time of the audit, eight of the ten compliance activities had been adopted in the NCF, while six of these activities were mainly delivered in regions (see Appendix B). Five of the six regionally delivered compliance activities were standardized using common practices and national procedural guidance. These activities included the SC6 Street-Level Scan, SC6 Site Audits, Verifying Continued Operations and Broadcasting Inspection Requirement Analysis and Broadcast Inspection. However, while standard forms and templates were used to perform these activities, regions varied in how they captured information within these templates.

Prevention of Non-Compliance, which is largely an outreach and education activity, is delivered differently across the regions. The purpose and method for this activity is understood differently between offices; while some endeavour to hold information sessions for large groups of stakeholders, others may conduct a single site visit for the year, while other offices await further instruction before adopting the activity.

NHQ conducts SC6 Industry Reporting and Spectrum Licence Terms and Conditions Audits, with additional work performed by regions on an as-needed and directed basis. As such, national standardisation for these activities does not hold the same challenges as the others.

Radio Licence Terms & Conditions Audits and Monitoring of Unlicensed Operations do not have common processes and practices in place, as they are not yet standardized; DGSO plans to standardize Radio Licence Terms & Conditions Audits in fiscal 2019-20, while the implementation date for Monitoring of Unlicensed Operations has not yet been confirmed. It is important to note, however, that regions continue to perform these activities even without standardisation.

As a good practice, the Western Region (WR) has implemented quality control practices and detailed procedural guides to support greater consistency from office to office in the delivery of compliance activities, and the content of their activity reports. However, without further standardisation of compliance activities and quality control practices in place across the entire program, DGSO may not be able to achieve nationally consistent delivery of compliance activities across all regions.

Recommendation 5 (Medium Risk)

The DGSO should further standardize the remaining compliance activities by implementing national procedural guidelines with quality control practices to ensure compliance activities are delivered and documented consistently across regional offices.

4.6 IT application system

The licensing module within the Spectrum Management System (SMS), has been adapted for the compliance program, and is used to log all compliance activities and results, including measurements, observations, follow-ups, and corrective actions taken by SMOs and other technical support staff.

While the intention was to leverage the SMS, which was already in place, the licensing module is not designed to guide users through each step of a process, and instead acts as an information repository only, and regions maintain separate systems for tracking their activities. As such, a number of standard IT controls are missing or are insufficient:

• No process controls exist, which would be used to help manage each step in the process, embed approvals, and ensure consistency of delivery;
• There are few input controls, which means that any type of information can be entered into any field;
• No completeness checks are built into the system, which means that no information is essentially "required" to complete a file;
• A number of drop down lists can be over-written and changed by users; and
• The free-form text boxes used for providing measurement results and explanations have space limitations, resulting in users creating additional documentation outside of the system.

Further, the licensing module is not integrated with other supporting systems, and requires that all information be manually entered. For example, data must first be downloaded to portable devices, and then uploaded to personal workstations for further data cleaning and organisation. The data is then manually entered into a Word document template, and uploaded to the system as an attachment. Systems previously used to support automated analysis, such as the Technical Measurements Reporting System, which was used to cross-reference licensing data against usage data, are no longer supported, and is not integrated with the SMS. Management indicated that a decision was made to not integrate TMRS and its functionality when SMS was implemented, as the cost of integration would have driven the project budget higher. However, SMOs continue to use these legacy systems, as they provide key sources of data for compliance and licensing activities, and their function has not been replaced.

Additionally, users spoke of regular system crashes and unavailability during extended periods. To work around some system limitations, a user manual has been developed to provide guidance on which fields to use, and for which purpose. This document is essential to the process, as the same data fields are used for every compliance activity, but each activity has different requirements. This has led to a challenge in ensuring consistency, as 79% of general mandatory fields (account numbers, dates, category of activity, relevant office) in sample files were accurately completed, and 47% of data mandatory fields (measurements and inquiries) were accurately completed. As a good practice, the Western Region performs quality reviews of information entered into the system , and walks through exceptions with SMOs as they arise.

Without adequate system controls, the information captured may not accurately reflect the results of compliance activities. As well, with significant manual interventions, the possible errors could also lead to significant losses in efficiency. Lastly, these potential issues could also lessen the reliability of information from supporting enforcement activities in the framework currently under development.

Recommendation 6 (High Risk)

The DGSO should implement quality control and assurance practices over compliance activities to ensure all information entered in the system is accurate and appropriately coded. As a long-term solution, the DGSO should also consider implementing an information management capability that can guide the processes for each compliance activity, and accurately and efficiently capture the required information for analysis, decision-making and reporting.

4.7 Equipment and fleet

An asset investment planning process is established and takes place annually. The process is led by NHQ in collaboration with the regions. Operational needs for specific assets such as radiofrequency and safety measurement equipment and vehicles are brought forward by the regional representatives through the National Equipment Working Group (NEWG). The acquisition of equipment and the fleet is centralized at NHQ, but involves regional staff for their input on specifications. There are dedicated service centres that perform the equipment and fleet modification and configurations based on the specifications and operational needs.

On a regular basis, the equipment and fleet are inspected visually following a standard procedure, and sent to the service centres for maintenance (e.g. calibrations, oil changes, tire inspections and changes). This follows the departmental policy and guidelines for acquiring and managing tangible assets.

While the fleet is maintained, they are kept and used longer than their recommended life span. The average age of the fleet in operations is 9.5 years old, while the existing guideline suggests a maximum age of vehicles-in-use as 7 years; the oldest vehicle in the fleet is 16 years old.

The meters that are used for measuring safety limits to support SC6 activities are calibrated according to manufacturer guidelines. The Spectrum Analyzers for analysing and investigating radio frequency usage, however, were purchased over 10 years ago, beyond the suggested investment guideline of replacement after five to seven years of service. In addition, 12 of 13 analyzers were not calibrated based on the recommended schedule. While analyzers do not need to be calibrated for all tasks performed by SMOs such as interference direction there are no controls in place to ensure calibrated analyzers are used for those compliance activities that required precise measurements, and there is no documentation maintained to support the traceability of which analyzers had been used after the fact. These analyzers are in use and continue to support the SMOs in carrying out other non-SC6 compliance activities. Further, the current investment plan does not provide details on the funding for equipment replacement and calibration.

Since the time of the audit, funding to support the Spectrum program and the move towards the adoption of newer technologies was announced in Budget 2019. The program is currently working toward implementing the future investment plans.

Without adequate and reliable equipment and fleet, the DGSO may not be able to effectively carry out their Spectrum compliance and investigative activities and meet the legislative requirements for ensuring the safe and efficient use of the Spectrum resource.

Recommendation 7 (High Risk)

The DGSO should implement a medium- and long-term investment plan to ensure the Spectrum compliance program can continue to meet its objectives, for both current technologies and those anticipated in the adoption of 5G services & technology.

4.8 Monitoring and reporting

On an annual basis, the DGSO prepares a national compliance report summarizing the overall achievement of the program, its performance against planned targets for compliance activities, and the general observations across the country. The report is presented to the DGSO's senior management team for review and discussion. Additionally, the regions produce semi-annual reports highlighting their ongoing performance against planned targets for specific compliance activities. These reports are presented to SOC for review and discussion. The three regional operations directors also held bilateral meetings on an ongoing basis to discuss their progress and overall findings-to-date in their regions.

The process to generate these reports is lengthy and requires extensive levels of effort. Because of the limitations in the IT system – where information is held in attachments – the compliance results and data cannot be generated automatically. Instead, results are manually collated and entered into a spreadsheet, and later emailed for review and national roll-up. This manual process also prevents continuous, moment-in-time reporting and analysis.

While the annual report is produced each year for senior management review, the information presented is largely limited to the volume of compliance activities completed; apart from the SC6 activities, compliance rates are not reported. As well, not all of the performance metrics outlined in the NCF are reported against the actual performance. While only two years of data were available at the time of the audit, reports did not compare results year-over-year. Moving forward, as more data becomes available, the program will also be in a position to identify possible trends for each compliance activity.

If Spectrum compliance program reports do not describe compliance rates and their year-to-year changes, the DGSO management team may not be able to determine the program's performance or progress toward the program's long-term goals and outcomes or make corresponding adjustments to activity planning.

Recommendation 8 (Medium Risk)

The DGSO should strengthen its annual reporting to include compliance rates for all standardized activities; year-over-year comparisons and trends; and performance against the established performance metrics.

4.9 Training and development

A national Spectrum Technical Training Committee (STTC), composed of regional representatives, is established to oversee the design and delivery of the formal technical training and development program to newly recruited SMOs. Key responsibilities of the committee include developing competency requirements and yearly training plans, and to establish the national course schedule, expected delivery formats and enrollment. The committee also updates course materials as needed, based on regional and NHQ inputs; the STTC is currently in the process of updating all of the training course modules. Training courses are delivered by National Designated Instructors (NDIs).

A formal new recruit training and development program known as Spectrum Management Officer Recruitment and Professional Development Program (SMORPDP) is in place and candidates' competency requirements are formally documented and have been incorporated into yearly performance appraisals. New recruits progress through a three-year period, during which they are expected to complete eight courses. On-the-job training such as job shadowing is also provided to the new recruits, allowing them to acquire technical knowledge and skills and gain first-hand experience.

However, there is no course module explicitly designed for the compliance program and its activities. While there is significant overlap between the training to perform Spectrum operations and compliance assessments, there are elements of the compliance program that remain unique, and are not taught in the classroom.

Without clearly defined training related to the Spectrum compliance program, staff may lack the required knowledge to carry out their compliance duties and tasks effectively and efficiently.

Recommendation 9 (Low Risk)

The DGSO should strengthen its training program to include compliance activity training.

4.10 Continuous improvement

The Spectrum Operations Management groups in the regions share good practices and lessons learned through their regional operational committees on a regular basis. Additionally, three regional Spectrum Program Management directors hold regular informal meetings to discuss and address challenges and lessons learned from implementing the national compliance framework.

The representatives of the regional operational committees and working groups also attend the national committees and working groups such as the SOC, STTC, and NCEWG, where they bring forward discussion on notable good practices, lessons learned from special cases and other improvement opportunities for endorsement. Once approved, changes are then implemented across the entire compliance program. Further, a dedicated team of staff at NHQ are also responsible for reviewing and updating the national procedural guidelines to reflect all proposed changes, as well as to incorporate regional feedback.

In the regions, wikis are used extensively by staff to share information: processes, operational procedures, and guidelines are documented and regularly updated in the wikis. Key contact information is also provided for staff to reach out to subject matter experts if additional technical assistance is needed, and staff can post questions on the wiki in order to seek clarification from their colleagues.

Management in all three regions, as well as NHQ, regularly collaborate in order to share information and ideas for future activities. Internal procedural circulars for compliance activities that are nationally standardized are reviewed periodically and updated to reflect process changes, technical changes, and regional feedback.

4.11 Management response and action plan

The findings and recommendations of this audit were presented to the Director General, Spectrum Management Operations Branch (DGSO), and representatives from the DGSO management team. Management has agreed with the findings included in this report and will take action to address all recommendations by March 31, 2024.

Management Response and Action Plan

5.0 Overall conclusion

The Spectrum program is making progress in implementing its five-year National Compliance Framework. Future efforts will be required in order to ensure a nationally consistent approach to all compliance activities, including enhancing the methodology for planning compliance activities, further defining procedures, building quality controls over the capture of information, and producing more comprehensive annual reporting.

Appendix A: Audit criteria

Audit of the Spectrum Compliance

Appendix B: National compliance framework activities