supply chain pilot project

RELIABILITY | RESILIENCE | SECURITY

NERC | Report Title | Report Date I

Supply Chain Pilot Project Network Interface Controller Identification

NERC Staff Report – June 2021

NERC | Supply Chain Pilot Project: Network Interface Controller Identification | NERC Staff Report | June 2021 ii

Table of Contents

Preface ........................................................................................................................................................................... iii

Executive Summary ........................................................................................................................................................ iv

Recommendations and Next Steps .......................................................................................................................... v

Introduction ................................................................................................................................................................... vi

Chapter 1: Project Overview ........................................................................................................................................... 1

Chapter 2: Example ......................................................................................................................................................... 2

Question 1 ................................................................................................................................................................... 2

Questions 2 and 3 ........................................................................................................................................................ 2

Questions 4, 5, and 6 ................................................................................................................................................... 2

Question 7 ................................................................................................................................................................... 3

Question 8 ................................................................................................................................................................... 4

Appendix A: Survey Process with Questions ................................................................................................................... 7

Appendix B: Survey Responses ....................................................................................................................................... 9

NERC | Supply Chain Pilot Project: Network Interface Controller Identification | NERC Staff Report | June 2021 iii

Preface Electricity is a key component of the fabric of modern society and the Electric Reliability Organization (ERO) Enterprise serves to strengthen that fabric. The vision for the ERO Enterprise, which is comprised of the North American Electric Reliability Corporation (NERC) and the six Regional Entities (REs), is a highly reliable and secure North American bulk power system (BPS). Our mission is to assure the effective and efficient reduction of risks to the reliability and security of the grid.

Reliability | Resilience | Security Because nearly 400 million citizens in North America are counting on us

The North American BPS is made up of six RE boundaries as shown in the map and corresponding table below. The multicolored area denotes overlap as some load-serving entities participate in one RE while associated Transmission Owners (TOs)/Operators (TOPs) participate in another.

MRO Midwest Reliability Organization

NPCC Northeast Power Coordinating Council

RF ReliabilityFirst

SERC SERC Reliability Corporation

Texas RE Texas Reliability Entity

WECC WECC

NERC | Supply Chain Pilot Project: Network Interface Controller Identification | NERC Staff Report | June 2021 iv

Executive Summary Over the past several years, authorities have recognized that supply chain vulnerabilities are a growing risk to national security. The U.S. government directed its agencies to not use specific equipment manufactured by companies that are geographically located in nations that are considered foreign adversaries. Recently, the Federal Communications Commission (FCC) published a list of communications equipment and services (Covered List) that are deemed to pose an unacceptable risk to the national security of the United States.1 The Covered List specifically cites telecommunications equipment produced or provided by Huawei [Technologies Company] or ZTE [Corporation] and video surveillance and telecommunications equipment produced by Hytera Communications Corporation, Hangzhou Hikvision Digital Technology Company, and Dahua Technology Company. While these restrictions are mandatory for U.S. federal agencies, similar protective measures are also recommended to the private sector. NERC responded to these developments by issuing three alerts to industry, seeking data to determine the magnitude of threat that foreign manufactured equipment posed to the BPS. Industry responses to those alerts suggested that embedded technologies from foreign manufacturers could be present in their environment, but the data response itself did not provide sufficient detail to support definitive conclusions. NERC recently undertook a collaborative engagement with industry stakeholders to understand the pervasiveness of Huawei and ZTE components in industry networks; this document describes the results of that effort. The project leveraged a 2020 joint report2 from NERC and the Federal Energy Regulatory Commission (FERC) that described noninvasive techniques that entities could use to identify device manufacturers. The joint report focused specifically on the network interface controller (NIC), a device common to many types of electronic equipment in both the information technology (IT) and operational technology (OT) environments. While the primary intent of the collaboration project was to evaluate the prevalence of foreign manufactured equipment in the North American (NA) electricity sector, it also was expected to provide lessons learned and best practices that industry can employ to identify and mitigate risks that are associated with the use of foreign adversary technologies. This document describes processes for acquiring data, assesses the foreign manufactured equipment present in the NA electricity sector, and concludes with relevant observations and recommendations to the electric industry. The following are key findings from this project:

• Little to no exposure to operational networks: More than 425,000 NICs were analyzed by study participants, and no NIC controllers manufactured by Huawei or ZTE were discovered on operational networks. These results align with the findings gleaned from responses to the NERC alerts. In one corporate (non-operational) environment, wireless devices from those manufacturers were identified, but mitigating factors make those results less significant.

• Information about subsidiaries: Unless a subsidiary used a name that was a derivative of the two manufacturers, any affiliation with those companies was difficult to ascertain. No list of subsidiary or partner companies was available for participants’ reference.

• Network Analysis: Entities were encouraged to analyze networks they deemed to be appropriate for the purposes of the study. Ultimately, participants analyzed operational networks by more than a two to one margin over the number of corporate IT networks. Some entities tested multiple networks with many of the analyzed OT networks being operational.

1 https://docs.fcc.gov/public/attachments/DA-21-309A1.pdf 2 Joint Staff White Paper on Supply Chain Vendor Identification - Noninvasive Network Interface Controller: https://www.nerc.com/pa/comp/CAOneStopShop/Joint%20Staff%20White%20Paper%20on%20Supply%20Chain_07312020.pdf

https://www.nerc.com/pa/comp/CAOneStopShop/Joint%20Staff%20White%20Paper%20on%20Supply%20Chain_07312020.pdf

v

• Project Value: Several participants expressed the opinion that scanning networks for specific equipment manufactured in foreign adversary nations was helpful and could be valuable as on ongoing process for identifying risks that would otherwise be undetected or unrecognized.

• Participation: Participants represented a variety of organization sizes, types, locations, and regions. The ratio of entities aware of the project compared to those that participated was lower than expected: 22% (17 out of 78). Nonetheless, the study results are consistent with the responses from the earlier NERC alerts.

Recommendations and Next Steps The following recommendations were derived from the key findings of the project:

• Tools and Practices: Entities should use automated tools and management practices to actively monitor networks for evidence of unacceptable risks and malicious activity. Such an approach would be especially helpful for detecting equipment from organizations designated by authorities as known foreign adversaries. Monitoring activities should include procurement and commissioning processes.

• Data collection: Future efforts to collect data for this type of study should be organized via the NERC alert process with mandatory reporting. Alerts should be detailed enough so responses include the quantity and quality of information needed for a complete analysis.

• Adversarial Data: Network hardware is assigned a unique identifier called the media access control (MAC) address. A portion of that address is the organizational unique identifier (OUI), which identifies a device’s manufacturer. A centrally managed list of relevant MAC addresses and OUIs would help entities identify equipment manufacturers and subsidiaries that are located in adversarial countries. Several study participants recommended that such a resource with consistent, timely, and accurate information would be readily usable.

• Participation: Technical groups, such as the Security Integration and Technology Enablement Subcommittee (SITES) and Security Working Group (SWG), should develop work products that describe best practices for security posture without focusing only on compliance activities.

NERC | Supply Chain Pilot Project: Network Interface Controller Identification | NERC Staff Report | June 2021 vi

Introduction Government agencies and industry security experts have expressed concerns for several years about potential supply chain threats posed by equipment manufacturers associated with foreign adversaries.3 These reports have cited evidence of backdoors or security vulnerabilities in a variety of devices and published these findings in the following examples: In 2012, a report from a committee of the U.S. House of Representatives recommended against use of equipment manufactured by Huawei or ZTE, two Chinese telecommunication companies that were cited because of their close ties to the Chinese government. The report emphasized concerns that equipment from these manufacturers could be used to surreptitiously intercept communications.4 A 2013 report from the U.S. Government Accounting Office, “Addressing Potential Security Risks of Foreign-Manufactured Equipment,” highlighted ways to exploit vulnerabilities in the communications equipment supply chain by injecting malicious code in network components.5 A 2019 publication that was prepared by the Defense Innovation Board, “Risks & Opportunities for DoD” that highlighted threats posed by China and other nation-state adversaries as wireless technology evolves to the fifth generation (i.e., “5G”).6 To address these threats, government legislative and executive actions imposed restrictions on the purchase and use of telecommunications equipment from specified manufacturers for certain critical programs and industries. Among the most significant of these actions was Executive Order (E.O.) 13920, “Securing the United States Bulk-Power System”,7 which was issued in 2020. The concerns highlighted by that Order were also addressed in an FCC list of communications equipment and services (Covered List) that are deemed to pose an unacceptable risk to the national security of the United States.8 While EO 13920 was suspended for 90 days by E.O. 13990 and ultimately revoked on April 20, 20219, the underlying security concerns remain relevant to the findings from the pilot project and the development of this report. As supply chain threats became more conspicuous and far-reaching, the need for industry response became increasingly apparent. NERC issued alerts in 2017, 2019, and 2020 that included strong recommendations for NERC registered entities. The alerts focused on gathering the extent of condition information regarding the risk of equipment installed on the BPS that was manufactured or supplied by certain foreign entities of concern.10 Each alert

3 The U.S. Department of Energy (DOE) defines foreign adversaries as “any foreign government or foreign nongovernment person engaged in a long-term pattern or serious instance of conduct significantly adverse to the national security of the U.S. or its allies or the security and safety of U.S. persons.” The DOE has identified China, Russia, Iran, Cuba, North Korea, and Venezuela as foreign adversaries. Refer to: https://www.govinfo.gov/content/pkg/FR-2020-07-08/pdf/2020-14668.pdf 4https://republicans-intelligence.house.gov/sites/intelligence.house.gov/files/documents/huawei-zte%20investigative%20report%20(final).pdf 5 https://www.gao.gov/assets/gao-13-652t.pdf 6 https://media.defense.gov/2019/Apr/03/2002109302/-1/-1/0/DIB_5G_STUDY_04.03.19.PDF 7 https://trumpwhitehouse.archives.gov/presidential-actions/executive-order-securing-united-states-bulk-power-system/ 8 https://docs.fcc.gov/public/attachments/DA-21-309A1.pdf 9https://www.federalregister.gov/documents/2021/01/25/2021-01765/protecting-public-health-and-the-environment-and-restoring-science-to-tackle-the-climate-crisis 10 https://www.nerc.com/pa/rrm/bpsa/Pages/Alerts.aspx

https://www.govinfo.gov/content/pkg/FR-2020-07-08/pdf/2020-14668.pdf

https://republicans-intelligence.house.gov/sites/intelligence.house.gov/files/documents/huawei-zte%20investigative%20report%20(final).pdf

https://republicans-intelligence.house.gov/sites/intelligence.house.gov/files/documents/huawei-zte%20investigative%20report%20(final).pdf

https://www.gao.gov/assets/gao-13-652t.pdf

https://media.defense.gov/2019/Apr/03/2002109302/-1/-1/0/DIB_5G_STUDY_04.03.19.PDF

https://trumpwhitehouse.archives.gov/presidential-actions/executive-order-securing-united-states-bulk-power-system/

https://docs.fcc.gov/public/attachments/DA-21-309A1.pdf

https://www.federalregister.gov/documents/2021/01/25/2021-01765/protecting-public-health-and-the-environment-and-restoring-science-to-tackle-the-climate-crisis

https://www.federalregister.gov/documents/2021/01/25/2021-01765/protecting-public-health-and-the-environment-and-restoring-science-to-tackle-the-climate-crisis

https://www.nerc.com/pa/rrm/bpsa/Pages/Alerts.aspx

Introduction

vii

focused on specific supply chain concerns regarding the BPS and the associated cyber assets11 that may have been manufactured or supplied in countries of specific concern. NERC and FERC jointly published a white paper, “Supply Chain Vendor Identification – Noninvasive Network Interface Controller,” in July 2020.12 The report provided example approaches of assessing networks for the deployment of foreign adversary components that could be used to impact the BPS, specifically focusing on the network interface controller (NIC). Huawei and ZTE hold a significant global market share of network devices and often resell networking subcomponents (e.g., the NIC) that are often unlabeled (or “not branded”) yet were suspected to be embedded in cyber assets that may be associated with BPS operational technology (OT) systems. The pilot project described in this white paper and the results obtained from the project are intended to follow-up on previous activities by obtaining details about the pervasiveness of possible NICs within utility informational technology (IT) or operational technology (OT) networks.

11 For example, BES Cyber Assets, Electronic Access Control or Monitoring Systems, Physical Access Controls Systems, Protected Cyber Assets. 12 https://www.nerc.com/pa/comp/CAOneStopShop/Joint%20Staff%20White%20Paper%20on%20Supply%20Chain_07312020.pdf


NERC | Supply Chain Pilot Project: DRAFT Network Interface Controller Identification | NERC Staff Report | April 2021 1

Chapter 1: Project Overview Rather than seeking project volunteers by publishing a general announcement to industry, NERC selected a set of entities that was reasonably representative of the overall industry based on criteria like geographic area, Regional Entity footprint, reliability function(s), size, and Interconnection. NERC staff worked with subject matter experts (SMEs) from these organizations and engaged with Regional security groups to garner additional participation from other interested Registered Entities. Project information was provided via emails and formal presentations, describing the purpose and process for conducting the study and encouraging participation in order to support the ERO’s efforts to gather useful data for the analysis. The SMEs were informed that participation was strictly voluntary and that any submitted information would be anonymously reported. NERC held an informational webinar on November 30, 2020, with representatives from entities that had been asked to participate. The webinar highlighted the reason for the pilot study, elaborated on the recommendations from the FERC-NERC report regarding NIC identification techniques, described the questionnaire, and provided a project time line. The ratio of entities that chose to participate in the study as compared to the number that were contacted was lower than expected. Only 22% (17 out of 78) of the entities contacted actually participated in the study by compiling data and completing the questionnaire. Table 1 indicates the participation rate and regional representation for this pilot project.13 The participants included generation, transmission, and distribution entities. Regional representation was inconsistent; for RF and SERC, the response was relatively high while NPCC and WECC were low. There was no explanation from non-participants about their decision, but our conclusion is that “volunteering” was perceived to be expensive, time consuming, and/or a compliance risk (since “NERC” was requesting the data). Nonetheless, results from those that did participate and the representative environments that were analyzed does indicate that findings from the alert data responses are consistent with the results of this more detailed analysis.

Table 1.1: Overview of Contacts/Participants by Regional Entity Region Entities Contacted Entities Volunteered Participation Rate Regional Representation

MRO 33 7 21% 41%

NPCC 12 0% 0%

RF 7 3 43% 18%

SERC 10 4 40% 24%

Texas RE 6 2 33% 12%

WECC 10 1 10% 6%

Totals 78 17 22% 100%

Entities were asked to analyze the network(s) they considered to be applicable, and to provide questionnaire responses to NERC via a secure portal. NERC compiled the questionnaire results, aggregated and anonymized the information, and analyzed the responses to produce this report. 13 Entities that operate in multiple Regional Entities are reported according to the location of the environment tested for this pilot project.


Chapter 2: Questionnaire Responses This section provides a brief synopsis and analysis of information collected during the pilot project. Appendix A provides the list of survey questions and Appendix B contains the aggregate data collected. Question 1 Each participant was asked to assess one or more networks in support of the project. While OT networks (test, development, or operational) were recommended and preferred, entities were encouraged to use any network(s) they considered suitable and attainable. The goal was to increase information collection and participation from industry rather than to restrict participants to a specific type of network that was not necessarily practical. Results showed that participants evaluated NICs on OT networks versus corporate IT networks by more than a two-to-one margin with many entities able to obtain data from operational environments. In addition, several participants analyzed multiple networks. However, reports also indicated that substation networks were typically not included in the results due to the impracticality traveling to a number of physical locations. Figure B.1 provides more details. Questions 2 and 3 The FERC-NERC joint white paper14 on noninvasive techniques for NIC discovery provided examples of possible methods for performing assessments to include ARP scan using NMAP, an ARP cache table,15 the DHCP client table, and port mirroring. Participants were asked in Question 2 to indicate what method was used for performing the assessment. Similarly, Question 3 asked about the specific tools or techniques used to perform the assessment, citing examples that included Wireshark, NMAP, SolarWinds, GFI LanGuard, or some other type of tool. More than half the participants reported using a NIC discovery method other than those previously mentioned. These methods included accessing switches and gateways to list active ARP tables and using asset management tools that used methods such as those mentioned above or with custom software. Multiple entities stated that the use of existing network management tools and records offered a significant advantage in performing the analysis, both in terms of time spent on data collection as well as the usability of the information that was obtained. For example, one entity reported that their intrusion detection system had replicated OT network traffic, thereby enabling them to look at more than 20 different operational OT environments. One participant stated that the use of IT system command line interface (CLI), such as NMAP commands, helped them to effectively detect all component hardware information that existed on devices. Questions 4, 5, and 6 Questions 4–6 pertained to quantities, asking participants to report the number of NICs that were assessed and a breakdown of how many were attributed to Huawei or ZTE by number and type. Equipment manufactured by ZTE and Huawei use nearly 1,200 MAC addresses, and some participants limited their analysis to searching for those specific addresses. Overall, study participants indicated that more than 425,000 unique MAC addresses were analyzed, however. Since corporate IT networks are typically far larger than OT networks and with fewer disparate locations, the majority of MAC addresses unsurprisingly came from IT networks. Nonetheless, study participants were able to assess many OT operational, development, and test environments as well, so it is believed that the results presented in this report are a fair representation of the numbers and types in entities across North America. See Table B.1 for details. 14 https://www.nerc.com/pa/comp/CAOneStopShop/Joint%20Staff%20White%20Paper%20on%20Supply%20Chain_07312020.pdf 15 Some entities mentioned that ARP cache tables could be manipulated or poisoned. However, since the devices tested or ARP tables in question are typically inside the ESP, there should be reasonable assurance that the tables have not been modified. Entities concerned with ARP cache poisoning should assess their network security to mitigate this possibility.


Chapter 2: Questionnaire Responses

3

Two entities detected Huawei or ZTE equipment during their analyses; in both cases, the components were associated with wireless devices (i.e., cellular phones and mobile hotspots) on corporate (i.e., non-operations) networks. These results were not deemed to be statistically significant, especially since the vast majority of the detected devices appeared on system logs that had been retained on a participant’s network management device for five years and pertained to a wireless guest network. The organization does not support Android devices so they surmised that their NetDisco utility detected mobile devices that “auto-connect[ed] to any open network.” None of the study participants indicated that they found subsidiaries of Huawei or ZTE. One participant’s response was succinct: “[If] subsidiaries used names that were not derivatives of the two main companies, their affiliations were difficult to ascertain.” Another participant provided more details: “… searching for subsidiaries took some effort, and after several [Internet] searches for who those subsidiaries actually are yielded several different answers. Had to spend some time figuring out the ’best‘ list to use and then turn those into OUI’s that could be searched against. While most of the subsidiaries (at least in the list I used) had names that were derivatives of the two main companies, there were several that were not and I wanted to make sure I could confidently assert that I had indeed looked for ‘their subsidiaries’.” Question 7 Each entity was asked to describe the resource commitments associated with their analysis (i.e., how much time was required). Participants with network asset management tools and records indicated that it was relatively easy for them to extract data by using automated tools or scripts that were already available. This response was most common with the transmission entities and OT environments where elevated levels of network visibility and scanning capabilities appear to exist. Some entities reported recent improvements to generating assets that provided the means to extract this information although they estimated that the effort would have taken hundreds of hours for travel time, coordination, and data extraction without that ability. For entities that reported a rapid turn-around time, the majority of them noted that the time devoted to the project was dedicated primarily to data filtering and formatting with the readily available output from an entity’s network management system or process used to compare to relevant MAC addresses. Figure B.4 reflects entity reports regarding man hours. Most were able to perform the network scan, analyze results, and complete the questionnaire in under 20 total man hours with the most time dedicated to correlating extracted data with specific MAC addresses that was provided on a separate list. However, two entities did indicate that they had to spend a significant amount of time on the process because the tool’s output could not be used to automate the process for extracting and analyzing data. Some entities indicated that time constraints made it difficult to perform a more thorough evaluation. Explanations included competing priorities that made it difficult for a limited number of personnel to devote more time to the project or because the organization had to initiate a separate process or involve multiple departments to obtain data because network asset management tools, records, and processes were not available to facilitate data collection more effectively.


4

Question 8 The project questionnaire concluded with a multi-part open-ended question regarding lessons learned, recommendations to industry, and next steps that might result from being part of the project. The following are key takeaways from information provided by the respondents:

• Unexpected Components: Participants reported finding no unexpected components in their analysis, but one entity did provide information about equipment used in their environment that came from other Chinese manufacturers. These companies have no known affiliation with Huawei or ZTE; however, the entity was aware that the devices were in use and although there has been no indication of increased risk, they wanted to share the information.

• Tools and Expertise: Overall, data gathering was reported to be relatively easy and non-intrusive with respondents indicating that they had the tools and expertise needed to perform the analysis. Several stated that they were able to leverage existing network tools and system records to complete their report. Recommendations for enhancing the process include the following:

A centrally managed list of MAC addresses for equipment manufacturers and subsidiaries that are located in adversarial countries: The information would be valuable for device on-boarding checklists to verify that something has not slipped through the purchasing process. However, the MAC address information would have to be continually maintained by a trusted source and readily available to users, or the process would quickly become irrelevant.

Authoritative organizationally unique identifier (OUI) listings for vendors in a format that is readily usable: When there are a large number of addresses to analyze, an automated search process would be the most practical approach. Thus, a list formatted so it could be imported to a spreadsheet and used as a lookup table would be far more useful than a manual “stare and compare” process.

• Obstacles or Challenges: In their analyses, participants noted the following challenges they faced as they gathered and reviewed data:

Segmented networks required configuration changes to permit Orion/SolarWinds commands to obtain the needed information.

In at least one case, disabling SolarWinds caused the analysis to take days longer than would have otherwise been the case. Previously, the use of SolarWinds Device Tracker could have reported the relevant data and possibly enabled the analysis to be complete in less than an hour.

Information that was obtained required parsing to be useful in analysis.

There were instances when it was not possible to associate host names with the IP addresses and MAC addresses; when this occurred, additional research was necessary to identify and report accurate results. Similarly, one participant was uncertain about the accuracy of results because data in the NMAP-MAC-Prefixes file was incomplete or unclear without another source available. The participant did further analysis for unresolved MAC address prefixes.

A participant noted that they faced scheduling conflicts with competing security projects since those efforts also required high-level expertise and resources. Those conflicts had to be resolved before the NIC project could be completed.

While many study participants found the availability and use of network management tools helpful in their analyses, one entity indicated that their tool required a manual look-up of each NIC; this took a considerable amount of time.

• Unexpected Results or Findings during Analysis: Most participants indicated no unexpected results or findings from their analysis, but there were a few comments, listed here:


5

There were instances where private space MAC addresses were in use or MAC addresses were detected that did not associate with known IEEE OUIs; participants did not indicate how or whether further analysis informed the results contained elsewhere in their questionnaire responses.

MAC address randomization was observed for some devices that utilize it for privacy (e.g., iPhones, and Windows 10 devices).

The question regarding components “attributed to Huawei, ZTE, or any of their subsidiaries” was not easy to answer. Where subsidiaries used names that were not derivatives of the two main companies, their affiliations were difficult to ascertain. As a consequence, more research was needed to determine the “best” list of relevant OUIs. If industry could rely on a single trusted source that maintained relevant and current information about corporate relationships, industry could use it to make informed decisions about supply chain risks in their own environments.

• Plans for Periodic Assessments: A query regarding “next steps” was included in Question 8 to gauge the impact of the study on its participants. Responses ranged from planning future assessments only “as needed or when directed” to changing existing practices to incorporate assessment as a regular practice. The following were the responses:

The study was informative and useful.

There are existing practices to continually monitor equipment in the organization’s IT environment on an automated basis while manual spot checks are used in the OT environment.

A similar analysis would be a useful addition to the annual vulnerability assessment that is mandated by CIP Reliability Standards.

NIC manufacturer information will be added to the cyber asset inventory attributes so it can be part of the analysis process prior to installation.

Monitoring tools and techniques that are in place are sufficient for detecting potentially malicious activity, adding an assessment similar to what was done for the study is not necessary.

Unless the assessment process is automated, making it repeatable is unlikely.

Even if a similar analysis was scripted and performed periodically to detect "suspicious" MAC addresses, a sophisticated attacker could and would easily disguise MAC addresses to appear “safe.”

Tools and processes already monitor networks and equipment that would detect the addition of new assets or changes to existing assets.

• Impact on Procurement Process: While the previous question pertained to the impact that participation in the study could have on network and equipment management practices, study participants were also asked whether their supply chain/procurement process could be affected as well. Several participants had considered that move, as indicated by these responses:

A data analytics tool like Microsoft Power BI could be used to create a data model for future reviews that can condition data outputs from the scripts run in Orion and matched to MAC OUIs.

Adding manufacturer verification as a step in the predeployment commissioning process would be especially appropriate for cyber assets that are subject to compliance with CIP Reliability Standards.

Internal processes will address future acquisitions of technology from any foreign country to ensure there is no undetected or unacceptable risk.


6

Information gleaned from acquisition and supply chain activities will be shared with industry via a peer sharing process (e.g., regional security group discussions or within vendor-supported tools such as the Asset to Vendor Network16).

The same data could be leveraged to identify NIC vendors on new hardware during the procurement process, prior to being introduced to production environments.

There are no plans to change the procurement process at this time, but that decision will be reevaluated as more information is received from NERC, the E-ISAC, the Department of Homeland Security, etc.

16 Asset to Vendor Network is a mutual assistance platform for utilities that share the cost of vendor risk assessments and cyber asset vulnerability patches and solutions to reduce duplication and meet compliance requirements. It is a subscription service not endorsed by NERC; information is listed here for information purposes only.


Appendix A: Survey Process with Questions

1. What type of network environment was assessed? Please select the option that best applies.

a. OT Test Environment

b. OT Development Environment

c. OT Operational Environment (test prior to deployment)

d. Corporate IT Environment (test prior to deployment)

2. What noninvasive technique was used to analyze your network?

a. NMAP ARP Scan

b. ARP Cache Table

c. DHCP Client Table

d. Port Mirroring

e. Other (please specify)

3. What application or tool was used to analyze your network (select all that apply)?

a. Wireshark

b. NMAP

c. SolarWinds

d. GFI LanGuard

e. Other (please specify)

4. How many components (i.e., total number of unique MAC addresses) were assessed as part of the analysis?

5. How many components (i.e., unique MAC addresses) were attributed to Huawei, ZTE, or any of their subsidiaries within the network that was analyzed?

6. Please note the number of equipment, devices, or components that were identified for each manufacturer listed in [the table].

7. Please describe the resource commitments associated with this analysis (i.e., man hours, on-site time required, etc.).

8. Please describe any lessons learned from this exercise or any opportunities for enhancement to these types of network analysis techniques. Please address each of the following questions:

a. Were any other unexpected components found during testing that NERC and industry should be made aware of that may be manufactured or supplied by foreign adversaries?

b. Did you have the appropriate tools and expertise available to effectively complete this analysis?

c. Were there any obstacles or challenges you encountered when performing the analysis?

d. Did anything unexpected occur during the analysis?

Appendix A: Survey Process with Questions


e. Do you plan to periodically assess your environment using any of the techniques used above?

f. Have you considered evaluations for discovery of network components in your supply chain procurement process?


Appendix B: Survey Responses This appendix provides the aggregated responses from the survey as well as the key takeaways for each question. The values in the charts that follow reflect the aggregated information that was reported by the entities that participated in the pilot project. Question 1 “What type of network environment was assessed?”

Figure B.1: Responses to Question 1

Question 2 “What noninvasive technique was used to analyze your network?”


Key Takeaway–Question 1: OT networks comprised approximately two-thirds of the results including OT testing, development, and operational environments. The remaining one-third of results included IT environments.

Appendix B: Survey Responses


Question 3 “What application or tool was used to analyze your network?”


Key Takeaway–Question 2: The most common method for network scanning involved gathering information on ARP cache tables, followed by use of NMAP ARP scanning, DHCP client tables, and port mirroring. A number of entities reported that the network asset management tools they presently use are able to quickly identify MAC addresses that could then be correlated to specific equipment manufacturers.

Key Takeaway–Question 3: Most entities network asset management tools other than the tools listed as examples. These were heavily leveraged by entities to gather the data being requested, and some manual formatting and analysis of the data was required to compare the MAC address information with the MAC addresses under question.



Questions 4 and 5 “How many components (i.e., unique MAC addresses) were assessed as part of the analysis?” “How many components (i.e., unique MAC addresses) were attributed to Huawei, ZTE, or any of their subsidiaries within the network that was analyzed?”

Table B.1: Responses to Questions 4 and 5

Entity

Q4: Number of Components Q5: Attribution

# of MAC Addresses Analyzed

OT Test

OT Dev [or DMZ]

OT Op

Corporate IT

Huawei ZTE Subsidiaries

A 3,111 1,258 1,853 0 0 0

B 12,300 X X X X 0 0 0

C 1,218 1,218 0 0 0

D 115,982 X X 165 447 0

E 18 X X 0 0 0

F 580 X X X 0 0 0

G 9,304 811 300 8,193 0 0 0

H 250,000 250,000 0 0 0

I 6,040 6040 0 0 0

J 1,181 X X 0 0 0

K 19,455 4,966 14,489 0 0 0

L 71 0 0 0

M 107 107 0 0 0

N 5,359 5,359 0 0 0

O 192 X X 0 0 0

P 615 X X 0 0 0

Q * X X X X 6 0 0

Total 425,553 171 447 0



Question 6 “Please note the number of equipment, devices, or components that were identified for each manufacturer listed in [the table].” Only two entities reported identifying any NICs or other devices manufactured or supplied by Huawei, ZTE, or their subsidiaries, as shown in Table B.1. Entity D extracted a report from a network management tool spanning over five years. 612 unique MAC addresses were identified and all “appear to have belonged to devices only seen on [the] guest wireless network.” Entity Q reported that all six Huawei devices were LTE E8372 hotspot turbo sticks. Both OT and corporate IT networks were scanned by Entity Q; however, it was not specified which network contained these devices.

Question 7 “Please describe the resource commitments associated with this analysis (i.e., man hours, on-site time required, etc.).”


Key Takeaway–Questions 4 and 5: Over 425,000 unique MAC addresses were tested as part of this pilot across 17 entities. While the majority of MAC addresses were obtained from corporate IT networks, many different OT operational, development, or test environments were included in the pilot project. Only two entities identified any Huawei or ZTE components. One entity noted that this was from a wireless guest network and the other noted that the six devices identified were mobile hotspot devices.

Key Takeaway–Question 6: Two entities identified equipment from Huawei, ZTE, or their subsidiaries. One entity reported that, over a five year time frame, about 600 unique MAC addresses had connected to their guest wireless network. The other entity reported six Huawei mobile hotspot devices. Both of these entities reported wireless connectivity devices as the only equipment from these suppliers.



Question 8 “Please describe any lessons learned from this exercise or any opportunities for enhancement to these types of network analysis techniques.” A recap of the responses regarding Question 8 is provided in the analysis section of this report. Raw survey data is not provided here for confidentiality reasons.

Key Takeaway–Question 7: Most entities were able to perform the network scan, analyze results, and complete the questionnaire in under 20 total man hours. These entities had automated tools to extract the necessary information quickly and most time was dedicated to correlating the extracted information with specific MAC addresses. Two entities noted a significant amount of time needed to extract. The primary cause of the increased time was that some tools required manual searching against the MAC addresses of interest while most tools had automated capabilities.

supply chain pilot project

Documents