IoT Solar Inverters & Trickle-Down Vulnerabilities

3 September 2019, by Jos Wetzels, Principal Security Consultant at Secura

Omniksol

Omnik is a Chinese-German manufacturer of photovoltaic (PV) inverters and accessoires popular in the Netherlands, Germany and Belgium.

The Omnik inverters have two RJ-45 ports through which up to 50 inverters to be connected in series in order to allow for configuration and data transfer over RS-485. In addition, Wi-Fi and GPRS connectivity modules are available as either internal cards or external kits (connected to the inverter via RS-485) for remote status monitoring, datalogging and configuration of an entire inverter cascade.

Keep in mind that these are only the public accounts and only the ones visible in the Omnik portal. Other brands reselling the SolarMAN solution might utilize their own portals.

In addition to the above map of Wi-Fi kit users, we can find dozens of them publicly exposed over the internet by using the Shodan and Censys search engines:

The SolarMAN / iGEN Wi-Fi Kit

Alright, let's take a closer look at the Wi-Fi kit. The first thing to notice is that the open AP is the default option. While the setup allows you to 'hide' the AP, set WEP / WPAPSK / WPA2PSK encryption and change the web server username & password it doesn't require the end user to do so and as such the device seems typically deployed insecure-out-of-the-box. In addition, if the open AP is merely used for initial device setup it is not disabled afterwards leaving it open on many 'setup and forget' devices.

So what can we do with access to the AP here? Well, first of all it allows us to interact with the PV inverters via various interfaces about which more later. Secondly the Wi-Fi kit needs to be able to connect to the internet in order to communicate with the SolarMAN cloud backend which ingests the inverter data and makes it available to you via a mobile app. As such the kit is connected to an internal network either via Ethernet cable or a Wi-Fi AP (for which the SSID and credentials need to be entered in the web configuration interface). Apart from the fact that this means trusting the cloud backend with both your PV inverter data and potentially your personal Wi-Fi credentials, it also means that the kit's own AP now acts as a point of entry into your private network. A risk affecting many IoT devices.

In order to get a better idea of the kit's security posture, i decided to take a look at its various services. A quick nmap scan reveals the MAC address corresponds to the Shenzhen-based Drogoo Technology Co. and the following services are exposed:

* 53/UDP, DNS
* 80/TCP, HTTP
* 8899/TCP, Datalogger Info
* 48899/UDP, HF AT Interface

Web Interface (80/TCP)
On port 80 we find a web server which, after a HTTP basic authentication prompt is passed, provides a configuration interface supporting typical configuration options (settings for wireless & cable network, datalogger, cloud server, serial communications) as well as the ability to change the web interface username & password and upgrade both the Wi-Fi kit and inverter firmwares.

Datalogger Info (8899/TCP)
The service on this port is used for datalogger communications for which there exist several open-source community scripts (http://www.mb200d.nl/wordpress/2015/11/omniksol-4k-tl-wifi-kit/, https://github.com/Woutrrr/Omnik-Data-Logger, https://github.com/XtheOne/Inverter-Data-Logger) but from a security perspective this service is not terribly interesting so i decided not to investigate any further.

HF AT Interface (48899/UDP)
So let's assume that the end user configured a strong password for the web interface and we don't want to go the route of memory corruption exploitation in any of the exposed services. In that case this interface, which i called the 'HF AT interface' (for reasons that will become clear later), seems like our best shot.

The interface is used for network discovery by responding with identifying information to specific UDP broadcast messages containing the string "WIFIKIT-214028-READ" as can be seen when reverse engineering the SolarMAN mobile app:

The FCC ID seems to belong to the HF-A11 embedded Wi-Fi module manufactured by Shanghai Hi-Flying Technology. The presence of this module, which is based on the MIPS architecture and runs the eCos RTOS, can be observed on the kit's PCB after opening the enclosure:

In addition, a quick analysis of the kit's firmware confirms it runs eCos on MIPS and shows Hi-Flying and HF-A11 strings:

Hi-Flying

Shanghai High-Flying Electronics Technology Co., Ltd is a wireless communications products manufacturer and cloud services provider who produce various IoT modules and devices. These modules and devices are also sold by other vendors such as USR IOT and are adopted by various low-cost IoT ODMs and OEMs in a wide variety of products ranging from smart bulbs such as the well-known MiLight, smart power plugs, webcams, coffee machines and security alarms to industrial RTUs, serial-to-ethernet converters, gateways and solar inverter data loggers all of which are again rebranded by sprawling networks of company names.

A slide from a Hi-Flying company profile presentation shows some prominent international customers (including Omnik) and a broad portfolio of modules:

These modules typically consist of an off-the-shelf wireless communications chip (eg. the MediaTek MT5931) bundled together with an RTOS (eg. eCos, FreeRTOS, Contiki, ARM mbed) or Linux, several communications and networking libraries, embedded web, ftp and telnet servers and proprietary network discovery, remote configuration and OTA firmware update functionality. The modules have the ability to function as access point (AP) and/or station (STA) and can be interacted with via a UART AT command interface.

There seem to be two different configuration / OTA protocols both of which have an interface on port 48899/UDP:

* HF AT Interface (alternatively referred to as SmartConfigure Smart Link), used by the HF-A, HF-LPT, HF-LPB, etc. series modules
* IOTService / IOTManager, used by the Eport and Wport series modules

In addition, some modules (such as the HF-LPB125 and HF-LPT330) support WeChat AirKiss which is, as documented in the Espressif ESP-TOUCH guide, essentially a covert channel interface for configuring Wi-Fi settings on embedded devices which have no other interfaces. It does this by having the device start in packet capture mode and have the AirKiss app send a series of UDP packets to the Wi-Fi AP, encoding the SSID and password into the packet length fields. Because the length field is not encrypted, the listening node can parse out the SSID and password and join the Wi-Fi network after which configuration can begin. While typically AirKiss is only active during intial configuration, the approach is still fundamentally a security risk.

HF AT Interface / SmartConfigure Smart Link

Rather than merely being used for network discovery, this service actually exposes the module's AT interface over UDP and functions in the following fashion:

* Scan network with UDP broadcast string on port 48899
* Nodes will respond with IP,MAC,MID
* Answer with "+ok", node enters AT mode
* We can now execute arbitrary AT commands
* Keep a node in command mode by sending AT+W at 1 min intervals, the node wont accept other connection requests for 1 min
* Send AT+Q to quit AT command mode

The AT interface provides access to various kinds of sensitive functionality such as:

* Upgrading firmware
* Getting / Setting Wi-Fi configuration data (including keys)
* Getting / Setting service (eg. web server) credentials
* Getting / Setting 'user' configuration (device-specific info)
* Setting GPIO pin status
* Navigating to URLs in a proxy-like fashion

In addition, OEMs can add device-specific commands (such as upgrading firmware for modules or devices connected to the Wi-Fi module like the PV inverter in the case of the SolarMAN kit).

The only protection for this service is a 'password' which doubles as the network discovery string. The password is hardcoded in the firmware and cannot be changed and since it is part of the discovery protocol it cannot be considered confidential. By default this password is 'HF-A11ASSISTHREAD'.

In some cases (eg. the Omniksol WiFi kits & cards) OEMs have changed this password but that adds little additional security as all it takes is for an attacker is to obtain a device firmware image, mobile app or integration gateway and extract the password. OEMs can change the password in new firmware releases but this would break compatibility with certain apps and tools using the interface for device discovery and the attacker can simply repeat the trick.

Extracting the AT password for a HF-based device firmware image is simple and involves unpacking the firmware (extracting the LZMA compressed blob from the U-boot image) and fetching the hardcoded password from the decompressed image. This can be done in a 'clever' way (loading it in IDA, identifying the AT interface authentication function with a FLIRT signature and finding the reference to the password) or in an easy way which uses the fact that for the eCos based images the password is always sandwhiched between the same two strings:

As such the following oneliner is enough to get the AT password from a HF eCos based firmware image:

For images based on FreeRTOS, Contiki or mbed the following pattern seems to come after the password:

Which gives us the oneliner:

As it turns out, the issues with this interface have been independently discovered by multiple parties looking at different IoT products over the past few years such as CSL and Orvibo S20 smart sockets and Flux, LimitlessLED and Zengge Wi-Fi lightbulbs, with some prior work showing how to harvest SSIDs and keys from internet-facing Hi-Flying based devices and then geolocating the networks in question.

IOTService / IOTManager

The IOTService protocol is used by the IOTService tool and app to discover and configure Hi-Flying Eport and Wport based devices. It is very similar to the AT interface and essentially exposes the HF CLI, which can also be made available over telnet, wrapped in JSON on port 48899/UDP without authentication. Communication is initiated by a client who sends a broadcast discovery message after which nodes respond with their device information and the client can start sending JSON-wrapped CLI commands. The functionality exposed is similar to that of the AT interface and allows for dumping and changing Wi-Fi and service credentials, modifying firmware, etc.

In order to illustrate how this protocol can be abused i chose the Hi-Flying HF2211 serial converter / gateway as a target.

Serial converters / gateways are used in a variety of environments (such as industrial & building automation) as an interface between different serial (eg. fieldbusses over RS-232 or RS-485) and Ethernet protocols and are typically hooked up to automation equipment (like PLCs) or field devices. Such converters have historically been highly insecure and Hi-Flying's products are not an exception in this regard.

The HF2211 is based on the Wport module series and like other Hi-Flying based devices by default sets up an open AP named 'HF2211_XXXX' where XXXX are the last 2 bytes of the MAC. The HF2211 runs eCos on a Ralink RT5350f MIPS SoC and exposes the following services:

* 23/TCP, Telnet (Eport/Wport CLI)
* 53/UDP, DNS
* 80/TCP, HTTP
* 8899/TCP, Serial Converter Service
* 48899/UDP, IOTService

By using the IOTService management tool and intercepting the (unauthenticated, plaintext) traffic we can analyse the protocol and build our own scripts to dump WiFi credentials or modify converter firmware to carry out efficient MITM attacks on the processed serial traffic (and thus adversely affect the industrial processes being controlled by it).

About Those Inverters

Alright, back to those solar inverters which started this investigation. As we saw an attacker with access to one of the WiFi kit management interfaces can download a new firmware image to the connected inverters over the serial connection. And given that there's no firmware authentication mechanism on the inverter we can make arbitrary modifications, ranging from bricking them to more sophisticated actions.

Affecting Electrical Grid Stability

Solar inverters exist to convert the variable direct current (DC) output of photovoltaic systems into a utility frequency alternating current (AC) that can be fed into a commercial electrical grid or a local 'off-grid' electrical network. The power thus generated can be either fed directly into the utility grid, as would be the case with a commercial solar park, or can be used to supply household power consumption demands either directly with only excess power fed into the grid or indirectly by having net metering equipment help settle the bill.

A balance between power supply and demand is crucial to the proper functioning of the grid in order to prevent instability. Photovoltaic systems influence grid balance in the form of lessening demand and increasing supply, typically by directly powering local needs and feeding excess power to the grid. Solar inverters are central here for maintaining the balance between the DC generation of the solar array(s) and the AC load of the household and utility grid. Peaks or dips resulting from things like solar eclipses, cloud cover, temporary faults or peak-hour production (eg. at noon) are dealt with by a combination of utility companies planning ahead, providing buffers and intervention capabilities and inverters ensuring they accurately match the voltage, frequency and phase of the grid AC sine wave, smooth out fluctuations and provide anti-islanding capabilities by disconnecting from the grid if it goes down to ensure utility worker safety.

As pointed out in prior work on this subject (https://www.mdpi.com/1996-1073/12/4/725/pdf), (https://horusscenario.com/), an attacker with sufficient control over an inverter's functionality (such as by means of downloading modified firmware) could manipulate it to potentially affect grid stability. An attacker could seek to manipulate inverter power output (and potentially affect the battery management system) to cause a drop in overall supply as well as negatively affect the individual financial compensation(s) received for power supplied to the grid or seek to inject excessive power and/or manipulate the voltage in order to amplify undesirable grid states.

Of course the complexity of such attacks is not to be underestimated. In addition to the defensive presence of grid protection systems and operator intervention, such an attack would require scale and coordination. These factors in turn are influenced by the degree of solar penetration in the overal grid's power generation sources and attacker control over a significant share of PV inverters, the latter in turn being influenced by inverter solution heterogeneity and exposure.

Regardless, even in the absence of real-world demonstrations of feasibility on representative testbeds the hardening of connected systems underpinning various Distributed Energy Resources (DER) such as solar, wind, biomass and small hydro is of crucial importance to overal grid hardening efforts.

Affecting Inverter Safety

Another potentially malicious scenario would involve an attacker modifying the inverter firmware in order to adversely impact safety, such as by causing a fire. Whether it is possible or not to reliably cause an inverter fire purely through software means remains to be demonstrated and as such caution is required in judging the realism of this scenario, but given the typical degree of inverter firmware control over internal fans, voltage adjustment temperature monitors, etc. a scenario like that is not outside the realm of possibilities.

The most common causes of inverter fires are arcing and resistive heating (either as precursor to arcing or by itself which are typically the result of poor manufacturing and/or installation, ambient conditions and ageing components. Adversely affecting inverter safety would require an attacker to artificially induce a materials breakdown (eg. through resistive heating and high ambient temperatures) as a precursor to arcing or as a direct cause of ignition in addition to overcoming safety mechanisms like arc fault detectors and temperature monitoring.

Omnik Inverter Firmware Internals

There are several different lines of Omnik inverters but generally speaking they all have firmware for a master CPU module, a slave CPU module and a display module. In order to illustrate the firmware internals, let's take a look at the Omniksol-13k/17k/20k-TL triple-phase transformerless inverter.

The master and slave modules are based on the TMS320F2802x running the Micrium µC/OS-II RTOS, with strings in the firmware hinting at a toolchain relation to the TMS320C2000 single frequencypower line communication (SFPLC) modem. The display module is based on the AT91SAM9261 microcontroller running Windows CE.

As we can see from the above disassembly snippets, the firmware has granular control over a wide array of functionality from the fan, inverter output and various aspects of grid synchronization to Maximum power point tracking (MPPT) thus allowing an attacker capable of modifying the firmware to maliciously adjust inverter operations in a very precise manner. Whether this is truly sufficient for achieving the desired results would require further research.

Conclusion

While the potential impact of the discussed vulnerabilities on connected solar inverters and serial converters is worrying, they are not the real issue in this story. It is (or ought to be) common knowledge that most connected embedded systems (whether consumer or industrial IoT) are typically woefully insecure and over the past decade awareness of 'cyber-physical attacks', where security issues in the digital world directly affect the physical realm, have also increased.

In my opinion the most troubling aspect here is the illustration of the detrimental effects of supply chain opacity on vulnerability management. The complicated supply chains in connected embedded systems involve many players across many tiers supplying and integrating different hardware and software components all the way from SoCs and libraries to the same product ending up with multiple different OEMs who do minimal customization before passing it on to another layer of resellers who basically slap a new logo on it. A vulnerability in any component at any level can 'trickle down' an increasingly opaque supply chain to end up in ICS equipment, POS terminal and internet-connected doll alike and the further up the supply chain vulnerabilities originate, the wider the range of products they tend to end up and the harder it is to track and mitigate them without active collaboration of all parties involved. Case in point: if you think vulnerabilities like Broadpwn, Blueborne, Devil's Ivy, Shellshock or Heartbleed can't be found in the wild anymore because the parties directly responsible for the vulnerable code issued a patch you'd be in for a nasty surprise.

Since no single entity oversees the entire software development and maintenance lifecycles, there is often no single, coherent set of security requirements and the further up the chain you go (such as SoC or RTOS vendors), the broader the customer base that needs to be catered to and that typically means catering to the lowest common denominator in terms of capabilities, overhead and cost constraints. On top of that, there is no clear single entity who has the ability to directly and centrally issues patches for any given vulnerability in the end product. Instead, OEMs have to wait for their suppliers (or their suppliers' suppliers) to issue their own patches and integrate them into new firmware versions which they can then distribute to their customers who are often left having to manually apply them to all affected products (or trust some small IoT vendor with arbitrary firmware update capabilities on a bunch of devices in their networks, which might be worse.

In summary, what's needed is a more holistic approach to IoT security covering all relevant aspects of the product lifecycle and ensuring the products meet a common minimum baseline.

Vulnerability Disclosure Timeline:

* 6 October 2018 - First report to Hi-Flying
* 26 May 2019 - Reports to SolarMAN and Omnik as well as NCSC-NL
* 26 May to 22 July 2019 - NCSC-NL attempted to contact manufacturers but no response, escalation to CNCERT/CC
* 15 August 2019 - Still no response from any manufacturer, decision to publish

Thanks to Peter-Jan Pinxteren for spotting the original open APs and making the link with the Omnik connectivity kits.