A containerized data scraper for APSolar ECU's written in node.js by Ryan Jacobs. It has the following features:
-
Prometheus metrics exporter
-
JSON data endpoint
-
MQTT data export
-
Data export to MQTT that is follows the topic format required for auto-discovery in Home Assistant.
Since most of us that have installed APSolar ECU's in the home don't have access to APSolar's monitoring site or API to get information and may be curious about exporting generation and all other data provided by the ECU into something a bit more query-able (at the very least), this data scraper scrapes the data from the ECU's local web page and provides a number of methods to ingest that data.
This project started life as an unofficial Prometheus exporter for the data provided by the APSolar ECU as Prometheus was the time-series database of choice for me at the time (and it's a little easier to get started with for something simple like this vs. InfluxDB). It became apparent quickly that having the scraper provide a number of different data formats for whatever someone might want to do with it was the way to go, and so it grew a JSON endpoint and not one, but two MQTT data formats.
At this time, providing support for InfluxDB is being considered for a future release.
NOTE: The APSolar ECU (at this time) only updates the statistics every 5 minutes. As a result, the scraper will only scrape every 5 minutes. Repeated hits to the scraper will produce a cached copy for those 5 minutes. Please set any downstream ingestion to respect this time limit.
You will need to define at least the environment variable ECUHOST in order to run the image and ensure that port 3000 is forwarded somehow. Further configuration information is available below.
docker run --env ECUHOST=<your-ecu-host-or-ip> -p 3000:3000 djryanj/solar-scraper:latest
Configuration is done via environment variables only.
The following environment variables directly affect the operation of the scraper. Only one is required to be set; the rest are optional depending on what features you want to do (and the configuration required for them). There are defaults for all.
| Variable | Required | Description | Default |
|---|---|---|---|
ECUHOST |
Yes | IP or hostname of the APSolar ECU to scrape. | 192.168.1.1 |
PORT |
No | The port that the container listens on (internally). Be sure to change the port Docker forwards if you change this. | 3000 |
USE_MQTT |
No | Set to true if you want to use generic MQTT. See the MQTT section for more information. Be sure to also set MQTT_HOST!! | False |
USE_HA_MQTT |
No | Set to true if you want to use Home Assistant to auto-discover solar information via MQTT. Note that USE_MQTT and USE_HA_MQTT are independent; you do not need to set USE_MQTT to true if USE_HA_MQTT is true. Be sure to also set MQTT_HOST!! | False |
MQTT_HOST |
No | Set to the IP address or DNS name of your MQTT server. Both USE_MQTT and USE_HA_MQTT use the same host. IMPORTANT! The default host could potentially expose information to a public server you don't want exposed. Be sure to set this if you set USE_MQTT or USE_HA_MQTT to true! NOTE: At this time, providing a self-signed certificate file for TLS communications is not supported. If your MQTT server uses PKI from a well-known certificate authority such as letsencrypt (be sure that the WHOLE chain is sent), it should work. At this time, self-signed cert behaviour is untested. | test.mosquitto.org |
MQTT_PORT |
No | The TCP port on which to connect to the MQTT server. | 1883 |
MQTT_USERNAME |
No | If you need to supply a username for your MQTT server, set it here. | null/unset (works for anonymous servers) |
MQTT_PASSWORD |
No | If you need to supply a password for your MQTT server, set it here. IMPORTANT! This will be visible in the logs and if you get container information from the Docker host. You could provide this via Docker secrets. | null/unset (works for anonymous servers) |
MQTT_TOPIC |
No | Base topic for standard MQTT data. See MQTT section for more information. | home/solar |
HA_MQTT_TOPIC |
No | Base topic for Home Assistant auto-discovery. This must match the base topic in your Home Assistant configuration.yaml. Note: this can only be a single label (e.g. homeassistant or hass) or auto-discovery will fail. |
homeassistant |
The following environment variables provide information into the scraper but do not affect operation beyond the cosmetic (they are exported with metrics and are visible on the browseable home page). None of these are required.
| Variable | Description | Default |
|---|---|---|
SITENAME |
A friendly site name. Mostly visible on the home page of the scraper but also exported with metrics. | Your House! |
BUILDID |
The BUILDID is intended to be passed in from an automation pipeline such as Azure DevOps and should be equivalent to the git hash (this was mostly used during development; a future release may hide this for production builds). The BUILDID can be read from the local .git directory, but that is not present in the Docker container. | missingBuildId (when nothing is passed to Docker or the .git directory is not present locally); otherwise the 7-character git hash of the |
BUILDNUMBER |
The BUILDNUMBER is intended to be passed in from an automation pipeline such as Azure DevOps and should be equivalent to the automated build number, e.g. 20190910.1. | local |
SOURCEBRANCHNAME |
The SOURCEBRANCHNAME is intended to be passed in from an automation pipeline such as Azure DevOps and should be equivalent to the git branch that the container was built from, e.g. master. | local |
Exposes port tcp/3000 by default. Change the PORT environment variable if you want to change it. Leverage something like jwilder/nginx-proxy to automatically forward a hostname from the Docker host's port 80 or 443 to this container.
There is a default cosmetic endpoint at / (e.g. http://docker-host:3000/) that shows all data in a human-readable form.
Prometheus expects an endpoint at /metrics, which is where it is. In addition to the custom metrics provided by this scraper, there are node.js-specific metrics exported as well that are not presented here. You can browse the endpoint if you wish.
All metrics are prefixed by solar_. The list of metrics is below (including the prefix for clarity).
| Metric | Description |
|---|---|
solar_total_generated |
Total solar power generated over the lifetime of the array in kilowatt hours (kWh). |
solar_daily_generated |
Total solar power generated today in kilowatt hours (kWh). |
solar_current_total_power |
Current solar power output in watts (W). |
solar_carbon_offset |
Estimated carbon offset in kg of CO2 saved over the lifetime of the solar array. |
solar_trees_planted |
Estimated carbon offset by the equivalent number of trees that have been planted over the lifetime of the solar array. |
solar_gallons_offset |
Estimated carbon offset in gallons of gasoline saved over the lifetime of the solar array. |
solar_scraper_version_info{version="<version>",hostname="<hostname>",buildId="<buildId>",ecuHost="<ecu-host>"} |
Version and other information. |
The following metrics will be one per panel in your array, labelled by inverter ID (see the Administration->ID Management or the Real Time Data links on the ECU to see what inverters are configured). Inverters that have multiple panels connected (e.g. the YC500/600 or YC1000) will have an -A, -B etc. appended to the end.
| Metric | Description |
|---|---|
solar_panel_power{inverterId=<inverterId>} |
Current power output of the panel in watts (W). |
solar_panel_grid_frequency{inverterId=<inverterId>} |
Current grid frequency measured by the panel in hertz (Hz). |
solar_inverter_temperature{inverterId=<inverterId>} |
Current inverter temperature in °C. |
solar_inverter_grid_voltage{inverterId=<inverterId>} |
Current grid voltage output by the inverter in volts A/C (V). |
Although only one MQTT host is allowed, you can output 2 kinds of MQTT data. One is more generically formatted and published when USE_MQTT is true; the other is formatted for use by the MQTT auto-discovery function in Home Assistant, enabled by setting the USE_HA_MQTT environment variable to true.
The generically-formatted data could be suitable for consumption into Prometheus through something like MQTTGateway for Prometheus. This would achieve a decoupling of the scraper and Prometheus, which could be useful in large systems with many ECU's (though I doubt such a thing would need this software ;) ).
The MQTT_TOPIC and HA_MQTT_TOPIC environment variables, set to home/solar and homeassistant respectively by default, are the base topics upon which the rest of the topic labels are built.
IMPORTANT: Do not use more than a single label (e.g. homeassistant or hass) for the Home Assistant auto-discovery base topic or auto-discovery will fail.
The following metrics are exported to the MQTT_TOPIC (e.g. home/solar):
| Metric | Description |
|---|---|
/dailyGen |
Total solar power generated today in kilowatt hours (kWh). |
/totalGen |
Total solar power generated over the lifetime of the array in kilowatt hours (kWh). |
/currentSystemPower |
Current solar power output in watts (W). |
/treesPlanted |
Estimated carbon offset by the equivalent number of trees that have been planted over the lifetime of the solar array. |
/gallonsSaved |
Estimated carbon offset in gallons of gasoline saved over the lifetime of the solar array. |
/carbonOffset |
Estimated carbon offset in kg of CO2 saved over the lifetime of the solar array. |
/scraper/version |
Running version number of the scraper software. |
/scraper/hostname |
Hostname on which the software is running. |
/scraper/azureBuildNumber |
Pipeline build number. |
/scraper/ecuHost |
Configured ECU IP address or hostname. |
The following topics will be one per panel in your array, labelled by inverter ID (see the Administration->ID Management or the Real Time Data links on the ECU to see what inverters are configured). Inverters that have multiple panels connected (e.g. the YC500/600 or YC1000) will have an -A, -B etc. appended to the end.
| Metric | Description |
|---|---|
/panels/<inverterId>/currentPower |
Current power output of the panel in watts (W). |
/panels/<inverterId>/gridVoltage |
Current grid voltage output by the inverter in volts A/C (V). |
/panels/<inverterId>/temperature |
Current inverter temperature in °C. |
/panels/<inverterId>/gridFrequency |
Current grid frequency measured by the panel in hertz (Hz). |
/ping |
Unix-timestamp for the last published MQTT data. |
The following metrics are exported to the HA_MQTT_TOPIC (e.g. homeassistant).
All of these will be discovered as sensor within Home Assistant, due to the topics' full format e.g. homeassistant/sensor/totalGen/state. For more information, see MQTT auto-discovery function in Home Assistant. Therefore, all of the below are relative to HA_MQTT_TOPIC/sensor.
Note: For brevity, the /state suffix for each topic is not added on to the topic name (this is usually not seen in Home Assistant anyway). /config topics are not presented.
| Topic | Data | Home Assistant Sensor Name |
|---|---|---|
/carbonOffset/carbon |
Estimated carbon offset in kg of CO2 saved over the lifetime of the solar array. | sensor.carbon_offset_kg_of_co2 |
/carbonOffset/trees |
Estimated carbon offset by the equivalent number of trees that have been planted over the lifetime of the solar array. | sensor.carbon_offset_trees_planted |
/carbonOffset/gallons |
Estimated carbon offset in gallons of gasoline saved over the lifetime of the solar array. | sensor.carbon_offset_gallons_of_gasoline_saved |
/dailyGen |
Total solar power generated today in kilowatt hours (kWh). | sensor.power_generated_today |
/currGen |
Current solar power output in watts (W). | sensor.current_solar_power_output |
/totalGen |
Total solar power generated over the lifetime of the array in kilowatt hours (kWh). | sensor.lifetime_solar_power_generated |
Each of the below metrics will exist one per panel/inverter ID. The panel # for sensors should be the same every run, unless additional inverters are added.
| Topic | Data | Home Assistant Sensor Name |
|---|---|---|
/solar_panel_<inverterID>/power |
Current power output of the panel in watts (V). | sensor.solar_panel_<#>_power |
/solar_panel_<inverterID>/temperature |
Current inverter temperature in °C. | sensor.solar_panel_<#>_temperature |
/solar_panel_<inverterID>/frequency |
Current grid frequency measured by the panel in hertz (Hz). | sensor.solar_panel_<#>_frequency |
/solar_panel_<inverterID>/voltage |
Current grid voltage output by the inverter in volts A/C (V). | sensor.solar_panel_<#>_voltage |
The following additional topics are published:
- A JSON-formatted string of all data is published to
HA_MQTT_TOPIC/json(e.g not atHA_MQTT_TOPIC/sensor/json. Unsure at this time how useful that is, but Home Assistant can read JSON-formatted MQTT topics. MQTT_TOPIC/pingis always published if eitherUSE_MQTTorUSE_HA_MQTTistrue
NOTE: Scraper versioning info is not currently published to a Home Assistant topic. This is a decision I made because at this time, I can't see a reason to include it. It's mostly useless information for most users, more so for Home Assistant users because of the way the data is surfaced and likely to be used in Home Assistant, and if it's desired it's available on the other topic for manual addition anyway.
The scraper will return a JSON-formatted string by browsing /json, e.g. http://docker-host:3000/json. This is useful for consumption into a wide variety of downstream things.
There is no other API available.
Here is a sample config block for Prometheus:
- job_name: solar_ecu
scrape_interval: 5m
static_configs:
- targets:
- <hostname>