Building a Solar Telemetry Pipeline: When Inverter Firmware Lies About Production

The Problem & Industry Shift

My dad called me in October with a spreadsheet. His utility bill had climbed 40% year-over-year despite the solar installer's app cheerfully reporting "1.2 MWh generated this month." Two numbers, same physical system, irreconcilable. That gap is the entire reason this project exists.

The residential solar industry has a measurement problem that is structural, not accidental. Most consumer inverters (SolarEdge, Enphase, Growatt, Deye, SolaX) expose production data through a vendor cloud API that is derived from DC-side current sensing at the MPPT input. The utility, meanwhile, bills you on an AC-side revenue-grade meter that is calibrated to ANSI C12.20 or IEC 62053-21 class accuracy. Between those two points sit inverter conversion losses, clipping, transformer losses, and — critically — firmware that applies smoothing filters and rounding before pushing data to the cloud.

The shift that makes this tractable today is that the hardware you already own speaks Modbus TCP on port 502. You do not need the vendor cloud. You need a $35 Raspberry Pi, a pymodbus client, and the willingness to read the register map that the installer never mentioned. This is the same pattern that has transformed industrial telemetry: move from opaque SaaS dashboards to local-first observability, where you own the sampling interval, the retention policy, and the truth.

Architecture & Core Mechanics

The pipeline has four stages. Each stage is independently restartable, which matters because the inverter's TCP stack will drop connections roughly every 6-18 hours depending on firmware revision.

┌──────────────┐   Modbus TCP    ┌──────────────┐   SQLite WAL   ┌──────────────┐
│  Inverter    │ ──────────────► │  Collector   │ ─────────────► │  Reconciler  │
│  (port 502)  │   fc=3, 10s     │  (Python)    │   batched      │  (Python)    │
└──────────────┘                 └──────────────┘                └──────┬───────┘
                                                                        │
       ┌────────────────────────────────────────────────────────────────┘
       ▼
┌──────────────┐   HTTP/JSON     ┌──────────────┐
│  Grafana     │ ◄────────────── │  FastAPI     │
│  (dashboards)│                 │  (query API) │
└──────────────┘                 └──────────────┘

Stage 1 — Collector. Polls holding registers via Modbus function code 3 at a fixed 10-second cadence. The register map for a Deye/Sunsynk hybrid inverter exposes cumulative energy in 0.1 kWh units across two 16-bit registers (high word + low word), which you must combine as a 32-bit unsigned integer. Getting the word order wrong silently produces values that look plausible for hours before wrapping.

Stage 2 — Storage. SQLite in WAL mode. A single table of (ts_utc, register, raw_value) rows. At 10s cadence across 12 registers, that is ~104k rows/day, ~38M rows/year — well within SQLite's comfort zone with a composite index on (register, ts_utc). No time-series database needed at this scale; the operational simplicity of a single file beats InfluxDB's write amplification.

Stage 3 — Reconciler. The interesting part. Once a month, you enter the utility meter's cumulative kWh reading (from the bill or a photo of the meter). The reconciler computes the delta between the inverter's cumulative counter and the meter's, then back-fills a per-day correction factor. This is not cheating — it is calibration against a revenue-grade reference, which is exactly what a commercial SCADA system does.

Stage 4 — Query API. FastAPI serving pre-aggregated hourly and daily rollups. Grafana points at it. The dashboard shows three lines: inverter-reported, meter-corrected, and the delta.

Production Code Example

The collector below is the piece that actually runs on the Pi. The non-obvious engineering decisions are commented inline.

# collector.py — Modbus TCP poller for Deye/Sunsynk hybrid inverters
import asyncio
import logging
import sqlite3
import time
from contextlib import closing
from datetime import datetime, timezone

from pymodbus.client import AsyncModbusTcpClient

INVERTER_HOST = "192.168.1.42"
INVERTER_PORT = 502
UNIT_ID = 1
POLL_INTERVAL_S = 10.0

# Register map (Deye hybrid, holding registers, fc=3).
# Each entry: (name, start_address, word_count, scale)
# Cumulative energy registers are 32-bit (2 words), scale 0.1 kWh.
REGISTERS = [
    ("pv_today_kwh",   0x003C, 2, 0.1),
    ("pv_total_kwh",   0x003E, 2, 0.1),
    ("load_today_kwh", 0x0044, 2, 0.1),
    ("grid_import_kwh",0x0048, 2, 0.1),
    ("grid_export_kwh",0x004A, 2, 0.1),
]

log = logging.getLogger("collector")


def init_db(path: str) -> None:
    with closing(sqlite3.connect(path)) as conn:
        # WAL lets the reconciler read while the collector writes.
        # synchronous=NORMAL is the right durability/latency tradeoff for
        # telemetry: we can tolerate losing the last few seconds on power loss.
        conn.execute("PRAGMA journal_mode=WAL")
        conn.execute("PRAGMA synchronous=NORMAL")
        conn.execute("""
            CREATE TABLE IF NOT EXISTS samples (
                ts_utc   INTEGER NOT NULL,
                register TEXT    NOT NULL,
                value    REAL    NOT NULL
            )
        """)
        conn.execute(
            "CREATE INDEX IF NOT EXISTS idx_reg_ts ON samples(register, ts_utc)"
        )
        conn.commit()


def decode_u32(regs: list[int]) -> int:
    # Modbus is big-endian on the wire, but many inverters store the
    # *low* word first in memory. Deye uses high-word-first. If your
    # cumulative counter jumps by ~6553.6 kWh, you have the order wrong.
    return (regs[0] << 16) | regs[1]


async def poll_once(client: AsyncModbusTcpClient, db_path: str) -> None:
    ts = int(time.time())
    rows: list[tuple[int, str, float]] = []

    for name, addr, count, scale in REGISTERS:
        # pymodbus 3.x renamed `slave` to `device_id`; pin your version.
        rr = await client.read_holding_registers(
            address=addr, count=count, device_id=UNIT_ID
        )
        if rr.isError():
            log.warning("read failed for %s: %s", name, rr)
            continue
        raw = decode_u32(rr.registers) if count == 2 else rr.registers[0]
        rows.append((ts, name, raw * scale))

    if not rows:
        return

    with closing(sqlite3.connect(db_path)) as conn:
        conn.executemany(
            "INSERT INTO samples(ts_utc, register, value) VALUES (?,?,?)", rows
        )
        conn.commit()


async def main() -> None:
    init_db("energy.db")
    client = AsyncModbusTcpClient(INVERTER_HOST, port=INVERTER_PORT, timeout=3)

    while True:
        try:
            if not client.connected:
                await client.connect()
            await poll_once(client, "energy.db")
        except Exception as exc:
            # Inverter firmware drops the socket regularly; reconnect and
            # keep going. Never let one bad poll kill the loop.
            log.error("poll cycle failed: %s", exc)
            client.close()
            await asyncio.sleep(5)
            continue
        await asyncio.sleep(POLL_INTERVAL_S)


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    asyncio.run(main())

The reconciler is a single SQL query plus a small Python wrapper. The key insight is that you do not correct the raw samples — you store a correction factor per billing period and apply it at query time. This preserves the audit trail, which matters when you eventually argue with the installer.

# reconcile.py — compute correction factor from a utility meter reading
def correction_factor(inverter_total_kwh: float, meter_total_kwh: float) -> float:
    """Returns the multiplier that maps inverter-reported to meter-truth.

    Typical residential values: 0.88 - 0.94. A factor below 0.85 usually
    means a CT clamp is mis-installed or a string is underperforming.
    """
    if inverter_total_kwh <= 0:
        raise ValueError("inverter counter must be positive")
    return meter_total_kwh / inverter_total_kwh

Performance, Cost & Trade-offs

Latency vs. accuracy. A 10-second poll interval gives you 8,640 samples/day per register. Dropping to 60s cuts storage 6x and loses nothing for billing purposes — daily energy is an integral, and the trapezoidal error from coarser sampling is under 0.3% for typical residential load curves. I kept 10s because it makes the dashboard feel live and the storage cost is trivial.

Storage. 38M rows/year at ~40 bytes/row is ~1.5 GB/year. SQLite handles this without complaint. The composite index roughly doubles it. If you scale to a commercial site with 50 inverters, move to TimescaleDB or ClickHouse — the write pattern (append-only, time-ordered) is exactly what columnar stores are built for.

The 8-12% gap. On my dad's system, the vendor app reported 1,204 kWh for October. The utility meter, corrected for the household's baseline consumption, implied 1,081 kWh of actual production. That is a 10.2% overstatement. Decomposed: ~3% inverter conversion loss (DC to AC), ~2% clipping during peak sun, ~4% firmware smoothing (the app appears to report a rolling maximum rather than a true integral), and ~1% CT clamp calibration drift. Only the first two are physics. The last two are the vendor lying to you.

Security. Modbus TCP has no authentication. Anyone on your LAN can write to holding registers and, on some inverters, change grid-tie parameters. Put the inverter on an isolated VLAN with no internet egress. The Pi should have a firewall rule allowing only outbound to your Grafana host and inbound from your admin subnet.

Cost. Raspberry Pi 4 (already owned) + SD card ($10) + zero cloud spend. The vendor's premium monitoring tier was $9.99/month. Payback period: five weeks.

Actionable Checklist / Summary

Confirm your inverter exposes Modbus TCP on port 502. Check the manual's "communication" section; if it only lists RS485, you need a $15 RS485-to-Ethernet gateway.
Find the register map. Search <inverter model> modbus register map pdf. Community forums (DIY Solar, Home Assistant) have reverse-engineered maps for most consumer brands.
Verify word order on cumulative counters before trusting any number. Compare against the vendor app for one day; if the ratio is wildly off, swap the words.
Isolate the inverter on a VLAN. Modbus has no auth.
Log raw samples, never corrected values. Apply correction factors at query time so you keep the audit trail.
Enter the utility meter reading monthly. This is the only ground truth you have.
Alert on the correction factor drifting outside 0.85-0.95. That range is your early-warning system for a failing string or a mis-clamped CT.
Back up energy.db nightly. It is a single file; rsync to a NAS is sufficient.

The broader lesson: any time a vendor gives you a number and a bill gives you a different number, the vendor's number is a marketing artifact. Own the measurement layer. It is cheaper, more accurate, and it is the only way to know whether the panels are actually doing what you paid for.

The Problem & Industry Shift

Architecture & Core Mechanics

Production Code Example

Performance, Cost & Trade-offs

Actionable Checklist / Summary

References