Skip to main content

When the NSPanel won't boot after an update: anatomy of one outage

This article is different from the rest of this site. It isn't a "do this and it'll work" guide but a breakdown of one specific failure — from a black display to a working panel, including every dead end I wandered into along the way.

I'm writing it because the fix itself turns out to be trivial. What's valuable is how you get there — and above all, how you recognise that the hypothesis you're banking on is wrong. The techniques used here apply to any ESP32 device running ESPHome, not just the NSPanel.

You're in the middle of this and just want the fix

Panel black, not on Wi-Fi, and reflashing didn't help? It's most likely latched in safe mode, and factory.bin won't fix that - the failed-boot counter lives in the NVS partition, which a factory image never touches.

esptool --port /dev/cu.usbserial-0001 erase-flash
esptool --port /dev/cu.usbserial-0001 --baud 460800 write-flash 0x0 your-firmware.factory.bin

The full command with an explanation is in The fix, prevention in Prevention. The rest of the article is about how you arrive at that diagnosis.

Who this is for

I assume you're comfortable with a serial line, esptool and reading logs. If you just want to get an NSPanel running, start with Sonoff NSPanel - eWeLink or ESPHome. The practical recipes without the narrative live in its Troubleshooting section.

Environment​

  • Sonoff NSPanel EU, ESP32-D0WD-V3 rev 3.1, 4 MB flash
  • NSPanel_HA_Blueprint version 2026041, pulled from the main branch with refresh: 300s
  • ESPHome 2026.7.2, ESP-IDF v5.5.5
  • bluetooth_proxy: enabled in the configuration — six months without trouble
  • A hidden SSID on a separate IoT network

NSPanel board with the serial header

Symptoms​

The panel first hung after an update on a blue screen reading Initializing with all fields empty. Rebooting just repeated it. After disconnecting from mains, nothing came up at all — black display, no response, and the panel never appeared on Wi-Fi.

Reflashing the firmware over serial did not help.

Timeline​

I reconstructed this only at the end, from Home Assistant's history. I'm putting it up front, but during the diagnosis I didn't have it:

TimeWhat happened
Jul 21, 18:54Last successful boot, panel running
Jul 23, 19:06Panel went quiet and never came back
Jul 25, 10:45Diagnosis begins, first serial log
Jul 25, 13:01First usable log - from a proper supply
Jul 25, 13:20Flash erased and firmware rewritten
Jul 25, 13:31Panel fully working again

The outage lasted 42 hours; the diagnosis itself just under three.

Dead end 1: weak power from the 3V3 pin​

The first serial log looked like this (abridged):

[C][safe_mode:189]: Unsuccessful boot attempts: 2
[I][app:060]: Running through setup()
[C][wifi:649]: Starting
ets Jul 29 2019 12:21:46
rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)

The panel restarted every second and always right after wifi: Starting.

What rst:0x1 (POWERON_RESET) and rst:0xc (SW_CPU_RESET) mean​

Here's the first transferable skill. That rst: line is the fastest way to cut the problem in half:

CodeMeaningWhere to look
rst:0x1 (POWERON_RESET)Chip lost power, or was reset via the EN pinPower, hardware
rst:0xc (SW_CPU_RESET)Software crashed and restartedFirmware, configuration
rst:0x8 (TG1WDT_SYS_RESET)Watchdog fired — something got stuckBlocking code
rst:0xf (RTCWDT_BROWNOUT_RESET)Brownout detectorPower

POWERON_RESET in a loop, always at the moment Wi-Fi starts — the point of highest current draw — is a textbook symptom of a weak supply. I closed it as a power problem.

And it was a dead end. Not because I misread the log, but because the measurement setup was faulty.

Why that log was worthless​

At that point the panel was powered from the 3V3 pin on the USB-TTL adapter. That pin comes from a tiny regulator on the adapter board good for 50–100 mA. That's enough for the flash itself, which is exactly what makes it treacherous - the firmware writes without a hitch. But at boot, Wi-Fi asks for multiples of that and the supply collapses. So the log faithfully documented the limits of my adapter, not a fault in the panel.

Lesson

Before you trust data, validate the rig you measured it with. Moving the power lead from 3V3 to the +5V pin of the same adapter is enough - it comes essentially straight from USB VBUS and the panel converts it with its own regulator. No external supply required.

One interesting detail: the crash also happened in safe mode, where nothing from the Blueprint runs. At the time I took that as proof of a hardware cause — and with that faulty rig, it genuinely was one.

Dead end 2: exhausted memory​

The second hypothesis rested on the Blueprint's own documentation, which describes the symptom verbatim:

During Startup: If your device runs out of memory at startup, it may not load the firmware, resulting in a black screen and an unresponsive device.

  • Be cautious when adding memory-intensive components like bluetooth_proxy.

Black display, unresponsive device. An exact match. And I did have bluetooth_proxy in my configuration.

Measuring instead of guessing: what bluetooth_proxy really costs​

Rather than speculate, I compiled the same configuration twice with bluetooth_proxy: as the only difference. ESPHome prints actual usage at the end of a compile, and Download firmware binary is enough to get it — nothing is uploaded to the panel.

with bluetooth_proxywithoutdifference
Flash1,529,391 B (83.3%)1,110,539 B (60.5%)−409 kB
DRAM used69,352 B57,384 B−12 kB
Total DRAM pool124,580 B180,736 B−56 kB
Free DRAM55,228 B123,352 B+66 kB

Here's something I didn't expect. The total DRAM pool isn't the same in both cases. The BLE controller carves its memory out of the pool at link time — so it isn't merely a matter of consumption. Free internal DRAM drops from 123 kB to 55 kB, less than half.

And PSRAM won't save you, even though the panel has 2 MB of it and uses a few percent. Wi-Fi buffers must be DMA-capable and on the ESP32 they cannot live in PSRAM. Wi-Fi and Bluetooth coexisting is therefore the most internal-DRAM-sensitive spot in the whole system.

Addendum: an OTA rollback confirmed it​

After the fix I tried putting bluetooth_proxy back and installing it over OTA. The log then said:

[W][safe_mode:094]: OTA rollback detected! Rolled back from partition 'app1'
[W][safe_mode:094]:  The device reset before the boot was marked successful

The Bluetooth build uploaded, failed to boot, and the bootloader reverted to the previous firmware by itself. I then found the panel working fine — but running the old build without the proxy, not the new one.

In hindsight this is the strongest evidence for the memory hypothesis I have. It's also a fourth safety net I didn't know about during the diagnosis: ESP-IDF watches a freshly uploaded firmware, and if it resets before ESPHome marks it good, it restores the previous version. That's why an OTA update can't kill the panel as easily as a serial flash can.

How to spot it

If you see OTA rollback detected after an OTA, your new version is not running. Check compiled on in the log - if it's older than your last compile, you're looking at the old firmware and the new one never took hold. A second test: dump_config must contain [C][esp32_ble_tracker:...] and [C][bluetooth_proxy:...]. If they're absent, the proxy isn't running.

On the second attempt, this time with safe_mode: storage: rtc in the configuration, the build held on long enough to check in - and ESPHome captured the crashes:

*** CRASH DETECTED ON PREVIOUS BOOT ***
  Reason: Fault - Unknown
  Crashed core: 1
  PC:  0x4011A490     ← first crash
  PC:  0x4000BFF0     ← second crash, this address lies in ROM

Two consecutive boots crashed at different addresses, the second one inside the chip's ROM. Both crashes also pass through the same function - 0x4011A4AB in the second backtrace sits 27 bytes past the first crash site.

That is diagnostically valuable in itself: a deterministic code bug crashes at the same address every time. Varying crash sites, one of them in ROM, point instead to memory corruption or failing allocations claiming whichever victim happens to be next. It also fits the first crash, where LBEG/LEND pointed into the ROM memory-routine area.

Why there's no decoded backtrace here

I wanted to translate those addresses into function names, and couldn't - and the reason is instructive. I was compiling via a remote build, so there was no local ESP-IDF toolchain (CMAKE_ADDR2LINE not found) and firmware.elf was overwritten by the very next build. When I tried resolving the addresses against the ELF left on disk, I got an unrelated mix of functions from NVS, lwip and the API - because it was the ELF of a different build. I spotted it because its symbol table contained not a single occurrence of bluetooth_proxy.

The lesson: a backtrace is only worth anything with the ELF of exactly the build that crashed. If you want to decode one, compile locally and set that ELF aside before you reproduce the crash.

So, what is and isn't proven: a build with bluetooth_proxy repeatedly fails to boot on this panel - evidenced by the rollback and by two captured crashes. That this is what took the panel down on 23 July is not proven and remains a hypothesis.

Why this is only half an answer​

I removed bluetooth_proxy, flashed, powered up — and the panel still didn't boot. That memory pressure was real and measured, but it wasn't the thing keeping the panel dead.

Lesson

A measured factor and a proven cause are two different things. Finding something objectively marginal did not mean I'd found the culprit.

Finally, a usable log​

Only with power from the +5V pin and the serial adapter wired to just TX, RX and GND did I get a log worth having:

[C][safe_mode:189]: Unsuccessful boot attempts: 10
[E][safe_mode:201]: Boot loop detected
[I][app:060]: Running through setup()
[C][wifi:649]: Starting
[D][wifi:1325]: Starting scan
[C][component:209]: Setup wifi took 98ms
[I][app:117]: setup() finished successfully!
Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.

Core 1 register dump:
PC      : 0x400e111b  PS      : 0x00060530  A0      : 0x8018ac74  A1  : 0x3ffb8520
...
EXCVADDR: 0x00000000  LBEG    : 0x4000c349  LEND    : 0x4000c36b
...
rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)

Everything changed. rst:0xc (SW_CPU_RESET) instead of POWERON_RESET — this is a software crash, not a power problem.

What Guru Meditation LoadProhibited and EXCVADDR: 0x00000000 mean​

The second transferable skill. Two lines are enough:

  • LoadProhibited / EXCCAUSE: 0x1c — the program tried to read from an address it isn't allowed to read
  • EXCVADDR: 0x00000000 — and that address was zero

Together that means a null pointer dereference. Somewhere there's a pointer assumed to be valid that isn't.

The other registers help too: A8: 0x00000000 shows which register carried the null, and PC: 0x400e111b is the address of the instruction that crashed.

What this narrowed down​

The crash happened in safe mode, where ESPHome starts only logger, wifi, ota and safe_mode. Nothing from the Blueprint, no display, no scripts. So:

  • it isn't a panel configuration error
  • it isn't a missing TFT firmware
  • it's a crash inside ESPHome's own code

Dead end 3: the ESPHome version​

A crash after Starting scan plus a fresh ESPHome update made a regression the obvious suspect. I decided to check this one in the source, not by guessing.

Mapping logs to source code​

The third transferable skill, and possibly the most useful. Notice the numbers in the brackets:

[C][wifi:649]: Starting
[D][wifi:1325]: Starting scan

Those aren't arbitrary — they're the line in the source file that emitted the log. Fetch the source for your exact version and look:

gh api "repos/esphome/esphome/contents/esphome/components/wifi/wifi_component.cpp?ref=2026.7.2" \
  --jq '.content' | base64 -d > wifi_component.cpp
sed -n '649p;1325p' wifi_component.cpp

Which gives:

649:  ESP_LOGCONFIG(TAG, "Starting");
1325:  ESP_LOGD(TAG, "Starting scan");

It matches. That confirms I'm looking at both the right version and the right file — not a given, because the wifi tag is also used by wifi_component_esp_idf.cpp. That one has only 1288 lines, so line 1325 can't be in it.

Timing killed the hypothesis​

I suspected the scan-result sorting that starts just below line 1325 in that file. But look at the timestamps:

[13:01:51][D][wifi:1325]: Starting scan
[13:01:51][C][component:209]: Setup wifi took 98ms
[13:01:51][I][app:117]: setup() finished successfully!
[13:01:51]Guru Meditation Error

All within the same second. And the log is missing both Found networks: and No networks found. A scan normally takes two seconds — so it never finished and the code handling its results was never reached.

Lesson

Log timestamps are evidence. When a hypothesis claims something happened inside a function that hasn't run yet, the hypothesis is dead — however good its explanation sounded.

Comparing versions​

That left checking whether the code in question had changed at all:

diff -u 2026.6.5_wifi_component.cpp 2026.7.0_wifi_component.cpp | grep -c "^[+-]"
# 34
diff -u 2026.7.0_wifi_component.cpp 2026.7.2_wifi_component.cpp | grep -c "^[+-]"
# 0

The Wi-Fi component is byte-identical across 2026.7.0, 7.1 and 7.2. Within the 7.x series it cannot be the difference. And the whole 6.5 → 7.0 transition is 34 lines containing no unguarded dereference — a new roaming condition, renaming USE_RP2040 to USE_RP2, suppressing a reboot during provisioning (which is != nullptr guarded), and a change in where the fast_connect preference is stored.

The source did not confirm the hypothesis. That's a legitimate result even if an unsatisfying one — I eliminated a suspect rather than finding a culprit.

The trap with no way out: three locks of safe mode​

Here's the heart of it. In hindsight the important question wasn't "what knocked the panel over" but "why couldn't it recover". A one-off crash turned into a 42-hour outage because three locks lined up.

Lock 1: the counter hit ten in fifteen seconds​

ESPHome has boot loop protection: after ten failed boots it switches to safe mode. But the restarts came 1.5 seconds apart, so ten attempts took no time at all. Before I even noticed something was wrong, the panel was locked.

Lock 2: the counter lives in flash and survives a power cycle​

The safe_mode default on ESP32 is storage: flash. The documentation says it plainly:

Flash storage: persists through power loss

So every power cycle I tried in order to revive the panel was pointless. The counter stayed at ten and the panel went straight into safe mode — where the display isn't started, hence the black screen.

Lock 3: factory.bin doesn't overwrite NVS, only erase-flash does​

And this is the most insidious one. Look at the partition table from the boot log:

0 otadata          OTA data         01 00 00009000 00002000
1 phy_init         RF data          01 01 0000b000 00001000
2 app0             OTA app          00 10 00010000 001c0000
3 app1             OTA app          00 11 001d0000 001c0000
4 nvs              WiFi data        01 02 00390000 00070000

The counter lives in the nvs partition at 0x390000. The factory image is written from 0x0 and covers the bootloader, the partition table and app0. It never touches NVS.

That's why I could reflash the panel as often as I liked. The counter sat there untouched, the panel kept entering safe mode — and safe mode kept crashing on that null pointer.

Any one of those three locks could have been worked around. All three together meant nothing I tried could possibly have worked.

The fix​

Once the diagnosis is done, the fix is three commands:

# optional: read the previous firmware's version from the second OTA slot
esptool --port /dev/cu.usbserial-0001 read-flash 0x1d0000 0x200 app1.bin

# erase the ENTIRE flash including NVS - this is the step that matters
esptool --port /dev/cu.usbserial-0001 erase-flash

esptool --port /dev/cu.usbserial-0001 --baud 460800 \
  write-flash 0x0 nspanel01-firmware.factory.bin

Wi-Fi credentials are compiled into the firmware, so erasing NVS costs you nothing.

And the first boot afterwards:

[C][safe_mode:189]: Unsuccessful boot attempts: 0
[I][app:060]: Running through setup()
...
[I][nspanel.core:189]: Blueprint progress: 100.0%
[I][nspanel.boot:165]: Progress: Completed
[C][wifi:1259]:   IP Address: 192.168.1.50

Counter at zero, a full setup instead of safe mode, Blueprint at one hundred percent.

Prevention​

Three configuration changes, each addressing one part of this story:

# 1) Counter into RTC memory - a power cycle wipes it
safe_mode:
  storage: rtc

# 2) Hidden SSID: connect directly, skip scanning
wifi:
  fast_connect: true

# 3) Pinned Blueprint version instead of 'main' + refresh
packages:
  remote_package:
    url: https://github.com/Blackymas/NSPanel_HA_Blueprint
    ref: 05ecefde2bac5db8bc96c02148058518d78a06fb
    refresh: never

safe_mode: storage: rtc is the most valuable of the three. RTC memory survives a software reset but a power cycle wipes it. The difference is fundamental:

storage: flash (default)storage: rtc
Transient crashcounter climbs to 10 and latches permanentlyevery power cycle gives the firmware a clean attempt
Persistent crashdead, serial onlystill crashes, but never latches

Safe mode remains available as a safety net, but the panel can never get stuck in it so badly that only a serial cable saves you.

You'll find the whole configuration in context in Step 2 of the main guide.

A pinned ref means the Blueprint version won't shift under you on every recompile. You stop receiving fixes — that's the point. Updates become deliberate: you check the memory figures first and only then upload.

What doesn't work as a rescue​

I considered whether a rescue action could be mapped to a physical button. It can't — and it's worth explaining why.

A button stands no chance. In a boot loop the firmware never reaches button handling, and in safe mode the Blueprint's binary sensors don't run at all. Any in-firmware rescue requires the firmware to run; and the firmware is precisely what's crashing.

The factory_reset component with resets_required looks like the exact answer — power cycle five times and the device wipes its preferences. But look at the implementation:

static bool was_power_cycled() {
  return esp_reset_reason() == ESP_RST_POWERON;
}
...
if (was_power_cycled()) { count++; ... }
else { this->save_(0); }   // ANY other reset zeroes the counter

It counts only genuine power-ons. A crash reboot is SW_CPU_RESET, which falls into the else and zeroes the counter. So in a boot loop it goes: power on → 1 → crash → 0 → power on → 1 → crash → 0. You never reach five.

That component is useful for a different situation: the device runs fine but is misconfigured and you can't reach it over the network.

Home Assistant as a forensic tool​

I reconstructed the timeline at the top of this article from HA via its REST API. It's an underrated tool — and above all, it lets you disprove hypotheses.

When exactly the device went away comes from the history of any of its entities. Note that without end_time the endpoint returns only one day:

curl -H "Authorization: Bearer $TOKEN" \
  "$HA/api/history/period/$START?end_time=$END&filter_entity_id=sensor.nspanel01_temperature&minimal_response"

That gave me the exact second the panel went silent.

Disproving the power-outage hypothesis was more elegant. If mains had failed, other devices would have restarted too. All it took was looking at the uptime of another ESP32 on the same network:

sensor.esp32_bluetooth_proxy_uptime = 407702 s ≈ 4.7 days

That value spans the entire incident, so mains never failed. One number, hypothesis settled.

A guess about the host running HA rebooting fell the same way — sensor.node_pve_last_boot pointed at a date four months old.

And the logbook showed what actually happened at the moment of the outage: Home Assistant restarted, and everything came back from it except the panel.

What I never found out​

Honestly: the original trigger could not be determined, and now it never will be.

The panel ran normally on that firmware for two days, so it can boot. Something knocked it over on 23 July at 19:06, but what, I'll never know — all the evidence has been overwritten:

  • app0 was reflashed
  • app1, the second OTA slot, was blank (0xffffffff)
  • NVS was erased, which was necessary for the fix
  • the ELF file needed to decode the backtrace was overwritten by new compiles
  • sw_version in HA's device registry updated the moment the panel reconnected

So a hypothesis remains: the memory pressure from bluetooth_proxy, with 55 kB of free DRAM, was exactly the kind of headroom a single update swallows. It isn't proven, though, and I won't dress it up as more than it is.

The ESPHome crash in safe mode on a null pointer is a thing in its own right. There's no report of it in esphome/issues, so either it's a rare combination or few people walk that path. Without the ELF from the original build, though, it can't be reported usefully.

Takeaways​

Twelve things that transfer elsewhere:

  1. Read the rst: code first. It splits power from software faster than anything else.
  2. EXCVADDR: 0x00000000 in a Guru Meditation means a null pointer dereference.
  3. Validate your measurement rig before trusting data. Powering from the adapter's 3V3 pin manufactured a fake fault for me.
  4. The numbers in log tags are source lines. [D][wifi:1325] can be looked up in your ESPHome version, and the guessing stops.
  5. Timestamps disprove hypotheses. A missing scan log proved the crash was elsewhere.
  6. Measure, don't guess. Two compiles produced a 66 kB DRAM difference. An estimate would have produced nothing.
  7. Distinguish pool from consumption. BLE takes 56 kB out of the total pool at link time, not just out of usage.
  8. Know which partition your flashing tool overwrites. A factory image leaves NVS alone, and NVS held the lock.
  9. HA is a forensic tool. Another device's uptime disproves a power-outage theory with a single number.
  10. Varying crash sites mean memory, not a code bug. A deterministic bug crashes at the same address every time. When the fault PC moves between reboots, look for exhausted or corrupted memory.
  11. A backtrace is worthless without the ELF of exactly the build that crashed. Decoding against a different build produces plausible-looking nonsense. You spot it because the call chain makes no sense - and you confirm it by searching the ELF's symbols for a component that build was supposed to contain.
  12. OTA is safer than a serial flash. ESP-IDF watches the new version and reverts on failure. The price is that you're then looking at the old firmware and have to notice OTA rollback detected.

And finally, the least technical one: leave yourself headroom. The panel ran on 55 kB of free memory for six months and worked. The mistake wasn't that the configuration was wrong — it was that it had no margin, and one routine update swallowed it.

Sources​

Do you like this article?

Your support helps create the best smart home guides!

Send a tip

Comments​