Spark 3 hangs every few hours

Bob_Steers · November 29, 2021, 6:54pm

Does this still happen with today’s firnware release?

Clrwl04 · December 2, 2021, 6:54am

Updated last night, no problems so far, looking good.

Clrwl04 · December 2, 2021, 6:59am

Forced wifi to change channel and the Spark hung.

Bob_Steers · December 2, 2021, 12:19pm

If you switched to the earlier experimental release, did you switch back to edge (or ${BREWBLOX_RELEASE})?

Clrwl04 · December 3, 2021, 2:47pm

Am using the following.

Bob_Steers · December 3, 2021, 2:58pm

Then we won’t feel silly later at least.

For now I’d recommend keeping a fixed channel, and we’ll continue digging through the Wifi code.
Alternatively, if you have a spare Pi of any kind, I can write a quick guide for setting it up as Wifi to USB repeater. This lets you bypass the Wifi stack on the Spark itself.

Clrwl04 · December 3, 2021, 4:39pm

The Wifi AP was and still is set to fixed channel.

Elco · December 5, 2021, 2:06pm

I have found something. It seems that Particle made a change to the hard fault handler, without ever testing that it still worked. Simply unacceptable.

So this is caused by 2 Particle bugs:

An old bug in the wifi stack causes a hard fault (unrecoverable error) when the WiFi is lost. The code that contains the bug is closed-source, so I cannot help them fix it.
A recently introduced bug in the hard fault handler causes the device to completely lock up instead of rebooting.

I have created a PR with a fix at Particle:

github.com/particle-iot/device-os

Fix hardfault handler to SOS again

particle-iot:develop ← BrewBlox:fix/hardfault-handler

opened 07:08PM - 04 Dec 21 UTC

elcojacobs

+6 -3

This fixes the broken hardfault handler as reported in #2380 I added the `.a…lign 4` line, which I think might be the actual fix for the compiler warning that the commit that introduced the bug intended to address. It is very easy to test the hard fault handler, so it seems that the compiler warning was 'fixed' in the commit below without any testing! :fearful: https://github.com/particle-iot/device-os/commit/7b6e2086af907668d395c63404ecc754ee36fe41 ## Test app: ```cpp void setup() { int (*bad_instruction)(void) = nullptr; bad_instruction(); } void loop() { HAL_Delay_Milliseconds(1); } ```

And reported the WiFi bug again:

github.com/particle-iot/device-os

WiFi dropout causes hardfault, hardfault handler is broken and P1 doesn't SOS and reset

opened 05:00PM - 04 Dec 21 UTC

elcojacobs

## Bug Report ### Expected Behavior Hardfault handler should blink RED sta…tus LED and reboot ### Observed Behavior Hardfault results in complete system hangup and system doesn't reset. I built my app with a non-modular build with SWD enabled in an attempt to find the cause of complete system hangups when the wifi becomes unavailable. The cause of the hangup is inaccessible to me because it is in the closed source WICED files. However, I noticed that I couldn't break the debugger on hardfaults anymore in 'prvGetRegistersFromStack'. I did looked at the history of the hardfault handler and reverted a change. This made SOS and hardfault work again. ### Steps to Reproduce - Clone our repo - In the <repo root>/build directory, run `make program-dfu MODULAR=n PLATFORM=p1 PARTICLE_DEVELOP=y USE_SWD=y APP=brewblox` - Turn WiFi on and off on router until turning it off causes a hardfault ### Test App https://github.com/BrewBlox/brewblox-firmware/tree/develop/app/brewblox-particle ### Analysis Since recently, I had issues with complete hangups of our temperature controllers. Even the watchdog didn't reset the system anymore. While trying to step debug using SWD, I noticed the system hung in the `HardFault_Handler` function, without jumping into `prvGetRegistersFromStack. In this commit: https://github.com/particle-iot/device-os/commit/7b6e2086af907668d395c63404ecc754ee36fe41#diff-cd3f7ac0caef6976249cf63ff8b1dbcfe1c3150d09e3b694a45cfbc3354c85d7 ```cpp void HardFault_Handler(void) { __asm volatile ( " tst lr, #4 \n" " ite eq \n" " mrseq r0, msp \n" " mrsne r0, psp \n" " ldr r1, [r0, #24] \n" " ldr r2, handler2_address_const \n" " bx r2 \n" " handler2_address_const: .word prvGetRegistersFromStack \n" ); } ``` was changed into: ``` void HardFault_Handler(void) { __asm volatile ( " tst lr, #4 \n" " ite eq \n" " mrseq r0, msp \n" " mrsne r0, psp \n" " ldr r1, [r0, #24] \n" " ldr r2, =handler2_address_const \n" " bx r2 \n" " handler2_address_const: .word prvGetRegistersFromStack \n" ); } ``` When I revert that change and rebuild the firmware, the `prvGetRegistersFromStack` is entered again, the LED shows SOS flashes and the system reboots instead of completely locking up. Reading the registers in pulFaultStackAddress, I get the following: link register is at ``` 0x802ec35 is in worker_thread_main (WICED/RTOS/FreeRTOS/WICED/../../wiced_rtos_common.c:173). ``` pc is 0. Is this perhaps a function pointer that is zero? See the screenshot below. ![image](https://user-images.githubusercontent.com/1708921/144717761-14f8f925-f221-492c-85a5-74e0ccaa3010.png) If my analysis is correct, this is a double bug report: - Hardfault handling is broken - There is a bug in WICED causing the hardfaults

The hard fault handler bug is in the system layer, that the Spark 2 and 3 download directly from Particle.
We can build our own system binaries to flash over USB, but not for over-the-air updates. Therefore, I’ll try to push get Particle to do a hotfix release tomorrow.

Arnt · February 21, 2022, 4:52pm

@Bob_Steers @Elco
My spark still having the hiccups, with the latest release.
It is no longer going into freeze, but is using too long time to restart. And when it do, it has to adjust outputs from zero.
Since my fermentation chamber is in my garage and I am living north of the polar circle and the temperature in the garage is below zero, the temperature in the chamber dropps like a stone.
Not so good for my fermentation

Would have been nice if the spark could check if it was doing any regulation before it was rebooted and then start with the same outputs again.

I am relly looking forward for a final solution on this issue.

Log: https://termbin.com/63qj1

Bob_Steers · February 21, 2022, 5:35pm

We can have a look at storing and then loading some semi-volatile values, but the reboot is a response to a system crash. At that point, any PID calculations are no longer accessible.

To prevent hiccups until we either find a fix (or a workaround for a bug in system software), we recommend making sure that your wifi router uses a fixed channel, and only has a single access point in range.

If desired, I can do a writeup of instructions for a Pi (any model, including zero W) USB to Wifi bridge. This would prevent hiccups, at the drawback of having an additional Pi attached to your Spark.

Arnt · February 22, 2022, 6:46am

I use google wifi mesh and can not set either channel or frequency, so then the only solution will be USB to wifi bridge. For that I need your help. I use RPI4.

Bob_Steers · February 22, 2022, 7:23am

Will do. The basic idea is straightforward enough, but I’ll need to double check the required configuration.

Arnt · February 22, 2022, 7:41am

In the meantime, I’m trying to set up one of my older routers in fixed channel repeater mode and only 2.4 MHz

Edit: Could not get it right, so I awaits USB solution.

Bob_Steers · February 22, 2022, 2:37pm

Creating a USB-to-LAN bridge for a Spark 3

To bypass the Spark Wifi, without having to connect your Brewblox server to your Spark over USB, you can use a second Pi as a bridge. This Pi will be discovered as a Wifi Spark, and can be used as such by Spark services.

Required:

Pi (any model)
USB cable

Installation

First, install the Raspberry Pi OS on your Pi.
After this is done, run:

sudo apt update
sudo apt install -y avahi-daemon socat

You do not need to install Brewblox on this Pi.

Preparation

First, we need to get the Spark device ID. To get this, run brewblox-ctl discover-spark on your Brewblox server, and copy the Device ID value.
Example device ID for a Spark 3: 400032001047383531363134.

Now stop your Brewblox services with brewblox-ctl down.

We want the Spark to no longer connect to Wifi. To clear Wifi credentials, hold the SETUP button until it first blinks dark blue, and then rapidly blinks light blue.

Configuration

We’re creating two configuration files:

a systemd service to start the bridge when the Pi starts.
an Avahi service to let the Pi be discovered as a Spark.

Create the systemd service file at /etc/systemd/system/spark-bridge.service

[Unit]
Description=Brewblox Spark Bridge

Wants=network.target
After=syslog.target network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/socat tcp-listen:8332,reuseaddr,fork file:/dev/ttyACM0,raw,echo=0,b115200
Restart=on-failure
RestartSec=10
KillMode=process

[Install]
WantedBy=multi-user.target

To load and start the systemd service, run

sudo chmod 640 /etc/systemd/system/spark-bridge.service
sudo systemctl daemon-reload
sudo systemctl enable spark-bridge
sudo systemctl start spark-bridge
sudo systemctl status spark-bridge

Create the Avahi service configuration at /etc/avahi/services/spark-bridge.service

<?xml version="1.0" standalone="no"?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="yes">%h Spark Bridge</name>
<service>
<type>_brewblox._tcp</type>
<port>8332</port>
<txt-record>ID=${DEVICE_ID}</txt-record>
<txt-record>HW=Spark 3</txt-record>
</service>
</service-group>

Replace ${DEVICE_ID} with the device ID you copied in the Preparation step.

To load and start the service, run

sudo systemctl restart avahi-daemon
sudo systemctl status avahi-daemon

Startup

Now if you run brewblox-ctl discover-spark on your Brewblox server, it should show a LAN Spark 3 with the IP address of your bridge Pi.

Start Brewblox with brewblox-ctl up, and your service should connect as normal.

Bob_Steers · February 22, 2022, 2:40pm

As a side note: your logs show parsing errors in your tilt calibration files. Could you please copy the contents of your brewblox/tilt/SGCal.csv file?

Arnt · February 22, 2022, 2:46pm

Can this be installed on a virtual machine?

Bob_Steers · February 22, 2022, 2:50pm

Your virtual machine would need access to the USB port, and the host network interface.

Arnt · February 22, 2022, 2:53pm

Black, 1.062, 1.064
Black, 1.042, 1.044
Black, 1.022, 1.024
Black, 1.012, 1.014

Bob_Steers · February 22, 2022, 2:54pm

Are there any empty lines in the file? It should handle those, but maybe there’s a bug.

Arnt · February 22, 2022, 2:56pm

Yes, there where 2 of them
Have fixed it, thx