Spark 3 keeps restarting

#1

In the last ten minutes the spark 3 (on the other side of the room) has restarted itself 3-4 times. I don’t even have an active fermentation going and haven’t used the BrewBlox UI or interacted with any BrewBlox services/tools in a few days. I have noticed it occasionally restart on its own, but I assumed it was a power fluctuation or something and I never noticed this many in a row.

How should I go about troubleshooting this issue? I can’t tell if the log tool is capturing what is happening on the spark side.

#2

That sounds like the Spark is encountering hard faults.

The LED will flash a status code in red if this happens: https://docs.particle.io/tutorials/device-os/led/photon/#red-flash-sos

Does the problem persist after power cycling the Spark?

There is an open issue for logging reboot reasons. I’m not aware of earlier reports where the Spark suddenly started rebooting, after being fine for days.

#3

How is the spark powered? Is the connection to the service working normally?

#4

Hi, for info I have a similar situation with my V2.

It only seems to manifest when it is idle i.e. between ferments and the PID’s are disabled. I didn’t give it too much attention given all the updates that are being issued. I can’t see any pattern to it as it seems quite random. I noticed it in particular yesterday whilst brewing, it sounds a beep and the screen resets and all appears normal again. I also noticed on the service page the Spark widget would show varying up times (minutes/hours) despite it being on but idle for extended periods.

Now it is “on” as I am fermenting I haven’t noticed any restarts.

My V2 is currently connected via USB to the RPi without the external power supply. I did notice it recycle also when on wifi only and with the external supply. (Since all the updates/firmware I have left them connected)

Dave.

#5

USB vs WiFi might be related. We could trigger a hard fault reboot when we sent a lot of data during a block import but fixed that last week by limiting chunks on the python side. The problem is a usb buffer overrun.

@adempewolff are you on WiFi or USB?

#6

Sorry, was working against a deadline for the last couple days.

I don’t notice any LED status codes, but I could be missing them. Similar to Trig, I just hear a beep and when I look over to the spark it is on the boot screen and the LED is flashing the normal boot-up connect to WiFi colors/sequence. It seems to work perfectly fine both before and after the reboot, but not managing an active fermentation it’s hard to tell to what extent there are no effects.

I’m actually not sure whether it is using USB or WiFi. It was set up to use Wifi and be powered by a Meanwell MDR-60-12, however, recently I’ve left the pi nearby and them both connected by USB because of the steady stream of firmware updates. I don’t think power is cycling to the cabinet as the pi takes a while to boot up and I can connect to it immediately after the reboot. While it’s theoretically possible that the 12V circuit is cycling without cabinet power cycling, I don’t find that especially likely.

#7

My Spark V1 Rev C with a Photon upgrade has been doing the same thing for a few weeks now. It randomly reboots when connected over only WiFi and over only USB. Sometimes it goes up to 4 days before a reboot, other times its rebooting every hour or less. It started while I was adding fermenter #3 and at first it seemed that disabling all temperature profiles and beer settings then reenabling the temperature profiles would stop it for a few days, but now it appears to only slow the multiple times an hour reboots. All 3 fermenters are being controlled by temperature profile and the rebooting doesn’t seem to have any adverse affects on temperature control at this point. I have been watching the community posts and upgrading at every release with no improvement. Let me know what you need and Ill send over. Thanks again, I’m loving the multi-chamber setup and flexibility of the new BrewBlox.

#8

We’ve been having trouble reproducing this issue (current uptime on our test fridge: 11 days). A possible correlation with Setpoint Profiles is new though: we’ve set up a test case to see whether that makes the difference.

Apart from that, we’re implementing that after a reboot, the controller sends a (logged) message to the service with the reason. This should at least provide us with some more information.

2 Likes
#9

Plus 1 on random reboots. Confirmed this is happening when controller idol. My reboots happen about every 5 minutes. Pi v2. I have powered the Pi via USB as well as an external power supply

#10

Could you please export your current blocks? We’re still trying to triangulate the issue.

#11

Bob, my blocks are currently default if the classic pi. Is there a log I can give you?

#12

Spark page -> actions -> Import/Export blocks.
This can do a dump to file.

brewblox-ctl log generates an URL for backend logs. This may be useful as well.

#13

Another restarting spark. Mine is restarting every few minutes. I’ve just run an update and now the blocks aren’t showing at all and the actions drop down is shaded and won’t let me export anything.
Powered over usb (it’s own supply).

#14

The troubleshooting widget is also not visible in the Spark page?

In the UI sidebar is a bug menu (bottom left). Could you please run the “Export API errors” button there, and brewblox-ctl log on the Pi?

#15

I’ll do that when I get back home.

1 Like
#16

brewblox-errors.json (29 Bytes)

#17

brewblox-errors (1).json (461 Bytes)
brewblox.log (175.6 KB)

#18

Thanks! That should be pretty helpful to get more insight in what’s happening.
The current theory is that it’s caused by a concurrency issue in the Wifi stack. We’re sure there’s a bug there, and are now figuring out whether this is the one causing the reboots, or that we should keep looking after we fix it.

1 Like
#19

Finding the cause of the reboots was a deep rabbit hole. I found some issues in the firmware that might be the cause and are fixed in the edge release we just pushed.

The biggest one seems to be a thread-safety bug in Particle’s device-os, which could cause a hard fault (and thus a reboot) when WiFi was reconnecting in the background while trying to read the IP address for displaying on the LCD.

So please update your firmware and let us know if the reboot issues are still present.

1 Like
#20

updated to the new version, so far so good.

1 Like