Spark 3 randomly rebooting

There’s a screen cap of the time since reboot. It appears to be stable now.

Thought it was good but had another reboot. https://termbin.com/bh12

Thanks for the log. Those really help with finding the problem.

The reboots are triggered by the firmware watchdog detecting that the firmware hasn’t done anything for a minute.

The last bug we fixed with this behavior was caused by the Wifi stack: it would hang if it couldn’t connect.
We’re not sure yet it’s the culprit again, but it’s a good place to start looking.

You can try removing the Wifi credentials by holding down the setup button for ten seconds until it rapidly blinks blue. If Wifi is the problem, that should stop the reboots.

In the meantime, we’re still trying to reproduce the bug, or find possible culprits in the source code. We’ll let you know if know something more.

His last log shows a hard fault reset, not a watchdog time out.
So I’ll try to reproduce it by running the same blocks.

@blkmagik98 Can you export your blocks from the Spark service page?

I can’t seem to grab them from the log export. (@Bob_Steers we should make that export compatible).

Here are the blocks. brewblox-blocks-spark-one.json (9.7 KB)

BrewBlox had been up almost 30 hours with no issues but had the screen up while I was cleaning my beer lines and noticed that it lost the IP address.

https://termbin.com/ppj3

And now the reboot… https://termbin.com/c5vw

The Spark 3 is currently up over 38 hours, which is the longest by far. The only things I changed were- rebooted the router and changed from a fermentation profile to a beer constant as fermentation was complete.

The Spark 3 has now been up for over 4 days with no issue, and I switched it back to the fermentation profile but with same temp over multiple days with no issues. Evidently the router reboot fixed whatever the issue was.

2 Likes

My Spark 3 keeps rebooting as well when directly connected to a power source instead of via the raspberry. I double checked my wifi and nothing seems wrong. (The whole house is running on a Unifi system and the Spark has a proper connection, channel is set to 6 and is not being interfered by the neighbours)

When connecting the raspberry to the same power adapter and the Spark to a USB port of the raspberry, everything runs fine. But when removing the Pi from the equation (moving it elsewhere), the Spark starts to reboot. The reset_reason_hex = ‘8C’ and reset_data_hex = ‘01’.

My log: https://termbin.com/h0r7

Thanks in advance and happy new year! (and thanks for all your work)

Thanks for the report!

Right now it’s hard to say exactly what causes this. The reset reason / data only tell us that there was a hangup, and the watchdog automatically restarted the system. You already eliminated other common reasons for Wifi issues (channel hopping / signal drops).

We already fixed multiple independent bugs that could cause this behaviour (there’s another fix in today’s release), and we’re working on adding better logging to the controller itself to improve feedback.

Actually, this might be a new data point. We have not been able to reproduce the reboots for debugging so far.

Can you add --discovery=wifi to your the command arguments of the spark-one docker container?
This would force connecting over wifi, even when connected to the Spark with USB.

I expect the issue to be related to the USB buffers on either the pi or the Spark side. If it occurs consistently on USB and not on WiFi in your tests, this would narrow it down. If they still occur with the WiFi argument, it is more likely a power issue.

Hey Elco, can you elaborate on how to add “discovery=wifi” to the docker container? I’m having an issue where the service is running on the pi and I can ping my spark but the ui says it is timing out and won’t connect. I’m getting this in the spark log:

spark-one_1 <date/time> brewblox_service.events Closing <EventListener for “eventbus”>
spark-one_1 <date/time> ox_devcon_spark.api.sse_api Starting SSEPublisher
spark-one_1 <date/time> _devcon_spark.communication Connection failed: ConnectionRefusedError([Errno 111] Connect call failed (‘192.168.12.167’, 8332))
spark-one_1 <date/time> ox_devcon_spark.api.sse_api Initial subscription push failed: TimeoutError()

It is described in more detail here: https://brewblox.netlify.com/user/connect_settings.html

Let me try this tomorrow, including a more powerful power supply. (It works with the raspberry in between, with the same power supply, but who knows!). thank you!

I upgraded the power supply to an iPad charger (10W) and added --discovery=wifi and the crashes seem less now. But still several times today: https://termbin.com/34ea

I have an dashboard from the fermentation and RIMS wizard and a home dashboard with a graph and 4 widgets that show my 4 digital actuators on the DS2413 chips. Do you want me to export my blocks?

Your log already includes blocks =)

This weekend we’re actually busy re-flooring the office. We’ll do some digging in your logs monday/tuesday when we’re back at work.

Thanks! I didn’t expect it any sooner :slight_smile: Good luck re-flooring!

Hi, I’m having the same random reboots. It doesn’t seem to be more than once every day or 2, but sometimes it can be multiple times a day. I’m connecting over wifi.

I’m on version 98ffcff1 which I think is the latest. I just wanted to give you guys a heads up and to let me know how I can help you guys track this down.