I have been getting disconnections from the BrewBlox Raspberry Pi every day now even though everything points to the Spark being online and connected to the network.
The LED does not blink and the ip address shows connected with a valid IP. I can also see the MAC address connected to my network with a stable connection. I currently have it set with a static IP on my local network as well. The issue seems to be with the service on the spark rather than a hardware or wifi issue.
I am able to use it for an entire day but reliably, the next day, the BrewBlox UI shows it as unconnected and continuously restarts the service trying to connect. When I try to connect to the IP address via web it won’t load and the brewblox-ctl http command shows an error as well but I am able to ping the IP.
I can fix the connection every time by just forcing the spark to reconnect to wifi via my router’s management interface.
Logs: https://termbin.com/mofj
You may want to run brewblox-ctl disable-ipv6.
There’s a bug in the Docker network stack where if the router supports IPv6, a starting container will interrupt all other network interfaces.
It may not be the only issue, but I’m seeing a lot of the kind of events that are caused by this in your dmesg output:
[928759.849234] eth0: renamed from veth4431494
[928759.910556] br-326df2cc55a6: port 7(veth2007a81) entered blocking state
[928759.910580] br-326df2cc55a6: port 7(veth2007a81) entered forwarding state
[928765.738342] br-326df2cc55a6: port 8(veth9ea4ad2) entered disabled state
[928765.739080] eth0: renamed from veth3cb891f
[928765.809130] br-326df2cc55a6: port 8(veth9ea4ad2) entered blocking state
[928765.809148] br-326df2cc55a6: port 8(veth9ea4ad2) entered forwarding state
[928776.358309] br-326df2cc55a6: port 10(veth4d64154) entered disabled state
When your service connects to your Spark, it logs that around 14:35 UTC, the Spark crashed and rebooted:
spark_1 | 2021-01-19T15:33:45.316242761Z 2021/01/19 15:33:45 INFO ...lox_devcon_spark.connection HandshakeMessage(..., reset_reason='PANIC', reset_data='WATCHDOG')
spark_1 | 2021-01-19T15:33:45.491680774Z 2021/01/19 15:33:45 INFO ...evcon_spark.synchronization System uptime: 0:57:04.511000
Was this an unattended crash, or something caused by you tinkering with the system?
I haven’t been doing any tinkering, just monitoring fermentation at the moment.
The network interface messages are because I use the hardware as a VPN interface for my home network as well so it is doing network forwarding, should be unrelated to this issue as the error seems to be with connecting the the spark and not the pi.
I have also already disabled ipv6 and running the command again just says it is already disabled.
It may cause connection resets. I don’t think I ever combined a VPN and Brewblox on the same Pi, so it certainly is a new scenario. @Elco is Spark-side connection exhaustion a possibility (overloaded with simultaneous / aborted connections)?
I can’t draw too many conclusions from the log, as it only goes back a few minutes.
Ah that would certainly be interesting, it probably is the multiple reconnections causing the spark to finally crash, I was thinking that the data was being polled but if it is using a constant connection then that would probably tax the spark more than it was meant to
The Spark is -very- memory constrained, and indeed uses a constant connection to stream data to reduce overhead.
If it’s an option, could you check whether the problem persists when VPN is disabled?
We can’t do much about network / network interface interrupts, but reconnects triggering a Spark crash does sound like something we can and should fix.