Spark service continuously reconnecting after CommaindTimeout() errors

Troubleshooting
1

I keep losing sync between the spark-one service and my brewpi.
Firmware has been flashed yesterday and I’ve reinstalled brewblox and the tilt integration from scratch to eliminate any prior config issues.

https://termbin.com/v3k3

Getting a lot of this sort of thing every 2-3 minutes…

spark-one_1  | 2021/02/24 17:54:08 INFO     ...evcon_spark.synchronization  Service synchronized!
spark-one_1  | 2021/02/24 17:55:11 ERROR    brewblox_service.repeater       <Broadcaster> error during runtime: CommandTimeout(ListObjectsCommand)
spark-one_1  | 2021/02/24 17:55:11 INFO     brewblox_devcon_spark.spark     Checking connection...
spark-one_1  | 2021/02/24 17:55:17 ERROR    ...blox_devcon_spark.commander  Error parsing message `7A0005BA|0000,010080FEFF81C4,02008000010A0C23002A000A473531383831381208636166356564646618063A083639356364626631420A323032302D31312D30324A0A323032302D31302D30380099,0300800101088FEAA31A10EBA5DA8106180128053001002C,0400800201008E,05008038010A08626161687977656828BEFFFFFFFFFFFFFFFF01320D3139322E3136382E342E323434000E,060080390108012040284000A2,0700803A010A1308021203DF2B351A08486561742050494470700A1308011203037CD51A08436F6F6C20504944706F0A1708031203D66C001A0C426565722053657474696E6758680A1408061203A3A3001A09426565722054656D7050650A17080412034C4C4C1A0C416D6269656E742054656D7050640A16080512034169E11A0B4672696467652054656D7050661218427265776572792038343535204B6567657261746F722031180100DD,13008040010A060A04080210010A0212000A021A000A042202080140020041,6400012E01088088082128FF7E5E9015012F0041,6500012E0108FF1F2128FF451C901503800035,6600012E010880800D2128FF1F8590150166002B,6700012F0110662880800A30EEFD0C38014080800A48015080C0025880800D00D1,6800012F011065288E4730FF1F3801408E4748015080C00258FF1F00E7,690001360110809F49004C,6A00013E01081310042A190A0408E0A7120A0410C0A9070A0B2209086910E0E5A401180100A6,6B00013E010813100118012A0E0A0C220A086910809F4918012001300100FD,6C00013301086A18C0EE6D386A40010058,6D00013301086B18904E2080803228808032386B4001488080320025,6E000137010A04108080050A080880F524108080050A080880DE341080E00520683096F3D9810600C3,6F000130010868106C28FF1F308E475801600168FFFF1870E0A80178880E8001BBA32888011D98018E67B0016CD8010600F3,70000130010868106D28FF1F308E473880803240808032580160016880803270E0A80178880E8001F8C65088013C98018E67B0016DD8010600DC` : ValueError(Unexpected message)
spark-one_1  | 2021/02/24 17:55:17 INFO     ...lox_devcon_spark.connection  HandshakeMessage(name='BREWBLOX', firmware_version='caf5eddf', proto_version='695cdbf1', firmware_date='2020-11-02', proto_date='2020-10-08', system_version='2.0.0-rc.1', platform='photon', reset_reason_hex='8C', reset_data_hex='02', device_id='23002A000A47353138383138', reset_reason='USER', reset_data='CBOX_RESET')
spark-one_1  | 2021/02/24 17:55:20 INFO     brewblox_service.repeater       <Broadcaster> resumed OK
2

Spotty wifi can trigger a soft hangup in the controller. It seems to deal badly with router channel switches, but we’re also aware of a bug in the Wifi network stack.
For the first, we recommend setting your router to a fixed channel. For the second, we’ll release a fix soon. If your Spark is close to the Pi, you may also just switch to USB for stability.

3

WiFi in that part of the basement is probably the “worst”, though only about 10 ft from an access point with a wall between… Signal strength for both devices are between -68 and -58 dBm for the last 24+ hours on both devices. From the client perspective, there has been no roaming or channel changes, and zero Tx drops/retries for either device, though the AP is reporting a fair amount (60-90 packets/sec) of Rx drops in the 2.4Ghz band. Probably due to the low tx power of the rpi2 and spark WiFi chipset/antennae?

First, I’ll relocate the rpi running brewblox to see if it’s the rpi that’s having the wifi issues. If that doesn’t do it, I’ll see if I can relocate the in-ceiling access point. Moving the fermentation chamber and attached brewpi is harder than the ceiling tiles and structured wiring. :slight_smile:

Is there a thread or issue for the wifi stack bug I can link to and keep tabs on?

4

Likely, but unconfirmed. I’ve been looking into it, but right now my data is n=1 in a system without any problems. I live in an apartment, so no bad wifi coverage spots either.
We’re adding some wifi diagnostics to brewblox-ctl log next release, and packaging some tool people can run to collect wifi stats. We have no feasible way to collect this data for the controller though - memory restraints are much too severe.

I put up an issue for it with informal discussions from slack.

5

I have just merged some fixes that seem to help.
It is running on our test setups now and will be released soon.

6

wonderful. Hope the tests pass. Thanks Elco.