[Brewblox-Tilt] Tilt data stops showing on fermentation graph

Plugins
1

@j616s
Reposting a continuation here from the already-too-long “Breblox Tilt Service” thread in an attempt to separate distinct issues.

Previously, BrewBlox + Tilt on a RPi3: the tilt service runs for a while (10’s of minutes to a few hours) but eventually data stops showing up in the graph. I “assume” that data is not being collected. The reason is unknown. Since data will come back (usually) upon restart of the tilt service, I presume the tilt battery is okay.

New information:
On the RPi3 build, I created a crontab to restart the tilt service once per hour. This improved data showing up on the graph, but eventually still ran into an issue with either tilt data still not showing up on the graph or the spark service becoming less reliable. Here, “less reliable” means both tilt and spark services reported shorter up-times than all the other containers). I find this strange since I only restarted the tilt service. Note also that the up-times for the tilt and spark services seldom matched … if everything else was up for 36 hours, the spark might be up for 9h and tilt up for 5h.

Second, just to rule out issues with RPi + bluetooth, I installed BrewBlox + Tilt on an x64 PC running Ubuntu 20.04 LTS. Unfortunately, the same (lack of stability) issues still occur, namely tilt data stops showing up on the graph after some number of minutes or hours.

Lastly, something I found interesting is that I left the whole thing alone for a day and the tilt data started showing up again, on it’s own, after a 14 hour hiatus … data stopped at 02:58 and started again at 20:13.

I’m happy to provide more specific details and/or try other troubleshooting steps. At this point, I’m just reporting my findings in the hopes something makes sense to someone else or triggers a new idea about a possible cause.

2 Likes
2

Is IPv6 currently enabled? I doubt it’d be the cause of this, but we do know that when IPv6 is enabled on the host, all containers get a network interrupt whenever one in the same bridge network restarts.

What is the uptime of the Spark controller? You can check this in the service page. Whenever the service loses connection and fails to reconnect within a minute, it will restart.

3

Excellent point. I think I forgot to do that. I’ll double-check the setting this evening and let it run overnight. I’ll post results back here.

(2020-05-21 UPDATE)
Yesterday afternoon I did

  1. disable IPv6
  2. stop brewblox
  3. reboot the machine

Tilt data has been collected successfully for the last 22 hours (at least, I don’t see any gaps on the graph, which is set to display duration of 1d).
All containers report 22h uptime EXCEPT spark, which reports 48 minutes. The small spark uptime isn’t too surprising and I attribute it to Spark hardware issues (as per: Spark V2 beeping repeatedly, now appears dead)

(2020-05-23 UPDATE)
Tilt data has been reported consistently on the graph for the past 70+ hours. No visible gaps on the graph. All containers report > 2d uptime except Spark, which reports 16h. Certainly looks to me that disabling IPv6 was what was needed for stable Tilt connection.

Reminder this is on Ubuntu 20.04. If this setup remains stable, I may try again on the RPi3 for comparison.

4

Not entirely sure it’s related, but I had some conversations with Noah about this and we landed on the baud rate of the UART being a contributing factor. With a heavily-used Pi 2 and multiple Tits I could pretty easily kill an aioblescan thread reliably. When I dial the baud rate WAY back, the problem nearly goes away.

Not sure Brewblox uses aioblescan or anything like it, but the btuart should be underneath.

1 Like
5

That sounds plausible. The brewblox Spark service uses a baud rate of 115200 over USB, but if memory serves, that’s not mandatory. We’d have to check whether Spark connection type impacts reliability of the Tilt service.

I don’t have a Tilt at home, but can probably set up an experiment with Sparks and Arduinos spamming serial, and BLE scans to detect phones / speakers / toasters that happen to have BT enabled.

We’ve been planning for a while to build a BT <-> TCP relay application that can be installed on either the Brewblox host, or on a Pi Zero W. This primarily to help with BT range issues, but if UART is a bottleneck, it would help with that as well.

The ETA on getting more involved with the Tilt is Soon™.
Right now we’re busy pushing the automation service through alpha. After that we intend to spend some time making the learning curve less vertical.

6

This is the baud rate on the btuart itself. it can be as fast as 921600 on the Pi 4. This is what I did for BrewPi Remix:

7

To add another data point: I’ve also been having the issue that tilt data recording stops after a few minutes.

I seems to be somehow related to Bluez, but I don’t know enough about it to debug it further. Running aioblescan from the command line in the background seems to keep data collection alive. This is as far as I got:

These are the last data brewblox_tilt received and since then it has been frozen:

tilt_1       | 2020/05/21 09:59:32 DEBUG    brewblox_tilt                   {'Temperature[degF]': 51, 'Temperature[degC]': 10.5555555555556, 'Specific gravity': 1.03, 'Signal strength[dBm]': -94, 'Plato[degP]': 7.558229019000095}
tilt_1       | 2020/05/21 09:59:32 DEBUG    brewblox_tilt                   {'Temperature[degF]': 51, 'Temperature[degC]': 10.5555555555556, 'Specific gravity': 1.03, 'Signal strength[dBm]': -92, 'Plato[degP]': 7.558229019000095}
tilt_1       | 2020/05/21 09:59:32 DEBUG    brewblox_tilt                   {'Blue': {'Temperature[degF]': 51, 'Temperature[degC]': 10.5555555555556, 'Specific gravity': 1.03, 'Signal strength[dBm]': -92,
'Plato[degP]': 7.558229019000095}}
tilt_1       | 2020/05/21 09:59:32 DEBUG    aioamqp.channel                 Exchange declared
tilt_1       | 2020/05/21 09:59:38 DEBUG    brewblox_tilt                   {'Temperature[degF]': 51, 'Temperature[degC]': 10.5555555555556, 'Specific gravity': 1.03, 'Signal strength[dBm]': -91, 'Plato[degP]': 7.558229019000095}
tilt_1       | 2020/05/21 09:59:38 DEBUG    brewblox_tilt                   {'Blue': {'Temperature[degF]': 51, 'Temperature[degC]': 10.5555555555556, 'Specific gravity': 1.03, 'Signal strength[dBm]': -91,
'Plato[degP]': 7.558229019000095}}
tilt_1       | 2020/05/21 09:59:38 DEBUG    aioamqp.channel                 Exchange declared

Now as soon as i run aioblescan manually, brewblox-tilt also starts collecting data again. As soon as I stop aioblescan, brewblox-tilt also stops collecting data:

pi@brewpi:~ $ date -u;sudo python3 -m aioblescan -T
Thu 21 May 10:09:22 UTC 2020
{"uuid": "a495bb60c5b14b44b5121370f02d74de", "major": 51, "minor": 1029, "tx_power": -59, "rssi": -94, "mac": "fb:2b:e2:c5:aa:77"}
{"uuid": "a495bb60c5b14b44b5121370f02d74de", "major": 51, "minor": 1029, "tx_power": -59, "rssi": -92, "mac": "fb:2b:e2:c5:aa:77"}
{"uuid": "a495bb60c5b14b44b5121370f02d74de", "major": 51, "minor": 1029, "tx_power": -59, "rssi": -96, "mac": "fb:2b:e2:c5:aa:77"}
^Ckeyboard interrupt

And the brewlblox logs:

tilt_1       | 2020/05/21 10:09:29 DEBUG    brewblox_tilt                   {'Temperature[degF]': 51, 'Temperature[degC]': 10.5555555555556, 'Specific gravity': 1.029, 'Signal strength[dBm]': -94, 'Plato[degP]': 7.312401509832824}
tilt_1       | 2020/05/21 10:09:29 DEBUG    brewblox_tilt                   {'Temperature[degF]': 51, 'Temperature[degC]': 10.5555555555556, 'Specific gravity': 1.029, 'Signal strength[dBm]': -92, 'Plato[degP]': 7.312401509832824}
tilt_1       | 2020/05/21 10:09:29 DEBUG    brewblox_tilt                   {'Temperature[degF]': 51, 'Temperature[degC]': 10.5555555555556, 'Specific gravity': 1.029, 'Signal strength[dBm]': -96, 'Plato[degP]': 7.312401509832824}
tilt_1       | 2020/05/21 10:09:29 DEBUG    brewblox_tilt                   {'Blue': {'Temperature[degF]': 51, 'Temperature[degC]': 10.5555555555556, 'Specific gravity': 1.029, 'Signal strength[dBm]': -96, 'Plato[degP]': 7.312401509832824}}
tilt_1       | 2020/05/21 10:09:29 INFO     brewblox_service.events         Closing <EventPublisher for "172.17.0.1">
tilt_1       | 2020/05/21 10:09:29 INFO     aioamqp.protocol                Recv open ok
tilt_1       | 2020/05/21 10:09:29 DEBUG    aioamqp.channel                 Channel is open
tilt_1       | 2020/05/21 10:09:29 DEBUG    aioamqp.channel                 Exchange declared
8

Just an update from me.

I updated my Pi on Saturday during my brew day (full OS update as well as Brewblox). I noticed when running apt, that there was an update for bluez-firmware (to 1.2-4+rpt3) and pi-bluetooth (to 0.1.13), along with a few other packages.

Since the upgrade on Saturday, by Tilt has been logging continuously ever since (now Wednesday afternoon). This is the longest I have seen it continuously run since the move from BrewPi to Brewblox. Not sure what changes have been made either to the above packages, or to the Tilt service, but this is looking pretty positive so far.

image
image714×590 36.1 KB

Also, really pleased with the input sanitation - no longer getting the big spikes which made the graph look flat! (I have two Tilts in here as only one brew on at the moment. I need to look at the calibration again as they never agree, but then I mainly use for trends rather than absolute values).

Thanks!
Jerry

1 Like
9

That’s fantastic news. Cheers for letting us know!