BrewBlox Tilt Service

frizzo · August 7, 2019, 1:30am

Thank you. No worries & No rush. Not brewing this week++. Sorry to be a pain. This is all much more complex and time-consuming than the BrewPi software ever was.

Also, just tried a reset of the env to clear old test data as I figured out what I wanted to name the various blocks… since, every time I changed a name the old metrics/names persisted (expected this based on the design.)

How do I get the tilt service back in the UI?
Also, should the Tilt service be displayed in the left-hand “Services” panel like “spark-one” is?

I ran:
curl -k -X DELETE https://localhost/spark-one/objects
brewblox-ctl setup
Do you want to check whether ports are already in use? [Y/n]
n

Do you want to update brewblox-ctl? [Y/n]
n

This directory already contains a docker-compose.yml file. Do you want to keep it? [Y/n]
y

This directory already contains datastore files. Do you want to keep them? [Y/n]
n

This directory already contains history files. Do you want to keep them? [Y/n]
n

This directory already contains Traefik files. Do you want to keep them? [Y/n]
n

After restart and adding a simple graph to the home dashbord, there is no “tilt” metrics tree displayed in the Graph widget config (which was there before I reset things). The docker container is running, and the tilt docker logs look the same as before (above). I’ve tried taking things down and up again.

Bob_Steers · August 7, 2019, 8:26am

As these are mostly BrewBlox questions:

Graph / Metrics widgets will only show fields that have been updated the last day. The actual data will not disappear, but stale fields will no longer be shown in the selection tree. (There’s an issue for mentioning this in the graph settings popup)

The Tilt doesn’t have a dedicated service or widget in the UI (for now). If the container is publishing data, it will automatically be available in the Graph/Metrics widgets.

Your approach to wiping everything is correct: this removed everything except the docker-compose.yml file.

After a data wipe, the tilt metrics are shown as soon as the container publishes data to the history database. The history keys are refreshed when you open the graph/metrics settings windows.

If you open either the Graph or Metrics settings after a few minutes, and the Tilt is still not visible, then there’s likely an issue in the Tilt service. I’m afraid I can’t help much with these.

frizzo · August 7, 2019, 2:44pm

Thanks Bob. Will wait for James to have a look when time permits.

So far in analyzing the logs, not sure I see any smoking guns - other than the tilt logs: " Failed to execute management command ‘scanend’ (code: 11, error: Rejected)" of course.

Did notice an eventbus connection error as well. Likely due to an async parallel startup issue, or something wrong?

history_1 | 2019-08-07T01:47:47.186341973Z /usr/local/lib/python3.7/site-packages/brewblox_service/events.py:241: UserWarning: Connection error in <EventListener for “eventbus”>: ConnectionRefusedError([Errno 111] Connect call failed (‘172.24.0.5’, 5672))

Bob_Steers · August 7, 2019, 2:51pm

The eventbus errors at startup are nothing to worry about. The Tilt service is a bit quicker than the RabbitMQ eventbus, and will automatically retry until it works.

frizzo · August 7, 2019, 3:19pm

@Bob_Steers, Figured it was something like that.

@j616s, Additional triage:
Scenario 1: If I down all the services and reboot the rpi, then up all the services, the tilt service event data points are stored and metrics are visible, though still getting the “scanend” errors in logs.

Scenario 2: If I down all the services, then bring them back up, tilt service event data points are not getting stored and no metrics are visible, yet can find nothing different or indicative of a unique error/warning in the logs.

Edit: T+30m… Tilt service spontaneously stopped logging approx. 6m after it began working upon startup in Scenario 1.

Note: Currently the Tilt is just in a jar of water, separate from the ferm chamber so temp offsets are expected.

update 2: blescan from tilt docker container fails too.

pi@rpi2:~/brewblox$ docker exec -it brewblox_tilt_1 /usr/local/bin/blescan
Scanning for devices…
Traceback (most recent call last):
File “/usr/local/bin/blescan”, line 10, in
sys.exit(main())
File “/usr/local/lib/python3.7/site-packages/bluepy/blescan.py”, line 122, in main
devices = scanner.scan(arg.timeout)
File “/usr/local/lib/python3.7/site-packages/bluepy/btle.py”, line 854, in scan
self.stop()
File “/usr/local/lib/python3.7/site-packages/bluepy/btle.py”, line 803, in stop
self._mgmtCmd(self._cmd()+“end”)
File “/usr/local/lib/python3.7/site-packages/bluepy/btle.py”, line 312, in _mgmtCmd
raise BTLEManagementError(“Failed to execute management command ‘%s’” % (cmd), rsp)
bluepy.btle.BTLEManagementError: Failed to execute management command ‘scanend’ (code: 11, error: Rejected)

j616s · August 7, 2019, 8:55pm

Yeh. It looks like the same issue as before. It’s the BT stack on the host. The BT stack on the Pi can be a bit unstable. Could you let me know which model Pi you have, which version of bluez you’re running on the host, if you installed it with the command on the Readme for the Tilt service or direct using apt?

frizzo · August 7, 2019, 9:26pm

Pi 3 Model B (China rev. a22082)
Linux rpi2 4.19.58-v7+ #1245 SMP Fri Jul 12 17:25:51 BST 2019 armv7l GNU/Linux
Raspbian GNU/Linux 10 (buster)

pi@rpi2:~/brewblox$ apt -a list bluez
Listing… Done
bluez/testing,now 5.50-1+rpt1 armhf [installed]
bluez/stable 5.50-1+b12 armhf

I can back down to the stable version, but had problems with hcitool until I went to 5.50-1+rpt1.
After that, BT seemed solid on the base OS.

Also had some fun waiting for a stable build of docker.
pi@rpi2:~/brewblox$ apt list docker-ce
Listing… Done
docker-ce/buster,now 5:0.0.0-20190727010531-15bdbd7-0~raspbian-buster armhf [installed]

Could it be an issue that the tilt docker image is using an older version still and we’ve got some sort of compatibility issue on top of Buster?

frizzo · August 8, 2019, 12:40am

Ok, to eliminate the potential issues with Buster, I rebuilt on Stretch and used the nightly build of bluez per the the Github Readme.
e.g.
wget http://http.us.debian.org/debian/pool/main/b/bluez/bluez_5.50-1_armhf.deb && sudo dpkg -i bluez_5.50-1_armhf.deb && rm bluez_5.50-1_armhf.deb

Noticed a Spark FW update tonight too

Current OS/package details:

Raspbian GNU/Linux 9 (stretch)
Linux raspberrypi 4.19.58-v7+ #1245 SMP Fri Jul 12 17:25:51 BST 2019 armv7l GNU/Linux
docker-ce/stretch,now 5:19.03.1~3-0~raspbian-stretch armhf [installed]
bluez/now 5.50-1 armhf [installed,local]

Logs: https://termbin.com/bvay

Same problem with only a few minutes of BT data from the Tilt logged to Brewblox.
The iPhone app continues to update, so I know the Tilt is still transmitting it’s beacon announcements. I was a slightly delayed before setting up the Fridge wizard, and by then, the Tilt service had malfunctioned already. Here’s the full auto-zoomed graphs of the data set on my latest rebuild with Tilt added before first startup:

j616s · August 8, 2019, 6:40am

This is more or less my setup now, too. Except I think I’m on the stable bluez now and used the Stretch docker because buster wasn’t available when I last installed. But this problem has been around on-and-off longer than buster…

I’m wondering if the Pi’s can have one of many different BLE chips on where some are more stable than others and its pot luck which you get… It’s the only reason I can see that explains why it would work for some people but not others.

frizzo · August 9, 2019, 1:51pm

In case it’s worth tracking for this issue, here’s my HW info.
I don’t have another rpi to try or a USB BT adapter sitting around to experiment with.

Edit: I should note that back in the old pre-docker brewpi days, @sbowler’s code and BT integration was working flawlessly with my HW.

pi@rpi2 : ~ $ sudo hciconfig -a
hci0: Type: Primary Bus: UART
BD Address: B8:27:EB:F9:B7:FF ACL MTU: 1021:8 SCO MTU: 64:1
UP RUNNING
RX bytes:67872 acl:0 sco:1 events:2995 errors:0
TX bytes:7465 acl:0 sco:0 commands:482 errors:0
Features: 0xbf 0xfe 0xcf 0xfe 0xdb 0xff 0x7b 0x87
Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3
Link policy: RSWITCH SNIFF
Link mode: SLAVE ACCEPT
Name: ‘raspberrypi’
Class: 0x000000
Service Classes: Unspecified
Device Class: Miscellaneous,
HCI Version: 4.1 (0x7) Revision: 0x168
LMP Version: 4.1 (0x7) Subversion: 0x2209
Manufacturer: Broadcom Corporation (15)

pi@rpi2 : ~ $ cat /proc/cpuinfo | grep Revision
Revision : a22082

Which is an Embest Ltd. Chinese board.

j616s · August 11, 2019, 12:41pm

Sorry I haven’t picked this up yet. I’ve caught tonsillitis so out of action atm.

Bob_Steers · August 11, 2019, 1:17pm

Oh, that’s not nice. Hope you get well soon!

frizzo · August 18, 2019, 12:06am

Oh!!! Sorry to hear that. Take care and get well soon.

jerryp · August 22, 2019, 1:30pm

Hell @j616s,
I hope you have recovered!

Thank you for the work getting this together. I used to use the Tilt with my BrewPi. Finally got around to rebuilding my RPi as BrewBlox and getting my head around it.

I followed your instructions at https://github.com/j616/brewblox-tilt - the only difference is that I am now running the latest bluez following an update.

At first I could not log anything, running:

docker logs brewblox_tilt_1 -f

Just returned a lot of lines like:

2019/08/22 12:48:19 ERROR brewblox_tilt Encountered an error: Failed to execute management command ‘scanend’ (code: 11, error: Rejected)

Rebooting appears to have started logging fine, but after about 30 minutes, this appears to have stopped and the logs are back to the above error. So, it does look like the Bluetooth drops after some time.

For information, I believe I have a Raspberry Pi 3, Model B and two Tilts - Black and Red…

Docker logs for the Tilt service: https://termbin.com/zlp6

Kernel, hardware and system information:

pi@fridgepi:~ $ cat /proc/version
Linux version 4.19.66-v7+ (dom@buildbot) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1253 SMP Thu Aug 15 11:49:46 BST 2019
pi@fridgepi:~ $ cat /proc/cpuinfo
processor : 0
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 76.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

processor : 1
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 76.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

processor : 2
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 76.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

processor : 3
model name : ARMv7 Processor rev 4 (v7l)
BogoMIPS : 76.80
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 4

Hardware : BCM2835
Revision : a02082
Serial : 00000000a5326403
pi@fridgepi:~ $ cat /etc/os-release
PRETTY_NAME=“Raspbian GNU/Linux 10 (buster)”
NAME=“Raspbian GNU/Linux”
VERSION_ID=“10”
VERSION=“10 (buster)”
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL=“http://www.raspbian.org/”
SUPPORT_URL=“http://www.raspbian.org/RaspbianForums”
BUG_REPORT_URL=“http://www.raspbian.org/RaspbianBugs”

Let me know if you need any further information. I am happy to help test where I can.

Thanks!

j616s · August 22, 2019, 5:41pm

Hello. I’m much better, thanks

Just trying to find time to investigate this. Sorry I haven’t been able to pick it up sooner. If anyone else finds time and fancies taking a look, I do accept pull requests

jerryp · August 22, 2019, 9:29pm

I would love to help, but it has been many years since I did much development work! I can potter about with Python and some others, but would probably take me a long time to work anything out…

j616s · August 26, 2019, 9:19am

FYI. Finally got time to look at this. Investigating now

j616s · August 26, 2019, 1:05pm

Sounds like a long standing race condition in bluepy. Tried experimenting with an alternative but will need to play with it more. I’ve added to the issue on bluepy but doesn’t look like it (or any other bluetooth libs for that matter…) are under active development… I’d like to avoid the solution from the old brewpi tilt implementation as it’s a bit verbose. But I might have to adapt that if I can’t find an alternative.

Bob_Steers · August 26, 2019, 2:53pm

I believe the race condition is only present in the bluetooth scan?

An alternative hack is to find another language with a correct bluetooth implementation, and let that dump the scan results to stdout.

j616s · August 26, 2019, 4:02pm

Yeh. There was an issue where scanning once would result in a few updates then nothing, though. I think given the Tilt is BLE, it has to constantly “scan” anyway as it doesn’t pair. Working with bluetooth in general just feels like unstable hack on unstable hacks tbh…

And that’s another option. But I’d really like to do things right.