External Plugin basics

I have a BBQ Controller that has a terrible interface and associated phone app, I have worked out how to get the data from it and want to feed it into Brewblox purely for UI/ graphing and metrics tracking.

Is https://github.com/BrewBlox/brewblox-boilerplate still the right place to start?

I will also draw a lot from https://github.com/j616/brewblox-tilt which is does the same as it seems to be well maintained.

Yes, brewblox-boilerplate is the right place to get started for backend services.

The Plaato plugin is another boilerplate-based service simple enough to be used as example.

gold, Plaato seams like it is pretty clean, great starting point.

I have my Repeater collecting and pushing metrics to influx, which was super easy to setup.

So I have:

           await mqtt.publish(
                    'key': f"ShareMyCook[{device_data.name}]",
                    'data': device_data.serialize()

key = ShareMyCook[UltraQ]
data is:

    'active': 1, 
    'fan_duty[%]': 100, 
    'pit_temp[c]': 17.0, 
    'pit_target[c]': 210.0, 
    'food1_temp[c]': 18.0, 
    'food1_target[c]': 65.0, 
    'food2_temp[c]': 20.0, 
    'food2_target[c]': 65.0

I would like to have it like how a spark is shown where the main container is ShareMyCook, then each of my devices has the keys that are in the data. This way I can detail multiple devices under the one data source.

If you understand you correctly, you want to publish to key=ShareMyCook, and data={'UltraQ': {...} }.

https://brewblox.netlify.app/dev/reference/event_logging.html#history describes how nested data is flattened.

You can publish different contents of data, and they will be merged automatically.

Side note: our system automatically recognizes degC / degF in brackets.

Cheers for all of that, I have a functional docker image now.

What platform are you building on?

Some Interesting things with building the docker image:

  • For the life of me, I couldn’t get buildx to become available on unbuntu 20 LTS, enabling experimental mode via the env variable or docker json just wouldn’t work, so I ended up doing it on my Mac.

  • readline -f doesnt work the same as gnu readline, so I ended up installing greadline via homebrew, which is the same.

  • cp dist/ docker/ doesn’t seem to copy anything on a mac, but cp dist docker/ does, which should also work under most linux distros too.

Not sure what you mean by platform. Our CI builds are on Azure, using x86.

What version Docker are you using on ubuntu?

There are two experimental flags in docker: one for the engine, one for the CLI. As far as I know, only the CLI flag is required, but you can try setting them both.

https://github.com/BrewBlox/brewblox-firmware/blob/develop/docker/enable-experimental.sh is the script in firmware we use to set the flags.

https://github.com/BrewBlox/brewblox-firmware/blob/develop/docker/prepare-buildx.sh sets up buildx. This is pretty much the same as the commands described in boilerplate.

Note that in Azure CI (and probably others), buildx setup is subtly different, because they use Docker EE.
The required config for that can be found in the boilerplate pipelines file.

I’ll change the dist vs. dist/ call. Where does the readline -f happen?

I’ll look into the firmware setup stuff and see.

Not sure what you mean by platform.

I meant development. I was following your guide, and lots of little things didn’t match up or work out of the box, so I was just wondering what your development environment looks like, as I assume you iterate with docker locally, prior to committing and running the full CI process.

Where does the readline -f happen

Sorry, readlink -f

What version Docker are you using on ubuntu?

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04 LTS
Release:	20.04
Codename:	focal

$ DOCKER_CLI_EXPERIMENTAL=enabled docker version
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Tue Jun 23 22:26:12 2020
 OS/Arch:           linux/amd64
 Experimental:      true

  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Thu Jun 18 08:26:54 2020
  OS/Arch:          linux/amd64
  Experimental:     false
  Version:          1.3.3-0ubuntu2
  Version:          spec: 1.0.1-dev
  Version:          0.18.0

$ DOCKER_CLI_EXPERIMENTAL=enabled docker buildx version
docker: 'buildx' is not a docker command.
See 'docker --help'

Elco and me both use Ubuntu 20.04, and we both have separate machines at home and at the office.

We use a few shortcuts to save time, and our CI pipeline is set to build feature branches, but yes, we do commonly start local image builds.

According to docs, buildx is available in docker 19.03, which you have. How did you install Docker? (Snap, install guide, get.docker script, etc)

I suspect any omissions in the install guide are caused by the “it works on my machine” issue, where we forgot about one or more config changes, or happen to have a required package pre-installed for an unrelated purpose.
Pyenv / poetry typically are the most fiddly to get set up. Does this match your experiences?

So far, we’ve spent quite a lot of time improving and automating dependency/config management in dev, CI, and production - and it has been more than worth it.
I’ll make an issue to do a squeaky clean OS install on a spare disk to verify the instructions.
Feel free to mention any and all speed bumps you encounter.

I assume in docker/before_build.sh? It looks to be overkill. That line can be replaced with:

pushd "$(dirname "$0")/.." > /dev/null

Just apt install docker, I spun up an EC2 instance with Ubuntu 20.03, just to test fresh, and it wanted to install it via snap, which I am still getting my head around, so I gave up when it acted the same.

Yeah especially on my mac with homebrew, I use zshell too, so there were a couple of things that I needed to sort out. I definitely attribute it to my setup though.

Poetry kept wanting to use my system interpreter, which by default on Mac is still Python2.7.

When I took out my specific variance, it was only those two points that I brought up that I thought worthwhile contributing back.

For development, I use IntelliJ IDEA with the Python plugin, so basically PyCharm. With the poetry plugin it found the venv no problems, and I just ended up just debugging with the entrypoint as __main__.py. Breakpoints and interactive debugging worked perfectly, so iterating quickly was no issue at all. I only went to docker after I was code complete.

I just followed your documentation for setting up Azure pipelines. The only section that I needed to stray from was this:

DOCKER_PASSWORD is your Docker Hub password. Make the value secret by clicking the lock icon.

I have two factor auth setup, so I needed to go into Account Settings -> Security and create an access token and use that for the password.

Otherwise, very complete and helpful.

I can see myself building more of these things.

One thing that I would like to build is a notifications plugin so that I can send myself telegram messages when thresholds are exceeded, or have it join a channel as a bot so I can send it requests for up-to-date metrics. The BBQ for example, I leave overnight, so it is good to know if the fan kicks up, or the temp rises/drops too much.

Oh, I case you were wondering, here is my fork of boilerplate https://github.com/clarkadamp/brewblox-sharemycook

Thanks, I’ll add a note about that.

Sounds very useful. If you wish to decouple the functionality and/or include data from other services: the brewblox_service.mqtt module also includes subscribe/listen functions.

That looks really nice. Do be careful if you run brewblox-ctl log: you’re currently logging credentials if you have a login error.

Edit: I ran a fresh Ubuntu install, and then followed the boilerplate instructions.
If you install docker with sudo apt install -y docker.io, buildx is not available.

To fix this, run:

sudo apt purge docker.io
curl -sL get.docker.com | sh

I’ll update the boilerplate instructions to include docker.

After getting my head around aresponses I finally have unit testing complete, but I get this warning

  /Volumes/dev/projects/brewblox-sharemycook/.venv/lib/python3.7/site-packages/brewblox_service/features.py:48: DeprecationWarning: Changing state of started or joined application is deprecated
    app[FEATURES_KEY] = dict()

But I can’t find the deprecation warning.

Is that something I need worry about?

I have brewblox-service = "^0.30.1" in my toml file (IntelliJ prompted me to update), boilerplate had "^0.28.0"

Also, what do you use to document use cases and design decisions? I would like to start gathering some use cases for alerting and actions.