Hooking into another Traefik instance

I have my brewlox installed on an ubuntu machine and I run a bunch of other docker images other than brewblox on a range of ports so I can co-host them.

I started looking into Traefik and I would like to us it for other use cases, so I have a few options:

  1. start my own instance seperate to BrewBlox, modify BrewBlox to suite
  2. make my other instances hook into the BrewBlox instance
  3. Configure another IP and bind a seperate instance to that, keeping them seperate.

I haven’t found any documentation on how to do 3, the ability to scope/assign docker labels to a given Traefik instance.

So then, the next choice is probably based on how the docker-compose.shared.yml is maintained.

I know I could probably remove the traefik service from the shared yaml, but will this get overwritten with the next brewblox update?

I ideally I would like to run my own Traefik and have brewblox hook into it via it’s labels.

Do you want your other images to share ports with Brewblox?

Do you want Traefik to handle SSL certs?

I would not recommend option 3. It would not solve your problems, and create a whole set of new ones.

We do overwrite the shared.yml every update, but you can override it in the docker-compose.yml file.

Important context is that Traefik migrated to a new configuration syntax. We will adopt the new syntax pretty soon: we figured out required changes already, and are now just waiting for a suitable release to add it to.

Scoping is part of these changes. I think that the most elegant approach would be to override the Brewblox services to use the new, scoped, configuration.

You can then use separate Traefik runtimes for Brewblox and your other containers, or have them share Traefik, and configure ports/routing with labels.

I’ll look up my implementation notes for Traefik config, and do a quick write up tomorrow or so.

Yeah, ideally I would just seperate them by DNS names brewblox.home, other-thing.home using traefik.frontend.rule=Host: rules.

How are my docker-compose and the docker-compose.shared combined? Been looking for the code that does that. So far the best I can do is https://github.com/BrewBlox/brewblox-ctl-lib/search?q=read_shared_compose&unscoped_q=read_shared_compose

I’ll look up my implementation notes for Traefik config, and do a quick write up tomorrow or so.
Awesome, thanks :smiley:

It’s part of docker-compose itself, but we do override the used file names by setting COMPOSE_FILE in .env

It looks like I can “disable” the packaged Traefik by the following in my docker-compose.yml

services:
  traefik:
      image: tianon/true # Super small image
      command: "true"
      entrypoint: "true"

And also move the ports that it binds to by updating .env

BREWBLOX_PORT_HTTP=40080
BREWBLOX_PORT_HTTPS=40443

This would allow me to standup my instance on the well known ports.

Correct on both counts.

Further considerations:

  • if you also override the ports in traefik to be empty, you don’t need to move them in .env.
  • http/2 + https is required for the Brewblox UI. There are too many simultaneous connections for http/1.1.
  • your traefik server needs to be connected to the brewblox bridge network (created by docker-compose) in order to forward to brewblox services.

Here you go. This was surprisingly fun to set up. We may adopt the concept of publishing localhost as brewblox.local to reduce the hassle with IP addresses.

Known constraint: the http -> https redirect will also be active for your services.
You can configure this by tweaking routing between web and websecure entrypoints, but better take one step at a time.

The implementation still places the traefik container in the Brewblox project.
This can be changed, but may cause some issues with brewblox-ctl update. I haven’t tried out all edge cases there.

Edit: Guide is now available at https://brewblox.netlify.app/dev/tutorials/subrouting.html

This is awesome. I’ll play tomorrow.

Cheers!

Is this also new?

 eventbus:
    image: brewblox/mosquitto:${BREWBLOX_RELEASE}
    labels:
      - traefik.http.services.eventbus.loadbalancer.server.port=15675

My current eventbus configuration is

  eventbus:
    image: brewblox/rabbitmq:${BREWBLOX_RELEASE}
    restart: unless-stopped
    labels:
      - "traefik.port=15675"
      - "traefik.protocol=ws"
      - "traefik.frontend.rule=PathPrefix: /eventbus"

So websockets access. Just want to make sure this isn’t leaking the new release :smiley:

Adam

You can leave out the image if you want: we configured RabbitMQ and Mosquitto to use the same port for MQTT over websockets.

Traefik v2 is smart enough to automatically proxy WS requests, so no need for explicit configuration there.

Had a successful night, my brewblox is part of an external traefik instance and accessible from the public internet (Google OAuth protected). I wanted brewblox to join an external one as I didn’t want updates to brewblox to impact my other running services.

I ended up just changing docker-compose.shared.yml to get it going, tomorrow night I will look to clean it up and see if I can drive it entirely from docker-compose.yml

Once clean, I will show you what I needed to do, to see if there are any better ways to represent this as so I can have the least amount of impact with brewblox updates.

Thanks for all the guides, the prefix-strip middleware would have had me going for a while with datastore if I didn’t know about it upfront.

1 Like

Today’s release includes the migrate to Traefik v2.
For you, this means you don’t need to override the datastore / history / eventbus services.

I updated the guide I posted earlier, and it’s now available at https://brewblox.netlify.app/dev/tutorials/subrouting.html

Just updated the stack with the new release which definitely made it easier and cleaner.

I made the following additions to my external traefik docker-compose.yml

networks:
...
  brewblox:
      external:
        name: brewblox
..
  traefik:
    networks:
      - brewblox

the following changes to my .env

DOMAIN_NAME=mydomain.com
COMPOSE_PROJECT_NAME=brewblox
BREWBLOX_PORT_HTTP=4080
BREWBLOX_PORT_HTTPS=40443

And this is my docker-compose.yml for brewblox

version: '3.7'

networks:
  default:
    driver: bridge
    external:
      name: brewblox
services:
  brewpi:
    command: --name=brewpi --mdns-port=${BREWBLOX_PORT_MDNS} --discovery=all --device-id=40002f001847383434353030
    image: brewblox/brewblox-devcon-spark:${BREWBLOX_RELEASE}
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.brewpi.rule=Host(`brewblox.$DOMAIN_NAME`) && PathPrefix(`/brewpi`)
    - traefik.http.routers.brewpi.tls=true
    - traefik.http.routers.eventbus.middlewares=chain-oauth@file
    privileged: true
    restart: unless-stopped
  datastore:
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.datastore.rule=Host(`brewblox.$DOMAIN_NAME`) && PathPrefix(`/datastore`)
    - traefik.http.routers.datastore.tls=true
    - traefik.http.routers.datastore.middlewares=prefix-strip,chain-oauth@file
  eventbus:
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.eventbus.rule=Host(`brewblox.$DOMAIN_NAME`) && PathPrefix(`/eventbus`)
    - traefik.http.routers.eventbus.tls=true
    - traefik.http.routers.eventbus.middlewares=prefix-strip,chain-oauth@file
    ports:
    - 5672:5672
  history:
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.history.rule=Host(`brewblox.$DOMAIN_NAME`) && PathPrefix(`/history`)
    - traefik.http.routers.history.tls=true
    - traefik.http.routers.history.middlewares=chain-oauth@file
  traefik:
    entrypoint: /true
    image: tianon/true
    restart: 'no'
  ui:
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.ui.rule=Host(`brewblox.$DOMAIN_NAME`) && (PathPrefix(`/ui`)
      || Path(`/`))
    - traefik.http.routers.ui.tls=true
    - traefik.http.routers.ui.middlewares=chain-oauth@file

The extra traefik labels (enable, tls, chain-oauth middlewares) are to work with my setup.

The docker-compose.share.yml is vanilla as of the the current version.

It all works well!

The only issue that is that when the update script runs and is verifying services is that it attempts to goto https://localhost:40443 which is not started, and even if it went through the current entrypoint it wouldn’t pass authentication. I might work on a simple nginx reverse proxy that doesn’t have all the authentication and bind it to localhost:40443.

That looks like a pretty clean setup!

I do spot eventbus having a 5672:5672 port mapping. That is the AMQP port from RabbitMQ. MQTT/Mosquitto doesn’t use it.
If you want to expose the MQTT broker, you can change it to 1883:1883, otherwise you can remove it.

You can repurpose the traefik service for this. If you configure it to only use a file provider, it will ignore the label-based configuration.

Edit: you can also skip the service migrate by using brewblox-ctl update --no-migrate. It’ll also skip any automatic migrations we’re performing, but that’s probably not a bad thing with your custom setup.

The eventbus is for a tilt service running on a RaspberryPi in my fridge, which I have to admit, I haven’t tested yet.

Do I still need the --mdns-port=${BREWBLOX_PORT_MDNS} on the brewpi command? Looking at the doco for the latest release, it seem like it can be removed.

I’ll look into reusing traefik, I thought it might just work out of the box, given it it is on different ports and the constraint

--providers.docker.constraints="LabelRegex(`com.docker.compose.project`, `${COMPOSE_PROJECT_NAME}`)"

I even tried add localhost into the rule

    - traefik.http.routers.brewpi.rule=(Host(`localhost`) || Host(`brewblox.$DOMAIN_NAME`)) && PathPrefix(`/brewpi`)
    - traefik.http.routers.datastore.rule=(Host(`localhost`) || Host(`brewblox.$DOMAIN_NAME`)) && PathPrefix(`/datastore`)
    - traefik.http.routers.eventbus.rule=(Host(`localhost`) || Host(`brewblox.$DOMAIN_NAME`)) && PathPrefix(`/eventbus`)
    - traefik.http.routers.history.rule=(Host(`localhost`) ||  Host(`brewblox.$DOMAIN_NAME`)) && PathPrefix(`/history`)
    - traefik.http.routers.ui.rule=(Host(`localhost`) || Host(`brewblox.$DOMAIN_NAME`)) && (PathPrefix(`/ui`)

localhost shows up in my main traefik instance, but given that it’s all auth’d it’s ok.

But 404s with curl --insecure https://localhost:40443/datatore

I’ll keep looking into it.

The Tilt by default uses MQTT over websockets (targets HOST:443/eventbus). I suspect that OAuth breaks that, and that the easiest solution is to open up 1883, and add --mqtt-protocol=mqtt to the tilt args.

You can remove the --mdns-port argument.

datatore -> datastore. Not sure whether that’s a typo in your request, or just in the forum post.

There can also be multiple Router / Middleware / Service combos for each docker-compose service.
Untested, but I suspect that this may work:

labels:
...
- traefik.http.routers.internal-datastore.rule=Host(`localhost`) && PathPrefix(`/datastore`)
- traefik.http.routers.internal-datastore.middlewares=prefix-strip

That works perfectly.

My final configuration

External Traefik docker-compose.yml

networks:
...
  brewblox:
      external:
        name: brewblox
..
  traefik:
    networks:
      - brewblox

brewblox .env

DOMAIN_NAME=mydomain.com
COMPOSE_PROJECT_NAME=brewblox
BREWBLOX_PORT_HTTP=4080
BREWBLOX_PORT_HTTPS=40443

brewblox docker-compose.yml

networks:
  default:
    driver: bridge
    external:
      name: brewblox
services:
  brewpi:
    command: --name=brewpi --discovery=all --device-id=40002f001847383434353030
    image: brewblox/brewblox-devcon-spark:${BREWBLOX_RELEASE}
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.brewpi.rule=Host(`brewblox.$DOMAIN_NAME`) && PathPrefix(`/brewpi`)
    - traefik.http.routers.brewpi.tls=true
    - traefik.http.routers.brewpi.middlewares=chain-oauth@file
    - traefik.http.routers.internal-brewpi.rule=Host(`localhost`) && PathPrefix(`/brewpi`)
    privileged: true
    restart: unless-stopped
  datastore:
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.datastore.rule=Host(`brewblox.$DOMAIN_NAME`) && PathPrefix(`/datastore`)
    - traefik.http.routers.datastore.tls=true
    - traefik.http.routers.datastore.middlewares=prefix-strip,chain-oauth@file
    - traefik.http.routers.internal-datastore.rule=Host(`localhost`) && PathPrefix(`/datastore`)
    - traefik.http.routers.internal-datastore.middlewares=prefix-strip
  eventbus:
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.eventbus.rule=Host(`brewblox.$DOMAIN_NAME`) && PathPrefix(`/eventbus`)
    - traefik.http.routers.eventbus.tls=true
    - traefik.http.routers.eventbus.middlewares=prefix-strip,chain-oauth@file
    - traefik.http.routers.internal-eventbus.rule=Host(`localhost`) && PathPrefix(`/eventbus`)
    - traefik.http.routers.internal-eventbus.middlewares=prefix-strip
    ports:
    - 1883:1883
  history:
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.history.rule=Host(`brewblox.$DOMAIN_NAME`) && PathPrefix(`/history`)
    - traefik.http.routers.history.tls=true
    - traefik.http.routers.history.middlewares=chain-oauth@file
    - traefik.http.routers.internal-history.rule=Host(`localhost`) && PathPrefix(`/history`)
  ui:
    labels:
    - traefik.enable=true
    - traefik.docker.network=brewblox
    - traefik.http.routers.ui.rule=Host(`brewblox.$DOMAIN_NAME`) && (PathPrefix(`/ui`)
      || Path(`/`))
    - traefik.http.routers.ui.tls=true
    - traefik.http.routers.ui.middlewares=chain-oauth@file
    - traefik.http.routers.internal-ui.rule=Host(`localhost`) && (PathPrefix(`/ui`)
      || Path(`/`))
version: '3.7'

no changes to docker-compose.shared.yml, so I assume unless there is major changes to traefik, updates should still work. As shown below:

$ brewblox-ctl update
Do you want to remove old docker images to free disk space? [Y/n]: y
Command is about to: Download and apply updates.
Do you want to continue? (yes, no, verbose, dry-run) [press ENTER for default value 'yes']
INFO       Updating brewblox-ctl...
INFO       Updating brewblox-ctl libs...
edge: Pulling from brewblox/brewblox-ctl-lib
Digest: sha256:3e2e5fcbdccb8aaa7cbb9250499744309ba3ec2f7337f2b010bfd7b3b99ad55c
Status: Image is up to date for brewblox/brewblox-ctl-lib:edge
docker.io/brewblox/brewblox-ctl-lib:edge
46975e3cc1bec9f874858493f1700ae421b6414f22f659840a5f5cae1b1d2e5e
ctl-lib
Command is about to: Download and apply updates.
Do you want to continue? (yes, no, verbose, dry-run) [press ENTER for default value 'yes']yes
INFO       Updating configuration files...
INFO       Checking Avahi config...
INFO       Stopping services...
Removing brewblox_influx_1    ... done
Removing brewblox_traefik_1   ... done
Removing brewblox_ui_1        ... done
Removing brewblox_brewpi_1    ... done
Removing brewblox_datastore_1 ... done
Removing brewblox_history_1   ... done
Removing brewblox_eventbus_1  ... done
Network brewblox is external, skipping
INFO       Migrating configuration files...
INFO       Checking .env variables...
INFO       Pulling docker images...
Pulling influx    ... done
Pulling ui        ... done
Pulling brewpi    ... done
Pulling eventbus  ... done
Pulling datastore ... done
Pulling history   ... done
Pulling traefik   ... done
INFO       Starting services...
Creating brewblox_history_1   ... done
Creating brewblox_influx_1    ... done
Creating brewblox_eventbus_1  ... done
Creating brewblox_datastore_1 ... done
Creating brewblox_traefik_1   ... done
Creating brewblox_brewpi_1    ... done
Creating brewblox_ui_1        ... done
INFO       Migrating service configuration...
Connecting https://localhost:40443/history/ping, attempt 1/60
Success!
Connecting https://localhost:40443/datastore, attempt 1/60
Connecting https://localhost:40443/datastore, attempt 2/60
Success!
INFO       Updating version number to 0.5.0...
INFO       Pruning unused images...
Total reclaimed space: 0B
INFO       Pruning unused volumes...
Deleted Volumes:
...

Total reclaimed space: 238.9kB

I must say, you have made some great design choices and put together a flexible architecture, I bet you never expected for anyone to try to do this.

Thanks again

1 Like