Cannot start service traefik

Bizmo · January 1, 2021, 8:20pm

Hi,

Long time user of Arduino and then spark brewpi, finally had some spare time over xmas and decided to update to BrewBlox.

Gone through the setup via the wiki, fresh install on a new SD on a Raspi 3A.

Everything goes fine, ‘brewblox-ctl update’ shows I have the latest version but when I run the ‘up’ command I get the following error.

Any ideas?

ERROR: for brewblox_traefik_1 Cannot start service traefik: driver failed programming external connectivity on endpoint brewblox_traefik_1 (d07d79b52d8ec499688b46b40f2e52e603e678ab016cd340f1d6c34f44aacdf5): Bind for 0.0.0.0:443 failed: port is already allocated

Bob_Steers · January 1, 2021, 8:39pm

It’s complaining that something else is already using the HTTPS port (443).

Are you running something else with a web UI?
What is the output from sudo netstat -tulpn?

If you’re not running anything else, run:

brewblox-ctl down
sudo reboot

After reboot, run up again, and it should work.

Bizmo · January 1, 2021, 8:43pm

nothing else with a web, was a fresh SD.

This is the output of net stat. seems to be Docker running on 443

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      501/sshd            
tcp6       0      0 :::22                   :::*                    LISTEN      501/sshd            
tcp6       0      0 :::443                  :::*                    LISTEN      645/docker-proxy    
tcp6       0      0 :::80                   :::*                    LISTEN      659/docker-proxy    
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           343/avahi-daemon: r 
udp        0      0 0.0.0.0:68              0.0.0.0:*                           491/dhcpcd          
udp        0      0 0.0.0.0:53648           0.0.0.0:*                           343/avahi-daemon: r 
udp        0      0 0.0.0.0:59831           0.0.0.0:*                           343/avahi-daemon: r 
udp6       0      0 :::5353                 :::*                                343/avahi-daemon: r 
udp6       0      0 :::42519                :::*                                343/avahi-daemon: r 
udp6       0      0 :::57744                :::*                                343/avahi-daemon: r

Tried the down command then reboot but come back up to the same error.

Bob_Steers · January 1, 2021, 9:00pm

Docker occasionally fails to clean up. You can run brewblox-ctl kill to force close everything. Then wait a minute, or reboot, and netstat should report 443 to be unused.

Bizmo · January 1, 2021, 9:50pm

really odd, tried that and same error.

Have ejected the SD and followed the install again to the letter and again got the below

pi@raspberrypi:~ $ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      503/sshd            
tcp6       0      0 :::22                   :::*                    LISTEN      503/sshd            
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           329/avahi-daemon: r 
udp        0      0 0.0.0.0:57877           0.0.0.0:*                           329/avahi-daemon: r 
udp        0      0 0.0.0.0:68              0.0.0.0:*                           493/dhcpcd          
udp        0      0 0.0.0.0:52683           0.0.0.0:*                           329/avahi-daemon: r 
udp6       0      0 :::5353                 :::*                                329/avahi-daemon: r 
udp6       0      0 :::47888                :::*                                329/avahi-daemon: r 
udp6       0      0 :::42915                :::*                                329/avahi-daemon: r 
pi@raspberrypi:~ $ cd ~/brewblox/
pi@raspberrypi:~/brewblox $ brewblox-ctl -v up
SHELL      docker-compose up -d --remove-orphans
Starting brewblox_influx_1 ... 
Starting brewblox_history_1 ... 
Starting brewblox_ui_1      ... 
Starting brewblox_redis_1   ... 
Starting brewblox_traefik_1 ... 
Starting brewblox_spark-one_1 ... 
Starting brewblox_eventbus_1  ... 
Killed

Bob_Steers · January 1, 2021, 10:26pm

Seems like the bind problem is solved. You may need to retry the up - Killed typically is a memory error. I noticed it as well since last Docker update. We’ll look into doing some optimization to avoid startup problems.

Bizmo · January 1, 2021, 10:37pm

Strange, it seems to be the same each time I launch it. I’ve stopped the swap file to see if that makes a difference but doesn’t seem to change.

I can make a log unfortunately as the pi locks up after the killed message

Bob_Steers · January 1, 2021, 11:13pm

You could try starting in batches:

docker-compose up -d influx redis eventbus
docker-compose up -d history spark-one
docker-compose up -d

It’s rather annoying, and I’ll be sure to look into both issues. Something as basic as “start system” should consistently work.

Bizmo · January 1, 2021, 11:31pm

Just tried that and it works fine.

How odd!

Bob_Steers · January 1, 2021, 11:50pm

Good to hear! The cause seems to be that peak memory use when starting all services is more than what the Pi has available. Starting up a few services at a time spreads the load.

Bizmo · February 19, 2021, 5:34pm

Just wanted to follow this up,

Just brought a brand new RasPi 4B 4gb with a sandisk extreme pro as i was annoyed with this error, fresh install and get the same problem.

Bob_Steers · February 19, 2021, 5:57pm

Are you using the Raspberry Pi OS, or a 64-bit version?
Is this still something that is resolved by starting services in batches?

Could you please run this command, and post the URL?

cat /var/log/syslog | nc termbin.com 9999

In a recent release we added a mitigation for this problem, so I’m indeed rather surprised to see it again on a 4GB system.

Bizmo · February 19, 2021, 6:20pm

Yes still works in batches just not with the full up command

Have messaged you link to termbin

Bob_Steers · February 19, 2021, 6:28pm

This looks like it is caused by an IPv6-related bug in docker.
Previously we bypassed this by disabling IPv6 altogether, but I’ve been tinkering with that lately, and a better fix seems to be to make some changes to the docker settings.

brewblox-ctl down
nano /etc/docker/daemon.json

In the editor, paste the following:

{
  "ipv6": true,
  "fixed-cidr-v6": "2001:db8:1::/64"
}

Press ctrl-X to save and exit.

Then run:

systemctl reload docker
brewblox-ctl up

Bizmo · February 19, 2021, 7:48pm

brilliant! that works perfectly, fermentation fridge mk2 is now all up and running! going to do a test run with a vessel of water to check I have it all tuned right.

Could have saved €60 on the raspi 4, but good to have a more up to date faster one anyway