Services issue after Ubuntu upgrade

rdharles · January 10, 2022, 5:48pm

Hi,
When I upgraded Ubuntu, I broke a bunch of services apparently? When i run brewblox-ctl up, I get this? Any ideas?
Thanks!
RD

rdh@AirUbuntu:~/brewblox$ brewblox-ctl up
Starting brewblox_traefik_1 …
Starting brewblox_redis_1 … error
Starting brewblox_spark-one_1 …
Starting brewblox_victoria_1 …
Starting brewblox_ui_1 …
Starting brewblox_history_1 …
Starting brewblox_eventbus_1 …
Starting brewblox_ui_1 … error
Starting brewblox_spark-one_1 … error
Starting brewblox_traefik_1 … error

ERROR: for brewblox_ui_1 Cannot start service ui: OCI runtime create failed: container with id exists: c41ccfb650ac4e6f7454e0f51f5b2c9e280461fd949fe0afd0e92a71b96ba334: unknown

ERROR: for brewblox_spark-one_1 Cannot start service spark-one: OCI runtime create failed: containeStarting brewblox_victoria_1 … error

ERROR: for brewblox_traefik_1 Cannot start service traefik: OCI runtime create failed: container with id exists: a4dd2d4bde172c0618f036d15dd09f3b01cc51b8db7ecb46e7cab868109b251e: unknown

ERROR: for brewblox_victoria_1 Cannot start service victoria: OCI runtime create failed: container Starting brewblox_eventbus_1 … error

Starting brewblox_history_1 … error
with id exists: 24318bd596e780e638f8e47cdf3ebe78866cd546d8d87860f67d7d4e90177411: unknown

ERROR: for brewblox_history_1 Cannot start service history: OCI runtime create failed: container with id exists: 1336fee33a9e0604b8ebd8fbd5dd478350842284fb1e531e3916b16deb590b8c: unknown

ERROR: for redis Cannot start service redis: OCI runtime create failed: container with id exists: 935e6b8b96a10cd117d994ffb6d5e826a5c36e37ef26912147c4ad6455b2fe42: unknown

ERROR: for ui Cannot start service ui: OCI runtime create failed: container with id exists: c41ccfb650ac4e6f7454e0f51f5b2c9e280461fd949fe0afd0e92a71b96ba334: unknown

ERROR: for spark-one Cannot start service spark-one: OCI runtime create failed: container with id exists: a0640c868702150553d6917150a6952fe118074f07751f4666e0536575919dfc: unknown

ERROR: for traefik Cannot start service traefik: OCI runtime create failed: container with id exists: a4dd2d4bde172c0618f036d15dd09f3b01cc51b8db7ecb46e7cab868109b251e: unknown

ERROR: for victoria Cannot start service victoria: OCI runtime create failed: container with id exists: 9a1c6f148d6fc85ca5e99d20cc4d2a14de45986efc85b5dd88d6012037e10c24: unknown

ERROR: for eventbus Cannot start service eventbus: OCI runtime create failed: container with id exists: 24318bd596e780e638f8e47cdf3ebe78866cd546d8d87860f67d7d4e90177411: unknown

ERROR: for history Cannot start service history: OCI runtime create failed: container with id exists: 1336fee33a9e0604b8ebd8fbd5dd478350842284fb1e531e3916b16deb590b8c: unknown
ERROR: Encountered errors while bringing up the project.
Command 'docker-compose up -d ’ returned non-zero exit status 1.
rdh@AirUbuntu:~/brewblox$

Bob_Steers · January 10, 2022, 7:18pm

I suspect the restart left some leftovers in place. Does it work if you first run brewblox-ctl kill?

rdharles · January 10, 2022, 7:37pm

Hi,
Looks like it killed some stuff but still errors…

rdh@AirUbuntu:~/brewblox$ brewblox-ctl kill
a4dd2d4bde17
9a1c6f148d6f
a0640c868702
1336fee33a9e
24318bd596e7
935e6b8b96a1
c41ccfb650ac
rdh@AirUbuntu:~/brewblox$ brewblox-ctl up
Creating brewblox_redis_1 …
Creating brewblox_history_1 …
Creating brewblox_victoria_1 …
Creating brewblox_redis_1 … error
Creating brewblox_eventbus_1 …
Creating brewblox_traefik_1 …
Creating brewblox_spark-one_1 …

ERROR: for brewblox_ui_1 Cannot start service ui: cgroups: cgroup mountpoint does not exist: unknown

ERROR: for brewblox_redis_1 Cannot start service redis: cgroups: cgroup mountpoint does not exist: Creating brewblox_spark-one_1 … error
Creating brewblox_history_1 … error
ERROR: for brewblox_spark-one_1 Cannot start service spark-one: cgroups: cgroup mountpoint does not exist: unknown

Creating brewblox_victoria_1 … error
xist: unknown

ERROR: for brewblox_history_1 Cannot start service history: cgroups: cgroup mountpoint does not exist: unknown

Creating brewblox_traefik_1 … error
xist: unknown

ERROR: for brewblox_traefik_1 Cannot start service traefik: cgroups: cgroup mountpoint does not exist: unknown

ERROR: for ui Cannot start service ui: cgroups: cgroup mountpoint does not exist: unknown

ERROR: for redis Cannot start service redis: cgroups: cgroup mountpoint does not exist: unknown

ERROR: for spark-one Cannot start service spark-one: cgroups: cgroup mountpoint does not exist: unknown

ERROR: for eventbus Cannot start service eventbus: cgroups: cgroup mountpoint does not exist: unknown

ERROR: for history Cannot start service history: cgroups: cgroup mountpoint does not exist: unknown

ERROR: for victoria Cannot start service victoria: cgroups: cgroup mountpoint does not exist: unknown

ERROR: for traefik Cannot start service traefik: cgroups: cgroup mountpoint does not exist: unknown
ERROR: Encountered errors while bringing up the project.
Command 'docker-compose up -d ’ returned non-zero exit status 1.
rdh@AirUbuntu:~/brewblox$

Bob_Steers · January 10, 2022, 7:46pm

Some googling reveals more people are having this problem.
What is your new Ubuntu release, and what is your docker version (run docker version)?

rdharles · January 10, 2022, 8:01pm

Docker: 19.03.12
Ubuntu: 21.10

Bob_Steers · January 10, 2022, 9:11pm

Docker release notes indicate this should be fixed in Docker 20.04 and up. From what source did you install Docker?

rdharles · January 10, 2022, 10:12pm

Honestly, I don’t even know it’s been so long…does it get installed as part of the Brewblox install?
If so, this is what I did…

Bob_Steers · January 10, 2022, 10:33pm

Yes, we run the Docker install script. If you got it from there, a apt update + apt upgrade would do the trick.

rdharles · January 10, 2022, 11:04pm

Sorry but I’m a complete novice, do you have a link to the exact commands I need to run.
I found a lot of articles online on what to type and they’re all different?!?
Thank you!

Bob_Steers · January 11, 2022, 12:10am

That’d be

sudo apt update
sudo apt upgrade -y

Then recheck docker version. If it’s still at 19, we’ll have to look at the apt settings.

rdharles · January 11, 2022, 12:13am

Looks like it didn’t update/upgrade…

rdh@AirUbuntu:~/brewblox$ sudo apt update
[sudo] password for rdh:
Hit:1 Index of /ubuntu impish InRelease
Get:2 Index of /ubuntu impish-updates InRelease [110 kB]
Get:3 Index of /ubuntu impish-security InRelease [110 kB]
Get:4 Index of /ubuntu impish-backports InRelease [101 kB]
Get:5 Index of /ubuntu impish-security/main amd64 c-n-f Metadata [2,044 B]
Fetched 323 kB in 1s (352 kB/s)
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
All packages are up to date.
rdh@AirUbuntu:~/brewblox$ sudo apt upgrade -y
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
Calculating upgrade… Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
rdh@AirUbuntu:~/brewblox$ docker version
Client: Docker Engine - Community
Version: 19.03.12
API version: 1.40
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:45:44 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:44:15 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

Bob_Steers · January 11, 2022, 1:25pm

After upgrading, it disabled the source for Docker packages.
To re-enable it, edit the config file by running

sudo nano /etc/apt/sources.list.d/docker.list

I suspect it now looks something like this:

# deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable

Remove the # at the start, and change focal to impish. End result:

deb [arch=amd64] https://download.docker.com/linux/ubuntu impish stable

Press Ctrl-X and then enter to exit and save the file.
Then re-run

sudo apt update
sudo apt upgrade -y

rdharles · January 11, 2022, 2:28pm

Looks like everything went perfectly (until I tried to bring brewblox up again…?!)

rdh@AirUbuntu:~/brewblox$ sudo vi /etc/apt/sources.list.d/docker.list
rdh@AirUbuntu:~/brewblox$
rdh@AirUbuntu:~/brewblox$ sudo apt update
Hit:1 Index of /ubuntu impish InRelease
Get:2 Index of /ubuntu impish-updates InRelease [110 kB]
Get:3 Index of linux/ubuntu/ impish InRelease [48.9 kB]
Get:4 Index of /ubuntu impish-security InRelease [110 kB]
Get:5 Index of /ubuntu impish-backports InRelease [101 kB]
Get:6 Index of linux/ubuntu/ impish/stable amd64 Packages [4,136 B]
Get:7 Index of /ubuntu impish-updates/main amd64 DEP-11 Metadata [18.9 kB]
Get:8 Index of /ubuntu impish-updates/universe amd64 DEP-11 Metadata [4,484 B]
Get:9 Index of /ubuntu impish-backports/universe amd64 DEP-11 Metadata [9,088 B]
Get:10 Index of /ubuntu impish-backports/universe DEP-11 64x64 Icons [1,789 B]
Get:11 Index of /ubuntu impish-security/main amd64 Packages [141 kB]
Get:12 Index of /ubuntu impish-security/main i386 Packages [48.8 kB]
Get:13 Index of /ubuntu impish-security/main Translation-en [36.4 kB]
Get:14 Index of /ubuntu impish-security/main amd64 DEP-11 Metadata [6,432 B]
Get:15 Index of /ubuntu impish-security/universe amd64 Packages [49.4 kB]
Get:16 Index of /ubuntu impish-security/universe Translation-en [20.2 kB]
Get:17 Index of /ubuntu impish-security/universe amd64 DEP-11 Metadata [2,308 B]
Fetched 714 kB in 1s (571 kB/s)
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
4 packages can be upgraded. Run ‘apt list --upgradable’ to see them.
rdh@AirUbuntu:~/brewblox$ sudo apt upgrade -y
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
Calculating upgrade… Done
The following NEW packages will be installed:
docker-ce-rootless-extras docker-scan-plugin libslirp0 slirp4netns
The following packages will be upgraded:
containerd.io docker-ce docker-ce-cli libexiv2-27
4 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
1 standard security update
Need to get 98.0 MB of archives.
After this operation, 28.2 MB of additional disk space will be used.
Get:1 Index of /ubuntu impish-updates/main amd64 libslirp0 amd64 4.4.0-1ubuntu0.21.10.1 [58.2 kB]
Get:2 Index of linux/ubuntu/ impish/stable amd64 containerd.io amd64 1.4.12-1 [23.7 MB]
Get:3 Index of /ubuntu impish/universe amd64 slirp4netns amd64 1.0.1-2 [28.2 kB]
Get:4 Index of /ubuntu impish-security/main amd64 libexiv2-27 amd64 0.27.3-3ubuntu4.1 [821 kB]
Get:5 Index of linux/ubuntu/ impish/stable amd64 docker-ce-cli amd64 5:20.10.12~3-0~ubuntu-impish [40.7 MB]
Get:6 Index of linux/ubuntu/ impish/stable amd64 docker-ce amd64 5:20.10.12~3-0~ubuntu-impish [21.2 MB]
Get:7 Index of linux/ubuntu/ impish/stable amd64 docker-ce-rootless-extras amd64 5:20.10.12~3-0~ubuntu-impish [7,920 kB]
Get:8 Index of linux/ubuntu/ impish/stable amd64 docker-scan-plugin amd64 0.12.0~ubuntu-impish [3,517 kB]
Fetched 98.0 MB in 13s (7,780 kB/s)
(Reading database … 186145 files and directories currently installed.)
Preparing to unpack …/0-containerd.io_1.4.12-1_amd64.deb …
Unpacking containerd.io (1.4.12-1) over (1.2.13-2) …
Preparing to unpack …/1-docker-ce-cli_5%3a20.10.12~3-0~ubuntu-impish_amd64.deb …
Unpacking docker-ce-cli (5:20.10.12~3-0~ubuntu-impish) over (5:19.03.12~3-0~ubuntu-focal) …
Preparing to unpack …/2-docker-ce_5%3a20.10.12~3-0~ubuntu-impish_amd64.deb …
Unpacking docker-ce (5:20.10.12~3-0~ubuntu-impish) over (5:19.03.12~3-0~ubuntu-focal) …
Selecting previously unselected package docker-ce-rootless-extras.
Preparing to unpack …/3-docker-ce-rootless-extras_5%3a20.10.12~3-0~ubuntu-impish_amd64.deb …
Unpacking docker-ce-rootless-extras (5:20.10.12~3-0~ubuntu-impish) …
Selecting previously unselected package docker-scan-plugin.
Preparing to unpack …/4-docker-scan-plugin_0.12.0~ubuntu-impish_amd64.deb …
Unpacking docker-scan-plugin (0.12.0~ubuntu-impish) …
Preparing to unpack …/5-libexiv2-27_0.27.3-3ubuntu4.1_amd64.deb …
Unpacking libexiv2-27:amd64 (0.27.3-3ubuntu4.1) over (0.27.3-3ubuntu4) …
Selecting previously unselected package libslirp0:amd64.
Preparing to unpack …/6-libslirp0_4.4.0-1ubuntu0.21.10.1_amd64.deb …
Unpacking libslirp0:amd64 (4.4.0-1ubuntu0.21.10.1) …
Selecting previously unselected package slirp4netns.
Preparing to unpack …/7-slirp4netns_1.0.1-2_amd64.deb …
Unpacking slirp4netns (1.0.1-2) …
Setting up docker-scan-plugin (0.12.0~ubuntu-impish) …
Setting up libexiv2-27:amd64 (0.27.3-3ubuntu4.1) …
Setting up containerd.io (1.4.12-1) …
Setting up docker-ce-cli (5:20.10.12~3-0~ubuntu-impish) …
Setting up libslirp0:amd64 (4.4.0-1ubuntu0.21.10.1) …
Setting up docker-ce-rootless-extras (5:20.10.12~3-0~ubuntu-impish) …
Setting up slirp4netns (1.0.1-2) …
Setting up docker-ce (5:20.10.12~3-0~ubuntu-impish) …
Processing triggers for man-db (2.9.4-2) …
Processing triggers for libc-bin (2.34-0ubuntu3) …
rdh@AirUbuntu:~/brewblox$
rdh@AirUbuntu:~/brewblox$ docker version
Client: Docker Engine - Community
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:45:33 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:41 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0
rdh@AirUbuntu:~/brewblox$
rdh@AirUbuntu:~/brewblox$ brewblox-ctl up
Starting brewblox_victoria_1 …
Starting brewblox_history_1 … error
Starting brewblox_spark-one_1 …
Starting brewblox_victoria_1 … error
Starting brewblox_ui_1 …
Starting brewblox_traefik_1 …
Starting brewblox_redis_1 …

ERROR: for brewblox_history_1 Cannot start service history: OCI runtime create failed: container wiStarting brewblox_eventbus_1 … error

Starting brewblox_spark-one_1 … error
with id exists: 6020c0e4f4b8109c1c4bfd8f9ba224c724660839fe4313d8a53c5359d52164ef: unknown

ERROR: for brewblox_eventbus_1 Cannot start service eventbus: OCI runtime create failed: container with id exists: 8530dbcb9d1d8cf7d3cfe72a42ae9d8ba4bd1dc9b3307cebf32575a698ebdf86: unknown
Starting brewblox_ui_1 … error
ERROR: for brewblox_spark-one_1 Cannot start service spark-one: OCI runtime create failed: container with id exists: 51b18b978a83ef89f0a2c6ca971857a24b698fc0d1f9287c8a543546a0ae1af8: unknown

ERROR: for brewblox_ui_1 Cannot start service ui: OCI runtime create failed: container with id exisStarting brewblox_redis_1 … error

Starting brewblox_traefik_1 … error
d exists: 1f6cd0d066cde76db7a119fdcea34bb0a1bcfadca7a200e26de01ddddc174529: unknown

ERROR: for brewblox_traefik_1 Cannot start service traefik: OCI runtime create failed: container with id exists: 4c380090ed5b8ce3e76871f2dbffc180c244abac2c77e0031cf6118fcd06878f: unknown

ERROR: for history Cannot start service history: OCI runtime create failed: container with id exists: a179a7fd5bf86a55c26271fca5c5b2acf5337980409eeb7eae03e41fb7de01fa: unknown

ERROR: for victoria Cannot start service victoria: OCI runtime create failed: container with id exists: 6020c0e4f4b8109c1c4bfd8f9ba224c724660839fe4313d8a53c5359d52164ef: unknown

ERROR: for eventbus Cannot start service eventbus: OCI runtime create failed: container with id exists: 8530dbcb9d1d8cf7d3cfe72a42ae9d8ba4bd1dc9b3307cebf32575a698ebdf86: unknown

ERROR: for spark-one Cannot start service spark-one: OCI runtime create failed: container with id exists: 51b18b978a83ef89f0a2c6ca971857a24b698fc0d1f9287c8a543546a0ae1af8: unknown

ERROR: for ui Cannot start service ui: OCI runtime create failed: container with id exists: af133295d0295f3be62c6d7dd4c6dd4af6b7f2076c9732baabc0c323658c7dfa: unknown

ERROR: for redis Cannot start service redis: OCI runtime create failed: container with id exists: 1f6cd0d066cde76db7a119fdcea34bb0a1bcfadca7a200e26de01ddddc174529: unknown

ERROR: for traefik Cannot start service traefik: OCI runtime create failed: container with id exists: 4c380090ed5b8ce3e76871f2dbffc180c244abac2c77e0031cf6118fcd06878f: unknown
ERROR: Encountered errors while bringing up the project.
Command 'docker-compose up -d ’ returned non-zero exit status 1.
rdh@AirUbuntu:~/brewblox$

Bob_Steers · January 11, 2022, 3:15pm

To fix that, run brewblox-ctl kill again.

rdharles · January 11, 2022, 3:48pm

OK, that seemed to work…

rdh@AirUbuntu:~/brewblox$ brewblox-ctl kill
4c380090ed5b
a179a7fd5bf8
51b18b978a83
8530dbcb9d1d
af133295d029
1f6cd0d066cd
6020c0e4f4b8
rdh@AirUbuntu:~/brewblox$ brewblox-ctl up
Creating brewblox_redis_1 … done
Creating brewblox_spark-one_1 … done
Creating brewblox_eventbus_1 … done
Creating brewblox_history_1 … done
Creating brewblox_traefik_1 … done
Creating brewblox_ui_1 … done
Creating brewblox_victoria_1 … done
rdh@AirUbuntu:~/brewblox$

…but i still get “Unable to connect” in the browser when I try to bring up the dashboard?

Bob_Steers · January 11, 2022, 3:53pm

Could you please run brewblox-ctl log?

rdharles · January 11, 2022, 4:01pm

Here you go…

https://termbin.com/sf08

Bob_Steers · January 11, 2022, 7:32pm

I don’t immediately see anything wrong, except some complaints from the proxy. I suggest restarting your machine to reset various system services.

rdharles · January 11, 2022, 8:24pm

Unfortunately, no change after reboot.

Just to confirm, this error doesn’t mean anything, I saw it a couple times in the log:

ERROR brewblox_service.service Unknown arguments detected: [’–mdns-port=5000’]

Thanks!

Bob_Steers · January 11, 2022, 10:51pm

Correct. The argument is obsolete, but also harmless.

I’ll put together some detailed troubleshooting commands for network issues in Docker, but I do have to ask: is this machine home to important software or configuration outside Brewblox?

If not, I would do a clean system upgrade as described in Upgrading your system | Brewblox

I’ll happily assist you debugging system issues, but for a dedicated machine the “nuke and reinstall” approach is likely to be quicker and simpler. The snapshot contains all Brewblox settings and history data.