Ferment graph broke on update to 2021/02/25 release

[hexamer]

My ferment graph just keep showing “No data (yet) for selected period.”

Not exactly sure what version I came from, but my logs shows a brewblox-ctl version of 0.17.0 - so maybe an April 2020 version?

I may have taken a less-than-conventional approach to updating and I’m on a Synology DS713+

1. Update Python from 3.5.6 to 3.7.7
2. Update pip to 21.0.1
3. Update brewblox-ctl because I wanted to take a snapshot (probably unnecessary)
4. Ended up making a backup
5. Run brewblox-ctl update

At that point I was surprised to see that my whole configuration was reset

6. Ran brewblox-ctl backup load to restore the backed-up config

Most of my config was back, but the ferment graph is not working and my tilt service seems to have disappeared too.

Any thoughts on what’s going on here?

[Bob_Steers]

Could you please run brewblox-ctl log? I’ll have a look at what happened to your tilt service.

If you check the graph settings, can you see all selectable fields in the tree view?

[hexamer]

Log is attached: brewblox.log (189.0 KB)

Yes, I can see the selectable fields in the tree view.

[Bob_Steers]

It looks like your previous version was still using the CouchDB configuration database. Some time last year, we switched to Redis.
brewblox-ctl backup includes a data migration, but that doesn’t seem to have happened. The variable with the configuration version also hasn’t been changed. At a guess, there was an error during the update.

To get things in order again, run the following commands in your brewblox dir:

brewblox-ctl down
sudo mv redis/ redis-bak/
brewblox-ctl update

The update should print some messages about it having migrated {num} entries from {database}, and afterwards, you should have your old config back.

I can’t say whether this will also fix your graph issues. Please let me know if it doesn’t.

Your tilt service is indeed not included in configuration. You can follow the instructions here to add it again.

[hexamer]

Thank you for helping out here, Bob. At a top level it looks like that basically took away the Temp Control Assistant and Tilt widget I had added after update. The Fermentation Graph is still not updating:

I’m attaching a terminal log from the commands you asked me to issue as well as another brewblox-ctl log. Sorry … no pastebin on Synology

Log of commands from Bob Steers 3-5-2021.txt (7.0 KB)

brewblox_3-5-2021.log (187.7 KB)

I tried a brewblox-ctl restart after this and still no change.

Also:

[Bob_Steers]

The update restored the old configuration, from before your earlier update.

It looks like Synology has its own alternative to the nc command we use to upload files: https://blog.codybunch.com/2020/10/08/Fast-Bulk-File-Copy-to-Synology-Tar-over-NetCat/

Have you restarted Docker (or your NAS) after running the update?

[hexamer]

I tried restarting the NAS and the problem persists. I also tried deleting the graph widget and adding it back, still no luck.

The only other potential clue is when I try different browsers I get different messages. On Edge, I get this:

But it doesn’t seem like a cache issue since nothing changes when I clear my cache on Firefox.

The issue with the tilt may not be an issue at all. I forgot that service is running on a different machine.

[Bob_Steers]

I saw the problem myself a few days ago on my Pi, but it went away when I did the equivalent of kicking it a few times, so not sure what caused it. I suspect there’s a bug in the error handling for the history data stream.

If you explicitly visit the http:// version (not https), does it still happen?

[hexamer]

I can’t do that externally because of the way I setup by nginx reverse proxy (see below), but sure enough when I access internally the graph starts working.

Setting up BrewBlox for password protected external access

@j616s VPN is a great secure solution, but personally I find it cumbersome to establish a VPN connection just to quickly check on my fermentation from my phone (on the train, at work, …). @hexamer I still think the nginx reverse proxy + basic auth route should be possible in your case with some tweaking of the nginx config. I don’t know much about the Synology, but just did a quick experiment (similar to your setup) on my raspberry pi: Installed nginx on the Pi that’s running Brewblox Replace…

Here’s what my site config looks like

server {
        listen 443;
        server_name brewblox.hexamer.haus;
        location / {
                auth_basic "Login to BrewBlox";
                auth_basic_user_file /etc/nginx/.htpasswd;
                proxy_set_header Authorization "";
                proxy_pass https://localhost:8443;
                proxy_ssl_verify off;
                proxy_ssl_session_reuse on;
                proxy_buffering off;
                proxy_http_version 1.1;
                proxy_set_header Host $http_host;
                proxy_set_header Upgrade $http_upgrade;
            	proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host localhost:443;
        }
}

[hexamer]

OK, reading that post a little closer now, it looks like my problem is described there too. I’ll see if I can use the later info in that post to fix the problem.

[Bob_Steers]

Reverse proxy settings are something I’d have to test. If I remember correctly, websockets (used by the history stream) can misbehave in specific circumstances when proxied.

It’s also possible that the basic auth headers are not set for the websocket request, or upgrade, and the proxy drops it.

Keep in mind that during the discussion in the topic you linked, graph streams still used SSE. We switched to websockets in the 2020/10/29 release.

[hexamer]

Looks like you were right, Bob. I simply added:

proxy_set_header Connection “Upgrade”;

and it seems back to normal.