Actuators turn on during firmware flash

When I flash the firmware, during the flash the actuators turn on. Is this the intended behavior? Ideally I would prefer this not to happen. Is there a way to change the actuators to “fail-closed”?

It’s certainly not intended behavior.

Right now we’re implementing a rework of actuator pins / DS2413 / DS2408. We’ll have a look whether the problem still occurs.

It looks like pins are briefly set to high during reboot, before our firmware starts.

We’ll notify the upstream vender, see whether we can work something out. We’re also working on having the service automatically update the firmware without the controller having to reboot.

Thanks for looking into it.

I’ve always noticed that the pins briefly all turn on during a reboot (usually less than one second). What is more concerning though is how during my last several firmware flashes, during the flashing process the pins go high for probably a good 10 seconds (it seems to happen every time so if you need logs/dumps and aren’t able to reproduce it on your end I would be happy to help).

It’s not the end of the world for my setup, but I could imagine this causing big problems on other setups (ie. dry firing of heating elements). If particle isn’t responsive, it might be worth adding guidance in the meanwhile that users might want to consider unplugging actuators during the flash if unintended actuation would cause problems.

Thanks for the feedback. I really recommend to install dry fire protection (a float switch) in your kettles anyway. It makes brew day a lot easier, because you can just leave them powered when filling or emptying the kettles. Less worrying and more relaxed brewing.

We saw pins being online slightly longer than the normal flicker, but nowhere near 10s. Could you please post your exported blocks (spark service page, actions)? I’ll have a look at reproducing it on monday.

If it becomes more common (and not fixed upstream), we’ll likely have to find some creative solutions: unplugging actuators is both inconvenient and easily forgotten.

@Elco Luckily this is just driving fans, heating pads and a glycol pump so it’s just an annoyance and not a threat for me.

@Bob_Steers I just timed it and it’s actually 24 seconds. Here you are:

Blocks: brewblox-blocks-hotpinsduringflash.json (7.3 KB)
Log: https://termbin.com/5wd1
Video: https://youtu.be/q7XRlcTaHRA (you can hear the fans/pumps on from 0:10 to 0:34)

Ironically, this also perfectly reproduced the separate “incompatible firmware” after flash issue.

Screenshot%20from%202019-06-22%2012-00-31

It also flashes a timeout error at the bottom of the screen. Too fast to screenshot, but here it is from the api error log.

{
“fetch”: [
{
“info”: “/spark-one/discover_objects”,
“body”: “500: TimeoutError()”,
“time”: “Sat Jun 22 2019 12:00:29 GMT-0400 (Eastern Daylight Time)”,
“status”: 500
}
],
“db”: []
}

Cycling the power on the Sparkv3 resolves the incompatible firmware issue, but then it hangs on “Service not syncronized”–which I can resolve with brewblox-ctl down and brewblox-ctl up.

Thanks for the info! The video is particularly helpful: it appears the pins become active when DFU mode (Device Firmware Upgrade, yellow blinking LED) is triggered.

We’ve already added some changes to improve the firmware check reliability and feedback. A temporary workaround is to disable the version check entirely. To do this, add --skip-version-check to the arguments for the Spark service in brewblox/docker-compose.yml

Example:

spark-one:
    image: brewblox/brewblox-devcon-spark:rpi-${BREWBLOX_RELEASE:-stable}
    privileged: true
    depends_on:
      - eventbus
      - datastore
    restart: unless-stopped
    labels:
      - "traefik.port=5000"
      - "traefik.frontend.rule=PathPrefix: /spark-one"
    command: >
      --name=spark-one
      --mdns-port=${BREWBLOX_PORT_MDNS:-5000}
      --skip-version-check

To complete this round of bug whack-a-mole: the hangup on service synchronization seems to be due to the datastore freezing. Running docker-compose restart datastore in the brewblox directory likely works as well as running down + up, and is significantly faster.

Needless to say, we’ll do some digging for the root cause, and a more permanent fix.

I just noticed something very interesting while flashing the firmware with the latest fix.

I was viewing my setup from a different angle and noticed power LEDs on actuated power outlets blinking during the DFU mode period. I had previously assumed that the pins were constantly high throughout that period due to the constant fan noise.

I can’t be sure, but it almost seemed that the actuated LEDs were blinking at the same tempo as the yellow status LED on the spark. This could have just been a coincidence though.

Tested it today, and the bug does reproduce here. On first sight it appears to be hardware, but I really can’t recall seeing it before this week.

I’ll flash some old firmware to verify. May as well check whether there’s some sort of intermittent signal going on.

Just a short note. I’ve noticed a very short burst of relay expansion board outputs pulsing on during reboots. Short enough that it could just be a single clock cycle but its enough to make my compressor go clunk. I never had this under the old BrewPi codebase but have always had it on BrewBlox (at least since very early releases). I haven’t updated in a few weeks now so unsure if they stay on longer now. I’ll probably be in a position to update & test later in the week, though.

It really seems to be hardware related: we flashed the old brewpi software, and it still reproduced.

The flickering is caused by the pin being in an undetermined state: it’s not really high and not really low.

As a solution we’ve increased the prio on something we wanted to do anyway: let the Spark firmware update itself over any connection (wifi or usb).

Implementation is to ship firmware binaries with the service, and ask to update if a mismatch is detected.