You are very welcome to share your recipe 
I got started on a refactor of the mutex constraints which will make the behavior easier to understand and more reliable.
In the new design, the actuator will take ownership of the mutex and it gets its own configurable extra hold time to block others from activating after it turns off.
It will take about a week to implement at all layers and test it in our dev setup.
@Elco
Effort really appreciated!
Sounds like a good measure to encapsulate the mutex during actuator hold.
For now the LargeFermentor is stable at 4C and crossing fingers for it to stay like that for another couple of days.
the recipe…
Had a FG test yesterday with a small tasting sample 
, real fruity flavor and a hazy mouthfeel.
2 Likes
Thanks for sharing the recipe. The link was malformed, this one should work:
Looks like a tasty recipe. How is the bitterness? The hop stand at 85C will add some bitterness. A lower temp, 70 or 75 would reduce that.
PR for mutex rework can be found here:
Will be released with the necessary UI changes after the weekend.
Thanks for sharing the Pull Request. I am not c++ proficient enough for any code review, however I believe the encapsulation and separate timing calculations are good safeguards.
The bitterness of the (NE)IPA is palatable but not dominating. Reducing the hop-stand temperature to 70C would probably benefit some of the more “juice favoring” recipients of the brew.
1 Like
Bug fix is now live. Would be cool if you can test it with your less reliable cable to see if the mutex now also works correctly under error conditions.
Have now updated UI and spark and reinstalled the “patchy” cable. No brew in fridge, “dry-running” beer constant mode. I have left the actuator constraints unchanged. Ill folow closely for the next days.
Again thank you for the effort.
Atle
2 Likes
@Elco, followup:
Had a look at the fridge this evening and its now at -21C!
Here is the log and the csv of the graph around the time the latch up happened.
(logs indicate the DS2413 is intermittently disconnecting and connectiong multiple times during the early hours of 4th of March)
The cool pin is static (ON) and the (virtual) beer is solid frozen!
This is definately not a desired situation and probably the watchdog should catch this as a loss of sane operation?
What do you think?
graph-LF Graph-spark-one-downsample_1m(2).csv (89.8 KB)
Atle
I assume Cool Pwm Setting / cool pin desired state correctly dropped to 0 when the actuator locked up?
Going by this graph/logs the mutex deadlock seems to have been solved, but DS2413 disconnects can cause a pin lockup.
The graph indicate LF_Cool pin is ON (1) and dose not chage state even if the beer temp falls below set point. The firmware should in this case at list have set the LF_Cool pin OFF (0) whitch the graph indicate is not the case.
The graph shows cool pin State. Pins have State and Desired State fields.
The same goes for Pwm: the graph shows achieved %, and not desired %.
Am I correct in that the “desired” fields for both pin and pwm dropped to 0 when it locked up?
The desired state of LF_Cool Actuator dropps to 0 and the desired PWM value follows down to zero.
Do I understand i correctly that you read-back the status of the DS2413 and graph this value? If so the watchdog should intevene at some point (mismatch between desired and value) in time to prevent the “freeze-in” condition ?
That’s one possible approach. We may indeed be more aggressive about disabling actuators if they misbehave, but I can’t promise solutions while still looking for the root cause.
Desired state / state is used for more purposes. If you put a Minimum On constraint on an actuator, and then try to toggle it on/off manually, it will have a state mismatch while the constraint prevents it from turning off.
Thanks for testing this for us! I have not been able to recreate the scenario myself. Can I log in on your system remotely sometime soon to dig around a bit?
The ds2413 has state itself which is kept while it has power. I suspect something to be wrong involving this.
I wondered about the state memory, since as I pulled the cable to the DS2413 intermittently the cool ssr did NOT loose its state (even) if the spark had been set to stop control. After a longer (several seconds) disconnect of the one wire cable the ssr did turn off.
Sure you can access the system. The RPI is a headless CLI only setup, what’s the preferred way (protocol) of enabling access?
Atle
I just fixed a bug in the interaction between pwm and constrained actuators. Can you test whether it also resolves this issue?
Back again with an update!
Fridge has been running for 14 days at a steady 6 °C in beer mode.
these are the logs
https://termbin.com/rfr6
The DS 2413 is disconnecting and reconnection quite frequently (patchy cable).
However, the system has been running stable the 14 days.
Seem like you nailed latch-up! Yeay!
Great work!
And thanks you for the effort!
graph-LF Graph-spark-one-downsample_1m(4).csv (112.7 KB)
1 Like
Awesome, thanks for confirming that the fix works!
@Elco
Followup after last firwareupdate 06032020 [d1b5ace1].
After an “outage” of the rpi late March 30, I upgraded the system til latests SW and firmware.
It seem like the controll algorithms has changed drastically in the latest firmware!
Looking at the graph before and after the upgrade, the change is evident!
The beer temp does not have the same degree of stability as befor the upgrade, nore does the fridge temp. (all PDI params are set equal before and after update).
What is causing the dramatic change?
Atle
[graph-LF Graph-spark-one-downsample_10m.csv|attachment]
Greåh export:
(upload://tXbfLkgk5RO5mjC27DzCcThQwpW.csv) (30.3 KB)
Logs:
https://termbin.com/c3ki