Development Update?

Software Dev
1

Hey @Elco. It’s been a while since we’ve had any update on how development is going for the complete architecture redesign. You used to give regular updates but I haven’t seen one for a while. Are you able to tell us a bit that’s been happening over the last few months?

Did you ever employ a new Developer to help you out?

I’m particularly interested in a better interface for managing brew day (ie easy switching between mash control and boil control). I realise this isn’t the highest priority though.

2

Hi Dan,

I don’t have time right now to write a big elaborate blog post, so here’s a quick update

  • I managed to hire a full time developer named Bob, who is working full time on the new stack and making great progress (github.com/brewblox). He’s done his internship at BrewPi for the past 5 months and started on a contract since July. He’s working mainly on the service layer: microservices, controller communication, data logging, rest api.
  • I have also hired a freelance front end developer (Gaya) to work on the Vue frontend for 2 days a week, for the past 5 months.
  • I’m tackling the firmware changes myself.

This has resulted in a lot of progress! The state of things now:

  • The service layer has been rewritten as microservices running in their own docker container, this is an entirely new stack. The services can be queried via REST and there’s an event bus for inter-service communication. We’re making it easy to integrate third party services and hardware too.
  • The UI is being developed as a Vue/Quasar (with Typescript!) dashboard which the user can fully customize by adding blocks. These are a live representation of the hardware and fully customizable charts. Currently Gaya is working on porting the interactive process view from this React prototype (https://brewpi-ui-demo.herokuapp.com/processview/herms-automated-valves). The basics for charts and simple hardware representation blocks are done.
  • The firmware is being rewritten to also be a flexible set of blocks (PIDs, sensors, actuators) that the user can create and link together in any way he pleases.

We’re currently integrating the layers and refactoring where needed for that.

Due to the work on the new stack, the old stack has not seen many updates. It’s not backwards compatible.

We’re trying to have a first alpha version out in September. It’s taking a long time, but then you’re getting everything at once. Multiple processes, customizable charts, mashing layout, etc.

5 Likes
3

Thanks @Elco. Appreciate the update. Glad to hear you’re making some good progress and looking forward to the alpha release in a few months. I realize it’s a bit task you’re aiming for.

4

Good news. With the third party integration are you also allowing for third party software to grab the data from the BrewPi? Specifically I was thinking about the new Brewfather software, being software that already has integration with Tilt/iSpindle hydrometers and BrewPiless, it seems to be really helpful to see all the data in one place for an ongoing brew and as it is cloud based and passive less prone to security issues.

5

Yes, all services have a rest interface. But writing a new container to integrate the Tilt into our system is probably less work than the other way around.

1 Like
6

I’m very much looking forward to this - I am finally getting my control box done this weekend (is the plan). I got my shelf with all the valves and pumps built. Get the hoses all measured and hooked up today. I just need to do my volume measurement markings in the kettles. :grinning: Then I will submit some pictures and stuff for the group when its all setup.

7

Is a September release a pretty sure thing or should I be looking at a second Spark for a second fermenter?

8

@Elco Just wanted to follow up with you and your time on progress / status of the “new” development update you guys have been working on? Looking forward to it - thanks
Ill add pictures of my setup after the control box is all done :smiley:

9

During my first week of holiday I did a bit of groundwork refactoring the new framework (laptop on the beach, I get bored quickly and prefer coding puzzles to sudoku :slight_smile: ). In the second week, I read 4 of Scott Meyers C++ books which gave me a lot of inspiration and I got excited about using the new C++14 features and much more of the STL.

So when I came back, I refactored the entire framework we use for managing objects on the controller (like sensors, pids, actuators, etc). I’m very happy with the new implementation and have 97% test coverage for it now!

If you are curious, look here (readme in repo is outdated),

This is the right moment to make some fundamental changes to how the control system works, so I’m not only switching to a new framework, but also change how objects reference each other and how calculations are done.

Last week I spent integrating a new fixed point library called CNL and I integrated a new filtering implementation I coded earlier which gives a much flatter passband, less delay and has step detection. This enables wider temperature ranges in the future, so we can add support for higher temperature sensors for example. I just refactored the PID class and my next goal is refactoring the actuators to simplify constraints (min/max, time limits, mutex, load balancing).

So done for firmware:

  • Framework for communcation, object management, settings storage.
  • Sending/receiving object data with protocol buffers
  • Refactoring of temperature sensors, setpoint, PID, mock objects

Todo:

  • Refactor actuators
  • New display code
  • More unit tests
  • More integration tests

See the commit log here. I’m a bit disappointed that a release in September is unlikely but the pace is great and I do expect to have a demo ready in the first half of October. We’ll start testing with a few selected testers and if that goes well roll it out for others too.

Meanwhile, Bob and Gaya have done a ton of work on the service layer and UI. The framework for dashboards and widgets is done. We have live updating charts, dashboard widgets and setup wizards. Most of the work to be done is implementing the actual widgets for the different block types (Pid, Setpoint, etc).
You can follow that development here.

I’m very sorry that we’ll overrun my estimate of September, but not by much. I’m 100% certain the new feature set will blow you away though.

6 Likes
10

Thanks, this is a really great update! While I have been following the github commits over the summer, it’s nice to get a big picture update with editorial comments every once in a while.

Definitely looking forward to the alpha in October. While I won’t lie about it being hard to wait, I’m sure down the line it will be worth it having all the code well designed and as generic/mod-able as possible.

1 Like
11

Any info available on how other projects can hook into the new software? For example ispindle being able to push info to brewblox?

12

We have boilerplate for writing your own docker containers that can hook into the platform.

There is a global RabbitMQ eventbus for generated data. If the iSpindle container pushes data onto that bus, it will automatically be logged by the history service and will be available to graphs in the UI.

For reading/writing data and settings to the container directly, it will provide its own REST API and we also have boilerplate for that.

So to integrate the iSpindle, you will have to write I think max 100 lines of code to handle communicating with it.
And perhaps you’ll want to add a new widget to use on the UI dashboards.

For a demo, Bob integrated an Xbox controller and displayed button state in Graphs. That didn’t require much code:

So to summarize:
We have a basic set of containers that provide the framework for data logging and managing the containers. It will be very easy to add a new container and it will run completely independent of the others but it can be integrated using the provided communication channels. There are no code dependencies.

13

Its now mid-late October…

14

Hi @matt_m,

I’m well aware of that. We’re working our asses off to deliver as soon as possible.
As you can see here:
https://github.com/BrewPi/firmware/commits/feature/brewblox
And here:

Latest developments:

  • Almost all control objects are ported to the new framework. Only some OneWire actuators are left
  • The firmware got some new exciting features:
    • All digital actuators will have a user configurable set of constraints. Minimum off time, minimum on time and/or mutually exclusive with other actuators.
    • All analog actuators (this includes PWM) will have a list of constraints, this includes min, max and load balancing
    • The constraints above can be used for example to run your HLT and your BK heating element at the same time when your fuses only support one at a time. The high cycle of one PWM will overlap with the low cycle of the other (mutex) and the load balancer will ensure that the sum of both doesn’t exceed 100%.
  • Temperature profiles have moved to run on the Spark instead of the server
  • PID and PWM classes were almost completely rewritten for better performance and reliability
  • Digital filtering has completely changed for better filtering with less delay

On the UI side:

  • UI got user configurable units (Celsius/Fahrenheit/etc)
  • Charts can be user configured for which data to show and over which period
  • Chart data is now streamed with SSE to be displayed live
  • New menus to create the constraints
  • New configuration wizards
  • Reordering of dashboards is possible

We have reached end-to-end and are ready to test on hardware. We will start doing this in the coming week with a select group of test customers. Things are moving forward fast, but at the same time it just a lot of new code. We’re working very hard. Check the commit logs and confirm for yourself.

We want to release a public beta as soon as the minimum set of requirements to run a brewery is met, but not sooner.

5 Likes
15

Well Done @Elco and team, this looks great, cannot wait for the public beta.

2 Likes
16

Awesome! Cant wait for the beta too

17

@Elco and team, just wanted to follow up with you guys and see what type of time frame you think the Beta might be looking at currently? I got mine updated to 5.10 you had mentioned October, and then additional coding you are incorporating. Just curious on what to set expectations around. Thanks again for all your time and hard work I know we all really appreciate it.

18

Hi @Elco,

Kudos to you and your team for all your hard work building a great product, running a company, re-writing the software from scratch, and remaining active answering questions in community forums. I too am excitedly awaiting the new software release and would like to share with you my sentiment that it [software] should be released only when it’s actually ready. Anything short of that will result in more time spent fixing issues and responding to complaints than if that work [testing] is done up front before release!

To anyone else that is finding it difficult to wait for the new software, I suggest spending your time reading: The Inmates Are Running The Asylum, which is full of real-world examples of failed companies due directly to rushed or poorly designed software.

The target release date is just a goal that was based on best estimates using information available at the time. If the original goal wasn’t met, a new goal can (and should) be made, but it’s not a guaranteed delivery date. Similarly, if a house isn’t ready by the deadline, one can’t simply move in early, a new goal must be made, based on new best estimates. Writing software is a highly complex venture that requires immense amounts of time and energy. A good product will last many years, and even then, is usually re-written after some time due to the nature of patching over time. Hence the very fact that @Elco is re-writing BrewPi from scratch right now and we are all waiting for it. It is the nature of software.

Cheers,
–Andy

1 Like
19

Thanks for all of your input and support :slight_smile:

Here is a short status update:

Real world hardware test

We started testing the full stack, including hardware and were happy with the results. Naturally, some bugs came to light but we had a successful test with the new platform (which will be called BrewBlox) controlling actual water temperatures.

In BrewBlox, there will be no static or pre-configured control loops or sensors. The user will have complete freedom to create blocks on the controller. A block can be for example a sensor, setpoint, PWM driver or PID.

In our test scenario we wanted to control the temperature of two water cookers, with the following caveat: we do not have enough power to turn both on at the same time. This is very much like most brewers having to choose between powering the HLT or the boil kettle.

So we set up the following blocks. This probably sounds really technical, so we doing our best to create easy to use wizards for setup.

To drive each water cooker, we set up these blocks:

  • A OneWire temperature sensor
  • A Setpoint
  • A digital output pin
  • A PWM driver to toggle that output pin
  • A PID to control the PWM driver based on the sensor and setpoint

In BrewBlox, a list of constraints can be set on each of the actuators. For digital actuators, this is the minimum ON time, minimum OFF time or mutual exclusivity. For analog actuators (actuators with a range), you can set a minimum, maximum or balanced.
So in this case we have the 2 digital outputs sharing the same mutex so they alternate.
The two PWM actuators shared the same balancer so they get an equal share when the sum of their values goes over 100%.

With this setup, we could keep the 2 water cookers at 90 degrees without problems, with live data streaming to the UI widgets. We also have live updating charts, not only of the temperatures but also all internal PID parameters and every single variable in the system. This will be very helpful to analyze PID response and tune the settings.

UI progress

Lots of UX. We simplified creating dashboard widgets, improved block creation wizards, auto discover sensors, moving/copy/rearrange/delete blocks and widgets and many small usability and architecture improvements.

Deployment on the Raspberry Pi.

We also ran the entire stack on a Raspberry Pi 3. Some small bugs arose that require attention, but overall no serious issues. The Raspberry Pi 2 did not run well, so we’ll have to figure out why because we don’t want to force users to buy a new Pi. The Raspberry Pi 1 is probably a bit underpowered, but we will see how far we get.

Next major steps

  • Hardware display: the Spark still shows a blank screen, we have not written new code to drive the TFT display yet.
  • Support for OneWire expansion boards: I still need to create the blocks for our valve control boards and SSR extension board.
  • Simplifying deployment. It is already pretty simple with a docker-compose file to create all the different microservices involved, but it is still complex. I think we should have a preconfigured raspberry pi image that requires only some small changes on the SD card before it is transferred to the pi.
  • Simplify initial setup by having a lot of sensible defaults that work for most users.
  • Default settings for common use cases (like PID settings for mashing, fridges, glycol)
  • Styling of many UI widgets to be easy to recognize and use. The widgets look very much alike. A single glance should be enough to get the info you need. Lesser used settings and values should hidden on the dashboard and only shown in the modal popups.
  • Lots of help buttons and text to explain what all the features are and how they should be used.

When can I get my hands on it?

As Andy said above, releasing too early is bad. And I think it is too early because you probably don’t want to spend a lot of time dealing with small frustrations that together eat up a lot of time.
Theoretically, the system can already be used to control a fermentation or mashing setup. But getting it set up is not smooth yet so we would like to take a bit more time before releasing a public beta.

We’re currently working with some test users that have been part of the development process and know their way around Docker, Linux, compiling firmware and debugging in general. Their feedback is very helpful and enough to improve our software and process. The software development process is completely open source, so we can’t stop anyone from trying the new stuff, but it is probably best to wait until we have made it a somewhat smoother experience.

We don’t want to hold back the software until it is perfect or complete, but a smooth deployment an absolute requirement. Once we have that, the first brave test users can try the new stack. My best estimate for this early beta release is 1-2 weeks.

If you are a UX/web designer who’d love to help us with icons, colors and widget design in Vue, get in touch for early access. Our firmware can also be simulated as a virtual controller, so doing UI work doesn’t require any specific hardware.

What will come later

Process management including the flow views, brewing steps, and valve control will not be part of the first release. The system is usable without it and it should not hold back the release. Our first release will do everything the current BrewPi does and more though.

I’ll try to make a demo video on our next hardware test!

7 Likes
20

When you say valve control won’t be in the first release, do you just mean for mash setups or do you mean at all? If I read right, glycol should be possible with the first release?

Will multi-process be possible in the first release?