Web Stack Help Wanted

Software Dev
1

Calling web developers!

As some of you may know, we plan to rewrite the web UI from scratch, focusing on making it a pure browser client implementation talking to a brewpi service via HTTP and websockets.

So, what’s the best javascript stack to use?

Here are some of our requirements:

  • testable
  • separation of concerns (content, behavior, presentation, UI state, application state, etc…)
  • visual and non-visual functionalities as components (keep things modular)
  • application logic in isomorphic JS. keeps the code clean (platform neutral)
  • (etc…more to come as I think of it.)

We’ve looked at many frameworks, ComponentJS, React (starter kit), Angular, Bootstrap etc… For charting, we’re considering Grafana.

The first step is to make a MVP using the chosen client stack, by implementing a service endpoint that mocks a Brewpi Spark being powered on/off and reflecting the availability/status of that in the web UI.

If you have any recommendations about frameworks , libraries etc… we’d love to hear them. And of course if you want to get involved coding just drop a note here.

Our main concern here is with the front end and application logic - the back-end services we’ll be coding in Python.

1 Like
2

This is something I do as part of my day to day role, where I design the back and frontend systems. For the backend, node.js can work quite well, especially if you wish to standardise on JS for all the application logic. It is rich and widely used, though I haven’t used it on a raspberry Pi, so a rapid prototype would be needed to do some speed measurements for the backend.

There are a number of testing frameworks out there. I can’t comment in detail here, but a quick google will point you in the right direction. I tend to do the majority of this on the backend, which isn’t in JS.

One issue with JS is the object model and concepts such as inheritance. It’s possible, but clunky from a language point of view. Here Typescript gives some richer support, while keeping backward compatibility, but I warn that the typings files you need to use to work with standard JS libraries can be a bit of a pain. Bear in mind the new JS will be much more like how Typescript is working right now, so something to have a think about at least.

For the system I’ve designed, I actually use .Net for the backend, as it’s incredibly rich and there are a myriad of strong REST server implementations that you can get through the open source community, which do a lot of the work for you. The core is going open source with the next release, as well as windows 10 running natively on the new raspberry pi’s. However, there are license implications of this approach and .net is not there yet from a platform agnostic point of view in my opinion.

The frontend opens up a lot of options. We used durandal.js, but angular.js is a great option with a lot of develop support. Both have a strong MVC approach and the binding system is very powerful. If I was doing it again, I’d probably use angular, simply for the larger community and knowledge base.

Do use something like require.js for injection here, which will help keep your JS modular and testable.

For look and feel, use bootstrap unless you have a good reason to do otherwise. The out of box css, especcially if you don’t do a lot of UI development, solves a lot of early issues and also gives you a strong adaptive UI which can run on many devices without needing to worry about it.

For the charts, I strongly suggest you look at highcharts.com. We use them for all our charting requirements and it’s very powerful. In particular, they have a highstock library which is great with time series data. The only issue here is there is a license cost for for-profit projects. Does the software qualify as open source and not for profit? If you can’t use these, just pick something that has native binding support for the frontend framework you choose.

Lastly, I’d suggest you setup an automatic linting and compile for the node.js server and client files if you go that direction. It’ll catch issues early in the development process and save you integration headaches as you go.

Anything else you’d like to know, just drop me a message and I can give my two cents.

3

Thanks for your input!

For charting, we were considering grafana, which doesn’t have licensing issues for FOSS projects.

The problem with having so many js frameworks is that it’s sometimes hard to know exactly what surface each framework covers. E.g. I was going to ask about bootstrap vs react, and durandal vs react, but a quick search finds that these can in fact be used together. So I dug more into the react starter kit to find out what all the pieces are:

  • EditorConfig - consistent formatting across IDEs
  • Babel - ES6 source code transpiler, so we can use new JS features on current JS engines
  • webpack - module bundler (with dependencies)
  • ESLint - linting for JS and JSX
  • CSSLint and CSSComb - linting/formatting for CSS
  • JSCS - code style linter and enforcer
  • Flow - static type checker
  • BrowserSync - cross device testing

Seems to be a strong focus on consistent code formatting, which is not a bad thing.

I’ve never liked the pseudo-OO in JS so nice to see Classes are added to ES6, even if it is just syntactic sugar, and the ES6 module system.

For synchronization between the application model and views, I’ve just discovered Flux - a design pattern for applications. MVC has multi-directional couplings, flux has a single data flow.) I’ve seen MVC become problematic in large apps with cascading changes causing an explosion in updates, so appreciate the simplicity of the Flux pattern.

For modules, I just listened to this presentation. Requirejs is an option, also webpack, browserify, jpms, As the webcast shows, modules/loaders/bundlers is a complex topic, I’m sure we’ll continue to evolve our thoughts since it’s a problem that can be delayed until we need to deploy remotely.)

I think Bootstrap is a definite, and it’s well-regarded.

There’s also meteorjs - a javascript application framework (full stack.)

So much choice!

4

grafana looks interesting. A short prototype with your data in a test page, doing both a small set and also a full 3 week log would be a good test to see how well it would work.

For JS, there are a LOT of frameworks and libraries. Everytime I look, another 3 seem to have popped up. It’s useful to isolate the problems you need to solve first and then see what fits. If you start with the frameworks, you’ll be pulled everywhere and slowly lose your mind :smile:

You’ve listed a lot of different technologies, but they are solving different cross sections of the problem but not the giving a whole solution, as I can see it at least. As I see it, you need the following:

  • Build server. This compiles/lints all your JS and gives some consistency to the process, as well as early syntax error reporting.
    Example: Can’t remember off the top of my head

  • REST server: Running on the spark, which allows the system to pull information from the device, as well as send signalling commands. You want something light here, like a self hosted rest.
    Example: Node.Js could do this well, but there’s others out there.

  • Backend web server. This will serve all assets to the frontend. It can communicate with the backend rest server(s) running on the sparks and also server as a REST server for the frontend. This would run on the pi/web server. An option here is to use a framework for a self hosting REST server instead of using and needing to manage a more complete web server such as apache. You can also have this server run web sockets for the frontend if required. This is where something like node.js becomes very powerful.
    Example: Node.Js

  • Frontend framework. This gives you MVC or any other structure you wish to use. It is served to the user from the backend web server and basically loads a simple HTML page into memory, then switching over to the framework which takes care of loading all other assets as needed. The core idea here is to have a modular structure from the start, using injection to pull the modules in as needed, which means you need to use an injection system right from the start, such as require.JS. It’s not that bad though, as the framework will force you to do the right thing.
    Example: Angular.JS

  • Frontend Styling
    Bootstrap basically :smile:

*JS libraries for features
You’ve already identified one, with the charting solution and more will popup as you work. Toastr is great for popup messages for example. These just get added to the injection system and you pull them in as required. I’d suggest you serve the libraries from your backend, as you don’t want to assume the user has a net connection to use your system.

Of course, then along comes React, which has a virtual DOM and figures out a lot of this stuff on the fly and, as far as I can see, generated the pages from the server. From what I can see, it means you code the frontend side on the server codebase and then the generated pages are shipped to the client. It would need a prototype to see how well it works with using the additional JS libraries. Personally I’m used to have control of the DOM when needed, so for complex problems I can’t comment which is the better solution.

Personally, I think the best approach would be to get all your data from the sparks into your backend and be able to package it into JSON and fetch it with a simple REST call. You can test that from any browser. Once you’re there, any of the frameworks you’ve mentioned will work and you can make it as simple or as hard as you wish. However, at that point, you can pick your main contenders, identify a prototype spec and do a set of sprints to compare how easily you can get a really basic, unstyled application up and running with each approach.

5

Yep, that’s the plan is to expose services via REST API. The services will be coded in python, so that’s a different technology stack. Hence, here, let’s focus on the browser client. The build server, integration testing, system testing, web server we have covered. (We’re purposely avoiding node for the services since the programming paradigm of callbacks for everything to implement co-operative threading is pretty evil imo, and probably one of the worst cases of premature optimization ever since it permeates an entire platform.)

Totes, there’s so many JS libraries, it makes my head spin, and that’s the reason I started this thread. I think a plan is to keep a page listing all the libraries that are currently in use or being considered, and any that were actively considered but rejected in favor of something else. That’s kind of what I started above, but will push it to a github page soon in the brewpi web repo.

Good idea with the MVP. That’s what I’d like help with getting started. Since the back-ends aren’t even coded yet, any back-end functionality will be mocked for the MVP client.

I’m also keen to get an end-to-end test implemented - the MVP for this is to show a list of connected devices, which updates as devices are connected/disconnected.

6

Yeah we don’t use node either as it didn’t suit the business needs and results in a lot of overhead that’s unnecessary for just communicating with a simple end device. Using Python is a good call there and makes things a lot easier.

It sounds like all the backend is covered in your design except your web server for the frontend. I take it you’re just going to go with simply serving up the JS and having it all run on the client from there?

A simple mock python service running in a separate process is all you’d need for testing, with a simple hello world result to show it as connected.

For the MVP, looks like it’s just down to picking a webserver to run on the Pi and get it to serve up your framework. Here, take a look at the simple start mockups from the ones you are considering and you’ll be up and running in no time. I did it before with durandal and angular.js and was up and running in a few hours with a simple test page for a starter app. So much comes out of the box with these. React should also have the same.

Really, with the frameworks, an awful lot comes out of the box at the start. Then the learning starts in earnest :wink:

Pick which MVP frameworks you want to try. If I get a free few moments (rare these days) and I know the web server you’re using, I’ll try to throw something up on my pi at home and see how hard it is to get the starter application up and running.

7

I’ve done front end development on some pretty large scale sites my stack has typically looked something like this:

  1. Angular + Restangular(creates easy to work with mappings to
    endpoints) + Browserify for dependencies + Chart.js for graphs(not sure if this is my favourite, would be interested to try alternatives)
  2. SASS + Bourbon(define your own grids/components behind the scenes rather than placing presentation in markup)
  3. Karma + Jasmine for unit tests
  4. Gulp tasks to watch/build
    -Compile/Minify CSS
    -Lint everything, JSCS with airbnb style guide or similar
    -Run JS unit tests
    -Concat, minify, obfuscate JS, transpile if using typescript or ES6
    -Usemin(replaces references in templates)
    -Usually lots more depending on the project
  5. Jenkins or similar as build server to run above grunt tasks and fail build if tests/lint fail

I think Angular or React would be the best choice just for community size, available libraries and approachability. Durandal would be fairly similar to Angular but the founder of Durandal joined the Angular 2.0 team then left Angular and started a new framework so I question it’s future a little bit.

You may want to run through the examples at TodoMVC to see how you might approach the same problem with the different frameworks and see which ones resonate the most. Framework fatigue is real javascript and I don’t think you’d go wrong in choosing any that have been mentioned in this thread. I’d like to tackle some issues and submit PRs when this gets going.

Sorry for the long first post, I’m just getting into BrewPi and this is the first post I felt compelled to reply to.

8

Yes, was thinking for the rpi of just serving the JS as static resources. For cloud deployment later we can look at other options. (I’m sure some nodejs will creep in at some point, perhaps as part of the backend web server, but we will be using it with purpose rather than being a default choice.)

9

Welcome to the brewpi discourse!
What a first post! Thanks very much for your inputs. No worries for the long post, much useful info there. :smile:

1 Like
10

I was thinking either Jenkins (which we currently use) or travis-ci, to save us the headache of maintaining a jenkins instance. But travis is also a bit of a pain since you can’t have a local copy for testing builds before pushing to release.

I\d not heard of SASS/Bourbon so that’s a nice find.

Any thoughts on ComponentJS?

11

That looks like a great stack to use for a web project. The framework testing sight you mentioned is new to me but a great little find. Next time I’m starting a web app from scratch, I’ll def. Be trying that out.
I agree with Durandal. It’s fully featured but has been abandoned once and most likely will be again. The community trust is not there anymore.

Personally I prefer Teamcity to Jenkins for ease of configuration but I haven’t used it for linting or with Jasmine. I’m sure it’s possible, but would be very interesting to investigate.

12

Yeah I was thinking Jenkins for local builds though the gulp scripts could easily be run without a build server on the local system, not sure what implications that would have on your backend though.

I’ve not looked into ComponentJS before but I like it’s principles, I have had some issues with NPM availability in the past so it may be a worthwhile option. Browserify and NPM seem to work for me 99% of the time though so I haven’t had a reason to look elsewhere yet.

13

Edit: going to read more on componentjs before commenting

14

I imagine it’s possible with Teamcity as well, we have a .NET backend at work and use AppVeyor which basically just ties into msbuild and we have prebuild task to run our gulp tasks.

I do prefer most options to Jenkins for ease of use but Jenkins has that nice plus of being free for all projects, commercial or not. :smiley:

1 Like
15

To my mind, getting the job done and the right license (free for commercial use) takes precedence over minor features or inconveniences.

The download page says TeamCity is free for the professional edition with 3 build agents. I’ve not looked into the product in any depth so don’t know how much or limiting that is. But since we already have a Jenkins instance running I feel we need a good reason to invest time learning and switching to a new build tool.

1 Like
16

The 3 agents is fine for most projects, but if you already have a build system you are comfortable using, no point in changing.

1 Like
17

If you already use jenkins for the backend it will do fine to run the front end tasks.

Does the API have a specification defined at all yet? If it was something that could be mocked already I’d be interested in playing with it. This is a fun little tool if you want to test your REST schema. You can set up your endpoints in a few minutes and it just writes/reads from a json flat file. Would allow for front end work to begin right away if you have a spec already defined.

18

There is no formal spec yet. We’ve hashed out some ideas but not really had a good vehicle to write them down/express them. The json-server looks very useful, thanks for sharing! Nice easy way to map routes to resources. We’ve also have glanced at apiary which combines mocking, testing, docs but need more time to assess that. Also swagger, There’s a blog post comparing the different tools.

For testing the client, I’m a believer that even the REST API can be mocked on the client so testing can happen entirely on the client. (Consider it a unit/integration test of components on the client.)

19

Yes, absolutely, if you use Angular it comes with a utility to intercept all requests through its $http service and mock the responses as well as perform expects and whens on them in your unit tests to make sure everything is behaving as it should depending on the requests sent out. Here is a simple plnkr(not written by me) showing a mocked backend using the above mentioned service.

1 Like
20

Thanks for this link, I think I’m going to use this at the day job now. Hadn’t heard of it before. Also how do I get involved in helping with the web app? Should I go through the issue log of the existing web app or is this going to be a complete rebuild?

1 Like