Workflow-Based Visual Programming with Node-RED

Hacking with IBM’s Node-RED

Whenever we want to build a system, we usually find that everything is a workflow. If we want to build a web application, we have to make a lot of boiler-plating components to capture user input, validate it. Then it has to be sent over the wire, the service needs to have components that validate authentication, validate and sanitize data, talk to different services to get additional information, write to the database, and so on.

Even if it’s not such a complex application, maybe you just want to write a script that knows how to connect to a database, knows http, and knows REST.

Every time I had to write such script or system, I had to initialize a package manager, npm init for example, write a bunch of dependencies, create a web-server, connect to a database, perhaps create a nice CLI to start the script. This is somehow time consuming and I haven’t even started to write the custom login and the workflow logic that I want.

Then, even if I have the boilerplate, I need to pay attention to be consistent with the interface of the data that is flowing trough the workflow.

After all that is done, then if other people will want to use or alter the same script, they have to understand the code style, the logic that is in place, and this all seems that there must be a better way to write workflow-based applications.

Good news. There is a great tool that takes care of all this! It’s called Node-RED and was created by IBM under their IOT (Internet of Things) set of tools. Node-RED is a visual programming tool that is relying a lot on I/O and it does that by having modules / components that talk to each other via messages.

The entire platform is written in JavaScript/Node.js, and it’s simply great. It has I/O components, storage, social components, custom functions, GPIO and microcontrollers modules, and you can even do your own custom modules.

Once you create a I/O workflow by tying all these components together, you can save the workflow in JSON format and share only that. It is the most amazing tool ever created for programmers, scripters and IOT fans.

For the rest of this article I will go over how to do migration MongoDB script using Node-RED.

The flow from the above picture does the following:

Interval Operation is an Input node that is timer-based, and every 0.2 seconds sends a signal trough the wire.

The Inject Query node is preparing the mongoDb find query to read one record at a time. For this, you can use the default msg.limit property to set the db.collection.limit() but there is no skip property to do full pagination.

For that I had to hack a little bit the Node-RED mongo component to chain skip into the equation. This most likely will resolve into a pull request in node red.

collection.find(selector,msg.projection).sort(msg.sort).limit(msg.limit).skip(msg.skip);

The next node, Find in MongoDB is a very simple mongoDb connection node that uses a find query. The trick and there is not so much documentation about this, actually I had to read the source code to figure this one out is that find takes the input of the previous node and validates if it’s a valid mongo query and if so executes it and messages to the next node.

The Edit Record is a function node and is where you would want to do extra work on the node before saving it again in a new or existing collection.

After doing some extra manipulation on the record data, we use the dispatch records, switch node, which is used with two outputs. The first output is to send to the MongoDB insert node, and only forwards messages that match a regex. For my purpose I’ve regex-ed that the email should be yahoo, gmail or hotmail.

So again the logic is simple, if an valid email pipe to write to MongoDB node otherwise pipe to Output no 2 which is a default debug that writes the message.payload. Below is the mongodb out node.

Some of the things and the advantages of using Node-RED to write your workflows are:

• 0% boilerplate
• strict contract between data that is flowing, they talk only trough messages and you decorate messages if you want structure on the data
• visual programming tool that has a very low learning curve
• improves productivity and its simply a joy to use it
• data exchange format for the workflows is JSON

So without further ado, try Node-Red. And Happy New Year!