Xideco

A Network Enabled Software Backplane

Artwork from http://www.terrancesimien.com/creole-for-kidz/

When looking for a new project to work on, I began to reflect on Xi, a project I created back in the Fall of 2014. 

Xi allows a user to control a set of of individual  microcontrollers as if they were a single "megacontroller" using either Scratch or Snap! as a graphical user interface.

Although Xi works very well, it is not without its limitations. Here are some of them:

• The code does not allow boards to directly communicate with each other. All communication must go through either Scratch or Snap!. 
• Xi is closely coupled to Scratch and Snap! - replacing the control interface is not simple.
• The code was written in JavaScript and has no easy way of incorporating features written in other computer languages.
• It is tightly coupled to the Johnny Five library, making it difficult to extend beyond the features of Johnny Five.

For my new effort I put together a set  of "wants":

• Python 3.5 should be the system's language of choice, however if a user wishes to develop using some other language, they should have the freedom to do so.
• The system should be portable with the ability to be executed on:
• Windows.
• Mac.
• Linux.
• The system should be distributable:
• Any component should be able to run on any computer within a local area network, using any operating system or combination of operating systems.
• Configuration should be simple with minimal user interaction, but flexible enough to allow modification if it becomes necessary to do so.
• The system should be extensible and flexible for physical as well as other types of computing:
• It should support current boards types.
• Have the capability to support future board types.
• Have the capability to run general applications (like sending Tweets).
• Be able to interface to, and make use of, non Python software.
• Have the ability to support a set of services that could span multiple boards.
• The ability to describe current and future capabilities through a portable, extensible and language neutral messaging protocol.

From Xi, to Xideco

How Do You Pronounce Xideco?

Xideco is pronounced like Zydeco. Zydeco is a form of  Louisiana Cajun music.
What can I say? Whenever I hear Zydeco music, it makes me smile - take a listen.

What does the Xi-d-eco acronym stand for?

Xi = Cross Platform Interconnect

Cross platform meaning support for multiple single board computer types,on multiple operating systems, using multiple programming languages.

d = Distributable

Having the ability to cooperatively execute Xideco software components residing on a single computer as well as on multiple computers. 

eco

Eco-system. Suggests that the design should allow easy growth and be a living and extensible system. 

Getting from Xi To Xideco

In order to develop Xideco two main challenges needed to be overcome:

• The First Challenge
• Finding a flexible networking library.
• This library should be able to support a system that resides on a single computer as well as having the ability to distribute the components across multiple computers, at the user's discretion.
• The Second Challenge
• Finding a way to describe system functionality in an abstract and language neutral manner.
• Decouples the system from a specific vendor centric implementation.
• From a higher level, control point of view, all hardware platforms "look alike" - consistency is good.
• Forces all hardware specific development to the "edges" of the system, minimizing coding churn for all other portions of the system.
• Requirements for the messaging library:
• Messages must be easy to compose and  parse.
• Library must provide an easy way to serialize messages so that they can be easily and consistently transported throughout the network.

Overcoming the Challenges

Challenge one was solved by selecting ZeroMQ , and challenge two was solved by selecting MessagePack.

So, What is ZeroMQ ?

From the ZeroMQ guide, here is a description of ZeroMQ in "a hundred words".

ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ZeroMQ is from iMatix and is LGPLv3 open source.

Things that attracted me to ZeroMQ:

• Asynchronous I/O Model
• Multi-language support (bindings for many languages)
• Multi-OS support
• Scalable
• Fast
• Open Source
• No need for an intermediate broker.

Xideco uses PyZMQ,  a Python binding API for ZeroMQ. Want to learn more about PyZMQ? Check out the PyZMQ tutorial.

A JavaScript binding was also employed for the Node.js Bridge described below.

And What is MessagePack ?

MessagePack is a language-neutral, platform-neutral, extensible mechanism for serializing data. To quote the Message Pack web site, "It's like JSON, but fast and small." Message Pack makes message creation and parsing easy and natural.

MessagePack was selected since it does not require schema definitions to describe and create messages. Xideco messages are short and the JSON like format  seemed to be a natural fit.

It too, like ZeroMQ, has bindings for many computer languages.

Xideco Proof Of Concept Architecture

Here are the goals for the proof of concept system:

• Run Xideco software components cooperatively on multiple computers.
• Control 2 Arduino boards using a Scratch GUI.
• "Plug in" a JavaScript application to log specific Xideco protocol messages.
• And a bonus application (not shown in the diagram below) to send Tweets from Scratch while monitoring and controlling the Arduino boards. A detailed tutorial on the implementation of the "Xideco Twitter Bridge" can be found here.

For the proof of concept,  Arduino boards were used.  Subsequent releases will provide Xideco "bridges" for other board types (Raspberry Pi, BeagleBone Black, etc.) and other user interfaces (Blockly, tkinter, etc.). No matter what the microcontroller or user interface, the Xideco messaging protocol stays consistent so adding new components does not affect existing components. To create a new component, the component is coded against the existing messaging protocol or new protocol messages are added as needed. Changes are made at the edges of the system and not at its internals.

Test Bed Configuration

The drawing above depicts the Xideco Software Components used in the proof of concept test bed. The Xideco system was distributed across 4 computers running simultaneously:

1. A Windows 10  computer
2. An Ubuntu 14.04 computer
3. An Ubuntu 15.10 computer
4. A BeagleBone Black Microcomputer. 

Functionality  is distributed as follows:

• A Scratch off-line editor  is running on the Windows 10 computer serving as the systems GUI.
• The HTTP Bridge is running on the same Windows 10 Computer (Scratch and the HTTP Bridge must reside on the same computer due to a limitation built into Scratch).
• An Uno is physically connected to a computer running Ubuntu 14.04. Pin 6 is configured as a digital output and is connected to an LED. Pin 12 is configured as a digital input and is connected to a push button switch.
• A Leonardo is connected to a BeagleBone Black (running Debian Wheezy). It has pin 13 configured as a digital output to control the built-in Arduino LED.
• The Xideco Router (described below), is running on the Ubuntu 15.10 computer.
• An experimental JavaScript Bridge is running on the Ubuntu 15.10 computer as well. It monitors and prints any commands that are "sent"  to the Uno on the Ubuntu 14.03 computer. It may be "plugged" into the system at anytime without disrupting any ongoing activity of the system. It may be run on any of the 3 other computers.

The component distribution used for the test bed was picked at random. Xideco software components may be freely distributed at the user's discretion. The components can all be running on a single computer, or across multiple computers as shown in the test bed above, without any change to configuration files. You can put any component on any computer regardless of operating system. 

The Xideco Software Components

Scratch HTTP

Scratch off-line was used to drive the test bed using a set of Xideco Scratch GPIO blocks.

In addition to the default English language extension blocks, blocks for the following languages are included with the Xideco package.

Here is a screen shot of Scratch with the English language custom blocks loaded.

And here is a screen shot of the Scratch test scripts used in verifying the test bed.

Xideco Network Components

HTTP Bridge

The HTTP bridge provides protocol translation to and from HTTP and the Xideco messaging protocol. The test bed uses a single instance of the HTTP Bridge that communicates directly with Scratch.

The HTTP Bridge sends the translated Scratch commands to the Xideco router using a bidirectional  ZeroMQ PAIR socket. 

When translating an incoming HTTP message, the HTTP bridge creates a multi-part ZeroMQ message. The first part is the "message topic" or envelope (other components can choose to "subscribe" to receive all messages of that topic), and the second part is the payload of the command containing details such as pin number, and other relevant parameters. Forming the multi-part messages in the HTTP Bridge allows for direct publishing of these messages by the Xideco Router.  The router simply publishes the messages without modification.

Xideco Router

The router is responsible for distributing messages originating both from Scratch (via the HTTP Bridge) and data updates received from the Arduino Bridges. Additional bridges may be dynamically added to the network without the need for network configuration (configuration is built into the bridge components) and without disrupting current system operations.

There is a single Xideco Router instance in the test bed. 

Pub/Sub 

The router distributes messages to the Arduino and other Bridges by using ZeroMQ PUB sockets. In turn the Bridges subscribe to the specific messages they are interested in.

To receive  data updates from the Arduino Bridges, the router subscribes to receive those messages and then forwards the messages upstream to the HTTP Bridge. The bridge translates the Xideco protocol messages back into HTTP and sends them to the Scratch editor for processing.

Arduino Bridge

The Arduino bridge translates Xideco Protocol messages to and from the Firmata Protocol.

Each Arduino board has its own associated Arduino Bridge.

The Arduino Bridge creates a ZeroMQ SUB socket to listen for all messages of interest.

Arduino data updates, as well as error message notifications, are published by the Arduino Bridge using a ZeroMQ PUB socket. The router subscribes to receive these messages from all connected Arduino Bridges.

For the test bed, there are 2 Arduino Bridge instances. One for the Uno and the other for the Leonardo.

Node.js Bridge

The node.js Bridge was created in order to demonstrate just how easy it is develop a plug-in,  non-Python component for Xideco. This bridge is invoked like any other node.js program. It will attach itself to Xideco and becomes part of the Xideco network. 

The node.js bridge subscribes to receive all control messages that have an  "A1" topic. Messages with an "A1" topic are Scratch control messages identified to be processed by board #1. However, any process can subscribe to receive any message topic. The Node.js Bridge elects to receive the same messages that the Arduino Bridge does and thus, effectively making the Node.js bridage a message logger for "A1" messages.

Here is the JavaScript code for the node.js bridge:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

// This is a simple JavaScript Xideco Bridge that subscribes to // Scratch control messages destined for board #1. var zmq = require('zmq') , sock = zmq.socket('sub'); var msgpack = require("msgpack-lite"); var data; sock.connect('tcp://192.168.2.194:50502'); sock.subscribe('A1'); console.log('Subscriber connected to port 50502'); sock.on('message', function(topic, message) { data = msgpack.decode(message); console.log('received a message related to:', topic, 'containing message:', data); });

Extending Xideco - The Twitter Bridge Tutorial

To further demonstrate just how easy it is to add functionality to an existing Xideco system, check out this tutorial on the Xideco Twitter Bridge. This bridge let's you send out Tweets directly from a Scratch command block.

Summing It All Up

This is just the beginning for Xideco. I am hoping to add support for many more microcontrollers and applications in the future. 

I invite you to try it out and add and then share your own module. 

If you have any comments, questions or suggestion, please feel free to share them.

So Where Do You Get Xideco?

You can find the Xideco code here and with instructions on how to install here.

And when you use it, I hope it makes you smile.