What's Next For Xideco?

I've begun working on the BeagleBone Black Bridge. I will be using the Debian 7.9 2015-11-12 image. I wanted to use the 8.3 image, but there are some device tree overlay issues when using PWM.

There is a new branch on Github where I will be checking in code from time to time until the work is complete. This is a very volatile branch, so use it at your own risk.

I have some notes on installing Python 3.5.1 on the clean 7.9 image, and I will be updating this wiki page as I make progress. You can check that page to see the current status and read my notes. The page will be cleaned up before the branch is merged into the master branch.

I will be using the Adafruit_BBIO library as a board interface. One caveat - it must be run as root or with sudo.

Stay tuned!

Xideco 0.2.0 Published

Update Contains Full Support For

 Raspberry Pi

Information about the new Raspberry Pi Bridge may be found here.

All Scratch blocks are supported including SONAR, Servo Control, and Analog Read (using an A/D converter).

Raspberry Pi Diaries

Jan 26, 2016

Part 7

Servo is now functioning and changes uploaded to Github.

Next is sonar (hc-sr04).

Evening Update

Sonar using an  HC-SR04 is functional - I can turn reports on and off and have Scratch display the report values (distance in centimeters).

Next is support for the PCF8591 YL-40. This is an i2c device that supports A/D conversion.

Once this is complete, I will be merging the Raspberry Pi Bridge into the Github Master Branch and and start to look at the BeagleBone Black Bridge. 

Another great day!

Raspberry Pi Diaries

Jan 24, 2016

Part 6

Last night, my Raspberry Pi Model B Rev 2 started acting erratically and then it decided to not even boot. I decided to rebuild the SD card and reinstall everything.

After flashing the SD card with the Raspian Jessie release, the PI booted up. I then proceeded to install everything I needed for the Raspberry PI bridge using my own instructions here.

I then installed the pigpio library.

Doing this on the Model B rev 2 (not to be confused with the Pi2) was a very slow process. To do everything took over 3 hours. I will be doing the same on a Pi2 in the future just to compare the differences in time to do a complete reinstall.

I installed the SD card into the Pi, it booted up and everything seemed right with the world again. I did  a shutdown of the Pi to make sure not to corrupt the image, unplugged it and went to bed. This morning I powered up the Pi and it wouldn't boot, so I thought I would swap the SD card into a Pi 2 to see if it would boot, and it did - the image was not corrupted. I swapped it back to the B and no boot. So I have a flaky Raspberry Pi B rev 2. I will deal with this later, but will continue work on the Pi 2.

Tone and Tone Off have been implemented. I can't believe how easy it was to do this using pigpio - what a wonderful library. I used some of the "wave" methods. The only confusing part was figuring out how to populate the "pulse" class entries to create audio frequencies.

Here is some commented test code using pigpio waves:

import time  # import the time library

import pigpio  # import the GPIO library

pin = 18

# establish 2 pulse definitions to create a 1k hz tone
# the first entry sets pin 18 high for a duration of 1000 microseconds
# and the second entry set pin 18 low for a duration of 1000 microseconds
tone_1k = [pigpio.pulse(1 << pin, 0, 1000), pigpio.pulse(0, 1 << pin, 1000)]


# remember to execute "sudo pigpiod" on the Pi before executing

#instantiate pigpio
pi = pigpio.pi()

# set the buzzer pin to output
pi.set_mode(pin, pigpio.OUTPUT)

# clear any previous wave descriptions
pi.wave_clear()

# create a wave descriptor
pi.wave_add_generic(tone_1k)  # 100 ms flashes

# create the tone and save the id
tone = pi.wave_create()

# play tone continuously for one second
pi.wave_send_repeat(tone)
time.sleep(1)
pi.wave_tx_stop()  # stop waveform

#clean up
pi.wave_clear()  # clear all waveforms

pi.write(pin, 0)

On to controlling servos next.

Raspberry Pi Diaries

Jan 23 2016

Part 5

I am testing the error paths on some of the blocks (enter a gpio pin number of 99, trying to do a digital write to a pin that was previously setup as an input, etc.). In my previous diary post I mentioned that I will be displaying debugger information as detailed text instead of reference numbers.

I have reconsidered this decision and am going back to reference numbers. The reasoning behind this is that approach is a burden on non-English speakers. If I use reference numbers and provide a cross reference of error number to English text string on the Wiki page, that page could be translated using something like Google translate. 

I am continuing to test error conditions and once I am satisfied with the error processing for the currently functioning blocks, I will go on to block functionality I have not yet addressed.

Here are the items yet to be addressed:

• Tone
• Servos
• Sonar
• Analog Input

For Analog Input, I just purchased an a/d converter module that is already supported by PIGPIO.  The module has a potentiometer and light sensor so it will be fun to play with.

Stay tuned.

The Raspberry Pi Diaries

Jan 22, 2016

Part 4

Some more progress to report. Uploaded the latest Rpi bridge to Github. The bridge is able to detect the board type (B, B+, etc.) and create a pin map of available GPIO pins for that board type.

If an operation is attempted on an illegal GPIO for the board type connected, it is detected, denied.

I am able to set the pin mode for Digital Input and Output from Scratch and the Enable/Disable function is now working.

If Digital Input is set to Disabled, reports are not sent back to Scratch.

If Digital Output is set to Disabled, performing a digital write has no affect on the pin.

The user can toggle back and forth between enabled and disabled.

If a digital or analog (PWM) write is attempted and the GPIO was not set to OUTPUT mode, the write is denied. 

To Do: Come up with a Raspberry Pi set of error messages and verify that they are being reported to Scratch for each error condition.

LATE EVENING UPDATE

Modified debug error reporting to be meaningful strings instead of reference numbers.

To Do: Update Arduino Bridge to do the same.

Control C handling working consistently well now.

SOLVED:

Xideco Installation Problems On Windows

Thanks to the quick action of the pyzmq developers, the problem has been solved.

Xideco Installation Problem On Windows

The pyzmq library was recently updated by its authors and installing this version (15.2.0) fails on windows. 

I have entered an issue on Github. Until the problem is fixed, I have put an installation workaround on the wiki. 

If you have any questions or problems, please let me know.

Thanks,

Alan

The Raspberry Pi Diaries

Jan 20 2016

Part 3

Great progress! I have updated the Raspberry Pi Bridge (code is on Github).

First a word about remote debug on the PI using PyCharm. Turns it out it was trivial. There are 2 methods to do remote debugging, a complicated method and an easy method. Of course I chose the easy way. It is called "Remote debug with a remote interpreter". 

I am able to do all of my edits on my PC, do a code inspection, upload the code to the PI and run a debug session (code running on PI), set breakpoints, etc., all on my PCthrough SSH.

Using the PIBrella as a test vehicle, I successfully:

• Enabled code to connect to xideco
• Connect the RPi to Xideco (all other components are running on my PC)
• Configured LED pins as Input and PWM
• Performed digital and analog(PWM) writes to the LEDs

Still lots to do, but I was able to get some basic functionality up and running in about an hour.

Not bad if I say so myself.

The Raspberry Pi Diaries

Jan 19 2016

Part 2

Not a bad day. Got all of the following working:

• get_hardware_revision
• get_pigpio_version
• initializing a pigpio session
• configuring a pin for digital input
• registering a callback
• monitoring the pin for changes
• releasing PIGPIO resources upon exit.
• able to edit on my pc in Pycharm and upload changes to Raspberry Pi and download any changes made on the Raspberry Pi back to Pycharm

Not Accomplished:

• I need to refresh my memory on how to configure Pycharm for remote debugging. Time to check my notes from 2 years ago ;-).

Here is the output from the run:

pi@raspberrypi:~/xideco/xideco/xideco/raspberrypi_bridge$ python3 xirb.py
board num: 1
HW REV: 14
PIGPIO REV: 43
11 1 1685339352
11 0 1685437103
^Cdone done
cleaning up
pi@raspberrypi:~/xideco/xideco/xideco/raspberrypi_bridge$ 

The output that starts with "11" is from the callback.  I have a pibrella hooked up and it has a switch on pin 11 (BCM). You can see the value going to 1 and then to a 0. The value "1685339352" is a time stamp provided by the callback.

The Raspberry Pi Diaries

Jan 19, 2016

After publishing Xideco and having a good night's sleep, I am about to start on the Xideco Raspberry Pi Bridge.

I am creating this "diary" in the hopes that it may encourage others to create Xideco plug-in modules. I have created a Github branch so that my changes can be tracked by anyone interested.

I will be updating the diary and the branch on an irregular basis, but will try to do so on an almost daily basis until the project is complete.

So let's get started.

GPIO Library

After looking at several readily available and popular Raspberry Pi GPIO libraries I have decided to use PIGPIO. My reasons for choosing this library over the others were:

• A well thought out Python API.
• It worked out of the box.
• Provides callbacks.
• User does not have to be root to affect the GPIO (after starting a daemon as root)
• Seems to support ALL of the features I need.

Development Environment

RPi Board

I am starting off using a Raspberry PI 1 Model B, but have a B+ and a 2 on hand as well. Hopefully the Model B won't run out of steam, but time will tell.

RPi Operating System

I am starting with Raspbian Wheezy, May 2015 release. I will be moving on to the Jessie release sometime in the future.

IDE

I will be using Pycharm 5 Professional running on my main PC for editing, Github support and remote debugging of code residing on the RPi.

Getting Started

I will be using the Arduino Bridge as a template, but there will be major changes. All Arduino specific code will be disabled or removed. So my first goals are:

Try out some of the PIGPIO API calls, starting with something simple, like get_hardware_revision and get_pigpio_version, initializing a pigpio session, configuring a pin for digital input and registering a callback, monitoring the pin for changes and releasing PIGPIO resources upon exit.

I will report back any issues that come up, and will publish the code on the Github branch when I have something worth sharing.

Xideco

A Network Enabled Software Backplane

Artwork from http://www.terrancesimien.com/creole-for-kidz/

When looking for a new project to work on, I began to reflect on Xi, a project I created back in the Fall of 2014. 

Xi allows a user to control a set of of individual  microcontrollers as if they were a single "megacontroller" using either Scratch or Snap! as a graphical user interface.

Although Xi works very well, it is not without its limitations. Here are some of them:

• The code does not allow boards to directly communicate with each other. All communication must go through either Scratch or Snap!. 
• Xi is closely coupled to Scratch and Snap! - replacing the control interface is not simple.
• The code was written in JavaScript and has no easy way of incorporating features written in other computer languages.
• It is tightly coupled to the Johnny Five library, making it difficult to extend beyond the features of Johnny Five.

For my new effort I put together a set  of "wants":

• Python 3.5 should be the system's language of choice, however if a user wishes to develop using some other language, they should have the freedom to do so.
• The system should be portable with the ability to be executed on:
• Windows.
• Mac.
• Linux.
• The system should be distributable:
• Any component should be able to run on any computer within a local area network, using any operating system or combination of operating systems.
• Configuration should be simple with minimal user interaction, but flexible enough to allow modification if it becomes necessary to do so.
• The system should be extensible and flexible for physical as well as other types of computing:
• It should support current boards types.
• Have the capability to support future board types.
• Have the capability to run general applications (like sending Tweets).
• Be able to interface to, and make use of, non Python software.
• Have the ability to support a set of services that could span multiple boards.
• The ability to describe current and future capabilities through a portable, extensible and language neutral messaging protocol.

From Xi, to Xideco

How Do You Pronounce Xideco?

Xideco is pronounced like Zydeco. Zydeco is a form of  Louisiana Cajun music.
What can I say? Whenever I hear Zydeco music, it makes me smile - take a listen.

What does the Xi-d-eco acronym stand for?

Xi = Cross Platform Interconnect

Cross platform meaning support for multiple single board computer types,on multiple operating systems, using multiple programming languages.

d = Distributable

Having the ability to cooperatively execute Xideco software components residing on a single computer as well as on multiple computers. 

eco

Eco-system. Suggests that the design should allow easy growth and be a living and extensible system. 

Getting from Xi To Xideco

In order to develop Xideco two main challenges needed to be overcome:

• The First Challenge
• Finding a flexible networking library.
• This library should be able to support a system that resides on a single computer as well as having the ability to distribute the components across multiple computers, at the user's discretion.
• The Second Challenge
• Finding a way to describe system functionality in an abstract and language neutral manner.
• Decouples the system from a specific vendor centric implementation.
• From a higher level, control point of view, all hardware platforms "look alike" - consistency is good.
• Forces all hardware specific development to the "edges" of the system, minimizing coding churn for all other portions of the system.
• Requirements for the messaging library:
• Messages must be easy to compose and  parse.
• Library must provide an easy way to serialize messages so that they can be easily and consistently transported throughout the network.

Overcoming the Challenges

Challenge one was solved by selecting ZeroMQ , and challenge two was solved by selecting MessagePack.

So, What is ZeroMQ ?

From the ZeroMQ guide, here is a description of ZeroMQ in "a hundred words".

ZeroMQ (also known as ØMQ, 0MQ, or zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It's fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ZeroMQ is from iMatix and is LGPLv3 open source.

Things that attracted me to ZeroMQ:

• Asynchronous I/O Model
• Multi-language support (bindings for many languages)
• Multi-OS support
• Scalable
• Fast
• Open Source
• No need for an intermediate broker.

Xideco uses PyZMQ,  a Python binding API for ZeroMQ. Want to learn more about PyZMQ? Check out the PyZMQ tutorial.

A JavaScript binding was also employed for the Node.js Bridge described below.

And What is MessagePack ?

MessagePack is a language-neutral, platform-neutral, extensible mechanism for serializing data. To quote the Message Pack web site, "It's like JSON, but fast and small." Message Pack makes message creation and parsing easy and natural.

MessagePack was selected since it does not require schema definitions to describe and create messages. Xideco messages are short and the JSON like format  seemed to be a natural fit.

It too, like ZeroMQ, has bindings for many computer languages.

Xideco Proof Of Concept Architecture

Here are the goals for the proof of concept system:

• Run Xideco software components cooperatively on multiple computers.
• Control 2 Arduino boards using a Scratch GUI.
• "Plug in" a JavaScript application to log specific Xideco protocol messages.
• And a bonus application (not shown in the diagram below) to send Tweets from Scratch while monitoring and controlling the Arduino boards. A detailed tutorial on the implementation of the "Xideco Twitter Bridge" can be found here.

For the proof of concept,  Arduino boards were used.  Subsequent releases will provide Xideco "bridges" for other board types (Raspberry Pi, BeagleBone Black, etc.) and other user interfaces (Blockly, tkinter, etc.). No matter what the microcontroller or user interface, the Xideco messaging protocol stays consistent so adding new components does not affect existing components. To create a new component, the component is coded against the existing messaging protocol or new protocol messages are added as needed. Changes are made at the edges of the system and not at its internals.

Test Bed Configuration

The drawing above depicts the Xideco Software Components used in the proof of concept test bed. The Xideco system was distributed across 4 computers running simultaneously:

1. A Windows 10  computer
2. An Ubuntu 14.04 computer
3. An Ubuntu 15.10 computer
4. A BeagleBone Black Microcomputer. 

Functionality  is distributed as follows:

• A Scratch off-line editor  is running on the Windows 10 computer serving as the systems GUI.
• The HTTP Bridge is running on the same Windows 10 Computer (Scratch and the HTTP Bridge must reside on the same computer due to a limitation built into Scratch).
• An Uno is physically connected to a computer running Ubuntu 14.04. Pin 6 is configured as a digital output and is connected to an LED. Pin 12 is configured as a digital input and is connected to a push button switch.
• A Leonardo is connected to a BeagleBone Black (running Debian Wheezy). It has pin 13 configured as a digital output to control the built-in Arduino LED.
• The Xideco Router (described below), is running on the Ubuntu 15.10 computer.
• An experimental JavaScript Bridge is running on the Ubuntu 15.10 computer as well. It monitors and prints any commands that are "sent"  to the Uno on the Ubuntu 14.03 computer. It may be "plugged" into the system at anytime without disrupting any ongoing activity of the system. It may be run on any of the 3 other computers.

The component distribution used for the test bed was picked at random. Xideco software components may be freely distributed at the user's discretion. The components can all be running on a single computer, or across multiple computers as shown in the test bed above, without any change to configuration files. You can put any component on any computer regardless of operating system. 

The Xideco Software Components

Scratch HTTP

Scratch off-line was used to drive the test bed using a set of Xideco Scratch GPIO blocks.

In addition to the default English language extension blocks, blocks for the following languages are included with the Xideco package.

Here is a screen shot of Scratch with the English language custom blocks loaded.

And here is a screen shot of the Scratch test scripts used in verifying the test bed.

Xideco Network Components

HTTP Bridge

The HTTP bridge provides protocol translation to and from HTTP and the Xideco messaging protocol. The test bed uses a single instance of the HTTP Bridge that communicates directly with Scratch.

The HTTP Bridge sends the translated Scratch commands to the Xideco router using a bidirectional  ZeroMQ PAIR socket. 

When translating an incoming HTTP message, the HTTP bridge creates a multi-part ZeroMQ message. The first part is the "message topic" or envelope (other components can choose to "subscribe" to receive all messages of that topic), and the second part is the payload of the command containing details such as pin number, and other relevant parameters. Forming the multi-part messages in the HTTP Bridge allows for direct publishing of these messages by the Xideco Router.  The router simply publishes the messages without modification.

Xideco Router

The router is responsible for distributing messages originating both from Scratch (via the HTTP Bridge) and data updates received from the Arduino Bridges. Additional bridges may be dynamically added to the network without the need for network configuration (configuration is built into the bridge components) and without disrupting current system operations.

There is a single Xideco Router instance in the test bed. 

Pub/Sub 

The router distributes messages to the Arduino and other Bridges by using ZeroMQ PUB sockets. In turn the Bridges subscribe to the specific messages they are interested in.

To receive  data updates from the Arduino Bridges, the router subscribes to receive those messages and then forwards the messages upstream to the HTTP Bridge. The bridge translates the Xideco protocol messages back into HTTP and sends them to the Scratch editor for processing.

Arduino Bridge

The Arduino bridge translates Xideco Protocol messages to and from the Firmata Protocol.

Each Arduino board has its own associated Arduino Bridge.

The Arduino Bridge creates a ZeroMQ SUB socket to listen for all messages of interest.

Arduino data updates, as well as error message notifications, are published by the Arduino Bridge using a ZeroMQ PUB socket. The router subscribes to receive these messages from all connected Arduino Bridges.

For the test bed, there are 2 Arduino Bridge instances. One for the Uno and the other for the Leonardo.

Node.js Bridge

The node.js Bridge was created in order to demonstrate just how easy it is develop a plug-in,  non-Python component for Xideco. This bridge is invoked like any other node.js program. It will attach itself to Xideco and becomes part of the Xideco network. 

The node.js bridge subscribes to receive all control messages that have an  "A1" topic. Messages with an "A1" topic are Scratch control messages identified to be processed by board #1. However, any process can subscribe to receive any message topic. The Node.js Bridge elects to receive the same messages that the Arduino Bridge does and thus, effectively making the Node.js bridage a message logger for "A1" messages.

Here is the JavaScript code for the node.js bridge:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

// This is a simple JavaScript Xideco Bridge that subscribes to // Scratch control messages destined for board #1. var zmq = require('zmq') , sock = zmq.socket('sub'); var msgpack = require("msgpack-lite"); var data; sock.connect('tcp://192.168.2.194:50502'); sock.subscribe('A1'); console.log('Subscriber connected to port 50502'); sock.on('message', function(topic, message) { data = msgpack.decode(message); console.log('received a message related to:', topic, 'containing message:', data); });

Extending Xideco - The Twitter Bridge Tutorial

To further demonstrate just how easy it is to add functionality to an existing Xideco system, check out this tutorial on the Xideco Twitter Bridge. This bridge let's you send out Tweets directly from a Scratch command block.

Summing It All Up

This is just the beginning for Xideco. I am hoping to add support for many more microcontrollers and applications in the future. 

I invite you to try it out and add and then share your own module. 

If you have any comments, questions or suggestion, please feel free to share them.

So Where Do You Get Xideco?

You can find the Xideco code here and with instructions on how to install here.

And when you use it, I hope it makes you smile.

UPDATE #2 - pymata-aio and pyserial

The author of pyserial has released version 3.0.1 and this resolves the incompatibility issues with pymata-aio.

To upgrade pyserial to 3.0.1, please use the following "pip" command:

pip install pyserial --upgrade

UPDATE: pyserial 3.0 and pymata-aio

There have been some changes implemented in the pyserial github repository. I did a local pip install using the latest repository code and it solves the problems. 

These changes are not yet available on pypi, but as soon as the upcoming release is available I will verify that it is compatible with pymata-aio, and announce the results here.

pyserial 3.0 and pymata-aio

When pip installing pymata-aio, the latest version of pyserial is also installed.

pymata-aio was developed and tested using pyserial 2.7. The latest version of pyserial is now version 3.0. This latest release of pyserial changed the writeTimeout parameter to write_timeout, resulting in an exception to be thrown as shown below.

Here is a simple example based on using the pyserial 2.7 api:

import serial

my_serial = serial.Serial('/dev/ttyACM0', 57500, timeout=1, writeTimeout=1)

And here is output when using pyserial 3.0:

$ python3 sertest.py
Traceback (most recent call last):
  File "sertest.py", line 3, in 
    my_serial = serial.Serial('/dev/ttyACM0', 57500, timeout=1, writeTimeout=1)
TypeError: __init__() got an unexpected keyword argument 'writeTimeout'

I am working with the pyserial author to resolve the issue. If you are seeing this error, you can force pyserial 2.7 to be loaded using the following pip command:

pip install pyserial==2.7