Asyncio is Coming to PyMata!

The next version of PyMata, scheduled for release later this summer, incorporates the Python 3 asyncio library. "Why?", you might. ask. Let's compare results of running two similar projects that control a servo motor

Both tests were run using Python 3.4.3 on Linux. The CPU used in the tests is an Intel® Core™ i5-2500K CPU running at 3.30GHz.

First, here is the PyMata 2.08 project code:

And now the system monitor results:

Application code for the new release:

Its system monitor results:

CPU utilization went down from 28% to 6% !

Comparing the code, you might be wondering why there is no import of the asyncio library. There are 3 APIs  included with the new release.

• pymata_core
• This is a pure asyncio API for those that wish to code directly in asyncio. It is a method based API.
• pymata3
• This is a "pymata plug-in api" that acts as a proxy for pymata_core. It shields the user from having to deal with asyncio. This is the API used for the sample code above. It is method based.
• pymata_iot
• This API provides an Autobahn based Websocket interface. It too is a "pymata plug-in" API that uses JSON messaging between client and server.
• Here is a screen shot of a web application taking advantage of this API:

• This rather busy screen shot shows a page that allows control of the digital inputs and the PyMata data latching feature. Pin 12 and 13 are enabled for reporting. Pin 12 is currently low and pin 13 is high. A latch was set for pin 12 that was triggered on 27 June 2015 at 13:27:24. Pin 13 also has a pending latch to fire when it goes low and then high again.
• Here is a sample of the JavaScript code used in this screen shot for setting pin mode:

• This code uses the browser's built-in Websocket client and therefore JavaScript was used. You can easily communicate using a Python Websocket client as well. First here is the JSON message template

• And the Python code to set the pin mode via Websocket:

Here is a summary of the features of the new release, its current status and a "to-do" list.

Major Release Features

• Utilizes the Python asyncio library for concurrency implementation.
• Full implementation of the StandardFirmata version 2.4.3 protocol.
• Continued support for FirmataPlus protocol(Sonar, Stepper Motor, Tone, and Encoder support).
• 3 APIs provided for application flexibility.
• Auto-detection of the Arduino COM port.
• Built-in Control-C detection.

Current Status

• pymata_core API is code complete and tested.
• pymata3 API is code complete and tested.
• pymata_iot is 70% complete and tested.

To Be Done

• Finish pymata_iot.
• Code cleanup.
• Documentation.

PyMata Version 2.07 

(Bug Fix Release)

Now Available on Github and Pypi

Tim Slator identified a bug that manifested itself in an exception being thrown when certain analog pins were being used as digital pins. Tim's fix is incorporated into version 2.07, and tested on an Uno, Leonardo and Mega 2650.

You can obtain the latest changes using pypi or download them here.

Thanks Tim!

Xi Is Now Available on ScratchX

The Scratch team has released an experimental web site that supports hardware extensions for Scratch and a Xi client is available at:

http://scratchx.org/?url=http://MrYsLab.github.io/xi4s.js

A French translation is available at:

http://scratchx.org/?url=http://MrYsLab.github.io/xi4s_fr4.js

Xi allows Scratch to simultaneously control mulitple Arduino, Raspberry PI and Beagle Bone Black microcontrollers and allow them to share sensor data.

To use these clients, you will need to install the Xi servers that can be found at:

https://github.com/MrYsLab/xi

Futures and Coroutines and Loops! Oh my!

I have just published an updated version of esp4s, the project that uses Scratch to interface to an Arduino Esplora microcontroller board. This latest version uses Python 3.4.3 and its associated asyncio library. The new program is called esp4s-aio and is available on Github.

Going from proof of concept to a finished project was not as simple as I had originally thought. The Python documentation for asyncio is really great for reference once you understand how to use the library, but at least for me, was not very instructive for a first time user.

But once I figured out how to convert Pyserial's readline() and write() to "futures", and was able to instantiate and communicate with an aiohttp HTTP server, I was well on my way to finishing the task at hand.

But those weren't my only educational challenges. I also needed to learn how to unit test the code and came upon pytest-asyncio. After asking the package authors several rookie questions, and receiving quick and accurate answers (without any snark I might add) the last pieces of the educational puzzle were in place.

Now on to the code. I started with the same architectural design I used for the original esp4s. However, working through the code, I quickly realized that I could take a simpler, more sequential code flow approach than I used for the original multi-threaded esp4s project. This not only simplified the design, but the testing and debugging, as well.

One very major design departure from the original project, was to go from having the Esplora pushing out all sensor data every 25 ms. to a polled approach.I was concerned that performance would suffer but that turned out not to be the case. 

Now on to the results. I used a common Scratch script called test.sb2, ( which is provided in the esp4s-aio distribution) to compare esp4s and esp4s-aio. The Scratch script toggles an LED on the Esplora while concurrently displaying status for all 16 Esplora sensors. Scratch polls for status data at a rate of approximately 30 times per second.

It was interesting to find that the esp4s-aio version shows no display lag as sensor data varies, but the original version does (which means that the design of the original was sub-optimal). 

However, more dramatic was the difference in CPU utilization. First, here is  CPU utilization for the original:

And now the CPU utilization for esp4s-aio:

In this graph, the first 20 seconds are before Scratch was started and esp4s-aio is running idle. The increase begins when the Scratch script starts polling for data.

So overall, I am very pleased with the results, and plan to use asyncio coroutines for several future projects.  Since my projects tend to be more I/O intensive than CPU intensive, asyncio and coroutines are definitely in my future.

And Now, For Something Completely Different

The Grand Experiment

In writing Xi, I learned the importance and power of non-blocking asynchronous io. Xi is written in JavaScript, but my language of preference is Python, so I decided to explore Python's alternatives to node.js.

I looked at many options. Gevent, Stackless Python, Twisted,  the Python asyncio library, as well as others. What interested me most was asyncio. My reasoning here is that is part of the Python 3.4.2+ library (batteries included), and it supports concurrency without having to resort to callbacks. It performs this magic through coroutines.

Knowing the direction I wanted to go towards, I read the literature, looked at examples, and gave myself a simple task to solve. The task was to be able to read and process the relentless data stream sent by an Arduino Esplora running the Arduino sketch included with esp4s program, using nothing but asyncio coroutines.

Believing I was fully educated, I wrote some code, started up Python, and low and behold, I got a screen full of exceptions. It became abundantly clear that I was clueless when it came to coroutines, especially asyncio coroutines.

Back to the Classroom with My Tail Between My Legs

Time to go on a Google hunt for examples to emulate.  All I could find were ways to concurrently calculate a Fibonacci sequence, or open up thousands of web connections. Neither of these examples helped me solve my self-assigned problem of continuously receiving a stream of serial data without pinning the CPU utilization meter.

It became perfectly clear that looking at the available examples just wasn't enough. It was time to really learn about the nuts and bolts of coroutines and the related topic of generators. There is a ton of literature on the interwebs on these subjects, but I could find nothing that directly related to helping me solve my problem. I finally came across a link to a Dave Beazley talk entitled, "Generator Tricks for System Programmers". After reviewing the slides, I began to understand the real utility of using generators. My education was partly complete. I checked out another Dave Beazley talk, "A Curious Course on Coroutines and Concurrency". Dave explains coroutines in a way that a mere mortal like me could readily understand. Now I was hooked. So I continued with Dave's trilogy and not only looked at the slides for "Generators - The Final Frontier", but watched the YouTube video as well - from beginning to end - a first for me. Now, I have to warn you, this is a 3 hour video, and it took me a couple of days to get through it. Not because I needed to take a nap between viewings, but because the  material was so mind blowing, that I needed to take breaks to absorb it. The last piece of my coroutine crash course was to watch Dave's 2015 PyCon presentation, "Python Concurrency From the Ground Up". A mere 30 minute presentation that is chock full of goodies.

The Grand Experiment, Take 2

Now Dave does't cover the asyncio library per se (hey Dave, if you are reading, how about an asyncio talk for PyCon 2016?),  but feeling I was armed with a pretty solid understanding of coroutines, I would try and slay the asyncio dragon once again.

I started with something simple. A consumer/producer program using asyncio coroutines. It actually worked - no exceptions mocking me this time. Encouraged,  I went on to solving my original problem of processing streaming data coming from an Arduino Esplora using eps4s. Surprise, surprise - this actually worked too!

Ecstatic, I decided to prototype a total asyncio rendition of esp4s. The esp4s program allows a user to control and monitor an Arduino Esplora board using the Scratch or Snap! programming languages.

The esp4s software is sandwiched in-between the serial data stream coming from the Esplora, and the HTTP status polls and actuator commands coming from a Scratch project. A new Esplora status frame arrives every 25 ms, and a new Scratch poll request arrives every 33 ms. Neither are flow controlled and both are fully asynchronous. The esp4s processes these streams and acts as a data translator between the Esplora and Scratch project.

For the prototype of esp4s-aio, the software accepts all status frames from Esplora, processes 4 of the 16 status types (to add load), and sends them to the Scratch script. Concurrently, it receives the poll requests and responds with the latest status data and in addition, concurrently receives a separate stream of commands from the Scratch script to control the Esplora. For the prototype, I created a Scratch script to blink the red LED on the board.

The esp4s program is being pushed and pulled from all directions!

The Results

So was all this effort worth it? Here is a screen shot of CPU utilization while running the threaded version of  esp4s:

And here is the prototype using asyncio coroutines:

It looks like I can have my cake (writing in Python), and eat it too (have the concurrency of node.js)!

What's Next?

In the coming weeks I will be completing and unit testing the code.

Once this is all complete, I will be publishing esp4s-aio. 

Stay Tuned!

A Shameless Plug

PyCharm

I have been using 2 extraordinary tools to help in developing esp4s-aio. The first is a Python IDE called PyCharm. There is a free (really free - no expiration date) community addition available for download. I am using the professional version, but used the community edition for most of my previous work.

PyVmMonitor

The second tool is PyVmMonitor, a profiler that plugs into PyCharm.

Spanish Translation for esp4s

Now Available on Github

Thank you to José Manuel Ruiz Gutiérrez for providing the Spanish translation files.

Attention Scratch Xi Users

Scratch 2.0 Goes From HTTP to HTTPS Preventing the Xi Server From Connecting to the Scratch Xi Client

This change prevents Xi from making a websocket connection between the Xi server and the Xi extension client loaded into Scratch.

Snap! is unaffected and will continue to work as normal.

We are working on providing a solution using secure websockets, but in the meantime, there is a workaround for both Chrome and Firefox by setting a flag in the browser.

For Firefox, enter this "url" into the browser:

about:config

When the warning screen appears, click on the "I'll be careful I promise" button.

Next, search for network.websocket.allowinsecureFromHTTPS and click on the line to set it to true. Then close and reopen Firefox.

For Chrome, enter this "url" into the browser:

chrome://flags

and then search for Allow insecure WebSocket from https origin.

Then click on "enable". Close and reopen Chrome.

If you have any questions or concerns, just send us an email using the "Contact" feature on the left side of this page (you may have to scroll down).