300x250 AD TOP

Search This Blog


Featured Post

Reusing a Laptop Camera

A few months ago I came across this instructable , its showing how to reuse a laptop camera module as a usb device, which some of them use. ...

Paling Dilihat

Powered by Blogger.

Feature Label Area

Tuesday, January 23, 2018

Reusing a Laptop Camera

A few months ago I came across this instructable, its showing how to reuse a laptop camera module as a usb device, which some of them use. so later when I've decided to clean up and get rid of old and defunct laptops, I've found two camera modules and attempted to repurpose them.

The module's quality is nothing to be impressed by but they are UVC compatible, which should be good for linux use as well, but I'll test them at a later time.

The connectors seem like a standard, the following diagram is from the instructable, one of my modules even had the pinouts on the silkscreen!

While my devices are:

I've had a few USB connectors laying around, so I've looked up the pinouts

Soldered the connectors

and tested the cameras are not a short circuit risk with a USB tester

The first showed up as VID_05E3&PID_0505, which is a Genesys Logic USB2.0 PC Camera, its power consumption is around 80ma.

The second one showed up as VID_04F2&PID_B106, which is a Chicony® USB M UVC WebCam, its power consumption is around 110ma.

Next, testing them with an Orange PI....

Tags: ,

Wednesday, January 3, 2018

IDesign - The Architect's Master Class - Review

I've had the privilege of taking Juval Lowy's - The Architect's Master Class last week and while I've been a Senior Software Developer for the last 10 years, I've also wore the architect's hat in some of my past positions.

Juval takes the class on a roller coaster of  realizations, explaining the difference between computer science and software engineering which requires an understanding of what engineering really is about as well as the business aspects of change and growth and what's the architect's role in a software project that is on time, on budget while maintaining high quality.

I don't consider myself easily persuaded, I do believe that I learned some new mental and practical tools. Some participants complained that the class was a bit overly dramatic but I think it was on purpose, most if not all participants listened to Juval for a week from 9:00 until 20:30 with only small breaks for recuperation, lunch, coffee and the occasional cookie, so "drama" did the trick.

The first 4 days looked like they were planned to break habits, open the participants mind to new (or old to be precise, see David Parnas's On the criteria to be used in decomposing systemsinto modules,  and Designing Software for Ease of Extension and Contraction), and eventually present Juval's arguments towards a proper software engineering project planning and execution, e.g. "The Method".

Some of the quotes that stuck:

"The only way for a project to be on time, on budget and on quality at the end is to be on time, on budget and on quality throughout"

"The purpose of testing is to prove something doesn't work"

"Being Agile, not DOING Agile"

"Agile forgot the Andon Cord"

"Gold Plating"

"Features are the result of integration, not implementation"

But I won't ruin it for you, if you have the time, budget and mental readiness, you should go, you don't have to do anything about it... but you'll probably learn a thing or two and if not, the interaction with other architects or aspiring architects will probably be fruitful.

Further Reading:


Saturday, December 16, 2017

Machine Learning Automatic License Plate Recognition

I'm starting to study deep learning, mostly for fun and curiosity but following tutorials and reading articles is only a first step.

Though I know and programmed multiple languages in the past, somehow deep learning is associated with Python and as someone who likes C like languages it was always a dislike for me, the whole concept of using spaces to control program blocks looked ridiculous to me, but what the hell, lets try to learn it, it makes things a lot less complicated than compiling Tensorflow, Caffe or OpenCV from source and then trying to get them to talk to each-other, where in python these issues have already been solved.

Learning neural networks have been on my mind for quite a while, I've even read a few neurology books to understand the origins of these ideas but only when I've attended GTC Israel 2017 and had the chance for hand-on guided Nvidia DIGITS session I've started to take active interest, though not really achieve anything new for a while.

10 points if you can locate me in this clip

So I thought about a cool project, though I'm not sure what its usefulness is going to be now, so how about recognizing and registering all the vehicle license plates around one's car?

Algorithmic Approach

At first I've tried OpenALPR's approach, finding a large rectangle with multiple rectangles inside it. it works if the license plate is a major object in the image, but not if there are multiple vehicles, not to mention an unstructured scene, like driving on the highway or a mobile camera of some sort, though I might have not implemented it correctly in my code.

Image Segmentation

So the 2nd approach I've thought about is image segmentation. I've been reading a lot about ENet, SegNet and ICNet lately and was eager to try it. and so I've began to look for a Keras model to get things started. But then I realized, I don't necessarily need the localized polygon of the license plate, a bounding box should be more than enough. then I can pass the cropped image to tesseract and get a license plate.

Object Detection

So I've looked up a few object detection models, such as SSD, YOLO, Faster RCNN, R-FCN, RetinaNET and more are being designed as we speak. I've decided to go with YOLO, being biased to it after seeing a demo I liked.

But to train any kind of machine learning model, you need data and lots of it. I've started to look for a license plate dataset but couldn't find anything that has both the images and the polygons... but then I remembered I've seen that in the Cityscapes Dataset there is an unmarked license plate class, so theoretically all I needed to do was generate the right mask/polygons for the training.

I've cloned the basic-yolo-keras repo by Huynh Ngoc Anh, updated it to work with python 3 and ran a training session on the dataset.

Having a laptop, its a bit of a problem to train on it, since its not always on, I need to take it with me etc' etc'. so I've looked for an online solution. eventually I ended up using Azure NC6 machine at $0.90/hour, it has Nvidia K80 with 12GB of RAM so I could increase the batch size to make things run a bit faster, eventually training took less than 24 hours on a ~2400 images, some with more than one sample.

(on my 1050TI, this video was created at about 9fps)

As you can see the license plate should be readable, otherwise it doesn't really detect it, I didn't plan this, so I'm guessing YOLO training is really good or its a side effect of using the Cityscapes Dataset quality.


My next task was OCRing the license plates so I can get data I can list and log, I've had some experience with tesseract in the past, so I chose to try it this time as well.

Well.. this didn't go as smooth as I wanted... while many license plates are readable by a human, the noise is just too high for tesseract to recognize reliably.

The following video was shot with 4K camera, high shutter speed and high bitrate (SJCAM M20), but the recognition quality has marginally increased.

(creating this video was even slower, the GPU didn't work as hard, but tesseract did a lot of work (CPU), about 1.5fps)

I've had my fun with this project, but I think the next step could be another deep learning  object detection, only this time it should be the license plate numbers in case of Israel in addition, letters - for many others.

If I may guess further, the reason this project was not a complete success is the OCR process, the camera is an action camera, so very wide lens, that means very low resolution for each license plate.

I'm pretty sure further pre-processing effort might raise tesseract's recognition quality, they do look readable. I did discover that Israeli license plates are just too tall for tesseract's english detection, which is somewhat amusing.
If pre-processing doesn't work as desired, this little project has taught me that machine learning can probably do this task as well and probably with high precision.

Source Code

I'm still not ready to publish any python code, I will need to familiarize with more of it before being ready to do so.
In any case, there is nothing new there, the code for building the CityScapes dataset extract is basically just parsing the JSON files and producing VOC format XML, the YOLO code is the code from basic-yolo-keras with some adjustments. and lastly the cleanup code for the license plate is just a simple auto-levels like code on the V channel in HSV.


This was a fun project, I'm sure that with further research it can be a pretty cool and reliable software, using YOLO for license plate detection seemed to work pretty good, perhaps cleaning up the dataset and further optimizing the training and inference processes will make it even better, perhaps using a machine learning based number/letter recognition will make reading the plates more reliable. perhaps it can all be coded with an algorithm rather than a model.... maybe the next thing should be recognizing car maker and color?....

Further Reading:

Speed/accuracy trade-offs for modern convolutional object detectors by Felix Lau
Cityscapes Dataset
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
ICNet for Real-Time Semantic Segmentation on High-Resolution Images
SSD: Single Shot MultiBox Detector
You Only Look Once: Unified, Real-Time Object Detection
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
R-FCN: Object Detection via Region-based Fully Convolutional Networks
RetinaNET - Focal Loss for Dense Object Detection


Tuesday, December 5, 2017

Flashing ATtiny85 with USBasp and Making a PWM Generator

I've ordered a few Digispark clones (originally made by DigiStump) from AliExpress for use in low pinout, low power, tiny projects that don't require much code as they use the ATtiny85. These devices looked very cool and suitable for my needs and the fact that they can be programmed without the cumbersome ISP programmers made it even more appealing (spoiler: the clones did not work properly)

At first I expected them to be able to communicate over USB and expose a com port, which would make programming them even easier but they don't work that way.


The firmware bootloader is based on micronucleus and the USB interface  is V-USB, it is a firmware/software only implementation of low-speed USB device, so any functionality you wish to implement needs to be in software, even virtual com ports, for examples you can look here.

Something to note about the micronucleus firmware is the 5 seconds startup delay. If you need the device to start up immediately, you'll have to try a different approach where shorting pin 5 to ground enabled programming, otherwise it starts immediately. There is however a solution for it, but I did not test it.

So I've hooked up the Digispark, started the Arduino IDE and loaded blink and...

\micronucleus\2.0a4/launcher -cdigispark --timeout 60 -Uflash:w:Blink.ino.hex:i 
Running Digispark Uploader...
Plug in device now... (will timeout in 60 seconds)
> Please plug in the device ... 
> Press CTRL+C to terminate the program.
> Device is found!
connecting: 16% complete
connecting: 22% complete
connecting: 28% complete
connecting: 33% complete
> Device has firmware version 1.6
> Available space for user applications: 6012 bytes
> Suggested sleep time between sending pages: 8ms
> Whole page count: 94  page size: 64
> Erase function sleep duration: 752ms
parsing: 50% complete
> Erasing the memory ...
erasing: 55% complete
erasing: 60% complete
erasing: 65% complete
> Starting to upload ...
writing: 70% complete
writing: 75% complete
writing: 80% complete
> Starting the user app ...
running: 100% complete
>> Micronucleus done. Thank you!

no dice.

Apparently the clones were flashed with micronucleus but either an old version or wrong fuses.


Fixing Digispark Clones with USBasp

For a different purpose I've ordered a cheap USBasp 2.0 programmer from AliExpress, although it comes with a firmware already flashed, its using an old usbasp firmware so its vendor id and product id are not compatible with current avrdude which is being used by Arduino.

If you insist on updating the USBasp firmware, you can follow Darell Tan post, which is based on work made by Uwe Zimmermann.

otherwise you can use PROGISP  v1.72 which does a great job.

I've connected the USBasp to the Digispark

First, connect the pins according to this:

RST <> PB5

Next, I've downloaded the lastest micronucleus firmware for ATTiny85 and flashed the bootloader:

- Select the chip as ATTiny85
- Click RD to see it is able to communicate appropriately
- Load the appropriate micronucleus firmware
- Select the fuses according to the documentation.
- Click Auto to flash the bootloader

Note: Note fuse RSTDISBL, You might not be able to use PB5 as its used as external reset pin.

We then retry to flash using Arduino IDE / USB cable and its working!

Running Digispark Uploader...
Plug in device now... (will timeout in 60 seconds)
> Please plug in the device ... 
> Press CTRL+C to terminate the program.
> Device is found!
The upload process has finished.
connecting: 16% complete
connecting: 22% complete
connecting: 28% complete
connecting: 33% complete
> Device has firmware version 2.1
> Device signature: 0x1e930b 
> Available space for user applications: 6522 bytes
> Suggested sleep time between sending pages: 7ms
> Whole page count: 102  page size: 64
> Erase function sleep duration: 714ms
parsing: 50% complete
> Erasing the memory ...
erasing: 55% complete
erasing: 60% complete
erasing: 65% complete
> Starting to upload ...
writing: 70% complete
writing: 75% complete
writing: 80% complete
> Starting the user app ...
running: 100% complete
>> Micronucleus done. Thank you!

ATtiny85 as a PWM generator

I've wanted a decent pwm generator that can display the duty cycle width for diagnosing problems with brushless ESCs and servo motors, so the cheap ones would not do. I've decided to build my own, as always for education and fun.

We'll start with the basics, the schematics:

As mentioned previously, the ATtiny85 does not have a dedicated USB port, nor does it have a dedicated serial port, so how does it communicate over USB? It does so with V-USB, which is a software emulation for a USB port/device.

But the USB connection comes at a cost, you'll most likely encounter problems if pins 4-5 (PB3-4) will have any contact with other components. so either flash and test this chip on a breadboard or flash once and forget you did, or just don't use these pins.

So lets start with our PWM generator.

I'm using:
Digispark ATtiny85 Clone - This chip is good enough for this purpose, only drawback is the limited timers.
10k potentiometer - As an analog input for the 10bit ADC, which is then mapped fromo 400-2400us.
TM1637 7 segment 4 digits display - To display the selected duty cycle.
1k resistor and 1N4004 Diode - for output protection, I've just had to replace the ATtiny85 due to feedback from a servo, so its my attempt to protect it.

The source code is very simple, its actually the example from Adafruit SoftServo with display code.

Timer Limitation

The ATtiny85 has a limited timer, which can't be used to control high resolution PWM signal. but Adafruit SoftServo is a good solution for this problem, rather than use the timer to control the PWM directly, it uses the timer to call a function that simulates a PWM signal by writing the pin directly with a delayMicroseconds between.

ADC Noise

At first I've tried to read the potentiometer directly and push an update to the servo, but I got so much noise from the ADC that the servo shaked a lot. I think most if not all ADCs have a noise problem. usually a capacitor and a low pass filter can eliminate some of the noise and while the ATtiny85 has a ADC Noise Reduction Mode, I've resorted to use a simple solution:

long avg = 0;
for (int i = 0; i < 100; i++) {
avg += analogRead(POT_PIN);
val = avg / 100;

It will never be complete without a printed enclosure :-)

Further References
- I had to check which vendor/device ids to find out how to use the USBasp first as the device didn't come with any information. so I've used NirSoft's USBLogView to see which device was being plugged in/out.
- I looked into having a virtual com port with these devices, Osamu Tamura @ Recursion Co started AVR-CDC and its emulating virtual com port, but I didn't get around to test it. Two more source code libraries can be found here and here.
- Official USBasp firmware is written by Thomas Fischl


Sunday, September 24, 2017

ESP32 as Professional Grade/Industry 4.0 Device

According to Forbes, IoT is a $264 Billion market while Industry 4.0 will be $152 billion by 2022, these predictions can lead us to believe in IoT as more than a fleeting trend and should prepare us for the future to come where embedded developers will be plucked from other software development fields, as micro-controllers get more powerful they can also host a more modern programming paradigms and we’ll explore some of these for the sake of making embedded development more robust as well as faster time to market development.

Many words have been written about the advantages and the future that Industry 4.0 devices can achieve for manufacturing, factories, plants and countless ideas have been in the open, hackathons have been performed to find the next killer device, services and applications. This article is not about that, its about how to achieve the fastest way to develop such applications for wide scale deployment.

Industry 4.0 plays a significant role in industrial optimization, by monitoring systems and analyzing the sensor inputs either by statistics or by learning algorithms, failure prediction, load analysis and and optimizations can be achieved. In turn cost cutting and growth are two byproducts that every business needs.

While some of the big time players offer ready made solutions, all the parts have been in the open source community for quite a while, IoT no longer a toy for the geeks but a growing movement to improve a person's life and can be applied in industrial setting.

In my humble opinion, IoT and Industry 4.0 are not there to take jobs but to take the mundane and boring work and allow a better working condition, more interesting and less erosion and by that increase engagement and productivity.

Industrial Usage

While the ESP32 is not a replacement for commercial PLC/SCADA devices, it can achieve that goal pretty fast, sensors and relay modules or SSR modules can make connectivity relatively easy task and if you’re missing a few inputs or outputs, a multiplexer can be a solution as long as your switching speed is within the tolerances.

But what about software?


At this moment, node-red is the most popular IoT control software, it is very easy to use and many tutorials have been done in both video and articles. Other options are Eclipse Kura and Project Flogo.

If your needs are more industrial, a few SCADA open source project:


RapidSCADA - Rapid SCADA is free, open source, full featured SCADA software.

Eclipse NeoSCADA™ is flexible. It is not an out of the box solution, but a set of tools that can be combined in many different ways. It provides development libraries, interface applications, mass configuration tools, front-end and back-end applications.

IndigoSCADA - A small footprint SCADA system entirely developed in C and C++ with multiple OS support and multiple front end protocol drivers.

ScadaBR is free, open source free software for the development of Automation, Data Acquisition and Supervisory Control applications.

SZARP is a full-featured SCADA system designed for monitoring slowly changing industrial processes, for example operations of urban heating systems. It is completely free software, published under terms of the GNU General Public License 2.0 (or any later version).

Why ESP32

While all of the ideas in this article are common to most of the IoT devices with a decent hardware, this article is about ESP32, which is even more suitable than most of the other micro-controllers I’ve encountered lately.

While Espressif is well known in the hobbyist community, it is less used at the professional and industrial sectors, in my opinion not rightfully so.

Espressif ESP32 gives a fair fight against many micro-controllers. Maxim, Texas Instruments, Microchip, Nordic Semiconductor, NXP, Silicon Labs, ST and Atmel all provide microprocessors that can compete, some more popular than others, some provide a free IDE, some provide very expensive SDKs and most of them has a sort of RTOS. What Espressif did right is opening up their SDK and development environments so others like PlatformIO can make development for their hardware accessible, their choice of FreeRTOS allowed them to open source the entire framework and by that help anyone willing to learn about embedded development to get the information, details and insights that only a source code can provide.

Around accessibility to hardware and software a community can be created and in turn the hardware can get support, software, components and drivers like no other, essentially by giving the community esp-idf, design documents and specs, this gift keeps on giving back, volunteers contribute back their time and effort while Epressif can sit, relax and plan their next killer chips.

To give a good example for how not to push a product we can take Intel Curie, Galileo, Joule and Edison, while Intel did make an effort to push their maker chips into the market, most of them were way overpriced and under-performing and in spite of their efforts by giving many many of these devices as gifts, prizes, participating in hackathons and more, their downfall was the lack of documentation and community support. The companies that did try to build a business around Intel maker chips are now left hanging and need to redesign their products.


ESP32 Ecosystem

Some of the components that compose the esp32 ecosystem are:

esp32 microprocessor - A dual core 240Mhz microprocessor, 520kb ram, RTC, ULP, 34 GPIOs, network connectivity and multiple UARTs, I2Cs, I2Ss, SPIs, CANBUS, ADCs and DACs with external SPI EEPROM that can be from 1-16MB.

esp-idf – Espressif made a great effort to build a Software Development Kit (SDK) which exposes most if not all of the hardware’s capabilities.

Networking - WiFi/BT/Ethernet connectivity, lwIP TCP/IP Stack.

Security/Cryptography – Encryption acceleration instructions (AES (128, 192, 256) / SHA (1,256,384,512) Acceleration), Networking encryption, Flash encryption, Secure Boot and JTAG Disabling. As well as mbedtls library (formally known as PolarSSL), OpenSSL API translation and libsodium.

Cloud Ready – both AWS IoT and Azure IoT components exists and tested.

FreeRTOS – A full implementation including Tasks and real-time task scheduler, Timers, Queues, Interrupt handling, critical sections and events.

Modern C++ 11/14 GCC Compiler – Modern C++11 and up offers many advantages over C++98, among them is increased productivity, speed, stability, safer code and portability are the right buzzwords but smart pointers, improved collections, lambdas and threads and more are the most common advantages anyone should start looking into.

Industrial/Professional Grade

By combining all these components, Espressif made sure that esp32 will fit many applications, including industrial and professional grade, but what does it mean to be able to write industrial or professional grade applications?

First, lets define industrial and professional grade applications.

Industrial/Professional Hobbist
Paid by corporations/factories  Paid by private consumers/Free/Open Source
Multiple teams/developers are working for a common goal One or more developers are working on their spare time or with very little budget
Mission-Critical Systems, essential for a business to grow Occasional downtime is expected, nothing extreme will happen if the system is down.
Reliable, Robust application, should prove business value Reliability/Robustness is not guaranteed.
Fault Tolerant/High Availability/Recoverability Repair/Support if any, can be at business hours
Predictable/Repeatable Performance and Timing Performance is not defined and may not have a negative effect
Multi Layered, Discreet Components, Separation of concerns Boundaries are not clear, programming paradigms are flexible.
Queued Work, Interrupt Handling Slow Response and inconsistent results might be acceptable.
Connectivity Essential Connectivity Optional
Higher Costs Medium Costs

Lets get things straight, industrial/enterprise applications do not mean higher quality of code and hobbyist software does not mean lower quality, all it means is that they both have different requirements, for example, a hobbyist application developer might spend more time on user experience and UI, a commercial developer can and probably will spend more time on stability, load testing and making sure the software won't fail when a 30,000 RPM drill is coming at an aluminum block at a very fast speed. different requirements, that's all. More over, these days the boundaries between industrial and hobbyist products are getting more and more fuzzy because there is more awareness.

Now, lets imagine you need to write a program that almost never fail, imagine you need it to be up and running all the time, imagine you need to know what its doing, how its doing, its health, what happened to it when it failed, you need to update it when you find a new problem or find a more efficient way to do something, imagine its as secure as possible.

“a program that almost never fail”

While it is a nice dream, software does fail. But what can we do to solve the most common failures and improve stability?

  • Use Modern Compilers and programming paradigms, object oriented development, RAII, smart pointers, collections, lambdas, anything that can help you get your job done in a safer manner. Raw pointers have been around since C, but smart pointers are safer for use. Prefer dynamic casts and static casts over reinterpret cast and c casts.
  • Tests are essential to a stable application, while running tests for xtensa processors on a PC inside QEMU could get complicated, you can use abstraction layers and mocks to run and test most of your code on your development OS, esp-idf and PlatformIO have testing facilities to ease the pain. Also, as the code is a standard C++ code, you can use cppcheck for static code analysis.
  • Always use source control, even for POCs.
  • Build Automation, Versioning.
  • Crash Dump Analysis/Stack Overflow Detection are part of esp-idf, use them.
  • Logs are important for diagnosing a problem or a program misbehave, while esp-idf provides a logging mechanism which can be used for tracing, its also good to be able to save these messages to files and later transmit for analysis and post mortem or use a network syslog.
  • Enable OTA updates, use versioning, certificate signing and a staging environment that contains all the hardware this firmware will be deployed.

What can we do to make the software more reliable? robust? secure?

  • Input Validation - always verify your inputs, buffer overrun is still one of the major problem that leads to hacking, SQL injection is still not dead.
  • Tests - test your code with invalid input, long WiFi names, external hardware glitches. disconnect power, reconnect power. worse case errors should lead to crashes, not compromise security.
  • Use more advanced paradigms, like queues, events, callbacks, exceptions, micro components, layers, inversion of control, control weaving.
  • Logs should have everything you need to diagnose a problem from remote, you shouldn't fly someone across an ocean just to find out a configuration problem.

What can we do to make the software fault tolerant, highly available and recoverable?

  • Fail Fast - failing fast leads to development finding the problem at development time, never swallow errors, if you must, log them first.
  • Exception Handling - an exception happens at an exceptional state, it should not happen, if it did, find out why. don't use exceptions for non-exceptional errors.
  • Reboot upon failure - esp-idf is configured to reboot on panic, invalid pointers access will lead to reboots, . Stack overflow leads to a reboot.
  • Watchdog - sometimes failure does not mean a crash, sometimes it means that a software no long responds at the designed time, sometimes a task enters an infinite loop and waste all its time on doing nothing. watchdogs and heartbeats are designed to detect these situations, use them.
  • High CPU usage detection - while not part of FreeRTOS failures, if you see a poorly performing software, add it, it's not hard. 
  • Brownout detection - brownout is a state of too low power for the hardware to work properly, the system should reboot on brownout detection. Fault injection could be just another way to hack the hardware.
  • If appropriate, use more than 1 device, perform election to select the better device, if it fails, failover to the redundant device, alternatively use heartbeat to select the master device. An example usage would be a redundant controller on a CCR breathing system.

What can we do the to make the software more predictable? High performance? Accurate Timing?

  • RTOS - RTOSs provide multi tasking capabilities, queues, interrupt handling in a safe way.
  • Multicore, faster microcontroller - if your core is busy all the time, it won't be able to react to changes in time, leading to reduced performance in the best case and dependent hardware crash/accident in the worse case.
  • Use Interrupts and Timers - its the age old Polling vs Push.

A Technical Drill Down

esp-idf is a very interesting SDK, Espressif made a huge effort to make it as easy to use as possible, you can find APIs controlling the hardware, drivers, pthread compatible APIs and C++ stdlib, even glob, regex and tar and miniz, all of this makes programming for the ESP32 relatively easy with many features that makes porting existing code a breeze.

esp-idf includes a few interesting features and components that aids building a more robust application, among them is SPI filesystems, both FAT and SPIFFS (spoiler: FAT is faster), you can use the filesystems to store log and data files, the SPI EEPROM include a wear leveling layer which helps to prolong the amount of writes possible. Settings can be stored in NVS so formatting the filessytem won't affect it, lastly if you need more than the SPI memory, you can always use the SD-Card Library.

Cloud Connectivity is essential to large scale monitoring and control, esp-idf includes AWS IoT connectivity and Microsoft Azure IoT connectivity works as well, TelegramBot API, CoAP and MQTT are available as well.

lwIP is a IPv4/IPv6 TCP/IP Stack which have a few interesting features, among them is DNS/mDNS resolution, SNMP, DHCP, AUTOIP, PPP and L2TP , all of them are the basics of networking, among the things you can do with it are DHCP servers, SNTP, PING, HTTP(s) Client/Server, CoAp Client/Server, OpenSSL Client/Server.

nghttp2 is an http2 client/server.

mbedTLS/libsodium are cryptographic libraries, using them is mostly preference, both are very similar, and have their own set of features and quirks, you should make an informed decision by researching them.

FreeRTOS APIs include Task management, including a scheduler and monitoring, data structrures that incude Queues/Stacks, Ring Buffer, Locking Mechanisms like Semaphores and Mutexes. Interrupt Aware APIs

But free is not always good, free means you can use it without paying but sometimes you need it with support, sometimes you need it with a guarantee, FreeRTOS have a commercial alternative called OpenRTOS, it’s the exact same RTOS just with a different licensing. Another Alternative is the SAFERTOS. For more information you can see a comparison here.

Power Management is an important features for low power applications, the ESP32 have multiple sleep levels with an ULP processor that can reduce power requirements even further.

Debugging – while personally I don’t believe in straight up debugging as its usually reducing the quality of logging and on production you can mostly work with logs and crash dumps, Espressif got OpenOCD with JTAG working. esp-idf comes with a unit test app based on Unity and  If you choose to use PlatformIO, you can use its testing facilities as well.


Other RTOSs are not bad, on the contrary, but SDKs have a very steep learning curve, while getting an example up and running could take anywhere from 5 minutes to a few hours, being proficient in that SDK takes time, learning its quirks and overcoming its limitations takes study, but if you take a look at other RTOSs , most of them have almost the same structure and APIs.

But why use a popular RTOS? What does it mean?
Learning about a device from its datasheet could take a long time, especially those 100 pages datasheets for complex devices, on the other hand, using a library that is already working and tested can take significantly less time.

One of the advantages of being a part of the Arduino community is that its a huge community, so many developers already read the datasheet and wrote drivers and sample codes, some of them are very low quality but like every kind of community, some are very good.

ESP32 can be used two additional RTOS that I know of:

Simba http://simba-os.readthedocs.io

NuttX http://www.nuttx.org/

For more information, you can look at this comparison.


While the ESP32 SDK is Apache License, always check license for other components and libraries when you use them. FreeRTOS have a different license, some LCD display drivers have another etc' etc'

Always consult a license lawyer when using for commercial applications

Rapid Prototyping

SDKs are a great way to explore the device’s capabilities but although C++ these days is very easy to learn, sometimes the developers who do the proof of concept and the exploration come from a different field, electronics, mathematics, physics and biology to name a few and C++ might not be their preferred language, a few other languages have been implemented on the ESP32

- LUA - Lua RTOS is a real-time operating system designed to run on embedded systems, with minimal requirements of FLASH and RAM memory.
- MicroPython - aims to put an implementation of Python 3.x on microcontrollers and small embedded systems.
- Basic - Basic interpreter hidden in esp32 silicon

A Few Interesting Projects

Web Radio - MP3 web radio project.

Camera Demo - With OV7725 camera module.

Alexa Voice Service (AVS) client for ESP32- https://github.com/GOLDELEC/Huan

WebSocket example project - https://github.com/ThomasBarth/WebSockets-on-the-ESP32

WiFiManager - WiFi Connection manager with fallback web configuration portal

libesphttpd is a HTTP server library. It supports integration in projects running under the non-os and FreeRTOS-based SDK. Its core is clean and small, but it provides an extensible architecture with plugins to handle a flash-based compressed read-only filesystem for static files, a tiny template engine, websockets, a captive portal, and more.

CAN Bus Driver

Low Powerdeep-sleep functionality if the ESP32

Console component - create an interactive shell.

ESP32 QEMU - https://github.com/Ebiroll/qemu_esp32/

ADNS3080 - Is a mouse optical flow, while you can find example code at ArduPilot, I couldn't find a source code but its a standard SPI and should be relatively easy to port, here's an example use.

Ethernet Connection - Using LAN8720

Development Environment

Eclipse https://exploreembedded.com/wiki/Setting_up_Eclipse_for_ESP-IDF

VSCode http://docs.platformio.org/en/latest/ide/vscode.html

Visual Studio http://docs.platformio.org/en/latest/ide/visualstudio.html

ATOM http://docs.platformio.org/en/latest/ide/atom.html

ESP32 Books, Tutorials and Samples

Neil Kolban has invested a lot of time writing a book, samples and tutorials and should get a special thank you due to the amount of effort he made.


https://leanpub.com/kolban-ESP32 (https://www.robolinkmarket.com/datasheet/kolban-ESP32.pdf)


Espressif Technical Reference: http://espressif.com/sites/default/files/documentation/esp32_technical_reference_manual_en.pdf

Development Kits

ESP-WROVER-KIT looks very interesting at $47.00, comes with external 4GB RAM, LCD and JTAG .

ESP32 is also available in many flavors and configurations on the cheap:

A Kit, In a simple breadboard form, An alternative breadboard form, With OLED, In an easy way to change the SPI EEPROM, or this.

In theory you can use this chip to expand the EEPROM to 16MB (untested).

With a Battery, With a LoRa module and OLED displayWith Soil, Temperature and humidity sensorsBattery and display in a compact form.

If you need a Flashing Fixture.

Further Reading and References

You can always explore other options and find the best suitable for your needs, you can find a list here (http://www.softwareforenterprise.us/list-of-free-and-open-source-scada-m2m-systems/)


In my opinion 3 things drive the Arduino community, cost, accessibility and open source. While an Enterprise or a Factory might not care so much to spend $10,000 - $15,000 on a development environment, they will start caring when they can easily find software developers that already know the hardware being used, in financial terms, the cost of training a new employee are far greater than their hardware.

While I did not bring any real world examples of using ESP32 for industrial purposes, I think I did bring a lot of information on how to do it, the future is inevitable, many software developers are not familiar with hardware but their knowledge and expertise in an professional software practices is mostly far greater than the average hardware programmer, but different requirements will make that previously clear boundary a thing of the past....
Tags: , ,

Monday, September 4, 2017

Looking To The IoT Future With PlatformIO And ESP32

Looking To The Future With PlatformIO And ESP32 or Why I Think the ESP32+PlatformIO is a game changer.

I've started doing electronics and other technical stuff since I was very little but I've left this hobby in the late 90's. Since then it all went under the radar for me, I would still buy the occasional Elektor magazine or browse the kits section in an electronics shop but it lost my serious attention.

About 10 years ago I've wanted to try building something and so bought the immortal Hakko 936 and a Power Supply from CircuitSpecialists and later made an order of a few transistors and ICs from DigiKey but that project did not really work as I forgot too much and needed to learn too much but without the proper books and it caused me to lose interest again.

At 2012 I've had another idea how to build an all budget house automation system and so found myself browsing AliExpress and it rekindled my love for soldering smells and building things.

So why am I telling you this?

Around that time the Arduino ecosystem started flourishing and everyone could start doing microprocessor work easily and at home and although my first microprocessor was a Microchip PIC, it was around 2005, the IDE sucked, you could do either assembler only or limited C code, I've lost interest pretty fast and I bought about 3 processors for about $25...

MPLAB 6.10
Attractive eh?
At the time assembly was understandable, the thing had about 1-8kb for code and while you can write optimized C code or the compiler can optimize for code size, assembly was the normal thing to do.

So again, why am I telling you this?

Like Steve Ballmer immortal words "Developers Developers Developers!", a good development environment is one of the primary reasons a product can get popular.

While Arduino IDE is a perfect for beginners, you quickly reach the limit of what you can do without a modern IDE, such as Visual Studio, Eclipse or any real code parsing, autocompletion, quickinfo IDE, the fun part is discovering that Atmel AVR (the processor many Arduinos are based on) already has an IDE, Atmel Studio.

Its even better to find out someone has already made the Arduino IDE compatible with Visual Studio IDE, which began to be my favorite since Visual Studio 2013 when the C++ parser was improved tremendously, that plugin is Visual Micro.


But lately I've found myself limited with what I can do in Visual Micro. I've started programming the ESP32, which is my favorite microprocessor at the moment. The more I learn about it, the more I realize this is an enterprise grade, industry 4.0, IIoT, whichever buzzword you want to stick on it, it can probably do it with its dual core, 520kb memory, 4MB flash, Wifi, BT, 18 channel ADC, 4SPI, 2 I2C, 3 UARTs CAN bus, IR, PWM, sleep modes, encryption and cryptographic acceleration. A thing of beauty for $5 module or $8 development board with USB Programmer.

So why am I telling you this?

I've recently decided to take another look at PlatformIO, while their github started at about 2014, I'm only taking interest now since it can probably do what I want it to do, give me access to esp-idf with a decent editor and make the noise go away until I need it (build environment).

I'll be blunt, the IDE is still rough around the edges, Atom is not as fun as Visual Studio, VS Code does not have a C++ parser as good as Visual Studio's and overall development experience is about 7 out of 10 but that thing does its magic and enables me to explore the entire FreeRTOS and esp-idf framework so I'll bear the quirks.

In any case, I've installed the plugin about a week ago, since then I've patched and did a pull request for allowing custom partitions and I've even found a way to measure task CPU usage in FreeRTOS which eluded me in-spite of all my efforts in Visual Micro/Arduino.

Do I recommend it?

Only if you've surpassed the capabilities Visual Micro/Arduino gave you since there's still no replacement for Visual Studio C++ parser in my opinion. and Yes, I did try Eclipse some time ago but was not satisfied with the results so I've abandoned it.

But wait, there's good news, I've found a way to use Visual Studio with PlatformIO. At first I saw issue 543, it did not work so well for me, so I combined it with my own solution and it worked, I now have a project that works with PlatformIO! :-)

PlatformIO Use

Lets continue working on an example of what I could do with PlatformIO that I couldn't do with Visual Micro/Arduino, first lets discuss how projects are built in PlatformIO as far as I could tell in this early stage.

With PlatformIO the do-it-all command is... you guessed it - platformio (or platformio.exe in windows) or pio.

In our case, for esp32 with esp-idf:
platformio init --board esp32dev --project-option "framework=espidf"

Now the project is ready for development, you can open it with VSCode or Atom and start working.

But there are two enhancements we can do right now, first we should edit platformio.ini and add the following line in the end of env:esp32dev so the "platformio device monitor" will not crap out:
monitor_baud = 115200

Then we can create our Visual Studio project (not vscode!):
platformio init --ide visualstudio

But we're not done yet, Visual Studio intellisense is designed to work with Microsoft Visual C++, we'll need to add a file and set it in Project Properties -> NMake -> Forced Includes

At first I've attempted to use the solution in Issue 543, but it did not deliver, so I've modified it to this but its a work in progress, so most of the intellisense is working but there are still some quirks.

#ifndef __GNUC__
#define __GNUC__ 2

#ifndef __STDC__
#define __STDC__

#ifdef _WIN32
#define __attribute__(A) /* do nothing */

#ifdef _MSC_VER#define __asm__(x)
#define __extension__(x)
#define __attribute__(x)
#define __builtin_va_list int
#define __extension__
#define __inline__
#define __builtin_constant_p
#define _Bool bool
typedef int __INTPTR_TYPE__ ;

typedef unsigned int __UINTPTR_TYPE__;

So now that we have a good environment, we can start working.

Custom Partition Table

One of the first features I wanted to try and implement with PlatformIO which I couldn't with Visual Micro is a custom partition table. We add a new partitions_table.csv and follow the instructions.

You will also need to modify sdkconfig.h and update these defines:
#define CONFIG_PARTITION_TABLE_FILENAME "coredump_partitions.csv"
#define CONFIG_PARTITION_TABLE_CUSTOM_FILENAME "coredump_partitions.csv"

In this case I wanted to try a partition table with ota, a fat filesystem and a coredump.

# Name,   Type, SubType, Offset,   Size
# Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs,      data, nvs,     0x9000,  0x5000,
otadata,  data, ota,     0xe000,  0x2000,
app0,     app,  ota_0,   0x10000, 0x140000,
app1,     app,  ota_1,   0x150000,0x140000,
eeprom,   data, 0x99,    0x290000,0x1000,
fat,      data, fat,     0x291000,0x15F000,
coredump, data, coredump,0x3F0000,0x10000

I've wrote a small snippet of code to display the partition table once the esp boots up and now we can upload the code to esp32:

platformio run -t upload

If we're successful, we'll see:

Hard resetting...

==== [SUCCESS] Took 20.79 seconds ====

Now we can monitor esp32:

platformio device monitor

And see the results:

- label           type/subtype   address:size            encrypted
- app0            0x00/0x10      0x00010000:0x00140000   0
- app1            0x00/0x11      0x00150000:0x00140000   0
- nvs             0x01/0x02      0x00009000:0x00005000   0
- otadata         0x01/0x00      0x0000e000:0x00002000   0
- eeprom          0x01/0x99      0x00290000:0x00001000   0
- fat             0x01/0x81      0x00291000:0x0015f000   0
- coredump        0x01/0x03      0x003f0000:0x00010000   0

Great! First success with the esp-idf and PlatformIO!

You can find the sources for this POC here:

Core Dumps

One of the features that an enterprise grade product needs is to be able to analyze crashes, not just the ones that a developer is able to attach a debugger to or the ones that can be reproduced but the ones that seldom happen, the ones that happen only in production, the hard ones.

To be able to do that, Alexey Gerenkov implemented saving the core dumps to a special flash partition, but I wanted to do something a bit different, I wanted to send the core dumps back to my server for monitoring, so whenever the device would start it will go to the core dump partition, check if it not empty, then be able to send it to the server, save it to the SPI fat file system or anything else that might be needed.


I had a good starting point for a crash, I've used this code.

You need to select a partition table with a core dump partition, this is where the previous POC came into play as the PlatformIO esp-idf framework can only use partitions_singleapp.csv as far as I can tell without my modification.

I then started to look for a way to retrieve the core dump partition, you will need to use

I've then used esp_partition_mmap to easily access the partition and check its length.

I think it might be better to check for the start and end markers instead of a "not empty" (0xFF) characte, but its good enough for the POC.

So now we have the start address and length of the core dump, we can do whatever we want with it, dump it to screen as base64, there's also a built in function that does that on crash, but you need to configure it:

Again, this is not what I wanted as there will be no one connected to the device to grab that core dump from the console.

Eventually we can assume we have a core dump, make sure you're uploading the core dump along with the firmware version it happened in otherwise you won't know where it crashed and you'll get the wrong results.

We can now run the command to get the core dump in text form:
espcoredump.py info_corefile -t b64 -c test_crash.txt -g toolchain-xtensa32\bin\xtensa-esp32-elf-gdb.exe ".pioenvs\esp32dev\firmware.elf" 

And the result:
==================== ESP32 CORE DUMP START ====================

================== CURRENT THREAD REGISTERS ===================
pc             0x400d0901       0x400d0901 <recur_func()+73>
lbeg           0x400014fd       1073747197
lend           0x4000150d       1073747213
lcount         0xffffffff       4294967295
sar            0x0      0
ps             0x60f20  397088
threadptr      <unavailable>
br             <unavailable>
scompare1      <unavailable>
acclo          <unavailable>
acchi          <unavailable>
m0             <unavailable>
m1             <unavailable>
m2             <unavailable>
m3             <unavailable>
expstate       <unavailable>
f64r_lo        <unavailable>
f64r_hi        <unavailable>
f64s           <unavailable>
fcr            <unavailable>
fsr            <unavailable>
a0             0x400d08e0       1074596064
a1             0x3ffc7f20       1073512224
a2             0x2      2
a3             0x3f403578       1061172600
a4             0x3ffc7f60       1073512288
a5             0x3ffc271c       1073489692
a6             0x0      0
a7             0x0      0
a8             0x5      5
a9             0xffffffad       -83
a10            0x20     32
a11            0x3ffc656c       1073505644
a12            0x1      1
a13            0x80     128
a14            0x1      1
a15            0x0      0

==================== CURRENT THREAD STACK =====================
#0  0x400d0901 in recur_func () at src\ est_core_dump.cpp:80
#1  0x400d08e0 in recur_func () at src\ est_core_dump.cpp:73
#2  0x400d08e0 in recur_func () at src\ est_core_dump.cpp:73
#3  0x400d0924 in unaligned_ptr_task (pvParameter=0x0) at src\  est_core_dump.cpp:90

======================== THREADS INFO =========================
  Id   Target Id         Frame
  9    process 8         xQueueGenericReceive (xQueue=0x3ffc2ecc, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at .platformio\\packages\\framework-espidf\\components\\freertos\\queue.c:1452
  8    process 7         xQueueGenericReceive (xQueue=0x3ffc2904, pvBuffer=0x0, xTicksToWait=4294967295, xJustPeeking=0) at .platformio\\packages\\framework-espidf\\components\\freertos\\queue.c:1452
  7    process 6         prvTimerTask (pvParameters=0x0) at .platformio\\packages\\framework-espidf\\components\\freertos\     imers.c:445
  6    process 5         failed_assert_task (pvParameter=0x0) at src\   est_core_dump.cpp:103
  5    process 4         0x400d0c12 in init_analysis_task (pvParameter=<optimized out>) at src\ est_core_dump.cpp:130
  4    process 3         bad_ptr_task (pvParameter=0x0) at src\ est_core_dump.cpp:51
  3    process 2         0x400d1838 in esp_vApplicationIdleHook () at .platformio\\packages\\framework-espidf\\components\\esp32\\freertos_hooks.c:52
  2    process 1         0x400d1838 in esp_vApplicationIdleHook () at .platformio\\packages\\framework-espidf\\components\\esp32\\freertos_hooks.c:52
* 1    <main task>       0x400d0901 in recur_func () at src\    est_core_dump.cpp:80

======================= ALL MEMORY REGIONS ========================
Name   Address   Size   Attrs
.rtc.text 0x400c0000 0x0 RW
.iram0.vectors 0x40080000 0x400 R XA
.iram0.text 0x40080400 0x7860 R XA
.dram0.data 0x3ffc0000 0x1d20 RW A
.flash.rodata 0x3f400010 0x3e64 RW A
.flash.text 0x400d0018 0xed20 R XA
.coredump.tasks 0x3ffc6504 0x164 RW
.coredump.tasks 0x3ffc7e60 0x1d4 RW
.coredump.tasks 0x3ffc559c 0x164 RW
.coredump.tasks 0x3ffc5430 0x160 RW
.coredump.tasks 0x3ffc5028 0x164 RW
.coredump.tasks 0x3ffc4eb0 0x16c RW
.coredump.tasks 0x3ffc6398 0x164 RW
.coredump.tasks 0x3ffc76e0 0x14c RW
.coredump.tasks 0x3ffc622c 0x164 RW
.coredump.tasks 0x3ffc6ed0 0x154 RW
.coredump.tasks 0x3ffc8848 0x164 RW
.coredump.tasks 0x3ffc86f0 0x14c RW
.coredump.tasks 0x3ffc6004 0x164 RW
.coredump.tasks 0x3ffc5ea0 0x158 RW
.coredump.tasks 0x3ffc2d60 0x164 RW
.coredump.tasks 0x3ffc2bd0 0x184 RW
.coredump.tasks 0x3ffc3328 0x164 RW
.coredump.tasks 0x3ffc3190 0x18c RW

===================== ESP32 CORE DUMP END =====================

CPU Utilization

This is a bit of a hack, since I would rather not change esp-idf code without commiting the change back to the main repo, I've tried many workarounds, eventually the last nail in the coffin was the fact there's a definition for portGET_RUN_TIME_COUNTER_VALUE in freertos/portmacro.h, making it impossible to override from outside the framework.

So.... I had to modify portmacro.h, a change I plan to ask to push back into esp-idf:

#define portGET_RUN_TIME_COUNTER_VALUE()  xthal_get_ccount()


Then I needed to add the functions for initializing the counter and getting the counter values in sdkconfig.h:

#ifndef __ASSEMBLER__
#ifdef __cplusplus
extern "C"{

#ifdef ESP32
extern unsigned long get_run_time_counter_value();
extern void init_run_time_counter();

#ifdef __cplusplus

#define portGET_RUN_TIME_COUNTER_VALUE() get_run_time_counter_value()
#define portCONFIGURE_TIMER_FOR_RUN_TIME_STATS() init_run_time_counter()


#define configUSE_TRACE_FACILITY 1

(In case you're wondering about the #ifndef __ASSEMBLER__ its because the sdkconfig.h is included in .S files)

I then went ahead and used system_get_time to get the value for the counter as it was the only stable one for now, the others are core dependent so every time a task would switch core due to context switching, the counter value will be different showing skewed results.

I know system_get_time is deprecated but I've yet to find a better function with stable results that won't crash, I've tried gettimeofday but it crashed the device.

In any case, this should not be used in production software according to FreeRTOS.

After running for a few cycles, we get these results:

Tasks currently running:
Printing Stats:
high_load1      502167712        49
medium_load2    15551453         1
display_stats   235900          <1
medium_load1    18987019         1
low_load2       384767          <1
low_load1       422990          <1
IDLE            34476           <1
IDLE            977132          <1
high_load2      505107417        50
Tmr Svc         60              <1
ipc0            340010          <1
ipc1            22441           <1

iterations low: 70 medium: 3156 high: 100096

What I would like you to notice is the amount of iterations each function executed, bear in mind this is an RTOS, so higher priority tasks run first, if the high and medium tasks are not in a wait section such as vTaskDelay, the low priority won't get any time, taskYIELD might not be enough here so make sure you don't have high priority tasks that never release CPU time.


This was a short demonstration of a few things the ESP32 can do in terms of enterprise grade devices, the more I read esp-idf code, the more I realize its potential. 

I think this development board is one of the most fun ones I played with, in the upcoming weeks I'll see if its possible to apply enterprise grade architecture and design with it, I've already implemented a lean dependency injection for it and did a POC controlling it with Azure IoT Hub. 

You must have seen that:

In my opinion ESP32 + FreeRTOS + Azure IoT Hub + PlatformIO is overlooked and has the potential of realizing many Industry 4.0 projects.

To understand the potential, this is a small section from Microsoft Build 2017 First day Keynotes, you will notice what analyzing seemingly naive data can give you in the long run a change in power consumption, vibrations, temperatures can all point to an imminent failure, predicting maintenance, short product life and much much more.

I think the next best step will be to learn about PlatformIO unit testing capabilities.

I'm looking forward to see what espressif will think of next, just imagine the PC we had about 20 years ago is now the size of a coin and can be programmed by anyone.

Tags: , , ,