Saturday, March 4, 2017

OpenCV Node js Bindings - Background Subtraction Demo - on Raspberry PI !

Finally, the arm build is working, some of the problems I had to overcome, well? waiting for Raspberry PI to compile, why this computer is so popular when RK3288 is available I will never know. I've ordered a few RK3288 to accelerate my work on arm, meanwhile I think my RK3188 will be faster, I'll update as soon as I can.

Another problem is some issues I've had with file names, while working on the linux port, the typescript import file names were case insensitive and this time they were.

Lastly, different version of gcc than the linux test machine, so c++14 wasn't the default, I've added it to the build, hopefully there won't be any issues now.

In any case, this demo is for background subtraction, on arm, about 3-4 fps for two background subtraction algorithms, CPU  is about 25-30%, mostly single core, in the video its higher because ffmpeg attempts to capture and compress the screen capture in the background, the x11 took some more CPU.

Not bad but if you're planning to run OpenCV on Raspberry PI, you'll probably need to do some serious optimizations by resizing or changing algorithm parameters.

Saturday, February 25, 2017

Node js C++ Addon Overload Resolution - v1.0.0

Module compiles and all tests are passing cross platform, windows/linux/arm (raspberry pi) were tested!

Lessons learned:
- gcc is a lot less forgiving than visual c++, cryptic error messages when there are circular references between header files.
- gcc 6 is missing some minor c++17 implementations (std::size for example).
- c++ 11 (and up) - windows header includes some headers by default, in gcc you have to include them explicitly, like cmath.
- node-gyp adds c++11 compilation flag which kills gcc's ability to compile c++ 14. it can be removed but why was it even there...?

edit 2017-02-26:
- gcc multi-line comment warning makes sense when you realize the backslash at the end ( \ ) might be used to comment the line following the comment.
- gcc doesn't like extra (and useless) typename keyword
- gcc doesn't like you not using typename where it should be clear its a type definition. 
- gcc doesn't like to mix definitions from headers and code files. 

and many more... 

I guess gcc makes you write better code :-)

OpenCV Node js Bindings - HOG Demo

This is a simple demo showing that the addon is starting to work, many of the basic objects are implemented and working, including Rect, Point, Matrix, and more.

The HOG demo is actually a ported version of pedestrian detection demo from opencv_extras, since the CPU version of VideoCapture only plays mpeg files, I've modified utility.ts (which is ported from utility.h/cpp) to use my own ffmpeg bindings instead.

The porting work I've done attempts to include the GPU APIs as well, but until I'll implement cuda/GPU work in node-alvision, its only a stub, but the CPU version works as well as I could see, the code runs pretty slow since I've tested it in in debug mode while developing, but it should run faster in release and these APIs could run in node js threadloop, so in theory it could be a lot faster, but it remains to be seen if it will be.

ported typescript file is pedestrian_detection.ts

Saturday, February 18, 2017

Node js C++ Addon Overload Resolution - v0.7.0

Another minor release but with a major new type!

- AsyncCallback - this is a new type the module exposes, it allows a stand alone thread to do callbacks into v8 by queuing the call data and signalling libuv to execute it next time the event loop executes, by far the most useful feature of this release!

Note that there is an option to change the callback from weak to strong and by that making node js wait until the callback is destroyed before exiting making it useful in many situations.

- expose get_type to classes outside the node-overload resolution - while its very beneficial to hide most of the type information from classes using this module, in some cases such as property accessors which don't pass through the module but still need to make sure the data types passed to them is the one expected, get_type exposes this functionality.

- fix array convertible checks - when using arrays, the old method wasn't checking the correct conversion is possible from array types to other array types. while it is possible to convert array to number (in JavaScript anything is possible ;-) ), it should not be considered as a valid convertible but now it is also allowed to convert an array of derived to array of base.

Saturday, February 4, 2017

Node js C++ Addon Overload Resolution - v0.5.0

After being stuck with a few tests for node-alvision, I've decided to see why they take so much time.

From what I could gather while doing performance analysis, the major bottlenecks were the functions that determine the appropriate overload in general and the ones that do the actual type analysis and convertible checks.

Doing type analysis in v8 is not so straight forward as it seems, numbers are convertible to strings and back, almost anything can be converted to boolean and on top of that, checking if a certain object belongs to a C++ class also needs some work.

What the POC did is go over all the registered types and then determine which type the class belongs to but that can be shortened to checking only the names of the prototype chain, so if we have a type system of 230 types give or take, it will go over 3-5 the most after the change.

Now, if we have a function with 10 parameters and with 10 overloads, it would go over 100 type checks the most. so I've took out the arguments checks out of the overload matching function and created another object called function_arguments, which queried the argument type only once instead of multiple times and returned a cached result.

Another performance improvement is the array type checks, which went over the entire array and checked each element type, instead of it, I've decided to make it a little less reliable for the sake of performance, so now it will check 10 the most, so if we have a 1000 items in the array, it will skip every 100 items and check only their types.

So all of these are actually a sort of type system, which I refactored it into a new type_system class.

While it did boost the performance, I wasn't pleased with the boost, so instead of waiting about a minute for results, it would take only 20 seconds.

I went ahead and analyzed the bottlenecks further and found it that while the type checks themselves are no longer an issue, going over 10-20 overload for certain functions is still time consuming and after doing it once, if I know the argument types, there's no reason to do it again for the same function/classes.

This is where function_rank_cache class came into play, it caches the correct function given the same conditions apply, same class/function and argument types.

This improved the performance greatly and now I'm left with about 5 seconds of actual testing time for the said test.

All of these improvements were made in Debug, compiling it as Release improved things beyond my expectations for the moment.

In the end, what helped the most is doing only the work necessary :-)

Monday, January 30, 2017

OpenCV Node js Bindings - Progress

I've decided to try and take a break today from implementing APIs, optimizing the node-overload-resolution and trying to execute OpenCV tests and just see if the fruits of my labor are actually working.

So I've started with this piece of code:

 var m = new alvision.Mat(500, 500, alvision.MatrixType.CV_8SC3);, new alvision.Point(250, 250), 100, new alvision.Scalar(0, 0, 255), 3, alvision.LineTypes.FILLED);  
 alvision.imshow("test window", m);  

Seems simple enough no?

but then I've discovered that I didn't implement circle yet:, new alvision.Point(250, 250), 100, new alvision.Scalar(0, 0, 255), 3, alvision.LineTypes.FILLED);
Error: error executing +circle not implemented
    at Error (native)
    at Object. (node-alvision\test\tmp.ts:73:10)
    at Module._compile (module.js:556:32)
    at Module.m._compile (node-alvision\node_modules\ts-node\src\index.ts:299:25)
    at Module._extensions..js (module.js:565:10)
    at Object.require.extensions.(anonymous function) [as .ts] (node-alvision\node_modules\ts-node\src\index.ts:302:14)
    at Module.load (module.js:473:32)
    at tryModuleLoad (module.js:432:12)
    at Function.Module._load (module.js:424:3)
    at Function.Module.runMain (module.js:590:10)

but now that I got the overload resolution working, I've added these lines:

  auto img<IOArray*>(0)->GetInputOutputArray();
  auto center<Point*>(1)->_point;
  auto radius<int>(2);
  auto color<Scalar*>(3)->_scalar;
  auto thickness<int>(4);
  auto lineType<int>(5);
  auto shift<int>(6);
  cv::circle(img, *center, radius, *color, thickness, lineType, shift);


Took about 20 seconds including compile time. 

And the result:

So far so good!

Friday, January 13, 2017

Node OpenCV Addon

I'm glad to finally say I have a good start as far as I can tell for implementing a good OpenCV addon for node js.

Its not a sprint but a marathon, OpenCV is one of the largest, most complex libraries I had to deal with and I've tested many ways to implement it. finally found a sound way to actually do it.

The adventure started some time ago when I attempted to use a few functions to display an augmented reality diffusion tensor file on a marker with a tablet (I got some help from Frank from DSI Studio, Thanks Frank!), it later became a challenge as I learned more and more about C++, Node js and OpenCV.

The first few things I've tried was using ffmpeg api in node js, ffmpeg is written in C, which lacks automatic object lifetime management (constructors/destructors) but contains allocators where you need to explicitly free unused objects and buffers.

I've decided the best way was to try and implement a C++ wrapper on top of these APIs. it was challenging to find the appropriate lifecycle from the documentation but reading the C code helped a lot, eventually I've completed a C++ wrapper which could access most of ffmpeg API with an intuitive API.

The adventure didn't end there, I've used OpenCV Matrix (cv::Mat) as the base container for both image and audio frames which allowed me to explore the OpenCV API with both video and passthrough audio but with a possibility to eventually process the audio as well since its easily accessible.

I've started to explore the OpenCV API and I really liked the idea to manipulate the video frames with Node js, so I decided to implement a few of OpenCV APIs but then I thought how hard would it be to implement the entire OpenCV API in node js, so I went ahead and learned about Node Addons, V8 and NAN.

Many people think that Javascript is not suitable for processing Video/Audio but the truth of the matter is that Javascript is not really processing anything, its just a scripting engine. The real magic stays in C/C++ domain and the delay Node js/v8 adds is negligible when you see how much time compressing a frame or executing canny on it takes, those microseconds do add up, but in terms of percentage, it should be nearly invisible in the total program execution time, besides, this is more for fun than for science.

But... Javascript is not entirely suitable for scripting complex APIs and if you add the number of function overloads possible in OpenCV, it will probably be hell to use not to talk about development, this made me think in the Typescript direction, which can both help the intellisense and make development easier since it enforces some type checking at development time and that could eventually be used to make the API a lot more readable and intuitive.

A few things made developing the API not the most fun thing in the world, for example, the way v8 exposes its internal objects, everything looks like an object, all numbers are actually a double and there is no possibility to overload function implementation internally.

These reasons and more made me think and develop node-overload-resolution project, which addressed most of these problems by allowing me to add overloads to functions, add parameter validations, automatic conversion between v8 data types and C++ data types, automatically execute function implementations in the libuv threadpool as async functions and handling return values and exceptions without explicitly writing a single line of code in the addon itself.

So why am I writing all of this today?

Today the first test passed, not my tests, not all the tests, but the first OpenCV test which I ported to typescript to make sure the API is working properly.

Hopefully its not the last :-)

But let me be completely honest, it is by no way or shape ready for anything, the amount of APIs implemented are very little, even the build at the moment is complex and takes anywhere between 30 minutes to an hour. Since I've created the opencv_ts branch I didn't even check if the code compiles on linux, which it did before, on linux32/64 and even arm and NDK, sorry linux guys, in terms of IDE for C++, I have yet to see anything which comes close to Visual Studio ;-)