Hand Gesture Teachable Machine Model w/ Arduino Connection/Soon-to-be Arduino Kinetic Sculpture

So, I started out doing something kind of boring where I was going to teach a model to recognize when I touch my hair (nervous habit of mine) and make an annoying sound. This hopefully would’ve helped me break that habit. However! I started thinking about a project I had started in my physical computing class where I was going to write a hand tracking/gesture recognition program to control an Arduino kinetic sculpture. I am currently using the openCV library for Processing… but Teachable Machine offers up a different approach. My idea was: if I could use TM as my hand gesture recognition model… this project would probably be much simpler on the programming end and then I could focus more on the visual/sculpture. So… I brought my model into p5.js and researched how to get serial output from there. This is where I found P5.serialport, which is  a p5.js library that allows your p5 sketch to communicate with Arduino.

Teachable Machine Model Demonstration

So, this model is pretty buggy. Luckily redoing the teachable machine part is fairly simple, so I will continue to test and refine this until it works properly and easily. I also want to try out some other classes that I didn’t include in this demo. I had made a hand waving and finger gun class but it seemed to mess things up a bit, so I disabled them for this demo.

P5.js Code

Arduino Code

Arduino Circuit Set Up

I have not been able to work on the actual arduino sculpture much yet. So far, I just have the Arduino controls set up to 3 basic servos so I know things are working and its receiving a connection.

Downloading the P5.serialserver

For P5 to communicate with the Arduino, I need the webSocket server (p5.serialcontrol). Unfortunately, I tried to download the GUI from github but my computer does not recognize it as safe software – so that didn’t work.

Then I tried to run the other option of using it without the GUI and running it from my command line… that also didn’t work…

So basically, I cannot go any further with this project until I figure out how to work around my computer blocking me from running/downloading this app. But! I am optimistic!

Also, I apologize for this extremely long post!