SECRET OF CSS

Using AI To Automate A Pinball Machine | by Arhan Jain | Sep, 2022


Would computer vision work?

Photo by Heather McKean on Unsplash

During our senior year of high school, my peers and I attempted to automate an antique pinball machine sitting in our physics teacher’s classroom.

A naïve approach had been used before with a laser to detect the ball and fire the flippers as needed. However, we wanted to take it to the next level with the help of artificial intelligence. With this in mind, there were two main challenges to tackle: being able to detect the pinball and making decisions based on the pinball detection.

0*NcfIpQyRgrXHLWL8
Pinball table computer vision setup diagram

We used a camera mounted from the top of the pinball table to capture the entire playfield at 60 FPS. There were a few problems with this — it was really hard to limit the capture field of view to only the table itself, slight movement of the camera would shift the entire playfield in the frame, and the table was slanted, which captured a non-rectangular playfield.

Rather than creating a more robust hardware solution for the camera rig, we chose to solve all of these problems through software.

Left: Original frame captured with green circles highlighting the tape we added. Right: Processed frame

With the power of OpenCV, we could manipulate what the camera captured however we wanted.

We came up with the idea to place pieces of contrasting colored tape on the four corners of the playfield to use as anchors for cropping the camera capture.

For every frame, we ran OpenCV’s contour detection after applying HSV color thresholding masks for the blue and yellow colors of the tape. From the returned contours, we filtered for shapes with four sides (rectangles) and an area within a specified interval. Our implementation to detect these corners with OpenCV is as follows:

Transforms and crops original capture to a frame with rectangular playfield only

With this, we had a foundation to detect the pinball without all the visual noise around the table and less dependence on the positioning of the camera. The playfield would be processed to take up the full frame if the four corner tape pieces are in the frame.

This was especially important for the decision-making part of the project, as the decision-making models would rely on the ball’s position from the detection algorithm. Without the anchors, the ball’s position would be from different reference points if the camera moved. Training the decision models would rely on all the position data from the same reference point.

In the next part of this series, I’ll cover how we detected the pinball in such a visually colorful and noisy environment.

Our first thoughts were to try a machine learning strategy to detect the pinball position by running every frame through a convolutional neural network; however, there were two problems that we realized in trying this approach — it was computationally expensive, and we would need tons of labor towards labeling data to train the model.

Instead, we began working on a non-machine learning approach. Luckily, OpenCV has very powerful computer vision algorithms implemented that are more computationally efficient than machine learning algorithms. After doing the image preprocessing (blurring, HSV thresholding, and converting to a binary image), we used OpenCV’s cv2.findContours(...) function. This function uses the algorithm proposed in the paper by Satoshi Suzuki and Keiichi Abe, which can be found here.

0*a1LJ5IKGVddrSNB7
1) Gaussian blurred frame 2) HSV-thresholded frame, 3) Detected Contours 4) Ball outlined in red

Out of the many contours detected, we filter them based on an interval of area, maximum perimeter, and maximum number of sides. With this, we could do a decent job detecting the ball when nothing obstructed the camera’s view.

Our final step to get the x and y coordinates of the ball was to get the “center of mass” of our contour. This is pretty simple, thanks to OpenCV’s implementation of cv2.moments(...). You can get the x and y center of mass coordinates with the following code:

m = cv2.moments(c)  # given that c is the contour of the ball

x = int(m['m10'] / m['m00'])
y = int(m['m01'] / m['m00'])



News Credit

%d bloggers like this: