ERO Rev. 3.8

ERO sat in her box for a long time (again), this time from July 2016 to Jan. 2024. After the Oct. 2009 milestone Get (empty) Beer of ERO 3.5, there was significant investment of time and money in 2 areas: Image processing Arm Image Processing Motivation: The CMUcam color blob detection scheme was highly unreliable mostly due to the Auto White Balance (AWB) algorithm of the camera firmware. The definition of a 'color' was constantly changing, so recognizing a 'color' was unreliable. Proposed solution: Switch to OpenCV-based image processing on 32-bit Raspberry Pi-like ARM platform. Chose Beaglebone over Raspberry Pi for lower power consumption. Initially BeagleBone White, then upgraded to BeagleBone Black which added HDMI capability for watching the object recognition algorithm on a small LCD monitor. Results: Though OpenCV provides powerful image processing capability and examples, it doesn't make you an image processing expert. Could not get the algorithms to work even as well as the CMUcam method. Arm Motivation: The V1 Arm (SG5-UT) was small and nimble with basic analog servos. It performed beautifully with an empty beer bottle, but lacked the strength to lift a full beer. Proposed solution: Switch to a more powerful arm based on advanced digital servos (actuators) by Dynamixel (Robotis). Initially AX-12A Smart Arm, then Pro Series with dual MX-28 servos for the bicep joint. Results: Arm V2 (AX-12A Smart Arm) was still not strong enough to lift a full beer, and was significantly less 'nimble' than Arm V1. Arm V3 (with MX-28 servos for the bicep joint) could barely lift a full beer, but not from the fully extended position. See Arm Fetching Full Beer video.

Where does that leave us?

• The above summarizes the ERO Rev. 3.6 and 3.7 investments that failed to improve ERO's prime directive performance over the 3.5 baseline. That, among other things, landed ERO in her box for 8 years.
• After the Rip Van Winkle break, I'm motivated to try again.
• Unfortunately, most of ERO's parts and software are horribly out of date, her main processor is now obsolete, and many of the companies that made her parts are out of business.
• Unlike most of my professional career, I've decided not to do a mass upgrade of parts & software to the latest and greatest, but to do what I can with what she has.

ERO v3.8 Progress

1. Image processing
• Upgraded linux development PC to Ubuntu 22.04 LTS and actually managed to install OpenCV 2.4.3 which is ~10 years old.
• Able to continue development of the OpenCV vision application and cross-compile for the BeagleBone Black platform which is running Ubuntu linux 12.04 LTS (12 years old).
• Made great progress on understanding the object detection aglorithms and was able to tweak one of them to work pretty well. ERO can reliably find the fridge, the fridge handle, and one specific type of beer.
• Attempts at an algorithm to generically detect the shape of a can or bottle are a fail so far.
2. Arm
• One advantage of the 8 year break is that cans have become the predominant beer container again. A 12 oz. can of beer weighs almost 8 oz. less than a 12 oz. bottle of beer. ERO's Arm V3 can lift it, even from full extension!
• Discovered that with a full bottle of beer, if ERO starts lifting while sliding it out of the fridge, it does not 'drop' as in the Arm Fetching Full Beer video.
• So the arm objective has been met, she can lift a full bottle or a full can of beer. She just can't tell a can from a bottle yet!
• Added a switch to set beer container type to can or bottle. The switch affects which image processing reference images are loaded in Step 6: "Arm fetch beer", as well as changes to certain arm movements.
• Not able to deposit the beer into a shallow cup on ERO's top deck because the arm can't tuck that low. Longer servo brackets on the elbow joint would allow that to work. Unfortunately, CrustCrawler is out of business so I can't get the extended brackets anymore, and a lengthy search on robotics forums and manufacturers websites came up empty.
• Due to the larger arm, the fridge has to be propped up higher so that it's the right height for ERO's full arm extension.
• Due to the larger vertical distance between the webcam and the gripper, and the horizontal offset of it, ERO has to go in blind for the last 2-4" of an approach.
• Added a complex backdrop image at back of fridge to aid in image detection accuracy of a beer.

Current Status as of May 2024

• Due to moving to a new house that doesn't have a race track painted on the basment floor, the line tracking steps have been removed from the "Get Me A Beer" state machine.
• Due to negative feedback received on the v3.5 video, a new step has been inserted to close the fridge.
• ERO can now detect 2 specific kinds of beer, 1 can and 1 bottle (switch selectable).
• The full "Get Me A Beer - complete" sequence, with the modifications described above, is significantly more repeatable than the v3.5 baseline [see Get (empty) Beer video] -- maybe even 50% repeatable!

Progress & Videos







return to ERO 3 page

return to ERO Home