Towards Visually-Guided Neuromorphic Robots

Beobots in Action

We are developing a number of neuromorphic visually-guided behaviors for the Beobot. Please see our publication page for details.

One important aspect of our research is in learning to navigate. To this end, we feed decoded signals from the radio control of the robot into the sound card input of one of the motherboards. This allows the robot to measure a teaching signal sent by human operators helping it in its learning process.

Testing under human control

Here we tested the mechanical stability of the robot under human radio control.

• Beobot Action Video 1 (5.4 MB AVI)
• Beobot Action Video 2 (4.6 MB AVI)

Learning to navigate

Christopher Ackerman developed a nifty piece of software, by which the robot learns to navigate based on a global analysis of the gist of the scene it perceives through its video camera.

• Going to the test track and setup (5.4 MB DivX).
• Training to drive around our running track (8.7 MB DivX). Chris closely follows the robot, which is already operating in autonomous mode, but makes frequent mistakes. Using the radio, Chris corrects the robot's actions as necessary to ensure a smooth ride.
• A bit more traning and we'll be ready (3.8 MB DivX). Chris has trained the robot for about half the circumference of the track and it looks like we are ready to turn the radio control off.
• No hands - the beobot now drives in autonomous mode (2.7 MB DivX).
• Some close-up views of the beobot at rest (6.8 MB DivX).
• Some more close-up views of the beobot at rest (2.0 MB DivX). The beobot constantly learns from comparing what it sees to possible human corrective commands from the radio control.
• Off it goes (13.0 MB DivX). By now, the beobot is far, far away, and this video was taken with very high optical and digital zoom, which explains why the camera seems so unstable (it is quite challenging to film at such high zoom, as the smallest camera motion yields huge shifts in the filmed video).

The interesting aspect of the approach used here is that it makes no assumption about the contents or structure of the images seen by the Beobot. Instead, a generic analysis of the image is performed (using a Fourier transform), converting each frame into a 40-number 'signature'. A two-layer backprop network then learns the appropriate actions to take for given signatures. As an example, let's train the Beobot to navigate along a different path, and under very different lighting conditions (dusk).

• Let's train for a bit (21 MB DivX). Here Chris corrects the Beobot as it makes mistakes while learning about this new (never seen before) environment.
• Much better! (18 MB DivX). After just one run along the path from start to finish, now the Beobot is already making much fewer mistakes while driving back along the path, from finish to start (although it is the first time the Beobot is driving along this path in this direction).
• Autonomous mode (23 MB DivX). The Beobot now is fairly comfortable driving along the path almost entirely on its own.
• Mission accomplished (14 MB DivX). The Beobot is now comfortable at navigating in this difficult environment.