Robolocust
From ILabWiki
| Table of contents |
Executive Summary
Project Goal: Use locusts as a bio-sensor to detect impending collisions.
Here is a quick status report on where we are at present and a plan of where we expect (or would like) to be in the coming few weeks. Later sections provide more details on the motivations behind the project, links to similar research, etc.
Current Status
- The software framework for "plugging and playing" with different LGMD models and integration algorithms is largely in place.
- The LGMD model described by (Stafford, et al. 2007) has been implemented.
- We have a stub in place for the LGMD model by (Gabbiani, et al. 1999)
- We have implemented a particle filter for combining multiple LGMD's.
- The hardware framework for the mobile robot is also largely ready.
- The R/C car platform along with the mini-ITX box running Linux and the INVT toolkit is in place. Three FireWire cameras and a laser range scanner have also been mounted.
- A microcontroller for talking to the R/C car's motor driver is working.
Demo Videos and Interim Report
- Basic motion tracking: (http://ilab.usc.edu/~mviswana/robolocust/demo-tracking.mpeg) This video illustrates how the LGMD model tracks motion (red bars). The "danger curve" (cyan curve at bottom) should not respond too much as we walk by.
- Hand-waving: (http://ilab.usc.edu/~mviswana/robolocust/demo-handwaving.mpeg) We expect to see the danger curve go up and down as we wave our hands near the cameras and bring them in towards the cameras.
- Fast collisions: (http://ilab.usc.edu/~mviswana/robolocust/demo-fastcoll.mpeg) The danger curve stays low (for the most part) as we come in but shoots up as we try to grab/hit the cameras.
- Slow collisions: (http://ilab.usc.edu/~mviswana/robolocust/demo-slowcoll.mpeg) The LGMD spikes and danger curve both remain depressed if we come in very slowly. This is a failure mode of the LGMD-based collision sensor.
- CSCI-547 Project Report: (http://ilab.usc.edu/~mviswana/robolocust/cs547-report.pdf) This class paper details our particle filter and state estimation for the above demos. CSCI-547 (http://asimov.usc.edu/~gaurav/CS547/) is a course on Probabilistic Robotics.
Next Steps
- Done: We already have far too many parameters that need to be tweaked! Before things get any worse, we should add support for reading a config file so that we don't have to constantly hack code and rebuild. This can be implemented with a central (singleton) config "database" that all other objects in the system can consult as required.
- Tuesday, Feb 3rd 2009: Add support for reading input frames from AVI and MPEG files. This will allow us to test the spike train generators with artificial stimuli (e.g., black box moving in/across FOV on a white background).
- Wednesday, Feb 11th 2009: Implement the (Gabbiani, et al. 1997) model for generating spikes and test with artificial as well as real data. The image processing for this will require some kind of optic flow. Some possibilities include the Green's theorem based stuff described in (Poggio, et al. 1991) and the log-polar stuff from (Tistarelli and Sandini, 1993).
- Thursday, Feb 19th 2009: Build Reichart compound eye for avoidance using multiple LGMD spikes using either the (Gabbiani, et al. 1997) or (Stafford, et al. 2007) models. This one should work with both whereas the particle filter based integration algorithm will probably only work for the Gabbiani model (see the CSCI-547 Project Report (http://ilab.usc.edu/~mviswana/robolocust/cs547-report.pdf)) for more details on the particle filter state estimation and why it might not work with the Stafford model).
- Friday, Feb 27th 2009: Install new motor driver, encoder and other necessary paraphernalia on robot to try and slow it down and get better control over its motion. Right now, the R/C car has two speed settings, viz., faster and fastest. Letting it loose with over $3,000 worth of equipment on board seems like a bad idea...
- Friday, Feb 27th 2009: Figure out how to interface with the laser range finder and write the glue code to make this work.
- Thursday, Mar 5th 2009: Write and test basic behaviours for moving robot and avoiding/stopping with the laser range finder. This will serve as an emergency shutoff system that will (hopefully) prevent disastrous collisions in case the locust-based avoidance turns out to be a bust.
- Tuesday, Mar 10th 2009: Implement a DAMN (or DAMN-like) arbiter for allowing active behaviours to vote on all possible actions and move the robot based on the results of the voting process.
- Friday, Mar 13th 2009: Insert the locust/vision modules into the sensorimotor loop by wrapping them into a suitable behaviour. This would include the locust models and integration algorithms all packaged into a behaviour that control the robot based on the camera inputs and artificially generated spike trains.
- Put reinforcement (or other) learning into the mix to get best results (whatever that means)...
Robolocust Motivations and Challenges
The goal of this project is to use several locusts as a bio-sensor for a mobile robot that detects impending collisions and then steers the robot away safely.
The reason for our interest in using locusts is that they possess a visual interneuron known as the Lobula Giant Movement Detector (LGMD) that responds preferentially to objects on a collision course. As its name suggests, the LGMD is a fairly big neuron. This makes it relatively easier to isolate for the purpose of taking electrophysiological recordings.
Unfortunately, despite the attractiveness of the LGMD as a collision detector, its applicability to mobile robots is limited by the fact that these neurons fire at their peak rates extremely close to the time-to-collision (in the order of 200ms). Locusts can get by just fine with such short warnings because, when they jump to evade predators and other approaching objects, their musculoskeletal systems are capable of inducing accelerations up to 17 gees. Needless to say, a mobile robot would be rather hard-put to respond similarly.
Another problem that needs to be overcome in order to successfully use the LGMD is that we need several of them to be able to make disriminative judgments regarding the approach vectors of oncoming obstacles. A locust does not make very fine distinctions about where something is coming from; when its LGMD fires, it simply jumps. In the case of a mobile robot, we need to know with greater accuracy where an obstacle (or obstacles) lies so that the controller may steer the robot in a suitable direction.
This means that we need, not just one locust, but several, with each looking in a different direction. Thus, the second challenge for using the LGMD as a collision detector for a robot involves combining the spike trains streaming in from multiple LGMD's.
Overall Plan
Before we mount actual locusts onto our robot, we would like to first test our ideas on how to combine the multiple signals to yield a good steering angle. To this end, we are developing a software framework that allows us to use different LGMD models and integration algorithms simply by supplying appropriate command-line options.
In addition to the software simulation framework, we also want a hardware simulator that will enable testing on a real robot once we are confident about the different algorithms our software supports. To enable this, we have an R/C car onto which we have mounted three FireWire cameras, a laser range scanner, a mini-ITX box running Linux and the INVT toolkit (http://ilab.usc.edu/toolkit/), and some motor driver interface hardware plus other paraphernalia such as a wireless network card.
Once we have satisfied ourselves that we have a good control policy on hand, we will work with our collaborators, the locust electrophysiology experts (http://glab.bcm.tmc.edu/), to do the actual integration between the live animals and our robot.
Long-term Vision
Eventually, we want to be able to culture the LGMD's and other neurons making up the locust's visual system and hook just this little clump of cells to a robot instead of having to use the entire animal.
Extending this even further, we would like to investigate other fruitful integrations between biological subsystems and robotic hardware and software. For example, instead of DC motors, can we grow muscles and use them as actuators instead? What might the limits and limitations of such techniques be?
Design Notes
Software Architecture
All the images streaming in from the cameras are tiled to create a single panaromic image.* This composite image is then split into several vertical strips as described in (Michels, et al. 2005).
Each such strip corresponds to a possible steering angle and also to a virtual locust. The idea here is to simulate a population of locusts looking in multiple directions. Each virtual locust "monitors" its "main" strip and also the ones immediately to the left and right. This allows us to simulate multiple locusts with overlapping fields of view.
Each virtual locust reads its assigned strips and generates an LGMD spike train by implementing an LGMD model such as the one by (Stafford, et al. 2007). These spikes will correspond to motion, especially that indicating collisions, within the FOV. The integration algorithm will combine the LGMD spike trains from multiple locusts into a reasonable steering angle or other control action.
At present (circa November 2008), we have two main ideas for combining multiple LGMD signals. The one that is currently being implemented is a Bayesian state estimator. It works (or rather, will work) as follows: to each vertical strip, we assign a suitable likelihood that there is an approaching obstacle given the current LGMD reading by applying a sensor model to a the current belief. The sensor model measures the causal probabilities, i.e., the likelihood of getting a particular reading (or range of readings) given that there is an obstacle approaching from various distances. These causal probabilities are (i.e., will be) obtained empirically.
The second approach involves building a model of an insectival compound eye. Thus, the LGMD signal from each strip will be propagated with appropriate time delays through a neural network that will produce a vector sum indicating the overall magnitude and direction of spiking activity, which will tell us where approaching obstacles are.
Hardware
Presentations
Here are some presentations I have made on this project:
- August 22nd 2008: (http://ilab.usc.edu/~mviswana/robolocust/ilab-lunch-20080822.pdf) Robolocust intro and discussion of (Guest and Gray, 2006).
- October 10th 2008: (http://ilab.usc.edu/~mviswana/robolocust/ilab-lunch-20081010.pdf) Discussion of (Michels, et al. 2005).
References
Gabbiani, F., Krapp, H. G., Laurent, G. "Computation of object approach by a wide-field motion sensitive neuron." J. Neuroscience, 19(3):1122–1141, 1999.
Guest, B. B., Gray, J. R. "Responses of a Looming-Sensitive Neuron to Compound and Paired Object Approaches." J. Neurophysiology, 95:1428–1441, November 2006.
Michels, J., Saxena, A., Ng, A. Y. "High Speed Obstacle Avoidance Using Monocular Vision and Reinforcement Learning." Proc. 21st Int'l Conference on Machine Learning, 2005.
Poggio, T., Verri, A., Torre, V. "Green theorems and qualitative properties of the optical flow." Technical report, MIT, Internal Lab. Memo 1289, 1991.
Stafford, R., Santer, R. D., Rind, F. C. "A bio-inspired visual collision detection mechanism for cars: combining insect inspired neurons to create a robust system." BioSystems, 87:162–169, 2007.
Tistarelli, M., Sandini G. "On the advantages of Polar and Log-polar Mapping for Direct Estimation of Time-to-Impact from Optical Flow." IEEE PAMI, Aplir 1993.
Notes
A very crude panaromic image because we don't really stitch images together based on any keypoint matching, etc. Instead, we just take all the input images and put them together side-by-side. This is done to make it easy for the rest of the framework to access the input stream without having to worry about the details of exactly which camera or video source the images are coming in from.
