Locust Virtual Reality System

Panoramic Virtual Reality System for Locusts

A motion compensated sphere and panoramic perspective correct VR

LocustVR was jointly funded by The Department of Collective Behavior at MPI-AB and the Centre for the Advanced Study of Collective Behaviour. They approached us to create the worlds first, virtual reality system to study freely walking locusts. Using our experience creating sophisticated and unique VR systems for numerous species, we developed LocustVR.

LocustVR is a combination of a locomotion compensator sphere for freely moving animals, and a perspective correct virtual reality (VR) projection. The system simultaneously tracks the position of the locust as it walks on a large sphere, rotates the sphere to keep the locust always near the center of the arena, and projects a panoramic virtual reality from 3 projectors mounted above.

System and Specifications and Architecture

LocustVR consists of three large subsystems:

Locomotion compensator
Detection, tracking, and closed-loop control using Motif
VR software

Locomotion Compensator

The 600mm diameter sphere was surface treated to increase its friction to make locomotion easier. It is controlled using 2 servo motors with rotary encoders (based on [1] and in conjunction with PhenoSys). The locomotion compensator acts as a spherical treadmill and can compensate for the movement of the subject in all directions - always returning the subject back to the center of the arena. This allows the locust to freely move without escaping from the experimental apparatus.

Oblique view of the system and computer hardware (left rear).

Front view showing sphere mounting support and operator console (right rear).

Detection, Tracking, and Closed-loop Control

Video capture and processing use our Motif software. Two cameras were present, a color camera for observation, and a 100fps NIR camera for tracking. Using Motif realtime processing support a contrast-based tracking method detected the center of mass and the orientation of the locust. After Kalman filtering, a control-law keeps the locust near the center of the sphere by adjusting the motor velocity. Simultaneously, the position of the animal on the sphere and the rotation of the sphere was also fed into the VR system.

Motif controls two cameras, the tracking camera (left) and a color observation camera (right).

Virtual Reality

3 x 120fps projectors project downwards onto a cylindrical projection surface (800mm diameter, 700mm high) and also onto the flat floor which surrounds the sphere. The overlap between projectors was corrected and blended together to achieve uniform brightness. The VR system is based on the FreemoVR engine [2]. The virtual environment can be designed in Blender software and experimental logic is implemented in Python.

Three rigidly mounted projectors oriented downwards. The tracking and observation cameras are visible above.

View of the sphere indicating supporting mounts and X-axis drive wheel (right, upper) .

System Operation

Introducing the Subject

LocustVR includes an air-supported hinged upper frame. This makes it easy to change subjects by simply lifting the lid and placing the locust on top of the sphere. As soon as the locust is detected, motion compensation is enabled and the lid can be closed.

The hinged upper element allows easily introducing new subjects.

A sample virtual world with two vertical posts in front of an infinite chequerboard background.

Sample Experiment

In this video the locust explores a virtual world in an experiment to understand decision making (described in [3]). Locusts are highly attracted to the edges of objects so a virtual world featuring two identical vertical posts against a white background was designed in 3D modelling software. In this paradigm, as the locust approaches the two identical targets, the animal reaches a point at which it 'decides' which post to continue approaching.

By systematically studying the dynamics of this decision process as the relative angular position and the number of posts changes, the researchers hope to understand fundamental processes of how decisions are made.

Video footage graciously provided by NTR. For more examples please see the documentaries from NTR (at 6m15s and 8m28s), Xenus/ARTE (at 24m00s), and PM Wissen (Servus TV).

Coverage in the Media

The LocustVR system has been featured in international media prominently. Have a look at the videos below to see the system in action, and to learn about how LocustVR is used to study collective behavior. Click on the image to be redirected to watch the video, the times where the LocustVR is featured in the video is indicated in parentheses below.

loopbio