Animal Tracking and Detection in Natural Environments Using Deep Learning

Do you need to detect and track animals in a video? Have conventional image processing techniques been insufficient for your needs due to difficult or changing video conditions? Deep-learning approaches have taken computer vision by storm and at loopbio we make them available to you, creating state of the art deep-learning based object detectors customized to your scientific needs.

Conventional image processing requires tedious hand tuning of parameters and strong visual differences between the animal and the background, usually resulting in poor solutions. In the meanwhile, deep-learning models learn how to locate the animals when provided with a small fraction of the videos annotated. Our solution empowers comfortable and efficient video annotation, includes in-house improvements to tackle problems like crowding and varying light conditions, and works well even on challenging recordings, like those obtained in natural environments.

This video shows A complete end-to-end demonstration of training and evaluation of a custom animal detector.

Guided Creation of Custom Detectors

Loopy features a simple step-by-step wizard for creating your own object detectors. Without specialist training you can configure your own deep-learning model and start training it on your annotated data.

Select which types of annotated objects Loopy should learn to detect.
See real performance of the detector, as it trains, on different annotated segments of your videos.

Not sure when you have annotated enough videos? Loopy includes sophisticated feedback for evaluating the performance of your custom detector on videos; both quantitative feedback such as the algorithm precision and loss, but also qualitative display of how well it detects the objects in real videos you have annotated. Loopy is careful to follow best-practices by not testing on video segments on which it was trained.

Its simple to explore how different deep-learning settings affect performance.
Quantitative measures of detector performance are also provided to help more experienced users.

Use Your Custom Detector to Track in Videos

Once your model has been trained and you have decided that it's performance is sufficient for your needs, you can use it on your normal videos. Using your detector, Loopy can find all objects in your videos. These results can then be downloaded or analysed in loopy using our analysis and plotting tools.

Generated video showing the detected objects in every frame.
A 2D histogram generated from the position of the detected objects, and the ability to download the raw data.

Tracking Single or Multiple Animals in Videos

Loopy includes powerful single and multiple subject tracking algorithms, with more added all the time. If you are unsure about the correct algorithm to use, Loopy includes a novel 'tracking wizard' which helps set various algorithm parameters to their optimum values.

Tracking of multiple individuals with identity.
One example type of analysis that Loopy offers for tracked data

Scoring and Coding of Subject Behavaiour

Many studies utilise manual coding of videos in order to record the behaviour of subjects in videos. Loopy includes a powerful modules which lets you code behaviour efficiently and flexibly from the comfort of your web browser.

  • Code an unlimited number behaviours from one or more videos
  • Play back videos at 1x - 8x real speed, and single frame step forwards and backwards
  • Configure extensive and customised keyboard shortcuts - you can code an entire video without touching the mouse
  • Supports modifiers for behaviours and multiple coding types (focal, ad libitum, etc)
  • Includes a powerful social scoring module allowing you to code any behaviour as the interactions between subjects
  • Behaviours may be events (single point in time) or durations (have a start and end time)
  • Can score synchronised videos - multiple views from multiple cameras - of the same scene at once
  • Includes analyses such as behavioural ethograms, network diagrams of interactions, and summary statistics
  • Raw coded data can be downloaded in a simple csv format, or h5, for further analysis

Powerful Analyses

Many powerful and peer-reviewed plotting and analyses types are already available within loopy, such as behavioural ethograms per video or subject, robust summary statistics for both social and non-social behaviours, and much more. We are constantly adding more analyses to Loopy, and because it is updated every 30 days, all new features are immediately available to subscribers. If you have an existing analysis in R, Python, or excel, the raw data can be downloaded in a number of formats.

Summary statistics for multiple behaviour types
Behavioural ethograms are available for all coded videos.

Annotation of Subject Positions and Attributes

Deep learning needs training data show the location of objects or object attributes. Our web-based annotation tool lets you collect rich data from videos in a comfortable way. If you are an AI researcher then Loopy could be the perfect solution to your ground-truth video data needs. If you are a scientist who wishes to code spacial attributes from videos then the annotation module of Loopy allows you to do so. If you are intending to use Loopy for training your own object detector then you will use the annotation module for this task. The annotation tool allows you to

  • Annotate points and boxes in videos (polygons coming soon)
  • Assists selecting appropriate temporal resolution to annotate
  • Attach modifiers to all annotated objects (difficulty score, occlusion, custom tags, etc)
  • Guided / predictive box annotation - Loopy minimises the amount of annotation work by sensible and configurable prediction and interpolation of annotated objects between frames
  • Powerful keyboard shortcuts, mouse-free annotation possible
  • Raw annotation data available in industry standard (h5, VOC-XML, etc) formats
  • Annotation statistics, coverage, size, progress are shown

Loopy shows clearly which segments of a video are annotated - important for showing the network a diverse class of examples to learn from.
Multiple objects can be annotated per frame and can differ based on class, ID or attribute.
Annotated data can also be used for immediate analysis elsewhere in the site.

Image Processing and Video Editing

Loopy comes with a number of general purpose image processing algorithms including slow-motion and summary video generation, optical character recognition, color and white-balance correction and much more. These operations can be used to pre-process videos before tracking, or as stand-alone tools.