Skip to content

Commit

Permalink
Small formatting changes
Browse files Browse the repository at this point in the history
  • Loading branch information
Swathi Sheshadri authored and Swathi Sheshadri committed Aug 1, 2019
1 parent db5c6d6 commit 9d03738
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 12 deletions.
6 changes: 3 additions & 3 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# 3D reconstrution toolbox for behavior tracked with multiple cameras

## What is pose3d?
pose3D is a toolbox for researchers working on computer vision based behavior tracking. Functions necessary to reconstruct behavioral features tracked from multiple cameras in 2D to 3D are provided with this toolbox. The core of this toolbox is a one shot implementation of triangulation for 3D reconstruction applicable to n cameras. It includes two main modes of 3D reconstruction to be selected based on the design of one's experiment. The first mode uses, 2D data from all n cameras that are used to track the feature in 2D while the second mode selects the best camera pair for every time point and feature of interest. Furthermore, validation of the approach is included in example implementations showcasing the capabilities of the this toolbox. The examples also include illustration of 2D to 3D reconstruction workflow with preprocessing steps such as image undistortion, automatic selection of cameras containing well tracked 2D features and application of suitable temporal filters on the reconstructed 3D trajectories.
## pose3d?
pose3d is implemented in Matlab for 3D reconstruction of features tracked in 2D using DLC or any other tracking software. It provides a semi-automated 3D reconstruction workflow that takes users through the process of camera calibration, undistortion, triangulation as well as post processing steps such as filtering to reduce outliers. In addition to providing an easy to use workflow, the key component in our implementation is the ‘n’ camera triangulation function for 3D reconstruction available for users in three modes. The first mode of triangulation is referred to as ‘all’ and uses 2D data from all ‘n’ cameras that are used to track the feature in 2D. The second mode is referred to as ‘avg’ and performs triangulation over all pairs of cameras in the setup and provides the average over all pairs as the result. The third mode is referred to as ‘best-pair’ and selects the best camera pair for every time point and feature of interest for triangulation.

## Table of Contents
* [Why pose3d?](#Why-pose3d)
Expand All @@ -16,7 +16,7 @@ pose3D is a toolbox for researchers working on computer vision based behavior tr

## Why pose3d?
Behavior tracking in lab environments was recently revolutionized by [DeepLabCut (DLC)](https://github.com/AlexEMG/DeepLabCut): a markerless tracking software.
While DLC provides excellent tools for 2D tracking, scope for development of 3D reconstruction workflows especially for 2D behavior tracking made from more than two cameras still exists. [Anipose](https://github.com/lambdaloop/anipose) and [DLC](https://www.nature.com/articles/s41596-019-0176-0) offer implementations in python using OpenCV package in Python for camera calibration and 3D reconstruction. Camera calibration is a necessary step for 3D reconstruction in which checkboard images acquired from a camera are used to estimate its lens properties such as focal length and principal point. When checker board images are acquired simultaneous from 2 cameras, the position of one camera with respect to the other can also be obtained. In comparison to openCV, Matlab (The MathWorks Inc., Natick, Massachusetts), has a very user-friendly and graphical user interface (GUI) based camera calibration routine which makes camera calibration much easier to perform. recon3D takes advantage of this function in Matlab and provides a toolbox for 3D reconstruction of features tracked using DLC or any other 2D tracking software as illustrated in Figure 1. Given the popularity of Matlab in academia, we believe this toolbox will help make 3D reconstruction of tracked 2D behavior easier for researchers to use.<br/>
While DLC provides excellent tools for 2D tracking, scope for development of 3D reconstruction workflows especially for 2D behavior tracking made from more than two cameras still exists. [Anipose](https://github.com/lambdaloop/anipose) and [DLC](https://www.nature.com/articles/s41596-019-0176-0) offer implementations in python using OpenCV package in Python for camera calibration and 3D reconstruction. Camera calibration is a necessary step for 3D reconstruction in which checkboard images acquired from a camera are used to estimate its lens properties such as focal length and principal point. When checker board images are acquired simultaneous from 2 cameras, the position of one camera with respect to the other can also be obtained. In comparison to openCV, Matlab (The MathWorks Inc., Natick, Massachusetts), has a very user-friendly and graphical user interface (GUI) based camera calibration routine which makes camera calibration much easier to perform. pose3d takes advantage of this function in Matlab and provides a toolbox for 3D reconstruction of features tracked using DLC or any other 2D tracking software as illustrated in Figure 1. Given the popularity of Matlab in academia, we believe this toolbox will help make 3D reconstruction of tracked 2D behavior easier for researchers to use.<br/>

![](ExampleGrasping_2Dto3D_whitebg.gif)<br/>
*Figure 1: Reconstruction in 3D of grasping behavior tracked in 2D using DeepLabCut*
Expand Down
11 changes: 2 additions & 9 deletions paper/paper.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,19 +28,12 @@ bibliography: paper.bib
---

# Summary
Markerless tracking is a crucial experimental requirement for behavioral studies conducted across species in different enviroments. A recently developed toolbox called DeepLabCut (DLC) (@Mathis2018) leverages Artificial Neural Network (ANN) based computer vision to make precise markerless tracking possible for scientific experiments. DLC uses a deep convolutional neural network, ResNet(@He2016) pre-trained on ImageNet database(@Imagenet) and adapts it to make it applicable for behavioral tracking tasks. <br/>
To track complex behaviors such as grasping with object interaction in 3D, experimental setups with multiple cameras have to be developed. Development of such systems can largely benefit from a robust and easy to use camera calibration and 3D reconstruction toolbox. <br/>
To map features tracked from multiple cameras using DLC to 3D world coordinates there exists OpenCV (@opencv) based implementations (@Nath2019). In comparison to OpenCV, Matlab (The MathWorks Inc., Natick, Massachusetts) provides a feature-rich graphical user interface (GUI) for camera calibration: an essential step for 3D reconstruction. The GUI provides visual feedback and helps to quickly detect and reduce errors during the calibration process. However, existing functions in Matlab cannot be trivially used to reconstruct 3D behavior from 2D tracked data for more than two cameras. <br/>

Our toolbox pose3d is implemented in Matlab for 3D reconstruction of features tracked in 2D using DLC or any other tracking software. Our toolbox provides a semi-automated 3D reconstruction workflow that take users through the entire process of camera calibration, undistortion, triangulation as well as some post processing steps such as filtering to reduce outliers. pose3d also allows users to try different pre- and post-processing parameters that can be set by simply editing a configuration file and running the main function of the repository called main_pose3d. In addition, pose3d visualizes the results for every run of the main program and helps perform manual parameter sweeping before deciding on saving results. <br/>
The core of our 3D reconstruction workflow is the ‘n’ camera triangulation function available for users in three modes that users can select from based on the knowledge of their experimental design. The first mode of triangulation is referred to as ‘all’ and uses 2D data from all n cameras that are used to track the feature in 2D. The second mode is referred to as ‘avg’ and performs triangulation over all pairs of cameras in the setup and provides the average over all pairs as the result. The third mode is referred to as ‘best-pair’ and selects the best camera pair for every time point and feature of interest for triangulation. While the first and second modes can be used for 3D reconstruction of features tracked with any software, the third mode is applicable only when 2D features are tracked with DLC.
Furthermore, we provide two demo datasets (details of which are provided in the next paragraph) illustrating the different capabilities of pose3d that users can refer to get a quick introduction to using the toolbox. <br/>
Given the popularity of Matlab in academia and its well documented and easy to use core functions for camera calibration, we believe this toolbox will help make 3D reconstruction of tracked 2D behavior easier for researchers to use. <br/>
Markerless tracking is a crucial experimental requirement for behavioral studies conducted across species in different enviroments. A recently developed toolbox called DeepLabCut (DLC) (@Mathis2018) leverages Artificial Neural Network (ANN) based computer vision to make precise markerless tracking possible for scientific experiments. DLC uses a deep convolutional neural network, ResNet(@He2016) pre-trained on ImageNet database(@Imagenet) and adapts it to make it applicable for behavioral tracking tasks. <br/> To track complex behaviors such as grasping with object interaction in 3D, experimental setups with multiple cameras have to be developed. Development of such systems can largely benefit from a robust and easy to use camera calibration and 3D reconstruction toolbox. <br/> To map features tracked from multiple cameras using DLC to 3D world coordinates there exists OpenCV (@opencv) based implementations (@Nath2019). In comparison to OpenCV, Matlab (The MathWorks Inc., Natick, Massachusetts) provides a feature-rich graphical user interface (GUI) for camera calibration: an essential step for 3D reconstruction. The GUI provides visual feedback and helps to quickly detect and reduce errors during the calibration process. However, existing functions in Matlab cannot be trivially used to reconstruct 3D behavior from 2D tracked data for more than two cameras. <br/>
Our toolbox pose3d is implemented in Matlab for 3D reconstruction of features tracked in 2D using DLC or any other tracking software. It provides a semi-automated 3D reconstruction workflow that take users through the entire process of camera calibration, undistortion, triangulation as well as post processing steps such as filtering to reduce outliers. pose3d also allows users to try different pre- and post-processing parameters that can be set by simply editing a configuration file and running the main function of the repository called main_pose3d. In addition, pose3d visualizes the results for every run of the main program and helps perform manual parameter sweeping before deciding on saving results. <br/> The core of our 3D reconstruction workflow is the ‘n’ camera triangulation function available for users in three modes that users can select from based on the knowledge of their experimental design. The first mode of triangulation is referred to as ‘all’ and uses 2D data from all n cameras that are used to track the feature in 2D. The second mode is referred to as ‘avg’ and performs triangulation over all pairs of cameras in the setup and provides the average over all pairs as the result. The third mode is referred to as ‘best-pair’ and selects the best camera pair for every time point and feature of interest for triangulation. While the first and second modes can be used for 3D reconstruction of features tracked with any software, the third mode is applicable only when 2D features are tracked with DLC.Furthermore, we provide two demo datasets (details of which are provided in the next paragraph) illustrating the different capabilities of pose3d that users can refer to get a quick introduction to using the toolbox. <br/>Given the popularity of Matlab in academia and its well documented and easy to use core functions for camera calibration, we believe this toolbox will help make 3D reconstruction of tracked 2D behavior easier for researchers to use.
# Demo datasets and error measurement in 3D reconstruction
Demo datasets provided in pose3d were acquired by moving a Rubik’s cube on a table along different directions and by recording it simultaneously from a 5-camera tracking system. For the demos, we track the corners of the cube using DLC in the first example and manually in the second example. Manual annotations are used to mimic the usage of pose3d for any other 2D tracking software. <br/>
The key difference between the two examples is as follows. DLC, in addition to providing 2D tracking provides users with a likelihood value for every tracked feature that informs the users on how confident the network is about the inferred location of that particular feature of interest at any given time point. pose3d makes use of this information, to apply a threshold and automatically select the cameras that cross this threshold for 3D reconstruction.
From the 2D tracked corners we use pose3d to track corners in 3D over 1000 example frames with DLC and 20 with manual annotations. Following this, we reconstruct the edges of the cube and compare it to the standard edge length of Rubik’s cube (57 mm). In the demo data using DLC based 2D annotations, we obtain an on average error of 1.39 mm in 3D reconstructed edge lengths computed over all 12 edges of the cube across 1000 example frames. For the demo data using manual annotations across 20 frames we obtain an on average error of 1.16 mm. Furthermore, using ‘all’ mode of triangulation provided significantly better results in both our demo datasets than the other two modes of triangulation (tests included in the demo functions). <br/>
For further reading on details of triangulation for 3D reconstruction please refer to our <br/>
[supporting document](Appendix.pdf)

# References

0 comments on commit 9d03738

Please sign in to comment.