Skip to content

Commit

Permalink
editor comment fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
Swathi Sheshadri authored and Swathi Sheshadri committed Aug 15, 2019
1 parent eb8d8c3 commit 5126773
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions paper/paper.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ bibliography: paper.bib
---

# Summary
Markerless tracking is a crucial experimental requirement for behavioral studies conducted in many species in different environments. A recently developed toolbox called DeepLabCut (DLC) (@Mathis2018) leverages Artificial Neural Network (ANN) based computer vision to make precise markerless tracking possible for scientific experiments. DLC uses a deep convolutional neural network, ResNet(@He2016) pre-trained on ImageNet database(@Imagenet) and adapts it to make it applicable for behavioral tracking tasks. To track complex behaviors such as grasping with object interaction in 3D, experimental setups with multiple cameras have to be developed. Development of such systems can largely benefit from a robust and easy to use camera calibration and 3D reconstruction toolbox. To map features tracked from multiple cameras using DLC to 3D world coordinates there exist OpenCV (@opencv) based implementations (@Nath2019, @Anipose). <br/>
Markerless tracking is a crucial experimental requirement for behavioral studies conducted in many species in different environments. A recently developed toolbox called DeepLabCut (DLC) (@Mathis2018) leverages Artificial Neural Network (ANN) based computer vision to make precise markerless tracking possible for scientific experiments. DLC uses a deep convolutional neural network, ResNet (@He2016) pre-trained on ImageNet database (@Imagenet) and adapts it to make it applicable for behavioral tracking tasks. To track complex behaviors such as grasping with object interaction in 3D, experimental setups with multiple cameras have to be developed. Development of such systems can largely benefit from a robust and easy to use camera calibration and 3D reconstruction toolbox. To map features tracked from multiple cameras using DLC to 3D world coordinates there exist OpenCV (@opencv) based implementations (@Nath2019, @Anipose). <br/>

In comparison to OpenCV, Matlab (The MathWorks Inc., Natick, Massachusetts) provides a feature-rich graphical user interface (GUI) for camera calibration which is an essential step for 3D reconstruction. The GUI provides visual feedback and helps quickly detect and reduce errors during the calibration process. However, existing functions in Matlab cannot be trivially used to reconstruct 3D behavior from 2D tracked data for more than two cameras. Our toolbox pose3d is implemented in Matlab for 3D reconstruction of features tracked in 2D using DLC or any other tracking software. It provides a semi-automated 3D reconstruction workflow that takes users through the entire process of camera calibration, undistortion, triangulation as well as post processing steps such as filtering to reduce outliers. pose3d also allows users to try different pre- and post-processing parameters that can be set by simply editing a configuration file and running the main function of the repository called main_pose3d. In addition, pose3d visualizes the results for every run of the main function and helps perform manual parameter tuning before saving final results. <br/>

Expand All @@ -41,5 +41,6 @@ Demo datasets provided in pose3d were acquired by moving a Rubik’s cube on a t

The key difference between the two examples is as follows. DLC, in addition to 2D tracking provides users with a likelihood value for every tracked feature that informs the users on how confident the network is about the inferred location of that particular feature of interest at any given time point. pose3d makes use of this information by applying a threshold and automatically selecting the cameras that cross this threshold for 3D reconstruction. From the 2D tracked corners we use pose3d to track corners in 3D over 1000 example frames with DLC and 20 with manual annotations. Following this, we reconstruct the edges of the cube and compare it to the standard edge length of a Rubik’s cube (57 mm). In the demo data using DLC based 2D annotations, we obtain on average an error of 1.39 mm in 3D reconstructed edge lengths computed over all 12 edges of the cube across 1000 example frames. For the demo data using manual annotations across 20 frames we obtain an average error of 1.16 mm over all 12 edges computed over 20 manually annoted frames. Furthermore, using ‘all’ mode of triangulation provided significantly better results in both our demo datasets than the other two modes of triangulation (comparison tests for the 3 modes of 3D reconstruction included in the demo functions for reference). <br/>

For further reading on the details of triangulation for 3D reconstruction please refer to our [supporting document](Appendix.pdf)
For further reading on the details of triangulation for 3D reconstruction please refer to our [supporting document](Appendix.pdf) <br/>

# References

0 comments on commit 5126773

Please sign in to comment.