Real-time Sign Language Interpreter

We are a team of volunteers from PyTorch Scholarship Challenge from Facebook hosted by Udacity creating awesome Deep Learning models.

Dataset

Sign Language Dataset

File Description

• main.py
  • Entry point of the application for detecting sign language gesture
• collect_data.py
  • Entry point of the application for creating new sign language data

Setup

Run the following command to install all required python packages

pip install -r requirements.txt

How to Detect Gesture

Run the following command to install all required python packages

python main.py

Instructions

• Place your hand on the screen around the green boxes as shown on the figure below
• Press 'z' button to start hand detection
• Press 'z' button once again to stop hand detection
• Press ESC button to exit the application

How to Add new image data

See the sign language number gesture in https://github.com/ardamavi/Sign-Language-Digits-Dataset

Run the following command to install all required python packages

python collect_data.py

Instructions

• Press 'z' on your keyboard to start detecting hand
• Press 0-9 on your keyboard to change active label
• Press 'c' on your keyboard to capture a single data
• Press 's' on you keyboard to save all the captured data into file, the file will be located in the current application as X_{datetime}.npy and Y_{datetime}.npy
• Press 'd' on your keyboard to delete last recorded data
• Press ESC button to exit the application

Watch the example of adding new image data on https://youtu.be/lEJL5Xflwjo

References

[1] Kaggle Example Script

[2] Finger Detection

[3] Hand Detection

Notes

We also developed a sign language interpretation android application. Please check our repository on https://github.com/forfireonly/SignLanguage

More about our project

• Pitch Deck

• Landing Page

• Introduction Video

• Demo Desktop Application

• MacOS Application

• Windows Application

• Mobile Application