Skip to content

A browser extension for using and developing the web handsfree, powered by Handsfree.js

License

Notifications You must be signed in to change notification settings

Yonet/handsfree-browser

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

48 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

The Handsfree Browser Extension

🚨 This project useable but not officially released yet

This Handsfree.js powered browser extension is a designed to help you browse the web handsfree through face and/or hand gestures. The aspiration is to develop a "User Script Manager" like Tampermonkey and Greasemonkey, but for handsfree-ifying web pages, web games, web apps, and any other type of content found online.

This browser extension also includes handsfree Dev Tools, primarily to assist in handsfree WebXR development. In short, this extension is designed to help you use your entire browser and web totally handsfree!








Handsfree Browsing

By default, each page will get a "Face Pointer" or a set of "Palm Pointers" for you to browse pages handsfree.

However, in addition to the pointers you can add custom handsfree interactions. For example, here's a demonstration of handsfree-ifying Jason Webb's Reaction Diffusion Playgound where each individual finger pinch does something different:








Handsfree WebXR DevTools

This project is a started as a fork of the Mozilla WebXR Emulator Extension. This section of the browser extension was initiated from conversations with Alex Ionkov.

Currently the DevTools helps you emulate WebXR headsets with basic controller support.








Development Guide

Folder structure

Each of the files are located in their respective context folders in /src/. Handsfree.js specific scripts are named handsfree.js and the Mozilla WebXR specific ones are labeled webxr.js. Other than organizing the files the WebXR code is mostly untouched.

How it works

  • When you first install the extension, /src/background/handsfree.js checks if you've approved the webcam. If not, then it'll open the options page in src/options/stream-capture.html
  • The popup panel has a "Start/Stop Webcam" button that communicates with the background script to start the webcam: /src/popup/index.html
  • The background page is where the models are stored and run. This keeps everything isolated and only asks for webcam permission once (vs on every domain): /src/background/handsfree.js
  • The background page also uses the "Picture in Picture" API to "pop the webcam" out of the browser. It renders the webcam feed and debug canvases into a single canvas, and uses that as the srcObject to a separate video element which is the PiP'ed

How to install

  • Chrome: Install as an unpacked chrome extension. Visit chrome://extensions and enable Developer Mode on the top right, then click Load unpacked and select this project's root folder

  • Firefox: Coming soon, I switched computers and didn't realize I continued developing on Chrome 😆







Acknowledgemetns

About

A browser extension for using and developing the web handsfree, powered by Handsfree.js

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages