Skip to content

Realtime VRM Humanoid Avatar Animation using Human Library and BabylonJS - FORK

License

Notifications You must be signed in to change notification settings

BasedAnon/human-bjs-vrm-fork

 
 

Repository files navigation

Human: 3D Detection to VRM Avatar

WIP!!!

  • Input
    Video file
    Live webcam
  • Analyze and track all keypoints in a web worker thread
    Using Human library
    Face, Body, Hands
  • Display
    Video Overlay with keypoints
    VRM Avatar with pan/zoom/rotate functionality

Run


Online

Live Demo

Offline

  • Clone or fork the repository
  • Install dependencies: npm install
  • Start HTTPS server using precompiled code: npm start
  • Navigate to https://localhost:8001

Development Mode

  • Clone or fork the repository
  • Compile code on-demand & start HTTPS server: npm run dev
  • Navigate to https://localhost:8001

Notes

  • All heavy-lifting (machine learning processing) is performed using Human library
  • 2D visualization is performed using built-in draw methods in Human library
  • 3D visualization is performed using BabylonJS

Processing

Process flow is separated into two loops:

  • Process loop
    Runs in web-worker so its not blocking the main thread
    Supports WASM, WebGL and WebGPU backends
    Runs as fast as possible: frame is submited to worker and as soon as results are available a new frame is submitted
  • Render loop
    Runs at fixed 60 FPS
    Draws both 2D canvas overlay and 3D GL mesh
    Uses intepolated (for smoother output) results based on last known processed results from process loop

About

Realtime VRM Humanoid Avatar Animation using Human Library and BabylonJS - FORK

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • TypeScript 97.6%
  • HTML 2.4%