Deep Dive into WebXR Hand Tracking: Building Physics-Based Virtual Hand Interactions

Virtual Reality (VR) interfaces are rapidly shifting from handheld plastic controllers to natural, controller-less hand tracking. The browser-native WebXR Device API exposes highly detailed hand skeleton data directly to JavaScript, tracking 25 individual joints (joints, knuckles, tips) per hand.

However, rendering a 3D hand model on the screen is only half the battle. If your virtual hand passes straight through tables, walls, and objects like a ghost, the illusion of immersion collapses.

To build tactile spatial interfaces, you must map the hand's tracking skeleton to a physics simulation engine.

In this guide, we'll design a physics-based hand mapping pipeline using Three.js and Rapier.js (the high-performance Rust-compiled WASM physics engine) to enable realistic grabbing, pushing, and physical button presses.

⚡ 1. The Physics Mapping Pipeline

In a standard game engine, hand controllers are treated as kinematic rigid bodies. In WebXR, the browser updates joint coordinates every frame based on camera observations, but does not apply forces.

To make hands interact physically:

1. 2.
   Read WebXR Joint States: Every frame, request joint poses (positions and orientations) from the WebXR frame context.
2. 4.
   Translate to Kinematic Colliders: Spawn 25 small sphere colliders in Rapier.js corresponding to each hand joint.
3. 6.
   Sync Positions via Velocities: Instead of teleporting the physics colliders (which breaks physics collisions), calculate the velocity vector required to move the collider from its current physics position to the new WebXR target position.
4. 8.
   Resolve Collisions: The physics engine automatically computes contact forces against active dynamic bodies (like blocks, buttons, or levers).

[WebXR Frame Poses] ──> [Calculate Joint Velocities]
                                    │
                       [Apply to Kinematic Colliders]
                                    │
                        [Rapier.js Physics Step]
                                    │
  [Dynamic Body Collisions] <───────┴────────> [Update 3D Meshes]

🏗️ 2. Coding the XR Hand Tracker

Let's initialize our WebXR session with hand tracking requested, and loop through the skeleton structure.

javascript
import * as THREE from 'three';
import { ARButton } from 'three/addons/webxr/ARButton.js';

let renderer, scene, camera;
let hand1, hand2;

function initXRApp() {
  scene = new THREE.Scene();
  camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 20);

  renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
  renderer.setSize(window.innerWidth, window.innerHeight);
  renderer.xr.enabled = true;
  document.body.appendChild(renderer.domElement);

  document.body.appendChild(ARButton.createButton(renderer, {
    optionalFeatures: ['hand-tracking', 'physics']
  }));

  // Request hand instances from the WebXR manager
  hand1 = renderer.xr.getHand(0);
  hand2 = renderer.xr.getHand(1);

  scene.add(hand1);
  scene.add(hand2);

  setupHandMeshes(hand1);
  setupHandMeshes(hand2);
}

💻 3. Integrating Rapier.js Physics Colliders

Now, let's write the bridging logic to map the 25 joints of an XRHand into kinematic rigid bodies inside Rapier.js.

javascript
import RAPIER from '@dimforge/rapier3d-compat';

let physicsWorld;
const jointColliders = new Map(); // Joint Name -> Rapier Collider

async function initPhysics() {
  await RAPIER.init();
  physicsWorld = new RAPIER.World(new RAPIER.Vector3(0.0, -9.81, 0.0));
}

// Map key joint indices to physical spheres
const trackedJointIndices = [
  0,  // Wrist
  2, 3, 4, 5,       // Thumb
  6, 7, 8, 9,       // Index Finger
  10, 11, 12, 13,   // Middle Finger
  14, 15, 16, 17,   // Ring Finger
  18, 19, 20, 21    // Pinky
];

function setupHandPhysics(hand) {
  hand.addEventListener('connected', (event) => {
    const xrHand = event.data.hand;
    
    trackedJointIndices.forEach((jointIndex) => {
      // Create kinematic body so Rapier lets us control it manually
      const rigidBodyDesc = RAPIER.RigidBodyDesc.kinematicPositionBased();
      const rigidBody = physicsWorld.createRigidBody(rigidBodyDesc);

      // Sphere collider approximating knuckle size
      const colliderDesc = RAPIER.ColliderDesc.ball(0.012); // 1.2cm radius
      const collider = physicsWorld.createCollider(colliderDesc, rigidBody);

      const mapKey = `\${event.data.handedness}-\${jointIndex}`;
      jointColliders.set(mapKey, { rigidBody, jointIndex, xrHand });
    });
  });
}

🚀 4. Executing the Real-Time Sync Loop

On every frame, we query the spatial positions of our WebXR joints and compute the linear translation vectors for Rapier.js.

javascript
function tickPhysics(frame, referenceSpace) {
  // Step the physics engine simulation
  physicsWorld.step();

  jointColliders.forEach((data, mapKey) => {
    const { rigidBody, jointIndex, xrHand } = data;
    
    // 1. Get the joint handle from the XRHand structure
    const joint = Array.from(xrHand.values())[jointIndex];
    if (!joint) return;

    // 2. Query joint pose relative to our reference coordinate space
    const pose = frame.getJointPose(joint, referenceSpace);
    
    if (pose) {
      const targetPos = pose.transform.position; // vec3 { x, y, z }
      
      // Calculate kinematic movement
      const nextPosition = new RAPIER.Vector3(targetPos.x, targetPos.y, targetPos.z);
      
      // Teleport the kinematic body to match the physical hand
      rigidBody.setNextKinematicTranslation(nextPosition);
    }
  });
}

🛠️ 5. Handling Advanced Grabbing & Grasp Detection

While raw physics colliders allow you to push objects, grabbing requires checking distance relationships:

1. 2.
   Pinch Gesture: Calculate the distance between the index-finger-tip (index 9) and the thumb-tip (index 4).
2. 4.
   Distance Threshold: If the distance drops below 1.5cm and the hand is overlapping an interactive object collider, trigger a dynamic joint constraint (e.g. PrismaticJoint or FixedJoint in Rapier) between the hand's index rigid body and the object.
3. 6.
   Release: When the distance exceeds 2.5cm, delete the physics joint constraint, restoring gravity and momentum to the object.

🏁 6. Conclusion

WebXR Hand Tracking transitions spatial apps from abstract controller inputs to organic human movements. By linking tracked joint matrices directly to WASM-compiled engines like Rapier.js via kinematic position mapping, you construct high-fidelity VR experiences capable of true physics-based interactions natively in standard browsers.