Learn Creative Coding (#78) - WebXR: Creative Coding in VR/AR
Last episode we built a complete procedural 3D world -- terrain, trees, water, atmosphere, day/night cycle. You could walk through it with WASD and mouselook. But here's the thing: you were still looking at it through a rectangle. A flat monitor. The world existed in 3D but your experience of it was fundamentally 2D. You moved a flat mouse cursor across a flat screen to control a virtual camera. There's a glass wall between you and the scene.
WebXR removes that wall. It's the browser API that connects your Three.js scenes to VR headsets and AR devices. Put on a Quest, a Vive, an Apple Vision Pro, or hold up your phone, and suddenly you're not looking at the scene anymore -- you're inside it. Your head movements control the camera. Your hands are tracked in 3D space. You can reach out and touch things. The same Three.js code we've been writing this entire arc works in VR with surprisingly few changes.
This isn't some exotic future technology. WebXR shipped in Chrome in 2020 and is supported in every major browser today (except Safari on iOS, because Apple does what Apple does). If you have a Quest headset -- even the standalone Quest 2 or 3 without a PC -- you can open the Oculus browser, navigate to your localhost tunnel or deployed page, and walk into your scene. No app store, no sideloading, no native SDK. Just a URL.
For creative coding specifically, VR changes the stakes. The particle galaxy from ep065? Imagine being surrounded by it. Particles above, below, behind you. Turn your head and the constellation shifts. Reach out with a controller and particles scatter away from your hand. Scale that would feel cramped on a monitor becomes infinite when you're standing inside it. AR is equally transformative -- place a reaction-diffusion sculpture on your desk. Watch an L-system grow out of your floor. Your generative art escapes the screen and inhabits physical space.
Enabling WebXR in Three.js
The setup is almost embarrassingly simple. Three.js has built-in WebXR support. You enable it on the renderer, add a VR or AR button, and your existing scene works:
import * as THREE from 'three';
import { VRButton } from 'three/addons/webxr/VRButton.js';
const scene = new THREE.Scene();
scene.background = new THREE.Color(0x0a0a1a);
const camera = new THREE.PerspectiveCamera(
70, window.innerWidth / window.innerHeight, 0.1, 200
);
camera.position.set(0, 1.6, 3);
const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.setPixelRatio(window.devicePixelRatio);
renderer.xr.enabled = true;
document.body.appendChild(renderer.domElement);
// add the "Enter VR" button to the page
document.body.appendChild(VRButton.createButton(renderer));
// some objects to look at
scene.add(new THREE.AmbientLight(0x334455, 0.6));
const sun = new THREE.DirectionalLight(0xffeedd, 1.8);
sun.position.set(5, 8, 4);
scene.add(sun);
for (let i = 0; i < 30; i++) {
const mesh = new THREE.Mesh(
new THREE.IcosahedronGeometry(0.3, 1),
new THREE.MeshStandardMaterial({
color: new THREE.Color().setHSL(Math.random(), 0.6, 0.45),
roughness: 0.35,
metalness: 0.1
})
);
mesh.position.set(
(Math.random() - 0.5) * 6,
Math.random() * 3 + 0.5,
(Math.random() - 0.5) * 6
);
scene.add(mesh);
}
// animation loop -- note: use setAnimationLoop, NOT requestAnimationFrame
renderer.setAnimationLoop(function () {
renderer.render(scene, camera);
});
Three things changed compared to our normal setup:
renderer.xr.enabled = true-- tells the renderer to support XR sessionsVRButton.createButton(renderer)-- adds a button to the DOM that starts an immersive VR session when clickedrenderer.setAnimationLoop()instead ofrequestAnimationFrame-- this is important. In XR mode, the headset's refresh rate drives the render loop, not the browser's.setAnimationLoophandles both cases: normal browser rendering when not in VR, and headset-synced rendering when in VR. If you userequestAnimationFramedirectly, your scene won't render in VR.
That's it. If you open this on a Quest browser and tap "Enter VR", you're inside the scene. The 30 floating icosahedrons surround you. Turn your head and the camera follows. No extra code needed -- the headset provides head tracking, and Three.js automatcally applies it to the camera.
Scale matters: thinking in meters
On a flat screen, scale is abstract. A cube that's 1 unit wide could represent anything -- a sugar cube, a house, a planet. Your brain doesn't have a physical reference. In VR, 1 unit equals 1 meter. Your body provides the reference. A 1-unit cube feels like a box you could pick up. A 10-unit cube is a room. A 0.01-unit cube is a marble.
This changes how you design scenes. The camera starts at y = 1.6 in the code above because the average human's eyes are about 1.6 meters above the floor. If you set the camera at y = 0, you'd be lying on the ground. At y = 10, you'd be floating above everything. In VR, the headset overrides the camera position with your actual head position relative to the play space origin. So camera.position.y = 1.6 is really just the initial position before the headset takes over.
// a floor at y=0 feels like standing on it
const floor = new THREE.Mesh(
new THREE.PlaneGeometry(20, 20),
new THREE.MeshStandardMaterial({
color: 0x333344,
roughness: 0.85
})
);
floor.rotation.x = -Math.PI / 2;
floor.receiveShadow = true;
scene.add(floor);
// objects at y=1.0 to y=2.0 are at arm height
// objects at y=3.0+ are overhead -- you look up to see them
// objects below y=0 are underground (invisible with a floor)
For the procedural world from ep077, this means the terrain scale makes physical sense in VR. Hills that were 6 units high? Those are 6-meter hills -- about two stories. The trees at 3-5 meters tall are realistic tree height. The water plane at -1.5 meters is ankle-deep in a valley. The scale we chose for a flat screen happens to work well in VR because we were already using roughly real-world proportions. If your creative coding projects use wildly different scales (particle positions in the thousands, or everything crammed into 0.01 units), you'll need to rescale for VR.
VR controllers: your hands in the scene
The headset tracks your head. Controllers track your hands. Three.js gives you controller objects that you add to the scene like any other Object3D:
const controller0 = renderer.xr.getController(0);
const controller1 = renderer.xr.getController(1);
scene.add(controller0);
scene.add(controller1);
// visual representation: a line pointing forward from each controller
function buildController() {
const geo = new THREE.BufferGeometry();
geo.setAttribute('position', new THREE.Float32BufferAttribute(
[0, 0, 0, 0, 0, -1], 3
));
const line = new THREE.Line(geo, new THREE.LineBasicMaterial({
color: 0x00ddff
}));
line.scale.z = 3;
return line;
}
controller0.add(buildController());
controller1.add(buildController());
Each controller is a Group that Three.js updates every frame with the real-world position and rotation of that hand. You add child objects to it (the line above) and they move with the controller. The line points forward from the controller like a laser pointer -- this is the standard interaction paradigm for VR: point and shoot.
getController(0) and getController(1) return the left and right hand controllers. The index mapping depends on the hardware and which controller the user picked up first. Don't hardcode "0 = left, 1 = right" -- instead check the handedness property if you need to know which is which.
Controller events: trigger, squeeze, buttons
Controllers fire events when you press the trigger, squeeze the grip, or press buttons. The most important ones are selectstart (trigger pressed) and selectend (trigger released):
controller0.addEventListener('selectstart', function () {
console.log('left trigger pressed');
});
controller0.addEventListener('selectend', function () {
console.log('left trigger released');
});
controller1.addEventListener('selectstart', function () {
console.log('right trigger pressed');
});
// squeeze events (grip button on the side)
controller0.addEventListener('squeezestart', function () {
console.log('left grip pressed');
});
controller0.addEventListener('squeezeend', function () {
console.log('left grip released');
});
These are the building blocks of VR interaction. Trigger press = select, like a mouse click. Grip press = grab. You combine these with raycasting from the controller's position to build "point and click" and "grab and move" interactions. We covered raycasting in ep076 -- the exact same Raycaster works in VR, you just cast the ray from the controller instead of the mouse.
Raycasting from controllers
In ep076 we raycasted from the camera through the mouse position. In VR there's no mouse -- you raycast from the controller forward:
const raycaster = new THREE.Raycaster();
const tempMatrix = new THREE.Matrix4();
const interactiveObjects = []; // fill this with your clickable objects
function onSelectStart(event) {
const controller = event.target;
// build ray from controller position, pointing forward
tempMatrix.identity().extractRotation(controller.matrixWorld);
raycaster.ray.origin.setFromMatrixPosition(controller.matrixWorld);
raycaster.ray.direction.set(0, 0, -1).applyMatrix4(tempMatrix);
const intersects = raycaster.intersectObjects(interactiveObjects);
if (intersects.length > 0) {
const hit = intersects[0].object;
// do something with the hit object
hit.material.color.setHSL(Math.random(), 0.7, 0.5);
}
}
controller0.addEventListener('selectstart', onSelectStart);
controller1.addEventListener('selectstart', onSelectStart);
extractRotation gets the controller's orientation. The direction (0, 0, -1) is "forward" in the controller's local space. Apply the rotation matrix and you get the world-space direction the controller is pointing. The ray origin is the controller's world position. Same Raycaster API, different source.
The laser pointer line we added earlier aligns with this ray visually -- you see where you're pointing and the raycast tests that exact direction. Point at an object, pull the trigger, and it changes color. Simple but it's the foundation for all VR interaction.
Grabbing objects in VR
Pointing and clicking is fine but VR is about reaching out and grabbing. When the trigger is pressed and the controller overlaps an object, attach the object to the controller. It moves with your hand. Release the trigger, detach it:
let grabbed0 = null;
let grabbed1 = null;
function handleGrab(controllerIndex) {
const controller = controllerIndex === 0 ? controller0 : controller1;
const grabbed = controllerIndex === 0 ? grabbed0 : grabbed1;
if (grabbed) return; // already holding something
// check proximity: is any object close to the controller?
const controllerPos = new THREE.Vector3();
controller.getWorldPosition(controllerPos);
let closest = null;
let closestDist = 0.3; // grab range: 30cm
for (const obj of interactiveObjects) {
const dist = controllerPos.distanceTo(obj.position);
if (dist < closestDist) {
closest = obj;
closestDist = dist;
}
}
if (closest) {
controller.attach(closest);
if (controllerIndex === 0) grabbed0 = closest;
else grabbed1 = closest;
}
}
function handleRelease(controllerIndex) {
const controller = controllerIndex === 0 ? controller0 : controller1;
const obj = controllerIndex === 0 ? grabbed0 : grabbed1;
if (!obj) return;
// detach: move back to scene, keep world transform
scene.attach(obj);
if (controllerIndex === 0) grabbed0 = null;
else grabbed1 = null;
}
controller0.addEventListener('selectstart', function () { handleGrab(0); });
controller0.addEventListener('selectend', function () { handleRelease(0); });
controller1.addEventListener('selectstart', function () { handleGrab(1); });
controller1.addEventListener('selectend', function () { handleRelease(1); });
controller.attach(object) is the key method. It reparents the object from the scene to the controller while maintaining the object's world-space transform. The object stays at its visual position but is now a child of the controller -- so when you move your hand, the object moves with it. On release, scene.attach(object) reparents it back to the scene, again preserving world position. The object stays wherever your hand left it.
The proximity check (30cm radius) means you have to physically reach close to an object to grab it. This feels natural -- you don't grab things from across the room, you reach out and pick them up. For creative coding this means your objects need to be at reachable distances, which ties back to the scale discussion. Objects 10 meters away can't be grabbed directly -- you'd use teleportation or a tractor beam instead.
AR mode: augmented reality in the browser
Swap VRButton for ARButton and the session type changes to augmented reality. Instead of replacing your vision entirely, AR overlays your Three.js objects onto the real world via the device's camera:
import { ARButton } from 'three/addons/webxr/ARButton.js';
const renderer = new THREE.WebGLRenderer({ antialias: true, alpha: true });
renderer.xr.enabled = true;
document.body.appendChild(renderer.domElement);
document.body.appendChild(ARButton.createButton(renderer));
Two differences from VR setup: the renderer needs alpha: true so the background is transparent (the camera feed shows through), and we use ARButton instead of VRButton. You also typically don't set scene.background -- leave it transparent so the real world is visible behind your objects.
AR works on Android phones with Chrome (via WebXR), and on Quest devices in passthrough mode. Apple's Safari doesn't support WebXR on iOS, but you can use Apple's QuickLook for simple 3D model viewing (not interactive though).
Hit testing: placing objects on real surfaces
The killer feature of AR is placing digital objects on real-world surfaces. You look at your desk, tap, and a 3D object appears on the desk surface at the exact right position and height. WebXR's hit testing API makes this possible:
let hitTestSource = null;
let hitTestSourceRequested = false;
const reticle = new THREE.Mesh(
new THREE.RingGeometry(0.05, 0.07, 24),
new THREE.MeshBasicMaterial({
color: 0x00ff88,
side: THREE.DoubleSide
})
);
reticle.rotation.x = -Math.PI / 2;
reticle.visible = false;
scene.add(reticle);
renderer.setAnimationLoop(function (timestamp, frame) {
if (frame) {
const referenceSpace = renderer.xr.getReferenceSpace();
const session = renderer.xr.getSession();
// request hit test source once
if (!hitTestSourceRequested) {
session.requestReferenceSpace('viewer').then(function (refSpace) {
session.requestHitTestSource({ space: refSpace }).then(function (source) {
hitTestSource = source;
});
});
hitTestSourceRequested = true;
}
// perform hit test each frame
if (hitTestSource) {
const hitTestResults = frame.getHitTestResults(hitTestSource);
if (hitTestResults.length > 0) {
const hit = hitTestResults[0];
const pose = hit.getPose(referenceSpace);
reticle.visible = true;
reticle.matrix.fromArray(pose.transform.matrix);
reticle.matrix.decompose(
reticle.position, reticle.quaternion, reticle.scale
);
} else {
reticle.visible = false;
}
}
}
renderer.render(scene, camera);
});
// tap to place an object at the reticle position
const controller = renderer.xr.getController(0);
controller.addEventListener('select', function () {
if (reticle.visible) {
const geo = new THREE.IcosahedronGeometry(0.05, 2);
const mat = new THREE.MeshStandardMaterial({
color: new THREE.Color().setHSL(Math.random(), 0.6, 0.5),
roughness: 0.3,
metalness: 0.1
});
const mesh = new THREE.Mesh(geo, mat);
mesh.position.copy(reticle.position);
mesh.quaternion.copy(reticle.quaternion);
scene.add(mesh);
}
});
scene.add(controller);
scene.add(new THREE.HemisphereLight(0xffffff, 0xbbbbff, 1));
The reticle (green ring) hovers on real-world surfaces detected by the device's depth sensor or plane detection. It follows your gaze -- look at the floor and it sits on the floor. Look at a table and it jumps to the table surface. Tap and an icosahedron spawns at that exact position. It sits on the real surface, correctly positioned in 3D space relative to your room.
This is where creative coding gets wild. Instead of spawning a simple shape, spawn a particle system. A generative sculpture. A reaction-diffusion simulation running on a mesh. An L-system tree growing out of your desk. The hit test gives you the anchor point in physical space. What you build from that anchor is up to you.
Performance in VR: the 72fps floor
VR has a hard performance requirement that flat-screen rendering doesn't. On a monitor, dropping from 60fps to 30fps looks janky but is otherwise harmless. In VR, dropping below the headset's refresh rate (72fps for Quest 2, 90fps for Quest 3 and most PC headsets, 120fps for Quest Pro) causes motion sickness. Your brain expects head movements to be reflected instantly. When frames drop, the world stutters relative to your head motion and your vestibular system registers a mismatch. That mismatch is nausea.
And you're rendering twice per frame -- once per eye. The two cameras are offset by the interpupillary distance (~63mm). That's double the draw calls, double the fill rate, double everything. Your effective budget per eye is roughly half what you'd have on a flat screen.
This means:
// Use instancing aggressively -- ep070
// 1000 individual meshes = 1000 draw calls
// 1000 instanced meshes = 1 draw call
const instancedMesh = new THREE.InstancedMesh(geo, mat, 1000);
// Lower geometry resolution
// On flat screen: SphereGeometry(1, 64, 64) -- 8192 tris
// In VR: SphereGeometry(1, 16, 16) -- 512 tris
// You're close enough to notice? Maybe. But 16x fewer triangles.
// Cap pixel ratio
renderer.setPixelRatio(1); // VR headsets handle their own resolution
// Frustum culling is automatic in Three.js
// But for custom systems (particles, instanced), cull manually
// Simpler materials save fragment shader time
// MeshBasicMaterial is cheapest (no lighting)
// MeshLambertMaterial is cheap (per-vertex lighting)
// MeshStandardMaterial is the default PBR (per-pixel, expensive)
// MeshPhysicalMaterial is heaviest (clearcoat, subsurface, etc)
For the procedural world from ep077: 40,000 grass blades, 800 trees, 4000 dust particles, terrain with 180x180 segments, a bloom post-processing pass. On a desktop with a decent GPU, that runs fine at 60fps. In VR on a standalone Quest 2 (which has a mobile GPU), you'd need to cut aggressively -- 5000 grass blades, 200 trees, 500 particles, terrain at 60x60, no bloom. The visual quality drops, but being inside the world compensates. Immersion makes up for lower fidelity.
Teleportation: moving around in VR
Walking around in VR is limited by your physical play space (usually 2x2 meters for standing, maybe 3x3 for room-scale). For larger scenes you need artificial locomotion. The most comfortable method is teleportation -- point at a spot on the ground, click, and instantly move there. No smooth motion means no motion sickness.
const teleportMarker = new THREE.Mesh(
new THREE.CircleGeometry(0.3, 24),
new THREE.MeshBasicMaterial({
color: 0x00ff88,
transparent: true,
opacity: 0.5,
side: THREE.DoubleSide
})
);
teleportMarker.rotation.x = -Math.PI / 2;
teleportMarker.visible = false;
scene.add(teleportMarker);
// floor or terrain to teleport onto
const teleportTargets = [floor]; // add your terrain mesh here
function updateTeleportTarget(controller) {
tempMatrix.identity().extractRotation(controller.matrixWorld);
raycaster.ray.origin.setFromMatrixPosition(controller.matrixWorld);
raycaster.ray.direction.set(0, 0, -1).applyMatrix4(tempMatrix);
const intersects = raycaster.intersectObjects(teleportTargets);
if (intersects.length > 0) {
teleportMarker.position.copy(intersects[0].point);
teleportMarker.visible = true;
return intersects[0].point;
}
teleportMarker.visible = false;
return null;
}
let teleportTarget = null;
// update in animation loop
renderer.setAnimationLoop(function () {
// show teleport marker while pointing at ground
teleportTarget = updateTeleportTarget(controller0);
renderer.render(scene, camera);
});
// teleport on trigger release
controller0.addEventListener('selectend', function () {
if (teleportTarget) {
// move the XR camera rig to the target position
const offsetY = 0; // keep camera at current height
const baseReferenceSpace = renderer.xr.getReferenceSpace();
const transform = new XRRigidTransform(
{ x: -teleportTarget.x, y: -teleportTarget.y - offsetY, z: -teleportTarget.z, w: 1 },
{ x: 0, y: 0, z: 0, w: 1 }
);
const newRefSpace = baseReferenceSpace.getOffsetReferenceSpace(transform);
renderer.xr.setReferenceSpace(newRefSpace);
}
});
The trick is getOffsetReferenceSpace. In WebXR, you can't just move the camera -- the headset controls the camera. Instead, you offset the reference space, which shifts the entire coordinate system relative to the user. The effect is the same: you're standing in a different spot. But conceptually it's the world moving around you rather than you moving through the world.
This is a simplified version -- a production teleport system would add an arc trajectory (parabolic curve from controller to ground), show a path preview, handle collision with walls, and animate the transition. But the core mechanic is the same: raycast to find a target position, offset the reference space to teleport there.
Creative exercise: immersive particle space
Allez, time to build something that only makes sense in VR. A particle space you stand inside -- thousands of particles surround you in a sphere, drifitng slowly, reacting to your controllers. Point a controller and nearby particles flee. Squeeze the grip and they're pulled toward your hand. Two-handed spread pushes particles apart. The experience is about presence -- being physically inside a generative system, affecting it with your body.
import * as THREE from 'three';
import { VRButton } from 'three/addons/webxr/VRButton.js';
const scene = new THREE.Scene();
scene.background = new THREE.Color(0x030308);
const camera = new THREE.PerspectiveCamera(
70, window.innerWidth / window.innerHeight, 0.1, 100
);
camera.position.set(0, 1.6, 0);
const renderer = new THREE.WebGLRenderer({ antialias: true });
renderer.setSize(window.innerWidth, window.innerHeight);
renderer.xr.enabled = true;
document.body.appendChild(renderer.domElement);
document.body.appendChild(VRButton.createButton(renderer));
// subtle ambient light
scene.add(new THREE.AmbientLight(0x112233, 0.3));
// controllers
const ctrl0 = renderer.xr.getController(0);
const ctrl1 = renderer.xr.getController(1);
scene.add(ctrl0);
scene.add(ctrl1);
// controller visuals: glowing spheres at hand positions
const handGeo = new THREE.SphereGeometry(0.02, 8, 8);
const handMat = new THREE.MeshBasicMaterial({ color: 0x00ddff });
ctrl0.add(new THREE.Mesh(handGeo, handMat));
ctrl1.add(new THREE.Mesh(handGeo, handMat.clone()));
ctrl1.children[0].material.color.set(0xff4488);
// track squeeze state
let squeezing0 = false;
let squeezing1 = false;
ctrl0.addEventListener('squeezestart', function () { squeezing0 = true; });
ctrl0.addEventListener('squeezeend', function () { squeezing0 = false; });
ctrl1.addEventListener('squeezestart', function () { squeezing1 = true; });
ctrl1.addEventListener('squeezeend', function () { squeezing1 = false; });
// particles
const particleCount = 8000;
const positions = new Float32Array(particleCount * 3);
const velocities = new Float32Array(particleCount * 3);
const colors = new Float32Array(particleCount * 3);
for (let i = 0; i < particleCount; i++) {
const i3 = i * 3;
// distribute in a sphere around the viewer
const theta = Math.random() * Math.PI * 2;
const phi = Math.acos(2 * Math.random() - 1);
const r = 1.0 + Math.random() * 4.0;
positions[i3] = r * Math.sin(phi) * Math.cos(theta);
positions[i3 + 1] = r * Math.sin(phi) * Math.sin(theta) + 1.6;
positions[i3 + 2] = r * Math.cos(phi);
velocities[i3] = 0;
velocities[i3 + 1] = 0;
velocities[i3 + 2] = 0;
const hue = 0.55 + Math.random() * 0.15;
const col = new THREE.Color().setHSL(hue, 0.6, 0.5);
colors[i3] = col.r;
colors[i3 + 1] = col.g;
colors[i3 + 2] = col.b;
}
const particleGeo = new THREE.BufferGeometry();
particleGeo.setAttribute('position', new THREE.BufferAttribute(positions, 3));
particleGeo.setAttribute('color', new THREE.BufferAttribute(colors, 3));
const particleMat = new THREE.PointsMaterial({
size: 0.02,
vertexColors: true,
transparent: true,
opacity: 0.7,
sizeAttenuation: true,
depthWrite: false,
blending: THREE.AdditiveBlending
});
const particles = new THREE.Points(particleGeo, particleMat);
scene.add(particles);
// interaction
const ctrlPos0 = new THREE.Vector3();
const ctrlPos1 = new THREE.Vector3();
function updateParticles(delta) {
ctrl0.getWorldPosition(ctrlPos0);
ctrl1.getWorldPosition(ctrlPos1);
const pos = particleGeo.attributes.position.array;
for (let i = 0; i < particleCount; i++) {
const i3 = i * 3;
const px = pos[i3];
const py = pos[i3 + 1];
const pz = pos[i3 + 2];
// controller 0 influence
const dx0 = px - ctrlPos0.x;
const dy0 = py - ctrlPos0.y;
const dz0 = pz - ctrlPos0.z;
const dist0 = Math.sqrt(dx0 * dx0 + dy0 * dy0 + dz0 * dz0);
if (dist0 < 1.5) {
const strength = (1.0 - dist0 / 1.5) * 0.8;
if (squeezing0) {
// attract toward controller
velocities[i3] -= dx0 * strength * delta * 3;
velocities[i3 + 1] -= dy0 * strength * delta * 3;
velocities[i3 + 2] -= dz0 * strength * delta * 3;
} else {
// repel away from controller
const norm = Math.max(dist0, 0.1);
velocities[i3] += (dx0 / norm) * strength * delta * 2;
velocities[i3 + 1] += (dy0 / norm) * strength * delta * 2;
velocities[i3 + 2] += (dz0 / norm) * strength * delta * 2;
}
}
// controller 1 influence (same logic)
const dx1 = px - ctrlPos1.x;
const dy1 = py - ctrlPos1.y;
const dz1 = pz - ctrlPos1.z;
const dist1 = Math.sqrt(dx1 * dx1 + dy1 * dy1 + dz1 * dz1);
if (dist1 < 1.5) {
const strength = (1.0 - dist1 / 1.5) * 0.8;
if (squeezing1) {
velocities[i3] -= dx1 * strength * delta * 3;
velocities[i3 + 1] -= dy1 * strength * delta * 3;
velocities[i3 + 2] -= dz1 * strength * delta * 3;
} else {
const norm = Math.max(dist1, 0.1);
velocities[i3] += (dx1 / norm) * strength * delta * 2;
velocities[i3 + 1] += (dy1 / norm) * strength * delta * 2;
velocities[i3 + 2] += (dz1 / norm) * strength * delta * 2;
}
}
// gentle drift toward center (prevents particles flying away forever)
velocities[i3] += (0 - px) * 0.01 * delta;
velocities[i3 + 1] += (1.6 - py) * 0.01 * delta;
velocities[i3 + 2] += (0 - pz) * 0.01 * delta;
// damping
velocities[i3] *= 0.97;
velocities[i3 + 1] *= 0.97;
velocities[i3 + 2] *= 0.97;
// apply velocity
pos[i3] += velocities[i3];
pos[i3 + 1] += velocities[i3 + 1];
pos[i3 + 2] += velocities[i3 + 2];
}
particleGeo.attributes.position.needsUpdate = true;
}
// color shift based on particle velocity
function updateParticleColors() {
const cols = particleGeo.attributes.color.array;
for (let i = 0; i < particleCount; i++) {
const i3 = i * 3;
const speed = Math.sqrt(
velocities[i3] * velocities[i3] +
velocities[i3 + 1] * velocities[i3 + 1] +
velocities[i3 + 2] * velocities[i3 + 2]
);
// slow = cool blue, fast = warm pink
const t = Math.min(speed * 20, 1.0);
const hue = 0.6 - t * 0.45;
const col = new THREE.Color().setHSL(hue, 0.5 + t * 0.3, 0.4 + t * 0.2);
cols[i3] = col.r;
cols[i3 + 1] = col.g;
cols[i3 + 2] = col.b;
}
particleGeo.attributes.color.needsUpdate = true;
}
const clock = new THREE.Clock();
renderer.setAnimationLoop(function () {
const delta = Math.min(clock.getDelta(), 0.05);
updateParticles(delta);
updateParticleColors();
renderer.render(scene, camera);
});
window.addEventListener('resize', function () {
camera.aspect = window.innerWidth / window.innerHeight;
camera.updateProjectionMatrix();
renderer.setSize(window.innerWidth, window.innerHeight);
});
On a flat screen this works too -- the particles surround the camera and drift around. But in VR it's a completely different experience. You're standing in a cloud of glowing particles. They react to your hands. Reach out and they scatter. Squeeze and they swirl toward your palm. The color shifts as they accelerate -- blue calm, pink excited. Two hands working together, pushing and pulling the swarm around you. This is the kind of thing that can only exist in immersive space. On a monitor it's a nice particle sim. In VR it's an experience.
8000 particles is conservative for a particle system but VR is demanding. On a Quest 2 you could push to maybe 15,000-20,000 points with this simple material, but past that the fragment fill rate on transparent additive particles starts to hurt. On a PC-tethered headset with a decent GPU you could go much higher. Profile on your target hardware -- what runs at 72fps matters more than what looks impressive at 30fps.
Hand tracking: no controllers needed
Newer headsets (Quest 2+, Vision Pro) support hand tracking -- the cameras see your actual hands and track individual finger joints. Three.js supports this through XRHand:
const hand0 = renderer.xr.getHand(0);
const hand1 = renderer.xr.getHand(1);
scene.add(hand0);
scene.add(hand1);
// visual: small spheres at each joint
const jointGeo = new THREE.SphereGeometry(0.008, 8, 8);
const jointMat = new THREE.MeshBasicMaterial({ color: 0x00ddff });
for (const hand of [hand0, hand1]) {
for (let i = 0; i < 25; i++) {
// 25 joints per hand (wrist, fingers, tips)
const joint = new THREE.Mesh(jointGeo, jointMat);
joint.name = 'joint-' + i;
hand.add(joint);
}
}
Each hand has 25 tracked joints. You can read individual finger positions to detect gestures: pinch (index tip near thumb tip), point (index extended, others curled), fist (all fingers curled). This opens up interaction design that controllers can't match -- sculpt in the air, paint with your fingertips, conduct an orchestra of particles with hand waves.
Hand tracking is less precise than controllers (cameras can lose sight of fingers, especially when hands overlap or face away) and has higher latency. For creative coding experiments it's magical. For production-quality interaction, controllers are still more reliable.
What's ahead
WebXR puts your Three.js scenes into immersive space -- VR headsets for full immersion, AR for digital-physical blending. The same renderer, the same scene graph, the same materials. What changes is the interaction model (controllers instead of mouse, head tracking instead of OrbitControls) and the performance constraint (72-90fps mandatory, rendered twice per eye).
We've been building 3D worlds for sixteen episodes. WebXR is the gateway that lets you step inside them. But the 3D arc is wrapping up -- next we're changing direction entirely. Instead of creating visuals from code, we'll start creating visuals from data. Real-world datasets, API responses, files full of numbers and text -- all of it can become visual material for creative coding. The techniques we've built (geometry, color, motion, interaction, 3D) are the tools. Data is the next source of inspiration.
Allez, wa weten we nu allemaal?
- WebXR is the browser API that connects Three.js scenes to VR headsets and AR devices. Enable it with
renderer.xr.enabled = true, add aVRButtonorARButton, and switch torenderer.setAnimationLoop()instead ofrequestAnimationFrame. Existing Three.js scenes work in VR with minimal changes - In VR, 1 unit = 1 meter. Camera at y=1.6 is eye height. Scale affects how the scene feels physically -- a 1-unit cube is a box you could hold, a 10-unit cube is a room. Scenes designed with roughly real-world proportions (like our procedural world from ep077) translate well to VR
- Controllers are tracked 3D objects:
renderer.xr.getController(0)and(1). Add child objects for visual representation. They fireselectstart/selectend(trigger) andsqueezestart/squeezeend(grip) events. Combine with raycasting for point-and-click interaction - Raycasting from controllers: extract the controller's world rotation, set ray direction to (0,0,-1) in local space, transform to world space. Same
THREE.RaycasterAPI as mouse raycasting from ep076, just a different ray source - Grabbing: check proximity between controller and objects. On trigger press,
controller.attach(object)reparents the object to the controller (preserving world transform). On release,scene.attach(object)returns it to the scene. Objects move with your hand while grabbed - AR mode: use
ARButtoninstead ofVRButton, setalpha: trueon the renderer so the camera feed shows through. Hit testing (session.requestHitTestSource) detects real-world surfaces -- a reticle follows the surface under your gaze, tap to place objects on your desk, floor, or walls - Performance is critical: VR renders twice per frame (once per eye), and dropping below 72fps causes nausea. Use instancing, lower geometry resolution, simple materials, and cut particle counts. A scene that runs at 60fps on desktop may need significant optimization for standalone VR (Quest has a mobile GPU)
- Teleportation: raycast from controller to find a floor position, then
getReferenceSpace().getOffsetReferenceSpace(transform)to shift the coordinate origin. The world moves around the user rather than the user moving through the world - Hand tracking:
renderer.xr.getHand(0)gives 25 tracked joints per hand. Detect gestures (pinch, point, fist) from joint positions. Less precise than controllers but enables natural interaction -- sculpting, painting, conducting particles with hand movements - The creative exercise: an immersive particle space. 8000 particles surround you. Controllers repel nearby particles (scatter on approach), grip attracts them (swirl toward your palm). Color shifts with velocity -- blue calm, pink excited. Works on flat screen but the VR experience is the point -- presence turns a particle sim into something you inhabit
Sallukes! Thanks for reading.
X