Learn Creative Coding (#110) - Projection Mapping Basics
At the very end of last episode I said something a little bit cheeky - that once your code can drive lights, it can throw images onto whole buildings. I wasn't just being dramatic to sell you on charging your ESP32 :-). That's a real thing, it has a name, and today it's ours. We plotted with a pen, we cut with a laser, we lit up a strip of LEDs - and now we're going to take a projector and paint our generative sketches directly onto physical objects. Walls. Boxes. A sculpture. The side of your bookshelf if that's what you've got lying around.
It's called projection mapping, and the first time you do it right - the first time a shape you drew in p5 wraps perfectly around a real cardboard box and makes it look like it's breathing - you will make a noise out loud. I did. My flatmate came running. Allez, let me show you how the trick works, because underneath the magic it's just a bit of geometry we already half-know.
What projection mapping actually is
Here's the plain version. A projector throws a rectangle of light. Your object in the real world is almost never a perfect rectangle facing the projector dead-on - it's a box at an angle, a wall with a window in it, a weird lumpy sculpture. Projection mapping is the art of warping your image so it lines up exactly with the physical thing, so instead of a flat rectangle of light slapped over everything, the projection looks like it belongs to the surface. Like the object is glowing from inside.
That's the whole idea. The projector doesn't know or care what it's pointing at - it just fires pixels in a straight rectangle. All the cleverness happens in your code, before the light ever leaves the lens. You pre-distort the image so that when it hits the tilted, awkward surface, the distortion cancels out and it looks right from the viewer's spot.
// the core mental model of projection mapping:
//
// YOUR SKETCH (a clean rectangle)
// |
// | warp it so it fits the real surface
// v
// PRE-DISTORTED IMAGE --projector--> PHYSICAL SURFACE
// (looks perfect again!)
//
// the projector is dumb. it throws a rectangle. the magic is the WARP.
If that reminds you of anything, good - it should. Back in episode 32 we spent time bending images with shader uniforms, pushing pixels around based on math. And in episode 62, when we first stepped into 3D with Three.js, we learned how a flat thing gets placed and tilted in space. Projection mapping is those two ideas holding hands and walking out into your living room. See where this is going?
The heart of it: corner pinning
The simplest, most useful version of projection mapping is called corner pinning, and honestly it gets you 80% of the way for most projects. The idea is beautifully direct: your sketch is a rectangle with four corners. Your surface (say, one flat face of a box) also has four corners. Corner pinning just says - drag my four image corners onto those four real corners, and stretch everything in between to match.
// corner pinning in one breath:
// map the 4 corners of my canvas onto 4 points on the real surface.
//
// canvas corner (0,0) -> surface point (top-left of the box face)
// canvas corner (W,0) -> surface point (top-right)
// canvas corner (W,H) -> surface point (bottom-right)
// canvas corner (0,H) -> surface point (bottom-left)
//
// everything inside the rectangle follows along. that's it.
Now, you can't just move the corners and linearly stretch the middle - that gives you a wonky, wrong result the moment there's any perspective (and there's always perspective, because your projector is never perfectly square-on). What you actually need is a homography: a special kind of transform that handles perspective correctly. It's the same math a phone uses when it "flattens" a photo of a document you took at an angle. Let me unpack it, because it's less scary than it sounds.
The math: a perspective transform
A homography is a 3x3 matrix. Feed it a point (x, y) and it spits out a new point - but with a twist that ordinary 2D transforms (move, rotate, scale) don't have: it uses a third coordinate, w, to fake perspective. Things "further away" get divided down so they shrink, exactly like your eye does.
// a homography maps a point through a 3x3 matrix, then divides by w.
// that final divide is what gives us PERSPECTIVE (near = big, far = small).
function applyHomography(H, x, y) {
const nx = H[0]*x + H[1]*y + H[2];
const ny = H[3]*x + H[4]*y + H[5];
const nw = H[6]*x + H[7]*y + H[8]; // the perspective term
return [nx / nw, ny / nw]; // the divide is the whole secret
}
That little / nw at the end is the entire difference between "flat sticker" and "convincing perspective". A normal affine transform has nw always equal to 1, so nothing shrinks with distance. The homography lets nw vary across the image, and that is what makes a projection lie flat on a tilted surface.
So how do we find the nine numbers in H? We have four corner pairs - four source points and where each should land. Four points give us eight equations (an x and a y each), and a homography has eight free parameters (the ninth is just a scale we pin to 1). Eight equations, eight unknowns - it's a linear system we can solve. I won't make you derive it by hand, but here's the shape of it so it's not a black box.
// build the 8x8 linear system from 4 point correspondences.
// src = [[x,y]*4] (our canvas corners), dst = [[x,y]*4] (surface corners).
// this is the classic "direct linear transform" setup for a homography.
function homographyRows(src, dst) {
const A = [], b = [];
for (let i = 0; i < 4; i++) {
const [x, y] = src[i];
const [X, Y] = dst[i];
// two rows per point: one for X, one for Y
A.push([x, y, 1, 0, 0, 0, -X*x, -X*y]); b.push(X);
A.push([0, 0, 0, x, y, 1, -Y*x, -Y*y]); b.push(Y);
}
return { A, b }; // solve A * h = b for the 8 unknowns, then h9 = 1
}
Solving A * h = b is just Gaussian elimination - the same "solve a system of equations" you maybe met in school, only with a computer doing the boring arithmetic. Here's a compact solver so the pipeline is complete and you're not left hanging.
// tiny Gaussian-elimination solver for an 8x8 system. nothing fancy -
// forward eliminate, then back-substitute. good enough for corner pinning.
function solve(A, b) {
const n = b.length;
for (let col = 0; col < n; col++) {
// find the biggest pivot in this column (keeps it numerically stable)
let piv = col;
for (let r = col + 1; r < n; r++)
if (Math.abs(A[r][col]) > Math.abs(A[piv][col])) piv = r;
[A[col], A[piv]] = [A[piv], A[col]];
[b[col], b[piv]] = [b[piv], b[col]];
for (let r = col + 1; r < n; r++) {
const f = A[r][col] / A[col][col];
for (let c = col; c < n; c++) A[r][c] -= f * A[col][c];
b[r] -= f * b[col];
}
}
const h = new Array(n).fill(0);
for (let r = n - 1; r >= 0; r--) { // back-substitute
let s = b[r];
for (let c = r + 1; c < n; c++) s -= A[r][c] * h[c];
h[r] = s / A[r][r];
}
return h;
}
Wire those together and you've got a function that takes four corner pairs and hands you back a ready-to-use homography matrix. This is the engine room of every corner-pinning tool on earth, and now you own a little one.
// the full corner-pinning engine: 4 canvas corners + 4 surface corners
// -> a 3x3 homography you can apply to any point.
function computeHomography(src, dst) {
const { A, b } = homographyRows(src, dst);
const h = solve(A, b); // 8 numbers...
return [h[0], h[1], h[2],
h[3], h[4], h[5],
h[6], h[7], 1]; // ...plus the pinned 9th = 1
}
Making it practical in p5
Now, doing that homography math per-pixel in JavaScript would be painfully slow - that's the GPU's job, and there are kinder roads. The road I actually walk for quick projects is to render my sketch into an off-screen buffer, then draw that buffer onto a warped quad in WebGL. p5 gives us createGraphics() for exactly this - a hidden canvas you draw into, then use as a texture.
// render the "real" sketch into a hidden buffer, then map it onto a
// draggable quad. the buffer is your art; the quad is where it lands.
let art; // off-screen graphics buffer
let corners; // the 4 surface points we drag around
function setup() {
createCanvas(1280, 800, WEBGL); // WEBGL so we can texture a quad
art = createGraphics(640, 480); // our sketch lives in here
corners = [ // start as a plain rectangle...
{ x: -300, y: -200 }, { x: 300, y: -200 },
{ x: 300, y: 200 }, { x: -300, y: 200 },
]; // ...then you drag these onto the box
}
Each frame you draw your normal generative art into art (any sketch from the whole series - a flow field, a particle system, reaction-diffusion, whatever you love), and then you paint that buffer onto the four-cornered quad. In p5 WebGL you texture a quad by giving each vertex a position and a texture coordinate.
// paint the art buffer onto the warped quad.
// texture coords (u,v) stay a clean 0..1 rectangle;
// the vertex POSITIONS are the dragged corners. the GPU stretches between.
function drawMappedQuad() {
texture(art);
noStroke();
beginShape();
vertex(corners[0].x, corners[0].y, 0, 0, 0); // top-left -> uv (0,0)
vertex(corners[1].x, corners[1].y, 0, 1, 0); // top-right -> uv (1,0)
vertex(corners[2].x, corners[2].y, 0, 1, 1); // bot-right -> uv (1,1)
vertex(corners[3].x, corners[3].y, 0, 0, 1); // bot-left -> uv (0,1)
endShape(CLOSE);
}
One honest caveat: a plain textured quad like this uses bilinear interpolation, which isn't perfectly perspective-correct across a steeply tilted surface - straight lines can bow a touch. For most box faces and gentle angles you'll never notice. When you do need it dead-on, that's where the homography matrix from earlier comes in (feed it into a shader as the texture transform, riding straight on episode 32's uniform tricks) or you reach for a library like perspective.js that does the correct projective warp for you. Start with the quad, graduate to the matrix when your eye starts complaining.
Calibration: dragging the corners into place
Here's the part people don't expect: projection mapping is half code, half fiddling in a dark room. You will never type in the exact corner coordinates that match your real box - you find them by eye, dragging each corner until the projected edge sits perfectly on the physical edge. So every mapping tool, including yours, needs a calibration mode where you can grab and move those corners live.
// calibration: click-drag the nearest corner. mouse in p5 WEBGL is
// centered, so we shift by half width/height to match our vertex space.
let dragging = -1;
function mousePressed() {
const mx = mouseX - width / 2, my = mouseY - height / 2;
for (let i = 0; i < corners.length; i++) {
if (dist(mx, my, corners[i].x, corners[i].y) < 30) dragging = i;
}
}
function mouseDragged() {
if (dragging >= 0) {
corners[dragging].x = mouseX - width / 2;
corners[dragging].y = mouseY - height / 2;
}
}
function mouseReleased() { dragging = -1; }
And because you set this up once in a dark room and do not want to redo it every time you reopen the sketch, you save the calibration. A projector that doesn't move keeps the same corners forever, so this little bit of localStorage will save you a lot of grief.
// save / load your hard-won calibration so you never redo it by hand.
function saveCalibration() {
localStorage.setItem("mapCorners", JSON.stringify(corners));
}
function loadCalibration() {
const saved = localStorage.getItem("mapCorners");
if (saved) corners = JSON.parse(saved); // straight back to aligned
}
A trick from every VJ I've ever watched: project a grid while you calibrate, not your final art. A grid or a checkerboard makes misalignment jump out at you - a bent line is obvious, whereas a swirly generative pattern hides its own crookedness. Line the grid up first, then switch to your real content.
// draw a calibration grid INTO the art buffer. line these squares up
// with the real surface edges, then swap back to your actual sketch.
function drawCalibrationGrid(g, step = 40) {
g.background(0);
g.stroke(0, 255, 120);
for (let x = 0; x <= g.width; x += step) g.line(x, 0, x, g.height);
for (let y = 0; y <= g.height; y += step) g.line(0, y, g.width, y);
}
More than one surface
Once you can pin one quad, multiplying it is easy and that's where it gets genuinly exciting. A box has three visible faces - map each one with its own quad and its own homography, and suddenly the whole box is a display. A bookshelf becomes a grid of little screens, one per shelf. A room becomes walls-plus-floor-plus-ceiling. You just keep a list of mapped surfaces, each with its own corners and its own slice of art.
// a scene is just a list of independently-mapped surfaces.
// each one has its own corners and its own source sketch.
const surfaces = [
{ corners: [/*...*/], render: drawWaterfall }, // the front face
{ corners: [/*...*/], render: drawEmbers }, // the top face
{ corners: [/*...*/], render: drawVines }, // the side face
];
function drawScene() {
for (const s of surfaces) {
s.render(s.buffer); // draw that face's art into its buffer
drawMappedQuad(s); // pin it onto its patch of the real world
}
}
And here's the design idea that turns a technical demo into art: the content should know the architecture. Don't just splash the same loop everywhere. Project water that pools on the horizontal top and drips down the vertical sides. Fire that rises upward because fire rises. Vines that climb along the real edges of the object. When the content respects gravity and the shape of the thing, people stop seeing "a projection" and start seeing a box that is genuinely, impossibly alive. That's the whole game.
Masking: telling light where NOT to go
One more essential, because real surfaces are messy. Say you're mapping a wall that has a window in it - you do not want your projection spilling across the glass. Or your box has a gap between two faces. The fix is masking: you paint a black shape over the regions the light should skip. Black on a projector is just... no light, which is exactly what you want there.
// mask out a region so the projector goes dark there (e.g. a window).
// draw your art, then paint black polygons over the "keep out" zones.
function applyMask(g, holes) {
g.fill(0);
g.noStroke();
for (const poly of holes) {
g.beginShape();
for (const p of poly) g.vertex(p.x, p.y);
g.endShape(CLOSE); // black = no light hits the real surface
}
}
Masking is what separates a sloppy "big glowing rectangle over everything" from a clean map that hugs the object and leaves the surroundings dark. It's not glamorous, but it's the difference between amateur and tidy.
Getting a person into it
Want to make jaws drop? Let the projection react to whoever's standing in front of it. You don't need fancy hardware for a first taste - the webcam background-subtraction trick from our machine-learning arc is plenty. Grab the camera, compare each frame to an empty "background" frame, and wherever things changed, that's a person. Their silhouette becomes a mask, or a trigger, or a brush.
// crude presence detection: diff the live camera against a stored empty
// frame. big difference = something (someone) is there. drives the art.
function presenceMask(cam, background, threshold = 40) {
cam.loadPixels(); background.loadPixels();
const mask = [];
for (let i = 0; i < cam.pixels.length; i += 4) {
const d = Math.abs(cam.pixels[i] - background.pixels[i])
+ Math.abs(cam.pixels[i+1] - background.pixels[i+1])
+ Math.abs(cam.pixels[i+2] - background.pixels[i+2]);
mask.push(d > threshold ? 1 : 0); // 1 = "person here", 0 = empty
}
return mask; // feed this back into your sketch
}
Touch the wall and ripples spread from your hand. Walk past and the vines lean toward you. This is the doorway into interactive installations - a whole world we'll push much deeper into soon. For now, even a webcam and a threshold gets you a projection that notices you're in the room, and that alone feels like a small miracle.
The unglamorous practicalities
Same as with the LED power warning last time, let me put my serious face on for a minute, because the physical side will make or break your first attempt.
// femdev's projection reality-check (notes-as-code, not a program :-)
const projectionTips = {
brightness: "lumens matter. dark room: 2000+ is fine. any ambient light: 4000+.",
darkness: "the darker the room, the better. black paint on the object helps.",
distance: "short-throw projector for tight spaces - big image from up close.",
surface: "matte, light-coloured surfaces take projection best. gloss reflects.",
keepStill: "bolt the projector down. it moves 1cm, your whole calibration dies.",
viewer: "the illusion is perfect from ONE spot - set up where people will stand.",
};
That last one catches everyone. A corner-pinned projection looks flawless from the position you calibrated it, and gets progressively wonkier as you walk to the side - because you baked one particular viewpoint into the warp. Pros with depth sensors can compensate; for us, you just pick the spot the audience will stand and calibrate for there. Cheap projectors absolutely work for experiments, by the way - don't let anyone tell you you need a fancy rig to start. My first map was a 200-euro projector on a stack of books, and it still made my flatmate come running.
Your exercise: the living box
Time to build the classic, the one every projection-mapper cuts their teeth on: project onto a white box and bring it to life. Grab a cardboard box or a bit of foam core, paint it white or matte-grey, and sit it where your projector can see two or three faces. Then map each face separately.
// the living-box sketch skeleton: one buffer + one quad PER visible face,
// each running a different generative pattern. calibrate, save, enjoy.
const box = {
front: { buffer: null, corners: [], render: drawParticleFlow },
top: { buffer: null, corners: [], render: drawNoiseField },
side: { buffer: null, corners: [], render: drawSlowStripes },
};
function drawBox() {
for (const face of Object.values(box)) {
face.render(face.buffer); // a different sketch on every face
drawMappedQuad(face); // pinned to that physical face
}
}
Give each face a different visual - one face a particle flow (episode 11!), another a slow noise field (episode 12), a third some drifting stripes. Project the calibration grid first, drag the twelve corners (four per face) until every edge sits exactly on the real cardboard, save your calibration, then switch to the real content. The moment all three faces light up in register, that plain grey box stops being a box. It glows. It shifts. It becomes an object that shouldn't exist.
Stretch goals, in rising order of "whoa":
- Make the content obey the box: flow that spills over the top edge and runs down the front face, treating the corners as real geometry.
- Add the webcam presence mask so the box reacts when you wave at it.
- Map a second object in the same scene and have the two "talk" - a pattern that leaps from one to the other.
Do the basic three-face box if you do nothing else. There is a specific, ridiculous joy the first time your code wraps around a real object in the dark - it's the same feeling as your first plot coming off the bed or your first LED strip lighting up, but bigger, because now it's architecture. The screen isn't just behind you anymore. The screen is the room.
't Komt erop neer...
- Projection mapping is warping your image so it lines up perfectly with a real physical surface, so a dumb rectangle of projector light looks like it belongs to the object - like the thing is glowing from inside. The projector is dumb; all the cleverness is the warp you do in code first
- Corner pinning is the workhorse: map your canvas's four corners onto four points on the real surface and stretch everything between. It gets you most of the way for boxes, walls, and flat faces
- You can't just linearly stretch - perspective needs a homography, a 3x3 matrix with a divide-by-w that fakes near-big/far-small. Four corner pairs give eight equations for eight unknowns, and a little Gaussian-elimination solver hands you the matrix. Same math your phone uses to flatten a photo of a document
- The practical p5 route: render your sketch into an off-screen
createGraphicsbuffer, then texture it onto a draggable quad in WEBGL. A plain quad isn't perfectly perspective-correct on steep angles - graduate to the homography-in-a-shader (episode 32 uniforms) orperspective.jswhen your eye complains - Calibration is half the job and it happens in a dark room: drag the corners by eye onto the real edges, project a grid (not your art) so misalignment is obvious, then save the corners to localStorage so you never redo it
- Scale up by keeping a LIST of mapped surfaces - one quad per box face, per shelf, per wall. And make the content respect the architecture: water pools and drips, fire rises, vines climb the real edges. That's what turns a demo into art
- Mask out regions the light should skip (windows, gaps) by painting black - black on a projector is just no light. It's the difference between sloppy and tidy
- Get a person into it with the webcam background-subtraction trick from the ML arc: diff the live frame against an empty one, and the silhouette becomes a mask or a trigger. Touch the wall, ripples spread
- Practicalities decide everything: enough lumens for your room, a matte light surface, and above all bolt the projector down - it moves a centimetre and your whole calibration dies. The illusion is perfect from the one spot you calibrated for, so calibrate where people will stand
So that's four ways now our code has climbed out of the screen and into the physical world - drawn by a pen, cut by a laser, animated in LEDs, and now painted across real objects with light. And notice, again, how little brand-new maths there was today. A perspective transform (cousin of the 3D we met in episode 62), texturing a quad, reading pixels from a webcam, drawing into a buffer - all tools we already had, pointed at a projector instead of a monitor. The theme of this whole chapter keeps holding: the ideas don't change, only where the light lands.
But a projector paints on the outside of things. What if you want your generative shapes to become solid objects you can pick up and hold in your hand - not lit up, not drawn, but real, with weight and edges? That's a different kind of machine, and a different kind of magic, and it's where we're headed next. Bring your curiosity - and maybe start eyeing that white box, because we're not done making the physical world weird :-).
Sallukes! Thanks for reading.
X