A sheet of paper photographed on a desk comes back tilted, wider at one end than the other, grey rather than white, and usually with the shadow of the hand that held the phone lying across it. Everybody has taken this picture. Nobody has ever been pleased with it.
The strange part is that all three of those faults are completely determined. The tilt is geometry. The taper is geometry. The grey and the shadow are one lighting function that varies smoothly across the page. None of it involves guessing what should be there, because everything that should be there is already in the photograph, just distributed wrongly.
So this article is the odd one out in a series about running neural networks in the browser. This tool has no model in it at all, and that is the point worth making: not every problem with a picture is a problem for a model, and reaching for one here would be slower, larger and worse.
Four corners are eight equations
The transformation that maps a photographed quadrilateral back onto a rectangle is a projective transform, sometimes called a homography. It is the transformation that straight lines survive and parallel lines do not, which is exactly the property a photograph of a flat surface has.
It is described by eight numbers. Each corner you supply gives two equations, one for x and one for y, so four corners give eight equations in eight unknowns. That system has one solution and it is found by ordinary Gaussian elimination, the method from school, in about a microsecond. There is nothing approximate or iterative about it.
The mapping is then applied backwards, which is the part people find counterintuitive. You do not take each source pixel and work out where it lands, because that leaves gaps between the landing points. You take each destination pixel, run it back through the transform to find where in the photograph it came from, and read the colour there. That location is almost never a whole pixel, so the four surrounding pixels are mixed in proportion to how close each one is, which is bilinear sampling.
Everything above runs in this browser tab on the picture you handed it. Nothing is uploaded, and there is nowhere to upload it to.
Document scan from a photo
A sheet of paper photographed on a desk comes back tilted, wider at one end than the other, grey instead of white, and usually with the shadow of the hand that held the phone lying across it. This straightens the page onto a rectangle, takes the shading out and returns a PNG, one page at a time or a whole stack as a ZIP. The photograph is read inside this browser tab, and there is nowhere for it to be sent.
or press Enter to pick one. JPEG, PNG or WebP.
Sample photo, drawn in this browser.
Nothing is uploaded and nothing is downloaded: the whole thing is arithmetic that runs here, in this tab, on the picture you hand it.
White balance is how much of the estimated lighting is divided back out: nothing at zero, all of it at one hundred. Contrast is neutral at fifty and stretches around mid grey either side of it.
Nothing straightened yet. Put the four corners on the page and press the button.
| What happened | Value |
|---|
Why there is no model in here. Nothing on this page is a neural network and nothing is downloaded when you open the article. Four corners mapped onto a rectangle define a projective transform exactly: eight unknowns, eight equations, solved by Gauss elimination in about a microsecond. Every pixel of the output is then mapped back through that transform and read out of the photograph with bilinear sampling. The shading comes out by dividing the page by a heavily blurred copy of itself, and the black and white threshold is Otsu's, a walk through the 256 possible splits of the histogram looking for the one that separates ink from paper best. Not every problem with a picture is a problem for a model. This one is geometry and statistics, and it finishes before you have let go of the button.
How the corners are guessed. The photograph is reduced to 480 pixels on its long edge, turned to grey, and run through a Sobel filter, which answers one question per pixel: how fast is the brightness changing here. The steepest half per cent of those pixels set the scale, everything at least a third as steep as those counts as an edge, and of that set the tool takes the four extremes of the sums x plus y, x minus y, and their negatives. On a light page lying on a dark surface those four points are the four corners of the page. Measuring against the steepest edges rather than keeping a fixed quota matters more than it sounds: a page of writing holds far more strong edges inside it than along its border, and a quota gets used up by the text long before it reaches the corner of the paper.
What it cannot do. It finds a page against a dark background. White paper on a white table gives the edge finder nothing to hold on to, and then the corners have to be dragged into place by hand, which is why the handles sit on the picture rather than behind a menu. A curled or folded sheet stays curled: a projective transform can flatten a plane, not a curved surface, so the lines near a fold keep their bend. And the shading correction lifts every dark area, not only the shadow you meant it to remove, so a page carrying a photograph or a solid black panel comes back with that area washed out. Turning the white balance down is the answer when that happens.
Dividing the light out
The second fault, the grey page and the shadow across it, has an equally direct solution and it is the one that looks like magic when you see it work.
A photograph of a page is the page multiplied by the light falling on it. The ink is the page and the shadow is the light. If you could estimate the light on its own, you could divide it out and be left with the page.
You can estimate it, and the estimate is embarrassingly simple: blur the photograph heavily. Ink is small and detailed, so a strong blur removes it almost entirely. Lighting is large and smooth, so a strong blur keeps it almost perfectly. What survives a heavy blur of a page of text is essentially a picture of the illumination, and dividing the original by it flattens the page and makes the paper white.
background = heavy blur of the page
result = page / backgroundThe white balance slider is how much of that division to apply, from none to all of it. It is a slider rather than a switch because a page carrying a photograph or a solid dark panel has genuinely dark areas that are not shadow, and the division lightens those too.
Why a heavy blur is a picture of the light
The blur trick deserves a moment on its own, because it is one of those ideas that sounds like a hack and is actually a clean piece of reasoning about scale.
A photograph of a page contains information at two very different sizes. The ink is small: letters a few pixels across, strokes narrower still. The lighting is large: a shadow spans a third of the page, a fall off from a window covers all of it. Nothing about the illumination changes over the width of a letter, and nothing about the text persists over the width of a shadow.
A blur is a low pass filter, which is to say a device that keeps large things and discards small ones. Set its radius somewhere between the two scales, wider than any letter and much narrower than the page, and what comes out is the large component on its own. That is the lighting, isolated, without ever having modelled a light source or known where the window was.
It works because the two things you want to separate happen to live at different sizes. When they do not, the trick fails, and it fails in exactly the predictable way: a page with a large dark photograph on it has content at the same scale as the shadow, so the blur cannot tell them apart and the division treats the photograph as shade.
Otsu, for the black and white output
Turning a cleaned page into pure black and white needs a threshold, and picking one by hand is the kind of setting that is right for one photograph and wrong for the next.
Otsu’s method removes the choice. Build the histogram of the greyscale page, then walk through all 256 possible split points and, for each one, work out how spread out the two resulting groups are. The best threshold is the one where the two groups are internally most similar, which is the same as saying it is the split that separates ink from paper most cleanly. It is a single pass over 256 numbers and it is exact, in the sense that it is the optimum of the thing it optimises.
It fails in an interesting way that is worth knowing. Otsu assumes there are two groups. On a page that is almost entirely paper with three words on it, the histogram has one enormous peak and a barely visible bump, and the method will happily cut the paper peak in half. That is the case where the black and white output looks bizarre and the greyscale output is fine.
Finding the corners without being fooled by the text
Guessing the four corners is the only part of this that is genuinely a heuristic, and the first version of it in this tool was wrong in a way worth describing, because the mistake is a natural one.
The approach is to reduce the picture, convert to grey, run a Sobel filter, which answers one question per pixel about how fast the brightness is changing there, and then take the four extremes of the resulting edge mask. On a light page against a dark surface those four extremes are the four corners of the page.
The obvious way to build the edge mask is to keep the strongest few per cent of pixels. That fails on precisely the pictures this tool is for. A page of writing holds far more strong edges inside it than along its border, so a fixed quota is used up by the text long before it reaches the paper’s corner, and the detected quadrilateral collapses to a sliver somewhere in the middle of the paragraph.
The fix is to measure against the steepest edges rather than to ration by count: take the steepest half per cent as the scale and keep everything at least a third as steep as that, with an absolute floor so sensor noise is never promoted to an edge, and a guard that a mask covering more than a quarter of the picture is not a page. The tool explains this in its own text, and the handles sit on the picture rather than behind a menu because a heuristic that will sometimes be wrong needs to be correctable in one drag.
A stack, because one page is rarely the job
Nobody photographs one page. They photograph a contract, a form, a set of receipts, a letter that runs to three sides, and the awkward part has never been straightening any single one of them: it is ending up with a set of files that belong together.
So the tool keeps a stack. Straighten a page, add it, drop the next photograph in, repeat, and download the lot as one ZIP at the end. The settings stay where you put them between pages, which matters because a run of photographs taken at the same desk under the same lamp needs the same white balance, and re-finding it once per page is the kind of friction that makes people give up halfway.
It is a small feature and it is the difference between a demonstration and something usable on a Tuesday afternoon.
What it cannot do
It finds a page against a dark background. White paper on a white table gives the edge finder nothing to hold on to, and then the corners have to be dragged into place by hand, which is why they are on the picture and reachable with the keyboard rather than hidden.
A curled or folded sheet stays curled. A projective transform can flatten a plane and not a curved surface, so the lines near a fold keep their bend no matter how accurately the corners are placed. That is a mathematical limit rather than an implementation one, and the tools that do handle it are solving a substantially harder problem.
And the shading correction lifts every dark area, not only the shadow you meant to remove, so a page carrying a photograph or a solid black panel comes back with that area washed out. Turning the white balance down is the answer when that happens.
When a model would be the wrong tool
It is worth saying plainly why this one is arithmetic. A model would need to be downloaded, would be several megabytes, would run in tens or hundreds of milliseconds rather than a fraction of one, and would produce a result that is approximately right in a way that cannot be inspected. The transform here is exact, finishes before you have let go of the button, and every step of it can be checked against a number worked out by hand.
The general rule this suggests is not that models are overrated. It is that a model earns its place when the problem contains a judgement that cannot be written down. What was behind the object you removed is a judgement. Which pixel of hair is foreground is a judgement. How a tilted rectangle maps onto a straight one is not a judgement, it is eight equations, and the correct response to eight equations is to solve them.
The same reasoning is worth applying before you reach for any of the others. If the picture is only distorted rather than damaged, straightening it is pure geometry too. If it needs to end up somewhere sensible afterwards, the format matters more than the processing, and PNG is the right answer for a page of text for the same reason it is the wrong answer for a photograph. And if the document you photographed was somebody else’s, it is worth remembering that the picture carries where and when you took it until you take that out.
