Photo Editing

Cutting Things Out: Selections, Masks and the Art of the Edge

Most bad cut-outs are a hard-edge technique applied to a soft-edge problem.

Amateur cut-outs are not spotted by their shape. They are spotted by a halo of the old background and hair that has been shaved off in a curve no head ever had.

A panel with subject selection and background removal tools

Cutting something out of a photograph is the most common editing job there is, and the one where amateur work is easiest to spot. Not because the shape is wrong, but because of the edge: a halo of the old background, a rim of color that should not be there, hair that has been shaved off in a smooth curve no head has ever had.

Getting it right is less about the tool than about knowing which of three problems you are looking at.

Three kinds of edge

Hard edges belong to manufactured things: a phone, a box, a building against a sky. The boundary is one or two pixels wide and either you are on the object or you are not. These are easy, and the mistake is treating them as if they were difficult.

Soft edges come from focus and motion. A shoulder slightly out of focus does not have a boundary at all, it has a gradient twenty pixels wide. Cut it with a hard selection and the object looks pasted on, because you have invented a sharpness the photograph never had.

Partial edges are the hard case: hair, fur, smoke, glass, a wedding veil. Individual pixels are genuinely part background and part subject. No outline can be correct, because the truthful answer is a percentage rather than a yes or no.

Most bad cut-outs are a hard-edge technique applied to a partial-edge problem.

The tools, and what each is actually for

Marquee and lasso

Rectangles, ellipses and freehand outlines. Crude, and still the right answer more often than people expect, because a huge number of jobs are “isolate roughly this area so I can adjust only it”. You do not need a precise outline to darken a corner.

Wand and similar-color selection

Click a pixel, get everything nearby within a tolerance. Excellent on flat, even backgrounds. Fragile on anything else, because tolerance is a single number applied to a picture that has gradients in it. If the background is a studio sweep it works beautifully; if it is a sky that goes from pale at the horizon to deep at the top, it will grab half of it and you will spend longer fixing the selection than drawing one.

Subject detection

Machine learning models trained to recognise what the main object in a frame is, and to return a mask for it. In the last few years these have gone from novelty to first choice for most photographs, particularly for people and animals, because they understand hair as a category rather than as a color boundary.

They fail predictably: multiple candidate subjects, reflections, glass, and anything where the photographer’s idea of the subject differs from the obvious one. Treat the result as a good first draft, not a finished mask.

A panel with subject selection, background removal and edge refinement tools
Detecting the subject and refining the edge are two different operations. The second is what decides whether the result survives being looked at.

Pen and path tools

Slow, precise, and unbeatable for manufactured objects with clean curves. If you are cutting out a car, a bottle or a piece of furniture for a catalogue, a path will beat every automatic method, because the correct edge is geometric and you know where it is even where the photograph is ambiguous.

Feathering, and why the default is wrong

Feather softens a selection edge by blending across a few pixels. Every tool offers it, usually at zero by default, and zero is almost never right.

Real photographs have no infinitely sharp edges. Lenses have finite resolution, sensors have finite pixels, and even a perfectly focused hard object transitions over a pixel or two. A selection with no feather at all produces the jagged staircase everyone recognises as a bad cut-out.

Practical amounts: half a pixel to one pixel for a sharp object at full resolution, two or three for something slightly soft, and considerably more for a genuinely out of focus edge. Match the feather to the blur that is already in the picture and the cut becomes invisible.

The color fringe nobody mentions

Here is the detail that separates convincing composites from obvious ones. A pixel at the boundary of a subject standing in front of a green wall is not pure subject; it contains some green. Cut a perfect outline and those contaminated pixels come with you, giving your subject a thin green rim on its new background.

The fix is decontamination: replace the color of edge pixels with a color sampled from just inside the subject, while keeping their transparency. Most serious edge refinement offers this, sometimes as a separate control. It matters most when the old and new backgrounds differ strongly in color, which is most of the time.

If a cut-out looks wrong and you cannot say why, look at the edge pixels rather than the shape. It is almost always contamination or a missing feather.

Masks, not the delete key

The single most valuable habit in this whole subject: never delete the background. Turn the selection into a mask.

A mask is a grayscale image attached to a layer, where white means visible, black means hidden and gray means partially visible. Nothing is destroyed. The pixels are still there and can come back at any point, which matters more than it sounds, because a cut-out is almost never right the first time.

It also gives you a way to work that no selection tool can match: you can paint on the mask. Painting with a soft brush at low opacity lets you bring back a wisp of hair here, hide a stray pixel there, and gradually converge on something believable. Selections are made in one shot; masks are refined over several passes.

A selection menu with marquee, lasso, smart selection and feathering
Saving a selection sounds like housekeeping until the moment you need the same outline again three steps later.

Order of operations

A sequence that works for almost any subject:

  1. Get a rough selection fast, by whatever means. Do not perfect it.
  2. Convert it to a mask straight away, before doing anything else.
  3. Refine the edge: feather to match the photograph’s own softness, then decontaminate color.
  4. Put the subject on the new background at this point, not later. Edge problems are invisible against the old background and obvious against the new one.
  5. Paint on the mask to fix what you can now see.
  6. Check at 100 per cent and at thumbnail size. Different faults show at each.

The bit that is not technique

A cut-out fails just as often for reasons that have nothing to do with the edge. If the subject was lit from the left and the new background is lit from the right, no amount of masking will save it. If the subject was shot with a long lens and the background with a wide one, the perspective will not agree.

Before you start cutting, look at the light direction, the color temperature and roughly how compressed the perspective is. Where they disagree badly, fix that first, with a curve or a flip, or choose a different background. A technically perfect mask on a fundamentally mismatched composite still looks wrong, and viewers who could never explain why will simply not believe the picture.

Table of Contents

Learn it by building something

Every week one thing you can make the same afternoon, from dynamic templates to 3D type. Written down step by step.

One mail a week, and then it ends.
Unsubscribe in one click.

Free

Chaos Art

Autonomous painters make one-of-a-kind abstract art in 3D space - gestures, art movements, painterly media, and embeds that paint a new original for every visitor.

Download the free WunderPaint Plugin for WordPress

The Image Editor & Design Studio

Everything described here can be done in the browser, on your own site. The live demo runs the full editor with nothing to install.

Design Tools

Brush & Paint Styles

Paint with six styles that mix pigment instead of blending it, pick from thirty-seven tips shown as real strokes, draw a tip of your own out of shapes, and let the color travel along the stroke.

Workflow & Automation

Dynamic Templates: Build the Design Once, Let Your Posts Fill It In

You build the layout once, tell a few layers where to get their content, and from then on every post, product or page renders its own version.

Print & Craft

Making Things That Fold: Dielines, Duplex and Why Previews Matter

A gift box, a paper crane, a bunting banner. They look simple, and they fail in ways flat design never does, because there is a second geometry underneath.

WordPress Images

Why Should a Design Become a File at All?

Every static graphic on a page is a snapshot of what was true when somebody exported it. There is a whole category of quiet wrongness that follows from that.