Behind the mask
From a simple selection to a complex smart filter, masks are a big part of the digital workflow
By Stephen Romaniello
Story and images by Stephen Romaniello
Behind the mask
From a simple selection to a complex smart filter, masks are a big part of the digital workflow
By Stephen Romaniello
Story and images by Stephen Romaniello
Andres Mejia - stock.adobe.com
Isolating elements on a picture is an essential image editing skill. Accurate selections determine the quality of an edited image and can be among the factors that make or break the credibility of a manipulated picture. This article explores the types of masks available in mainstream image-editing software packages including Corel Paintshop Pro, Gimp, (a free, open-source package), Adobe Lightroom, Adobe Photoshop, and a few AI solutions. The techniques produce similar results from one package to the other. Only the interfaces are different. The software you choose will depend on what you are most comfortable with. I’ll be demonstrating masking techniques that assure seamless editing for accuracy and believability using Adobe Photoshop 2026.
What’s a mask?
According to my online dictionary, the first and most obvious definition of the word mask is “a form of disguise or concealment usually worn over or in front of the face to hide a person’s identity.” Of course, this term’s meaning has evolved to include several different applications, but as far as graphic software is concerned, masks cover two important functions; they hide or reveal image content, and they protect or expose areas of the image to editing processes.
Masks before computers
In the old days, before computers, a mask was a piece of semitransparent red film called Rubylith, (Fig. 1) designed to conceal portions of an image from the process camera’s light. Another type of mask was a sheet of frisket film used to shield a photo from the over-spray of an airbrush. These masks were cut by an expert hand wielding a razor-sharp X-Acto knife to ensure the exposed areas were precise. Though the process was time and labor intensive, believe it or not, all sorts of interesting effects could be achieved this way, including sharpening, blurring, increased contrast, solarization, and many other creative effects.
All things considered, Rubylith and frisket performed well in the realm of analog images, but compared to the digital masks of today, Rubylith and frisket are limited in their capabilities.
Software
Among image editing software’s mask-making features are a group of selection tools, channel-based masks to store selections, layer-based masks that conceal and reveal layer content, vector masks that are composed of paths or shapes, type masks that create character-based selection outlines, and filter masks that reveal and conceal the effects of filters. In addition, in some of the programs, there are AI features that perform automatic masking with a text prompt and the click of a mouse.
Figure 1. Rubylith is a semitransparent red film designed to conceal a portion of an image from the process camera’s light.
Figure 2. Most masks are displayed as either a black and white thumbnail in one of the panels, or as a colored overly on the image.
Mask basics
Digital masks share similar characteristics depending on the type of mask and the current viewing mode. Most are displayed as either a black and white thumbnail in one of the panels, or as a colored overlay on the image (Fig. 2).
Masks can be modified with specific tools or commands. Each image can present a different set of selection and editing problems. Consequently, masking techniques are designed to cover any imaginable situation. Masking can be quite complex and labor-intensive and sometimes situations arise where one masking technique is not quite sufficient and must be combined with other masking techniques.
Fortunately for the current generation of graphic artists and technicians, software has streamlined the masking process. Isolating image elements is now much faster, easier, and more precise. Each opened image can present a different set of selection and editing problems, consequently, built-in masking techniques are designed to cover any imaginable situation.
Figure 3. A selection outline is made with one of the selection tools found in the Tools panel.
Figure 4. The Select and Mask feature provides precise control over the mask’s edges.
The ants go marching
The most basic masks are temporary selections that define an area for instant editing. These selection outlines are sometimes called marching ants because the animated marquee looks like a column of tiny insects encircling the selected region. A selection outline is made with one of the selection tools found in the Tools panel (Fig. 3). These outlines enable the region within their boundaries to be edited by a tool or command. The area outside the outline remains protected from editing.
The 11 selection tools in Photoshop can be manual, requiring the hand encircling of a specific region, or automatic, requiring only the click of a mouse on a specific region. There are also AI text prompts that automatically select a subject or background.
Edges
One primary advantage of digital masking is the ability to control the edge of an affected area. The edge can be modified in several ways including feather: the softening of the outermost edge to gradually fade to transparency; border: modifying the selection edge to a specific width; smooth: the softening of stair-stepped pixels; expand: made bigger by a specific number of pixels; or contract: made smaller. When a selection is active, these options are found in the Select/Modify menu.
The Select and Mask feature (Fig. 4) provides even more precise control over the mask’s boundaries. Select and Mask works on both newly made selections and existing masks. It has a series of tools that refine the edges of an existing selection or mask akin to the Modify menu but with on-the-fly visual control. The most minute details such as the hair or fur can be included or excluded in the edge with these tools. There are also two AI controls: Select Subject and Refine Hair, that automatically simplify making difficult selections of ambiguous edges.
Quick mask
The selection tools are not by any means the only way to make selections. Sometimes they are not precise enough to do the job and need to be “custom refined” even before entering the Select and Mask workspace. Quick Mask is a fast and reliable way to refine a selection on the fly that takes advantage of the versatility of the Brush tool. After the initial selection is created, click the Quick Mask icon at the bottom of the Tools panel or press the Q key. The areas surrounding the marching ants are converted to a red colored, semitransparent field that, incidentally, resembles Rubylith (Fig. 5). This mask can be manipulated with the paintbrush and all the numerous brush characteristics that the software provides. Painting with black applies the red mask color to the selection. Painting with white, erases the mask color. Painting with any shade of gray produces semitransparent areas of the selection, depending on how dark or light the shade is. When the icon is clicked again (or the Q key is pressed) the quick mask is displayed as marching ants complete with revisions, ready for effects to be applied.
Figure 5. The areas surrounding the marching ants are converted to a red colored, semitransparent field that, incidentally, resembles Rubylith.
Figure 6. Selections can be saved as Alpha Channels.
Alpha channels
Creating masks can be a labor-intensive process in the image editing workflow and therefore, it is important to save the selection mask after it’s been drawn and refined. Completed masks can be saved as Alpha Channels (Fig. 6). Display the Channels panel and Click the Save Selection as Alpha Channel icon at the bottom of the panel. The selection is saved within the document and named Alpha 1. To load the selection press Command (Mac) or Control (Windows) and click the channel thumbnail. The marching ants appear, and the area is ready for editing.
Alpha channels are used far less frequently than in the past. Layer masks, vector masks, smart filter masks and layer styles have replaced many of the alpha channel techniques. Still, there are times when they can be quite useful in combining and duplicating selection marques.
Layer mask
The other types of masks that are commonly used in the nonlinear workflow are Layer masks. Where a selection marquee isolates a region for editing, layer and vector masks conceal and reveal areas of the image. Layer masks can be made by encircling a region of an image with a selection marquee and then clicking the Add Layer Mask icon at the bottom of the Layers panel. The Layer mask can then be painted or filled with white to reveal areas, or black to conceal areas (Fig. 7). If gray is painted on the mask, the area will be partially concealed or revealed depending on how dark the shade is.
Vector masks
The difference between layer and vector masks is how they are made and edited. Vector masks are made with the Path tools and edited with the Path Selection or the Direct Selection tool. The layer displays a gray and white mask thumbnail. Gray represents the area that is concealed and white represents area that is revealed. The path is the boundary between what is seen and what is concealed (Fig. 8).
Figure 7. Regions on the Layer mask can then be painted or filled with white to reveal areas, black to conceal areas, or gray to partially conceal areas.
Figure 8. With a vector mask, the layer displays a gray and white mask thumbnail. Gray represents the area that is concealed and white represents area that is revealed.
Figure 9. Areas of white on the mask are where the filter is applied. Areas of black are protected.
Figure 10.
Smart Filters
When a Smart Filter is applied to a layer, a mask is created. First, convert the layer to a smart object by choosing Convert to Smart Object from the Layer Options menu. Select an area where you want the filter to be applied. Then choose a filter from the Filters menu. Areas of white on the mask are where the filter is applied. Areas of black are protected, (Fig. 9) and areas of gray are partially protected. Refinements can be made by selecting the mask and painting with black, white, or gray.
AI masking
I would be remiss if I were to not mention the new AI masking technique that accelerate the workflow. When you open an image, the Contextual Task Bar (CTB) displays a prompt: Select Subject (Fig. 10). When clicked, the main subject in the image is automatically selected. If there is more than one object in the image the feature selects all the prominent subjects in the image and the selection will require editing to eliminate unwanted areas. Still, this may be a good place to start the masking process. The selections made by the CBT are based on the contrast of edge pixels in the image and have nothing to do with any intuitive understanding of the subject by the artificial intelligence.
Important skill
Masking is possibly the most important skill to learn in the image editing workflow. It certainly can be the most hands on. Thankfully, gone are the days of Rubylith, frisket, X-Acto knives, and process cameras. Instead, masks are available at your fingertips in image editing software packages. Learning to make accurate masks can be challenging because there are so many variables that determine what techniques to use. Of course, experimentation with the many mask features will get the perfect results you want. I hope this article has helped declaw creative masking. Naturally, using these tools and techniques will greatly enhance your ability to create amazing credible pictures, so give them a try.
Stephen Romaniello is an artist and educator who has taught graphics software for over 30 years. He is a certified instructor in several graphics and video applications and the author of numerous books and articles on the creative image manipulation, illustration and media programs. Stephen is the founder and CEO of GlobalEye Systems, a company that offers training and consulting in graphics software. He is also the founder of Fire Boy Productions, an art-book publishing company.
