Photoshop AI
An overview of the possibilities & pitfalls
By Stephen Romaniello
Artificial intelligence has become embedded in the cultural and intellectual milieu of contemporary life. If you have a question about your water heater, your garden, your pet, or, for that matter, anything, ask OpenAI’s platform, ChatGPT, or Anthropic’s Claude. You’ll get amazing answers with lots of detail, in a soothing, positive voice — and you will be encouraged to dig deeper by the AI platform when it asks questions about your questions. The fact is, like it or not, AI has become a mainstream research tool at universities and scientific institutions and an expert voice in the household. Why? Because it’s efficient and usually accurate, and it has not taken over the world … yet!
AI also has secured a vital role in image processing. Modern graphics software like Adobe Illustrator, Adobe Photoshop, or Corel PaintShop Pro, have embedded a wide range of AI techniques into their workspaces. These techniques aren’t just filters, they’re modeled on machine learning algorithms that automate or enhance complex editing tasks.
Let’s delve into the world of graphics software and see what mechanisms enable AI to modify and enhance images. We’ll also examine various AI techniques that can potentially enrich your workflow. Because of its extensive integration of this new technology, we’ll examine several of Photoshop’s AI features.
Mechanisms
Adobe has embraced AI and neatly integrated it into a lot of its software. But what is it? How does it work? What actually happens when you input text prompts, access Neural Filters or utilize any of the AI features? Here is an overview of the mechanisms that propel AI in the realm of graphics software:
Semantic Image Understanding (SIU): SIUs are algorithms that detect objects, people, backgrounds, and edges in an image. They’re used to automatically select a subject, to remove or replace backgrounds, and to generate masks. In Photoshop, there are several features that employ this technique including the and the Object Selection tool and the Contextual Task Bar with its Select Subject and Remove Background prompts. (Fig. 1)
To examine this in a bit more detail, SIU uses a machine learning technology called Convolutional Neural Networks (CNNs), a deep-learning model designed especially for analyzing images. Instead of looking at an entire image at once, CNNs function like a moving window that scans small regions and detects patterns, edges, textures, and shapes, and builds the new content from its analysis. It first detects edges and color differences, then it recognizes shapes objects like a face or a cat, based on an image database. CNNs technology is also used in face recognition software, medical imaging, and self-driving cars.
Segmentation Models Software: SIU builds a segmentation model to divide an image into meaningful parts. For example, it knows what a cat potentially looks like based on cat images in the database and it determines that “these exact pixels are the cat.” There are two types of segmentations. Semantic segmentation labels every pixel by category. For example, sky, road, mountain, cat etc. Instance segmentation separates, recognizes and labels individual objects. For example, it can distinguish and label individuals in a crowd.
These technologies work together in Photoshop. CNNs are the engine that extracts and recognizes features, and Segmentation Models Software uses those features to map pixels. For example, when you click Select Subject in the Contextual Task Bar, a CNN analyzes the image. A segmentation model identifies exactly which pixels belong to the subject. Then it instantly isolates and labels them. When you generate content with a text prompt, it identifies the area and replaces pixels with resources from the database.
Figure 1. The Object Selection tool and the Contextual Task Bar with its Select Subject and Remove Background prompts.
Figure 2. Using the Select Subject prompt and a simple Generative Fill text prompt “Parrot” transformed my brother, Chuck, into a parrot.
Adobe Firefly
In Photoshop, there are several types of features that utilize different types of AI systems. Generative AI generates entirely new pixels or images based on text prompts. It uses Adobe Firefly: a database of generative AI models designed specifically for creative work. Firefly embodies staples like images, graphics, text effects, and design elements. It is the engine behind many of the newer AI features inside Photoshop, Illustrator, and other Adobe apps. At its core, Firefly is integrated directly into Adobe’s creative tools as a text-to-image and image-to-image AI system built to generate and edit visual content.
Generative Fill
Let’s look at Photoshop’s most popular AI features that utilize these technologies. Generative Fill changes or adds elements to an image. The Select Subject prompt and a simple Generative Fill text prompt “Parrot,” produced the results in Fig. 2. My brother, Chuck, has been replaced by a parrot. Notice that there are three versions to choose from in the Properties panel. Further prompts produce different results. More specific prompts produce more refined results as in Fig. 3, in this case the prompt is “Green Parrot with King’s Crown.”
Generative Expand
This category of AI increases the borders of an image beyond its original borders and automatically fills the new space with realistic content. It’s used to fix bad cropping, reframe for different aspect ratios, or to simply enlarge the picture plane with matching content.
Start by selecting the Crop tool and expand the canvas to the desired size. Then, in the Contextual Task Bar, click Generative Expand. In the Properties panel, choose an option that has been generated by Firefly. (Fig.4) If you aren’t satisfied with the results, repeat the process and different image data will be added to the edges.
Figure 3. More specific prompts will produce more refined results. In this case the prompt is “Green Parrot with King’s Crown.”
Figure 4. Generative Expand enlarges the image’s borders and adds content generated by Firefly.
Generate Image
With this AI feature, you create images from scratch using text prompts. This feature is useful for envisioning concepts and layouts. First, open a new document. In the Contextual Task Bar, describe the image you want to create. In the case of figure five, the prompt, “Mountain Climbers on the Eiffel Tower” was entered. In the Generate Image dialog box you can choose content type, Art or Photo, and from several effects that alter the way the image will appear. (Fig. 5) Click Generate and a group of three images that fit the parameters of the prompt is displayed. (Fig. 6) Choose one, or if you want to see more variations, repeat the process.
AI selection features
Manually selecting areas of an image that require editing is frequently the most time consuming and labor-intensive part of the workflow. This list of AI-driven selection features has replaced manual techniques. Using SIU algorithms, they automatically detect target areas. They are not always one hundred percent accurate and often must be refined manually, but still, they are great time savers.
- Select Subject in the Contextual Task Bar instantly detects a main subject and surrounds it with a selection marquee
- Remove Background found in the Contextual Task Bar is a one-click background remover
- Object Selection Tool in the Tools panel. Hover over an area to auto-detect a person or object. It surrounds it with a pink outline. When the mouse is released, a selection marquee replaces the outline (Fig. 7)
- Select Hair/Refine Edge in the Select and Mask options bar. Automatically handles complex edges like hair, fur or smoke. Selecting ambiguous edges used to take minutes or hours, now it’s done seconds
AI Retouching Tools
From the Edit menu, Content-Aware Fill is a gamechanger. It seamlessly replaces unwanted content with textures, patterns and similarities sampled from the image itself. It is most useful for retouching smaller areas of textures and patterns. It can remove and replace objects like people, wires, and clutter with matching content of the surrounding areas. It’s also used to convincingly extend the edges of an image, similar to Generative Expand but with a little more control. Content Aware Fill replaces hours of cloning, masking, and manual retouching.
Figure 5. In the Generate Image dialog box you can choose the content type, Art or Photo, and from several effects that alter the way the image appears.
Figure 6. To generate this image, the prompt, “Mountain Climbers on the Eiffel Tower” was entered in the Contextual Task Bar.
Figure 7. Hover over an area with the Object Selection tool to auto-detect a person or object.
First, select an area you want to replace. From the interface, choose where the new content is sampled from by painting or erasing the green mask. (Fig. 8)
Two more AI powered retouching tools worth mentioning are the Remove Tool that brushes over unwanted areas and seamlessly replaces them with background content, and the AI enhanced Spot Healing brush that removes blemishes, dust, scratches and small irregularities.
Neural Filters
Found in the Photoshop’s Filter menu, Neural Filters are a set of AI-powered features that use machine learning models, Adobe Firefly, and Adobe’s internal research engine to automatically edit and transform images with multiple techniques that used to require advanced manual skills. Neural Filters analyze the content of your image; faces, lighting, textures, depth, ambiance and apply intelligent adjustments. Instead of manually masking, retouching, painting or colorizing, you adjust sliders. Photoshop handles the complex transformations. Unlike traditional Photoshop tools that require manual control, Neural Filters automate edits using learned patterns from millions of images. The Neural Filter interface (Fig. 9) offers five categories of filters that perform these complex tasks.
Portraits
These filters rely heavily on facial recognition and generative models to realistically rebuild and enhance facial and features.
- Skin Smoothing removes blemishes while preserving natural texture
- Smart Portrait adjusts facial features and expressions like the smile, age, gaze direction and tilt of the head
- Makeup Transfer applies makeup style from one face to another
Creative
These filters apply styles, artistic transformations, or color models to the target image to radically changing their appearance.
- Style Transfer applies the look of famous artworks or reference images to the target image
- Landscape Mixer transfers the colors, textures, and atmosphere from a menu of various landscapes to the target image. The time of day, (Day, Night, Sunset) and seasonal characteristics can be adjusted by dragging the sliders
Color
- Harmonization matches color and lighting between composited elements
- Color Transfer transfers the color theme from a menu of images to the target image
- Colorize automatically adds realistic color to black-and-white photos saving hours of masking and color adjustments
Photography
- Super Zoom enhances resolution using AI upscaling. It also sharpens edges and reduces noise
- Depth Blur simulates a shallow depth-of-field
Restoration These filters are particularly useful for repairing archival or damaged images.
- JPEG Artifact Removal cleans up compression noise and reduces artifact irregularities
- Photo Restoration repairs scratches, improves clarity, and enhances old photos avoiding hours of painstaking retouching
Advantages
There are several advantages to applying Neural Filters to images. First, they speed up the editing process. Tasks that took hours now take seconds. Furthermore, beginners can achieve pro-level edits with considerable ease. Most importantly, Neural Filters are non-destructive. They can be configured as Smart Filters.
Figure 8. From the Content Aware Fill interface, choose where the new content is sampled from by painting or easing the green mask.
Figure 9. The Neural Filter interface offers five categories of filters that perform complex tasks.
Limitations
There are limitations to Neural Filter performance that you should be aware of. For instance, the results can sometimes look unnatural, hyper-smooth, artificial or “AI-ish”, especially when the adjustments are over-applied. Some of the filters require an internet connection for cloud-based processing. Also, their performance depends on your hardware. You need a lot of energy to process these transformations, and device, and cloud processing can be slow.
Why Use AI?
The AI image generating features in Photoshop are fun to play with and can be a source of inspiration to your creative process. But Photoshop’s AI isn’t just about generating images. A lot of it has more to do with efficiency. It speeds up much of the technical work like selecting, masking, retouching, resizing, colorization etc. It reduces manual repetition, letting you focus on creative decisions instead of pixel pushing. AI doesn’t replace skill. It accelerates or eliminates tedious mechanical work, so you spend more time making aesthetic decisions and letting the creative juices flow.
Stephen Romaniello is an artist and educator who has seen the evolution of digital graphics since the beginning of the digital revolution in the 1980s. He is CEO of GlobalEye Systems, a graphics software and creative imaging consultancy, and founder of Gorilla Geeks, a creative consulting firm.
