Perceptual image analysis

Perceptual image analysis

Photos can be altered in multiple ways, some of them leave traces that can be revealed, commonly unless such altered photos get degraded by subsequent resavings (that frequently happens during sharing over the web though). Regardless of the algorithms-based analyzing of images, direct visual inspection for irregularities (e.g. a lack and/or oversupply of shadows and lights) can reveal a lot by itself.

This extension provides several ways of analysis. Besides extraction of meta data, it is:

directly by increasing contrast between tones (of possibly aligned slices),
PCA, via use of a global transformation that rotates within the color space;

direct search for altered local variations of colors and luminosity
(that can occur e.g. on borders of altered regions),
error-level analysis that highlights differences at variations (of colors)
that tend to occur between original vs. altered parts of images.

Images that are (claimed to be) original photos, should be searched for Exif information and for signs of alterations. Notice that here we deal with indirect evidences only, since anything can get faked. Even if an image (taken from web) is of a low quality, search for its occurrences over the web can help in its (in)validating, since it may had been put on web previously (at a different form).

The Perceptual image suite provides extensions that help with searching out image alterations. Along with this suite, there are other sites that provide free services, tools for and/or information on dealing with image analysis too. They are e.g. the Metadata Viewer, the FotoForensics (with blog), the Forensically (with blog), the Image Forensics for GIMP (code), the Ghiro, the izitru sites, and searching at ResearchGate (questions).

To help with using this extension, a few keyboard shortcuts are set. They are "Control <" and "Control >" for passing within the main menu, "Control comma" and "Control period" for: passing among the (sub)sections at the Meta and About parts, and for switching among views of the original photo, original-result overlay and the result image at the other (that is Levels/PCA/Slopes/ELA) parts. There the "Control semicolon" shortcut is for switching between non-scaled and scaled image views.

If the analyzed image contains Exif (that is meta information), it is extracted and presented in the exif section of the Meta part. If there is a thumbnail included in the image, it is presented as well (in the section below the textual Exif). Notice that the extraction of both the textual Exif and the thumbnail is done by a third-party library: exif-js.

The file section in the Meta part contains URL of the image and the headers that the respective web server sent (in case of an image taken from web), or image file name along with info on its last save time and size (in case of an image loaded locally).

When dealing with images taken by URLs, then at some cases the respective servers provide the last-modified header. It can be helpful on guessing when the image was taken (at the latest), though with some precautions: it should just mean the date when the image was put on (or last changed while being at) the server.

The levels part moves the image colors at a selected center value, and makes more light, dark, parts lighter, darker, respectively; thus separating color (light) levels around the chosen center value.

This is a quite simple technique that may help to find alterations in the analyzed image if the alterations are e.g. based on removing a slice from the image, with aligning the (originally displaced) parts together.
Notice that the spread option is for an approximation of the histogram equalization technique.

At some cases a ghost-like part can arise during changing the center value and/or the contrast: mostly when both the altered part and its surrounding are more light (or both more dark) than the real center luminosity value, since the forger may miss such a difference then. Notice that such cases are rare though, since software products used for image alterations tend to deal with the respective luminosities.

The PCA part uses the principal component analysis technique. It starts with putting individual pixels into the color space, that is a 3-dimensional space where axes are individual base colors (here red, green and blue) regardless of actual positions of the displayed stuff in the image.

Then it assumes that pixels (of a single image) form roughly baguette-like shaped cloud in the color space. And it aligns the color-wise axes along the main dimensions of the (expectedly baguette-like) cloud. The new axes – that is the main directions of the cloud – are called principal components. Aligning along these directions is the rotation of the PCA, here rotating from axes of individual red, green, blue colors into the directions of the main mixed colors of the expected image cloud.

By that we get the main variance within the colors along the first component. The second and the third components have then (lesser) variances within colors of the image that are independent from variances of the first component (and mutually too).

While the first component commonly provides roughly grayscaled version of the image, the second and the third components may expose irregularities within the image. The ←→ mode is for highlighting of outliers (within the colors), the spread modes are for highlighting of (unfitting) color changes within the image.

The outliers can be e.g. artificially brightened lips/teeth on portraits, and unfitting color changes can be artifacts due to edits of the image (that are not visible in the image itself, since the larger color variation along the first component effectively hides it). Notice that if a (lesser) region within the analyzed image contains atypical colors (with respect to the rest of the image), then such a region will be displayed as an outlier even if those colors are real though.

The PCA method is not applicable to images where it can not make the rotations. Thus it is not displayed for effectively single-color images, like those using e.g. blue color only, nor for grayscaled images.

Changes of colors are a common target of searches for signs of image alterations. The slopes part deals with local (pixel-wise) variations. The (pixel-wise) variations that are considered can be first, of any form, second, when they go a steady way over the pixels (sort-of approximation of the first derivative), or third, when the forms of variations evolve in a grainy/bumpy way (sort-of approximation of the second derivative). The blur mode is for differences between the image and its (generally) blurred version then.

A chance is that placing (several) image parts together would lead to altered local variations of colors at the regions of contacts; that tends to get vanished via image resaving though. Notice that signals (of image alterations) from the slopes part tend to be complementary to signals from the ELA part, though the ELA method commonly provides stronger signals (if it detects an alteration at first).

The changes are evaluated per-color, unless the luminosity option is set on – for considering variations of the overall lights. Apart from the blur mode, the luminosity way displays horizontal (red colored) vs. vertical (green colored) changes separately.

The error-level analysis method looks for dynamic properties of color variations, that is differences between the actual state and the image after it is blurred by resaving. Notice that while ELA may provide clear signals on image alterations when the images are of a high resolution, it commonly provide false signals on low-quality images.

ELA takes the analyzed image, exports it into JPEG, and computes the difference between the initial and the final state. Since common browsers provide methods for export into JPEG, this method can be implemented reasonably in JavaScript by reusing those methods.

By employing the JPEG export at ELA, the JPEG format itself can introduce artifacts, especially if the analyzed image is of a low quality. Such (low-quality) images bear color variations just due to the JPEG format, not necessarily due to any image alterations. Using the ELA method on such images can be equal to reading from coffee dreg. Notice that textures within images tend to lead to mean signals even at other images (of a middle quality).

To limit a load of the system while the blur parameter is slid, it has a delay set before it is applied.
During our testing of the error-level analysis, most of the cases when it (correctly) found a signal were for the blur parameter (used as 1-quality for the JPEG export) either varied around cca 0.10 or around 0.25 values, YMMV.

Image loading

This extension allows loading of images from websites via mouse-button menu. Within the extension page, images can be loaded both from local files (drag/drop or bottom-left button, Control o), and from URLs, see the input at bottom of this section.

Image search

Since search for images similar to the handled one is a part of image analyzing, the Perceptual image suite provides an extension that provides direct submitting of URLs and (modified) images to common search providers. Notice that whole-image modifying may be necessary for a successful search at some cases, since the analyzed image might had gone through some image-wise alterations at first.