• Making Photos Smaller Without Quality Loss
  • By Stephen Arthur
  • The Nuggets translation Project
  • Translator: Xat_MassacrE
  • Proofreader: Meifans, WindmXf

How to make the image smaller without losing it

Yelp already has more than 100 million user-posted photos, including photos from dinner, haircuts, and more, as well as our new feature # Yelfies. These images take up most of the bandwidth of users’ apps and websites, and also represent significant storage and transmission costs. In order to give our users the best user experience, we did our best to optimize our images, resulting in an average image size reduction of 30%. This not only saves time and bandwidth for our users, but also reduces our server costs. Oh, and the point is that our process is completely intact!

background

Yelp has been storing images uploaded by users for 12 years. We save PNG and GIF as PNG in lossless format and other formats as JPEG. We use Python and Pillow to save images, so let’s start by uploading images directly:

# do a typical thumbnail, preserving aspect ratio
new_photo = photo.copy()
new_photo.thumbnail(
    (width, height),
    resample=PIL.Image.ANTIALIAS,
)
thumbfile = cStringIO.StringIO()
save_args = {'format': format}
if format == 'JPEG':
    save_args['quality'] = 85
new_photo.save(thumbfile, **save_args)Copy the code

Let’s look at some ways to optimize file size without loss.

To optimize the

First we had to decide whether we went to ourselves or to a MAGICALLY change CDN provider to process our pictures. With the emphasis on quality content, it’s important to evaluate options and make trade-offs between image size and quality. Let’s take a look at some of the current image file size reduction methods, what changes we can make, and how much we can reduce the size and quality of each method. After completing the study, we decided on three main strategies. The rest of this article explains what we did and the benefits we gained from each optimization.

  1. Changes in the Pillow
  • Optimize the flag
  • Progressive JPEG
  1. Change the photo logic of the application
  • Big PNG detection
  • JPEG dynamic quality
  1. Replacing a JPEG encoder
  • Mozjpeg (Raster quantization, custom quantization matrix)

Changes in the Pillow

Optimize the Flag

This was one of the easiest changes we made: turn on the Pillow setting that saves extra file sizes at the cost of CPU time (optimize=True). Since the essence has not changed, this has no effect on the quality of the image.

For JPEG, the pair option tells the encoder to find the best Huffman encoding by performing an additional scan on each image. The first time, instead of writing to the file, the number of occurrences of each value is counted, along with the information necessary to calculate the desired encoding. PNG uses Zlib internally, so in this case the optimization option tells the encoder to use GzP-9 instead of GZP-6.

This was a simple change, but it turned out to be no silver bullet, as the file size was only reduced by a few percent.

Progressive JPEG

When we save an image as a JPEG, you can select different types from the following options:

  • Standard: JPEG images load from top to bottom.
  • Progressive: JPEG images load from blur to clear. The progressive option can be easily enabled in the Pillow (progressive=True). This is a noticeable performance improvement (that is, half-loaded images are more noticeable than half-loaded images that are not crisp).

There is also a small compression when progressive files are packaged. For a more detailed explanation, see Wikipedia’s article, the JPEG format uses a sawtooth pattern for entropy encoding on 8×8 pixel blocks. When the values of the blocks are unpacked and unpacked in order, you’ll find that the non-zero numbers usually come first, followed by the sequence of zeros, and that pattern will scan every 8×8 block of the image interlaced. With progressive encoding, the order of the unpacked blocks of pixels gradually changes. The larger values in each block will appear first in the file (the most differentiated areas of the image loaded in progressive mode will be scanned first), and a long string of small numbers, including many zeros, will be loaded last to fill in the details. This rearrangement of image data does not change the image itself, but it does increase the number of zeros in a line that can be more easily compressed.

A comparison of pictures of a delicious donut (click to enlarge) :

A mock of how a baseline JPEG renders.

Simulates the rendering of standard JPEG images.

A mock of how a progressive JPEG renders.

Simulate the rendering effect of progressive JPEG images.

Change the photo logic of the application

Big PNG detection

Yelp allows users to upload images in two main formats – JPEG and PNG. JPEG is a great format for photos, but not so great for high-contrast design content, like logos. PNG, on the other hand, is completely lossless, so it’s great for graphic-type images, but too big for images that don’t differ significantly. If the PNG image uploaded by the user is a photo (as identified by us), using THE JPEG format saves a lot of space. Typically, PNG images on Yelp are screenshots from mobile devices and “Meitu” apps.

(left) A typical composited PNG upload with logo and border. (right) A typical PNG upload from a screenshot.

(left) an obvious PNG composite. (Right) a screenshot of an apparent PNG.

We want to reduce the number of these unnecessary PNG images, but it is important to avoid over-interfering, changing the format or degrading the image quality. So, how do we recognize an image? Through pixels?

Using a sample of 2500 images, we found that the combination of file size and individual pixels worked well for us. We generate our candidate thumbnails at maximum resolution and then see if the output PNG file is larger than 300KB. If so, we check to see if the image content has more than 2^16 individual pixels (Yelp will convert RGBA images to RGB, even if they don’t).

In the experimental data set, manually adjusting the values that define large images can reduce the file size by 88% (that is, the storage we would expect to save if we converted all images) without harming the images.

JPEG dynamic quality

The first and most well-known way to reduce JPEG file size is to set quality. Many applications save jPeGs with a specific quality value.

Mass is actually a very abstract concept. In fact, each color channel of a JPEG image has a different quality. Quality levels from 0 to 100 correspond to different quantization tables on different color channels, and also determine how much information is lost. Quantization in the signal domain is the first step in losing information in JPEG encoding.

The easiest way to reduce file size is to reduce image quality and introduce more noise. But at a given quality level, not every image loses the same amount of information.

We can dynamically set the optimal quality level for each image, finding a balance between quality and file size. We can do this in two ways:

  • Bottom-up: These algorithms process images at the 8×8 pixel block level to generate tuned quantization tables. They calculate both theoretical mass loss and human visual information loss.
  • Top-down: These algorithms compare an entire image to its original version and detect how much information is missing. By continuously generating candidate images with different quality parameters, the one with the least loss is selected.

We evaluated a bottom-up algorithm, but so far it has not yielded satisfactory results in our experimental environment (although it seems to have a lot of potential for processing medium quality images, where more information can be discarded). Many academic papers on the algorithm were published in the early 1990s, but in this era of expensive computing power, bottom-up algorithms took shortcuts, such as failing to evaluate the interactions between pixel blocks.

So we choose the second method: use dichotomy to generate candidate images at different quality levels, and then use Pyssim to calculate its structural similarity matrix (SSIM) to evaluate the quality loss of each candidate image until this value reaches the non-statically configurable threshold. This approach allows us to selectively reduce file size (and file quality), but only for images that users would not notice if their quality were reduced.

In the chart below, we plot the SSIM values of 2500 images generated by 3 different quality levels.

  1. The blue line is zeroquality = 85The original graph generated.
  2. The red line is zeroquality = 80Generated graph.
  3. And finally, the orange diagram is the dynamic mass that we ended up using, with the parameterSSIM 80-85. Select a quality value between 80 and 85 inclusive for an image based on the convergence point or over the SSIM ratio (a static value calculated in advance so that the conversion occurs somewhere in the middle of the image range). This method can effectively reduce image size without breaking the bottom line of our image quality requirements.
SSIMs of 2500 images with 3 different quality strategies.

SSIM values of 2500 pieces of 3 different quality strategies.

SSIM

There are a number of image quality algorithms that can simulate the human visual system. After evaluating many methods, we believe that SSIM, although relatively old, is the most suitable method for iterative optimization of these features:

  1. Sensitive to JPEG quantization error.
  2. Fast, simple algorithm.
  3. Computations can be done on PIL native image objects without converting images to PNG format, and can also be run from the command line (see #2).

Example code for dynamic quality:

import cStringIO import PIL.Image from ssim import compute_ssim def get_ssim_at_quality(photo, quality): """Return the ssim for this JPEG image saved at the specified quality""" ssim_photo = cStringIO.StringIO() # optimize is  omitted here as it doesn't affect # quality but requires additional memory and cpu photo.save(ssim_photo, format="JPEG", quality=quality, progressive=True) ssim_photo.seek(0) ssim_score = compute_ssim(photo, PIL.Image.open(ssim_photo)) return ssim_score def _ssim_iteration_count(lo, hi): """Return the depth of the binary search tree for this range""" if lo >= hi: return 0 else: return int(log(hi - lo, 2)) + 1 def jpeg_dynamic_quality(original_photo): """Return an integer representing the quality that this JPEG image should be saved at to attain the quality threshold specified for this photo class. Args: Original_photoa prepared PIL JPEG image (only JPEG is supported) "" ssim_goal = 0.95 hi = 85 lo = 80 # working on a smaller size image doesn't give worse results but is faster # changing this value requires updating the calculated thresholds photo = original_photo.resize((400, 400)) if not _should_use_dynamic_quality(): default_ssim = get_ssim_at_quality(photo, hi) return hi, default_ssim # 95 is the highest useful value for JPEG. Higher values cause different behavior # Used to establish the image's intrinsic ssim without encoder artifacts normalized_ssim = get_ssim_at_quality(photo, 95) selected_quality = selected_ssim = None # loop bisection. ssim function increases monotonically so this will converge for i in xrange(_ssim_iteration_count(lo, hi)): curr_quality = (lo + hi) // 2 curr_ssim = get_ssim_at_quality(photo, curr_quality) ssim_ratio = curr_ssim / normalized_ssim if ssim_ratio >= ssim_goal: # continue to check whether a lower quality level also exceeds the goal selected_quality = curr_quality selected_ssim = curr_ssim hi = curr_quality else: lo = curr_quality if selected_quality: return selected_quality, selected_ssim else: default_ssim = get_ssim_at_quality(photo, hi) return hi, default_ssimCopy the code

Here are some other blogs about this technology, this one by Colt Mcanlis. Etsy published one too! Go and have a look!

Replacing a JPEG encoder

Mozjpeg

Mozjpeg, an open source offshoot of libjpeg-Turbo, is an encoder that swaps file sizes by execution time. This method is perfect for offline batch regenerating images. It takes 3 to 5 times longer than libjpeg-Turbo, and a bit more sophisticated algorithms to make images smaller!

One of the biggest differences in MozJPEG is the use of an additional quantization table. As mentioned above, quality is an abstract concept for each color channel quantization table. All signal points of the default JPEG quantization table are easy to hit. In the words of the JPEG guide:

These tables are for reference only and cannot be guaranteed to be applicable in any application.

So it is not surprising that most encoder implementations use these tables by default.

Mozipeg has taken the hassle out of using benchmark selection tables and created images using the best-performing generic alternative.

Mozjpeg + Pillow

Libjpeg is installed by default on most Linux distributions. So by default mozJPEG is not available in Pillow, but it’s not that hard to configure. When compiling with MozJPEG, use the –with-jpeg8 parameter and make sure the Pillow links and finds it. If you use Docker, you can also write a Dockerfile like this:

FROM ubuntu:xenial RUN apt-get update \ && DEBIAN_FRONTEND=noninteractive apt-get -y --no-install-recommends install \ #  build tools nasm \ build-essential \ autoconf \ automake \ libtool \ pkg-config \ # python tools python \ python-dev \ python-pip \ python-setuptools \ # cleanup && apt-get clean \ && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # Download and compile mozjpeg ADD https://github.com/mozilla/mozjpeg/archive/v3.2-pre.tar.gz/mozjpeg - SRC/v3.2 - pre. Tar. Gz RUN tar -xzf /mozjpeg-src/ v3.2-pre-tar. gz -c /mozjpeg-src/ WORKDIR /mozjpeg-src/mozjpeg-3.2-pre RUN autoreconf -fiv \ && ./configure --with-jpeg8 \ && make install prefix=/usr libdir=/usr/lib64 RUN echo "/usr/lib64\n" > /etc/ld.so.conf.d/mozjpeg.conf RUN ldconfig # Build Pillow RUN pip install virtualenv \ && virtualenv /virtualenv_run \ && /virtualenv_run/bin/ PIP install --upgrade PIP \ && /virtualenv_run/bin/ PIP install --no-binary=:all: Pillow==4.0.0Copy the code

That’s it! Once built, you can use the Pillow library with Mozipeg in your image processing workflow.

impact

So how much improvement did these methods bring? Let’s take a look at 2500 random images from Yelp’s photo library and use our workflow to see how file sizes change:

  1. Changing the Pillow Settings will reduce this by 4.5%
  2. Large PNG detection can reduce by 6.2%
  3. Dynamic mass can be reduced by 4.5%
  4. Change to MozJPEG encoder can reduce 13.8%

All of this adds up to an average image size reduction of about 30%, and when we apply it to the largest and most common image resolutions, not only does our web pages get faster for users, but we also save terabytes of data per day on average. This can be seen from CDN:

Average filesize over time, as measured from the CDN (combined with non-image static content).

Trend chart of time change versus average file size on CDN (including static content that is not a picture).

What we didn’t do

The purpose of this section is to introduce some other improvements that you might use, which Yelp doesn’t cover because of the toolchain we chose and some other trade-offs.

The second sampling

Secondary sampling is the main factor that determines the image quality and file size of web pages. Detailed instructions on secondary sampling can be found online, but for this blog the short version is that we have already used 4:1:1 secondary sampling (the default for Pillow in general), so we can’t get any improvement here.

Lossy PNG coding

After seeing what we did with PNG, you can choose to save a portion of the image as PNG using a lossy encoder like PNGMini, but we chose to save the image as JPEG. This is another good option that reduces the file size by 72-85% without the user making any changes.

Dynamic format

We are considering supporting more new image types such as WebP and JPEG2k. Even when scheduled projects come online, the long tail effect of user requests for optimized JPEG and PNG images continues to work, making the optimization still worthwhile.

SVG

SVG is used in many places on our website, such as static resources designed by our designers as style guides. This format, along with optimization tools like SvGo, can significantly reduce web page load, but it’s irrelevant to what we’re doing here.

The magic of suppliers

There are a number of vendors that can transfer, resize, crop and transcode images. Including the open source Thumbor. This is probably the easiest way for us to support responsive images, dynamic formatting and retaining borders in the future. But in the current situation our solution is sufficient.

read

The following two books definitely have something that they don’t mention in their blogs, and are highly recommended extensions to today’s topic.

  • High Performance Images
  • Designing for Performance

The Nuggets Translation Project is a community that translates quality Internet technical articles from English sharing articles on nuggets. Android, iOS, React, front end, back end, product, design, etc. Keep an eye on the Nuggets Translation project for more quality translations.