Developing a WebP Converter Script for Next.js

Ghost to Next.js Migration series

This post is part of a series of posts about migrating my blog from Ghost to Next.js:

Introduction

After migrating my blog from Ghost to Next.js, even though the site largely uses the Next.js image component for images, I wanted to also convert images to WebP format to optimise loading times.

To automate this process, I created a bash script that converts JPEG and PNG images to WebP and updates markdown files accordingly. While I use Contentlayer2 for markdown files in my setup, this script is versatile and can be used with any static site generator or framework that uses markdown and image files.

Script Overview

The script, available here, automates the process of converting images to WebP format and optionally updating references in markdown files. Here's a detailed look at how it works:

Initialization: The script starts by setting default values and parsing command-line arguments. This allows users to customize behavior such as directory traversal depth, WebP quality, and whether to include PNG files.
Image Search: Using the find command, the script recursively searches for JPEG (and optionally PNG) files in the specified directory structure. It respects the set depth limit and excludes hidden directories.
Conversion Process: For each image found, the script:
- Checks if the file is in a hidden directory (skipping if true)
- Logs the original file size
- Uses ImageMagick to convert the image to WebP format
- Logs the new WebP file size
- Updates the running totals for original and WebP sizes
Markdown Update: If enabled, the script then processes markdown files:
- Searches for .md and .mdx files
- Skips files like README.md, CHANGELOG.md, etc.
- Ignores files in the node_modules directory
- Uses sed to replace image references from .webp/.webp/.webp to .webp
Logging and Reporting: Throughout the process, the script maintains detailed logs:
- A conversion log showing before and after sizes for each image
- A found files log listing all images processed
- Real-time terminal output for monitoring progress
Final Summary: After processing all files, the script calculates and displays:
- Total number of files found and converted
- Total original size and new WebP size
- Total space saved and percentage reduction

Key Features

Configurable directory traversal depth
Option to include PNG files in the conversion
Customizable WebP quality setting
Exclusion of hidden directories and specific markdown files (e.g., README.md)
Ignores files in node_modules directory
Detailed logging and real-time terminal output

Technical Challenge: Global Variables in Bash

During development, I encountered an issue with global variables in Bash on macOS. Despite declaring variables outside functions and updating them inside, the values weren't persisting when accessed outside the functions.

The Problem

The issue stemmed from the use of pipes (|) in Bash, which create subshells for each command in the pipeline. Variables modified in a subshell are not visible to the parent shell.

For example:

find . -type f | while read file; do
    # Variables modified here aren't visible outside the loop
done
# Variables here retain their original values

The Solution

To overcome this limitation, I implemented a temporary file solution:

Create a temporary file at the start of the script:
```
TEMP_STATS_FILE=$(mktemp)
```
Initialize the file with default values:
```
echo "0 0 0 0" > "$TEMP_STATS_FILE"
```

Update values by reading from and writing to the file:

update_stats() {
    read total_original total_webp total_converted total_found < "$TEMP_STATS_FILE"
    # Update values
    echo "$total_original $total_webp $total_converted $total_found" > "$TEMP_STATS_FILE"
}

Read final values at the end of the script:

read TOTAL_ORIGINAL_SIZE TOTAL_WEBP_SIZE TOTAL_FILES_CONVERTED TOTAL_FILES_FOUND < "$TEMP_STATS_FILE"

Clean up the temporary file:
```
rm "$TEMP_STATS_FILE"
```

This approach ensures that the statistics are accurately maintained throughout the script's execution, regardless of subshells created by pipes.

Performance Considerations

The script is designed to be efficient, processing files in a single pass. However, for very large directories or sites with thousands of images, the conversion process can take some time. The script provides real-time feedback, so users can monitor progress.

The space savings can be significant. In my case, I saw an average reduction of about 70% in file size when converting from JPEG to WebP, while maintaining visual quality.

Conclusion

The WebP converter script provides an efficient way to optimize images for a Next.js blog or any static site using markdown and images. By addressing the global variable issue with a temporary file solution, I ensured reliable operation on macOS. This script significantly reduces image file sizes, improving load times and overall performance of the site.

While I created this script for my Next.js blog, its utility extends to any project where bulk image conversion to WebP is needed. The ability to automatically update markdown files makes it particularly useful for static site generators and documentation projects.