Orphaned memory-mapped files in PHP-FPM worker termination

Environment Baseline

The physical hardware is a dual-socket Intel Xeon Silver 4214R (24 cores, 48 threads total) provisioned with 128GB of ECC RAM. The host operating system is Debian 12 (Bookworm) running the standard 6.1.0-18-amd64 kernel. The environment is heavily containerized using LXC (Linux Containers), managed via LXD.

The specific container under investigation runs Alpine Linux 3.18. It is allocated 16 virtual CPUs and a strict memory limit of 32GB enforced via cgroups v2. The application stack within the container comprises Nginx 1.24.0, PHP 8.2 FPM, and MariaDB 10.11. The PHP layer relies on the native Imagick extension (compiled against ImageMagick 7.1.1-15) for graphical processing.

The application is a digital storefront dealing in high-resolution, print-ready files. The active template is the Postero - Wall Art & Poster WooCommerce Theme. Due to the nature of the products, the site routinely processes source image files (TIFF and JPEG) exceeding 80 megabytes in compressed size, often measuring up to 12,000 by 12,000 pixels, intended for 300 DPI physical reproduction.

The Indicator

At 04:15 UTC, an automated infrastructure audit script utilizing node_exporter recorded a steady depletion of available inodes on the /tmp mount point within the Alpine container. No other metric showed deviation from the established baseline. CPU utilization remained stable at an average of 14%, and physical RAM usage hovered at 18GB.

The /tmp directory in this container is mounted as tmpfs, a temporary file storage paradigm residing in volatile memory (and swap, though swap is disabled on this host).

I connected to the container and executed a filesystem check.

df -h /tmp

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           16G   2.1G   14G  13% /tmp

The storage capacity metric was well within limits. However, evaluating the index node (inode) capacity presented a different state.

df -i /tmp

Filesystem       Inodes IUsed    IFree IUse% Mounted on
tmpfs              4.0M  3.9M   102400   98% /tmp

The tmpfs instance was allocated 4 million inodes during the mount operation. 3.9 million of these were consumed. An inode is a data structure on a filesystem that stores all information about a regular file, directory, or other file system object, except its data and name. When a filesystem exhausts its inodes, no new files can be created, regardless of the available block storage capacity.

This specific condition results in silent application failures. Processes attempting to write temporary data to /tmp will receive an ENOSPC (No space left on device) error from the kernel's Virtual File System (VFS) layer, even if df -h shows gigabytes of available space.

I listed the contents of /tmp to quantify the objects.

ls -f /tmp | wc -l

The execution yielded 3924012. The directory was populated almost entirely by files following a specific nomenclature: magick-XXXXXXXX.

VFS and tmpfs Allocation Mechanics

To understand the interaction, it is necessary to examine how tmpfs handles inode allocation. When a tmpfs is mounted without explicit nr_inodes parameters, the kernel calculates the maximum number of inodes based on the allocated memory size. The default calculation permits one inode for every RAM page (usually 4KB on x86_64 architectures) allocated to the tmpfs instance, or half the physical RAM pages, whichever is lower.

In this container, the tmpfs mount was configured in /etc/fstab:

tmpfs   /tmp         tmpfs   rw,nosuid,nodev,size=16G,nr_inodes=4M   0 0

The 4 million inode limit was a hard boundary. The presence of nearly 4 million magick-* files, each consuming 0 bytes of data (as evident by the low total space utilization of 2.1GB), indicated that files were being created, their data segments potentially unlinked or truncated, but their file descriptors or directory entries were persisting.

I randomly sampled ten of these files and inspected their metadata using the stat command.

stat /tmp/magick-aB9zY1xW

  File: /tmp/magick-aB9zY1xW
  Size: 0           Blocks: 0          IO Block: 4096   regular empty file
Device: 1ah/26d Inode: 10485762    Links: 1
Access: (0600/-rw-------)  Uid: (   82/www-data)   Gid: (   82/www-data)
Access: 2023-11-04 02:14:12.000000000 +0000
Modify: 2023-11-04 02:14:12.000000000 +0000
Change: 2023-11-04 02:14:12.000000000 +0000

The files were consistently zero bytes in size, owned by the www-data user, and had strict 0600 permissions. The timestamp distribution showed these files were being generated steadily throughout the day, correlating with external HTTP requests triggering background processes.

Diagnostic Tracing with inotifywait

To observe the filesystem operations in real-time, I deployed inotifywait, a component of the inotify-tools package. This utility taps into the Linux kernel's inotify subsystem, which provides a mechanism for monitoring filesystem events down to the individual system call level.

I constructed a monitoring script to capture the exact lifecycle of these files.

#!/bin/sh
inotifywait -m -e create -e modify -e close_write -e delete -e moved_from -e moved_to --timefmt '%H:%M:%S' --format '%T %e %w%f' /tmp | grep magick > /var/log/tmpfs_trace.log

I allowed the script to run for 300 seconds, capturing a typical operational window. Afterward, I analyzed the resulting log.

...
10:14:02 CREATE /tmp/magick-qR4tU8vP
10:14:02 MODIFY /tmp/magick-qR4tU8vP
10:14:05 CLOSE_WRITE,CLOSE /tmp/magick-qR4tU8vP
10:14:05 DELETE /tmp/magick-qR4tU8vP
10:14:12 CREATE /tmp/magick-kL9mN2bV
10:14:12 MODIFY /tmp/magick-kL9mN2bV
... [Silence for 60 seconds]
10:15:13 CREATE /tmp/magick-xY7zA4cC
10:15:13 MODIFY /tmp/magick-xY7zA4cC
10:15:15 CLOSE_WRITE,CLOSE /tmp/magick-xY7zA4cC
10:15:15 DELETE /tmp/magick-xY7zA4cC
...

The standard, healthy operation sequence was visible: CREATE, MODIFY, CLOSE_WRITE, followed immediately by DELETE. The underlying application was creating the temporary file, processing data, and successfully instructing the VFS to unlink it.

However, the sequence for the file /tmp/magick-kL9mN2bV was anomalous. It registered CREATE and MODIFY, but the CLOSE_WRITE and DELETE events never materialized. The process that initiated the file creation abruptly ceased interaction with the file descriptor.

To identify the specific process responsible for the incomplete sequences, I cross-referenced the file creation timestamps with the active PHP-FPM processes and Nginx access logs.

Application Layer Interaction: PHP-FPM and WordPress Core

When administrators set up digital storefronts and Download WordPress Themes designed for visual media, the system relies heavily on the core WordPress function wp_generate_attachment_metadata(). This function is invoked whenever a new image is uploaded or when background regeneration tasks (like updating product thumbnails for WooCommerce) are triggered.

The execution path traverses the following components: 1. wp_generate_attachment_metadata() calls wp_get_image_editor(). 2. The core attempts to instantiate WP_Image_Editor_Imagick. 3. The Imagick extension loads the source file. 4. The multi_resize() method loops through the required registered image sizes.

The WooCommerce Postero theme registers several custom, high-resolution image sizes in its functions.php:

add_image_size( 'postero-hero', 2400, 1600, true );
add_image_size( 'postero-gallery-full', 1800, 1800, false );
add_image_size( 'postero-product-zoom', 3000, 3000, false );

I examined the Nginx logs for the exact second 10:14:12 corresponding to the orphaned magick-kL9mN2bV file.

192.168.10.55 - - [04/Nov/2023:10:14:12 +0000] "POST /wp-admin/admin-ajax.php?action=regenerate_thumbnails HTTP/2.0" 499 0 "-" "Mozilla/5.0..."

The HTTP status code 499 is an Nginx-specific code indicating Client Closed Request. The client terminated the TCP connection before Nginx could return a response.

Simultaneously, I checked the PHP-FPM error logs for the same timestamp.

[04-Nov-2023 10:15:12] WARNING: [pool www] child 14022, script '/var/www/html/wp-admin/admin-ajax.php' (request: "POST /wp-admin/admin-ajax.php") execution timed out (60.103092 sec), terminating
[04-Nov-2023 10:15:12] WARNING: [pool www] child 14022 exited on signal 15 (SIGTERM) after 1450.230000 seconds from start
[04-Nov-2023 10:15:12] NOTICE: [pool www] child 14088 started

The sequence of events was now clearly defined: 1. The PHP script began processing a massive source image (e.g., a 12,000 x 12,000 pixel TIFF) to generate the postero-product-zoom size. 2. The Imagick extension delegated the pixel manipulation to the underlying ImageMagick C library. 3. ImageMagick created a temporary cache file in /tmp. 4. The processing time exceeded the request_terminate_timeout defined in the PHP-FPM pool configuration (60 seconds). 5. The FPM master process sent a SIGTERM to the worker process 14022. 6. The worker process was immediately destroyed by the kernel.

Because the process termination was enforced externally by the master process via a signal, the PHP garbage collector never executed. More importantly, the internal destructors within the Imagick extension, which usually call DestroyMagickWand() and subsequently trigger the C library to unlink its temporary files, were bypassed. The kernel closed open file descriptors (which is why the files size showed as 0, as the buffers were flushed or dropped), but the directory entry in /tmp remained.

ImageMagick Memory Architecture and Pixel Cache

To fully resolve the issue, it is necessary to understand why ImageMagick was creating files in /tmp in the first place, rather than performing the image manipulation entirely in RAM.

Unlike standard file handling where data is streamed, image processing requires random access to pixel data. ImageMagick unpacks compressed formats (like JPEG or PNG) into an uncompressed memory array known as the Pixel Cache.

The memory requirement is independent of the compressed file size. It is calculated based on the image dimensions and the color depth. For a 12,000 by 12,000 pixel image processed at a 16-bit depth (which is standard for high-quality printing workflows) with an alpha channel (RGBA):

Width = 12,000 Height = 12,000 Pixels = 144,000,000

Each pixel requires: Red (16 bits = 2 bytes) Green (16 bits = 2 bytes) Blue (16 bits = 2 bytes) Alpha (16 bits = 2 bytes) Total = 8 bytes per pixel.

Total memory = 144,000,000 pixels * 8 bytes = 1,152,000,000 bytes (approximately 1.15 GB) of uncompressed pixel data per image instance.

During operations like resizing, ImageMagick often needs to hold both the original image and the destination image in the cache simultaneously, doubling the requirement to over 2.3 GB.

ImageMagick manages its resource consumption via an XML configuration file, typically located at /etc/ImageMagick-7/policy.xml. I dumped the contents of this file.

<policymap>
  <policy domain="resource" name="memory" value="256MiB"/>
  <policy domain="resource" name="map" value="512MiB"/>
  <policy domain="resource" name="width" value="16KP"/>
  <policy domain="resource" name="height" value="16KP"/>
  <policy domain="resource" name="area" value="128MB"/>
  <policy domain="resource" name="disk" value="1GiB"/>
  <policy domain="resource" name="file" value="768"/>
  <policy domain="resource" name="thread" value="4"/>
  <policy domain="resource" name="throttle" value="0"/>
  <policy domain="resource" name="time" value="3600"/>
  <policy domain="system" name="precision" value="6"/>
</policymap>

The policy.xml enforced strict limits. The memory limit was set to 256 MiB. The map limit (memory-mapped files) was set to 512 MiB.

When the PHP script requested the processing of the 1.15 GB uncompressed image, ImageMagick immediately hit the 256 MiB memory limit. Following its internal hierarchy, it fell back to memory-mapping. It hit the 512 MiB map limit. Finally, it fell back to the disk cache.

By default, ImageMagick utilizes the MAGICK_TMPDIR, TMPDIR, or /tmp environmental variables to locate its disk cache. It creates a .mpc (Magick Persistent Cache) file on disk. In this environment, /tmp is a tmpfs mount, meaning the data was still residing in physical RAM, but routed through the VFS layer as a file.

When the FPM worker was killed, the 1.15 GB data segment in RAM associated with the tmpfs file was freed by the kernel because the file descriptor was closed and there were no other links to the data blocks, but the inode itself remained registered in the directory structure.

Execution Path Reconfiguration

The resolution required a multi-tiered approach. Relying solely on increasing the PHP execution timeout is a flawed pattern; network anomalies or extremely large files will eventually trigger the timeout regardless. The system must gracefully handle process termination without leaking filesystem resources.

Tier 1: ImageMagick Resource Tuning

The container possessed 32GB of available memory. Restricting ImageMagick to 256 MiB was a legacy configuration unsuited for the workload of a wall art repository. By increasing the memory limits, ImageMagick processes the images entirely within the heap, bypassing the tmpfs VFS layer completely for standard operations. If the process is killed, the kernel reclaims the heap memory instantly, leaving no filesystem artifacts.

I edited /etc/ImageMagick-7/policy.xml:

<policymap>

  <policy domain="resource" name="memory" value="4GiB"/>

  <policy domain="resource" name="map" value="8GiB"/>

  <policy domain="resource" name="width" value="16KP"/>
  <policy domain="resource" name="height" value="16KP"/>
  <policy domain="resource" name="area" value="4GB"/>
  <policy domain="resource" name="disk" value="16GiB"/>
  <policy domain="resource" name="file" value="1024"/>

  <policy domain="resource" name="thread" value="1"/>
</policymap>

The reduction of thread to 1 is a critical adjustment for web server environments. ImageMagick uses OpenMP to parallelize processing. If FPM spawns 10 concurrent workers, and each worker utilizes 16 threads, the CPU undergoes severe context switching. Forcing single-threaded operation per worker yields higher overall throughput in a multi-tenant or concurrent HTTP environment.

Tier 2: PHP-FPM Configuration Adjustments

I accessed the FPM pool configuration at /etc/php/8.2/fpm/pool.d/www.conf.

The default request_terminate_timeout of 60 seconds was insufficient for resizing multiple 12,000 pixel images. However, simply raising it allows rogue processes to lock up workers indefinitely. I decoupled the PHP execution limits.

; /etc/php/8.2/fpm/pool.d/www.conf
request_terminate_timeout = 300s

In the core php.ini (/etc/php/8.2/fpm/php.ini), I adjusted the memory parameters:

memory_limit = 1024M
max_execution_time = 240

The request_terminate_timeout (300s) acts as a hard kernel-level kill switch, while max_execution_time (240s) allows PHP to attempt a graceful shutdown internally before the FPM master intervenes.

Furthermore, I explicitly defined a separate temporary directory for the PHP workers. Mixing application temporary files with system temporary files in /tmp complicates cleanup. I created a dedicated directory /var/cache/php_tmp and assigned ownership to the www-data user.

mkdir -p /var/cache/php_tmp
chown www-data:www-data /var/cache/php_tmp
chmod 0700 /var/cache/php_tmp

I then injected environment variables directly into the FPM pool configuration to override the default system paths.

; /etc/php/8.2/fpm/pool.d/www.conf
env[TMP] = /var/cache/php_tmp
env[TMPDIR] = /var/cache/php_tmp
env[TEMP] = /var/cache/php_tmp
env[MAGICK_TMPDIR] = /var/cache/php_tmp

This configuration isolates all ImageMagick disk cache operations to the persistent NVMe storage backing /var/cache, rather than the constrained tmpfs layer. While disk I/O is slower than tmpfs, the increased memory limit in policy.xml ensures that disk caching is only triggered for exceptionally massive anomalies, preventing memory exhaustion while ensuring stable completion.

Tier 3: Asynchronous Cleanup via systemd-tmpfiles

Despite configuring higher memory limits and isolating the temporary directory, operating system-level process termination (such as an Out-Of-Memory invocation by the kernel) can still result in orphaned files in the new persistent cache directory.

To ensure deterministic state management, I utilized systemd-tmpfiles. This native systemd component manages the creation, deletion, and cleaning of volatile and temporary files based on strict age rules.

I created a new ruleset at /etc/tmpfiles.d/php-imagemagick.conf.

# Type  Path                  Mode  User      Group     Age  Argument
d       /var/cache/php_tmp    0700  www-data  www-data  -    -
e       /var/cache/php_tmp/*  -     -         -         2h   -

The d directive ensures the directory exists with the correct permissions on boot. The e (empty) directive instructs the systemd timer to delete any file within the directory that has not been accessed or modified in the last 2 hours. This provides a guaranteed, fail-safe garbage collection mechanism independent of the PHP application state.

I reloaded the systemd daemon to parse the new rule and executed an immediate cleanup to verify syntax.

systemctl daemon-reload
systemd-tmpfiles --clean /etc/tmpfiles.d/php-imagemagick.conf

Verification and State Confirmation

To clean the existing state, I manually purged the 3.9 million orphaned files in /tmp. Standard rm -rf /tmp/magick-* would fail with an Argument list too long error due to shell expansion limits. I utilized find coupled with delete.

find /tmp -name "magick-*" -type f -delete

Following the deletion, I restarted the PHP-FPM service to apply the new pool variables and configuration.

systemctl restart php8.2-fpm

I re-evaluated the tmpfs inode capacity.

df -i /tmp

Filesystem       Inodes IUsed    IFree IUse% Mounted on
tmpfs              4.0M    14     4.0M    1% /tmp

The inode utilization returned to 1%.

To validate the execution path under load, I initiated a manual thumbnail regeneration task via the WordPress CLI tool (wp-cli) for a product containing a 180MB, 14,000 x 14,000 pixel source image.

sudo -u www-data wp media regenerate 4192 --force

During execution, I monitored the /var/cache/php_tmp directory using ls -la. No magick-* files appeared. I simultaneously monitored the process memory footprint using top -p $(pgrep -f "php-fpm: pool www"). The Resident Set Size (RSS) for the specific FPM worker processing the image scaled to 1.8GB, completing the task entirely in RAM without VFS interaction, and dropped back to 45MB immediately upon completion. The custom systemd timer rules and ImageMagick policy configurations successfully prevented disk cache usage while mitigating future file descriptor leaks.