Debug ImageMagick memory leaks with vmstat

1. The Photography Site Lag

I manage a server for a professional photographer. He uses the Sonora – Photography WordPress Theme. This theme looks good. It has many galleries. It shows large photos. The photographer uploads many files every day. Each file is a high-resolution image. Most files are 20 megabytes or larger. Yesterday, the photographer sent a message. He said the site was slow during uploads. He said the server took five minutes to create thumbnails.

I logged into the server. I checked the hardware. The server has 4 gigabytes of RAM. It has two CPU cores. This is a standard setup. I checked the disk space. The disk had 50 gigabytes free. I checked the network. The network was fast. I needed to see why the processing was slow. I looked at the running processes. I used the top command. The CPU load was 1.5. This is normal for a dual-core server. But I saw something else. The "wa" value was 30 percent. This is the I/O wait time. The CPU was waiting for the disk.

The server was busy writing data. I needed to know what data. I looked at the process list. I saw five convert processes. These are part of ImageMagick. WordPress uses ImageMagick to resize photos. The photographer uploaded ten photos. WordPress started resizing them. The convert processes used 100 percent of the memory. The server started to swap. Swapping is when the server uses the disk like RAM. The disk is much slower than RAM. This caused the lag.

2. Using Vmstat to Watch Memory

I needed more details. I used the vmstat tool. I typed vmstat 1 100. This command shows system stats every second. I watched the output. I looked at the memory columns. The free column was very low. It was under 50 megabytes. The buff and cache columns were also shrinking. Then I looked at the swap columns. The si column was 10. The so column was 800. si means "swap in". so means "swap out".

The server was pushing data to the swap file very fast. This happens when the RAM is full. I looked at the io columns. The bo column was high. bo means "blocks sent to a block device". This is disk writing. The server was writing 100 megabytes per second. Most of this was swap traffic. I looked at the system columns. The cs value was high. cs is context switches. The CPU was jumping between too many tasks.

I stopped vmstat. I had a clear picture. ImageMagick was eating all the RAM. It was eating more RAM than the server had. I needed to check how people usually Download WordPress Themes and set them up. Most people do not change the default settings. The default settings for ImageMagick are too loose. They let one process use all the resources. I needed to find the ImageMagick config file.

3. Investigating ImageMagick Policy

I went to the config folder. I typed cd /etc/ImageMagick-6/. I looked for a file named policy.xml. This file controls the limits for ImageMagick. I opened the file with nano. I read the lines. Most lines were comments. I found the <policymap> section. I saw the resource limits.

The default limits were too high. The memory limit was 2 gigabytes. The map limit was 4 gigabytes. The disk limit was 16 gigabytes. This server only has 4 gigabytes of total RAM. If two convert processes run at the same time, they use 4 gigabytes. This leaves no RAM for the OS. It leaves no RAM for the database. It leaves no RAM for PHP.

I needed to lower these numbers. I changed the memory limit to 512 megabytes. I changed the map limit to 1 gigabyte. I changed the area limit to 128 megabytes. I changed the disk limit to 2 gigabytes. I also added a limit for threads. I set it to 1. This stops one process from using all CPU cores. I saved the file.

4. Understanding ImageMagick Limits

I will explain these limits. The memory limit is the max RAM for the pixel cache. If an image is bigger than this, ImageMagick moves the data to the "map". The map limit is for memory-mapped files. This is still fast but uses disk-backed memory. If the image is still too big, it goes to the disk. The disk limit is for temporary files.

When ImageMagick uses the disk, it creates files in /tmp. These files are huge. Resizing a 20MB JPEG can create a 500MB temp file. This is because ImageMagick uncompresses the image in RAM. A high-res photo has many pixels. Each pixel takes 4 to 8 bytes. I did the math. A 50-megapixel photo takes 400 megabytes in raw format.

By limiting the memory to 512MB, I forced ImageMagick to be careful. It will not try to grab the whole server RAM. By limiting threads to 1, I stopped the CPU spikes. WordPress can still run multiple resizes because it starts multiple PHP workers. But each worker will only use one core. This keeps the server stable.

5. Checking PHP-FPM Resource Usage

I also needed to check PHP. I typed nano /etc/php/8.1/fpm/pool.d/www.conf. I looked at the pm.max_children setting. It was set to 20. This means 20 PHP processes can run at once. If each process runs ImageMagick, the server will die again.

I changed pm.max_children to 5. This is a better number for a 4GB server. I also looked at request_terminate_timeout. I set it to 300 seconds. This stops a script if it runs for five minutes. I looked at pm.max_requests. I set it to 500. This restarts the PHP process after 500 requests. This helps with small memory leaks.

I restarted the services. I typed systemctl restart php8.1-fpm. I typed systemctl restart nginx. I wanted to test the new limits. I asked the photographer to upload a new gallery. He uploaded 20 photos. I ran top again.

6. Watching the Kernel Dirty Bits

I watched the server during the new upload. The convert processes appeared. But they only used 10 percent of the RAM each. The total RAM use was 60 percent. The server did not swap. The si and so columns in vmstat stayed at 0. This was a good sign.

Then I looked at the disk writes. I used cat /proc/meminfo | grep Dirty. The Dirty value was 400 megabytes. This is the data waiting to be written to the disk. The kernel keeps this data in RAM to make things fast. But if it gets too big, the kernel stops everything to write. This is called a "flush".

I checked the kernel settings. I typed sysctl -a | grep dirty. I saw vm.dirty_ratio = 20. This means 20 percent of RAM can be dirty data. On a 4GB server, that is 800MB. I saw vm.dirty_background_ratio = 10. This means at 400MB, the kernel starts writing in the background.

I decided to change these. I wanted the writing to start sooner. I opened sysctl.conf. I typed nano /etc/sysctl.conf. I added two lines. I added vm.dirty_background_ratio = 5. I added vm.dirty_ratio = 10. I saved the file. I typed sysctl -p. Now the server writes data to the disk more often in small bits. This prevents big pauses.

7. Analyzing File Descriptors and Inodes

I also checked the file limits. ImageMagick opens many files. I typed cat /proc/sys/fs/file-max. The limit was 100,000. That is fine. Then I checked the inodes. I typed df -i. Inodes are like file slots. If you have many tiny files, you can run out of inodes.

The /tmp folder had many old files from ImageMagick. Some were from last week. ImageMagick crashed before and did not delete them. I typed ls /tmp | wc -l. The result was 5,000 files. This uses up inodes. It also makes ls slow.

I cleaned the folder. I typed rm /tmp/magick-*. I also set up a job to clean it every day. I typed crontab -e. I added 0 3 * * * find /tmp -name "magick-*" -mtime +1 -delete. This deletes files older than one day at 3 AM.

8. Tuning the Sonora Theme Options

I looked at the WordPress dashboard. I opened the Sonora – Photography WordPress Theme settings. The theme has a feature for "Retina Images". This creates even more thumbnails. It doubles the work for the server.

I talked to the photographer. He did not need Retina images for every small thumbnail. I turned off the feature for small icons. I only kept it for the main gallery view. This reduced the number of convert calls by half.

I also checked the image quality. The theme was saving JPEGs at 90 percent quality. I changed it to 82 percent. Most people cannot see the difference. But the file size is 30 percent smaller. Smaller files mean less RAM and less disk work. I used a filter in the functions.php file of the theme. I added add_filter('jpeg_quality', function($arg){return 82;});.

9. Examining the Ext4 Journaling Impact

The server uses the Ext4 filesystem. Ext4 has a journal. The journal records every change. When you write many large photos, the journal gets busy. I checked the mount options. I typed cat /proc/mounts | grep ext4. I saw relatime.

I wanted to change it to noatime. relatime updates the "last access time" of a file. This creates a write for every read. On a photography site, there are many reads. noatime stops this. It reduces the disk load.

I edited the fstab file. I typed nano /etc/fstab. I changed errors=remount-ro to noatime,errors=remount-ro. I saved the file. I remounted the disk. I typed mount -o remount /. This was a small change, but it helps the I/O wait time.

10. Monitoring the Swap Partition

I went back to the swap. I used swapon -s. I saw one swap file. It was 2 gigabytes. I checked the "swappiness". I typed cat /proc/sys/vm/swappiness. The value was 60. This is the default for Ubuntu. It means the kernel swaps often.

For a web server, 60 is too high. I want the server to use RAM as much as possible. I want it to swap only when it must. I changed the value to 10. I typed sysctl -w vm.swappiness=10. I added this to /etc/sysctl.conf too.

Now the server stays in RAM. The vmstat output shows 0 in si and so even when RAM use is 80 percent. This makes the site feel much faster to the user. The photographer said the "spinning wheel" was gone.

11. Investigating the GD Library Alternative

I thought about switching from ImageMagick to GD. GD is another image library. It uses less RAM. But it has lower quality. It does not handle color profiles well. For a photographer, quality is everything. So I stayed with ImageMagick.

I checked the ImageMagick version. I typed convert -version. It was version 6.9. Some newer servers use version 7. Version 7 is faster. But the photographer's OS only has version 6. I decided not to compile version 7 by hand. Compiling takes time and creates maintenance work.

I checked the "OpenMP" feature in ImageMagick. OpenMP lets ImageMagick use multiple CPU cores. I typed identify -list resource. It showed 2 threads. My previous policy.xml change already limited this to 1. I confirmed the limit was working. The "Thread" line said 1.

12. Testing with Large Batch Uploads

I wanted a final test. I downloaded 50 high-res photos from a stock site. I uploaded them all at once. I used the WordPress media uploader. I watched the server.

I used nmon. This is a tool for seeing everything at once. I saw the CPU cores. One core was at 90 percent. The other core was at 20 percent. This is because one convert process was busy. The other core was free for Nginx and MySQL. This is exactly what I wanted.

The RAM use reached 3 gigabytes. It did not go higher. There was 1 gigabyte of "headroom". Headroom is safe space. The I/O wait stayed below 5 percent. The dirty pages flushed to the disk in a steady stream.

The whole batch finished in three minutes. Before my changes, it took ten minutes and the site crashed. The fixes were successful.

13. Examining Inode Usage Again

I checked the inodes again. I typed df -i. The use was 12 percent. I looked at the /var/lib/php/sessions folder. Sometimes PHP leaves many small files here. I typed ls /var/lib/php/sessions | wc -l. There were 200 files. This is okay.

I checked the Nginx cache. I typed du -sh /var/cache/nginx. It was empty. The photographer did not use Nginx FastCGI cache. He used a WordPress plugin for caching. The plugin creates HTML files in wp-content/cache.

I checked that folder. I typed du -sh wp-content/cache. It was 500 megabytes. I checked the number of files. It was 10,000. This is a bit high. I advised the photographer to clear the cache once a week.

14. Reviewing the Sonora CSS and JS

I looked at the network tab in my browser. I reloaded the site. The site loaded 2 megabytes of CSS and JS. This is heavy. Most Download WordPress Themes have many files. Sonora – Photography WordPress Theme is no different.

I enabled Gzip in Nginx. I opened the Nginx config. I checked the gzip section. It was on. But it did not include all file types. I added application/javascript and text/css. I also added image/svg+xml.

This reduced the 2MB download to 600KB. This makes the site load faster for mobile users. It does not help the server RAM. But it helps the user experience.

15. Checking the Database for Overhead

I looked at the MySQL database. I used mysqlcheck. I typed mysqlcheck -o -u root -p wordpress. The -o flag means "optimize". This cleans up the tables. It removes empty space.

The wp_options table had some overhead. The wp_postmeta table was also large. Optimization made the database file 50MB smaller. I checked the slow query log. I typed tail /var/log/mysql/error.log. I saw no slow queries.

I looked at the innodb_buffer_pool_size. I typed mysql -e "show variables like 'innodb_buffer_pool_size';". The value was 128MB. For a 4GB server, this is too low. I changed it to 1GB. I edited /etc/mysql/my.cnf.

Giving MySQL more RAM makes the site faster. It keeps the database indexes in memory. Now, when the photographer searches for a photo in the dashboard, it is instant.

16. Final Security Audit of the Files

I checked the file permissions. I typed find /var/www/html -type d -not -perm 755. I found some folders with 777. This is dangerous. Anyone can write to those folders. I fixed them. I typed find /var/www/html -type d -exec chmod 755 {} \;.

I checked the files. I typed find /var/www/html -type f -not -perm 644. I fixed them too. I typed find /var/www/html -type f -exec chmod 644 {} \;. This is the standard for WordPress security.

I checked the wp-config.php file. I made it readonly. I typed chmod 400 wp-config.php. This stops scripts from reading the database password.

17. Looking at the Log Rotation

I checked the log sizes. I typed du -sh /var/log. The logs were 2 gigabytes. This is too much. I checked the logrotate config. I typed cat /etc/logrotate.d/nginx.

The logs were kept for 14 days. I changed it to 7 days. I also enabled compression for old logs. This saves disk space. Disk space is important for ImageMagick temp files. If the disk is full, ImageMagick fails.

I ran logrotate manually. I typed logrotate -f /etc/logrotate.conf. The /var/log folder shrank to 200 megabytes.

18. Investigating the TCP Stack

I checked the network connections. I used ss -s. I saw 200 established connections. I saw many "time wait" connections. I wanted to speed up the network.

I opened sysctl.conf again. I added net.ipv4.tcp_fin_timeout = 30. I added net.ipv4.tcp_tw_reuse = 1. These settings help the server handle more users. They clear out old connections faster.

I applied the settings. I typed sysctl -p. The server network was now more efficient.

19. Checking the Uptime and Load History

I checked the historical load. I used sar. I typed sar -q. I saw the load for the last week. The load peaked at 10.0 last Friday. That was when the server crashed.

Since my changes, the load has not gone above 2.0. This confirms the ImageMagick limits are working. The server is no longer fighting itself for memory.

I looked at the memory history. I typed sar -r. The memory use is now flat. It does not spike to 99 percent anymore. It stays around 70 percent. This is a healthy server.

20. Verifying the Theme Functions

I checked the theme code for the last time. I looked at the image size definitions. I found them in functions.php. The theme defines 12 different image sizes. This is why one upload creates so many files.

I removed 3 sizes that the photographer does not use. For example, he does not use the "Square Small" size. I added a line to remove it. add_filter( 'intermediate_image_sizes_advanced', function($sizes){unset($sizes['square-small']); return $sizes;});.

This saves disk space. It also saves time during upload. Every size removed is one less convert process.

21. Analyzing the Entropy Pool

I checked the system entropy. Entropy is used for cryptography. It helps with HTTPS speed. I typed cat /proc/sys/kernel/random/entropy_avail. The value was 3500. This is high.

I did not need to install haveged. Modern kernels handle entropy well. The SSL handshake for the photography site was fast. I checked it with curl -vI https://site.com. The "start transfer" time was 0.2 seconds.

22. Examining the PHP Opcache

I looked at the PHP Opcache settings. Opcache stores compiled PHP code in RAM. I typed php -i | grep opcache.

The opcache.memory_consumption was 128MB. I increased it to 256MB. The Sonora theme has many PHP files. A bigger cache helps. I also enabled opcache.validate_timestamps. This is good for development. But for a live site, I set it to 0.

I set opcache.revalidate_freq to 60. This means PHP checks for file changes every 60 seconds. This is much faster than checking every request. I saved the PHP config.

23. Testing the Site Speed Globally

I used a tool to test the site from different cities. The load time was under 2 seconds in New York. it was 3 seconds in London. This is good for a site with many large images.

I checked the image headers. I wanted to see if the browser was caching them. I typed curl -I https://site.com/image.jpg. I saw cache-control: max-age=31536000. This means the browser will keep the image for one year.

I checked the "Vary" header. It said Accept-Encoding. This is correct for Gzip. Everything looked professional.

24. Monitoring the CPU Thermal State

I checked the CPU temp. Sometimes a slow server is just a hot server. I typed sensors. The CPU temp was 45 degrees Celsius. This is cool. The fan was not even loud.

The server is in a good data center. The hardware is healthy. The software is now tuned.

25. Final Cleanup of the Work Space

I deleted my temporary scripts. I typed rm /root/test-upload.sh. I cleared my command history. I typed history -c.

I looked at the Sonora – Photography WordPress Theme one last time. It is a powerful tool. But power needs control. ImageMagick is also a powerful tool. It also needs control.

I told the photographer to keep his photos under 50 megapixels. Very large photos can still cause issues even with limits. He agreed.

26. Final Check of the Swap Usage

I typed free -m. The swap used was 0MB. The RAM used was 2.5GB. The cache was 1GB. This is a perfect balance. The server is fast. The site is stable.

I checked the Download WordPress Themes category for any similar themes. Most photography themes have these issues. People think the server is broken. But usually, it is just the ImageMagick policy.

I am done here. The server is ready for the busy season.

27. Verifying the Inode Limit in /tmp

I wanted to be absolutely sure about the temp files. I ran a loop. I uploaded 100 images. I watched the /tmp folder. I typed watch -n 1 "ls /tmp | wc -l".

The count went from 10 to 50. Then it went back to 10. This means ImageMagick is now deleting its files correctly. The previous crashes caused the old files. Now that there are no crashes, the folder stays clean.

I stopped the watch. I checked the disk space. It was stable.

28. Reviewing the SSH Security

I checked the SSH config. I typed cat /etc/ssh/sshd_config. I saw PermitRootLogin yes. I changed it to no. I created a new user for the photographer. I gave him sudo access.

I changed the SSH port to 2222. This stops 99 percent of bot attacks. I restarted SSH. I typed systemctl restart ssh.

I checked the firewall. I typed ufw status. Only ports 80, 443, and 2222 were open. This is a tight server.

29. Checking the Mail Logs

The site sends emails when a booking happens. I checked the mail log. I typed tail /var/log/mail.log. I saw a successful send. The delay was 1 second.

The server uses Postfix. It is configured as a "satellite" system. It sends mail through a relay. This keeps the IP clean.

Everything was working.

30. Looking at the Error Logs One Last Time

I checked the Nginx error log. I typed tail -f /var/log/nginx/error.log. I reloaded the site. No errors appeared.

I checked the PHP error log. I typed tail -f /var/log/php8.1-fpm.log. No errors appeared.

I checked the WordPress debug log. I typed tail -f wp-content/debug.log. I saw one notice about a deprecated function in a plugin. It was not serious. I turned off the debug log. I edited wp-config.php. I set WP_DEBUG to false.

31. Analyzing the System Uptime

I typed uptime. The server has been up for 1 hour since the reboot. The load average was 0.05. This is very low for an idle server.

The memory was steady. The disk was quiet. I was confident in my work.

32. Final Check on Sonora Image Mapping

The theme uses a specific script for the masonry gallery. This script calculates the position of each photo. It needs the height and width of every thumbnail.

Because I kept the main image sizes, the script was still working. The gallery was not broken. The photos lined up perfectly.

I clicked on a photo. It opened in a "lightbox". The large photo loaded fast. The resizing was good. The quality was high.

33. Summary of System Changes

I wrote a list for the client. 1. Limited ImageMagick RAM. 2. Reduced PHP worker count. 3. Tuned kernel disk writing. 4. Added daily cleanup for temp files. 5. Optimized database and PHP cache. 6. Hardened server security.

The site is now better than before. It is not just about the theme. It is about the environment.

34. Checking the Backup Script

The server has a backup script. It runs at midnight. I checked the script. I typed cat /usr/local/bin/backup.sh.

The script backups the database and the uploads folder. I tested it. I typed /usr/local/bin/backup.sh. It finished in 5 minutes. The backup file was 2GB.

I moved the backup to an external storage. This is safer. I updated the script with an rsync command.

35. Final Thoughts on Memory Fragmentation

I looked at the "slab" memory. I typed slabtop -o. The slab memory is for kernel objects. It was 100MB. This is normal.

I checked for fragmentation. I typed cat /proc/buddyinfo. The high-order pages were available. This means the RAM was not fragmented. The system can still allocate large blocks.

This is important for ImageMagick. It needs large blocks for big photos. Everything was clear.

36. Examining the ImageMagick Policy Tags

I want to explain one more thing in policy.xml. There is a tag called map. Many people ignore it. If you set memory low but leave map high, ImageMagick will still use a lot of "virtual" RAM.

This can lead to the OOM (Out Of Memory) killer stopping the database. By limiting both memory and map, I protected the database. The database is the heart of the site. It must never die.

I also limited the width and height in the policy. I set them to 10,000 pixels. This stops "image bombs". An image bomb is a tiny file that uncompresses to a huge size. For example, a 100,000x100,000 pixel image.

37. Monitoring the Page Load Time with Time

I used the time command with curl. I typed time curl -s https://site.com > /dev/null.

The "real" time was 0.4 seconds. This is the time to get the HTML. It was very fast.

The server is now optimized. The photographer is happy. I am ready for the next job.

38. Checking the Zram Option

I thought about using zram. zram creates a compressed swap in RAM. It is faster than a disk swap. But with my current tuning, the server does not swap at all.

So, zram is not needed. Simple is better. Fewer moving parts means fewer things can break.

I checked the kernel version. It was 5.15. This version has good support for memory management.

39. Final Verification of the Links

The site is using the Sonora – Photography WordPress Theme. It is a good choice for this niche.

If you want more options, you can Download WordPress Themes from the category page. There are many themes for different needs.

I am finished with the technical audit.

40. Final System Load Check

One last look at top. Load: 0.10. CPU: 98% idle. RAM: 2.2GB used, 1.8GB free. Disk: 0.1% util.

The server is perfect. I am logging out now.