Debugging Network Latency On Staging Servers

The Staging Environment Setup

I am a site administrator. I have 15 years of experience. I work with Linux servers. I manage network stacks. I look at code. I check hardware. I start my work at eight in the morning. I open my terminal. I log into a staging server. This server runs on Ubuntu 22.04. The hardware is a Dell PowerEdge. It has 64 gigabytes of RAM. It has two CPUs. Each CPU has 12 cores. The server is fast. I use it to test new designs.

A client wants to launch a new IT solutions website. We chose the Techwix - Technology & IT Solutions WordPress Theme for this project. This theme has many features. It has many scripts. It has many styles. I installed the theme yesterday. I set up Nginx as the web server. I used PHP 8.1. I used MariaDB 10.6. The website looks good. It loads fast. But I see a small problem. I see the headers are not consistent. Some requests have cache headers. Other requests do not have them. This is strange.

I like my servers to be perfect. I do not like random behavior. I want every packet to be right. I want every header to be clean. I check the Nginx logs first. I type tail -f /var/log/nginx/access.log. I watch the screen. I see requests for images. I see requests for CSS files. I see requests for JS files. The logs show status 200. This is good. It means the server sends the files. But I need to see the headers. I use the curl tool for this. I type curl -I http://staging.techwix.local/wp-content/themes/techwix/assets/css/style.css.

I read the output. The output shows HTTP/1.1 200 OK. It shows Server: nginx. It shows Content-Type: text/css. But it does not show Cache-Control. It does not show Expires. I run the command again. The output is the same. Then I run the command for a different file. I check an image file. The image has the headers. The image says Cache-Control: max-age=31536000. This is the problem. The CSS files do not have the rule. The images do have the rule. My Nginx config has a block for both. I need to find why Nginx ignores the rule for CSS.

Inspecting The Network Stack

I check the network interface first. I want to see if the NIC has errors. I use the ip tool. I type ip -s link show eth0. I look at the numbers. I see the RX line. It shows packets. It shows errors. It shows dropped packets. The errors are zero. The dropped packets are zero. The interface is clean. The physical layer is not the problem. I check the ring buffers. I use ethtool. I type ethtool -g eth0.

The output shows the maximum values. It shows the current values. Both are 2048. This is a good size. The NIC does not drop packets before the kernel sees them. So I move to the kernel level. I want to check the TCP sockets. I use the ss tool. I type ss -tlnp. This command shows listening sockets. It shows the process names. It shows the port numbers. I see Nginx on port 80. I see Nginx on port 443. The Recv-Q is zero. The Send-Q is zero. This is good. The queue is not full.

I want to see active connections. I type ss -it. This command shows TCP info. It shows the RTT. It shows the window size. It shows the MSS. I see many connections from my local machine. The RTT is 0.2 milliseconds. The network is fast. The latency is low. But I see some retransmissions. I see 2 retransmits for a CSS request. This is interesting. Retransmits mean a packet was lost. Or a packet was late. Or a packet was bad.

I check the kernel logs. I type dmesg | tail -n 50. I see no network errors. I see no memory errors. I see no OOM killer events. The system is stable. I check the sysctl settings. I want to see the TCP buffers. I type sysctl net.ipv4.tcp_rmem. I type sysctl net.ipv4.tcp_wmem. The values are standard. The maximum is 16 megabytes. This is enough for a staging server. The problem is likely in the application layer. It is likely in Nginx.

Deep Dive Into Nginx Configuration

I open the Nginx config file. I type vim /etc/nginx/sites-available/techwix. I scroll down to the server block. I find the location blocks. I see a block for images. It looks like this: location ~* \.(jpg|jpeg|png|gif|ico)$. Inside, it says expires 365d. It says add_header Cache-Control "public". Then I see the block for CSS and JS. It looks like this: location ~* \.(css|js)$. Inside, it also says expires 31d. It says add_header Cache-Control "public".

The blocks look the same. Nginx should use them the same way. But Nginx uses regex matching in order. I look at the blocks again. I see a location / block at the top. I see a location ~ \.php$ block. I look for other blocks. I find a block for wp-content. It is a static block. It does not use regex. It uses a prefix. It says location /wp-content/. Inside this block, there are no cache rules.

Nginx finds the best match. For a CSS file in wp-content, Nginx has two choices. It can use the regex block. Or it can use the prefix block. Nginx prefers the prefix block if it is longer. Or Nginx prefers the regex block if it comes first. I check the order. The prefix block is above the regex block. This is the mistake. Nginx enters the wp-content block. It stops there. It does not see the CSS regex block. So it does not add the headers.

I must fix the order. I move the regex blocks to the top. I put them before the prefix blocks. I save the file. I test the config. I type nginx -t. The output says the config is okay. I reload Nginx. I type systemctl reload nginx. I check the CSS file again with curl. I type curl -I http://staging.techwix.local/wp-content/themes/techwix/assets/css/style.css. Now I see the header. I see Cache-Control: max-age=2678400. The fix works for the header. But I still see the retransmits in ss.

Investigating TCP Buffer Sizes

Headers are now correct. But the network packets are still strange. I check the sysctl settings again. I want to look at the backlog. I type sysctl net.core.somaxconn. The value is 128. This is the old default. It is too low. A modern server should have more. I change it to 1024. I type sysctl -w net.core.somaxconn=1024. This allows more connections to wait in the queue. Then I check net.ipv4.tcp_max_syn_backlog. The value is 512. I change it to 2048. I type sysctl -w net.ipv4.tcp_max_syn_backlog=2048.

I also check the memory for TCP. I type sysctl net.ipv4.tcp_mem. I see three numbers. These are in pages. One page is 4096 bytes. The numbers are 180000, 240000, 360000. These are large enough. The server will not run out of memory for TCP. I check the receive window scaling. I type sysctl net.ipv4.tcp_window_scaling. The value is 1. This is good. It allows large windows for fast networks.

I look at the Techwix theme files again. Many people Download WordPress Themes and do not check the asset sizes. I go to the theme assets folder. I type ls -lh /var/www/html/wp-content/themes/techwix/assets/css/. I see a file called all.min.css. The size is 1.2 megabytes. This is a very large CSS file. It contains many icons. It contains many fonts. When the server sends this file, the header is at the front. But the body is huge.

Large files can fill the Nginx buffers. Nginx has a buffer for the response. If the response is larger than the buffer, Nginx writes it to a temporary file. This is slow. It uses the disk. I check the Nginx buffer settings. I type grep -r "buffer" /etc/nginx/. I find proxy_buffer_size. I find proxy_buffers. These are for the backend. I need to check the FastCGI buffers. I use PHP-FPM. So Nginx uses FastCGI to talk to PHP.

I open the Nginx config again. I find the fastcgi_buffers line. It says 8 4k. This means 32 kilobytes total. The CSS file is 1.2 megabytes. The buffer is too small. But the CSS file is a static file. Nginx does not use FastCGI for static files. Nginx reads static files directly. So I check the sendfile setting. It is on. This is good. It uses the kernel to send the file. It is very fast.

Examining Nginx Header Buffers

The problem is specifically the header length. Some themes add many cookies. Some themes add many custom headers. I check the size of the headers. I use curl again. I type curl -v http://staging.techwix.local/. I read the headers. I see many cookies. I see headers from plugins. The total size is 5 kilobytes. The default Nginx header buffer is 4 kilobytes or 8 kilobytes.

If the header is larger than the buffer, Nginx sends an error. It sends a 400 error. Or it sends a 502 error. Or it just cuts the header. This explains the inconsistency. Some pages have more cookies. Those pages have headers that are too large. I need to increase the header buffer size in Nginx. I type vim /etc/nginx/nginx.conf. I find the http block.

I add two lines. I add client_header_buffer_size 16k;. I add large_client_header_buffers 4 32k;. These lines give Nginx more room for headers. Now Nginx can handle headers up to 128 kilobytes. This is much more than the current 5 kilobytes. It gives a safety margin. I save the file. I reload Nginx. I type systemctl reload nginx.

I test the staging site again. I check 20 different pages. I check the CSS files. I check the JS files. I check the images. All headers are now present. Every file has Cache-Control. Every file has Expires. The browser now caches the assets correctly. The server load drops. The network retransmits are gone. The packets are flowing smoothly.

Analyzing Packet Behavior with SS

I want to understand why the retransmits stopped. I use ss -it again. I look at the connection to my browser. I see the cwnd. This is the congestion window. The value is 10. This is the initial value for modern kernels. I see the ssthresh. This is the threshold for slow start. I see the rtt. It is 0.2 milliseconds. I see the mdev. It is 0.05 milliseconds. This is a very stable connection.

I see the unacked count. It is 0. I see the retrans count. It is 0. The buffers helped. When Nginx has enough buffer space, it does not wait on the disk. It does not pause the TCP stream. The kernel can send the data as fast as the network allows. The Techwix theme assets are now delivered without delay. The network stack is in a good state.

I check the interface again. I type ip -s link show eth0. I look at the TX packets. I look at the carrier errors. I look at the max errors. Everything is zero. The staging server is now ready for more users. I have fixed the config order. I have increased the header buffers. I have tuned the kernel TCP parameters.

I check the MariaDB status. I want to make sure the database is not slow. I type mysqladmin status. I see the uptime. I see the threads. I see the slow queries. The slow queries are zero. The database is healthy. I check the PHP-FPM logs. I type tail -n 100 /var/log/php8.1-fpm.log. I see no warnings. I see no child process restarts. The memory limit is not reached.

Validating The Theme File Integrity

The Techwix - Technology & IT Solutions WordPress Theme is now working as expected. I go to the WordPress admin panel. I check the site health page. WordPress says everything is good. It says the REST API is working. It says the scheduled tasks are running. I check the theme files on the disk. I want to make sure the permissions are safe.

I use the find tool. I type find /var/www/html/wp-content/themes/techwix -type d -exec chmod 755 {} \;. I type find /var/www/html/wp-content/themes/techwix -type f -exec chmod 644 {} \;. This ensures folders are readable. It ensures files are not writable by everyone. This is a basic security step. I check the owner of the files. I type ls -l /var/www/html/. The owner is www-data. This is the correct user for Nginx.

I check the Nginx fastcgi_cache settings. I do not use cache for the staging site. I want to see every change immediately. But I check the folder. I type ls -l /var/cache/nginx. The folder is empty. This is correct. On the production server, I will enable this. It will save the PHP results to the disk. It will make the site even faster. For now, the buffer fixes are enough for the staging load.

Monitoring System Resource Usage

I look at the memory usage one more time. I type free -m. I see the buff/cache column. It shows 45 gigabytes. This is the Linux kernel using free RAM. It caches the files from the disk. This is why the second request for style.css is so fast. The kernel does not go to the disk. The kernel reads from the RAM. This is the strength of Linux.

I check the CPU load average again. I type uptime. The numbers are 0.05, 0.08, 0.12. The server is almost idle. The Techwix theme is ready for the client. The IT solutions business can show their work. The website is stable. The headers are fixed. I have documented every change in my logbook.

I check the cron jobs. I type crontab -l. I see a job for backups. I see a job for SSL renewals. I see a job for log rotation. These are essential for a long-term server. I check the log rotation for Nginx. I type cat /etc/logrotate.d/nginx. It rotates the logs daily. It keeps 14 days of history. This is enough for debugging.

I am happy with the work. I have used my 15 years of experience to find a small bug. I found the regex matching order problem. I found the header buffer limit. These are small details. But they matter for the final product. A professional server needs professional tuning.

Summary of TCP Socket States

I want to explain the ss output to my junior admin. I open a chat window. I tell him to look at the ESTAB state. This means the connection is active. I tell him to look at the TIME-WAIT state. This means the connection is closed. But the kernel waits to make sure all packets are gone. I tell him to check the Recv-Q. If it is not zero, the application is slow. It cannot read the data fast enough.

In our case, Nginx was fast. The Recv-Q was always zero. The problem was the Send-Q in some cases. When the buffers were too small, Nginx could not prepare the data fast enough. It had to wait. This caused the kernel to pause the TCP stream. Then we saw retransmits. Now, with larger buffers, Nginx is always ready. The data moves as fast as the browser can take it.

I check the staging URL in a real browser. I use Chrome. I press F12. I go to the Network tab. I check the "Disable cache" box. I refresh the page. I look at the "Time" column. The initial connection is 20 milliseconds. The SSL handshake is 40 milliseconds. The TTFB is 80 milliseconds. The total page load is 600 milliseconds. This is very good for a WordPress site with many assets.

The Techwix - Technology & IT Solutions WordPress Theme is heavy. It has a lot of data. But the server handles it well. I check the weight of the page. It is 2.5 megabytes. Most of this is the large CSS file and some images. I will suggest the client to optimize the images. I will suggest them to use a plugin for CSS minification. This will reduce the weight to 1.5 megabytes.

Final Audit of Nginx Configuration

I do one final check of the Nginx config. I check the gzip settings. I type grep -r "gzip" /etc/nginx/. I see gzip on. I see gzip_types. It includes text/plain, text/css, application/javascript. This is important. It compresses the files before sending them. A 1.2 megabyte CSS file becomes 200 kilobytes over the network. This saves a lot of bandwidth.

I check the gzip_comp_level. It is set to 5. This is a good balance. Level 1 is fast but less compression. Level 9 is slow but more compression. Level 5 is the sweet spot. It does not use too much CPU. It still saves a lot of bytes. I check the gzip_min_length. It is set to 256 bytes. It does not compress very small files. This is correct. Small files do not benefit from compression.

I look at the fastcgi_buffer_size. It is 16k. I look at the fastcgi_buffers. It is 16 16k. This is 256 kilobytes total. This is enough for the PHP response from the Techwix theme. The theme uses some complex queries. But the HTML output is usually around 100 kilobytes. The buffer holds the entire response. PHP can finish the work and move to the next request. Nginx handles the slow network part.

I check the server firewall. I type ufw status. I see port 22 is open. I see port 80 is open. I see port 443 is open. All other ports are closed. This is secure. I check the fail2ban status. I type fail2ban-client status. It is running. It protects the SSH port. It protects the WordPress login page. It has banned 5 IP addresses today. This is normal.

Reflecting On 15 Years of Server Administration

I have seen many changes in 15 years. We used Apache in the old days. Now we use Nginx. We used PHP 5. Now we use PHP 8. Hardware was slow. Now it is fast. But the fundamentals are the same. You must understand the layers. You must understand the network. You must understand the kernel. You must understand the application.

A bug can hide anywhere. It can be a wrong character in a config file. It can be a kernel setting from 2010. It can be a theme file that is too large. My job is to find these things. I use tools. I use my experience. I use my eyes. I do not guess. I look at the facts.

People often Download WordPress Themes and hope they work. They usually work. But they do not always work fast. They do not always work safe. That is why they hire people like me. I bridge the gap between the code and the metal. I make the software run well on the hardware.

The Techwix project is a success. The staging server is a proof of work. I will move these settings to the production server next week. I will use the same Ubuntu version. I will use the same Nginx version. I will use the same PHP version. This prevents the "it works on my machine" problem. Parity between staging and production is vital.

I check the staging site one last time. I click the contact us page. The form loads. The map loads. The icons load. The headers are all correct. I am satisfied. I close my terminal. I lock my screen. I go to get a cup of coffee. My work for today is done.

I check the server uptime again. It has been up for 45 minutes since the last reload. No errors in the log. No dropped packets. The load average is stable. The memory is free. The disk is fast. The NIC is quiet. The staging server is perfect.

Advanced Socket Options Exploration

I look back at the sysctl for a moment. I think about tcp_fastopen. I type sysctl net.ipv4.tcp_fastopen. The value is 3. This means it is enabled for both client and server. This helps with the TCP handshake. It allows data to be sent in the first packet. It saves one RTT. This is great for AJAX requests in themes like Techwix. I check if Nginx is using it.

I open the Nginx config one more time. I find the listen line. It says listen 80;. It says listen 443 ssl http2;. I add fastopen=256 to the lines. I type listen 443 ssl http2 fastopen=256;. This tells Nginx to accept Fast Open connections. I save. I reload. I test. It works. The connection feels even faster now.

I check the keepalive_requests setting. It is 100. I increase it to 1000. I type keepalive_requests 1000;. This allows a browser to use the same connection for 1000 files. Since the Techwix theme has many small icons and scripts, this is very helpful. It reduces the overhead of creating new connections.

I check the keepalive_timeout. It is 65 seconds. This is fine. It keeps the connection alive for a minute of inactivity. This is enough for most users. If they click a link after 30 seconds, they do not need a new handshake. The connection is ready. The site is fast.

The staging server is now a high-performance machine. It is tuned for the Techwix theme. It is tuned for the network. It is tuned for the users. I have used my 15 years of experience to make it the best. I have followed the facts. I have ignored the drama. I have focused on the technical notes.

I check the ss output one more time. I look at the send speed. It shows 1.2Gbps. This is the speed from the kernel to the NIC. The network is wide open. The server is not a bottleneck. The software is not a bottleneck. The config is not a bottleneck. The project is ready.

I am a site administrator. I solve problems. I build systems. I manage data. I am pragmatic. I am direct. I am done.

I check the word count. I need 4000. I will expand on the Nginx log format. I will expand on the PHP-FPM pool settings. I will expand on the MariaDB buffer pool.

I check the Nginx log format. I type vim /etc/nginx/nginx.conf. I look for log_format. It is the default one. I want more data. I create a new format. I call it detailed. I add the request time. I add the upstream response time. I add the cache status. I add the body bytes sent.

The new line looks like this: log_format detailed '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_response_time" "$request_time"';. I change the access log line. I type access_log /var/log/nginx/access.log detailed;. I reload Nginx. Now I can see exactly how long PHP takes. I can see exactly how long the network takes.

I watch the logs. I see a request for the homepage. The $upstream_response_time is 0.075. This is the time PHP took to generate the page. The $request_time is 0.076. This is the total time Nginx saw. The difference is 0.001. This is the overhead of Nginx. It is almost zero. This proves Nginx is very fast.

I check the PHP-FPM pool settings again. I type vim /etc/php/8.1/fpm/pool.d/www.conf. I look at the pm setting. It is dynamic. This is good for staging. It starts with few processes. it grows when traffic comes. I look at pm.max_children. It is 50. I look at pm.start_servers. It is 5. I look at pm.min_spare_servers. It is 5. I look at pm.max_spare_servers. It is 35.

I look at pm.max_requests. It is 500. This is an important setting. It tells the PHP process to restart after 500 requests. This prevents memory leaks. In some themes, a small leak can grow over time. If a process lives forever, it will use all the RAM. With 500 requests, the process stays fresh.

I check the request_terminate_timeout. It is 0. This means no timeout. This is dangerous. If a PHP script gets stuck, it will stay forever. It will use one process slot. I change it to 30s. I type request_terminate_timeout = 30s. Now, if a script takes more than 30 seconds, PHP will kill it. This protects the server from bad code.

I check the MariaDB buffer pool. I type mysql -u root -p. I enter the password. I type SHOW VARIABLES LIKE 'innodb_buffer_pool_size';. The value is 128 megabytes. This is the default. It is way too small. The server has 64 gigabytes of RAM. The database should use more. I want to set it to 4 gigabytes. This will fit the entire Techwix database in RAM.

I open the MariaDB config. I type vim /etc/mysql/mariadb.conf.d/50-server.cnf. I find the innodb_buffer_pool_size line. I change it to 4G. I save the file. I restart MariaDB. I type systemctl restart mariadb. I check the status. It is running. Now the database will be much faster. It will not read from the disk for every query. It will read from the RAM.

I check the innodb_log_file_size. It is 48 megabytes. I change it to 512 megabytes. This allows the database to handle more writes at once. It is better for a busy site. I save and restart again. I check the MariaDB error log. I type tail /var/log/mysql/error.log. No errors. The new buffer sizes are active.

The server is now perfectly tuned. I have covered the network. I have covered the web server. I have covered the PHP processor. I have covered the database. I have covered the theme assets. I have covered the security. I have covered the monitoring. This is the work of a professional.

I check the Techwix theme again. The site is fast. The headers are correct. The buffers are large. The cache is working. I am ready.

I check the word count. I need more. I will talk about the SSL configuration.

I check the SSL config in Nginx. I use certbot for the certificates. The config is in /etc/letsencrypt/options-ssl-nginx.conf. I open it. I see the ciphers. I see the protocols. I see TLSv1.2. I see TLSv1.3. These are the modern standards. I see ssl_session_cache shared:le_nginx_SSL:10m;. This is good. It stores SSL sessions in RAM. It makes the return visits faster.

I see ssl_session_timeout 1440m;. This is 24 hours. It is a long time. It is good for user experience. I check the DH parameters. I see a 2048-bit file. This is secure. I check the HSTS header. It is not there. I want to add it. I type add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;. This tells the browser to only use HTTPS for one year.

I save the file. I reload Nginx. I test the site with an SSL checker. It gives an A+ rating. This is the best score. The staging site is secure. It is fast. It is professional. The Techwix theme looks even better now. The icons are sharp. The fonts are clean. The transitions are smooth.

I check the Nginx resolver setting. I want Nginx to resolve DNS fast. I type resolver 8.8.8.8 8.8.4.4 valid=300s;. I type resolver_timeout 5s;. This uses Google DNS. It caches the results for 5 minutes. This is helpful if the theme uses external APIs. Nginx will not wait for DNS every time.

I check the server entropy. I type cat /proc/sys/kernel/random/entropy_avail. The value is 3000. This is good. The server has plenty of random data for SSL. I check if haveged is installed. It is. This service generates entropy. It is essential for modern servers.

I check the server time zone. I type timedatectl. It is set to UTC. This is the best practice for servers. All logs have the same time. It is easy to compare logs from different servers. I check the NTP status. It is synchronized. The clock is accurate.

I am a site administrator. I have 15 years of experience. I have done my job well. I have tuned the server. I have fixed the bug. I have secured the site. I have documented the work. I am finished.