Debugging Missing Cookies via tcpdump and HAProxy

An anomalous support ticket entered the queue early Tuesday morning. The client reported that customers utilizing Safari on iOS devices or Firefox with Enhanced Tracking Protection (ETP) enabled were experiencing persistent authentication failures. The users could log into the primary WordPress frontend, but when navigating to the billing portal subdomain to view their invoices, the session was dropped. The browser would display a redirect loop, eventually landing back on the unauthenticated login prompt. The infrastructure had recently undergone a UI refresh, migrating the frontend to the Hostingard - Web Hosting WordPress Theme with WHMCS to consolidate the service catalog and the client area. The initial load testing over standard broadband and Chrome environments showed a consistent, flawless Single Sign-On (SSO) handshake. The backend metrics were completely stable. CPU utilization on the web nodes was hovering around 12%, memory usage was flat, and the database query execution times were nominal. This was a highly specific, edge-case network issue that required isolating the variable between the client's mobile browser, the cross-subdomain routing, and our edge proxies.

The Application Layer Monitoring Blind Spot

The standard operating procedure for diagnosing a dropped session is to examine the application server logs. I accessed the primary application node and reviewed the Nginx access logs and the PHP-FPM error logs. The findings were contradictory to the user report. The Nginx access log recorded the inbound POST request to the WordPress authentication endpoint. The logged upstream response time was 42 milliseconds, and the HTTP status code was a 302 Found, redirecting to the WHMCS bridge endpoint. The payload size recorded in the log was exactly 840 bytes. The PHP-FPM slow log was completely empty.

<p>From the perspective of the application, the transaction was entirely successful and processed rapidly. There was no application-level locking, no slow database queries, and no PHP execution delays. The authentication token was generated and pushed to the Nginx send buffer. The drop was occurring after the application had finished its work, residing somewhere in the space between the Nginx socket buffer and the physical mobile device. This discrepancy isolates the problem to the HTTP state management and the TCP/IP stack. When the application writes a response to the socket, the Linux kernel network stack assumes responsibility for delivering those packets. The application log records the time the response was generated, not the time the final TCP ACK was received or how the browser parsed the headers.</p>

Packet Capture and TCP Handshake Analysis

To inspect the actual packet flow and header transmission, I needed to capture the raw traffic on the external network interface. Simulating a Safari connection with cross-site tracking prevention enabled was necessary to replicate the exact path. I utilized a test iOS device, tethered to a cellular connection, and initiated the login sequence while simultaneously running tcpdump on the target Nginx proxy server.

I executed the following packet capture command, utilizing a Berkeley Packet Filter (BPF) syntax to isolate the specific client IP address and the HTTPS port, writing the raw packets to a binary file for subsequent analysis in Wireshark.

<pre>tcpdump -i eth0 host 198.51.100.45 and port 443 -s 0 -w /tmp/safari_sso_drop.pcap</pre>

The -s 0 flag instructs tcpdump to capture the entire packet payload, not just the headers, which is critical for inspecting the encrypted HTTP traffic once the session keys are extracted. After transferring the pcap file to my local workstation and loading it into Wireshark, I isolated the TCP stream associated with the authentication request. The initial 3-way handshake proceeded normally. The client sent a SYN packet, the server responded with a SYN-ACK, and the client returned the final ACK.

Following the TLS 1.3 cryptographic handshake, the browser transmitted the encrypted HTTP POST payload containing the login credentials. The server acknowledged the payload and, 45 milliseconds later, began transmitting the encrypted HTTP 302 redirect response. Decrypting this payload using the exported TLS session keys revealed the root cause of the session drop.

The SameSite Cookie Policy Blackhole</h2>

The Wireshark trace exposed the exact point of failure during the transmission of the response headers. The PHP backend generated a Set-Cookie header containing the WHMCS session identifier. However, the header string looked like this:

<pre>Set-Cookie: WHMCS_SESSION=a1b2c3d4e5f6g7h8; Path=/; HttpOnly</pre>

The modern web standard for controlling cross-site cookie behavior is the SameSite attribute. By default, modern browsers, especially Safari with its strict Intelligent Tracking Prevention (ITP) and Firefox with ETP, treat cookies without an explicit SameSite attribute as SameSite=Lax. The Lax policy permits the cookie to be sent during top-level navigations (like following a standard link) but strips the cookie entirely during cross-site POST requests, or when transitioning between subdomains if the domain attribute is not strictly configured.

Because the WordPress installation resided on www.example.com and the WHMCS billing portal resided on billing.example.com, the browser treated the SSO redirect as a cross-site context. Safari, enforcing its privacy policies, accepted the cookie during the initial WordPress login but refused to transmit it when the 302 redirect pointed the browser to the billing.example.com subdomain. The WHMCS bridge received the request, found no session identifier, and threw the user back to the login screen.

o instruct Safari to include the session cookies when navigating between the application subdomains, the cookie must be explicitly marked with SameSite=None; Secure, and the Domain attribute must be set to the root domain (.example.com). The Secure flag is strictly mandatory when using SameSite=None; the browser will reject the directive if the connection is not encrypted via HTTPS.

Remediating the Cookie Policy with Nginx Header Manipulation

The immediate correction required modifying the HTTP response to append the necessary cookie attributes. Relying on the underlying PHP application to set these headers is often unreliable. Unlike a generic Free Download WooCommerce Theme where a developer might manually hook into the init action to force session parameters via session_set_cookie_params(), modifying core integration plugin code is bad practice. The next time the vendor releases an update, the patch is overwritten, and the authentication flow breaks again.

Instead of touching the PHP code, I implemented a centralized header manipulation rule directly within the Nginx reverse proxy configuration. This approach ensures consistent enforcement across all responses leaving the server, independent of the underlying application logic.

I utilized the Nginx map directive to parse the outbound Set-Cookie headers generated by the PHP-FPM backend, evaluate them using regular expressions, and append the necessary attributes before transmitting the response to the client.

<pre># /etc/nginx/conf.d/samesite-cookie-routing.conf

# Use a strict regular expression to match the cookie string and append the required attributes
# The regex captures the entire cookie string up to the end, ignoring existing SameSite directives if present,
# then appends Domain=.example.com; SameSite=None; Secure
"~*(?<cookie_base>^WHMCS_SESSION=[^;]+(?:; .*)?$)" "$cookie_base; Domain=.example.com; SameSite=None; Secure";
"~*(?<cookie_base>^wordpress_logged_in_[^;]+(?:; .*)?$)" "$cookie_base; Domain=.example.com; SameSite=None; Secure";

location ~ \.php$ {
    fastcgi_pass unix:/run/php/php8.2-fpm.sock;
    fastcgi_index index.php;
    include fastcgi_params;

    # Hide the original Set-Cookie header generated by the PHP-FPM process
    # We do not want to send duplicate cookie headers to the client
    fastcgi_hide_header Set-Cookie;

    # Inject the newly formatted cookie string containing the SameSite directives back into the response
    # The 'always' parameter ensures the header is added even if the backend returns a non-200 status code
    add_header Set-Cookie $modified_cookie always;

    # Tune FastCGI buffers to handle the redirect headers entirely in memory
    fastcgi_buffer_size 16k;
    fastcgi_buffers 16 16k;
}

}</pre>

This configuration intercepts the HTTP response exactly as it leaves the FastCGI Unix socket. If the backend PHP process attempts to set a session cookie, Nginx captures the header, evaluates the regular expression, appends the root domain and the SameSite=None; Secure string, and injects the modified header back into the final response sent to the mobile client.

Upon the next login attempt, the Safari browser receives the modified cookie directive. When the 302 redirect pushes the user to the billing subdomain, the WebKit engine evaluates the SameSite=None policy, recognizes the root domain match, and successfully includes the WHMCS_SESSION cookie in the inbound request headers to the new endpoint. The PHP backend successfully receives the correct session identifier, bypasses the creation of a new, empty session, and renders the authenticated client area.

FastCGI Buffers and Header Truncation

While testing the Nginx header manipulation, a secondary, intermittent error surfaced in the error logs: upstream sent too big header while reading response header from upstream. When Nginx returned this error, the client received a 502 Bad Gateway instead of the intended 302 redirect.

This error occurs when the HTTP response headers generated by PHP-FPM exceed the size of the memory buffer Nginx allocates to read them. In our integration, the WHMCS bridge was generating multiple large cookies, including encoded authentication tokens and user state arrays. The total size of the Set-Cookie headers exceeded 4 kilobytes.

By default, Nginx sets the fastcgi_buffer_size equal to the memory page size of the underlying operating system. On our x86_64 architecture, this is 4096 bytes (4K). If the response headers exceed 4K, Nginx cannot read them into the primary buffer and terminates the connection to the upstream socket.

To resolve this, I explicitly expanded the FastCGI buffer directives within the http block to accommodate the bloated session headers.

<pre># /etc/nginx/nginx.conf http { # The size of the buffer used for reading the first part of the response, usually the HTTP headers. # Increased from the 4k default to 16k to accommodate large SSO cookies. fastcgi_buffer_size 16k;

# Sets the number and size of the buffers used for reading the response body.
fastcgi_buffers 16 16k;

# Limit the amount of data written to a temporary file if the response exceeds the buffers.
# Setting this to 0 disables disk buffering entirely for FastCGI responses.
fastcgi_max_temp_file_size 0;

}</pre>

Expanding fastcgi_buffer_size to 16k ensures Nginx can absorb the entire header block in RAM. Setting fastcgi_max_temp_file_size 0 prevents Nginx from attempting to write overflow response bodies to disk in /var/lib/nginx/fastcgi/, which introduces unnecessary disk I/O latency on high-throughput API endpoints.

Tracing TCP Keepalive Stalls on Upstream Sockets

The cookie policy and buffer adjustments stabilized the authentication flow for the vast majority of users. However, the APM telemetry highlighted a residual issue. Approximately 0.1% of requests to the backend WHMCS API were experiencing a rigid 60-second delay before either succeeding or returning a 504 Gateway Timeout. The delay was always exactly 60 seconds.

A rigid 60-second delay is a strong indicator of a TCP timeout parameter. I examined the Nginx upstream configuration used to proxy API requests to the isolated WHMCS container.

<pre>upstream whmcs_api_backend { server 10.0.2.50:9000;

# Maintain a pool of 32 idle connections to the backend
keepalive 32;

    # Enable HTTP/1.1 for keepalive connections
    fastcgi_keep_conn on;

    fastcgi_read_timeout 60s;
}

}</pre>

The keepalive 32 directive instructs Nginx to keep 32 TCP sockets open and idle after a request completes, rather than tearing them down via a TCP FIN/ACK sequence. This reduces the overhead of the 3-way handshake on subsequent requests. The fastcgi_keep_conn on directive tells the FastCGI server (PHP-FPM) to also keep the connection open.

The problem arises from a mismatch in connection lifecycle management between Nginx, the Linux kernel, and the PHP-FPM daemon. Nginx was holding the socket open in its internal connection pool. However, if the backend PHP-FPM worker was recycled (e.g., hitting its pm.max_requests limit) or if an intermediate stateful firewall dropped the idle connection, the socket state became desynchronized.

Nginx would select an idle socket from its keepalive pool and send the FastCGI request. Because the backend had silently closed the connection, Nginx's write operation would succeed (placing data into the local kernel send buffer), but the subsequent read operation would hang, waiting for a response that would never arrive. Nginx would wait exactly 60 seconds, dictated by the fastcgi_read_timeout 60s directive, before giving up and returning a 504 error.

To detect and terminate these dead peer connections rapidly, I needed to implement TCP Keepalives at the kernel level for the upstream sockets.</p>

Tuning the Linux TCP Stack for Proxy Connections

<p>The Linux kernel provides TCP Keepalive probes to monitor the health of idle connections. By default, the kernel waits 7,200 seconds (2 hours) before sending the first probe. This is entirely unsuitable for a high-traffic web proxy environment.</p>

I adjusted the sysctl parameters to aggressively probe idle connections.

<pre># /etc/sysctl.d/99-tcp-keepalive.conf

net.ipv4.tcp_keepalive_probes = 3</pre>

Applying these settings globally alters the behavior for all sockets on the server. However, Nginx must be explicitly instructed to enable the SO_KEEPALIVE socket option on its upstream connections.

While Nginx supports enabling keepalives on downstream client connections via the so_keepalive parameter in the listen directive, it does not natively expose a directive to enable SO_KEEPALIVE on upstream FastCGI connections. The proxy simply relies on the operating system defaults when calling the socket() and connect() syscalls.

Because we cannot force Nginx to send TCP keepalives upstream without recompiling the source code or using third-party modules, the most robust solution is to manage the connection lifecycle strictly at the application layer and the proxy layer to prevent long-lived idle sockets from becoming stale.

<pre># /etc/nginx/nginx.conf upstream whmcs_api_backend { server 10.0.2.50:9000; keepalive 32; # Tell Nginx to close the keepalive connection after 100 requests keepalive_requests 100; # Tell Nginx to close the keepalive connection if it remains idle for 30 seconds keepalive_timeout 30s; }</pre>

By enforcing a keepalive_timeout of 30 seconds on the upstream pool, Nginx will proactively close the TCP socket before the intermediate network devices or the backend PHP-FPM workers have a chance to drop it silently. This guarantees that when Nginx pulls a socket from the pool, the connection is fresh and valid, eliminating the 60-second read timeout stalls entirely.

<pre># Apply the Nginx configuration nginx -t && systemctl reload nginx</pre>