Caching 301 redirects in Varnish while keeping the protocol

I have noticed a drop in our Varnish cache hit ratio and, upon investigation, found that the backend was not allowing 301 redirects to be cached:

Client request:

   42 RxRequest    c GET

   42 RxURL        c /img_resized/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

   42 RxProtocol   c HTTP/1.1

Backend request: 

   21 TxRequest    b GET

   21 TxURL        b /img_resized/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

   21 TxProtocol   b HTTP/1.1

Backend response: 

   21 RxHeader     b Cache-Control: no-cache

   21 RxHeader     b Location: http://static.example.com/img/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

   21 RxHeader     b Status: 301

I logged a bug for the application to be fixed and, in the meantime, added a VCL snippet to override the backend headers:

        # WSF-xxxx: App is not allowing redirects to be cached

        if ((req.http.Host == "static.example.com") &&

           (beresp.status == 301) &&

           (beresp.http.Cache-Control ~ "no-cache")) {

                set beresp.http.Cache-Control = "public, max-age=604800";

        }

There's a catch though.  We do SSL offloading for static.example.com in our F5 BigIP LTM load balancer.  In other words, the load balancer receives a https request and, in turn, makes a http request to Varnish.

As Varnish doesn't support SSL natively it is unaware of the protocol (http or https) being used; thus, the requests below get cached with the first answer it sees:

HTTP request

curl -I http://static.example.com/img/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

(...)

HTTP/1.1 301 Moved Permanently

Location: http://static.example.com/img/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

HTTPS request

curl -I https://static.example.com/img/au/images/gifts/2012/hero/christmas-homepage-hero-alyce.jpg

(...)

HTTP/1.1 301 Moved Permanently

Location: http://static.example.com/img/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

Notice that the HTTPS request was redirected to a HTTP URL.  This will most certainly cause issues, including mixed-mode SSL alerts in some browsers.

Our backend is a Ruby on Rails application and, to my surprise, it understands the X-Forwarded-Proto HTTP header.  This means that the application will use the value of X-Forwarded-Proto to build the URL in the Location header.

I improved the F5 iRule by instructing the load balancer to insert a X-Forwarded-Proto header in HTTPS requests which were being off-loaded to Varnish:

when HTTP_REQUEST {

      HTTP::header replace X-Forwarded-Proto "https"

}

In addition to this, I also had to instruct Varnish to use the X-Forwarded-Proto header in the cache hash:

sub vcl_hash {

if (req.http.X-Forwarded-Proto) {

set req.hash += req.http.X-Forwarded-Proto;

}

}

With these tweaks Varnish is now able to account for SSL offloading and correctly serve the appropriate cached version of a page.  An improvement on this configuration would be to only use X-Forwarded-Proto in the hash if the response is a redirect; at the moment I can't be bothered that Varnish is caching objects twice.

HTTP request

$curl -I http://static.example.com/img_resized/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

HTTP/1.1 301 Moved Permanently

Location: http://static.example.com/img/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

HTTPS request

$curl -I https://static.example.com/img_resized/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

HTTP/1.1 301 Moved Permanently

Location: https://static.example.com/img/au/images/gifts/2012/rectangle-75-58/christmas-promo-tile-alyce.jpg

Varnish and the Linux IO bottleneck

There are 2 main ways Varnish caches your data:

1. to memory (with the malloc storage config)
2. to disk (with the file storage config)

As Kristian explains in a best practices post, you get better performance with #1 if your cache size fits in RAM. With either method the most accessed objects end-up in RAM. While the malloc method puts everything in RAM and the kernel swaps it out to disk, the file storage method puts everything in disk and the kernel caches it in memory.

We have been using the file storage for a little more than 3 months on servers with 16GB of RAM and a file storage of 30GB. Recently, because of an application configuration error, the size of the cache grew from its usual maximum of 6GB to 20GB+.

Since the system only has 16GB, the kernel now needs to choose which pages to keep in RAM. When a dirty mmap'ed page needs to be released the kernel will write the changes to disk before doing so. More than that, the linux kernel will proactively write dirty mmap'ed pages to disk.

On a Varnish server with this setup or any other application that uses mmap() for large files, the situation translates to constant disk activity.

At some point the varnish worker process will be blocked by a kernel IO call. The Varnish manager process, with no way to tell what's happening to the child, believes it has stopped responding and kills it. The log entries below are typical:

00:30:10.395590 varnishd[22919]: Child (22920) not responding to ping, killing it.

00:30:10.395622 varnishd[22919]: Child (22920) not responding to ping, killing it.

00:30:10.417309 varnishd[22919]: Child (22920) died signal=3

At this point you lose your cached data and the system goes back to normal. After some time, the size of the cache will grow to be larger than your server's RAM again, repeating the kill-restart cycle.

You could choose to give the client more time to respond to the manager process' ping requests. This is done by increasing the default value of cli_timeout from 10 seconds (e.g. varnish -p cli_timeout=20) but this is merely masking the issue. The real issue is that the Linux kernel is busy writing dirty pages to disk.

There are parameters you can use to control how much time the kernel spends writing dirty mmap pages to disk. I have spent some time fine-tuning the parameters below with little results:

/proc/sys/vm/dirty_writeback_centisecs

/proc/sys/vm/dirty_ratio

/proc/sys/vm/dirty_background_ratio

In RHEL 5.x kernels you can completely disable committing changes to mmap'ed files:

echo 0 > /proc/sys/vm/flush_mmap_pages

Short of reducing the file storage size to fit in RAM, this is the best solution I found so far. Be aware that using this with the upcoming persistent storage in Varnish is a really bad idea as you risk serving corrupt and/or stale data. This is only acceptable because the mmap'ed data, in this setup, is superfluous: varnish throws away the cache on restart and it can always fetch the object from the backend.

I have asked Red Hat if they plan to keep the flush_mmap_pages setting in RHEL 6 but haven't received a response yet.  They did, however, confirm that msync() calls are honoured and dirty pages being evicted from RAM are committed to disk.