Squid v2.x (not 3.2 as far as I can tell?) has a feature they coined Collapsed Forwarding. In a nutshell, it allows the intermediary to piggy back multiple clients on a single origin server connections. There are a few ways we could accomplish this, but they all would share the same property: this is only safe to do for cacheable objects.

Problem space and issues

The problem to solve is a form of stampeding herd: When a cached object goes stale, instead of potentially letting 100's or 1000's of requests slip through to the origin, we want to minimize this as much as possible. In a best case scenario,  a redux to 1 origin connection could be achieved. One issue here is to know if a request is cacheable (and therefore shareable). On a complete cache miss (cold cache), this can be difficult at best.

ATS already has three (and a half) features that are in place to alleviate this problem, and we will discuss those three here as well. The discussion should also include options of improving upon these existing features, such that we would not need more complexity.

Fuzzy logic

ATS has configuration options related to pre-fetching objects before they go stale in cache. The intent here is that there's a small chance that a request for an object will be revalidated, starting 240 seconds before it goes stale.

However, in the current versions, as soon as the revalidation request happens, the requested object may no longer be served from cache. As a result, this feature ends up just removing an object from cache before it becomes stale.

Read While Write

This feature is pretty similar to how Squid actually implements Collapsed Forwarding. When ATS goes to fetch something from origin, and upon receiving the response, any number of clients can be allowed to start serving out of the partially filled cache object.

Open Read Retry Timeout

The open read retry configurations attempt to reduce the number of concurrent requests to the origin for a given object. While an object is being fetched from the origin server, subsequent requests would wait open_read_retry_time milliseconds before checking if the object can be served from cache.

Max Open Read Retries

The maximum number of retries before going upstream to parent or origin. This is related to the previous parameter. These settings should only be used if your content is cacheable as otherwise it would just serialize your connections to the origin.

 

Example: Lets say the typical response times is 50-100ms, try setting proxy.config.http.cache.max_open_read_retries to 3 and proxy.config.http.cache.open_read_retry_time to 50

Stale While Revalidate plugin

There is a plugin that implements the Stale While Revalidate cache-control directive, which is documented in RFC 5861. This directive is designed to allows caches to serve a stale object while a background revalidation is occurring.

However, in the current implementation, once the fetch of the object is initiated, the original object is not allowed to be served from cache.

Improvements

Each of these features above could benefit from some improvements. Here are some ideas, but please feel free to add more thoughts and ideas here.

Improve Fuzzy Logic with finer granularity

Allow for more precise configurations, other than using the overridable configurations. An example would be to for example allow cache.config to specify different fuzzy parameters based on matching regular expressions.

Allow for a true background fill. If the revalidation hits, ideally serve that (and future) requests from cache. Once the revalidation is complete, begin serving the new object.
Or, perhaps the request that triggered the revalidation would be proxied, while subsequent requests are still served from the previously cached object.
And/Or, submit an IMS request initially for the object. It seems likely that the headers will give a new max-age.

Allow read-while-writer to collapse immediately

We could perhaps allow read-while-writer to allow immediate collapsed forwarding, IF a cached objects was going stale. It's reasonable to think that the refreshed version will also be cacheable, and it would be pretty safe to do this. We would of course have to deal the case where the response comes back as non-cacheable, then any additional collapsed connections must dislodge from the first connection, and initiate its own origin session.

Implementing this would turn our existing read-while-writer to be identical to Squid v2.6.

Add intelligence to Open-Retry timeout

We could add a hash table keeping track of what origin sessions are outstanding, and allow the open-retry functionality only kick in when there is a session pending. This is a similar improvement to allowing read-while-writer collapse immediately, except it's probably easier to implement. With this hash table, we'd want to assure that only objects that were previously cached, and are now stale, are allowed to collapse and start the open-retry cycles.

Improve the cache to handle RFC 5861 better

Right now, the cached object is rendered inaccessible as soon as we initiate the first request to origin to revalidate the object. If we could fix the cache such that the old object is still the entry in the directory structure, until the response comes back, read-while-writer would work almost perfectly fine. As an optimization, it would be useful to allow serving the old version until the old one is completely filled; this makes large downloads faster and more efficient.

RFC 5861 also specifies a stale-if-error cache-control directive. This is not possible with the current cache – once the object becomes stale, it is no longer available to be served.

Both would be helped by a true background fill as described in the improving fuzzy logic section.

  • No labels