Page History

Squid v2.x (not 3.2 as far as I can tell?) has a feature they coined Collapsed Forwarding. In a nutshell, it allows the intermediary to piggy back multiple clients on a single origin server connections. There are a few ways we could accomplish this, but they all would share the same property: this is only safe to do for cacheable objects.

Problem space and issues

The problem to solve is a form of stampeding herd: When a cached object goes stale, instead of potentially letting 100's or 1000's of requests slip through to the origin, we want to minimize this as much as possible. In a best case scenario,  a redux to 1 origin connection could be achieved. One issue here is to know if a request is cacheable (and therefore shareable). On a complete cache miss (cold cache), this can be difficult at best.

ATS already has three (and a half) features that are in place to alleviate this problem, and we will discuss those three here as well. The discussion should also include options of improving upon these existing features, such that we would not need more complexity.

Fuzzy logic

ATS has three configuration options related to pre-fetching objects before they go stale in cache:

...

cache.

...

The intent here is that there's a small chance (fuzz.probability == 0.5%) that a request for an object will be revalidated, starting 240 seconds before it goes stale. For objects getting few requests per second, this would likely not trigger, but then this feature is not necessary anyways since odds are only 1 or a small number of connections would hit origin upon objects going stale. The defaults are a good compromise, for objects getting roughly 4 requests / second or more, it's virtually guaranteed to trigger a revalidate event within the 240s. These configs are also overridable per remap rule or via a plugin, so can be adjusted per request if necessary.

Finally, the fuzz.min_time is there to be able to handle requests with a TTL less than fuzz.time – it allows for different times to evaluate the probability of revalidation for small TTLs and big TTLs. Objects with small TTLs will start "rolling the revalidation dice" near the fuzz.min_time, while objects with large TTLs would start at fuzz.time. A logarithmic like function between fuzz.min_time and fuzz.time determine the revalidation evaluation start time. As the object gets closer to expiring, the window start becomes more likely. By default this setting is not enabled, but should be enabled anytime you have objects with small TTLs. Note that this option predates overridable configurations, so you can achieve something similar with a plugin or remap.config conf_remap.so configs.

However, in the current versions, as soon as the revalidation request happens, the requested object may no longer be served from cache. As a result, this feature ends up just removing an object from cache before it becomes staleThese configurations are similar to Squid's refresh_stale_hit configuration option.

Read While Write

This feature is pretty similar to how Squid actually implements Collapsed Forwarding. When ATS goes to fetch something from origin, and upon receiving the response, any number of clients can be allowed to start serving out of the partially filled cache object (it starts serving the object once background_fill_completed_threshold % of the object has been received). The difference (Ed: I think?) is that Squid allows this as soon as it goes to origin, whereas ATS can not do it until we get the complete response header. The reason for this is that we make no distinction between cache refresh, and cold cache, so we have no way to know if a response is going to be cacheable, and therefore allow read-while-write functionality.

The configurations necessary to enable this in ATS are:

...

cache

...

All four configurations are required, for the following reasons:

...

object

...

.

...

...

Once all this enabled, you have something that is very close, but not quite the same, as Squid's Collapsed Forwarding.

Open Read Retry Timeout

The open read retry configurations attempt to reduce the number of concurrent requests to the origin for a given object. While an object is being fetched from the origin server, subsequent requests would wait open_read_retry_time milliseconds before checking if the object can be served from cache. If the object is still being fetched, the subsequent requests will retry max_open_read_retries times. Thus, subsequent requests may wait a total of (max_open_read_retries x open_read_retry_time) milliseconds before establishing an origin connection of its own. For instance, if they are set to 5 and 10 respectively, connections will wait up to 50ms for a response to come back from origin from a previous request, until this request is allowed through.

These settings are inappropriate when objects are uncacheable. In those cases, requests for an object effectively become serialized. The subsequent requests would await at least open_read_retry_time milliseconds before being proxies to the origin.

Similarly, this setting should be used in conjunction with Read While Write for big (those that take longer than (max_open_read_retries x open_read_retry_time) milliseconds to transfer) cacheable objects. Without the read-while-write settings enabled, while the initial fetch is ongoing, not only would subsequent requests be delayed by the maximum time, but also, those requests would result in another request to the origin server.

Since ATS now supports setting these settings per-request or remap rule, you can configure this to be suitable for your setup much more easily.

The configurations are (with defaults):

...

Max Open Read Retries

The maximum number of retries before going upstream to parent or origin. This is related to the previous parameter. These settings should only be used if your content is cacheable as otherwise it would just serialize your connections to the origin.

Example: Lets say the typical response times is 50-100ms, try setting proxy.config.http.cache.max_open_read_retries

...

to 3 and proxy.config.http.cache.open_read_retry_time

...

The default means that the feature is disabled, and every connection is allowed to go to origin instantly. When enabled, you will try max_open_read_retries times, each with a open_read_retry_time timeout.

...

to 50

Stale While Revalidate plugin

There is a plugin that implements the Stale While Revalidate cache-control directive, which is documented in RFC 5861. This directive is designed to allows caches to serve a stale object while a background revalidation is occurring. When a popular object becomes stale, subsequent requests would be proxied to the origin for revalidation. With this plugin (and the appropriate Cache-control: directive), you are allowed to serve objects stale in cache, while one request goes to origin to fetch the new version. Combined with read-while-write, and the fuzzy logic feature, this is a good alternative to the Open-Retry feature. The downside is that the current implementation is not complete, and needs changes to the core / cache to solve the problem completely

However, in the current implementation, once the fetch of the object is initiated, the original object is not allowed to be served from cache.

Improvements

Each of these features above could benefit from some improvements. Here are some ideas, but please feel free to add more thoughts and ideas here.

...

Allow for more precise configurations, other than using the overridable configurations. An example would be to for example allow cache.config to specify different fuzzy parameters based on matching regular expressions.

Allow for a true background fill. If the revalidation hits, ideally serve that (and future) requests from cache. Once the revalidation is complete, begin serving the new object.
Or, perhaps the request that triggered the revalidation would be proxied, while subsequent requests are still served from the previously cached object.
And/Or, submit an IMS request initially for the object. It seems likely that the headers will give a new max-age.

Allow read-while-

...

writer to collapse immediately

We could perhaps allow read-while-writewriter to allow immediate collapsed forwarding, IF a cached objects was going stale. It's reasonable to think that the refreshed version will also be cacheable, and it would be pretty safe to do this. We would of course have to deal the case where the response comes back as non-cacheable, then any additional collapsed connections must dislodge from the first connection, and initiate its own origin session.

Implementing this would turn our existing read-while-write writer to be identical to Squid v2.6.

...

We could add a hash table keeping track of what origin sessions are outstanding, and allow the open-retry functionality only kick in when there is a session pending. This is a similar improvement to allowing read-while-write writer collapse immediately, except it's probably easier to implement. With this hash table, we'd want to assure that only objects that were previously cached, and are now stale, are allowed to collapse and start the open-retry cycles.

...

Right now, the cached object is rendered inaccessible as soon as we initiate the first request to origin to revalidate the object. If we could fix the cache such that the old object is still the entry in the directory structure, until the response comes back, read-while-writewriter would work almost perfectly fine. As an optimization, it would be useful to allow serving the old version until the old one is completely filled; this makes large downloads faster and more efficient.

RFC 5861 also specifies a stale-if-error cache-control directive. This is not possible with the current cache – once the object becomes stale, it is no longer available to be served.

Both would be helped by a true background fill as described in the improving fuzzy logic section.