Page History

Squid v2.x (not 3.x 2 as far as I can tell?) has a feature they coined Collapsed Forwarding. In a nutshell, it allows the intermediary to piggy back multiple clients on a single origin server connections. There are a few ways we could accomplish this, but they all would share the same property: this is only safe to do for cacheable objects.

Problem space and issues

The problem to solve is a form of stampeding herd: When a cached object goes stale, instead of potentially letting 100's or 1000's of requests slip through to the origin, we want to minimize this as much as possible. In a best case scenario,  a redux to 1 origin connection could be achieved. One issue here is to know if a request is cacheable (and therefore shareable). On a complete cache miss (cold cache), this can be difficult at best.

ATS already has three (and a half) features that are in place to alleviate this problem, and we will discuss those three here as well. The discussion should also include options of improving upon these existing features, such that we would not need more complexity.

Fuzzy logic

ATS has three configuration options related to pre-fetching objects before they go stale in cache:

...

cache

...

.

...

The intent here is that there's a small chance (fuzz.probability == 0.5%) that a request for an object will be revalidated, starting 240 seconds before it goes stale. For objects getting few requests per second, this would likely not trigger, but then this feature is not necessary anyways since odds are only 1 or a small number of connections would hit origin upon objects going stale. The defaults are a good compromise, for objects getting roughly 4 requests / second or more, it's guaranteed to trigger a revalidate event within the 240s. These configs are also overridable per remap rule or via a plugin, so can be adjusted per request if necessary.

Finally, the fuzz.min_time is there to be able to handle requests with a TTL less than fuzz.time; In this case, the revalidation happens in a function between fuzz.min_time and fuzz.time, and it gets closer to expiring, becoming more likely. By default this setting is not enabled, but should be enabled anytime you have objects with small TTLs. Note that this option predates overridable configurations, so you can achieve something similar with a plugin or remap.config conf_remap.so configs.

However, in the current versions, as soon as the revalidation request happens, the requested object may no longer be served from cache. As a result, this feature ends up just removing an object from cache before it becomes staleThese configurations are similar to Squid's refresh_stale_hit configuration option.

Read While Write

This feature is pretty similar to how Squid actually implements Collapsed Forwarding. When ATS goes to fetch something from origin, and upon receiving the response, any number of clients can be allowed to start serving out of the partially filled cache object. The difference (Ed: I think?) is that Squid allows this as soon as it goes to origin, whereas ATS can not do it until we get the complete response header. The reason for this is that we make no distinction between cache refresh, and cold cache, so we have no way to know if a response is going to be cacheable, and therefore allow read-while-write functionality.

The configurations necessary to enable this in ATS are:

CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
CONFIG proxy.config.cache.max_doc_size INT 0

All four configurations are required, for the following reasons:

1. Obviously the feature should be enabled to begin with. It's off by default (as of 4.0.x, we might want to modify this for 5.x)
2. The background fill feature must be allowed to kick in for every request. This is necessary, in case the owning consumer ("client session") goes away, someone needs to take over the session.
3. The max_doc_size must be unlimited, since we potentially don't know the size of the object, we can't allow it to disconnect due to going over this limit.

Once all this enabled, you have something that is very close, but not quite the same, as Squid's Collapsed Forwarding.

Open Retry Timeout

The third configuration option, which is really only suitable when you know everything is cacheable, are related how to deal with locking out cache access until the origin response has come back. Combined with read-while-write this would completely mimic the behavior of Squid's Collapsed Forwarding. The downside is that non-cacheable content would also be serialized, which can severely affect latency. Since ATS now supports setting these settings per-request or remap rule, you can configure this to be suitable for your setup much more easily.

The configurations are (with defaults):

...

Open Read Retry Timeout

The open read retry configurations attempt to reduce the number of concurrent requests to the origin for a given object. While an object is being fetched from the origin server, subsequent requests would wait open_read_retry_time milliseconds before checking if the object can be served from cache.

Max Open Read Retries

The maximum number of retries before going upstream to parent or origin. This is related to the previous parameter. These settings should only be used if your content is cacheable as otherwise it would just serialize your connections to the origin.

Example: Lets say the typical response times is 50-100ms, try setting proxy.config.http.cache.max_open_read_retries

...

to 3 and proxy.config.http.cache

...

.

...

open_read

...

_

...

retry_time

...

to 50

Stale While Revalidate plugin

There is a plugin that implements the Stale While Revalidate cache-control directive, which is documented in RFC 5861. With this plugin, you are allowed This directive is designed to allows caches to serve objects a stale in cache, while one request goes to origin to fetch the new version. Combined with read-while-write, and the fuzzy logic feature, this is a good alternative to the Open-Retry feature. The downside is that the current implementation is not complete, and needs changes to the core / cache to solve the problem completelyobject while a background revalidation is occurring.

However, in the current implementation, once the fetch of the object is initiated, the original object is not allowed to be served from cache.

Improvements

Each of these features above could benefit from some improvements. Here are some ideas, but please feel free to add more thoughts and ideas here.

...

Allow for more precise configurations, other than using the overridable configurations. An example would be to for example allow cache.config to specify different fuzzy parameters based on matching regular expressions.

Allow for a true background fill. If the revalidation hits, ideally serve that (and future) requests from cache. Once the revalidation is complete, begin serving the new object.
Or, perhaps the request that triggered the revalidation would be proxied, while subsequent requests are still served from the previously cached object.
And/Or, submit an IMS request initially for the object. It seems likely that the headers will give a new max-age.

Allow read-while-

...

writer to collapse immediately

We could perhaps allow read-while-writewriter to allow immediate collapsed forwarding, IF a cached objects was going stale. It's reasonable to think that the refreshed version will also be cacheable, and it would be pretty safe to do this. We would of course have to deal the case where the response comes back as non-cacheable, then any additional collapsed connections must dislodge from the first connection, and initiate its own origin session.

Implementing this would turn our existing read-while-write writer to be identical to Squid v2.6.

...

We could add a hash table keeping track of what origin sessions are outstanding, and allow the open-retry functionality only kick in when there is a session pending. This is a similar improvement to allowing read-while-write writer collapse immediately, except it's probably easier to implement. With this hash table, we'd want to assure that only objects that were previously cached, and are now stale, are allowed to collapse and start the open-retry cycles.

...

Right now, the cached object is rendered inaccessible as soon as we initiate the first request to origin to revalidate the object. If we could fix the cache such that the old object is still the entry in the directory structure, until the response comes back, read-while-writewriter would work almost perfectly fine. As an optimization, it would be useful to allow serving the old version until the old one is completely filled; this makes large downloads faster and more efficient.

RFC 5861 also specifies a stale-if-error cache-control directive. This is not possible with the current cache – once the object becomes stale, it is no longer available to be served.

Both would be helped by a true background fill as described in the improving fuzzy logic section.