Squid v2.x (not 3.x as far as I can tell?) has a feature they coined Collapsed Forwarding. In a nutshell, it allows the intermediary to piggy back multiple clients on a single origin server connections. There are a few ways we could accomplish this, but they all would share the same property: this is only safe to do for cacheable objects.
The problem to solve is a form of stampeding herd: When a cached object goes stale, instead of potentially letting 100's or 1000's of requests slip through to the origin, we want to minimize this as much as possible. In a best case scenario, a redux to 1 origin connection could be achieved. One issue here is to know if a request is cacheable (and therefore shareable). On a complete cache miss (cold cache), this can be difficult at best.
ATS already has three (and a half) features that are in place to alleviate this problem, and we will discuss those three here as well. The discussion should also include options of improving upon these existing features, such that we would not need more complexity.
ATS has three configuration options related to pre-fetching objects before they go stale in cache:
CONFIG proxy.config.http.cache.fuzz.time INT 240 CONFIG proxy.config.http.cache.fuzz.min_time INT 0 CONFIG proxy.config.http.cache.fuzz.probability FLOAT 0.005 |
The intent here is that there's a small chance (fuzz.probability == 0.5%) that a request for an object will be revalidated, starting 240 seconds before it goes stale. For objects getting few requests per second, this would likely not trigger, but then this feature is not necessary anyways since odds are only 1 or a small number of connections would hit origin upon objects going stale. The defaults are a good compromise, for objects getting roughly 4 requests / second or more, it's guaranteed to trigger a revalidate event within the 240s. These configs are also overridable per remap rule or via a plugin, so can be adjusted per request if necessary.
Finally, the fuzz.min_time is there to be able to handle requests with a TTL less than fuzz.time; In this case, the revalidation happens in a function between fuzz.min_time and fuzz.time, and it gets closer to expiring, becoming more likely. By default this setting is not enabled, but should be enabled anytime you have objects with small TTLs. Note that this option predates overridable configurations, so you can achieve something similar with a plugin or remap.config conf_remap.so configs.
These configurations are similar to Squid's refresh_stale_hit configuration option.
This feature is pretty similar to how Squid actually implements Collapsed Forwarding. When ATS goes to fetch something from origin, and upon receiving the response, any number of clients can be allowed to start serving out of the partially filled cache object. The difference (Ed: I think?) is that Squid allows this as soon as it goes to origin, whereas ATS can not do it until we get the complete response header. The reason for this is that we make no distinction between cache refresh, and cold cache, so we have no way to know if a response is going to be cacheable, and therefore allow read-while-write functionality.
The configurations necessary to enable this in ATS are:
CONFIG proxy.config.cache.enable_read_while_writer INT 1 CONFIG proxy.config.http.background_fill_active_timeout INT 0 CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000 CONFIG proxy.config.cache.max_doc_size INT 0 |
All four configurations are required, for the following reasons:
Once all this enabled, you have something that is very close, but not quite the same, as Squid's Collapsed Forwarding.
The third configuration option, which is really only suitable when you know everything is cacheable, are related how to deal with locking out cache access until the origin response has come back. Combined with read-while-write this would completely mimic the behavior of Squid's Collapsed Forwarding. The downside is that non-cacheable content would also be serialized, which can severely affect latency. Since ATS now supports setting these settings per-request or remap rule, you can configure this to be suitable for your setup much more easily.
The configurations are (with defaults):
CONFIG proxy.config.http.cache.max_open_read_retries INT -1 CONFIG proxy.config.http.cache.open_read_retry_time INT 10 |
The default means that the feature is disabled, and every connection is allowed to go to origin instantly. When enabled, you will try max_open_read_retries times, each with a open_read_retry_time timeout. Lets say you set these to 5 and 10 respectively, connections will wait up to 50ms for a response to come back from origin from a previous request, until this request is allowed through.
I don't know if Squid has similar features, it seems to have a 30,000ms timeout, which is not tunable. It also seems incredibly high.
There is a plugin that implements Stale While Revalidate, which is documented in RFC 5861. With this plugin, you are allowed to serve objects stale in cache, while one request goes to origin to fetch the new version. Combined with read-while-write, and the fuzzy logic feature, this is a good alternative to the Open-Retry feature. The downside is that the current implementation is not complete, and needs changes to the core / cache to solve the problem completely.
Each of these features above could benefit from some improvements. Here are some ideas, but please feel free to add more thoughts and ideas here.
Allow for more precise configurations, other than using the overridable configurations. An example would be to for example allow cache.config to specify different fuzzy parameters based on matching regular expressions.
We could perhaps allow read-while-write to allow immediate collapsed forwarding, IF a cached objects was going stale. It's reasonable to think that the refreshed version will also be cacheable, and it would be pretty safe to do this. We would of course have to deal the case where the response comes back as non-cacheable, then any additional collapsed connections must dislodge from the first connection, and initiate its own origin session.
Implementing this would turn our existing read-while-write to be identical to Squid v2.6.
We could add a hash table keeping track of what origin sessions are outstanding, and allow the open-retry functionality only kick in when there is a session pending. This is a similar improvement to allowing read-while-write collapse immediately, except it's probably easier to implement. With this hash table, we'd want to assure that only objects that were previously cached, and are now stale, are allowed to collapse and start the open-retry cycles.
Right now, the cached object is rendered inaccessible as soon as we initiate the first request to origin to revalidate the object. If we could fix the cache such that the old object is still the entry in the directory structure, until the response comes back, read-while-write would work almost perfectly fine. As an optimization, it would be useful to allow serving the old version until the old one is completely filled; this makes large downloads faster and more efficient.