• Created by Unknown User (zwoop), last updated on Oct 09, 2013 7 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 18 Next »

Squid v2.x (not 3.2 as far as I can tell?) has a feature they coined Collapsed Forwarding. In a nutshell, it allows the intermediary to piggy back multiple clients on a single origin server connections. There are a few ways we could accomplish this, but they all would share the same property: this is only safe to do for cacheable objects.

Problem space and issues

The problem to solve is a form of stampeding herd: When a cached object goes stale, instead of potentially letting 100's or 1000's of requests slip through to the origin, we want to minimize this as much as possible. In a best case scenario,  a redux to 1 origin connection could be achieved. One issue here is to know if a request is cacheable (and therefore shareable). On a complete cache miss (cold cache), this can be difficult at best.

ATS already has three (and a half) features that are in place to alleviate this problem, and we will discuss those three here as well. The discussion should also include options of improving upon these existing features, such that we would not need more complexity.

Fuzzy logic

ATS has three configuration options related to pre-fetching objects before they go stale in cache:

CONFIG proxy.config.http.cache.fuzz.time INT 240
CONFIG proxy.config.http.cache.fuzz.min_time INT 0
CONFIG proxy.config.http.cache.fuzz.probability FLOAT 0.005

The intent here is that there's a small chance (fuzz.probability == 0.5%) that a request for an object will be revalidated, starting 240 seconds before it goes stale. For objects getting few requests per second, this would likely not trigger, but then this feature is not necessary anyways since odds are only 1 or a small number of connections would hit origin upon objects going stale. The defaults are a good compromise, for objects getting roughly 4 requests / second or more, it's virtually guaranteed to trigger a revalidate event within the 240s. These configs are also overridable per remap rule or via a plugin, so can be adjusted per request if necessary.

If the revalidate occurs, the object will be immediately evicted from cache while it is refetched from the origin.

Finally, the fuzz.min_time is there to be able to handle requests with a TTL less than fuzz.time – it allows for different times to evaluate the probability of revalidation for small TTLs and big TTLs. Objects with small TTLs will start "rolling the revalidation dice" near the fuzz.min_time, while objects with large TTLs would start at fuzz.time. A logarithmic like function between fuzz.min_time and fuzz.time determine the revalidation evaluation start time. As the object gets closer to expiring, the window start becomes more likely. By default this setting is not enabled, but should be enabled anytime you have objects with small TTLs. Note that this option predates overridable configurations, so you can achieve something similar with a plugin or remap.config conf_remap.so configs.

These configurations are similar to Squid's refresh_stale_hit configuration option.

Read While Write

This feature is pretty similar to how Squid actually implements Collapsed Forwarding. When ATS goes to fetch something from origin, and upon receiving the response, any number of clients can be allowed to start serving out of the partially filled cache object. Note that for read-while-writer, you should allow background fill to always kick in (set the threshold to zero). The difference (Ed: I think?) is that Squid allows this as soon as it goes to origin, whereas ATS can not do it until we get the complete response header. The reason for this is that we make no distinction between cache refresh, and cold cache, so we have no way to know if a response is going to be cacheable, and therefore allow read-while-writer functionality.

The configurations necessary to enable this in ATS are:

CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.000000
CONFIG proxy.config.cache.max_doc_size INT 0

All four configurations are required, for the following reasons:

  1. enable_read_while_writer turns the feature on. It's off by default (as of 4.0.x, we might want to modify this for 5.x)
  2. The background fill feature should be allowed to kick in for every possible request. This is necessary, in case the writer ("first client session") goes away, someone needs to take over the session. Hence, you should set the background fill timeouts and threshold to zero; this assures they never times out and always is allowed to kick in.
  3. The proxy.config.cache.max_doc_size must be unlimited (set to 0), since we potentially don't know the size of the object, we can't allow it to disconnect due to going over this limit.

Once all this enabled, you have something that is very close, but not quite the same, as Squid's Collapsed Forwarding.

Open Read Retry Timeout

The open read retry configurations attempt to reduce the number of concurrent requests to the origin for a given object. While an object is being fetched from the origin server, subsequent requests would wait open_read_retry_time milliseconds before checking if the object can be served from cache. If the object is still being fetched, the subsequent requests will retry max_open_read_retries times. Thus, subsequent requests may wait a total of (max_open_read_retries x open_read_retry_time) milliseconds before establishing an origin connection of its own. For instance, if they are set to 5 and 10 respectively, connections will wait up to 50ms for a response to come back from origin from a previous request, until this request is allowed through.

These settings are inappropriate when objects are uncacheable. In those cases, requests for an object effectively become serialized. The subsequent requests would await at least open_read_retry_time milliseconds before being proxies to the origin.

Similarly, this setting should be used in conjunction with Read While Write for big (those that take longer than (max_open_read_retries x open_read_retry_time) milliseconds to transfer) cacheable objects. Without the read-while-writer settings enabled, while the initial fetch is ongoing, not only would subsequent requests be delayed by the maximum time, but also, those requests would result in another request to the origin server.

Since ATS now supports setting these settings per-request or remap rule, you can configure this to be suitable for your setup much more easily.

The configurations are (with defaults):

CONFIG proxy.config.http.cache.max_open_read_retries INT -1 
CONFIG proxy.config.http.cache.open_read_retry_time INT 10

The default means that the feature is disabled, and every connection is allowed to go to origin instantly. When enabled, you will try max_open_read_retries times, each with a open_read_retry_time timeout.

I don't know if Squid has similar features, it seems to have a 30,000ms timeout, which is not tunable. It also seems incredibly high.

Stale While Revalidate plugin

There is a plugin that implements the Stale While Revalidate cache-control directive, which is documented in RFC 5861. This directive allows caches to serve a stale object while a background revalidation is occurring. When a popular object becomes stale, subsequent requests would be proxied to the origin for revalidation. With this plugin (and the appropriate Cache-control: directive), you are allowed to serve objects stale in cache, while one request goes to origin to fetch the new version. Combined with read-while-writer, and the fuzzy logic feature, this is a good alternative to the Open-Retry feature. The downside is that the current implementation is not complete, and needs changes to the core / cache to solve the problem completely.

Improvements

Each of these features above could benefit from some improvements. Here are some ideas, but please feel free to add more thoughts and ideas here.

Improve Fuzzy Logic with finer granularity

Allow for more precise configurations, other than using the overridable configurations. An example would be to for example allow cache.config to specify different fuzzy parameters based on matching regular expressions.

Allow for a true background fill. If the revalidation hits, ideally serve that (and future) requests from cache. Once the revalidation is complete, begin serving the new object.
Or, perhaps the request that triggered the revalidation would be proxied, while subsequent requests are still served from the previously cached object.
And/Or, submit an IMS request initially for the object. It seems likely that the headers will give a new max-age.

Allow read-while-writer to collapse immediately

We could perhaps allow read-while-writer to allow immediate collapsed forwarding, IF a cached objects was going stale. It's reasonable to think that the refreshed version will also be cacheable, and it would be pretty safe to do this. We would of course have to deal the case where the response comes back as non-cacheable, then any additional collapsed connections must dislodge from the first connection, and initiate its own origin session.

Implementing this would turn our existing read-while-writer to be identical to Squid v2.6.

Add intelligence to Open-Retry timeout

We could add a hash table keeping track of what origin sessions are outstanding, and allow the open-retry functionality only kick in when there is a session pending. This is a similar improvement to allowing read-while-writer collapse immediately, except it's probably easier to implement. With this hash table, we'd want to assure that only objects that were previously cached, and are now stale, are allowed to collapse and start the open-retry cycles.

Improve the cache to handle RFC 5861 better

Right now, the cached object is rendered inaccessible as soon as we initiate the first request to origin to revalidate the object. If we could fix the cache such that the old object is still the entry in the directory structure, until the response comes back, read-while-writer would work almost perfectly fine. As an optimization, it would be useful to allow serving the old version until the old one is completely filled; this makes large downloads faster and more efficient.

RFC 5861 also specifies a stale-if-error cache-control directive. This is not possible with the current cache – once the object becomes stale, it is no longer available to be served.

Both would be helped by a true background fill as described in the improving fuzzy logic section.

  • No labels