Quick Drupal Cacherouter and Boost benchmarks

In the discussion following my last post about cron and the cache hit rate, I promised to do some testing of the different cacherouter backends, as well as Boost. Again, these tests focus on the needs of a smallish site with 500 nodes and 1200 requests per day. Boost is the clear winner for response time (which shouldn't be a surprise given that it allows the web server to deliver HTML files directly from disk). What's interesting though is that the response times are all close enough that it doesn't really matter what caching backend you choose (An end user cannot perceive the difference between 6ms and 2ms, and throughput isn't an issue at this scale). The only factor that's really relevant is how good your system's cache expiration and regeneration logic is. I haven't yet had a chance to explore this aspect in detail, but it seems like Boost is the clear winner here as well.

cached response times

authenticated response times

JMeter is used to generate a more realistic traffic profile that ab:

histogram of node popularity

A few people have brought up Pressflow and Varnish. I'm not sure that those technologies help solve this particular problem: Pressflow naturally sets the Cache-Control: max-age to the admin's chosen value, which Varnish will then honor. Varnish has dropped the idea of a prefetch, in favor of a grace period during which stale pages can be served while a fresh copy is retrieved from the backend. If the page is never requested after max-age + grace period, then that cache entry will grow cold. (That's not a criticism of Varsish; at scale that's exactly the behavior you'd want. However on a small site I'm willing to burn a little extra CPU power to regenerate cache entries that may never be used.)

Comments

The thing that is slightly misleading about what you are saying is that typically Memcache backend would be used in a multi-server environment and APC / File would be used in a single server environment. Also when you are hitting the same pages over and over and don't have cron running and don't have people submitting data, the database server can cache all the SQL queries and you are really only hitting memory caches for all of the cache technologies.

The other thing about Memcache is that there are two modes "shared" which means I keep an index of what is stored in the cache and have to do additional reads and writes each time a cache entry is received. There is also "non-shared" mode (in the settings it would be 'shared' => FALSE). This mode allows you to setup one "bin" or table (e.g. cache, cache_page, cache_filter, etc) per each memcached instance or daemon. This offers superior performance over shared mode due to less index lookups.

HTH.

-steve

I look forward to your exploration of the cache expiry logic. Many (myself included) feel that cache_clear_all on node create/edit is too harsh and inefficient. It has its reasons - obviously it's the safest way to never serve stale content. But we pay a big price for it.

Great articles, keep them coming.

-Robert

There is definitely a lot left out here. Steve, thanks for the tip about 'shared' => FALSE, and I admit I skipped over the many other differences of each backend. I think you raise a good point; your choice in cache backends should include other factors beyond hits / sec or response time.

I didn't mean to mislead, only focus on the small site which implies a single server. It's not that big scalability isn't important or interesting (Metal Toad built a clustered environment for emmys.com), rather I perceived a need for information that's useful for small sites too.

I also didn't include XCache, or alternative servers like nginx (I don't have either installed at the moment). And certainly the numbers are impacted by the fact that DB and file-based backends both have intermediate caches (the MySQL query cache and O/S kernel file cache, respectively).

It would be fun to work up a JMeter test plan that includes updates to nodes and comments; if anyone is willing to share jmx files that would be awesome! Still there will be no universal benchmark, every site has different usage patterns.

Cool to see that Boost is on top!

Boost can sorta emulate varnishes grace period with the correct boost settings. This means that you will always serve a page from the cache.
Enable these settings in boost given the standard settings.

  • Expire content in DB, do not flush file
  • Enable the cron crawler
  • Do not flush expired content on cron run, instead recrawl and overwrite it

This means that all pages marked as expired will be crawled. So right before the crawler hits the page, boost deletes that cached entry so the next request (most likely from the crawler) will be uncached.

If this window of a couple of seconds for an uncached page is too large then you can enable "Overwrite the cached file if it already exits" and update your htaccess rules from the boost-rules page. If your running in this mode, the server will always bypass the boost cache, thus when it is crawling its self, the crawler never hits a cached page (this setting is based off of the servers IP address, something to lookout for if using this setting on a local box).

Without the crawler, varnishes "grace period" emulation is not possible; also because of this, your minimum cache lifetime is how often cron is ran. Be aware of how long it takes to crawl the site (expired only) when playing around with how often cron runs for the crawler. If you have comments being posted, unless you have a short cron interval, I recommend disabling "Expire content in DB, do not flush file". On very large sites with a lot of comment action in many nodes may not perform well because of the time it takes to crawl all of those pages. If this is the case you might want to disable "Expire content in DB, do not flush file"; by doing this it allows for new comments to appear ASAP, but you lose having a minimum cache lifetime (how often cron runs is the lifetime). Leaving "Do not flush expired content on cron run, instead recrawl and overwrite it" enabled then still allows for the varnish "grace period" just that it only works for pages that where expired by the maximum cache lifetime setting, not by user interaction (comments, node edits, ect...).

In short the possibilities are nearly endless because every site is different. The above is a semi detailed explanation of 4 of the settings in boost; as you know there are many more ;)

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>, <cpp>, <java>, <php>. The supported tag styles are: <foo>, [foo].
  • Web page addresses and email addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Ready for transformation?