How Drupal's cron is killing you in your sleep + a simple cache warmer

But cach_clear_all does a check. If NULL is passed as the $cid then it does a check on some variables. It makes sure that cache_clear_all is only run on the tables once per "period" defined by the cache_lifetime settings.

Check your sites admin/settings/performance. Is the minimum cache lifetime set to "none"?

But even if it is set to none, it only deletes items which are expired...

Boost for 6.x is the answer!
It has a crawler built in so your site will never have an expired page. It can regenerate the cache before it expires. It solves all the issues you brought up. I can have a cache that lasts several weeks using Boost.

Please also get involved in getting finer grained cron controls into core
http://drupal.org/node/19173

Also consider setting your 'Minimum cache lifetime' setting in admin/settings/performance. That way if you run cron every hour it doesn't have to clear your caches every single run.

This won't solve everything, but it sure helps out alot.
Super Cron Module

Thanks for this post. I have my cron set to run every 5 minutes and it never occurred to me that there would be any harm in it. I'm going to investigate the options in the comments.

Thanks again!

Michelle

Thanks for the post Dylan, and to all who commented! This will definitely help, as a good portion of the sites we build are in the low-medium traffic range, and performance is always a concern.

Interesting stuff ... Mike @ Ixis pointed out this module in IRC:
http://drupal.org/project/elysia_cron

Would also help.

This is incorrect. This call is pretty innocous; it just prunes expired and unused items from the cache tables.

Whats called during system_cron() is cache_clear_all(NULL, $table). It is easy to assume that this wipes whole table. But not so - $cid is NULL in cache_clear_all() and thus empty($cid) is TRUE and thus only expired items get deleted. Follow the (twisted) logic of cache_clear_all and you will see it. The comments for this function are pretty confusiHere is the relevant section

function cache_clear_all($cid = NULL, $table = NULL, $wildcard = FALSE) {
  global $user;
 
  if (!isset($cid) && !isset($table)) {
    // Clear the block cache first, so stale data will
    // not end up in the page cache.
    cache_clear_all(NULL, 'cache_block');
    cache_clear_all(NULL, 'cache_page');
    return;
  }
 
  if (empty($cid)) {
    if (variable_get('cache_lifetime', 0)) {
      // We store the time in the current user's $user->cache variable which
      // will be saved into the sessions table by sess_write(). We then
      // simulate that the cache was flushed for this user by not returning
      // cached data that was cached before the timestamp.
      $user->cache = time();
 
      $cache_flush = variable_get('cache_flush_'. $table, 0);
      if ($cache_flush == 0) {
        // This is the first request to clear the cache, start a timer.
        variable_set('cache_flush_'. $table, time());
      }
      else if (time() > ($cache_flush + variable_get('cache_lifetime', 0))) {
        // Clear the cache for everyone, cache_lifetime seconds have
        // passed since the first request to clear the cache.
        db_query("DELETE FROM {". $table ."} WHERE expire != %d AND expire < %d", CACHE_PERMANENT, time());
        variable_set('cache_flush_'. $table, 0);
      }
    }
    else {
      // No minimum cache lifetime, flush all temporary cache entries now.
      db_query("DELETE FROM {". $table ."} WHERE expire != %d AND expire < %d", CACHE_PERMANENT, time());
    }
  }
 
<php>

Mikeytown2 - is there some cool video or tutorial that we can check out regarding how to best set up Boost to avoid all these issues?

While Boost can work wonders, I've ran into problems with it (on a D5 site) where the caching files were not clearing correctly, so buyer beware.

Memcache is definitely the way to go when possible, along with opcode caching. Beyond that I'm hoping to get a custom caching engine a co-worker built released to d.o to see what the feedback is..

In Drupal 7, the cron run has 2 processes to it: One is for "main tasks" (like the old cron hook), and the other is to run any cron entries queued up in the job queue. The latter you probably want to run fairly quickly, the former not so much.

So what if for Drupal 8 we split cron into 2 requests: One that can run on poormanscron or a separate cron run and handles the queued tasks, and has no cache clear. (That would be things like sending out mass emails, aggregator updates, etc.) The second could be set to run every 6-24 hours instead of every hour like most sites do, and handle things like log rotation, update_status updates, and other non-time-sensitive stuff, and then clear the cache.

That would keep high-volume tasks from piling up while keeping the wipe part of cron rare.

My super low-tech solution is to write a shell script to call specific cron hooks via drush, and then schedule that in the local Linux crontab at different times.

I posted basic instructions here:
http://drupal.org/node/606018#comment-2492668

Thanks for this post. I opened a d.o. issue for it: http://drupal.org/node/891600.

Thanks for sharing the info. I was wondering what module / software did you use to generate those cache hit graphs?