How to talk to your customers about Drupal Security updates
With the recent release of versions 7.2 and 6.22, a significant Drupal security flaw in 6.x has been identified and fixed. While I feel strongly this is illustrates the value of Drupal and Open Source, it can be a significant challenge to talk to your customers about this.
Here's the email that we drafted up and shared with our customers (please feel free to use it, rewrite and share if it proves useful):
Dear [Customer],
There was a significant security flaw identified in the version of Drupal your site is running that was fixed in a security patch that was released released on Drupal.org May 25. We're currently recommending implementing this ASAP patch to avoid any issues.
[Here we vary depending on whether our customer has a service agreement with us or not.]
A. Since you currently have an Extended Service Agreement with us, we're recommending scheduling the fix as part of our monthly allotment of hours.
OR
B. Since this represents a significant danger to the data on your site and machines within our hosting environment we are considering this update to be mandatory. Please let us know if you will be able to schedule a software update within the next few weeks yourself, or we can implement the patch on a time and materials basis.
We're currently estimating this task as a 1 hour line-item billed at your normal hourly rate, however should complications arise it's possible that it could take more time. There should be no downtime associated with the patch, but you may wish you review the site for possible issues/changes. If you need us to address any issues, they will be addressed on a T&M basis.
I feel strongly that this update should be viewed as a showcases the value of Drupal and Open Source projects. If your site were not built using Drupal, it's likely that this issue would have gone undetected and could have resulted in significant financial cost. The recent high profile Sony Playstation Network security breach being a potent example of what can go wrong.
Thank you for your understanding and continuing business! Please feel free to contact me should you have any questions.
Here's a link to the official announcement:
http://drupal.org/drupal-7.2
I'd love to hear how you deal with these potentially tough conversations, and what you've learned from them.
Comments
Great post there. Its always a tricky one, to get clients to understand why updates are so important, especially security updates. The way you explained it to your clients were great. I will be sending out a similar message to my clients today.
Thu, 05/26/2011 - 05:31
Really this security update is no different from the many that happen in contrib every year. While it may be exploitable for a given site (if the site has error-reporting-to-screen enabled - which goes against best practices), any of the many security updates to contrib may be exploitable for a given site as well depending on the site's configuration and depending on who the site defines as "untrusted" users.
We offer a maintenance plan (separate or included with a monthly retainer) to all our hosted clients that includes all security updates. We simply run them all (with the client doing q/a before deployment if it's more than a simple upgrade) rather than attempt to sort through whether or not a specific security issue applies to a given site at this time (which could change tomorrow if the site configuration changes).
Our hosted clients that are not on a maintenance plan, a retainer, or development project know that they are responsible for their own application-level (read Drupal) security. Though if they do encounter issues there are backups that can be restored. I do not consider this update mandatory for these clients (if by mandatory you mean "if you don't address this we'll shut down your site"). We can not be responsible for our clients shooting themselves in the foot. We do however make very certain that a client cannot shoot another client in the foot.
Thu, 05/26/2011 - 06:54
Hi,
We have included periodic updates in our hosting costs. So all sites that are on our servers will be updated periodically.
So I do not have to explain it after an update came out, but I try to sell it upfront when the client wants to take hosting from us.
We have some clients that do not care about it, but then we still place them on our own server and they will get updated.
We have outsourced the updating to our hosting company, so they plan, upgrade and check when new versions are applied. After that I do a double check. We can do a rollback if needed if things do not go smooth.
But it is hard to sell qualitative hosting, the clients are always comparing to a couple dollar hosting a month plans. Which is apples & pears.
We hammer on security, the smaller clients are the hardest to convince. Bigger clients love it and accept it much more easy.
Thu, 05/26/2011 - 07:27
We don't need that talk as Drupal patches are automatically applied as they are included in the hosting plan :-)
Thu, 05/26/2011 - 10:13
"I feel strongly that this update should be viewed as a showcases the value of Drupal and Open Source projects."
s/should be viewed as a //, perhaps?
Fri, 05/27/2011 - 15:32
Ha, I noticed that too. :)
Thu, 05/26/2011 - 12:14
Just a note, if you have your error settings set to 'write to the log', which is the recommended setting for production sites, then the Drupal 6 vulnerability is mitigated. So most production sites should not be affected by it - especially not in the sense of there being a publicly available exploit usable against every single Drupal 6 site. So while it's important to upgrade, and doesn't take much longer to do so than checking your error reporting settings, not all sites would be vulnerable.
I can't remember the last time there was a core security release for an XSS or SQL injection issue that didn't have mitigating factors like this. They should be taken seriously, but if assessing Drupal core security overall, it's important to read the fine print - many SAs only affect a handful of sites in practice, although the vulnerability is still critical if you're one of the affected sites.
This isn't an argument against telling people to upgrade regardless - that's why we have the SAs in the first place, it's more about using SAs to compare Drupal security to other open source projects (and closed source software, and services etc.) - often it is not apples to apples.
Fri, 05/27/2011 - 23:59
I agree the error reporting XSS is easily mitigated. For better or worse though, on-screen errors is the default in Drupal, and I have observed many live sites with this enabled. Anecdotally I'd even venture to say it's the majority. We're pretty vigilant about turning this off (it's on the deployment checklist), but even the best of us forget sometimes. And you can't completely dismiss potential attacks on dev or staging sites.
Because it affects the default install, 6.22 is by far the highest-impact update in recent memory.
Thu, 05/26/2011 - 17:20
Joaquin, great job with the format of that letter to clients. It introduces the reason to update Drupal in a solid way while highlighting the benefits of open source architectures!
Best,
Stephan
Tue, 09/27/2011 - 00:58
Thanks! :-)
Fri, 06/03/2011 - 17:57
Nice post.
I have recently thought about what would be a cost-effective and still safe way of doing core and module upgrades, how much I should estimate/charge for that, and how long a site should be left in an insecure state. I wonder, do you repeat this procedure with 1h+ estimated cost for every single core and contrib upgrade, or do you wait for a few of them to accumulate?
I also wonder, the "roll back to the latest backup" is a good solution for obvious wsod and data explosion, but it does not help a lot for issues that are only detected days after the upgrade. Rolling back would destroy all content produced on that day. So far there was never a situation where I had to roll back something.
Also, how do you deal with locally modified core and modules? git merge?
Thanks
Thu, 05/26/2011 - 03:16