How to talk to your customers about Drupal Security updates

With the recent release of versions 7.2 and 6.22, a significant Drupal security flaw in 6.x has been identified and fixed. While I feel strongly this is illustrates the value of Drupal and Open Source, it can be a significant challenge to talk to your customers about this.

Here's the email that we drafted up and shared with our customers (please feel free to use it, rewrite and share if it proves useful):

Dear [Customer],

There was a significant security flaw identified in the version of Drupal your site is running that was fixed in a security patch that was released released on Drupal.org May 25. We're currently recommending implementing this ASAP patch to avoid any issues.

[Here we vary depending on whether our customer has a service agreement with us or not.]

A. Since you currently have an Extended Service Agreement with us, we're recommending scheduling the fix as part of our monthly allotment of hours.

OR

B. Since this represents a significant danger to the data on your site and machines within our hosting environment we are considering this update to be mandatory. Please let us know if you will be able to schedule a software update within the next few weeks yourself, or we can implement the patch on a time and materials basis.

We're currently estimating this task as a 1 hour line-item billed at your normal hourly rate, however should complications arise it's possible that it could take more time. There should be no downtime associated with the patch, but you may wish you review the site for possible issues/changes. If you need us to address any issues, they will be addressed on a T&M basis.

I feel strongly that this update should be viewed as a showcases the value of Drupal and Open Source projects. If your site were not built using Drupal, it's likely that this issue would have gone undetected and could have resulted in significant financial cost. The recent high profile Sony Playstation Network security breach being a potent example of what can go wrong.

Thank you for your understanding and continuing business! Please feel free to contact me should you have any questions.

Here's a link to the official announcement:
http://drupal.org/drupal-7.2

I'd love to hear how you deal with these potentially tough conversations, and what you've learned from them.

Comments

Great post there. Its always a tricky one, to get clients to understand why updates are so important, especially security updates. The way you explained it to your clients were great. I will be sending out a similar message to my clients today.

Really this security update is no different from the many that happen in contrib every year. While it may be exploitable for a given site (if the site has error-reporting-to-screen enabled - which goes against best practices), any of the many security updates to contrib may be exploitable for a given site as well depending on the site's configuration and depending on who the site defines as "untrusted" users.

We offer a maintenance plan (separate or included with a monthly retainer) to all our hosted clients that includes all security updates. We simply run them all (with the client doing q/a before deployment if it's more than a simple upgrade) rather than attempt to sort through whether or not a specific security issue applies to a given site at this time (which could change tomorrow if the site configuration changes).

Our hosted clients that are not on a maintenance plan, a retainer, or development project know that they are responsible for their own application-level (read Drupal) security. Though if they do encounter issues there are backups that can be restored. I do not consider this update mandatory for these clients (if by mandatory you mean "if you don't address this we'll shut down your site"). We can not be responsible for our clients shooting themselves in the foot. We do however make very certain that a client cannot shoot another client in the foot.

Hi,

We have included periodic updates in our hosting costs. So all sites that are on our servers will be updated periodically.

So I do not have to explain it after an update came out, but I try to sell it upfront when the client wants to take hosting from us.

We have some clients that do not care about it, but then we still place them on our own server and they will get updated.

We have outsourced the updating to our hosting company, so they plan, upgrade and check when new versions are applied. After that I do a double check. We can do a rollback if needed if things do not go smooth.

But it is hard to sell qualitative hosting, the clients are always comparing to a couple dollar hosting a month plans. Which is apples & pears.

We hammer on security, the smaller clients are the hardest to convince. Bigger clients love it and accept it much more easy.

"I feel strongly that this update should be viewed as a showcases the value of Drupal and Open Source projects."

s/should be viewed as a //, perhaps?

Just a note, if you have your error settings set to 'write to the log', which is the recommended setting for production sites, then the Drupal 6 vulnerability is mitigated. So most production sites should not be affected by it - especially not in the sense of there being a publicly available exploit usable against every single Drupal 6 site. So while it's important to upgrade, and doesn't take much longer to do so than checking your error reporting settings, not all sites would be vulnerable.

I can't remember the last time there was a core security release for an XSS or SQL injection issue that didn't have mitigating factors like this. They should be taken seriously, but if assessing Drupal core security overall, it's important to read the fine print - many SAs only affect a handful of sites in practice, although the vulnerability is still critical if you're one of the affected sites.

This isn't an argument against telling people to upgrade regardless - that's why we have the SAs in the first place, it's more about using SAs to compare Drupal security to other open source projects (and closed source software, and services etc.) - often it is not apples to apples.

I agree the error reporting XSS is easily mitigated. For better or worse though, on-screen errors is the default in Drupal, and I have observed many live sites with this enabled. Anecdotally I'd even venture to say it's the majority. We're pretty vigilant about turning this off (it's on the deployment checklist), but even the best of us forget sometimes. And you can't completely dismiss potential attacks on dev or staging sites.

Because it affects the default install, 6.22 is by far the highest-impact update in recent memory.

Joaquin, great job with the format of that letter to clients. It introduces the reason to update Drupal in a solid way while highlighting the benefits of open source architectures!

Best,

Stephan

Nice post.
I have recently thought about what would be a cost-effective and still safe way of doing core and module upgrades, how much I should estimate/charge for that, and how long a site should be left in an insecure state. I wonder, do you repeat this procedure with 1h+ estimated cost for every single core and contrib upgrade, or do you wait for a few of them to accumulate?

I also wonder, the "roll back to the latest backup" is a good solution for obvious wsod and data explosion, but it does not help a lot for issues that are only detected days after the upgrade. Rolling back would destroy all content produced on that day. So far there was never a situation where I had to roll back something.

Also, how do you deal with locally modified core and modules? git merge?

Thanks

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>, <cpp>, <java>, <php>. The supported tag styles are: <foo>, [foo].
  • Web page addresses and email addresses turn into links automatically.
  • Lines and paragraphs break automatically.

About the Author

Joaquin Lippincott, CEO

Joaquin is a 20+ year technology veteran helping to lead businesses in the move to the Cloud. He frequently speaks on panels about the future of tech ranging from IoT and Machine Learning to the latest innovation in the entertainment industry.  He has helped to modernize software for industry leaders like Sony, Daimler, Intel, the Golden Globes, Siemens Wind Power, ABC, NBC, DC Comics, Warner Brothers & the Linux Foundation.

As the CEO and Founder of Metal Toad, an AWS Advanced Consulting Partner, his primary job is to "get the right people in the room".  This one responsibility is cross-functional and includes both external business development functions as well as internal delegation and leadership development.

A UCLA alumni, he also serves in the community as a Board Member for the Los Angeles Area Chamber of Commerce, the Beverly Hills Chamber of Commerce, and Stand for Children Oregon - a public education political advocacy group. As an outspoken advocate for entry-level job creation in tech he helped found the non-profit, P4TH, an organization dedicated to increasing the number of entry-level jobs in the tech industry, and is in the process of organizing an Advisory Board for the Bixel Exchange, a Los Angeles non-profit that provides almost 200 tech internships every year.

 

Ready for transformation?