Robotiquette Joaquin Lippincott Thu, 05/26/2016 - 17:26

I'm currently facing a dilemma when it comes to robots.

A long time ago in our business, we decided to not do a phone tree or an automated email response to anyone who decided to contact us.  One of our core values is Respect and another is a commitment to Great Experiences. Simply put, automated phone trees and email create terrible, frustrating experiences that we believe are disrespectful.  In other words they are a no go.  Fast forward more than a decade, and now the robots are coming to us.

Exclude crawlers server-wide with X-Robots-Tag

For a staging site, it's important to exclude crawlers. You wouldn't want your content to get indexed at the wrong URL! The conventional wisdom is to use HTTP Basic authentication. There are some disadvantages to this approach however, and I've found I prefer using a new HTTP header called X-Robots-Tag. Note that this assumes your only objective is to prevent indexing by benevolent crawlers.

Ready to get started?