cool tech graphics

Metal Toad Managed Services Dashboard - Part 2: Tools for Logging

Filed under:

In Part 1 we saw that the layout showed a glimpse of what is behind the scenes of our new managed services dashboard, the graphs were all generated with the help of a utility called Graphite. Graphite can't do the job on its own though, it takes a whole host of other programs to get the data from the various servers to Graphite in a form that it can read.

  1. Logstash: The first program in the chain is Logstash. Logstash is the program responsible for reading data from the various logs, changing the formatting, and then sending it to our log aggregation server. The best part is that logstash is almost endlessly versatile. Logstash can read data in several different formats, manipulate it, and then output data in other formats. Currently, we read custom logs for system stats like server load, memory, and Apache logs in json format. Logstash receives this data and sends it to the log aggregation server to be temporarily stored in Redis.
  2. Redis: Redis in itself, is a powerful tool used for storing key-value pairs. We use Redis to take the data stream from multiple sources and combine them into a single data stream.
  3. Logstash: After Redis has created the single data stream, Logstash sees the data a second time, this time it only deals with data on the log aggregation server, it takes output from Redis and outputs it to StatsD and Elasticsearch.
  4. Statsd: StatsD, a program developed by Etsy, is the data engine that allows us to send data to Graphite.
  5. Graphite: Finally, we end with Graphite. Graphite allows us to give a visual representation to our single stream data from StasD. With a few clicks of the mouse, you can perform mathematic formulas on the data. For example, the dashboard graphs bandwidth from a single server, the total errors from all servers, and the average resource usage across multiple servers.
Date posted: January 2, 2014

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • You can enable syntax highlighting of source code with the following tags: <code>, <blockcode>, <cpp>, <java>, <php>. The supported tag styles are: <foo>, [foo].
  • Web page addresses and email addresses turn into links automatically.
  • Lines and paragraphs break automatically.

Metal Toad is an Advanced AWS Consulting Partner. Learn more about our AWS Managed Services

About the Author

Nathan Wilkerson, Director of Engineering

Nathan started building computers, programming and networking with a home IPX network at age 13. Since then he has had a love of all things computer; working in programming, system administration, devops, and Cloud Computing. Over the years he's enriched his knowledge of computers with hands on experience and earning his AWS Certified Solutions Architect – Professional.

Recently, Nathan has transitioned to a Cloud Operations Manager role. He helps clients and internal teams interface with the Cloud Team using the best practices of Kanban to ensure a speedy response and resolution to tickets.

Have questions?