We built an ETL (extract, transform, load) system that would resolve the discrepancies. After pulling data from the sensors and automatically testing it using Simpletest, the biggest challenge was transforming it. The four different data monitoring services (Power One, DECK, Tigo, and Locus) used a variety of units for the same measurements—kelvin versus centigrade, kilowatt hours versus joules. Our system automatically transforms all the heterogeneous information into a single common format, using data modeling to create a unified vision.
An interesting extra challenge came from the rate limits set by the data monitoring companies—each source had a strict limit on the rate at which data could be extracted. So we created a solution that pulls data in a carefully controlled, automated speed to keep the info flowing uninterrupted.
Once we’d extracted and transformed the data, we used Highcharts to load it into interactive vector-based charts. Students and teachers can easily manipulate and analyze the data as part of their STEM curriculum.
.svg.png){kind=logo}
