Traditional “Data Lake” Approach May Not Be A Good Choice for IoT Data

The momentum around Apache Spark continues. Spark Summit East was a big success and Riak’s own Pavel Hardak was among the presenters at the recent conference held in Boston during one of this winter’s biggest US Northeastern storms.

Weather may be the original time series data. Conference attendees were talking about the storm and about IoT. Pavel’s presentation provided updates on the latest news on IoT data and provided some insights on the characteristics of data generated by IoT devices and how it differs from traditional enterprise or Big Data problems.

Pavel shows why the traditional “Data Lake” approach may not be a good choice for IoT data. He also discusses which architectural patterns are beneficial for IoT use cases and why some trusted methods eventually turn out to be “anti-patterns”.

In his presentation, Pavel shows how to combine best-of-breed open-source technologies, like Apache Spark, Riak and Mesos to build scalable IoT pipelines to ingest, store and analyze huge amounts of data, while keeping operational complexity and costs under control.