Temetra early on saw the potential of the Internet for collecting and analyzing meter data from water companies, allowing trouble spots such as leaks to be identified. Director Paul Barry’s foresight to see the niche opportunity within the emerging IoT environment has paid off. Temetra’s business has increased five-fold with this web-based Meter Data Management service.
Temetra’s modern Internet-based repository is designed for water, gas and heat meter utilities, storing a mixture of eye-ball, low-power radio AMR, and fixed network high-frequency meter index data, along with meter asset and network management data. Temetra’s mobile data collection, for Windows Mobile and Android, now ingests, stores and analyzes pressure and flow data from 10 million water meters.
Paul very kindly sat down with us to discuss how Riak KV is involved in helping Temetra meet their customer requirements and grow their business. Paul is the first to tell you that there’s no shortcut. “NoSQL has many benefits, but it also requires some new thinking if you’re used to non-distributed view of the world such as SQL. To get all the benefits of NoSQL, it’s necessary to design a distributed app, and deal with issues of availability and consistency in a controlled way.”
For background, Riak KV is now at the core of Temetra’s architecture and is used to manage all the high-frequency data, especially data that does not change frequently. It’s used to store meter index data, raw protocol data, audit trails, photos, and device events.
Paul, thanks for sitting down with us. Can you tell us how you came to Riak and how working with Riak and Riak KV has impacted your team?
We looked at MongoDB, Redis, Cassandra and Riak KV and ultimately we chose Riak’s Riak because I managed to get it up and running within an hour. We had an extended testing period of about two years, where Riak KV ended up being used in more and more places around our app. During this time we were using the open source version, but as we got more and more of our customers depending on it, we switched to the per-node support from Riak to provide expertise in the event of any problems.
To date, we’ve had no major incident, but I sleep a bit better knowing there’s 24/7 help available if we have an issue with our cluster that we can’t solve quickly ourselves.
We’re still a small team of developers, and I really like the low administrative cost of Riak KV. The burden of admin on a Riak cluster is also very low, aside from some basic health monitoring and occasional replacement / migration of nodes.
IoT is disrupting many traditional businesses as well as creating new business opportunities. Companies are struggling to either unearth these or leverage the technology available to help them become part of the evolution. What did Temetra do differently? How did NoSQL play a part?
We’re part of IoT, but servicing a particular narrow niche within that broad umbrella. Traditionally, the utility market we serve is very conservative in their technology choices and has strict on-premise setup requirements. The disruption we brought was switch to software as a service, allowing us to use the new technology as part of our service. Delivering our software as a service meant we could run a multi-tenant system and get customers on board very quickly. It also meant we could scale up easily. Behind the scenes we could change the tech but continue to deliver.
Many of our customers don’t really understand the difference in SQL/NoSQL, but they like how we offer regular updates, high-availability and high-speed analysis on their data. Our customers are after a boringly reliable service, which Riak KV supports.
What is the volume of data you are managing and how fast it is growing? How do you see Riak coping with this?
We’re adding more customers all the time – we’ve just added another with 1.2 million endpoints. The utility industry trend is towards more and more high-frequency data (short interval time-series) and rich data (photos and other metadata), so we’re seeing growth in both customers, and the size of their data.
One advantage we have is clear visibility on arrival of new data, so we don’t have to worry about unexpected peak data volumes and scrambling to provision extra capacity at very short notice. We have the luxury of being able to test and provision extra capacity prior to arrival of additional data. We use several monitoring tools to manage our infrastructure, and test to make sure we always have plenty of headroom to cope with predicted loads.
Knowing what you know now what advice would you give companies reviewing NoSQL technologies to help them make their decision?
There’s no perfect NoSQL platform for all use cases, so you need to be prepared to remodel your data to suit the one you choose. For our high-frequency index data, we’re on our second major iteration of storage model. With each iteration, we learn a bit and improve how to store best for our retrieval patterns.
We’ve ended up with several storage subsystems, SQL, Riak, Solr and in-memory KV. In my experience, NoSQL does not solve all the problems in modern applications, but in our case is a vital part of our overall service.
Is it a balance that you need to strike being an enterprise between safe, existing technologies and the new potentially exciting open source products or are there large competitive advantages to be gained by businesses for those willing to take advantage of these as soon as possible?
For a small organization like Temetra, the only way to cost-effectively scale our product is using innovative technologies, and open source is where all the interesting work is happening. The most likely candidates for addition to Temetra are Solr, Hazelcast. Spark and Storm and are currently all under evaluation as additions as they are very interesting.
What’s the future for Temetra?
Like a lot of companies, we’re interested in pulling in more sensor data, but in our case it will likely be specific data types that have value to utilities, that we can aggregate and analyze alongside the meter data. We now have a distributed application platform that can manage high volume time-series data, but want to keep our focus on the data we understand, and can add value to, rather than become a general purpose IoT data store. We’re also interested in seeing how Riak TS can add value in the future as it’s approach to IoT data may be able to help save us time and development hours.
In the short-term it’s about continuing the scalability so we can add another five million meters.
For additional insight click here to read Paul’s article in Computer Weekly.
