Our Journey into IOT has been always a learning experience. The changes our architecture had to go through to build a robust inhouse product has been nothing less than great. It all started with a project from our client to build a IOT application to gather and report on data.

We started out with a nodeJs application to handle the incoming HTTP request and a mysql database. All was well and in sometime our DB was not handling the huge number of inserts that was happening because the number of devices that we had to bring the data from, swelled from a few hundreds to a few thousands. So our search for a database ended with a close competition between MongoDb and Cassandra DB. We decided to go with cassandra given its inherent benefit with handling large amounts of data inserts and also because of its out-of-the-box fault tolerance capability. This change was not simple as as we had to migrate the data from MySql to cassandra. The migration was worth it . The application is able to handle lot many inserts form the IOT devices.

The data reports that we did first was getting out dated and client had more requirements to get more sophisticated reports. We initially thought cassandra would give us the speed to run large queries but it failed us when the client asked for more ad-hoc reports. Cassandra being a query based design DB we ended up creating a lot of tables for each report. We found that this was not a scalable solution as reports requirement kept on coming. So we looked for distributed computing technologies to save from the load and to query cassandra faster. We found apache spark. Armed with the spark cassandra connector we were able to bring power of distributed computing on top of cassandra. It was great to bring Apache spark into our ecosystem.

Some trivial requirement that came to us, was to show up real-time graphs. We went with socketio and data stream API for these. All looked good with what we did. The client now decided to add a new hardware vendor who had no HTTP endpoint support. They wanted support for MQTT. So we had to give support to lot of inputs like files, binary etc. We turned to Apache Nifi at this point. This made us to quickly support all inputs and we were able to enrich the data before sending to NodeJS backend. Great looks like we had solved the entire puzzle. Now we had nodeJS that was handling all the types of IOT devices. We did not have the ability to scale for different types of IOT devices. We had to create a endpoint to read from, depending on the type of the device. We had to introduce Kafka to split the incoming data based on types of devices and connect it to a particular NodeJs microservice.

All has been good till now. But we will wait to see what changes we will have to make in coming days when we hear more change request from the client.

We build sophisticated IOT applications and ML projects for clients in manufacturing and other verticals.