Serverless Data Integration & Monitoring

Overview

Companies use the Internet of things (IoT) to enhance customer experience and improve the efficiency of their digital transformation strategies. With the increase in the number of IoT devices, device management has become crucial. It's also critical to transforming the growing body of IoT data into business intelligence that can be acted upon. In this blog, the ability to ingest data, process it, monitor it, and facilitate decision-making calls for an architecture consisting of decoupled, single-purpose event processing components that receive and analyze events asynchronously is discussed.

Challenges

The challenges in the IoT device space are widely common, regardless of the business domain. Following are the common key challenges this article addresses

• Connecting IoT devices to an external infrastructure via a gateway that ensures clean, error-free data is collected and streamed through a secure channel using the de facto MQTT protocol.

• Automating the day-to-day management of these devices, including monitoring their health diagnostics, intervening if necessary in case of failure, and updating the firmware as required to ensure proper maintenance.

• Accommodate high traffic by scaling up on-demand in response to events in the environment.

• Automate and centrally securely manage the devices while allowing fine-grained control and configuration of individual devices.

GCP for IoT Data Ingestion and Monitoring

Using Google Cloud Platform’s (GCP) serverless architecture, data ingestion from IoT devices to analytics can be built as an event-driven system that scales seamlessly. GCP provides an end-end solution to collect, analyze, and visualize IoT data in real-time.

The solution provides a function-based backend implementation for the serverless environment that triggers business logic in response to events. Data streams into Cloud Pub/Sub, which in turn triggers Cloud Functions that can monitor and alert on data.

Solution Architecture

In the above architecture, data flows through three main layers, each serving an independent purpose:

• IoT Data Ingestion (from IoT Devices to Pub/Sub)

• Transformation (from Pub/Sub to Cloud Function/Cloud Dataflow for Data processing)

• Monitoring and Alerting (logging in to Cloud Logging for real-time data monitoring and alerting)

Cloud IoT Core is a fully managed service. It acts as a central hub for globally distributed IoT devices to which devices can securely connect, stream messages, and manage their data through the MQTT protocol. Cloud IoT Core can be used to route device messages to Cloud PubSub for aggregation. Based on the event types in devices, messages can be routed to different topics in PubSub.

By integrating Cloud PubSub, a simple and reliable event-driven real-time messaging service, with IoT Core, high volumes of IoT messages can be handled asynchronously and made available for downstream applications to process.

Cloud events need to be handled asynchronously without having to manage servers or run-time environments. Google Cloud Functions, GCP’s event-driven computed service, can be leveraged here to significant effect.

As shown in the reference architecture above, Cloud Functions that subscribe to different topics viz., one for application logs and another for device logs, are deployed. One of the cloud functions extracts message payload and device information from a PubSub message from IoT Core and then generates a structured log entry for Cloud Logging.

In order to gain further insight from PubSub data, the data may be fed into data analytics systems. User-defined metrics can be configured into a specific Cloud PubSub topic which can be further used to set thresholds and raise alerts, if necessary. Device managers in cloud IoT Core are capable of configuring and managing an individual device securely based on metrics.

Conclusion

Serverless computation is ideal for applications with rapid time-to-market and unpredictable scale requirements - today’s enterprise innovation projects and startups in the IoT space usually fall into this category. By abstracting server management, Cloud Functions automatically scale on-demand to identify meaningful events as fast as possible and respond to them quickly. They scale reliably and perform under high inflow making them an essential component in data integration architectures.