Pizza Detection using IoT CAM device and cloud service.

Background

The food production industry is a critical sector that caters to the growing demand for ready-to-eat meals and snacks. Large-scale food production facilities, such as vending machine operations, require efficient processes to ensure the quality and safety of the food products being dispensed to consumers. Traditional methods of monitoring and detecting issues in food products often rely on manual inspections, which can be time-consuming, prone to human error, and limited in terms of real-time analysis.

To address these challenges, an IoT-based food detection project was initiated to revolutionize the operations of a large-scale food production facility. The project aimed to leverage emerging technologies such as IoT sensors, data analytics, and machine learning algorithms to provide real-time monitoring and detection of food products within the facility's vending machines.

Problem Statement

Pizza vending machines rely on technology and automation to make the process efficient. However, the involvement of humans in the operation has introduced errors, causing a mismatch between the recorded number of pizzas by vending machine and the actual sales reflected in reports. To address this challenge, it's essential to establish a continuous monitoring system. This system will serve as a vigilant watchtower, promptly identifying any discrepancies between pizza counts and sales figures. Doing so, will not only rectify errors rooting from human actions but also detect any wrongful practices in the vending process. This vigilant monitoring will ultimately ensure that the sales reports remain accurate and trustworthy, benefiting both the business and its customers.

Solution

The solution involves utilizing an IoT CAM device and cloud service to enhance the monitoring of the pizza production process on the tray. By employing the IoT device, real-time video footage of the production flow is captured, enabling a detailed view of the entire process. This camera device serves as an efficient tool for gathering visual data.

The IoT CAM device and Cloud service create a powerful combination for enhancing insights. The camera captures production tray data, which is then processed and analyzed in the cloud. Machine learning models extract valuable information, offering a comprehensive understanding of production flow. This leads to improved decision-making and process optimization. By effectively monitoring and analyzing the pizza production process, end users gain enhanced efficiency and quality control.

Technology and device used

ESP-32 CAM

ESP32-CAM is a small-sized microcontroller board with a built-in camera module. The board is based on the ESP32 microcontroller and supports Wi-Fi and Bluetooth connectivity. The ESP32-CAM board is a popular choice for projects that require a small-sized, low-cost, and low-power device for capturing and processing images.

AWS

AWS (Amazon Web Services) offers a wide range of cloud-based services and solutions that cater to various computing, storage, networking, database, analytics, machine learning, and IoT needs.

Logical flow

• Placement of ESP32: The first step is to install an ESP32 device in each section of the pizza vending machine. ESP32 is a microcontroller that will be responsible for capturing the camera feed.

• Transmitting frames to AWS: The ESP32 devices will send the captured frames from the camera feed to the AWS platform. AWS provides a secure and scalable infrastructure for processing and storing data.

• Pizza detection with ML model: In AWS, we have deployed a pizza detection machine learning model. This ML model is specifically trained to analyze image frames received from ESP32 devices and identify the presence of pizza.

• Metadata storage: The metadata generated by the ML model, such as the identification and characteristics of the pizzas detected, will be stored in a structured manner. This metadata includes information about quantities, timestamps, and any other relevant data.

• Monitoring and analysis: By storing the metadata, we can effectively monitor and analyze the entire pizza production process. This data can provide insights into the performance of the vending machine, identify patterns or trends, optimize inventory management, and improve overall operational efficiency.

Technical workflow

Process of connecting ESP32-CAM with AWS IoT core

Step 1:

Create a new AWS IoT Core instance and register a device by creating a "thing", which represents the physical or virtual device connecting to AWS IoT Core.

AWS IoT Core is a managed cloud service provided by AWS that enables secure and reliable communication between IoT devices and AWS services.

Step 2:

Create an X.509 certificate and private key for your device to establish a secure connection with AWS IoT Core. Use the certificates to configure the device for secure communication. Once connected, the device can publish and receive messages using topics as communication channels with AWS services.

Flashing is the process of loading firmware onto a microcontroller's memory. For the esp32cam microcontroller, use the Arduino IDE and a USB cable to connect it to your computer. The IDE has a bootloader that enables uploading compiled firmware (.hex file) to the board via USB.

Step 3:

With Arduino Programming, we can manipulate the IoT device by flashing our code onto it. The code must include the MQTT topic name, certificate, private keys, and thing name provided by AWS IoT Core. This establishes a connection between the device and AWS IoT Core.

After flashing, the ESP32 CAM device will use the IoT MQTT protocol to publish image feed data in JSON format. The image frames will be in base64 format with the necessary metadata for the project.

Step 4:

Create an AWS IoT Core rule to forward all data from the MQTT topic to a Lambda function. When data is received by AWS IoT Core, it triggers the assigned Lambda function to handle the events.

AWS IoT Core allows you to define rules to process incoming messages and trigger actions based on specific conditions. For instance, you can use AWS Lambda to analyze device data or store it in an Amazon S3 bucket.

AWS Lambda is a serverless computing service from AWS. It runs your code without server management, scaling automatically to handle requests. Lambda functions can be triggered by events like S3 bucket changes or AWS IoT rules.

AWS Elastic Container Registry is a fully managed container image registry service by AWS. It securely stores and manages container images like Docker images for your applications.

Step 5:

Create a containerized Lambda function that incorporates an ML model trained to detect pizzas. The logic implemented in the Lambda function involves the following steps:

• The ML model is utilized to detect whether a pizza is present within the frame or not.

• If the ML model detects a pizza successfully, the Lambda function stores the image in an S3 bucket. Amazon S3 is a fully managed object storage service by AWS, enabling easy storage and retrieval of files, images, videos, and other data through a web interface or API calls.

• All metadata related to the detected pizza, including timestamps and relevant information, is stored in DynamoDB for efficient querying and data management. DynamoDB is a fully managed NoSQL database service by AWS, offering high scalability and reliability for storing and retrieving data.

Step 6:

Create a repository in the AWS ECR registry and pull the docker image that was previously created in the preceding step.

Step 7:

Create a Lambda function and retrieve the docker image from the AWS ECR Registry. Once this process is complete, the entire flow will function smoothly.

Detected Sample Images