Detecting solar panel damage with Amazon Rekognition Custom Labels
AWS Architecture BlogDetecting solar panel damage with Amazon Rekognition Custom Labelsby Ramakant Joshi, Shekar Tippur, and Anand Iyer | on20 FEB 2023| inAmazon Machine Learning,Amazon Rekognition,Architecture,Artificial Intelligence,Serverless,Technical How-to|Permalink|ShareEnterprises perform quality control to ensure products meet production standards and avoid potential brand reputation damage. As the cost of sensors decreases and connectivity increases, industries adopt real-time imagery analysis to detect quality issues.At the same time, artificial intelligence (AI) advancements enable advanced automation, reduce overall cost and project time, and produce accurate defect detection results in manufacturing plants. As these technologies mature, AI-driven inspections are more common outside of the plant environment.Overview of solutionThis post describes our SOLVED (Solar Roving Eye Detector) project leveraging machine learning (ML) to identify damaged solar panels usingAmazon Rekognition Custom Labelsand alert operators to take corrective action.As solar adoption increases, so does the need to detect panel damage. Applying AWS-managed AI services is a simpler, more cost-effective approach than human solar panel inspection or custom-built production applications.Customers can capture and process videos from the field and build effective computer vision models without creating a dedicated data science team. This approach can be generalized for use cases across industries to detect defects in wind turbines, cell phone towers, automotive parts, and other field components.Amazon Rekognition Custom Labelsbuilds off of existing service capabilities already trained to identify the objects and scenes in millions of cross-category images. You upload a small set of training images—typically a few hundred or less—into our console. The solution automatically loads and inspects the training data, selects the right ML algorithms, trains a model, and provides model performance metrics. You can then integrate your custom model into your applications through the Amazon Rekognition Custom Labels API.WalkthroughThis post introduces the SOLVED project featured at there:Invent 2021Builders Fair. It will:Review the need for solar panel damage detectionDiscuss a cloud-based approach to ingest, store, process, analyze, and detect damaged solar panelsPresent a diagram streaming videos from aRaspberry Pi, storing them onAmazon Simple Storage Service (Amazon S3), processing them using an AWS video-on-demand solution, and inferring damage using Amazon RekognitionIntroduce a console to mimic an operation center for appropriate actionDemonstrate the integration ofAWS IoT Corewith aPhilips Huebulb for operator alertsPrerequisitesBefore getting started, review the following prerequisites for this solution:This blog assumes familiarity withAWS Lambda,Amazon Kinesis Video Streams,AWS IoT Core, andAWS Identity and Access Management (IAM)Access to an AWS account with permissions to create the resources described in the installation steps sectionAnAWS IAM userwith the permissions described later in this postAccess to aFreenove 4WD Smart CarAccess to a Raspberry PiAccess to aPhilips Hue Smart BulbThe SOLVED projectThe SOLVED project leverages ML to identify damaged solar panels using Amazon Rekognition Custom Labels. It involves four steps:Data ingestion:Live solar panel video ingested from moving rover into an Amazon S3 bucketPre-processing:Captured video split into thumbnail imagesProcessing and visualization:ML models making real-time inferences to identify defective panels with a dashboard to review images and prediction scoresAlerting:Defective panels result in notification sent through MQTT messages to light a smart bulbFigure 1 shows the SOLVED project system architecture.Figure 1. The SOLVED project system architectureInstallation stepsLet’s review each of the steps in this use case.Data ingestionThe data ingestion layer of the SOLVED project consists of a continuous video stream captured as a rover moves through a field of solar panels.We used a Freenove 4WD Smart Car rover with Raspberry Pi. The mounted camera captures video as it moves through the field. We installed anAmazon Kinesis Video Streams Produceron the Pi and streamed the live video to a Kinesis Video Stream named reinventbuilder2021.Figure 2 shows the Kinesis Video Stream setup window for reinventbuilder2021.Figure 2. Kinesis Video Stream setup for reinventbuilder2021To start streaming, use the following steps.Create a new Kinesis Video Stream using thisAmazon Kinesis Video Streams Developer GuideMake a note of the Amazon Resource Name (ARN)On the Pi, access the command prompt and use aws sts get-session-token for temporary credentials. TheIAMuser should have thepermissionsforKinesis Video Streams PutMedia.Set the following environment variables:export AWS_DEFAULT_REGION="us-east-1"
export AWS_ACCESS_KEY_ID="xxxxx"
export AWS_SECRET_ACCESS_KEY="yyyyy"
export AWS_SESSION_TOKEN=“zzzzz”Start the streamer using the following command:cd ~/amazon-kinesis-video-streams-producer-sdk-cpp/build
./kvs_gstreamer_sample reinventbuilder2021Validate the captured stream by viewing the Media playback on the console.Figure 3 shows the video stream console, including the Media playback option.Figure 3. Video stream console with Media playback optionThere are two ways to clip video snippets, which we’ll do next.You can use the Download clip button on the video stream console as shown in Figure 4.Figure 4. Choose your video streaming clip durationAlternately, you can use a script from the following command line:ONE_MIN_AGO=$(date -v -30S -u "+%FT%T+0000")NOW=$(date -u "+%FT%T+0000")FILE_NAME=reinventbuilder-solved-$RANDOM.mp4echo $FILE_NAMES3_PATH=s3://videoondemandsplitter-source-e6lyof9qjv1j/aws kinesis-video-archived-media get-clip --endpoint-url $KVS_DATA_ENDPOINT \--stream-name reinventbuilder2021 \--clip-fragment-selector "FragmentSelectorType=SERVER_TIMESTAMP,TimestampRange={StartTimestamp=$ONE_MIN_AGO,EndTimestamp=$NOW}" \$FILE_NAMEecho "Running get-clip for stream"sleep 45aws s3 cp $FILE_NAME $S3_PATHecho "copying file $FILE_NAME TO $S3_PATH"The clip is available in the Amazon S3 source folder created usingAWS CloudFormation, as shown in Figure 5.Figure 5. Access your clip in the Amazon S3 source folderPre-processingTo process the video, we leverageVideo on Demand at AWS.This solution encodes video files withAWS Elemental MediaConvert. Out of the box, it:1. Automatically transcodes videos uploaded to Amazon S3 into formats suitable for playback on a range of devices using MediaConvert2. Customizes MediaConvert job settings by uploading a custom file and using different settings per input3. Stores transcoded files in a destination Amazon S3 bucket and uses CloudFront to deliver them to end viewers4. Provides outputs including input file metadata, job settings, and output details in addition to transcoded video. These outputs are stored in a separate JSON file, available for further processingFor our use case, we used the frame capture feature to create a set of thumbnails from the source videos. The thumbnails are stored in the Amazon S3 bucket with the video output.To deploy this solution, use theCloudFormation stack.Processing and visualizationEvery trained ML model requires quality training data. We began with publicly available solar panel images that were categorized as “good” or “defective” and uploaded the images to an Amazon S3 bucket into corresponding folders.Next, we configuredAmazon Rekognition Custom Labelswith the folders to indicate the labels to use in training and deploying the model. Using the rover images, we tested the model.We used the rover to record videos of good and damaged solar panels over an extended period and label the outcome favorably. The video was then split into individual frames usingMediaConvert, giving us a well-labeled dataset that we trained our model with using Amazon Rekognition Custom Labels.We used the model endpoint to infer outcomes on solar panels with varying damage footprints across multiple locations. AWS Elemental Mediaconvert expedited the process of curating the training set, and creating the model and endpoint using Amazon Rekognition was straightforward.As shown in Figure 6, we used a training set of 7,000 images with an even mix of good and damaged panels.Figure 6. A training set of imagesExamples of good panel images are depicted in Figure 7.Figure 7. Good panel imagesExamples of damaged panel images are depicted in Figure 8.Figure 8. Damaged panel imagesIn this use case, 90 percent model accuracy was achieved.To visualize the results, we leveragedAWS Amplifyto provide an operator interface to identify the damaged panels.Figure 9 shows screenshots from the operator dashboard with output from the Amazon Custom Labels Rekognition model for good and defective panels.Figure 9. Operator dashboard in AWS AmplifyAlertingMaintenance teams must be notified of defective panels to take corrective action. To create alerts, we configuredAWS IoT Coreto send MQTT messages to a Philips Hue smart bulb, with red bulbs indicating defective panels. To set up the Philips Hue API, use theHow to develop for Hue guide.For example, here’s the API to change color:PUT https://192.xx.xx.xx/api/xxxxxxx/lights/1/state{"on":true, "sat":254, "bri":254,"hue":20000}turns color to green{"on":true, "sat":254, "bri":254,"hue":1000}turns to red.We set up a client on the Pi that listens on an AWS IoT Core MQTT topic and makes an API request to Philips Hue.To connect a device to AWS IoT, complete these steps:Create an IoT thing, a device certificate, and anAWS IoT policy. An AWS IoT thing represents a physical device (in this case, Raspberry Pi) and contains static device metadata, as shown in Figure 10.Figure 10. AWS IoT Thing2. Create a device certificate, required to connect to and authenticate with AWS IoT. An example is shown in Figure 11.Figure 11. Device certificate3. Associate an AWS IoT policy with each device certificate. They determine which AWS IoT resources the device can access. In this case, we allowed iot.*, giving the device access to all IoT resources, as shown in Figure 12.Figure 12. IoT policyDevices and other clients use anAWS IoT root CA certificateto authenticate the server they’re communicating with. For more on how devices authenticate with AWS IoT Core, seeServer authenticationin theAWS IoT Core Developer Guide.Copy the certificate chain to the Raspberry Pi.For communication with the Philips Hue, we used theQhue wrapperas shown in Figure 13.Figure 13. Qhue wrapperThe authors presented a demo of this solution at re:Invent 2021 Builder’s Fair.Figure 14. Author demo at re:Invent 2021 Builder’s FairClean upIf you used the CloudFormation stack, delete it to avoid unexpected future charges. Delete Amazon S3 buckets and terminate Amazon Rekognition jobs to stop accruing charges.ConclusionAmazon Rekognition helps customers collect images in the field and apply AI-based analysis to interpret the condition of assets within the images.In this post, you learned how to configure the Kinesis Video Stream producer on a Raspberry Pi to upload captured videos to Amazon Kinesis Video streams. You also learned how to save video streams to Amazon S3 and leverage the Video on Demand at AWS solution.Using AWS MediaConvert, we transcoded the videos and create a set of thumbnails from the source videos. We then used Amazon Rekognition Custom Labels to train and deploy models for solar panel damage detection. Finally, we configured AWS IoT core to send MQTT messages to a Philips Hue smart bulb for notifications.In this post, we presented a serverless architecture on AWS to detect defective solar panels. The reference architecture diagram is adaptable to solve inspection and damage detection problems across other industries.TAGS:artificial intelligence,Customer Solutions,machine learning,serverlessRamakant JoshiRamakant Joshi is an AWS Solutions Architect, specializing in the analytics and serverless domain. He has a background in software development and hybrid architectures, and is passionate about helping customers modernize their cloud architecture.Shekar TippurShekar Tippur is a Data Architect with AWS Professional Services based in Southern California. Shekar specializes in designing and developing Analytics and ML solutions using AWS technologies for customers in various industries. Outside of work, he enjoys running, reading, cooking, and spending time with his family.Anand IyerAnand is a Principal Solutions Architect at AWS since 2016. He has helped global healthcare, financial services, and telco clients architect and implement enterprise software solutions using AWS and hybrid cloud technologies. He has an master's in Computer Science from Louisiana State University Baton Rouge, and an MBA from USC Marshall School of Business, Los Angeles. He is AWS certified in Security, Solutions Architecture, and DevOps Engineering.
Enterprises perform quality control to ensure products meet production standards and avoid potential brand reputation damage. As the cost of sensors decreases and connectivity increases, industries adopt real-time imagery analysis to detect quality issues. At the same time, artificial intelligence (AI) advancements enable advanced automation, reduce overall cost and project time, and produce accurate defect […]
Ramakant Joshi
Amazon Machine Learning Amazon Rekognition Architecture Artificial Intelligence Serverless Technical How-to