Thermal Point Cloud
ProjectThe purpose of this project was to combine depth data and thermal data into one point cloud. This was achieved using a RealSense D435i and a FLIR Lepton 3.1R. The raw thermal data from the Lepton was collected and transmitted via UDP using a Raspberry Pi. The thermal data was processed into images and streams which were used for calibration of the camera. The color stream of the RealSense and thermal stream of the Lepton were used to calculate the extrinsics between the camera frames. Finally, the depth data from the RealSense determined the position of the points in the point cloud while the thermal data determined the color of the points. This led to a 3D thermal point cloud.
Video Demo
Packages
The following packages were developed for this project.
Work Flow
Thermal Camera and Data
To begin with, thermal data had to be obtained. I settled on a Lepton 3.1R as it was radiometrically calibrated and had open-source software. It communicates through I2C to set its capabilities and functions and then provides lens data quickly through SPI. The resolution of the images were 160 X 120, capable of reading temperatures from -10°C to 140°C in high gain mode.
I rigged a Raspberry Pi 4 to connect to the Lepton camera and transmit the raw data to be processed later down the pipeline. I made a few changes to the existing code base, mainly removing the QT dependencies and adding a UDP socket to pipe the data over to a different machine.
Raspberry PI with Lepton 3.1R
Once the data was streamed, the next step was to verify if the temperatures read were accurate. This was done using 2 temperature baselines: boiling water and fridge temperature(-2°C). A thermocouple was used to verify the temperature being read. The pixel data is an unsigned 16-bit integer that hold the temperature in hectoKelvin (x102K).
Calibration
Next step I had looked at was to calibrate the camera to determine its intrinsics. This was integral to fixing the barrel distortion and determine the camera matrix to project pixel to 3D points. For calibration, I initially designed a square mesh pattern. This used rods of 3mm diameter but this was unfortunately too small to be picked up accurately by the pixels. The next design was using laser cut acrylic in traditional calibration patterns. The size of the squares or circles were large, 30mm to be exact, so as to be clearly picked up by the low resolution camera. Below are some sample images.
Calibration Patterns
Calibration Patterns as seen by the thermal camera
Using OpenCV’s calibrateCamera function the intrinsics were calculated with a reprojection error of 2.15 pixels. Once the intrinsics were calculated, the next step was to create a housing for the Raspberry Pi-Lepton device and a RealSense D435i to fix the relative positioning of the cameras. Then by capturing thermal and RGB image pairs of a chessboard pattern the extrinsic transformation between the two camera frames were determined.
RPi, Lepton and RealSense Housing
Extrinsic calibration image pair (thermal image has been stretched)
Process of Calibration
OpenCV’s stereoCalibrate was good enough to calibrate the extrinsics of the thermal camera frame in the RealSense’s RGB camera frame. The RealSense pipeline was set up specifically with its depth and color image frames aligned to ease the extrinsic calculation. The reprojection error from the calibration was 0.99 pixels. Now everything is ready to build point clouds.
Point Cloud
To build the point cloud, I looked at how Intel’s RealSense wrapper example code functioned. Using that as a template, I created a program that would process the raw thermal data and convert the temperatures based on a scale to a colormap based on the Lepton code base. The depth data from the RealSense determines the position of the 3D points while the temperature colormap determines the colors of the points. The temperature data is computed but for visuals the thermal RGB colormap is a better representation.
Future Work
The logical next step is to use this point cloud to perform some tasks and also to refine the data that is generated by the point cloud. Mainly I would like to work on reducing the noisy data using a median filter, offloading more code to the GPU to better the performance and then try classifying object to convert surface temperatures to internal temperatures.
References
Multiple Methods of Geometric Calibration of Thermal Camera - Ruixuan Liu, Hengrui Zhang, Sebastian Scherer Link to pdf
Acknowledgement
Grateful for your help and guidance, Matt Elwin!