Method
Auto-LIA Measurement System
Introduction
Auto-LIA is an efficient leaf inclination angle measurement system based on computer vision techniques and spatial geometric transformation. Specifically, first, the hardware part of the system consists of two relatively stationary RGB cameras with an inclinometer, and a set of input data consists of two RGB images with the angle between the hardware and the horizontal plane. Then, we combine object detection, as Yolov7, and 3D reconstruction such as RAFT for recovering leaf phenotypic information in the imaging space, which is the basis for subsequent calculations. Next, as for the accurate extraction of the key points of leaves, we propose a surface segmentation based on the gradient of spatial depth and angles, which implements the precision segmentation for leaves. Finally, we put forward a set of equations for the transformation from the spatial information of the leaves to LIA with the association between the image plane, the horizontal plane, and the target plant. The process we propose automates the entire process of non-invasive collection of leaf inclination angle, solves labor-intensive and invasive problems, and provides a strong correlation for individual plants.
Device parameter setting
| Device-Type | Numbers | Equipment Model |
|---|---|---|
| CPU | 32C | Intel(R) Xeon(R) Gold 6230R CPU @ 2.10GHz |
Usage
Auto-LIA based on the Yolov7 and RAFT, the parameters and the download link of pre-train model and dataset in the list：
Dependencies
- Linux with Python >= 3.7
- matplotlib>=3.2.2
- numpy>=1.18.5,<=1.24.0
- opencv-python>=4.1.1
- Pillow>=7.1.2
- PyYAML>=5.3.1
- requests>=2.23.0
- scipy>=1.4.1
- torch>=1.7.0,!=1.12.0
- torchvision>=0.8.1,!=0.13.0
- tqdm>=4.41.0
- protobuf<=4.21.3
- pandas>=1.1.4
- NOTICE: different versions of Pytorch package have different memory usages.
Instructions
Hardware parameter settings:
Change the following contents of def main() in main.py to the hardware settings for image capturing.
srcCamera = CameraCutParam(2875.3,2868.62,715.189,350.285)
rightCamera = CameraCutParam(3033.7,3020.2,707.717,328.642)
R = np.array([[0.998876,0.00216193,0.0473521],[-0.00175768,0.999962,-0.00857723],[-0.0473688,0.00848436,0.998841]])
t = np.array([-65.6515,-4.46654 ,-5.01777 ])
outerParam = CameraOuterParam(R , t)
Testing:
python main.py --source ./demo_output/17_1_rectify.jpg --source2 ./demo_output/17_2_rectify.jpg --floor_angle 29.35 --filter_pixels_min 2000 --filter_pixels_max 50000 --detect_model_path yolov7.pth --depth_model_path raft_kitti.pth
Evaluation
We also provide batch prediction methods with outside environment.
The performance of Auto-LIA under various parameters is shown in Figures.We suggest users to take 32.5° as the shooting angle with 0.5 as the segmentation threshold while adapting the Auto-LIA system to the real scene.
Results
Valuation of the accuracy of leaf inclination measurements.
Average error is a metric of the Auto-LIA's accuracy, with statistics on the minimum average error for each set of data across all parameters. The measurement accuracy of the system is optimal when the average error reaches the minimum, and its optimal measurement accuracy is obtained in group 2, which is 5.42°.
Sematic of LIA measurement when using different functions.
We verify the influence of the detection method and jump pixel in the measurement process. The evaluation statistics of the results include recognition rate, average error and calculation time.
Results of the real and estimated LIA at different shooting angles.
The environment around the plant includes the scene around the plant, and the light of the environment.
Process Visualization of Auto-LIA.
a) Two synchronous images of the input system. (b) Two calibrated input images. (c) Two cut images without losing too much semantic information. (d) The depth image of reconstructed (c). (e) Segmented area of the scene. (f) A schematic diagram of leaf vein extraction with leaves’ regions. (g) The result of Auto-LIA’s measurement. The whole visualization proves the soundness of our approach.
Visualization of functional effects.
(a) two cropped source images are used for LIA estimations. (b) the segmentation results and LIA measurements obtained by running the program without denoising operations. (c) the leaf results obtained by running the program with the jump pixels set to 2. The set of images demonstrates that the denoising operation has a positive effect on leaf segmentation as well as on LIA measurements.
Association of LIA with plant physiological status, taking hydration as an example.
(a) Physical drawings of representative plant morphology under well-watered situations in our collection. The plant on the left demonstrates the characteristic appearance of bent, shrinking, curly leaves under conditions of water deprivation. The plant on the right shows how the leaves look when well watered. (b) Changes over 8 hours in the different plants' LIAs. The mean LIAs of the two groups show different tendencies of change with time. The variances of LIAs for each group present the distribution of LIA in different time periods. } %These figures illustrate that leaves acquire different physical characteristics depending on their physiological state.