Objective
Collaborated in a team to conduct a comparative analysis of YOLOv4 and SalsaNext, two neural network based object detection techniques, for LiDAR data.
Approach
Benchmarking -- Both machine learning models, YOLOv4 and SalsaNext, were pre-trained on the PandaSet dataset, which we used as a benchmark. We evaluated their average precision for detecting cars in the test data. YOLOv4 achieved an average precision of 0.90904, while SalsaNext slightly outperformed it with 0.91004—indicating a marginal 0.1% improvement in precision by SalsaNext.
​
Experimental data -- For experimental analysis, we used the KITTI and WADS datasets. The KITTI dataset served as a representation of typical driving conditions on roads, while the WADS dataset introduced obscurities such as snow, providing a challenging environment for comparison. This dual-dataset approach allowed us to evaluate model performance in both standard and adverse conditions.
​
Analysis & Results -- In our implementation of YOLOv4, only 75% of the sensor's data around the field of view was visualized, compared to SalsaNext, which provided a full 360° perspective. To assess performance, we focused on the number of cars accurately detected by each model out of the total cars visible to it. This approach provided insight into the models’ effectiveness in varying environmental conditions.
YOLOv4
SalsaNext
Ten point cloud frames from the KITTI and WADS datasets were tested on both models to evaluate their performance. SalsaNext demonstrated consistency in its detections, reliably identifying the same cars across successive frames. Considering environmental obscurities such as snowfall, which can negatively impact object detection, we found that both YOLOv4 and SalsaNext performed comparably overall in our analysis. YOLOv4 detected 54.8% of cars, while SalsaNext detected 47.55% of cars.
​
When broken down by dataset, YOLOv4 identified 49.6% of cars in the KITTI dataset, whereas SalsaNext detected 55.1%. Conversely, in the more challenging WADS dataset, YOLOv4 outperformed SalsaNext, detecting 60% of cars compared to SalsaNext's 40%. This counterintuitive result highlights a tradeoff: YOLOv4 delivered better overall performance but lacked the detection consistency exhibited by SalsaNext. Ultimately, the choice between these models depends on the application's priorities—whether performance or consistency is more critical. Both models would benefit from further optimization to handle environmental challenges such as snow, fog, and rain more effectively.