Project Overview
Our goal is to enhance the panoptic segmentation model’s ability to accurately recognize and segment road lines, such as lane markings and road boundaries, which are essential for reliable road scene understanding.
We found that the Mapillary dataset provides very high-quality annotations for road lines, especially lane markings and road boundaries, which makes it an excellent dataset for improving road scene understanding.
Based on this advantage, we convert the Mapillary dataset into COCO format and fine-tune Mask2Former using NVIDIA TAO Toolkit. The trained model is then deployed on NVIDIA edge devices with TensorRT to achieve efficient and real-time panoptic segmentation.
Demo Video (inference on Nvidia Jetson Orin)
Mask2Former Introduction
Mask2Former is a universal segmentation framework that unifies semantic, instance, and panoptic segmentation in a single transformer-based model. With masked attention, it focuses on important image regions and generates high-quality masks efficiently, making it well suited for complex scene understanding in autonomous driving and robotics.
Explanation of the Mapillary dataset in COCO panoptic format
Training result
Inference Pipeline
