Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(T4Metric): fix incorrect matching of ground truths #158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
KSeangTan merged 14 commits into from
Dec 24, 2025
Merged

feat(T4Metric): fix incorrect matching of ground truths #158

KSeangTan merged 14 commits into from
Dec 24, 2025

Conversation

@KSeangTan
Copy link
Collaborator

@KSeangTan KSeangTan commented Dec 22, 2025 •
edited
Loading

Summary

This PR fixes a bug in T4Metric that it doesn't consider the ground truth case of merging truck and trailer. Before this PR, T4Metric loads ground truths directly from annotations without considering some preprocessing.
Since annotations are projected to the sensor coordinate in an info file, T4Metric will reproject annotations back to global coordinates when computing metrics. Therefore, it introduces slightly offset, as a result of that, only few boxes (1-2) might get filtered out compared to the previous results, especially, car and truck.
However, it doesn't significantly impact mAP overall.

This PR also significantly reduces inference time for metric computing about 1.5-2 times since we remove repeating loading .json files in T4Metric.

Change point

  • Load ground truths directly from _test_info.pkl instead from json files [1]
  • Load Gts and preds in sensor coordinates and reproject them back to global coordinates [2]
  • Add statistic about pred boxes [3]
  • Restructure the code t4_evaluate_all_scenes, where it doesn't need a for-loop [4]
  • Rename t4_evaluate to _evaluate_scenes, and call _load_eval_boxes to load Gts and Preds [5]
  • Add _load_eval_boxes to T4Metric to load Gts and Preds [6]
  • Format Gt boxes in format_result [7]
  • Add _format_gt_lidar_bbox to format Gt bboxes [8]
  • Update config details in CenterPoint for T4MetricV2 [9], [10], [11], [12]

Test performed

  • Before:
    2025/12/21 02:17:23 - mmengine - INFO - ==== GT boxes info after filtering in evaluation ====
    2025/12/21 02:17:23 - mmengine - INFO - class: car: 23898
    2025/12/21 02:17:23 - mmengine - INFO - class: pedestrian: 4556
    2025/12/21 02:17:23 - mmengine - INFO - class: bus: 957
    2025/12/21 02:17:23 - mmengine - INFO - class: truck: 1534
    2025/12/21 02:17:23 - mmengine - INFO - class: bicycle: 163
    2025/12/21 02:17:23 - mmengine - INFO - ===== End of GT Boxes info ====
    2025/12/21 02:17:23 - mmengine - INFO - ==== Est boxes info after filtering in evaluation ====
    2025/12/21 02:17:23 - mmengine - INFO - class: pedestrian: 15504
    2025/12/21 02:17:23 - mmengine - INFO - class: car: 36363
    2025/12/21 02:17:23 - mmengine - INFO - class: bicycle: 302
    2025/12/21 02:17:23 - mmengine - INFO - class: truck: 4033
    2025/12/21 02:17:23 - mmengine - INFO - class: bus: 1464
    2025/12/21 02:17:23 - mmengine - INFO - ===== End of Est Boxes info ====
    2025/12/21 02:17:45 - mmengine - INFO - ==== 30 scenes ====
    2025/12/21 02:17:45 - mmengine - INFO -
class_name mAP [email protected] [email protected] [email protected] [email protected]
car 85.5 79.3 85.6 88.3 88.6
truck 59.2 48.5 56.1 60.2 72.0
bus 74.0 66.6 74.3 77.4 77.6
bicycle 79.3 76.4 80.3 80.3 80.3
pedestrian 69.1 67.6 68.7 69.4 70.8

Total mAP: 0.734
mmengine - INFO - Elapsed time: 4183.9590 seconds

  • After:
    2025/12/21 13:05:56 - mmengine - INFO - ==== GT boxes info after filtering in evaluation ====
    2025/12/21 13:05:56 - mmengine - INFO - class: car: 23897
    2025/12/21 13:05:56 - mmengine - INFO - class: pedestrian: 4556
    2025/12/21 13:05:56 - mmengine - INFO - class: bus: 957
    2025/12/21 13:05:56 - mmengine - INFO - class: truck: 1548
    2025/12/21 13:05:56 - mmengine - INFO - class: bicycle: 163
    2025/12/21 13:05:56 - mmengine - INFO - ===== End of GT Boxes info ====
    2025/12/21 13:05:56 - mmengine - INFO - ==== Est boxes info after filtering in evaluation ====
    2025/12/21 13:05:56 - mmengine - INFO - class: pedestrian: 15504
    2025/12/21 13:05:56 - mmengine - INFO - class: car: 36363
    2025/12/21 13:05:56 - mmengine - INFO - class: bicycle: 302
    2025/12/21 13:05:56 - mmengine - INFO - class: truck: 4033
    2025/12/21 13:05:56 - mmengine - INFO - class: bus: 1464
    2025/12/21 13:05:56 - mmengine - INFO - ===== End of Est Boxes info ====
    2025/12/21 13:06:16 - mmengine - INFO - ==== 30 scenes ====
    2025/12/21 13:06:16 - mmengine - INFO -
class_name mAP [email protected] [email protected] [email protected] [email protected]
car 85.3 78.7 85.6 88.3 88.6
truck 72.8 62.6 72.9 76.6 79.2
bus 74.0 66.6 74.3 77.4 77.6
bicycle 79.3 76.4 80.3 80.3 80.3
pedestrian 69.1 67.6 68.7 69.4 70.8

Total mAP: 0.761
mmengine - INFO - Elapsed time: 2379.7419 seconds

  • T4MetricV2:

Summary:

Label Predict_num GT_nums Thresholds mean AP APs Mean APH APHs
car 36364 23898 0.50/1.00/2.00/4.00 0.8545 0.7925 / 0.8561 / 0.8832 / 0.8862 0.803 0.7547 / 0.8061 / 0.8221 / 0.8307
truck 4035 1548 0.50/1.00/2.00/4.00 0.7285 0.6266 / 0.7297 / 0.7660 / 0.7917 0.719 0.6193 / 0.7208 / 0.7559 / 0.7812
bus 1464 957 0.50/1.00/2.00/4.00 0.7407 0.6665 / 0.7442 / 0.7748 / 0.7773 0.707 0.6363 / 0.7128 / 0.7364 / 0.7441
bicycle 302 163 0.50/1.00/2.00/4.00 0.7943 0.7689 / 0.8028 / 0.8028 / 0.8028 0.761 0.7357 / 0.7688 / 0.7688 / 0.7688
pedestrian 15504 4556 0.50/1.00/2.00/4.00 0.6912 0.6758 / 0.6870 / 0.6939 / 0.7082 0.567 0.5572 / 0.5647 / 0.5688 / 0.5789

mAP: 0.7618, mAPH: 0.7116 (Center Distance BEV)
Elapsed time: 6353.7182 seconds

@KSeangTan KSeangTan requested a review from yamsam December 22, 2025 00:59
@KSeangTan KSeangTan requested a review from Copilot December 22, 2025 05:03
@KSeangTan KSeangTan self-assigned this Dec 22, 2025
Copilot AI reviewed Dec 22, 2025
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug in T4Metric where ground truth annotations were loaded directly from JSON files without considering preprocessing steps like truck-trailer merging. The fix loads ground truths from the preprocessed _test_info.pkl file instead, ensuring consistency with how annotations are handled during training. Additionally, the PR significantly improves performance by reducing redundant JSON file loading operations, achieving a 2-3x speedup in metric computation time.

Key changes:

  • Ground truths now loaded from preprocessed info files rather than raw JSON annotations
  • Boxes are loaded in sensor coordinates and reprojected to global coordinates for evaluation
  • Evaluation pipeline restructured to eliminate redundant file loading operations

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
autoware_ml/detection3d/evaluation/t4metric/t4metric.py Core metric implementation updated to load GTs from info files, add box statistics logging, and restructure evaluation pipeline
autoware_ml/detection3d/evaluation/t4metric/loading.py Removed deprecated GT/prediction loading functions that read from JSON files
projects/CenterPoint/configs/t4dataset/Centerpoint/second_secfpn_4xb16_121m_j6gen2_base_amp_t4metric_v2.py Updated config to provide required metadata for T4MetricV2
projects/CenterPoint/configs/t4dataset/Centerpoint/second_secfpn_4xb16_121m_j6gen2_base_amp.py Extended Pack3DDetInputs meta_keys to include required transformation matrices
projects/CenterPoint/configs/t4dataset/Centerpoint/second_secfpn_4xb16_121m_base_amp_t4metric_v2.py Updated config to provide required metadata for T4MetricV2
projects/CenterPoint/configs/t4dataset/Centerpoint/second_secfpn_4xb16_121m_base_amp.py Extended Pack3DDetInputs meta_keys to include required transformation matrices

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

vividf
vividf approved these changes Dec 24, 2025
View reviewed changes
Copy link
Collaborator

@vividf vividf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@KSeangTan KSeangTan merged commit bf2f3d2 into main Dec 24, 2025
2 checks passed
@KSeangTan KSeangTan deleted the feat/debug_perception_eval branch December 24, 2025 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@vividf vividf vividf approved these changes

@jacoblambert jacoblambert Awaiting requested review from jacoblambert

@SamratThapa120 SamratThapa120 Awaiting requested review from SamratThapa120

@yamsam yamsam Awaiting requested review from yamsam

Assignees

@KSeangTan KSeangTan

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@KSeangTan @vividf