Fix broken SAM and SAM-HQ due to tie weights and type validation#43420
Conversation
|
[For maintainers] Suggested jobs to run (before merge) run-slow: edgetam, edgetam_video, sam, sam2, sam2_video, sam3_tracker, sam3_tracker_video, sam_hq |
| self.assertTrue(iou_scores.shape == (1, 2, 3)) | ||
| torch.testing.assert_close( | ||
| iou_scores, torch.tensor([[[0.9105, 0.9825, 0.9675], [0.7646, 0.7943, 0.7774]]]), atol=1e-4, rtol=1e-4 | ||
| iou_scores, torch.tensor([[[0.9105, 0.9825, 0.9675], [0.7646, 0.7944, 0.7769]]]), atol=1e-4, rtol=1e-4 |
There was a problem hiding this comment.
This is due to the fast image processor being default now, and wasn't caught in the corresponding PR as it was failing at the time already (due to issues that are fixed in this PR)
There was a problem hiding this comment.
sounds good, drifts are minor
| input_boxes = torch.Tensor([[[620, 900, 1000, 1255]], [[75, 275, 1725, 850]], [[75, 275, 1725, 850]]]).cpu() | ||
| EXPECTED_IOU = torch.tensor([[[0.9773, 0.9880, 0.9522], [0.5995, 0.7658, 0.7936], [0.5995, 0.7658, 0.7936]]]) |
| self.assertTrue(torch.allclose(scores[0][-1], torch.tensor(0.8858), atol=1e-4)) | ||
| self.assertTrue(torch.allclose(scores[1][-1], torch.tensor(0.9092), atol=1e-4)) |
There was a problem hiding this comment.
This was never passing and the values were completely wrong
There was a problem hiding this comment.
indeed, that's my bad - I flagged it at the time but wasn't enforced https://github.com/huggingface/transformers/pull/35147/changes#r1967342982
thanks for updating!
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| self.assertTrue(iou_scores.shape == (1, 2, 3)) | ||
| torch.testing.assert_close( | ||
| iou_scores, torch.tensor([[[0.9105, 0.9825, 0.9675], [0.7646, 0.7943, 0.7774]]]), atol=1e-4, rtol=1e-4 | ||
| iou_scores, torch.tensor([[[0.9105, 0.9825, 0.9675], [0.7646, 0.7944, 0.7769]]]), atol=1e-4, rtol=1e-4 |
There was a problem hiding this comment.
sounds good, drifts are minor
| self.assertTrue(torch.allclose(scores[0][-1], torch.tensor(0.8858), atol=1e-4)) | ||
| self.assertTrue(torch.allclose(scores[1][-1], torch.tensor(0.9092), atol=1e-4)) |
There was a problem hiding this comment.
indeed, that's my bad - I flagged it at the time but wasn't enforced https://github.com/huggingface/transformers/pull/35147/changes#r1967342982
thanks for updating!
It looks like these models have been broken for a while due to incorrectly tie weights after the tie weight refactor, and more recently due to enforced typing in the processors
This PR replaces buffers that we can't tie anymore by nn.Parameters, and fixed the typing in processors