Trace when server provides image with duplicate metadata #1152
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The CDS will repeatedly download 'duplicate' images if the Contest API server says the file is different than what the CDS determines it to be after downloading. For example, if the server says it has an image that's 50x50, the CDS downloads it, and reads it as 51x51, then the next time you restart we'll have a 51x51 image, see that the source has a 'different' one, and download it a second time. This will happen every time the CDS restarts, leading to many duplicate images. The most obvious solution is to fix the bug in determining image size or file reference data in the CDS or the upstream server, but that takes time and is hard to track down, so this just starts by logging whenever this occurs to make it easier to track. In the future this code would make it easy to take the next step and delete the file after download, but I want to run it in a few contests before making that step. This change: - Moves the logic of whether a file reference matches an existing one from PlaybackContest to FileReferenceList. - Exposes the local file cache information from DiskContestSource. - After downloading in RestContestSource, check if the file matches one we already have. Most of icpctools#1125.
nickygerritsen
approved these changes
Jul 30, 2025
deboer-tim
added a commit
to deboer-tim/icpctools
that referenced
this pull request
Jul 31, 2025
The fix in icpctools#1152 fails in many cases due to a casting issue passing a list to FileReferenceList. Fixes that by adding the objects manually. Also noticed an NPE when a team name is null. Won't happen in a real contest, but currently exists in baku data.
deboer-tim
added a commit
that referenced
this pull request
Jul 31, 2025
The fix in #1152 fails in many cases due to a casting issue passing a list to FileReferenceList. Fixes that by adding the objects manually. Also noticed an NPE when a team name is null. Won't happen in a real contest, but currently exists in baku data.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The CDS will repeatedly download 'duplicate' images if the Contest API server says the file is different than what the CDS determines it to be after downloading. For example, if the server says it has an image that's 50x50, the CDS downloads it, and reads it as 51x51, then the next time you restart we'll have a 51x51 image, see that the source has a 'different' one, and download it a second time. This will happen every time the CDS restarts, leading to many duplicate images.
The most obvious solution is to fix the bug in determining image size or file reference data in the CDS or the upstream server, but that takes time and is hard to track down, so this just starts by logging whenever this occurs to make it easier to track. In the future this code would make it easy to take the next step and delete the file after download, but I want to run it in a few contests before making that step. This change:
Most of #1125.