Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Comments

GTC-3001: Pre-download and share sources#183

Merged
dmannarino merged 13 commits intodevelopfrom
gtc-3001_predl2
Feb 24, 2026
Merged

GTC-3001: Pre-download and share sources#183
dmannarino merged 13 commits intodevelopfrom
gtc-3001_predl2

Conversation

@dmannarino
Copy link
Member

@dmannarino dmannarino commented Oct 8, 2024 •
edited
Loading

Pull request type

Please check the type of change your PR introduces:

  • Bugfix
  • Feature
  • Build related changes

What is the current behavior?

  • Each worker process downloads a copy of the source, which effectively limits us to ~33GB datasets (because ~33GB * ~30 workers = ~1TB, the amount of storage on an instance).
  • Build is ancient and fails due to outdated deps and infra

Issue Number: GTC-2302

What is the new behavior?

  • Build succeeds, uses more modern tools
  • 1 copy of the source files is downloaded on instantiating a layer, and hard links are used to refer to it thereafter.
  • Uses newer Python (3.11)
  • I added in some basic OOM tolerance. Sorry, this one is feature creep.

Does this introduce a breaking change?

  • Yes
  • No

Other information

dmannarino and others added 10 commits September 29, 2024 18:07
@dmannarino
Update isort
6b3b8a8
@dmannarino
Update rasterio and incidentals
f5349af
@dmannarino
Move (unmodified) download functions to utils/sources.py
2bfda56
@dmannarino
docker-compose -> docker compose
dd018d9
@dmannarino
Make process_locally default to True
6b640fa
@dmannarino
WIP: source files disappear?
7b67273
@gtempus
test(raster_src_tile): create a layer in the test
500c313 
We no longer rely on the LAYER fixture.
@dmannarino
Improve test, use current dir in download path
1ad5bd4
@dmannarino
Fix troublesome use of lstrip to form link file paths
bf8cf4d
@dmannarino
Modernize project: Upgrade Python, rasterio, GDAL, build stuff (#186)
f5efc49 
* Switch to UV; Upgrade to GDAL 3.9.3, rasterio 1.4.3, PostgreSQL 14
* Remove unneeded cert stuff from Dockerfile
* Fix warnings about geom type
* Fix str, Enum issue with Python >3.10
* Fix bugs turned up by tests
* Modernize GHA files
* Pin TF, fix for new Docker
* Upgrade AWS prov. versions
@dmannarino dmannarino marked this pull request as ready for review February 23, 2026 15:47
@dmannarino dmannarino merged commit 37ac8ba into develop Feb 24, 2026
3 of 4 checks passed
@dmannarino dmannarino deleted the gtc-3001_predl2 branch February 24, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dmannarino @gtempus