Prework

• I understand and agree to help guide.
• I understand and agree to contributing guide.
• New features take time and effort to create, and they take even more effort to maintain. So if the purpose of the feature is to resolve a struggle you are encountering personally, please consider first posting a "trouble" or "other" issue so we can discuss your use case and search for existing solutions first.

Proposal

Related to #1055 because it deals with metadata files.

I propose optionally syncing _targets/meta/meta and _targets/meta/progress to an S3/GCS bucket. Updates could happen whenever a batched metadata write occurs (#1055).

I imagine this would involve increasing the default seconds_interval value. It would also become necessary to change how S3/GCS bucket prefixes are treated, which might invalidate existing pipelines that use custom prefixes.

But the advantages could be transformative. Shiny apps, runs on GitHub Actions, etc. would not need to save any part of the data store anywhere, which would remove the hardest challenge of using targets pipelines in these cases. Plus, it would be much easier to run a pipeline on one computer and check progress/results on another computer. No more need to manually copy the data store to get the metadata.