Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit bae694f

Browse files
ajeetdsouzazkochan
ajeetdsouza
and
zkochan
authored
fix(lockfile): compute tarball integrity upon download (#12491)
Some registries generate tarballs on demand and cannot list an integrity in their packument. pnpm then wrote integrity-less lockfile entries on the first install and failed the next one with ERR_PNPM_MISSING_TARBALL_INTEGRITY, unable to install from its own lockfile. Compute the missing integrity from the downloaded bytes and write it into the resolution before the lockfile is built: - Add an optional `resolutionNeedsFetch` contract to the fetcher API (backward compatible, since custom fetchers come from hooks). The remote-tarball fetcher reports it when a resolution lacks integrity; the picked fetcher's signal flows through PackageResponse -> ResolvedPackage so nothing re-derives it. - The package requester downloads such tarballs (including under --lockfile-only / skipFetch / not-installable) and fills the computed integrity onto the resolution via the already-running `fetching` promise, so dependency resolution isn't blocked. The deps-resolver awaits only the flagged entries before updateLockfile, because the integrity feeds the global virtual-store paths. - Move read-side enforcement into the npm resolver's lockfile verifier (MISSING_TARBALL_INTEGRITY): reject a registry/http(s) tarball entry whose integrity is missing/empty/non-string, fail-closed, before the URL-keyed and semver short-circuits. Drop the earlier read-side auto-heal (a missing-field bypass). Harden against tampered lockfiles (non-string tarball/integrity). - Reuse the fetcher picked during resolution on the fetch path instead of running pickFetcher (and a custom fetcher's async canFetch) twice per package. Mirrored in pacquet: PrefetchingResolver computes the integrity for integrity-less tarball resolutions during resolution (FetchTarballForResolution::run), deduped per URL with a singleflight cache. Closes #12145. --------- Co-authored-by: Zoltan Kochan <[email protected]>
1 parent d1635db commit bae694f

43 files changed

Lines changed: 1243 additions & 238 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.changeset/compute-missing-tarball-integrity.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
---
2+
"@pnpm/resolving.npm-resolver": minor
3+
"@pnpm/resolving.resolver-base": minor
4+
"@pnpm/fetching.fetcher-base": minor
5+
"@pnpm/hooks.types": minor
6+
"@pnpm/installing.package-requester": minor
7+
"@pnpm/installing.context": patch
8+
"@pnpm/installing.deps-resolver": patch
9+
"@pnpm/fetching.tarball-fetcher": patch
10+
"@pnpm/fetching.pick-fetcher": patch
11+
"@pnpm/store.controller-types": patch
12+
"@pnpm/lockfile.utils": minor
13+
"@pnpm/deps.graph-builder": patch
14+
"@pnpm/installing.deps-restorer": patch
15+
"@pnpm/patching.commands": patch
16+
"pnpm": minor
17+
---
18+
19+
Some registries generate tarballs on-demand and cannot provide an integrity checksum in their package metadata. In that case pnpm now computes the integrity from the downloaded tarball and stores it in the lockfile, so the entry is verifiable on subsequent installs instead of being written without an integrity (which would fail the next install). This also applies to `--lockfile-only`: the tarball is downloaded so its integrity can be computed. A lockfile entry that is still missing its integrity is rejected as a `ERR_PNPM_MISSING_TARBALL_INTEGRITY` lockfile verification violation (the install fails closed) rather than being silently re-fetched.

‎pacquet/crates/lockfile-verification/src/verify_lockfile_resolutions/tests.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -683,7 +683,7 @@ importers:
683683
packages:
684684
685685
[email protected]:
686-
resolution: {integrity: sha512-deadbeef, tarball: 'https://codeload.github.com/org/acme/tar.gz/abc123', gitHosted: false}
686+
resolution: {integrity: sha512-deadbeef, tarball: 'https://codeload.github.com/org/acme/tar.gz/0123456789abcdef0123456789abcdef01234567', gitHosted: false}
687687
688688
snapshots:
689689
@@ -756,7 +756,7 @@ importers:
756756
packages:
757757
758758
[email protected]:
759-
resolution: {integrity: sha512-deadbeef, tarball: 'https://CODELOAD.GITHUB.COM/org/acme/tar.gz/abc123', gitHosted: false}
759+
resolution: {integrity: sha512-deadbeef, tarball: 'https://CODELOAD.GITHUB.COM/org/acme/tar.gz/0123456789abcdef0123456789abcdef01234567', gitHosted: false}
760760
761761
snapshots:
762762

‎pacquet/crates/lockfile/src/resolution.rs‎

Lines changed: 81 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -452,21 +452,89 @@ impl From<ResolutionSerde> for LockfileResolution {
452452
}
453453
}
454454

455-
/// Best-effort URL-prefix check used to back-fill `gitHosted` on tarball
456-
/// resolutions written by older pnpm versions, and to gate trust on the
457-
/// tarball URL rather than the (tamper-prone) `gitHosted` flag. Mirrors
458-
/// upstream's `isGitHostedTarballUrl` at
459-
/// <https://github.com/pnpm/pnpm/blob/94240bc046/lockfile/fs/src/lockfileFormatConverters.ts#L23-L29>.
455+
/// Recognizes immutable archive URLs emitted by known git providers. The result
456+
/// gates integrity exemptions, so path shapes are matched explicitly and refs
457+
/// must be full commit SHAs.
460458
#[must_use]
461459
pub fn is_git_hosted_tarball_url(url: &str) -> bool {
462-
// Schemes and hostnames are case-insensitive, so match against a lowercased
463-
// copy: a tampered `https://CODELOAD.GITHUB.COM/...` must not slip past as a
464-
// non-git-hosted (and therefore registry-trusted) tarball.
465-
let lower = url.to_ascii_lowercase();
466-
(lower.starts_with("https://codeload.github.com/")
467-
|| lower.starts_with("https://bitbucket.org/")
468-
|| lower.starts_with("https://gitlab.com/"))
469-
&& lower.contains("tar.gz")
460+
let Some((host, path, query)) = parse_https_url(url) else { return false };
461+
if host.eq_ignore_ascii_case("codeload.github.com") {
462+
return is_github_codeload_archive(path);
463+
}
464+
if host.eq_ignore_ascii_case("bitbucket.org") {
465+
return is_bitbucket_archive(path);
466+
}
467+
if host.eq_ignore_ascii_case("gitlab.com") {
468+
return is_gitlab_archive(path, query);
469+
}
470+
false
471+
}
472+
473+
fn parse_https_url(url: &str) -> Option<(&str, &str, Option<&str>)> {
474+
const HTTPS_SCHEME: &str = "https://";
475+
if !url.get(..HTTPS_SCHEME.len())?.eq_ignore_ascii_case(HTTPS_SCHEME) {
476+
return None;
477+
}
478+
let rest = url.get(HTTPS_SCHEME.len()..)?;
479+
let (host, path_and_query) = rest.split_once('/')?;
480+
let path_and_query = path_and_query.split_once('#').map_or(path_and_query, |(path, _)| path);
481+
let (path, query) = path_and_query
482+
.split_once('?')
483+
.map_or((path_and_query, None), |(path, query)| (path, Some(query)));
484+
Some((host, path, query))
485+
}
486+
487+
fn is_github_codeload_archive(path: &str) -> bool {
488+
let segments = path_segments(path);
489+
segments.len() == 4 && segments[2] == "tar.gz" && is_full_commit_sha(segments[3])
490+
}
491+
492+
fn is_bitbucket_archive(path: &str) -> bool {
493+
let segments = path_segments(path);
494+
if segments.len() != 4 || segments[2] != "get" {
495+
return false;
496+
}
497+
let Some(commit) = segments[3].strip_suffix(".tar.gz") else { return false };
498+
is_full_commit_sha(commit)
499+
}
500+
501+
fn is_gitlab_archive(path: &str, query: Option<&str>) -> bool {
502+
let segments = path_segments(path);
503+
if segments.len() == 6
504+
&& segments[0] == "api"
505+
&& segments[1] == "v4"
506+
&& segments[2] == "projects"
507+
&& segments[4] == "repository"
508+
&& segments[5] == "archive.tar.gz"
509+
{
510+
return query_param(query, "ref").is_some_and(is_full_commit_sha);
511+
}
512+
let Some(archive_marker_index) =
513+
segments.windows(2).position(|window| window[0] == "-" && window[1] == "archive")
514+
else {
515+
return false;
516+
};
517+
if archive_marker_index < 2 || segments.len() != archive_marker_index + 4 {
518+
return false;
519+
}
520+
let commit = segments[archive_marker_index + 2];
521+
let archive_name = segments[archive_marker_index + 3];
522+
archive_name.ends_with(".tar.gz") && is_full_commit_sha(commit)
523+
}
524+
525+
fn path_segments(path: &str) -> Vec<&str> {
526+
path.split('/').filter(|segment| !segment.is_empty()).collect()
527+
}
528+
529+
fn query_param<'query>(query: Option<&'query str>, key: &str) -> Option<&'query str> {
530+
query?.split('&').find_map(|part| {
531+
let (part_key, value) = part.split_once('=')?;
532+
(part_key == key).then_some(value)
533+
})
534+
}
535+
536+
fn is_full_commit_sha(value: &str) -> bool {
537+
value.len() == 40 && value.as_bytes().iter().all(u8::is_ascii_hexdigit)
470538
}
471539

472540
impl From<LockfileResolution> for ResolutionSerde {

‎pacquet/crates/lockfile/src/resolution/tests.rs‎

Lines changed: 36 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,17 @@
11
use super::{
22
BinaryArchive, BinaryResolution, BinarySpec, DirectoryResolution, GitResolution,
33
LockfileResolution, PlatformAssetResolution, PlatformAssetTarget, PlatformSelector,
4-
RegistryResolution, TarballResolution, VariationsResolution, libc_matches,
5-
select_platform_variant,
4+
RegistryResolution, TarballResolution, VariationsResolution, is_git_hosted_tarball_url,
5+
libc_matches, select_platform_variant,
66
};
77
use crate::serialize_yaml;
88
use pretty_assertions::assert_eq;
99
use ssri::Integrity;
1010
use std::collections::BTreeMap;
1111
use text_block_macros::text_block;
1212

13+
const GIT_COMMIT: &str = "0123456789abcdef0123456789abcdef01234567";
14+
1315
fn integrity(integrity_str: &str) -> Integrity {
1416
integrity_str.parse().expect("parse integrity string")
1517
}
@@ -87,39 +89,37 @@ fn deserialize_tarball_resolution_with_git_hosted() {
8789
#[test]
8890
fn deserialize_tarball_resolution_backfills_git_hosted() {
8991
eprintln!("CASE: codeload.github.com");
90-
let yaml = text_block! {
91-
"tarball: https://codeload.github.com/foo/bar/tar.gz/abc1234"
92-
};
93-
let received: LockfileResolution = serde_saphyr::from_str(yaml).unwrap();
92+
let yaml = format!("tarball: https://codeload.github.com/foo/bar/tar.gz/{GIT_COMMIT}");
93+
let received: LockfileResolution = serde_saphyr::from_str(&yaml).unwrap();
9494
dbg!(&received);
9595
let expected = LockfileResolution::Tarball(TarballResolution {
96-
tarball: "https://codeload.github.com/foo/bar/tar.gz/abc1234".to_string(),
96+
tarball: format!("https://codeload.github.com/foo/bar/tar.gz/{GIT_COMMIT}"),
9797
integrity: None,
9898
git_hosted: Some(true),
9999
path: None,
100100
});
101101
assert_eq!(received, expected);
102102

103103
eprintln!("CASE: gitlab.com archive");
104-
let yaml = text_block! {
105-
"tarball: https://gitlab.com/foo/bar/-/archive/abc1234/bar-abc1234.tar.gz"
106-
};
107-
let received: LockfileResolution = serde_saphyr::from_str(yaml).unwrap();
104+
let yaml = format!(
105+
"tarball: https://gitlab.com/foo/bar/-/archive/{GIT_COMMIT}/bar-{GIT_COMMIT}.tar.gz",
106+
);
107+
let received: LockfileResolution = serde_saphyr::from_str(&yaml).unwrap();
108108
let expected = LockfileResolution::Tarball(TarballResolution {
109-
tarball: "https://gitlab.com/foo/bar/-/archive/abc1234/bar-abc1234.tar.gz".to_string(),
109+
tarball: format!(
110+
"https://gitlab.com/foo/bar/-/archive/{GIT_COMMIT}/bar-{GIT_COMMIT}.tar.gz",
111+
),
110112
integrity: None,
111113
git_hosted: Some(true),
112114
path: None,
113115
});
114116
assert_eq!(received, expected);
115117

116118
eprintln!("CASE: bitbucket.org archive");
117-
let yaml = text_block! {
118-
"tarball: https://bitbucket.org/foo/bar/get/abc1234.tar.gz"
119-
};
120-
let received: LockfileResolution = serde_saphyr::from_str(yaml).unwrap();
119+
let yaml = format!("tarball: https://bitbucket.org/foo/bar/get/{GIT_COMMIT}.tar.gz");
120+
let received: LockfileResolution = serde_saphyr::from_str(&yaml).unwrap();
121121
let expected = LockfileResolution::Tarball(TarballResolution {
122-
tarball: "https://bitbucket.org/foo/bar/get/abc1234.tar.gz".to_string(),
122+
tarball: format!("https://bitbucket.org/foo/bar/get/{GIT_COMMIT}.tar.gz"),
123123
integrity: None,
124124
git_hosted: Some(true),
125125
path: None,
@@ -153,6 +153,25 @@ fn deserialize_tarball_resolution_backfills_git_hosted() {
153153
assert_eq!(received, expected);
154154
}
155155

156+
#[test]
157+
fn is_git_hosted_tarball_url_rejects_false_positives() {
158+
assert!(is_git_hosted_tarball_url(&format!(
159+
"https://codeload.github.com/foo/bar/tar.gz/{GIT_COMMIT}"
160+
)));
161+
assert!(is_git_hosted_tarball_url(&format!(
162+
"https://gitlab.com/api/v4/projects/foo%2Fbar/repository/archive.tar.gz?ref={GIT_COMMIT}"
163+
)));
164+
assert!(!is_git_hosted_tarball_url("https://gitlab.com/foo/bar?download=tar.gz"));
165+
assert!(!is_git_hosted_tarball_url("https://codeload.github.com/foo/bar/tar.gz/main"));
166+
assert!(!is_git_hosted_tarball_url(
167+
"https://gitlab.com/foo/bar/-/archive/main/bar-main.tar.gz",
168+
));
169+
assert!(!is_git_hosted_tarball_url(
170+
"https://gitlab.com/api/v4/projects/foo%2Fbar/repository/archive.tar.gz",
171+
));
172+
assert!(!is_git_hosted_tarball_url("https://bitbucket.org/foo/bar/get/main.tar.gz"));
173+
}
174+
156175
#[test]
157176
fn serialize_tarball_resolution() {
158177
eprintln!("CASE: without integrity");

‎pacquet/crates/package-manager/src/prefetching_resolver.rs‎

Lines changed: 80 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -27,19 +27,25 @@ use crate::{
2727
install_package_from_registry::{extract_tarball, manifest_file_count, manifest_unpacked_size},
2828
retry_config::retry_opts_from_config,
2929
};
30-
use dashmap::DashSet;
30+
use dashmap::{DashMap, DashSet};
3131
use pacquet_config::Config;
32+
use pacquet_lockfile::{LockfileResolution, is_git_hosted_tarball_url};
3233
use pacquet_network::{AuthHeaders, ThrottledClient};
33-
use pacquet_reporter::Reporter;
34+
use pacquet_reporter::{Reporter, SilentReporter};
3435
use pacquet_resolving_resolver_base::{
35-
LatestQuery, ResolveFuture, ResolveLatestFuture, ResolveOptions, ResolveResult, Resolver,
36-
WantedDependency,
36+
LatestQuery, ResolveError, ResolveFuture, ResolveLatestFuture, ResolveOptions, ResolveResult,
37+
Resolver, WantedDependency,
3738
};
3839
use pacquet_store_dir::{
3940
SharedReadonlyStoreIndex, SharedVerifiedFilesCache, StoreDir, StoreIndexWriter,
4041
};
41-
use pacquet_tarball::{DownloadTarballToStore, MemCache, RetryOpts, SharedReportedProgressKeys};
42+
use pacquet_tarball::{
43+
DownloadTarballToStore, FetchTarballForResolution, MemCache, RetryOpts,
44+
SharedReportedProgressKeys,
45+
};
46+
use ssri::Integrity;
4247
use std::{marker::PhantomData, sync::Arc};
48+
use tokio::sync::OnceCell;
4349

4450
/// Borrowed-data bag handed to [`PrefetchingResolver::new`]. Everything
4551
/// the wrapper needs to drive a background tarball download:
@@ -92,6 +98,10 @@ struct OwnedFetchCtx {
9298
/// without this gate the bench saw ~3-5k redundant spawns per
9399
/// install on the alotta-files fixture (one per dependent edge).
94100
spawned_urls: Arc<DashSet<String>>,
101+
/// Per-URL singleflight cache for integrity-less tarballs. The first
102+
/// edge downloads and computes the integrity; later edges await the
103+
/// same cell instead of fetching the URL again.
104+
integrity_cache: Arc<DashMap<String, Arc<OnceCell<Integrity>>>>,
95105
}
96106

97107
/// Wraps an inner [`Resolver`] and, after each successful resolve that
@@ -146,10 +156,71 @@ impl<Reporter: self::Reporter + 'static> PrefetchingResolver<Reporter> {
146156
verify_store_integrity: config.verify_store_integrity,
147157
progress_reported: SharedReportedProgressKeys::clone(progress_reported),
148158
spawned_urls: Arc::new(DashSet::new()),
159+
integrity_cache: Arc::new(DashMap::new()),
149160
};
150161
PrefetchingResolver { inner, ctx, _phantom: PhantomData }
151162
}
152163

164+
/// Populate remote tarball resolutions whose integrity can only be
165+
/// learned from the downloaded bytes. `file:` and git-hosted tarballs
166+
/// are anchored by local bytes or a commit SHA and remain unchanged.
167+
async fn populate_missing_integrity(
168+
&self,
169+
result: &mut ResolveResult,
170+
) -> Result<(), ResolveError> {
171+
let LockfileResolution::Tarball(tarball) = &result.resolution else {
172+
return Ok(());
173+
};
174+
if tarball.integrity.is_some()
175+
// git-hosted tarballs are anchored by their commit SHA, not an integrity. Detect
176+
// them by URL, NOT by the `git_hosted` flag: the flag is tamper-prone lockfile
177+
// input, so trusting it would let a forged `git_hosted: true` on an arbitrary URL
178+
// skip the integrity computation. A real git-hosted archive (codeload/gitlab/
179+
// bitbucket) always has a matching URL.
180+
|| is_git_hosted_tarball_url(&tarball.tarball)
181+
|| tarball.tarball.starts_with("file:")
182+
{
183+
return Ok(());
184+
}
185+
let package_url = tarball.tarball.clone();
186+
// Scope credentials are selected from `name@version` when the
187+
// resolver knows it; direct URL tarballs fall back to URL identity.
188+
let package_id = result
189+
.name_ver
190+
.as_ref()
191+
.map_or_else(|| package_url.clone(), |nv| format!("{}@{}", nv.name, nv.suffix));
192+
193+
// Singleflight per URL: the same integrity-less tarball can arrive on many edges,
194+
// so compute its integrity once and share it. Clone the cell's `Arc` out of the map
195+
// before awaiting so the shard lock isn't held across the download.
196+
let cell = Arc::clone(&self.ctx.integrity_cache.entry(package_url.clone()).or_default());
197+
let integrity = cell
198+
.get_or_try_init(|| async {
199+
// This fetch warms the mem cache, so the prefetch path should not
200+
// spawn another task for the same URL.
201+
self.ctx.spawned_urls.insert(package_url.clone());
202+
let resolved = FetchTarballForResolution {
203+
http_client: &self.ctx.http_client,
204+
store_dir: self.ctx.store_dir,
205+
store_index_writer: self.ctx.store_index_writer.clone(),
206+
package_url: &package_url,
207+
package_id: &package_id,
208+
auth_headers: &self.ctx.auth_headers,
209+
retry_opts: self.ctx.retry_opts,
210+
}
211+
.run::<SilentReporter>(Some(&self.ctx.mem_cache))
212+
.await
213+
.map_err(|err| Box::new(err) as ResolveError)?;
214+
Ok::<_, ResolveError>(resolved.integrity)
215+
})
216+
.await?
217+
.clone();
218+
if let LockfileResolution::Tarball(tarball) = &mut result.resolution {
219+
tarball.integrity = Some(integrity);
220+
}
221+
Ok(())
222+
}
223+
153224
/// Inspect a fresh `ResolveResult` and, if it carries a tarball
154225
/// URL + integrity, kick off the download as a detached
155226
/// [`tokio::spawn`] task.
@@ -263,9 +334,10 @@ impl<Reporter: self::Reporter + 'static> Resolver for PrefetchingResolver<Report
263334
opts: &'a ResolveOptions,
264335
) -> ResolveFuture<'a> {
265336
Box::pin(async move {
266-
let result = self.inner.resolve(wanted_dependency, opts).await?;
267-
if let Some(result_ref) = result.as_ref() {
268-
self.maybe_kickoff_download(result_ref);
337+
let mut result = self.inner.resolve(wanted_dependency, opts).await?;
338+
if let Some(result_mut) = result.as_mut() {
339+
self.populate_missing_integrity(result_mut).await?;
340+
self.maybe_kickoff_download(result_mut);
269341
}
270342
Ok(result)
271343
})

‎pacquet/crates/resolving-tarball-resolver/src/tarball_resolver.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,6 +171,9 @@ impl TarballResolver {
171171
store_dir: ctx.store_dir,
172172
store_index_writer: ctx.store_index_writer.clone(),
173173
package_url: &resolved_url,
174+
// A direct https tarball has no resolver-known name@version, so the URL is the
175+
// only identifier; such tarballs carry no scoped-registry auth.
176+
package_id: &resolved_url,
174177
auth_headers: &ctx.auth_headers,
175178
retry_opts: ctx.retry_opts,
176179
}

0 commit comments

Comments
 (0)