In the case of AWS S3 and Azure, we set the cleanOnError to false, and for all other cases, we set the cleanOnError to true.

I thought it was because it wasn't implemented in the zedUpload library... I don't see any code to resume download in the HTTP lib there...

In the case of AWS S3 and Azure, we set the cleanOnError to false, and for all other cases, we set the cleanOnError to true.

I thought it was because it wasn't implemented in the zedUpload library... I don't see any code to resume download in the HTTP lib there...

In eve-libs, the downloader periodically records progress (the doneParts) and reports it to EVE for all datastore types—not just AWS and Azure. EVE then persists this progress. On restart or crash recovery, the downloader reads the saved doneParts and resumes from the last completed ranges. Therefore, unless there is a bug in how doneParts is produced, resumable downloads should for all datastore types.

You are right that this PR does not add new functionality for download resumability (it was already present). But we will support resuming downloads from HTTP/HTTPS datastores. We had a variable called cleanOnError which prevented the resumability if it was set to true. In the case of AWS S3 and Azure, we set the cleanOnError to false, and for all other cases, we set the cleanOnError to true. That is why the downloader resumed only when the datastore was AWS S3 or Azure.

It would make sense to update the release notes part of the description with that context (that it makes it more robust for S3, etc).

In terms of https:
Does the http/https downloader have the ability to download in chunks from a specified offset?
And do all of the https servers support this?
It might make sense to explore this a bit outside of this PR to see whether we can enable the resume for it.

Note that if a file changes or is deleted on the server (whether it is S3, Azure, or https) the result might be a failure of the interrupted and resumed download, or it will complete and the sha verification will fail. I think we have added code to automatically retry even when the sha verification fails, but it would make sense to check that as well.

On restart or crash recovery, the downloader reads the saved doneParts and resumes from the last completed ranges

I don't see the HTTP library handling any doneParts or similar entities.
https://github.com/lf-edge/eve-libs/blob/main/zedUpload/datastore_http.go

So, my understanding is that HTTP datastores do not support it in our case until we implement it in zedUpload.

On restart or crash recovery, the downloader reads the saved doneParts and resumes from the last completed ranges

I don't see the HTTP library handling any doneParts or similar entities. https://github.com/lf-edge/eve-libs/blob/main/zedUpload/datastore_http.go

So, my understanding is that HTTP datastores do not support it in our case until we implement it in zedUpload.

You are right @OhmSpectator, I got confused due to the function statsUpdater in the following code:

func (ep *HttpTransportMethod) processHttpDownload(req *DronaRequest) (error, int) {
	file := req.name
	if ep.hurl != "" {
		file = ep.hurl + "/" + ep.path + "/" + req.name
	}
	prgChan := make(types.StatsNotifChan)
	defer close(prgChan)
	if req.ackback {
		go statsUpdater(req, ep.ctx, prgChan)
	}
	hClient, err := ep.hClientWrap.unwrap()
	if err != nil {
		return err, 0
	}
	stats, resp := zedHttp.ExecCmd(req.cancelContext, "get", file, "",
		req.objloc, req.sizelimit, prgChan, hClient, ep.inactivityTimeout)
	return stats.Error, resp.BodyLength
}

func reqPostSize(req *DronaRequest, dronaCtx *DronaCtx, stats types.UpdateStats) {
	req.doneParts = stats.DoneParts
	dronaCtx.postSize(req, stats.Size, stats.Asize)
}

func statsUpdater(req *DronaRequest, dronaCtx *DronaCtx, prgNotif types.StatsNotifChan) {
	ticker := time.NewTicker(StatsUpdateTicker)
	defer ticker.Stop()
	var newStats, stats types.UpdateStats
	var ok bool
	for {
		select {
		case newStats, ok = <-prgNotif:
			if !ok {
				reqPostSize(req, dronaCtx, stats)
				return
			}
			stats = newStats
		case <-ticker.C:
			reqPostSize(req, dronaCtx, stats)
		}
	}
}

The function updates the doneParts of the drona request with the doneParts of the stats variable. However, in the case of http datastore we do not update the doneParts of the stat variable:

for {
	var copyErr error

	written, copyErr = io.CopyN(local, inactivityReader, chunkSize)
	copiedSize += written
	stats.Asize = copiedSize

	// possible situations:
	// err != nil && err == io.EOF - end of file, wrap up and return
	// err != nil && err == inactivityTimeout - begin a retry
	// err != nil - wrap up and return
	// err == nil - update stats and keep reading
	switch {
	case copyErr != nil && errors.Is(copyErr, io.EOF) && copiedSize != objSize && objSize != 0:
		appendToErrorList("premature EOF after %d out of %d bytes: %+v", copiedSize, objSize, copyErr)
		return stats, rsp
	case copyErr != nil && errors.Is(copyErr, io.EOF):
		// empty out the error list
		errorList = nil
		return stats, rsp
	case copyErr != nil && errors.Is(copyErr, &ErrTimeout{}):
		// the error comes from timeout
		appendToErrorList("inactivity for %s", inactivityTimeout)
	case copyErr != nil:
		appendToErrorList("error from CopyN after %d out of %d bytes: %v", copiedSize, objSize, copyErr)
		return stats, rsp
	default:
		// no error, so just continue
		types.SendStats(prgNotify, stats)
		continue
	}
	// every other case either returns or continues; if we made it here,
	// break io.CopyN loop, forcing a retry of the outer loop
	break
}

We only update the Asize of the stats, not the doneParts.

We need to make two changes: first update the doneParts of the stats, e.g.:

stats.Asize = copiedSize
if written > 0 {
    // adapt to your actual Part struct (Start/End or Offset/Size)
    stats.DoneParts = types.DownloadedParts{
        PartSize: copiedSize, // or your configured chunk size
        Parts:    []*types.PartDefinition{{Ind: i, Size: written}},
    }
}

and second start from the saved doneParts in the local file and add a range in the header of the request.

@jsfakian @OhmSpectator I took the liberty to add a pre-amble sentence to the Changelog notes
section. Please take a look and tweak it as needed.

@jsfakian @OhmSpectator I took the liberty to add a pre-amble sentence to the Changelog notes section. Please take a look and tweak it as needed.

LGTM

@jsfakian @OhmSpectator I took the liberty to add a pre-amble sentence to the Changelog notes section. Please take a look and tweak it as needed.

LGTM

I would be more specific

some datastores types (S3, Azure, ...)

@jsfakian The eden eve update (zfs) is failing and the log shows
"{"file":"/pillar/cmd/volumemgr/updatestatus.go:216","func":"github.com/lf-edge/eve/pkg/pillar/cmd/volumemgr.doUpdateContentTree","level":"error","msg":"doUpdateContentTree: BlobStatus(e7bf444fec8a84cbbb56f63c6199c4ea4c62a308394587b2a54de3e237532ba5) has error: open /persist/vault/downloader/pending/d78fe4a159e014bdce546470c5d557406e38ac3b08947d746265009317b87356.e7bf444fec8a84cbbb56f63c6199c4ea4c62a308394587b2a54de3e237532ba5.part: no such file or directory\n","pid":2520,"source":"volumemgr","time":"2025-10-02T15:29:05.292749332Z"}",

".part" sounds like it might be related to this PR. Can you take a look?

@jsfakian The eden eve update (zfs) is failing and the log shows "{"file":"/pillar/cmd/volumemgr/updatestatus.go:216","func":"github.com/lf-edge/eve/pkg/pillar/cmd/volumemgr.doUpdateContentTree","level":"error","msg":"doUpdateContentTree: BlobStatus(e7bf444fec8a84cbbb56f63c6199c4ea4c62a308394587b2a54de3e237532ba5) has error: open /persist/vault/downloader/pending/d78fe4a159e014bdce546470c5d557406e38ac3b08947d746265009317b87356.e7bf444fec8a84cbbb56f63c6199c4ea4c62a308394587b2a54de3e237532ba5.part: no such file or directory\n","pid":2520,"source":"volumemgr","time":"2025-10-02T15:29:05.292749332Z"}",

".part" sounds like it might be related to this PR. Can you take a look?

This is strange, I run the EDEN test for EVE upgrade zfs locally, and it works. One line that might be suspicious is the following:

f, ferr := os.OpenFile(tempLocFilename, os.O_RDWR, 0)

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 19.50%. Comparing base (d1b24f1) to head (fda7cb3).
⚠️ Report is 41 commits behind head on master.

@@            Coverage Diff             @@
##           master    #5254      +/-   ##
==========================================
+ Coverage   18.52%   19.50%   +0.98%     
==========================================
  Files          19       19              
  Lines        2705     3025     +320     
==========================================
+ Hits          501      590      +89     
- Misses       2120     2314     +194     
- Partials       84      121      +37     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jsfakian The eden eve update (zfs) is failing and the log shows "{"file":"/pillar/cmd/volumemgr/updatestatus.go:216","func":"github.com/lf-edge/eve/pkg/pillar/cmd/volumemgr.doUpdateContentTree","level":"error","msg":"doUpdateContentTree: BlobStatus(e7bf444fec8a84cbbb56f63c6199c4ea4c62a308394587b2a54de3e237532ba5) has error: open /persist/vault/downloader/pending/d78fe4a159e014bdce546470c5d557406e38ac3b08947d746265009317b87356.e7bf444fec8a84cbbb56f63c6199c4ea4c62a308394587b2a54de3e237532ba5.part: no such file or directory\n","pid":2520,"source":"volumemgr","time":"2025-10-02T15:29:05.292749332Z"}",

".part" sounds like it might be related to this PR. Can you take a look?

I have verified that this error comes from the following code:

// Finalize: fsync and atomically move tempLocFilename -> locFilename.
		f, ferr := os.OpenFile(tmpLocFilename, os.O_RDWR, 0644)
		if ferr != nil {
			log.Errorf("Failed to open file %s: %v", tmpLocFilename, ferr)
			return handleSyncOpResponse(ctx, config, status, locFilename,

But I cannot think why the tmpLocFilename might not be present.

@jsfakian The eden eve update (zfs) is failing and the log shows "{"file":"/pillar/cmd/volumemgr/updatestatus.go:216","func":"github.com/lf-edge/eve/pkg/pillar/cmd/volumemgr.doUpdateContentTree","level":"error","msg":"doUpdateContentTree: BlobStatus(e7bf444fec8a84cbbb56f63c6199c4ea4c62a308394587b2a54de3e237532ba5) has error: open /persist/vault/downloader/pending/d78fe4a159e014bdce546470c5d557406e38ac3b08947d746265009317b87356.e7bf444fec8a84cbbb56f63c6199c4ea4c62a308394587b2a54de3e237532ba5.part: no such file or directory\n","pid":2520,"source":"volumemgr","time":"2025-10-02T15:29:05.292749332Z"}",

".part" sounds like it might be related to this PR. Can you take a look?

I removed the part where we use the .tmp filename, and now the tests for the upgrade of ZFS and EXT4 have passed. Are we safe to merge?