chore(registry/storage/driver/s3-aws): refactor writer creation #4429

uhthomas · 2024-08-06T00:43:12Z

The logic is identical, but has been separated out and reorganised for clarity.

Whilst working on #4424, I found it really hard to understand the intention of what was happening, how and why. Hopefully this makes it more legible and easier to maintain.

uhthomas · 2024-08-06T00:56:51Z

Only concern I have is the use of key vs path (key := d.s3Path(path)). I can't see much consistency, though I imagine key is almost always correct to use.

uhthomas · 2024-08-06T10:09:10Z

Thanks for running the tests! As suspected, there was some stuff which had to be changed regarding key versus path. I've fixed that and have verified the failing test now works as expected locally.

uhthomas · 2024-08-06T11:20:29Z

Okay, I tried using key everywhere to avoid calling d.s3Path a bunch, but there are some edge cases and I'm not keen to change existing behavior. Should be good now.

milosgajdos

Thanks, very nice refactoring! I left some comments in line.

PTAL @Jamstah if you have any spare cycles 🙇‍♂️

milosgajdos · 2024-08-08T10:15:43Z

registry/storage/driver/s3-aws/s3.go

-			if key != *multi.Key {
-				continue
-			}
+func (d *driver) inProgressUpload(ctx context.Context, path string) (uploadID *string, err error) {


Please don't use named returns unless it significantly improves the code (it often doesn't and makes readers of the code miss important logic). Besides, you don't even take advantage of them through empty returns: all the returns in this func use explicit var names.

Named returns also serve as a form of documentation. The meaning of *string without the named return is unclear.

func (d *driver) inProgressUpload(ctx context.Context, path string) (uploadID *string, err error)

The meaning of *string without the named return is unclear.

Sure, but this is an unexported func, so one needs to read it anyway.

As I said, they generally make it "harder" to read the code, which is why we've been removing them from this codebase.

Do you feel in this case the named return parameter makes the code harder to understand? I'm not sure I see the benefit of:

func (d *driver) inProgressUpload(ctx context.Context, path string) (*string, error) { var ( uploadID *string empty bool )

compared to

func (d *driver) inProgressUpload(ctx context.Context, path string) (uploadID *string, err error) {

I think we could compare our opinions on code documentation using named returns all day -- @milosgajdos is stating that precedent has been set in the repo to not use named returns so it makes sense to just follow suite here.

milosgajdos · 2024-08-08T10:18:27Z

registry/storage/driver/s3-aws/s3.go

 		}
 	}
-	return nil, storagedriver.PathNotFoundError{Path: path}
+	return nil, nil


This rubs me the wrong way due to how I expect to consume return values from functions e.g. I would not expect the return value to be nil if the returned error is nil as well -- this often leads to very unexpected surprises for API consumers, especially when consuming unexported functions that often miss comments.

Do you have any other suggestions?

Is there a reason why we no longer return return nil, storagedriver.PathNotFoundError{Path: path} like the original code did?

If there is no upload found in path then it would maybe make sense to return it and let the consumer of the API decide what to do in such situation (we'd obviously add a code comment explaining what's what)

Hmm, yeah, sorry, I did have this in my original change but it somehow slipped when I switched it to using pages. I can fix that.

d29ef83

Have resolved it in the latest change. I still don't see many other ways of expressing this condition, it's sort of nice for the zero value to have meaning? An extra return value to indicate whether it was found or not just feels redundant when it would essentially just be the value of *uploadID != nil anyway.

Makes sense to me, thanks :) I'll add some comments.

So I'm wondering, if we should return something like this here instead of nil, nil @uhthomas. This will at least provide some signal to upstream API consumer of this function that the upload has not been found and prevent potential accidental nil memory access panics 🤷‍♂️

Suggested change

return nil, nil

return nil, storagedriver.Error{

DriverName: driverName,

Detail: fmt.Errorf("no in-progress upload found for empty file at path %s", path),

}

The nil upload ID is being used as a valid value. There is only one consumer of this function and it does not fail or propagate the error, instead, it will create a new upload instead. The Writer function would need to explicitly handle this error instead, and would be messy without something a bit more structured?

So instead of checking uploadID == nil it would need to instead check if the error is a storagedriver.Error and then check if err.Detail looks like "no in-progress upload found"? I'm not sure it even should be an error if there is no in progress upload, it's expected.

There is only one consumer of this function

At the moment. Technically, it's not an error per se, I agree, I'm just trying to be a good citizen to future generations 😄

registry/storage/driver/s3-aws/s3.go

milosgajdos · 2024-08-08T10:24:08Z

registry/storage/driver/s3-aws/s3.go

+	return nil, nil
+}
+
+func (d *driver) listParts(ctx context.Context, path string, uploadID *string) (parts []*s3.Part, err error) {


Similar to my previous comment regarding named returns: let's avoid them if possible; saving an extra line of slice var declaration is not worth it here IMHO.

milosgajdos · 2024-08-08T18:17:32Z

@uhthomas can you also please stop squashing while the PR is being reviewed? It's hard to track changes that way. We'll squash it before the MR is merged.

uhthomas · 2024-08-08T20:14:13Z

@uhthomas can you also please stop squashing while the PR is being reviewed? It's hard to track changes that way. We'll squash it before the MR is merged.

Sorry - it's what I'm used to at work and for other open source projects. I just amend my commits and force push rather than creating a long tail of commits. The diffs are still visible in the PR history if needed.

milosgajdos · 2024-08-09T10:16:10Z

Sorry - it's what I'm used to at work and for other open source projects

Sure, but please don't do it in this project. We can't be looking through the PR comment feed for commit history. The PR commit history tab is there for a reason.

The logic is identical, but has been separated out and reorganised for clarity. Signed-off-by: Thomas Way <[email protected]>

uhthomas · 2024-08-09T18:10:20Z

registry/storage/driver/s3-aws/s3.go

+		if uploadID != nil {
+			return uploadID, nil
 		}
+		return nil, storagedriver.PathNotFoundError{Path: path}


Just some questions about this behavior here - this is basically saying that if there is no multipart upload at the path, but there were results then a PathNotFoundError should be returned? Is it common for there to be multiple uploads at the path which don't exactly match the request key? Why is it fine if there aren't already in-progress uploads?

Should the logic change to:

if !empty && uploadID != nil

Maybe I'm missing something? If I am, then I'd like to document it!

Just some questions about this behavior here - this is basically saying that if there is no multipart upload at the path, but there were results then a PathNotFoundError should be returned?

IIRC this handles the situation where upload purging is cleaning up failed uploads 🤔

milosgajdos

LGTM. @thaJeztah @Jamstah PTAL 🙇‍♂️

uhthomas · 2024-08-09T21:24:27Z

Thank you again for your time :) I appreciate you reviewing this.

squizzi · 2024-09-10T19:28:27Z

registry/storage/driver/s3-aws/s3.go

+		}
+		return true
+	}); err != nil {
+		return nil, fmt.Errorf("list multipart uploads pages: %w", err)


Suggested change

return nil, fmt.Errorf("list multipart uploads pages: %w", err)

return nil, fmt.Errorf("failed to list multipart uploads pages: %w", err)

squizzi · 2024-09-10T19:36:29Z

registry/storage/driver/s3-aws/s3.go

-			if key != *multi.Key {
-				continue
-			}
+func (d *driver) inProgressUpload(ctx context.Context, path string) (uploadID *string, err error) {


I think we could compare our opinions on code documentation using named returns all day -- @milosgajdos is stating that precedent has been set in the repo to not use named returns so it makes sense to just follow suite here.

github-actions bot added area/storage area/storage/s3 labels Aug 6, 2024

uhthomas force-pushed the chore-s3-refactor-writer branch from 5ae4619 to cadfbea Compare August 6, 2024 10:08

uhthomas force-pushed the chore-s3-refactor-writer branch 4 times, most recently from 9aeacd6 to d29ef83 Compare August 6, 2024 11:19

uhthomas force-pushed the chore-s3-refactor-writer branch 3 times, most recently from 3909019 to d3462c6 Compare August 6, 2024 17:36

milosgajdos reviewed Aug 8, 2024

View reviewed changes

uhthomas force-pushed the chore-s3-refactor-writer branch from d3462c6 to d225efe Compare August 8, 2024 14:51

uhthomas force-pushed the chore-s3-refactor-writer branch from d225efe to 74a1a6b Compare August 8, 2024 20:02

milosgajdos requested review from Jamstah and thaJeztah August 9, 2024 16:33

chore(registry/storage/driver/s3-aws): refactor writer creation

052404a

The logic is identical, but has been separated out and reorganised for clarity. Signed-off-by: Thomas Way <[email protected]>

uhthomas force-pushed the chore-s3-refactor-writer branch from 74a1a6b to 052404a Compare August 9, 2024 18:06

uhthomas commented Aug 9, 2024

View reviewed changes

milosgajdos approved these changes Aug 9, 2024

View reviewed changes

milosgajdos requested a review from squizzi August 9, 2024 18:33

milosgajdos requested a review from corhere August 14, 2024 15:11

squizzi reviewed Sep 10, 2024

View reviewed changes

milosgajdos added the refactor label Dec 17, 2024

-	return nil, nil
+	return nil, storagedriver.Error{
+		DriverName: driverName,
+		Detail:     fmt.Errorf("no in-progress upload found for empty file at path %s", path),
+	}

	return nil, fmt.Errorf("list multipart uploads pages: %w", err)
	return nil, fmt.Errorf("failed to list multipart uploads pages: %w", err)

chore(registry/storage/driver/s3-aws): refactor writer creation #4429

Are you sure you want to change the base?

chore(registry/storage/driver/s3-aws): refactor writer creation #4429

Uh oh!

Conversation

uhthomas commented Aug 6, 2024

Uh oh!

uhthomas commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uhthomas commented Aug 6, 2024

Uh oh!

uhthomas commented Aug 6, 2024

Uh oh!

milosgajdos left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uhthomas Aug 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milosgajdos Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

uhthomas Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milosgajdos commented Aug 8, 2024

Uh oh!

uhthomas commented Aug 8, 2024

Uh oh!

milosgajdos commented Aug 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

milosgajdos left a comment

Choose a reason for hiding this comment

Uh oh!

uhthomas commented Aug 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

uhthomas commented Aug 6, 2024 •

edited

Loading

uhthomas Aug 8, 2024 •

edited

Loading

milosgajdos Aug 29, 2024 •

edited

Loading

uhthomas Aug 29, 2024 •

edited

Loading