@genevieve and @ljfranklin, I thought that this had been solved already a few months back. Are you all experiencing it again? If so, does this resolve the error for you in your pipeline?

@jtarchie no, our initial pass a while ago didn't help resolve the issue. More context here: #240 (comment). We haven't got a chance to put any mileage on this in our CI yet, but we were able to reproduce the error with an acceptance test included in the PR.

Just pushed one last commit to add the same retry logic to upload-stemcell as larger stemcells like vSphere might exhibit the same EOF error. I think we're ready for a review on this.

Some additional notes:

• We initially tried to add a RetryClient or similar to handle this retrying transparently but this ended up being too difficult. Main problem was you have an io.Reader as the req.Body which can't be rewound in order to retry. We opted for adding the retry higher up in the command package and adding a Reset method on the formcontent struct.
• Had to submit a PR to the uilive library as calling Stop() quickly followed by another Start() would sometimes panic: Allow Start to be called immediately after Stop gosuri/uilive#19. This hasn't been merged but that library is pretty low traffic so pointing at our fork doesn't seem terrible for now.
• Had to touch a bunch of files in the api package to add errors.Wrap from github.com/pkg/errors. This allowed us to unwrap the error back to io.EOF back in the network package.
• Had one other bit of weirdness where something was calling req.Body.Close() twice on retry which caused our progress bar to freak out because we call bar.Start() when the Body is closed. Tried to figure out where the multiple closes were coming from but ended up sidestepping it by modifying the progress ReadCloser to no-op on subsequent Close calls.

adding a Reset method on the formcontent struct.

I wanted to add that a while back but keeping forget... Nice work!

But meanwhile it seems still worth to investigate supproting resumable upload. Adding something like this into OpsManager and om would probably better solve the problem than retrying uploading the complete payload several times.

@fredwangwang @jtarchie could y'all give this a look when you get a chance? I agree resumable uploads would be great but getting the OpsMgr nginx config updated and backported to all supported versions seems like a tall order at this time.

@genevieve @ljfranklin we merged this in, can y'all let us know if you continue to have any problems with this moving forward? thank you!