Thanks to visit codestin.com
Credit goes to github.com

Skip to content

feat(logging/s3): add --file-max-bytes flag #952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Integralist merged 1 commit into from
Jun 15, 2023
Merged

feat(logging/s3): add --file-max-bytes flag #952

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ require (
)

require (
github.com/fastly/go-fastly/v8 v8.5.0
github.com/fastly/go-fastly/v8 v8.5.1
github.com/kennygrant/sanitize v1.2.4
github.com/mholt/archiver v3.1.1+incompatible
github.com/otiai10/copy v1.11.0
Expand Down
2 changes: 2 additions & 0 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ github.com/dustinkirkland/golang-petname v0.0.0-20191129215211-8e5a1ed0cff0 h1:9
github.com/dustinkirkland/golang-petname v0.0.0-20191129215211-8e5a1ed0cff0/go.mod h1:V+Qd57rJe8gd4eiGzZyg4h54VLHmYVVw54iMnlAMrF8=
github.com/fastly/go-fastly/v8 v8.5.0 h1:34CZP7IIjPnP4gVJYvEnNj44uMvHJ6t3+qK/43ijEmc=
github.com/fastly/go-fastly/v8 v8.5.0/go.mod h1:m/QWKyZsuH8Ow3tDkRLqlbvbJdxalRGkTT3ZKxZJ4eo=
github.com/fastly/go-fastly/v8 v8.5.1 h1:5fBBlpN2tFbrK4zXHYic2oLcPvxFqEIB9l+leH5UqGU=
github.com/fastly/go-fastly/v8 v8.5.1/go.mod h1:m/QWKyZsuH8Ow3tDkRLqlbvbJdxalRGkTT3ZKxZJ4eo=
github.com/fastly/kingpin v2.1.12-0.20191105091915-95d230a53780+incompatible h1:FhrXlfhgGCS+uc6YwyiFUt04alnjpoX7vgDKJxS6Qbk=
github.com/fastly/kingpin v2.1.12-0.20191105091915-95d230a53780+incompatible/go.mod h1:U8UynVoU1SQaqD2I4ZqgYd5lx3A1ipQYn4aSt2Y5h6c=
github.com/fatih/color v1.15.0 h1:kOqh6YHBtK8aywxGerMG2Eq3H6Qgoqeo13Bk2Mv/nBs=
Expand Down
6 changes: 6 additions & 0 deletions pkg/commands/logging/s3/create.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ type CreateCommand struct {
CompressionCodec cmd.OptionalString
Domain cmd.OptionalString
EndpointName cmd.OptionalString // Can't shadow cmd.Base method Name().
FileMaxBytes cmd.OptionalInt
Format cmd.OptionalString
FormatVersion cmd.OptionalInt
GzipLevel cmd.OptionalInt
Expand Down Expand Up @@ -77,6 +78,7 @@ func NewCreateCommand(parent cmd.Registerer, g *global.Data, m manifest.Data) *C
c.CmdClause.Flag("bucket", "Your S3 bucket name").Action(c.BucketName.Set).StringVar(&c.BucketName.Value)
common.CompressionCodec(c.CmdClause, &c.CompressionCodec)
c.CmdClause.Flag("domain", "The domain of the S3 endpoint").Action(c.Domain.Set).StringVar(&c.Domain.Value)
c.CmdClause.Flag("file-max-bytes", "The maximum size of a log file in bytes").Action(c.FileMaxBytes.Set).IntVar(&c.FileMaxBytes.Value)
common.Format(c.CmdClause, &c.Format)
common.FormatVersion(c.CmdClause, &c.FormatVersion)
common.GzipLevel(c.CmdClause, &c.GzipLevel)
Expand Down Expand Up @@ -159,6 +161,10 @@ func (c *CreateCommand) ConstructInput(serviceID string, serviceVersion int) (*f
input.Domain = &c.Domain.Value
}

if c.FileMaxBytes.WasSet {
Copy link
Collaborator

@Integralist Integralist Jun 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in the related Terraform PR: is this safe to send 0 to the API?

If yes, then nothing more to do here.

If no, then we could either add && c.FileMaxBytes.Value > 0 to the condition, or check the value is == 0 within the if block and return an error notifying the user that the input was unexpected.

input.FileMaxBytes = &c.FileMaxBytes.Value
}

if c.Path.WasSet {
input.Path = &c.Path.Value
}
Expand Down
1 change: 1 addition & 0 deletions pkg/commands/logging/s3/describe.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ func (c *DescribeCommand) Exec(_ io.Reader, out io.Writer) error {
lines := text.Lines{
"Bucket": o.BucketName,
"Compression codec": o.CompressionCodec,
"File max bytes": o.FileMaxBytes,
Copy link
Collaborator

@Integralist Integralist Jun 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the API always return the file_max_bytes field?

If no, then I'm wondering if the go-fastly implementation will cause a zero value (i.e. 0) to be set on the FileMaxBytes response field. In that case we don't want to show File max bytes: 0 in the CLI output (as demonstrated in your test file changes) because that's misleading the user into thinking something has gone wrong and that they won't be able to have any logs uploaded to S3.

So in that case I think we should only print the values here (and in the list file below) if the value is > 0.

This value could be 0 because the API didn't return the field

Copy link
Collaborator

@Integralist Integralist Jun 15, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignore my comment. Based on your feedback in the associated Terraform PR it seems the API understands 0 to mean "no size limit" and that's fine as I'm going to presume the API will always send a value for this if it was set and if it's not set it would default to zero.

"Format version": o.FormatVersion,
"Format": o.Format,
"GZip level": o.GzipLevel,
Expand Down
1 change: 1 addition & 0 deletions pkg/commands/logging/s3/list.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ func (c *ListCommand) Exec(_ io.Reader, out io.Writer) error {
fmt.Fprintf(out, "\t\tRedundancy: %s\n", s3.Redundancy)
fmt.Fprintf(out, "\t\tServer-side encryption: %s\n", s3.ServerSideEncryption)
fmt.Fprintf(out, "\t\tServer-side encryption KMS key ID: %s\n", s3.ServerSideEncryption)
fmt.Fprintf(out, "\t\tFile max bytes: %d\n", s3.FileMaxBytes)
fmt.Fprintf(out, "\t\tCompression codec: %s\n", s3.CompressionCodec)
}
fmt.Fprintln(out)
Expand Down
4 changes: 4 additions & 0 deletions pkg/commands/logging/s3/s3_integration_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -379,6 +379,7 @@ func listS3sOK(i *fastly.ListS3sInput) ([]*fastly.S3, error) {
PublicKey: pgpPublicKey(),
ServerSideEncryption: "aws:kms",
ServerSideEncryptionKMSKeyID: "1234",
FileMaxBytes: 12345,
CompressionCodec: "zstd",
},
}, nil
Expand Down Expand Up @@ -421,6 +422,7 @@ Version: 1
Redundancy: standard
Server-side encryption: aws:kms
Server-side encryption KMS key ID: aws:kms
File max bytes: 0
Compression codec: zstd
S3 2/2
Service ID: 123
Expand All @@ -442,6 +444,7 @@ Version: 1
Redundancy: standard
Server-side encryption: aws:kms
Server-side encryption KMS key ID: aws:kms
File max bytes: 12345
Compression codec: zstd
`) + "\n\n"

Expand Down Expand Up @@ -478,6 +481,7 @@ var describeS3Output = "\n" + strings.TrimSpace(`
Access key: 1234
Bucket: my-logs
Compression codec: zstd
File max bytes: 0
Format: %h %l %u %t "%r" %>s %b
Format version: 2
GZip level: 0
Expand Down
6 changes: 6 additions & 0 deletions pkg/commands/logging/s3/update.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ type UpdateCommand struct {
Path cmd.OptionalString
Period cmd.OptionalInt
GzipLevel cmd.OptionalInt
FileMaxBytes cmd.OptionalInt
Format cmd.OptionalString
FormatVersion cmd.OptionalInt
MessageType cmd.OptionalString
Expand Down Expand Up @@ -75,6 +76,7 @@ func NewUpdateCommand(parent cmd.Registerer, g *global.Data, m manifest.Data) *U
c.CmdClause.Flag("bucket", "Your S3 bucket name").Action(c.BucketName.Set).StringVar(&c.BucketName.Value)
common.CompressionCodec(c.CmdClause, &c.CompressionCodec)
c.CmdClause.Flag("domain", "The domain of the S3 endpoint").Action(c.Domain.Set).StringVar(&c.Domain.Value)
c.CmdClause.Flag("file-max-bytes", "The maximum size of a log file in bytes").Action(c.FileMaxBytes.Set).IntVar(&c.FileMaxBytes.Value)
common.Format(c.CmdClause, &c.Format)
common.FormatVersion(c.CmdClause, &c.FormatVersion)
common.GzipLevel(c.CmdClause, &c.GzipLevel)
Expand Down Expand Up @@ -150,6 +152,10 @@ func (c *UpdateCommand) ConstructInput(serviceID string, serviceVersion int) (*f
input.GzipLevel = &c.GzipLevel.Value
}

if c.FileMaxBytes.WasSet {
input.FileMaxBytes = &c.FileMaxBytes.Value
}

if c.Format.WasSet {
input.Format = &c.Format.Value
}
Expand Down