protobuf: generate code using a script #2382

s-urbaniak · 2016-04-05T14:24:32Z

Currently protobuf bindings have to be generated manually, the process
is being documented as a code comment in api/v1alpha/api.proto.

This automates protobuf generation by introducing a script and
also a make target "protobuf".

Fixes #2004

s-urbaniak · 2016-04-05T14:25:15Z

@alban @jonboulle PTAL

alban · 2016-04-05T14:37:12Z

scripts/genproto.sh

+GO_PROTOBUF_SHA="dda510ac0fd43b39770f22ac6260eb91d377bce3"
+
+echo "installing golang/protobuf using GOPATH=${GOPATH}"
+go get -u github.com/golang/protobuf/{proto,protoc-gen-go}


Does it mean we need Internet connectivity to run this script? Most Linux distributions (CoreOS Linux, Debian, Fedora...) have guidelines against using internet during the build. This would prevent using this script during the build.

The build normally downloads the CoreOS PXE image for building stage1's "coreos" flavor. To follow distribution guidelines, a configure option has been added: ./configure --with-coreos-local-pxe-image-path. CoreOS Linux uses it. Fedora & Debian only use the host flavor.

Slightly orthogonal, since we're continuing to commit the generated code for now: #2004 (comment)

... and another reason to keep doing so

I am inclined even not to create an issue for that since it becomes more and more obvious to check in the generated code.

Yes this script is a mere helper to have a reproducible way of regenerating the code by pinning the version of golang/protobuf.

jonboulle · 2016-04-05T14:40:24Z

doh: https://travis-ci.org/coreos/rkt/jobs/120903030#L336

s-urbaniak · 2016-04-05T14:46:38Z

ugh, the failed travis build is reproducible with a clean directory, investigating.

alban · 2016-04-05T14:54:30Z

api/v1alpha/api.pb.go

 	// for the image as nowadays all the image formats will be transformed to
 	// ACI.
-	BaseFormat *ImageFormat `protobuf:"bytes,1,opt,name=base_format" json:"base_format,omitempty"`
+	BaseFormat *ImageFormat `protobuf:"bytes,1,opt,name=base_format,json=baseFormat" json:"base_format,omitempty"`


Where does this change come from?

Does it stay compatible with previous versions? I don't know the default case for the json= attribute.

That's what upstream https://github.com/golang/protobuf generates, I had no intuition to what version the generated code was pinned at.

Since travis failed I have one now, namely the version referenced in Godeps, pinning the script to the very same version now and see if the generated code is the same.

Currently protobuf bindings have to be generated manually, the process is being documented as a code comment in api/v1alpha/api.proto. This automates protobuf generation by introducing a script and also a make target "protobuf". Fixes rkt#2004

s-urbaniak · 2016-04-05T15:48:35Z

@alban travis seems to be happy now, I also catched up with #2377; PTAL

krnowak · 2016-04-05T15:52:30Z

scripts/genproto.sh

+GO_PROTOBUF_SHA="2402d76f3d41f928c7902a765dfc872356dd3aad"
+
+echo "installing golang/protobuf using GOPATH=${GOPATH}"
+go get -u github.com/golang/protobuf/{proto,protoc-gen-go}


Another solution we could use is to vendor them in rkt. Just saying.

doh, even better, you're right.

You can add protoc-gen-go in vendoredApps and follow the Managing Dependencies guide.

Good luck with Godeps ;)

You copied my script and removed my TODO without doing it? :P

double doh :-D

s-urbaniak · 2016-04-06T12:19:49Z

rkt/rkt.go

 	"github.com/coreos/rkt/rkt/config"
 	rktflag "github.com/coreos/rkt/rkt/flag"
 	"github.com/spf13/cobra"
+	_ "github.com/spf13/cobra/doc" // transitive dependency for man page generation


I don't like this but it makes godep not remove cobra/doc's transitive dependencies.

I think it is ok as long as the comment is here to explain.

It reminds me of this dummy package in old rkt versions:
https://github.com/coreos/rkt/blob/v0.11.0/stage1/dummy.go#L15
which was removed in favor of the file vendoredApps.

What version of godep are you using?

What about a longer comment explaining the details you told me OOB about the "build ignore"?

I use godep v60.

Wait, I found something, godep explicitely ignores +ignore tags, if I use +doc tags in the _gen.go files, godep is happy.

right, that's where I was going - let's do that!

On Wed, Apr 6, 2016 at 2:42 PM Sergiusz Urbaniak [email protected]
wrote:

In rkt/rkt.go
#2382 (comment):

@@ -27,6 +27,7 @@ import (
"github.com/coreos/rkt/rkt/config"
rktflag "github.com/coreos/rkt/rkt/flag"
"github.com/spf13/cobra"

_ "github.com/spf13/cobra/doc" // transitive dependency for man page generation

I use godep v60.

Wait, I found something, godep explicitely ignores +ignore tags, if I use
+doc tags in the _gen.go files, godep is happy.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/coreos/rkt/pull/2382/files/25f9cd2d2ca12f99738919fe4740e1c9823bba92..c62a59a4661fa1059bd696a17d4afbf84bdf0730#r58697141

(tag support landed relatively recently via
tools/godep#271 )

On Wed, Apr 6, 2016 at 2:44 PM Jonathan Boulle [email protected]
wrote:

right, that's where I was going - let's do that!

On Wed, Apr 6, 2016 at 2:42 PM Sergiusz Urbaniak [email protected]
wrote:

In rkt/rkt.go
#2382 (comment):

@@ -27,6 +27,7 @@ import (
"github.com/coreos/rkt/rkt/config"
rktflag "github.com/coreos/rkt/rkt/flag"
"github.com/spf13/cobra"

_ "github.com/spf13/cobra/doc" // transitive dependency for man page generation

I use godep v60.

Wait, I found something, godep explicitely ignores +ignore tags, if I
use +doc tags in the _gen.go files, godep is happy.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/coreos/rkt/pull/2382/files/25f9cd2d2ca12f99738919fe4740e1c9823bba92..c62a59a4661fa1059bd696a17d4afbf84bdf0730#r58697141

Or the vendoredApps route would be fine actually...

On Wed, Apr 6, 2016 at 2:44 PM Jonathan Boulle [email protected]
wrote:

(tag support landed relatively recently via
tools/godep#271 )

On Wed, Apr 6, 2016 at 2:44 PM Jonathan Boulle [email protected]
wrote:

right, that's where I was going - let's do that!

On Wed, Apr 6, 2016 at 2:42 PM Sergiusz Urbaniak <
[email protected]> wrote:

In rkt/rkt.go
#2382 (comment):

@@ -27,6 +27,7 @@ import (
"github.com/coreos/rkt/rkt/config"
rktflag "github.com/coreos/rkt/rkt/flag"
"github.com/spf13/cobra"

_ "github.com/spf13/cobra/doc" // transitive dependency for man page generation

I use godep v60.

Wait, I found something, godep explicitely ignores +ignore tags, if I
use +doc tags in the _gen.go files, godep is happy.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/coreos/rkt/pull/2382/files/25f9cd2d2ca12f99738919fe4740e1c9823bba92..c62a59a4661fa1059bd696a17d4afbf84bdf0730#r58697141

Spoke with @krnowak about the vendoredApps route; tldr: meant for apps only, but why not. I am fine with either approach.

s-urbaniak · 2016-04-06T12:21:57Z

@alban @jonboulle PTAL

jonboulle · 2016-04-06T12:34:08Z

1,812 additions, 0 deletions not shown because the diff is too large. Please use a local Git client to view these changes.

seriously? smdh

s-urbaniak · 2016-04-06T13:19:20Z

@alban @jonboulle PTAL

alban · 2016-04-06T13:38:58Z

LGTM

jonboulle · 2016-04-06T13:40:26Z

Wait, we have both vendoredApps and different build tags? Isn't that redundant?

jonboulle · 2016-04-06T13:40:45Z

er, never mind, I'm blind.

jonboulle · 2016-04-06T13:40:52Z

LGTM

alban added this to the v1.4.0 milestone Apr 5, 2016

alban added area/developer tooling needs/review labels Apr 5, 2016

alban reviewed Apr 5, 2016
View reviewed changes

s-urbaniak added 3 commits April 5, 2016 17:29

protobuf: generate code using a script

f0e8170

Currently protobuf bindings have to be generated manually, the process is being documented as a code comment in api/v1alpha/api.proto. This automates protobuf generation by introducing a script and also a make target "protobuf". Fixes rkt#2004

downgrade protobuf to the same version as in godeps

05f7817

protobuf: catchup with PR rkt#2377

040d497

s-urbaniak force-pushed the genproto branch from a4ce4ce to 040d497 Compare April 5, 2016 15:38

krnowak reviewed Apr 5, 2016
View reviewed changes

jonboulle added needs/some-changes and removed needs/review labels Apr 5, 2016

Sergiusz Urbaniak added 2 commits April 6, 2016 14:06

Godeps: add golang/protobuf/protoc-gen-go

25f9cd2

protobuf: use vendored protoc-gen-go

c62a59a

s-urbaniak reviewed Apr 6, 2016
View reviewed changes

godep: use +doc in favor of +ignore in *_gen.go

9f297e7

alban added reviewed/lgtm and removed needs/some-changes labels Apr 6, 2016

jonboulle merged commit 40b11c0 into rkt:master Apr 6, 2016

alban mentioned this pull request Apr 15, 2016

Doc: API Service: generate doc from proto #2434

Closed

s-urbaniak deleted the genproto branch April 20, 2016 06:59

Uh oh!

protobuf: generate code using a script #2382

protobuf: generate code using a script #2382

Uh oh!

Conversation

s-urbaniak commented Apr 5, 2016

Uh oh!

s-urbaniak commented Apr 5, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonboulle commented Apr 5, 2016

Uh oh!

s-urbaniak commented Apr 5, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

s-urbaniak commented Apr 5, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

s-urbaniak commented Apr 6, 2016

Uh oh!

jonboulle commented Apr 6, 2016

Uh oh!

s-urbaniak commented Apr 6, 2016

Uh oh!

alban commented Apr 6, 2016

Uh oh!

jonboulle commented Apr 6, 2016

Uh oh!

jonboulle commented Apr 6, 2016

Uh oh!

jonboulle commented Apr 6, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants