Add Consul support #31622

soupdiver · 2016-08-29T16:46:17Z

An earlier discussion about Consul integration can be found here: #28508

Consul support integration

We worked on getting Consul support into Kubernetes. Our implementation is based on Consul 0.6.4 and should provide the same featureset as the etcd implementation does.
To ensure correct working of the implementation we wrote integration tests which are very similar to the ones for etcd. These can be found at `test/integration/master/consul_tools_test.go
However the implementation is not finished yet. Herein after I will address the remaining issue.

unit tests

Currently unit tests, which need to talk to a storage backend, are using the etcd implementation directly (example).
This means the tests can't directly be reused with Consul. We tried to solve this problem by refactoring all the etcd specific tests to more generic storage tests and also using a factory to create all the available storage backends and run the unit tests against each backend. In the end we had to drop this change because we could not keep up with the frequency of changes in these locations. So for now unit tests don't run against Consul.
We think this is probably a good discussion point on how to handle multiple storage backends.
This could be handled in a separate PR to keep the scope smaller for each one.

Our test implementation

In the following are snippets showing how our approach to run the tests for multiple backends looks like.

Running tests and creating storage

var factory storagefactory.TestServerFactory

func TestMain(m *testing.M) {
    storagefactory.RunTestsForStorageFactories(func(fac storagefactory.TestServerFactory) int {
        factory = fac
        return m.Run()
    })
}

func newStorage(t *testing.T) (*DeploymentStorage, storagefactory.TestClientServer) {
    internalStorage, server := registrytest.NewStorage(t, factory, extensions.GroupName)
    restOptions := generic.RESTOptions{Storage: internalStorage, Decorator: generic.UndecoratedStorage, DeleteCollectionWorkers: 1}
    deploymentStorage := NewStorage(restOptions)
    return &deploymentStorage, server
}

factory

func RunTestsForStorageFactories(iterFn func(TestServerFactory) int) {
    FactoryTypesList := os.Getenv("KUBE_STORAGE")
    if FactoryTypesList == "" {
        FactoryTypesList = "etcd2,consul"
    }
    types := strings.Split(FactoryTypesList, ",")
    retCodes := make([]int, len(types))
    for index, FactoryType := range types {
        factory, err := GetFactory(FactoryType)
        if err != nil {
            fmt.Printf("Skipping tests for %s because %v", FactoryType, err)
            retCodes[index] = 1
            continue 
        }
        fmt.Printf("Running tests in %s mode\n", factory.GetName())
        retCodes[index] = iterFn(factory)
    }
    for _, code := range retCodes {
        if code > 0 {
            os.Exit(code)
        }
    }
    os.Exit(0)
}

// Create creates a storage backend based on given config.
func GetFactory(Type string) (TestServerFactory, error) {
    switch Type {
    case storagebackend.StorageTypeUnset, storagebackend.StorageTypeETCD2:
        return NewETCD2TestServerFactory()
    //case storagebackend.StorageTypeETCD3:
        // TODO: We have the following features to implement:
        // - Support secure connection by using key, cert, and CA files.
        // - Honor "https" scheme to support secure connection in gRPC.
        // - Support non-quorum read.
        //return newETCD3Storage(c, codec)
    case storagebackend.StorageTypeConsul:
        return NewConsulSharedTestServerFactory()
    default:
        return nil, fmt.Errorf("unknown storage type: %s", Type)
    }
}

This change is

googlebot · 2016-08-29T16:46:19Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please let us know the company's name.

k8s-bot · 2016-08-29T16:46:31Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

k8s-bot · 2016-08-29T16:46:48Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

k8s-bot · 2016-08-29T16:47:06Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

k8s-bot · 2016-08-29T16:47:24Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

k8s-bot · 2016-08-29T16:47:41Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

k8s-bot · 2016-08-29T16:47:58Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

k8s-bot · 2016-08-29T16:48:20Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

soupdiver · 2016-08-29T16:51:27Z

I signed it!

mikejihbe · 2016-08-29T16:54:56Z

@soupdiver is now in our google group, so he should fall under MustWin's CLA

k8s-bot · 2016-08-29T23:25:23Z

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

lavalamp · 2016-08-29T23:45:02Z

ok to test

On Mon, Aug 29, 2016 at 4:25 PM, Kubernetes Bot [email protected]
wrote:

Can one of the admins verify that this patch is reasonable to test? If so,
please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update
configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-
jenkins-pull
https://github.com/kubernetes/test-infra/tree/master/jenkins/job-configs/kubernetes-jenkins-pull
instead.)

This message will repeat several times in short succession due to
jenkinsci/ghprb-plugin#292
jenkinsci/ghprb-plugin#292. Sorry.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#31622 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAnglsOxJIxTpiQ4EGUuv3nQLI5_49Xnks5qk2pugaJpZM4Jvqd9
.

soupdiver · 2016-08-30T08:28:54Z

I currently have a problem with adding dependencies properly.
Consul relies on a repository which is not available on github anymore.
See: https://github.com/hashicorp/consul/blob/26a0ef8c41aa2252ab4cf0844fc6470c8e1d8256/Godeps/Godeps.json#L6

It is properly vendored but when I execute ./hack/godep-save.sh it fails and complains that the repository can not be found.

How to deal with this scenario?

soupdiver · 2016-08-30T14:56:52Z

rebased

bgrant0607 · 2017-06-14T07:02:05Z

@soupdiver

The current project priorities are project health and stability:
https://groups.google.com/forum/#!topic/kubernetes-dev/PpgLgkffr3o

In addition to the costs and challenges of reviewing this PR, exposing this interface potentially incurs significant cost and risk to the project, so it would need to be considered carefully.

We have been focused on the transition to etcd3, which improves reliability and scalability:
#22448

etcd3 also provides features we requested here, including transactions:
#1957 (comment)

We're also moving a lot of code around. In particular, the apiserver and other API machinery needs to be made available to other repos, so the storage-related code is moving out.

Additionally, we're trying to craft extension points now such that extensions don't need to reside in our repos. Could an adapter API for consul be built instead, along the lines of https://github.com/coreos/zetcd?

We do need to figure out how to expose a storage API for extension apiservers
(#46351), but that hasn't been designed yet.

bgrant0607 · 2017-06-14T07:05:17Z

I see now that there is also a proposal to use etcd in mantl via zetcd:
mantl/mantl#2013

thomasvincent · 2017-06-14T08:30:47Z

@slackpad, @Theaxiom and @soupdiver it looks like we have feedback on a direction to take this. Thanks, @bgrant0607 will take a look at your suggestion and get back to you.

k8s-github-robot · 2017-06-23T00:41:56Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: soupdiver
We suggest the following additional approver: bgrant0607

Assign the PR to them by writing /assign @bgrant0607 in a comment when ready.

Associated issue: 28508

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

OWNERS

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-06-23T18:29:44Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: soupdiver
We suggest the following additional approver: bgrant0607

Assign the PR to them by writing /assign @bgrant0607 in a comment when ready.

Associated issue: 28508

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

OWNERS

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-06-23T22:29:53Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: soupdiver
We suggest the following additional approver: bgrant0607

Assign the PR to them by writing /assign @bgrant0607 in a comment when ready.

Associated issue: 28508

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

OWNERS

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-06-27T00:16:22Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: soupdiver
We suggest the following additional approver: bgrant0607

Assign the PR to them by writing /assign @bgrant0607 in a comment when ready.

Associated issue: 28508

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

OWNERS

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-06-28T20:40:11Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: soupdiver
We suggest the following additional approver: bgrant0607

Assign the PR to them by writing /assign @bgrant0607 in a comment when ready.

Associated issue: 28508

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

OWNERS

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-08-18T08:01:09Z

This PR hasn't been active in 64 days. It will be closed in 25 days (Sep 12, 2017).

cc @bgrant0607 @soupdiver

You can add 'keep-open' label to prevent this from happening, or add a comment to keep it open another 90 days

k8s-github-robot · 2017-08-30T21:43:35Z

Adding do-not-merge/release-note-label-needed because the release note process has not been followed.
One of the following labels is required "release-note", "release-note-action-required", "release-note-experimental" or "release-note-none".
Please see: https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md#write-release-notes-if-needed.

bgrant0607 · 2017-09-06T20:39:23Z

We're not prepared to move forward with this in the foreseeable future.

We need to think hard about what we want the storage API to look like, for pluggability but also for aggregated APIs.

We're still in the middle of the etcd2-to-etcd3 transition.

We can't afford to increase the dimensionality of our test and support matrix.

Etcd3 also added a number of features for us that we have yet to take advantage of, such as atomic transactions across multiple keys/values to give us more flexibility in how we represent resources.

Shims such as proposed by mantl/mantl#2013 could potentially provide a way to use other stores.

cc @lavalamp @kubernetes/sig-api-machinery-pr-reviews @kubernetes/sig-api-machinery-proposals

lavalamp · 2017-09-07T16:27:24Z

A few days ago I serendipitously added an agenda item for the next API machinery SIG meeting (Sep. 13th) to discuss making a more firm policy about this. If anyone wants to hear more reasoning behind Brian's above decision, it might be good to attend.

In my opinion we've waffled about this for too long, we should commit to speaking only etcd3 for the foreseeable future.

What if someone REALLLLLLLY has to run against another storage system? Etcd3 is open source. Fork it. Leave the interface, turn everything else into a shim that talks to your preferred store. Realize that you're signing up for a lifetime of pain maintaining 100% compatibility... (That would have been the case no matter how such a thing was implemented; but doing it as I suggest here makes it clear that the compatibility burden won't be born by the Kubernetes project.)

deads2k · 2017-09-07T20:09:21Z

What if someone REALLLLLLLY has to run against another storage system? Etcd3 is open source. Fork it. Leave the interface, turn everything else into a shim that talks to your preferred store. Realize that you're signing up for a lifetime of pain maintaining 100% compatibility... (That would have been the case no matter how such a thing was implemented; but doing it as I suggest here makes it clear that the compatibility burden won't be born by the Kubernetes project.)

Such a view is difficult to reconcile with "and now we will add a different kind of storage for aggregated API servers". If a different blob store is still under consideration, I'd be hard pressed to say, "we won't accept another... except for ours.". To properly support a different storage (blob store you've mentioned before), you'll need to separate an interface anyway.

bgrant0607 · 2017-09-07T20:18:16Z

@deads2k Let's please sort that out in #46351

chrrrles · 2017-11-07T07:06:50Z

@soupdiver - I will have a look at this PR.

@bgrant0607 @deads2k @lavalamp - Are there still any open PRs or proposals for a pluggable kv store? Interested in following this discussion moving forward.

Theaxiom · 2017-11-07T07:23:08Z

@chrrrles thank you!

bgrant0607 · 2017-11-07T07:32:03Z

@chrrrles No. It's off the table for now.

thomasvincent · 2017-11-07T07:33:13Z

Understood, you guys have lots on your plate.

axi92 · 2019-09-06T05:56:37Z

Is there anything new on that topic?

soupdiver · 2019-09-06T07:21:00Z

Is there anything new on that topic?

For complexity reasons, I don't think there is a plan to officially support Consul anytime (soon).

googlebot added the cla: no label Aug 29, 2016

mikejihbe mentioned this pull request Aug 29, 2016

docs: proposals: KV Support for Consul #28508

Closed

k8s-github-robot assigned lavalamp Aug 29, 2016

k8s-github-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. release-note-label-needed labels Aug 29, 2016

soupdiver force-pushed the consul-storage-integration branch 2 times, most recently from 1c7f9c1 to 2568ec7 Compare August 30, 2016 08:25

soupdiver force-pushed the consul-storage-integration branch 2 times, most recently from 7b855cd to 89d8e63 Compare August 30, 2016 10:05

k8s-github-robot mentioned this pull request Aug 30, 2016

[k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite} #30548

Closed

soupdiver force-pushed the consul-storage-integration branch from 89d8e63 to a933c18 Compare August 30, 2016 11:42

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 30, 2016

soupdiver force-pushed the consul-storage-integration branch from b7805b2 to e0fbf5a Compare August 30, 2016 14:56

soupdiver force-pushed the consul-storage-integration branch 3 times, most recently from 1a5e768 to 0e2f64f Compare August 30, 2016 16:20

bgrant0607 assigned bgrant0607 and unassigned lavalamp, brendandburns and wojtek-t Jun 14, 2017

cjwagner mentioned this pull request Jun 28, 2017

Fix approval notification matching bug. kubernetes/test-infra#3252

Merged

k8s-github-robot added the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Aug 30, 2017

bgrant0607 closed this Sep 6, 2017

k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. kind/design Categorizes issue or PR as related to design. labels Sep 6, 2017

lavalamp mentioned this pull request Sep 28, 2017

DynamoDb Support for API Server #53162

Closed

Add Consul support #31622

Add Consul support #31622

Uh oh!

Conversation

soupdiver commented Aug 29, 2016 • edited by k8s-oncall Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Consul support integration

unit tests

Our test implementation

Running tests and creating storage

factory

Uh oh!

googlebot commented Aug 29, 2016

Uh oh!

k8s-bot commented Aug 29, 2016

Uh oh!

k8s-bot commented Aug 29, 2016

Uh oh!

k8s-bot commented Aug 29, 2016

Uh oh!

k8s-bot commented Aug 29, 2016

Uh oh!

k8s-bot commented Aug 29, 2016

Uh oh!

k8s-bot commented Aug 29, 2016

Uh oh!

k8s-bot commented Aug 29, 2016

Uh oh!

soupdiver commented Aug 29, 2016

Uh oh!

mikejihbe commented Aug 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-bot commented Aug 29, 2016

Uh oh!

lavalamp commented Aug 29, 2016

Uh oh!

soupdiver commented Aug 30, 2016

Uh oh!

soupdiver commented Aug 30, 2016

Uh oh!

bgrant0607 commented Jun 14, 2017

Uh oh!

bgrant0607 commented Jun 14, 2017

Uh oh!

thomasvincent commented Jun 14, 2017

Uh oh!

k8s-github-robot commented Jun 23, 2017

Uh oh!

k8s-github-robot commented Jun 23, 2017

Uh oh!

k8s-github-robot commented Jun 23, 2017

Uh oh!

k8s-github-robot commented Jun 27, 2017

Uh oh!

k8s-github-robot commented Jun 28, 2017

Uh oh!

k8s-github-robot commented Aug 18, 2017

Uh oh!

k8s-github-robot commented Aug 30, 2017

Uh oh!

bgrant0607 commented Sep 6, 2017

Uh oh!

lavalamp commented Sep 7, 2017

Uh oh!

deads2k commented Sep 7, 2017

Uh oh!

bgrant0607 commented Sep 7, 2017

Uh oh!

chrrrles commented Nov 7, 2017

Uh oh!

Theaxiom commented Nov 7, 2017

Uh oh!

bgrant0607 commented Nov 7, 2017

Uh oh!

thomasvincent commented Nov 7, 2017

Uh oh!

axi92 commented Sep 6, 2019

Uh oh!

soupdiver commented Sep 6, 2019

soupdiver commented Aug 29, 2016 •

edited by k8s-oncall

Loading

mikejihbe commented Aug 29, 2016 •

edited

Loading