Thanks to visit codestin.com
Credit goes to buf.build

Skip to content

On-premise deployment with Kubernetes#

A self-hosted BSR runs on Kubernetes, distributed as a Helm chart with accompanying Docker images served from an OCI registry. The chart and the Docker images are versioned together, and the default values in the chart pull images that match the chart’s own version.

For the architecture and the services this page configures, see Architecture. For the upgrade procedure, see Upgrade and downgrade.

Prerequisites#

  1. Review the list of BSR dependencies before getting started.
  2. Ensure you have a license to operate the BSR. The license file should contain buflicense- followed by a base64 encoded string. License files are sent to customers during the onboarding process.
  3. Review the resource requirements listed below, ensuring you have an appropriately sized Kubernetes cluster for deployment.

Resource requirements#

The default resource limits and requests in the Helm chart work for the majority of customers; the typical deployment fits comfortably in 8 vCPUs and 32 GiB of memory. Apply standard high-availability practices: spread deployments across multiple nodes and availability zones.

Major dependencies:

  • 1x PostgreSQL (version 14+) instance with 4 CPU, 4 GiB memory, 10 GiB SSD disk. We recommend a high availability deployment. This stores application data such as users, organizations, repositories, and commit references
  • 1x Redis (version 6.2+, not Redis Cluster) instance with 1 GiB memory. Redis caches compiled modules and other performance-enhancing data.
  • S3-compatible storage or Azure Blob Storage. These usually don’t require pre-provisioning. The average BSR deployment uses less than 500GiB. Blob storage is used to persist BSR modules as binary representations of a set of Protobuf files.

The estimated monthly cost with this minimum configuration varies by cloud provider, but should be less than $1,000/month at list rates (not including any discounts). Pricing estimates are available for GCP, AWS, and Azure.

Create a namespace#

Create a Kubernetes namespace in the k8s cluster for the bsr Helm Chart to use:

$ kubectl create namespace bsr

Configure the BSR’s Helm values#

The BSR is configured using Helm values through the bsr Helm Chart.

Create a file named bsr.yaml to store the Helm values, which is required by the helm install step below. This file can be in any location, but we recommend creating it in the same directory where the Helm commands are run.

Set the desired host and configure the chart to use the license provided to you by Buf:

host: example.com # Hostname that the BSR will be served from
license:
  key: "buflicense-..." # License key that was provided to you by Buf

Put the values from the steps below in the bsr.yaml file. You can skip to Install the Helm Chart for a full example Helm chart.

Configure object storage#

The BSR requires either S3-compatible object storage, Azure Blob Storage, or GCS. Two services need bucket access: bufd (the main API) and the OCI image registry (oci-registry). Configuration in this section creates two Kubernetes service accounts (bufd-service-account and oci-registry-service-account) for them.

S3#

To allow access to the S3 bucket, the bufd and oci-registry services require EKS Pod Identity to be configured so the pods can assume an IAM role. To configure the storage, set the following Helm values, filling in your S3 variables:

storage:
  use: s3
  s3:
    bucketName: "my-bucket-name"
    region: "us-east-1"
    # forcePathStyle: false # Optional, use path-style bucket URLs (https://s3.amazonaws.com/BUCKET/KEY)
    # insecure: false # Optional, disable TLS
    # endpoint: "example.com" # Optional

With this configuration, the Helm chart creates two k8s service accounts that need to be bound to the IAM role by creating Pod Identity Associations: bufd-service-account and oci-registry-service-account.

To allow access to the S3 bucket, the bufd and oci-registry services require IRSA to be configured so the pods can assume an IAM role. To configure the storage, set the following Helm values, filling in your S3 bucket name and required annotations:

storage:
  use: s3
  s3:
    bucketName: "my-bucket-name"
    region: "us-east-1"
    # forcePathStyle: false # Optional, use path-style bucket URLs (https://s3.amazonaws.com/BUCKET/KEY)
    # insecure: false # Optional, disable TLS
    # endpoint: "example.com" # Optional
bufd:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<account id>:role/<role name>
ociregistry:
  serviceAccount:
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<account id>:role/<role name>

With this configuration, the Helm chart creates two k8s service accounts that needs to be bound to the AWS IAM role trust relationship: bufd-service-account and oci-registry-service-account.

The bufd client and oci-registry will attempt to acquire credentials from the environment. To configure the storage set the following Helm values, filling in your S3 variables:

storage:
  use: s3
  s3:
    bucketName: "my-bucket-name"
    region: "us-east-1"
    # forcePathStyle: false # Optional, use path-style bucket URLs (https://s3.amazonaws.com/BUCKET/KEY)
    # insecure: false # Optional, disable TLS
    # endpoint: "example.com" # Optional

Alternatively, you may instead use an access key pair.

  1. Add the accessKeyId to the configuration:

    storage:
  use: s3
  s3:
    accessKeyId: "AKIAIOSFODNN7EXAMPLE"
    bucketName: "my-bucket-name"
    region: "us-east-1"
    # forcePathStyle: false # Optional, use path-style bucket URLs (https://s3.amazonaws.com/BUCKET/KEY)
    # insecure: false # Optional, disable TLS
    # endpoint: "example.com" # Optional

  2. Create a k8s secret containing the s3 access secret key:

    $ kubectl create secret --namespace bsr generic bufd-storage \
  --from-literal=secret_access_key=<s3 secret access key>
Required permissions#

To interact with S3, the BSR IAM role needs to be granted the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:AbortMultipartUpload",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket-name>",
        "arn:aws:s3:::<bucket-name>/*"
      ]
    }
  ]
}

Azure Blob Storage#

A standard general storage account type is required to support block blobs.

The bufd client and oci-registry will attempt to acquire credentials from the environment. To configure the storage, set the following Helm values by filling in your Azure variables and adding the required annotations for the bufd and ociregistry service accounts and deployments:

storage:
  use: azure
  azure:
    accountName: "my-storage-account-name"
    container: "my-container"
    useAccountKey: false
bufd:
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: "my-client-id"
  deployment:
    podLabels:
      azure.workload.identity/use: "true"
ociregistry:
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: "my-client-id"
  deployment:
    podLabels:
      azure.workload.identity/use: "true"

The service accounts to be bound to the federated identity credentials are named bufd-service-account and oci-registry-service-account.

Alternatively, you may instead use the storage account key.

  1. Set the required Helm values:

    storage:
  use: azure
  azure:
    accountName: "my-storage-account-name"
    container: "my-container"
    useAccountKey: true

  2. Create a k8s secret containing an Azure storage account key:

    $ kubectl create secret --namespace bsr generic bufd-storage \
  --from-literal=account_key=<azure storage account key>

GCS#

Workload Identity Federation#

To allow access to the GCS bucket, the bufd and oci-registry services require Workload Identity Federation to be configured, with a GCP service account attached to the pods. To configure the storage, set the following Helm values, filling in your GCS bucket name and GCP service account:

storage:
  use: gcs
  gcs:
    bucketName: <bucket name>
bufd:
  serviceAccount:
    annotations:
      iam.gke.io/gcp-service-account: <gcp-service-account-name>@<gcp project>.iam.gserviceaccount.com
ociregistry:
  serviceAccount:
    annotations:
      iam.gke.io/gcp-service-account: <gcp-service-account-name>@<gcp project>.iam.gserviceaccount.com

With this configuration, the Helm chart creates two k8s service accounts that need to be bound to the GCP Service account: bufd-service-account and oci-registry-service-account. You also need to grant roles/storage.objectAdmin permissions on the GCS bucket to the GCP service account.

Create a Postgres database#

The BSR requires a PostgreSQL database. The BSR postgres user requires full access to the database, and additionally must be able to create the pgcrypto and pg_trgm extensions.

To configure Postgres, set the following Helm values:

postgres:
  host: "postgres.example.com"
  port: 5432
  database: postgres
  user: postgres
  # Optional, max postgres connection per pool, defaults to 10 if unset/zero
  # maxConnections: 20

Then create a k8s secret containing the Postgres user password:

$ kubectl create secret --namespace bsr generic bufd-postgres \
  --from-literal=password=<postgres password>

Note that if you’re using Azure Cosmos DB for PostgreSQL, it must be configured as a single-node cluster with high availability (HA) enabled.

You can configure the BSR to use GCP Cloud SQL IAM Authentication. The GCP service account needs the following permissions to connect to Cloud SQL properly:

roles/cloudsql.client
roles/cloudsql.instanceUser

Bufd also creates extensions as part of the migrations it runs in postgres, so you need to run the following command from an existing GCP Cloud SQL Superuser on the postgres shell:

GRANT cloudsqlsuperuser to "<gcp-service-account-name>@<gcp project>.iam";

Finally, if you were already running the BSR with another existing Cloud SQL user (eg, postgres), you need to reassign all ownerships:

GRANT "<gcp-service-account-name>@<gcp project>.iam" TO postgres;
REASSIGN OWNED by "postgres" TO "<gcp-service-account-name>@<gcp project>.iam";
REVOKE "<gcp-service-account-name>@<gcp project>.iam" FROM postgres;

Your configuration then looks like this:

postgres:
  cloudSqlInstance: <gcp project>:<gcp region>:<gcp cloud sql instance name>
  database: postgres
  user: <gcp-service-account-name>@<gcp project>.iam
  # Optional, if you need
  # impersonateServiceAccount: <gcp-service-account-to-impersonate-name>@<gcp project>.iam.gserviceaccount.com

For setup details, see the GCP docs.

Configure Redis#

The BSR requires a Redis instance.

  • Only the Redis Standalone deployment mode is supported.
  • Redis Cluster and Sentinel modes aren’t supported for the BSR.

To configure Redis, create a k8s secret containing the address:

$ kubectl create secret --namespace bsr generic bufd-redis \
  --from-literal=address=redis.example.com:6379

Optionally, we also support authentication and TLS for Redis. These can be set with the following Helm values:

redis:
  # Set to true to enable auth for redis.
  # The auth token will be read from the "auth" field in the "bufd-redis" secret
  auth: true
  tls:
    # Whether to use TLS for connecting to Redis
    # Set to "false" to disable TLS
    # Set to "local" to use certs from the "ca" field in the "bufd-redis" secret
    # Set to "system" to use the system trust store
    use: "false"
  • If authentication is enabled, the Redis auth string should be added to the bufd-redis secret in the auth field.
  • If TLS is enabled and use is set to local, the CA certificates to trust should be added to the bufd-redis secret in the ca field.

Example of a secret containing both an authentication token and a CA certificate:

$ kubectl create secret --namespace bsr generic bufd-redis \
  --from-literal=address=redis.example.com:6379 \
  --from-literal=auth=<redis auth string> \
  --from-file=ca=<redis ca.crt>

Example of a secret containing an authentication token, assuming a connection string like redis.example.com:6379,password=<password>,ssl=True,abortConnect=False:

$ kubectl create secret --namespace bsr generic bufd-redis \
  --from-literal=address=redis.example.com:6379 \
  --from-literal=auth=<redis password>

Configure authentication#

The BSR supports authentication using an external identity provider (IdP), through Security Assertion Markup Language (SAML) or OpenID Connect (OIDC).

In the SAML IdP, create a new application to represent the BSR. It should return a single sign-on URL and IdP metadata. Either a public URL or raw XML can be specified for the SAML config. For Okta SAML, see the Okta SAML guide.

To configure SAML authentication in the BSR, set the following Helm values:

auth:
  method: saml
  saml:
    # Endpoint where the XML metadata is available
    idpMetadataURL: "https://example-provider.com/app/12345/sso/saml/metadata"
    # If the authentication provider doesn't have a metadata url,
    # the raw XML metadata can be configured using the idpRawMetadata,
    # value instead.
    idpRawMetadata: |
      <?xml version="1.0" encoding="utf-8"?>
      <EntityDescriptor etc>
    # Optionally, configure the attribute containing groups membership information,
    # to enable support for automated organization membership provisioning.
    # Note that if configured, a user will not be permitted to log in to the BSR if the attribute is missing from the SAML assertion.
    # https://buf.build/docs/bsr/private/user-lifecycle#autoprovisioning
    groupsAttributeName: ""
  # Optional
  # A list of emails that will be granted BSR admin permissions on login
  # Note that this list is case-sensitive
  autoProvisionedAdminEmails:
    - "[email protected]"

SAML requires the application to have access to a certificate used for signing/encryption as part of the authentication process. For the BSR, this is stored as a Kubernetes TLS secret named bsr-saml-cert, and may be self-signed. For example, you can generate a certificate and create the required secret using OpenSSL.

$ openssl req -newkey rsa:2048 -nodes -keyout ca.key -subj "/CN=example.com" -x509 -days 3650 -out ca.crt
$ kubectl create secret --namespace bsr tls bsr-saml-cert \
  --key ca.key \
  --cert ca.crt

In the OIDC IdP, create a new application to represent the BSR and provide the callback URL. For Okta OIDC, see the Okta OIDC guide.

To configure OIDC authentication in the BSR, set the following Helm values:

auth:
  method: oidc
  oidc:
    # Only one of `issuerURL` or `discoveryURL` can be set. If `issuerURL` is set, the BSR
    # automatically discovers the OIDC configuration using the OIDC Discovery protocol. If
    # `discoveryURL` is set, the BSR uses the provided URL to fetch the OIDC Discovery document.
    issuerURL: "https://example.okta.com"
    discoveryURL: "https://example.okta.com/.well-known/openid-configuration"
    # The client ID of the OIDC application created in the IdP.
    clientID: "0oa2ho2ylo0HFI61d5d7"
    # Optional
    # groups:
    #   claim: "custom_claim" # The name of the OIDC claim containing groups information, default groups
    #   source: "userinfo" # Fetch group claim from the userinfo endpoint instead of id token claims
    #   requiredScope: "groups" # Additional scope to request from the IdP, to include groups information in the token/userinfo endpoint.
    # scopes: # Override the scopes to request from the IdP
    #   - openid
    #   - profile
    #   - email
    #   - offline_access
  # Optional
  # A list of emails that will be granted BSR admin permissions on login
  # Note that this list is case-sensitive
  autoProvisionedAdminEmails:
    - "[email protected]"

Additionally, a Kubernetes secret must be created for OIDC to function:

$ kubectl create secret --namespace bsr generic bufd-client-secret \
  --from-literal=client_secret=<oidc client secret>

Configure Ingress#

The BSR uses a Kubernetes Ingress resource to handle incoming traffic and for terminating TLS.

  • The domain used here must match the host set in the Helm values above.
  • TLS is required for the BSR to function properly.
  • HTTP2 is preferred to allow for gRPC support.
bufd:
  ingress:
    enabled: true
    className: "" # Optional ingress class to use
    annotations: {} # Optional ingress annotations
    hosts:
      - host: example.com
        paths:
          - path: /
            portName: http
    # Optional TLS configuration for the ingress.
    # May be omitted to configure TLS termination, depending on the ingress.
    # Requires a kubernetes TLS secret.
    tls:
      - secretName: bsr-tls-cert
        hosts:
          - example.com

Configure load balancer#

Make sure load balancer timeouts are set to at least 5 minutes.

If the load balancer doesn’t support H2C, TLS can optionally be used for communication between the load balancer and the BSR by enabling TLS on the listening ports of the bufd application. This requires a Kubernetes TLS secret named bsr-tls-cert.

bufd:
  tls:
    enabled: true
    # Optional. Secret name for the TLS cert
    # secretName: bsr-tls-cert
  # Optional. Used to add annotations to the ingress service.
  # May be needed for some ingress controllers to function correctly.
  service:
    annotations: {}

Configure observability#

The metrics block is used to configure the collection and exporting of metrics from your application using Prometheus:

observability:
  metrics:
    use: prometheus
    runtime: true
    prometheus:
      podLabels: # This is required if enabling network policies.
        app: prometheus
      port: 9090
      path: /metrics

Trusting additional certificates#

If you bump into issues regarding self-signed certificates, such as seeing the error tls: failed to verify certificate: x509: certificate signed by unknown authority, you can add your root certificates on the BSR. To trust additional certificates, mount the files on the bufd pod and include them in the client TLS configuration.

The clientTLSSecrets value will handle all the volumes and mounts. It is a map of secret name to key containing the certificate.

clientTLSSecrets:
  tls-cert: cert.pem

bufd:
  deployment:
    extraVolumeMounts:
      - mountPath: /config/secrets/certificates/cert.pem
        name: certificate
        readOnly: true
        subPath: cert.pem
    extraVolumes:
      - name: certificate
        secret:
          secretName: tls-cert
          items:
            - key: cert.pem
              path: cert.pem
  clientTLS:
    extraCerts:
      - /config/secrets/certificates/cert.pem

Install the Helm Chart#

After following the steps above, a complete bsr.yaml looks like the example below. This example uses S3, an OIDC IdP, and the clientTLSSecrets form for additional trusted certificates (introduced in chart v1.15.2):

host: example.com
license:
  key: "buflicense-..."

storage:
  use: s3
  s3:
    bucketName: "my-bucket-name"
    region: "us-east-1"

postgres:
  host: "postgres.example.com"
  port: 5432
  database: postgres
  user: postgres

# The Redis address is set in the "bufd-redis" secret's "address" field
# (e.g. "redis.example.com:6379"); see "Configure Redis" above.
redis:
  # Set to true if your Redis requires authentication. The auth token
  # is read from the "auth" field in the "bufd-redis" secret.
  auth: true
  tls:
    # "false" disables TLS, "local" uses the "ca" field in the
    # "bufd-redis" secret, "system" uses the system trust store.
    use: "system"

auth:
  method: oidc
  oidc:
    # Only one of `issuerURL` or `discoveryURL` can be set.
    issuerURL: "https://idp.example.com"
    clientID: "bsr"
  autoProvisionedAdminEmails:
    - "[email protected]"

bufd:
  ingress:
    enabled: true
    hosts:
      - host: example.com
        paths:
          - path: /
            portName: http
    tls:
      - secretName: bsr-tls-cert
        hosts:
          - example.com

# Mount additional trusted certificates (chart v1.15.2 and newer).
# Use bufd.deployment.extraVolumes / clientTLS on older charts.
clientTLSSecrets:
  tls-cert: cert.pem

observability:
  metrics:
    use: prometheus
    runtime: true
    prometheus:
      podLabels: # Required if enabling network policies.
        app: prometheus
      port: 9090
      path: /metrics

For SAML instead of OIDC, replace the auth.oidc block with the auth.saml shape from Configure authentication.

Install the chart with the bsr.yaml and the target chart version:

$ helm install bsr oci://us-docker.pkg.dev/buf-images-1/buf/charts/bsr \
  --version "1.x.x" \
  --namespace=bsr \
  --values bsr.yaml

The BSR is now running at https://<host>. Verify the deploy at https://<host>/-/status (admin auth) or, without authentication, at http://<bufd-pod-ip>:3003/-/status on any bufd pod. See Status page for the full reference.

Next steps#