feat: Admin service support for managed github repo APIs #7213

k-anshul · 2025-04-24T17:07:14Z

Subtask for : https://github.com/rilldata/rill-private-issues/issues/1569

Checklist:

Covered by tests
Ran it and it works as intended
Reviewed the diff before requesting a review
Checked for unhandled edge cases
Linked the issues it closes
Checked if the docs need to be updated
Intend to cherry-pick into the release branch
I'm proud of this work!

begelundmuller · 2025-05-02T15:14:35Z

admin/database/database.go

@@ -308,6 +308,13 @@ type DB interface {
 	InsertProvisionerResource(ctx context.Context, opts *InsertProvisionerResourceOptions) (*ProvisionerResource, error)
 	UpdateProvisionerResource(ctx context.Context, id string, opts *UpdateProvisionerResourceOptions) (*ProvisionerResource, error)
 	DeleteProvisionerResource(ctx context.Context, id string) error
+
+	FindManagedGithubRepoMeta(ctx context.Context, htmlURL string) (*ManagedGithubRepoMeta, error)


This comment will apply throughout the PR – we should call it "managed Git", not "managed Github".

Github is the provider, which we should be able to swap with another provider in the future by implementing a new integration, but without renaming APIs, database tables, etc..

I mentioned it in another reply but the problem is that a lot of code is already strongly coupled with Github and we don't know what common ground exists between Github and Gitlab and other providers so decoupling it completely is probably not as straightforward.

Hence as an extension it just felt intuitive to mark it Github and not Git but I am fine with removing Github from managed and made changes in the PR.

begelundmuller · 2025-05-02T15:16:12Z

admin/database/database.go

+	OrgID           string    `db:"org_id"`
+	ProjectID       *string   `db:"project_id"`


Storing org_id and project_id feels off, since one can be looked up from the other. Should the relationship to project_id be stored the reverse way? I.e. foreign key from the projects table, not the other way around?

I kept org_id to easily fetch repo for an org(normalization vs simplcity). But I can remove it as well. Just need an extra join.

Yeah I think storing the relationship other way is also good. That way we no longer need to maintain a bool indicating that this project uses a managed git repo. Though we can continue to duplicate remote in both places to keep rest of the code consistent.

begelundmuller · 2025-05-02T15:18:39Z

admin/database/database.go

+// ManagedGithubRepoMeta represents metadata about a Rill managed GitHub repository for projects deployed on Rill Cloud.
+type ManagedGithubRepoMeta struct {


Since this is the database package where many things are metadata, I think "meta" might be a redundant word. For example, we don't say DeploymentMeta or AssetMeta, just Deployment or Asset.

ManagedGitRepoMeta indicates that it just a metadata about the managed git repo and the actual data is stored somewhere and not in this table/database. That is why I added the extra meta suffix.

ManagedGitRepo sounds like the managed git repository is stored here itself.
However I am fine with dropping the suffix to make it inline with other tables.

begelundmuller · 2025-05-02T15:19:36Z

admin/database/postgres/migrations/0067.sql

@@ -0,0 +1,12 @@
+CREATE TABLE IF NOT EXISTS managed_github_repo_meta (


nit: remove IF NOT EXISTS. It's very unexpected if it already exists!

I just added to make it fail proof if we switch between versions. I can remove it as well.

admin/github.go

admin/server/projects.go

admin/server/assets.go

admin/github.go

begelundmuller · 2025-05-02T16:09:36Z

admin/database/database.go

+type InsertManagedGithubRepoMetaOptions struct {
+	OrgID           string  `validate:"required"`
+	CreatedByUserID *string `validate:"required"`
+	HTMLURL         string  `validate:"required"`


I know we've previously used html URLs, but I regret it a bit and it feels fragile. For this use case, should we perhaps store HTTP remotes instead? E.g. https://github.com/account/repo.git. It would make configuring remotes simpler, and would be better compatible with other Git providers than Github (e.g. Gitlab allows extra levels of nesting for repository paths).

In the follow up to this PR, we also need to set this in project URL and it may be inconvenient to see both http_url and clone_url in the same column.
I can make it remote and set clone_url and then do handling at places wherever it is required.

begelundmuller · 2025-05-02T16:13:36Z

cli/cmd/admin/start.go

@@ -230,7 +231,7 @@ func StartCmd(ch *cmdutil.Helper) *cobra.Command {
 			emailClient := email.New(sender)

 			// Init github client
-			gh, err := admin.NewGithub(conf.GithubAppID, conf.GithubAppPrivateKey)
+			gh, err := admin.NewGithub(cmd.Context(), conf.GithubAppID, conf.GithubAppPrivateKey, conf.RillManagedGithubOrg)


Not necessarily something to do now – but from an abstraction perspective, it feels weird to use the same client for user repositories and managed repositories. Would it make sense to initialize a separate client for managed Git repositories? E.g. admin.Github and admin.ManagedGit (incidentally backed by Github, but could be swapped for e.g. Google Cloud Source Repositories)?

Ideally we'd have an admin/git/provider.go interface with a common interface for Git providers. That'd also make supporting other providers easier.

I think we are in a complicated situation here.
Github and other providers do not confirm to a common implementation apart from the fact that they serve git repositories. So the serving part (which is mostly in runtime) is a common implementation but a lot of code in admin is already coupled to the provider.
Things like installationID, installation token and several such constructs may not be applicable to other providers and are kept in project table directly.

So we have two choices here :
a) Either embrace the fact that github is going to be an important part of our workflows and make it strongly coupled within our Admin code.
b) Try to find some middle ground for all these providers and keep it completely decoupled. But then we also need to make several changes in the code.

Maybe lets evaluate option 2 in a separate task.

admin/database/postgres/migrations/0067.sql

admin/github.go

admin/server/github.go

admin/server/server.go

admin/server/projects.go

Co-authored-by: Benjamin Egelund-Müller <[email protected]>

admin/database/postgres/postgres.go

admin/database/postgres/migrations/0068.sql

admin/github.go

admin/server/projects.go

begelundmuller · 2025-05-13T10:02:52Z

admin/server/projects.go

+// It assumes that the project is connected to github and necessary checks have been done by the caller.
+func (s *Server) githubRepoIDForProject(ctx context.Context, p *database.Project) (int64, error) {
+	if p.GithubRepoID != nil {
+		return *p.GithubRepoID, nil
+	}
+
+	client, err := s.admin.Github.InstallationClient(*p.GithubInstallationID)
+	if err != nil {


Maybe just to be safe, can it return an error if p.GithubInstallationID == nil (sounds like it would be an internal error)?

This is checked upstream but I can add here as well.

k-anshul added 2 commits April 24, 2025 22:28

create managed repo APIs

03bfa28

add mocks

cf14006

k-anshul self-assigned this Apr 24, 2025

k-anshul added 4 commits April 25, 2025 10:51

SQL fix

c60691a

use a retryable client

c03b0bc

created by user id

0e5f49d

self review

aabb542

k-anshul requested a review from begelundmuller April 25, 2025 10:24

k-anshul added 4 commits April 28, 2025 10:58

also include github repository id

d3dd154

set repo id in getting installation token

713b9ec

fix mock github

a63152f

small fix

2677ed1

begelundmuller requested changes May 2, 2025

View reviewed changes

k-anshul added 5 commits May 3, 2025 15:33

several review comments

0f5db9b

Merge remote-tracking branch 'origin/main' into create_mgd_repo_api

9932cf6

fix test cases

2027b2c

fix lint errors

887ee1a

Merge remote-tracking branch 'origin/main' into create_mgd_repo_api

a9087ec

begelundmuller requested changes May 5, 2025

View reviewed changes

k-anshul and others added 7 commits May 5, 2025 21:39

Update admin/github.go

4ec04cf

Co-authored-by: Benjamin Egelund-Müller <[email protected]>

Update admin/server/github.go

402bbac

Co-authored-by: Benjamin Egelund-Müller <[email protected]>

review comments

de0f9cf

Merge remote-tracking branch 'origin/main' into create_mgd_repo_api

b522e92

fix lint errors

5558411

go fmt fix

15ba4f4

small review

767feef

k-anshul requested a review from begelundmuller May 6, 2025 06:02

begelundmuller requested changes May 6, 2025

View reviewed changes

admin/database/postgres/postgres.go Outdated Show resolved Hide resolved

admin/database/postgres/postgres.go Outdated Show resolved Hide resolved

admin/database/postgres/migrations/0068.sql Outdated Show resolved Hide resolved

admin/github.go Outdated Show resolved Hide resolved

k-anshul added 2 commits May 8, 2025 10:38

Merge remote-tracking branch 'origin/main' into create_mgd_repo_api

1af0d0b

fix queries and add tests

6447a04

lint fix

4516be2

begelundmuller requested changes May 13, 2025

View reviewed changes

begelundmuller approved these changes May 13, 2025

View reviewed changes

k-anshul added 3 commits May 13, 2025 15:37

Merge remote-tracking branch 'origin/main' into create_mgd_repo_api

b1fed91

use superuser_force_access flag

5bb4771

Merge remote-tracking branch 'origin/main' into create_mgd_repo_api

b139a12

k-anshul merged commit b7f686d into main May 13, 2025
13 checks passed

k-anshul deleted the create_mgd_repo_api branch May 13, 2025 10:47

		OrgID string `db:"org_id"`
		ProjectID *string `db:"project_id"`

		// ManagedGithubRepoMeta represents metadata about a Rill managed GitHub repository for projects deployed on Rill Cloud.
		type ManagedGithubRepoMeta struct {

		@@ -0,0 +1,12 @@
		CREATE TABLE IF NOT EXISTS managed_github_repo_meta (

feat: Admin service support for managed github repo APIs #7213

feat: Admin service support for managed github repo APIs #7213

Uh oh!

Conversation

k-anshul commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-anshul May 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

k-anshul commented Apr 24, 2025 •

edited

Loading

k-anshul May 3, 2025 •

edited

Loading