ent · a8m · Nov 2, 2021 · Nov 1, 2021
diff --git a/doc/website/blog/2021-11-1-sync-to-external-data-systems-using-hooks.md b/doc/website/blog/2021-11-1-sync-to-external-data-systems-using-hooks.md
@@ -0,0 +1,306 @@
+---
+title: Sync Changes to External Data Systems using Ent Hooks
+author: Ariel Mashraki
+authorURL: https://github.com/a8m
+authorImageURL: "https://avatars0.githubusercontent.com/u/7413593"
+authorTwitter: arielmashraki
+image: https://entgo.io/images/assets/sync-hook/share.png
+---
+
+One of the common questions we get from the Ent community is how to synchronize objects or references between the
+database backing an Ent application (e.g. MySQL or PostgreSQL) with external services. For example, users would like
+to create or delete a record from within their CRM when a user is created or deleted in Ent, publish a message to a
+[Pub/Sub system](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) when an entity is updated, or verify
+references to blobs in object storages such as AWS S3 or Google Cloud Storage.
+
+Ensuring consistency between two separate data systems is not a simple task.  When we want to propagate, for example,
+the deletion of a record in one system to another, there is no obvious way to guarantee that the two systems will end in
+a synchronized state, since one of them may fail, and the network link between them may be slow or down. Having said
+that, and especially with the prominence of microservices-architectures, these problems have become more common, and
+distributed systems researchers have come up with patterns to solve them, such as the
+[Saga Pattern](https://microservices.io/patterns/data/saga.html).
+
+The application of these patterns is usually complex and difficult, and so in many cases architects do not go after a
+"perfect" design, and instead go after simpler solutions that involve either the acceptance of some inconsistency
+between the systems or background reconciliation procedures.
+
+In this post, we will not discuss how to solve distributed transactions or implement the Saga pattern with Ent.
+Instead, we will limit our scope to study how to hook into Ent mutations before and after they occur, and run our
+custom logic there.
+
+### Propagating Mutations to External Systems
+
+In our example, we are going to create a simple `User` schema with 2 immutable string fields, `"name"` and
+`"avatar_url"`. Let's run the `ent init` command for creating a skeleton schema for our `User`:
+
+```shell
+go run entgo.io/ent/cmd/ent init User
+```
+
+Then, add the `name` and the `avatar_url` fields and run `go generate` to generate the assets.
+
+```go title="ent/schema/user.go"
+type User struct {
+	ent.Schema
+}
+
+func (User) Fields() []ent.Field {
+	return []ent.Field{
+		field.String("name").
+			Immutable(),
+		field.String("avatar_url").
+			Immutable(),
+	}
+}
+```
+
+```shell
+go generate ./ent
+```
+
+### The Problem
+
+The `avatar_url` field defines a URL to an image in a bucket on our object storage (e.g. AWS S3). For the purpose of
+this discussion  we want to make sure that:
+
+- When a user is created, an image with the URL stored in `"avatar_url"` exists in our bucket.
+- Orphan images are deleted from the bucket. This means that when a user is deleted from our system, its avatar image
+  is deleted as well.
+
+For interacting with blobs, we will use the [`gocloud.dev/blob`](https://gocloud.dev/howto/blob) package. This package
+provides abstraction for reading, writing, deleting and listing blobs in a bucket. Similar to the `database/sql`
+package, it allows interacting with variety of object storages with the same API by configuring its driver URL. 
+For example:
+
+```go
+// Open an in-memory bucket. 
+if bucket, err := blob.OpenBucket(ctx, "mem://photos/"); err != nil {
+	log.Fatal("failed opening in-memory bucket:", err)
+}
+
+// Open an S3 bucket named photos.
+if bucket, err := blob.OpenBucket(ctx, "s3://photos"); err != nil {
+	log.Fatal("failed opening s3 bucket:", err)
+}
+
+// Open a bucket named photos in Google Cloud Storage.
+if bucket, err := blob.OpenBucket(ctx, "gs://my-bucket"); err != nil {
+	log.Fatal("failed opening gs bucket:", err)
+}
+```
+
+### Schema Hooks
+
+[Hooks](https://entgo.io/docs/hooks) are a powerful feature of Ent that allows adding custom logic before and after
+operations that mutate the graph.
+
+Hooks can be either defined dynamically using `client.Use` (called "Runtime Hooks"), or explicitly on the schema
+(called "Schema Hooks") as follows:
+
+```go
+// Hooks of the User.
+func (User) Hooks() []ent.Hook {
+	return []ent.Hook{
+		EnsureImageExists(),
+		DeleteOrphans(),
+	}
+}
+```
+
+As you can imagine, the `EnsureImageExists` hook will be responsible for ensuring that when a user is created, their
+avatar URL exists in the bucket, and the `DeleteOrphans` will ensure that orphan images are deleted. Let's start
+writing them.
+
+```go title="ent/schema/hooks.go"
+func EnsureImageExists() ent.Hook {
+	hk := func(next ent.Mutator) ent.Mutator {
+		return hook.UserFunc(func(ctx context.Context, m *ent.UserMutation) (ent.Value, error) {
+			avatarURL, exists := m.AvatarURL()
+			if !exists {
+				return nil, errors.New("avatar field is missing")
+			}
+			// TODO:
+			// 1. Verify that "avatarURL" points to a real object in the bucket.
+			// 2. Otherwise, fail.
+			return next.Mutate(ctx, m)
+		})
+	}
+	// Limit the hook only to "Create" operations.
+	return hook.On(hk, ent.OpCreate)
+}
+
+func DeleteOrphans() ent.Hook {
+	hk := func(next ent.Mutator) ent.Mutator {
+		return hook.UserFunc(func(ctx context.Context, m *ent.UserMutation) (ent.Value, error) {
+			id, exists := m.ID()
+			if !exists {
+				return nil, errors.New("id field is missing")
+			}
+			// TODO:
+			// 1. Get the AvatarURL field of the deleted user.
+			// 2. Cascade the deletion to object storage.
+			return next.Mutate(ctx, m)
+		})
+	}
+	// Limit the hook only to "DeleteOne" operations.
+	return hook.On(hk, ent.OpDeleteOne)
+}
+```
+
+Now, you may ask yourself, _how do we access the blob client from the mutations hooks?_ You are going to find out in
+the next section.
+
+### Injecting Dependencies
+
+The [entc.Dependency](https://entgo.io/docs/code-gen/#external-dependencies) option allows extending the generated
+builders with external dependencies as struct fields, and provides options for injecting them on client initialization.
+
+To inject a `blob.Bucket` to be available inside our hooks, we can follow the tutorial about external dependencies in
+[the website](https://entgo.io/docs/code-gen/#external-dependencies), and define the
+[`gocloud.dev/blob.Bucket`](https://pkg.go.dev/gocloud.dev/blob#Bucket) as a dependency.
+
+```go title="ent/entc.go" {3-6}
+func main() {
+	opts := []entc.Option{
+		entc.Dependency(
+			entc.DependencyName("Bucket"),
+			entc.DependencyType(&blob.Bucket{}),
+		),
+	}
+	if err := entc.Generate("./schema", &gen.Config{}, opts...); err != nil {
+		log.Fatalf("running ent codegen: %v", err)
+	}
+}
+```
+
+Next, re-run code generation:
+
+```shell
+go generate ./ent
+```
+
+We can now access the Bucket API from all generated builders. Let's finish the implementations of the above hooks.
+
+```go title="ent/schema/hooks.go"
+// EnsureImageExists ensures the avatar_url points
+// to a real object in the bucket.
+func EnsureImageExists() ent.Hook {
+	hk := func(next ent.Mutator) ent.Mutator {
+		return hook.UserFunc(func(ctx context.Context, m *ent.UserMutation) (ent.Value, error) {
+			avatarURL, exists := m.AvatarURL()
+			if !exists {
+				return nil, errors.New("avatar field is missing")
+			}
+			switch exists, err := m.Bucket.Exists(ctx, avatarURL); {
+			case err != nil:
+				return nil, fmt.Errorf("check key existence: %w", err)
+			case !exists:
+				return nil, fmt.Errorf("key %q does not exist in the bucket", avatarURL)
+			default:
+				return next.Mutate(ctx, m)
+			}
+		})
+	}
+	return hook.On(hk, ent.OpCreate)
+}
+
+// DeleteOrphans cascades the user deletion to the bucket.
+// Hence, when a user is deleted, its avatar image is deleted
+// as well.
+func DeleteOrphans() ent.Hook {
+	hk := func(next ent.Mutator) ent.Mutator {
+		return hook.UserFunc(func(ctx context.Context, m *ent.UserMutation) (ent.Value, error) {
+			id, exists := m.ID()
+			if !exists {
+				return nil, errors.New("id field is missing")
+			}
+			u, err := m.Client().User.Get(ctx, id)
+			if err != nil {
+				return nil, fmt.Errorf("getting deleted user: %w", err)
+			}
+			if err := m.Bucket.Delete(ctx, u.AvatarURL); err != nil {
+				return nil, fmt.Errorf("deleting user avatar from bucket: %w", err)
+			}
+			return next.Mutate(ctx, m)
+		})
+	}
+	return hook.On(hk, ent.OpDeleteOne)
+}
+```
+
+Now, it's time to test our hooks! Let's write a testable example that verifies that our 2 hooks work as expected.
+
+```go
+package main
+
+import (
+	"context"
+	"fmt"
+	"log"
+
+	"github.com/a8m/ent-sync-example/ent"
+	_ "github.com/a8m/ent-sync-example/ent/runtime"
+
+	"entgo.io/ent/dialect"
+	_ "github.com/mattn/go-sqlite3"
+	"gocloud.dev/blob"
+	_ "gocloud.dev/blob/memblob"
+)
+
+func Example_SyncCreate() {
+	ctx := context.Background()
+	// Open an in-memory bucket.
+	bucket, err := blob.OpenBucket(ctx, "mem://photos/")
+	if err != nil {
+		log.Fatal("failed opening bucket:", err)
+	}
+	client, err := ent.Open(
+		dialect.SQLite,
+		"file:ent?mode=memory&cache=shared&_fk=1",
+		// Inject the blob.Bucket on client initialization.
+		ent.Bucket(bucket),
+	)
+	if err != nil {
+		log.Fatal("failed opening connection to sqlite:", err)
+	}
+	defer client.Close()
+	if err := client.Schema.Create(ctx); err != nil {
+		log.Fatal("failed creating schema resources:", err)
+	}
+	if err := client.User.Create().SetName("a8m").SetAvatarURL("a8m.png").Exec(ctx); err == nil {
+		log.Fatal("expect user creation to fail because the image does not exist in the bucket")
+	}
+	if err := bucket.WriteAll(ctx, "a8m.png", []byte{255, 255, 255}, nil); err != nil {
+		log.Fatalf("failed uploading image to the bucket: %v", err)
+	}
+	fmt.Printf("%q\n", keys(ctx, bucket))
+
+	// User creation should pass as image was uploaded to the bucket.
+	u := client.User.Create().SetName("a8m").SetAvatarURL("a8m.png").SaveX(ctx)
+
+	// Deleting a user, should delete also its image from the bucket.
+	client.User.DeleteOne(u).ExecX(ctx)
+	fmt.Printf("%q\n", keys(ctx, bucket))
+
+	// Output:
+	// ["a8m.png"]
+	// []
+}
+```
+
+### Wrapping Up
+
+Great! We have configured Ent to extend our generated code and inject the `blob.Bucket` as an 
+[External Dependency](https://entgo.io/docs/code-gen#external-dependencies). Next, we defined two mutation hooks and
+used the `blob.Bucket` API to ensure our product constraints are satisfied.
+
+The code for this example is available at [github.com/a8m/ent-sync-example](https://github.com/a8m/ent-sync-example).
+
+:::note For more Ent news and updates:
+
+- Subscribe to our [Newsletter](https://www.getrevue.co/profile/ent)
+- Follow us on [Twitter](https://twitter.com/entgo_io)
+- Join us on #ent on the [Gophers Slack](https://entgo.io/docs/slack)
+- Join us on the [Ent Discord Server](https://discord.gg/qZmPgTE6RX)
+
+:::