This repository was archived by the owner on May 8, 2026. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 77
samples: firestore beam connector examples #966
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
a618c8d
samples: firestore beam connector examples
kolea2 6e0d228
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] 8137a96
randomize collection id
kolea2 49bbbe1
review feedback
kolea2 f3ac40e
Merge branch 'beam-samples' of github.com:kolea2/java-firestore into …
kolea2 71e791a
add README.md
kolea2 d729693
fix main class
kolea2 4b2e5c6
fix checkstyle
kolea2 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
135 changes: 135 additions & 0 deletions
135
samples/snippets/src/main/java/com/example/firestore/beam/ExampleFirestoreBeamRead.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| /* | ||
| * Copyright 2022 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.example.firestore.beam; | ||
|
|
||
| import com.google.cloud.firestore.FirestoreOptions; | ||
| import com.google.firestore.v1.DocumentRootName; | ||
| import com.google.firestore.v1.RunQueryRequest; | ||
| import com.google.firestore.v1.RunQueryResponse; | ||
| import com.google.firestore.v1.StructuredQuery; | ||
| import com.google.firestore.v1.Value; | ||
| import java.util.Objects; | ||
| import java.util.UUID; | ||
| import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions; | ||
| import org.apache.beam.sdk.Pipeline; | ||
| import org.apache.beam.sdk.io.gcp.firestore.FirestoreIO; | ||
| import org.apache.beam.sdk.io.gcp.firestore.RpcQosOptions; | ||
| import org.apache.beam.sdk.options.PipelineOptions; | ||
| import org.apache.beam.sdk.options.PipelineOptionsFactory; | ||
| import org.apache.beam.sdk.transforms.Create; | ||
| import org.apache.beam.sdk.transforms.DoFn; | ||
| import org.apache.beam.sdk.transforms.PTransform; | ||
| import org.apache.beam.sdk.transforms.ParDo; | ||
| import org.apache.beam.sdk.values.PCollection; | ||
|
|
||
| public class ExampleFirestoreBeamRead { | ||
|
|
||
| public static void main(String[] args) { | ||
| runRead(args, "cities-collection-" + UUID.randomUUID().toString().substring(0, 10)); | ||
| } | ||
|
|
||
| public static void runRead(String[] args, String collectionId) { | ||
| FirestoreOptions firestoreOptions = FirestoreOptions.getDefaultInstance(); | ||
|
|
||
| PipelineOptions options = | ||
|
kolea2 marked this conversation as resolved.
|
||
| PipelineOptionsFactory.fromArgs(args).withValidation().as(PipelineOptions.class); | ||
| Pipeline pipeline = Pipeline.create(options); | ||
|
|
||
| RpcQosOptions rpcQosOptions = | ||
| RpcQosOptions.newBuilder() | ||
| .withHintMaxNumWorkers(options.as(DataflowPipelineOptions.class).getMaxNumWorkers()) | ||
| .build(); | ||
|
|
||
| pipeline | ||
| .apply(Create.of(collectionId)) | ||
| .apply( | ||
| new FilterDocumentsQuery( | ||
| firestoreOptions.getProjectId(), firestoreOptions.getDatabaseId())) | ||
| .apply(FirestoreIO.v1().read().runQuery().withRpcQosOptions(rpcQosOptions).build()) | ||
| .apply( | ||
| ParDo.of( | ||
| // transform each document to its name | ||
| new DoFn<RunQueryResponse, String>() { | ||
| @ProcessElement | ||
| public void processElement(ProcessContext c) { | ||
| c.output(Objects.requireNonNull(c.element()).getDocument().getName()); | ||
| } | ||
| })) | ||
| .apply( | ||
| ParDo.of( | ||
| // print the document name | ||
| new DoFn<String, Void>() { | ||
| @ProcessElement | ||
| public void processElement(ProcessContext c) { | ||
| System.out.println(c.element()); | ||
| } | ||
| })); | ||
|
|
||
| pipeline.run().waitUntilFinish(); | ||
| } | ||
|
|
||
| private static final class FilterDocumentsQuery | ||
| extends PTransform<PCollection<String>, PCollection<RunQueryRequest>> { | ||
|
|
||
| private final String projectId; | ||
| private final String databaseId; | ||
|
|
||
| public FilterDocumentsQuery(String projectId, String databaseId) { | ||
| this.projectId = projectId; | ||
| this.databaseId = databaseId; | ||
| } | ||
|
|
||
| @Override | ||
| public PCollection<RunQueryRequest> expand(PCollection<String> input) { | ||
| return input.apply( | ||
| ParDo.of( | ||
| new DoFn<String, RunQueryRequest>() { | ||
| @ProcessElement | ||
| public void processElement(ProcessContext c) { | ||
| // select from collection "cities-collection-<uuid>" | ||
| StructuredQuery.CollectionSelector collection = | ||
| StructuredQuery.CollectionSelector.newBuilder() | ||
| .setCollectionId(Objects.requireNonNull(c.element())) | ||
| .build(); | ||
| // filter where country is equal to USA | ||
| StructuredQuery.Filter countryFilter = | ||
| StructuredQuery.Filter.newBuilder() | ||
| .setFieldFilter( | ||
| StructuredQuery.FieldFilter.newBuilder() | ||
| .setField( | ||
| StructuredQuery.FieldReference.newBuilder() | ||
| .setFieldPath("country") | ||
| .build()) | ||
| .setValue(Value.newBuilder().setStringValue("USA").build()) | ||
| .setOp(StructuredQuery.FieldFilter.Operator.EQUAL)) | ||
| .buildPartial(); | ||
|
|
||
| RunQueryRequest runQueryRequest = | ||
| RunQueryRequest.newBuilder() | ||
| .setParent(DocumentRootName.format(projectId, databaseId)) | ||
| .setStructuredQuery( | ||
| StructuredQuery.newBuilder() | ||
| .addFrom(collection) | ||
| .setWhere(countryFilter) | ||
| .build()) | ||
| .build(); | ||
| c.output(runQueryRequest); | ||
| } | ||
| })); | ||
| } | ||
| } | ||
| } | ||
92 changes: 92 additions & 0 deletions
92
samples/snippets/src/main/java/com/example/firestore/beam/ExampleFirestoreBeamWrite.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| /* | ||
| * Copyright 2022 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package com.example.firestore.beam; | ||
|
|
||
| import com.google.cloud.firestore.FirestoreOptions; | ||
| import com.google.firestore.v1.Document; | ||
| import com.google.firestore.v1.Value; | ||
| import com.google.firestore.v1.Write; | ||
| import java.util.UUID; | ||
| import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions; | ||
| import org.apache.beam.sdk.Pipeline; | ||
| import org.apache.beam.sdk.io.gcp.firestore.FirestoreIO; | ||
| import org.apache.beam.sdk.io.gcp.firestore.RpcQosOptions; | ||
| import org.apache.beam.sdk.options.PipelineOptions; | ||
| import org.apache.beam.sdk.options.PipelineOptionsFactory; | ||
| import org.apache.beam.sdk.transforms.Create; | ||
|
|
||
| public class ExampleFirestoreBeamWrite { | ||
|
kolea2 marked this conversation as resolved.
|
||
| private static final FirestoreOptions FIRESTORE_OPTIONS = FirestoreOptions.getDefaultInstance(); | ||
|
|
||
| public static void main(String[] args) { | ||
| runWrite(args, "cities-collection-" + UUID.randomUUID().toString().substring(0, 10)); | ||
| } | ||
|
|
||
| public static void runWrite(String[] args, String collectionId) { | ||
| // create pipeline options from the passed in arguments | ||
| PipelineOptions options = | ||
| PipelineOptionsFactory.fromArgs(args).withValidation().as(PipelineOptions.class); | ||
| Pipeline pipeline = Pipeline.create(options); | ||
|
|
||
| RpcQosOptions rpcQosOptions = | ||
| RpcQosOptions.newBuilder() | ||
| .withHintMaxNumWorkers(options.as(DataflowPipelineOptions.class).getMaxNumWorkers()) | ||
| .build(); | ||
|
|
||
| // create some writes | ||
| Write write1 = | ||
| Write.newBuilder() | ||
| .setUpdate( | ||
| Document.newBuilder() | ||
| // resolves to | ||
| // projects/<projectId>/databases/<databaseId>/documents/<collectionId>/NYC | ||
| .setName(createDocumentName(collectionId, "NYC")) | ||
| .putFields("name", Value.newBuilder().setStringValue("New York City").build()) | ||
| .putFields("state", Value.newBuilder().setStringValue("New York").build()) | ||
| .putFields("country", Value.newBuilder().setStringValue("USA").build())) | ||
| .build(); | ||
|
|
||
| Write write2 = | ||
| Write.newBuilder() | ||
| .setUpdate( | ||
| Document.newBuilder() | ||
| // resolves to | ||
| // projects/<projectId>/databases/<databaseId>/documents/<collectionId>/TOK | ||
| .setName(createDocumentName(collectionId, "TOK")) | ||
| .putFields("name", Value.newBuilder().setStringValue("Tokyo").build()) | ||
| .putFields("country", Value.newBuilder().setStringValue("Japan").build()) | ||
| .putFields("capital", Value.newBuilder().setBooleanValue(true).build())) | ||
| .build(); | ||
|
|
||
| // batch write the data | ||
| pipeline | ||
| .apply(Create.of(write1, write2)) | ||
| .apply(FirestoreIO.v1().write().batchWrite().withRpcQosOptions(rpcQosOptions).build()); | ||
|
|
||
| // run the pipeline | ||
| pipeline.run().waitUntilFinish(); | ||
| } | ||
|
|
||
| private static String createDocumentName(String collectionId, String cityDocId) { | ||
| String documentPath = | ||
| String.format( | ||
| "projects/%s/databases/%s/documents", | ||
| FIRESTORE_OPTIONS.getProjectId(), FIRESTORE_OPTIONS.getDatabaseId()); | ||
|
|
||
| return documentPath + "/" + collectionId + "/" + cityDocId; | ||
| } | ||
| } | ||
35 changes: 35 additions & 0 deletions
35
samples/snippets/src/main/java/com/example/firestore/beam/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # Firestore Beam Samples | ||
|
|
||
| ## Running instructions | ||
| These examples will show you how to read and write data from/into a Firestore database using the Firestore Beam connector. | ||
|
|
||
| 1. Set up the following environment variables | ||
| ``` | ||
| GOOGLE_CLOUD_PROJECT=<your-project-id> | ||
| REGION=<your-region> | ||
| TEMP_LOCATION=gs://<your-temp-bucket>/temp/ | ||
| NUM_WORKERS=<num-workers> | ||
| MAX_NUM_WORKERS=<max-num-workers> | ||
| ``` | ||
| 2. Run the command to write example "cities" data: | ||
| ``` | ||
| mvn compile exec:java -Dexec.mainClass=ExampleFirestoreBeamWrite \ | ||
| "-Dexec.args=--project=$GOOGLE_CLOUD_PROJECT \ | ||
| --runner=DataflowRunner \ | ||
| --project=$GOOGLE_CLOUD_PROJECT \ | ||
| --region=$REGION \ | ||
| --tempLocation=$TEMP_LOCATION \ | ||
| --numWorkers=$NUM_WORKERS \ | ||
| --maxNumWorkers=$MAX_NUM_WORKERS" | ||
| ``` | ||
|
|
||
| 3. Run the command to filter and read the data you wrote: | ||
| ``` | ||
| mvn compile exec:java -Dexec.mainClass=ExampleFirestoreBeamRead \ | ||
| "-Dexec.args=--project=$GOOGLE_CLOUD_PROJECT \ | ||
| --runner=DataflowRunner \ | ||
| --region=$REGION \ | ||
| --tempLocation=$TEMP_LOCATION \ | ||
| --numWorkers=$NUM_WORKERS \ | ||
| --maxNumWorkers=$MAX_NUM_WORKERS" | ||
| ``` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.