|
| 1 | +--- |
| 2 | +type: runners |
| 3 | +title: "Prism Runner" |
| 4 | +aliases: /learn/runners/prism/ |
| 5 | +--- |
| 6 | +<!-- |
| 7 | +Licensed under the Apache License, Version 2.0 (the "License"); |
| 8 | +you may not use this file except in compliance with the License. |
| 9 | +You may obtain a copy of the License at |
| 10 | +
|
| 11 | +http://www.apache.org/licenses/LICENSE-2.0 |
| 12 | +
|
| 13 | +Unless required by applicable law or agreed to in writing, software |
| 14 | +distributed under the License is distributed on an "AS IS" BASIS, |
| 15 | +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 16 | +See the License for the specific language governing permissions and |
| 17 | +limitations under the License. |
| 18 | +--> |
| 19 | + |
| 20 | +# Overview |
| 21 | + |
| 22 | +The Apache Beam Prism Runner can be used to execute Beam pipelines locally using [Beam Portability](/roadmap/portability/). |
| 23 | + |
| 24 | +The Prism runner is suitable for small scale local testing and provides: |
| 25 | + |
| 26 | +* A statically compiled, single binary for simple deployment without additional configuration. |
| 27 | +* A web UI when executing in stand alone mode. |
| 28 | +* A direct implementation of Beam execution semantics. |
| 29 | +* A streaming-first runtime that supports batch processing and data streaming programs. |
| 30 | +* Fast, in-memory execution for to simplify SDK, Transform, and Pipeline development. |
| 31 | +* Cross Language Transform support. |
| 32 | + |
| 33 | +Written in [Go](https://go.dev), it is the default runner for the [Go SDK](/roadmap/go-sdk/), but can be used in other SDKs as well (see below). |
| 34 | + |
| 35 | +# Capabilities |
| 36 | + |
| 37 | +While Prism already supports a great deal of Beam features, it doesn't yet support everything. |
| 38 | +Prism is under active development to close these gaps. |
| 39 | + |
| 40 | +With the exception of timer issues, use of unsupported features should fail the pipeline at job submission time. |
| 41 | + |
| 42 | +In the [2.59.0 release](/blog/beam-2.59.0/), Prism passes most runner validations tests with the exceptions of pipelines using the following features: |
| 43 | + |
| 44 | +OrderedListState, OnWindowExpiry (eg. GroupIntoBatches), CustomWindows, MergingWindowFns, Trigger and WindowingStrategy associated features, Bundle Finalization, Looping Timers, and some Coder related issues such as with Python combiner packing, and Java Schema transforms, and heterogenous flatten coders. |
| 45 | +Processing Time timers do not yet have real time support. |
| 46 | + |
| 47 | + |
| 48 | +See the [Roadmap](/roadmap/prism-runner/) for how to find current progress. |
| 49 | +Specific feature support information will soon migrate to the [Runner Capability Matrix](/documentation/runners/capability-matrix/). |
| 50 | + |
| 51 | +# Using the Prism Runner |
| 52 | + |
| 53 | +{{< language-switcher go java py >}} |
| 54 | + |
| 55 | +<span class="language-go">Prism is the default runner for the Go SDK and is used automatically. Set the runner with the flag `--runner=PrismRunner`. </span> |
| 56 | +<span class="language-java">Set the runner to `PrismRunner`. </span> |
| 57 | +<span class="language-py">Set the runner to `PrismRunner`. </span> |
| 58 | + |
| 59 | +For other SDKs, Prism is included as an asset on [Beam Github Releases](https://github.com/apache/beam/releases/tag/v{{< param release_latest >}}) for download and stand alone use. |
| 60 | + |
| 61 | +Here are some resources with information about how to test your pipelines. |
| 62 | +<ul> |
| 63 | + <li><a href="/documentation/pipelines/test-your-pipeline/">Test Your Pipeline</a></li> |
| 64 | + <li>The <a href="/get-started/wordcount-example/#testing-your-pipeline-with-asserts">Apache Beam WordCount Walkthrough</a> contains an example of logging and testing a pipeline with asserts. |
| 65 | + <!-- Java specific links --> |
| 66 | + <li class="language-java"><a href="/blog/2016/10/20/test-stream.html">Testing Unbounded Pipelines in Apache Beam</a> talks about the use of Java classes <a href="https://beam.apache.org/releases/javadoc/{{< param release_latest >}}/index.html?org/apache/beam/sdk/testing/PAssert.html">PAssert</a> and <a href="https://beam.apache.org/releases/javadoc/{{< param release_latest >}}/index.html?org/apache/beam/sdk/testing/TestStream.html">TestStream</a> to test your pipelines.</li> |
| 67 | +</ul> |
| 68 | + |
| 69 | +### Specify your dependency |
| 70 | + |
| 71 | +<span class="language-java">When using Java, you must specify your dependency on the Direct Runner in your `pom.xml`.</span> |
| 72 | +{{< highlight java >}} |
| 73 | +<dependency> |
| 74 | + <groupId>org.apache.beam</groupId> |
| 75 | + <artifactId>beam-runners-prism-java</artifactId> |
| 76 | + <version>{{< param release_latest >}}</version> |
| 77 | + <scope>runtime</scope> |
| 78 | +</dependency> |
| 79 | +{{< /highlight >}} |
| 80 | + |
| 81 | +<span class="language-py">This section is not applicable to the Beam SDK for Python. Prism is built in.</span> |
| 82 | +<span class="language-go">This section is not applicable to the Beam SDK for Go. Prism is built in.</span> |
| 83 | + |
| 84 | +Except for the Go SDK, Prism is included as an asset on [Beam Github Releases](https://github.com/apache/beam/releases/tag/v{{< param release_latest >}}) for automatic download, startup, and shutdown on SDKs. |
| 85 | +The binary is cached locally for subsequent executions. |
| 86 | + |
| 87 | +## Pipeline options for the Prism Runner |
| 88 | + |
| 89 | +Prism aims to have minimal configuration required, and does not currently present user pipeline options. |
| 90 | + |
| 91 | +## Running Prism Standalone |
| 92 | + |
| 93 | +Prism can be executed as a stand alone binary and will present a basic UI for listing jobs, and job status. |
| 94 | +This is an optional mode for Prism that is useful for demos or rapid iteration. |
| 95 | +It is not a requirement for using Prism in the Java or Python SDKs. |
| 96 | + |
| 97 | +This can be done in two ways, downloading an asset from the github release, or building the binary locally with Go installed. |
| 98 | + |
| 99 | +In either case, Prism serves a JobManagement API endpoint, and a Webpage UI locally. |
| 100 | +Jobs can be submitted using `--runner=PortableRunner --endpoint=<endpoint address>` and monitored using the webpage UI. |
| 101 | + |
| 102 | +Example output from the Prism binary: |
| 103 | + |
| 104 | +``` |
| 105 | +2024/09/30 09:56:42 INFO Serving JobManagement endpoint=localhost:8073 |
| 106 | +2024/09/30 09:56:42 INFO Serving WebUI endpoint=http://localhost:8074 |
| 107 | +``` |
| 108 | + |
| 109 | +The binary has the following optional flags: |
| 110 | + |
| 111 | +* `--job_port` sets the port for the Job management server (defaults to 8073) |
| 112 | +* `--web_port` sets the port for the web ui (defaults to 8074) |
| 113 | +* `--serve_http` enables or disables the web ui (defaults to true) |
| 114 | +* `---idle_shutdown_timeout` sets a duration that Prism will wait for a new job before automatically shutting itself down. Uses duration format like `10s`, `5m`,`2h`. Defaults to not shutting down. |
| 115 | + |
| 116 | +### Download a release asset |
| 117 | + |
| 118 | +This approach doesn't require other dependencies or runtimes installed. |
| 119 | +This is recommended if you want to deploy Prism on some other machine. |
| 120 | + |
| 121 | +Navigate to the latest [Beam Release Github page](https://github.com/apache/beam/releases/tag/v{{< param release_latest >}}), scroll to the bottom, and download the correct asset for where you want to run Prism. |
| 122 | + |
| 123 | +For example, if you want to execute Prism on a newer MacBook, you'd download the `darwin-arm64` asset. For executing on many cloud machines, you'd download the `linux-amd64` asset. |
| 124 | + |
| 125 | +This requires downloading the right asset for the machine Prism will run on, such as your development machine. |
| 126 | + |
| 127 | +Simply unzip, and execute. |
| 128 | + |
| 129 | +### Build from the release with Go. |
| 130 | + |
| 131 | +This approach requires a [recent version of Go installed](https://go.dev/dl/). |
| 132 | +This is recommended if you only want to run Prism on your local machine. |
| 133 | + |
| 134 | +You can insall Prism with `go install`: |
| 135 | + |
| 136 | +```sh |
| 137 | +go install github.com/apache/beam/sdks/v2/go/cmd/prism@latest |
| 138 | +prism |
| 139 | +``` |
| 140 | + |
| 141 | +Or simply build and execute the binary immeadiately using `go run`: |
| 142 | + |
| 143 | +```sh |
| 144 | +go run github.com/apache/beam/sdks/v2/go/cmd/prism@latest |
| 145 | +``` |
0 commit comments