Merge pull request #33061 from liferoad/beam-on-flink

Fixed the broken beam python on flink with PortableRunner
apache · Nov 11, 2024 · e0a5196 · e0a5196
2 parents d760383 + 4b7bcd8
commit e0a5196
Show file tree

Hide file tree

Showing 2 changed files with 11 additions and 4 deletions.
diff --git a/runners/flink/job-server-container/Dockerfile b/runners/flink/job-server-container/Dockerfile
@@ -28,4 +28,6 @@ COPY target/LICENSE /opt/apache/beam/
 COPY target/NOTICE /opt/apache/beam/
 
 WORKDIR /opt/apache/beam
-ENTRYPOINT ["./flink-job-server.sh"]
+
+# Add a conditional check for a mounted volume. This allows passing flink configs.
+ENTRYPOINT ["/bin/sh", "-c", "if [ -d \"/flink-conf\" ]; then /opt/apache/beam/flink-job-server.sh --flink-conf-dir /flink-conf; else /opt/apache/beam/flink-job-server.sh; fi"]
diff --git a/website/www/site/content/en/documentation/runners/flink.md b/website/www/site/content/en/documentation/runners/flink.md
@@ -207,12 +207,17 @@ To run a pipeline on an embedded Flink cluster:
 {{< /paragraph >}}
 
 {{< paragraph class="language-portable" >}}
-(1) Start the JobService endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest`
+(1) Start the JobService endpoint: `docker run --net=host apache/beam_flink1.18_job_server:latest`
 {{< /paragraph >}}
 
 {{< paragraph class="language-portable" >}}
 The JobService is the central instance where you submit your Beam pipeline to.
-The JobService will create a Flink job for the pipeline and execute the job.
+It creates a Flink job from your pipeline and executes it.
+You might encounter an error message like `Caused by: java.io.IOException: Insufficient number of network buffers:...`.
+This can be resolved by providing a Flink configuration file to override the default settings.
+You can find an example configuration file [here](https://github.com/apache/beam/blob/master/runners/flink/src/test/resources/flink-conf.yaml).
+To start the Job Service endpoint with your custom configuration, mount a local directory containing your Flink configuration to the `/flink-conf` path in the Docker container:
+`docker run --net=host -v <your_flink_conf_dir>:/flink-conf beam-flink-runner apache/beam_flink1.18_job_server:latest`
 {{< /paragraph >}}
 
 {{< paragraph class="language-portable" >}}
@@ -243,7 +248,7 @@ To run on a separate [Flink cluster](https://ci.apache.org/projects/flink/flink-
 {{< /paragraph >}}
 
 {{< paragraph class="language-portable" >}}
-(2) Start JobService with Flink Rest endpoint: `docker run --net=host apache/beam_flink1.10_job_server:latest --flink-master=localhost:8081`.
+(2) Start JobService with Flink Rest endpoint: `docker run --net=host apache/beam_flink1.18_job_server:latest --flink-master=localhost:8081`.
 {{< /paragraph >}}
 
 {{< paragraph class="language-portable" >}}