A Transform Service that uses Docker Compose

[chamikaramj]

A Docker Compose based transform service

This can be started to use portable transforms offered by Beam. Additionally this can be used to discover schema-aware transforms available in Beam.

After containers are released, this service can be started up by any Beam user with following steps:

• Install Docker

• Set environment variable BEAM_VERSION to the Beam version.

• Run following from the directory that contains the docker-compose.yml file
  $ docker-compose up

• Now the transforms service will be available at "<IP of the current machine>:5001"

Please see https://s.apache.org/beam-transform-service for the design.

Github issue: #26211

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[codecov]

Codecov Report

Merging #26023 (8205d0b) into master (21301b1) will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master   #26023      +/-   ##
==========================================
- Coverage   72.06%   72.06%   -0.01%     
==========================================
  Files         745      745              
  Lines      101203   101203              
==========================================
- Hits        72932    72928       -4     
- Misses      26811    26815       +4     
  Partials     1460     1460              

Flag	Coverage Δ
python	81.09% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...beam/runners/portability/expansion_service_main.py	0.00% <0.00%> (ø)

... and 4 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[github-actions]

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @AnandInguva for label python.
R: @Abacn for label java.
R: @damccorm for label build.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[chamikaramj]

R: @robertwb

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

[chamikaramj]

Tested by running both following pipelines against the same transform service.

• A Python pipeline that uses Java Kafka read and write transforms.
• A Java pipeline that uses the Python Dataframe transform.

[chamikaramj]

Run Java PreCommit

[chamikaramj]

Run Java_GCP_IO_Direct PreCommit

[chamikaramj]

@robertwb friendly ping.

Please note that I hope to add some advanced features (for example, sharing auth credentials between the local machine and the Docker containers) in future PRs.

[chamikaramj]

Run Java PreCommit

[chamikaramj]

Run Spotless PreCommit

[chamikaramj]

Also added support for sharing credentials between the local machine and the expansion service containers.

[chamikaramj]

Run Java PreCommit

[chamikaramj]

Run Java PreCommit

[lostluck]

If the goal is to keep the released instance of this service in sync with released beam, then you do not want to create a new Go module at this level (the go.mod and go.sum files.)

Otherwise this becomes another thing to try and keep up to date.

[chamikaramj]

Ack. Probably we can push the transform service under "sdks/java" to avoid adding a new Go module.

[chamikaramj]

These files were removed.

[Abacn]

LGTM. Just have a few minor comments (see above). Validated the command ./gradlew :sdks:java:transform-service:controller-container:docker running successfully on linux amd64.

Note: On apple M1 it fails with

#9 ERROR: "/target/launcher/linux_amd64/boot" not found: not found
------
 > [6/9] COPY target/launcher/linux_amd64/boot /opt/apache/beam/:
------
failed to compute cache key: "/target/launcher/linux_amd64/boot" not found: not found

because the generated image was in target/launcher/linux_arm64/boot

[chamikaramj]

Thanks. Updated to support both amd64 and arm64.

[chamikaramj]

Run Java PreCommit

[chamikaramj]

Run Spotless PreCommit

[chamikaramj]

Run Community Metrics Prober

[chamikaramj]

Run CommunityMetrics PreCommit

[chamikaramj]

Run Java_GCP_IO_Direct PreCommit

[chamikaramj]

Run Java_Pulsar_IO_Direct PreCommit

[chamikaramj]

Run Java_Amazon-Web-Services_IO_Direct PreCommit

[chamikaramj]

Run Java_Amazon-Web-Services2_IO_Direct PreCommit