Update pypi documentation 30145 #34329

rakeshcusat · 2025-03-18T05:37:38Z

Please add a meaningful description for your change here
#30145

Added Pypi friendly documentation for Python SDK.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

merging from upstream

github-actions · 2025-03-18T07:07:06Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

sdks/python/README.md

liferoad · 2025-03-23T13:25:10Z

sdks/python/README.md

+
+
+Instructions for building and testing Beam itself
+are in the [contribution guide](./CONTRIBUTING.md).


will this md link work?

good catch, this will not work. I need to adjust the path.

liferoad · 2025-03-23T13:25:47Z

Thanks a lot! Left some comments.

rakeshcusat · 2025-04-28T06:52:39Z

@liferoad / @damccorm can you take another look?

liferoad · 2025-04-28T14:04:33Z

@liferoad / @damccorm can you take another look?

LGTM. Rerunning some failed workflows.

rakeshcusat · 2025-05-05T00:15:13Z

@liferoad do we have any shortcut to re-run the failed test tasks?

liferoad · 2025-05-05T01:56:41Z

@liferoad do we have any shortcut to re-run the failed test tasks?

https://github.com/apache/beam/blob/master/.github/workflows/README.md#running-workflows-manually or using trigger commands:
https://github.com/apache/beam/blob/master/.github/workflows/README.md#precommit-jobs

rakeshcusat · 2025-05-05T06:02:33Z

Run Prism_Python PreCommit 3.9

codecov · 2025-05-05T06:10:31Z

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 56.53%. Comparing base (5950a17) to head (4e4f438).
Report is 384 commits behind head on master.

Files with missing lines	Patch %	Lines
sdks/python/setup.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #34329      +/-   ##
============================================
+ Coverage     56.34%   56.53%   +0.19%     
  Complexity     3286     3286              
============================================
  Files          1173     1182       +9     
  Lines        179037   181076    +2039     
  Branches       3398     3398              
============================================
+ Hits         100877   102375    +1498     
- Misses        74903    75444     +541     
  Partials       3257     3257

Flag	Coverage Δ
python	`81.10% <0.00%> (∅)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-05-05T08:10:09Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @tvalentyn for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

liferoad · 2025-05-05T14:47:02Z

Running pylint...
************* Module setup
setup.py:309:0: C0301: Line too long (94/80) (line-too-long)

rakeshcusat · 2025-05-06T06:44:54Z

Run Yaml_Xlang_Direct PreCommit

rakeshcusat · 2025-05-06T14:14:53Z

Run Python_Coverage PreCommit

rakeshcusat · 2025-05-06T14:18:30Z

Run Python_ML PreCommit 3.12

rakeshcusat · 2025-05-09T07:03:18Z

Run Python_ML PreCommit 3.9

rakeshcusat · 2025-05-10T06:01:24Z

Run Python_ML PreCommit 3.11

rakeshcusat · 2025-05-10T06:09:49Z

Run Python_ML PreCommit 3.10

liferoad · 2025-05-10T12:29:59Z

sdks/python/README.md

+
+## Python streaming pipelines
+
+Python [streaming pipeline execution](/documentation/sdks/python-streaming)


I think we should use the full url paths.

@liferoad are you noticing any error log that is complaining about the relative path?

I have been trying to find the logs from the CI/CD but I am not able to find it.

Is there any documentation that can help replicate this on my local machine?

I mean this will be added here: https://pypi.org/project/apache-beam/. So only full paths can work, right?

@liferoad I thought the setup tool will automatically figure that out based on this config and generate the the righgt link.

I am not sure. I checked https://github.com/numpy/numpy/blob/main/README.md?plain=1, which looks like using the full url paths.

I will update the document and provide the full path.

Is there way to check the generated files in CI/CD environment to make sure that it has generated the right formatted file?

I do not know since this is part of Beam release steps to PyPi.

+1 to using the full links. I'm not confident that the relative paths will definitely work, and it seems safer to use full links.

I actually validated that these don't get replaced, in the final text I see - Python [streaming pipeline execution](/documentation/sdks/python-streaming)

Steps I followed:

Wheels are automatically generated when we build wheels. First from this PRs checks, I clicked one of the build wheels steps

I went to summary, scrolled down to the bottom, and downloaded a set of wheels (I chose wheelhouse-cp312-macos-13, but I don't think this matters)

Extract the wheels from downloads with mac's archive extracter. These are just fancy zip files - https://stackoverflow.com/questions/32923952/how-do-i-list-the-files-inside-a-python-wheel

Navigated to apache_beam-2.66.0.dev0.dist-info/METADATA. This has the content that will be rendered in the pydoc I think.

@damccorm thank you for testing that out. I will change the link to the full link.

github-actions · 2025-05-23T12:15:03Z

Reminder, please take a look at this pr: @tvalentyn

github-actions · 2025-05-27T12:15:53Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @damccorm for label python.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

damccorm

Thanks - this looks like a good step, I think we can merge once the outstanding comment about paths is addressed

damccorm · 2025-05-27T20:21:25Z

sdks/python/README.md

+
+## Python streaming pipelines
+
+Python [streaming pipeline execution](/documentation/sdks/python-streaming)


+1 to using the full links. I'm not confident that the relative paths will definitely work, and it seems safer to use full links.

damccorm · 2025-05-27T20:29:34Z

sdks/python/README.md

+
+## Python streaming pipelines
+
+Python [streaming pipeline execution](/documentation/sdks/python-streaming)


I actually validated that these don't get replaced, in the final text I see - Python [streaming pipeline execution](/documentation/sdks/python-streaming)

Steps I followed:

Wheels are automatically generated when we build wheels. First from this PRs checks, I clicked one of the build wheels steps

I went to summary, scrolled down to the bottom, and downloaded a set of wheels (I chose wheelhouse-cp312-macos-13, but I don't think this matters)

Extract the wheels from downloads with mac's archive extracter. These are just fancy zip files - https://stackoverflow.com/questions/32923952/how-do-i-list-the-files-inside-a-python-wheel

Navigated to apache_beam-2.66.0.dev0.dist-info/METADATA. This has the content that will be rendered in the pydoc I think.

rakeshcusat · 2025-06-08T06:43:55Z

Thanks - this looks like a good step, I think we can merge once the outstanding comment about paths is addressed

@damccorm I have taken care of the comments and I am waiting for test to pass.

rakeshcusat · 2025-06-09T05:18:36Z

@damccorm / @liferoad Some of the precommit test are failing for random reasons which are not related with this change. What do you recommend?
I have already rebased my changes to ensure that I have all the changes on my branch.

liferoad · 2025-06-09T14:46:02Z

#35211 tracks one of the failed workflows.

liferoad · 2025-06-09T14:48:07Z

Re-triggered some workflows.

rakeshcusat and others added 5 commits April 9, 2019 15:55

Merge pull request #1 from apache/master

665570d

merging from upstream

Merge remote-tracking branch 'upstream/master'

f70a239

Merge remote-tracking branch 'upstream/master'

6bba28a

Merge branch 'apache:master' into master

beb3039

Merge branch 'apache:master' into master

154bd93

github-actions bot added the python label Mar 18, 2025

Merge branch 'apache:master' into master

3b5cccd

rakeshcusat mentioned this pull request Mar 18, 2025

[Task]: Improve project documentation on apache-beam pypi page. #30145

Closed

16 tasks

liferoad reviewed Mar 23, 2025

View reviewed changes

sdks/python/README.md Show resolved Hide resolved

liferoad requested review from damccorm, kennknowles, robertwb and tvalentyn March 23, 2025 13:22

liferoad reviewed Mar 23, 2025

View reviewed changes

sdks/python/README.md Show resolved Hide resolved

liferoad reviewed Mar 23, 2025

View reviewed changes

Merge branch 'apache:master' into master

f7bc0e4

github-actions bot added the Next Action: Reviewers label May 5, 2025

Merge branch 'apache:master' into master

df7ca4c

rakeshcusat force-pushed the update-pypi-documentation-30145 branch from ec25727 to 71b53cf Compare May 10, 2025 06:15

Add Pypi friendly documentation

e2c3b36

rakeshcusat force-pushed the update-pypi-documentation-30145 branch from 71b53cf to e2c3b36 Compare May 10, 2025 06:16

liferoad reviewed May 10, 2025

View reviewed changes

github-actions bot added the slow-review label May 23, 2025

github-actions bot added reassigned-reviewers and removed slow-review labels May 27, 2025

damccorm reviewed May 27, 2025

View reviewed changes

damccorm added Next Action: Author and removed Next Action: Reviewers labels May 27, 2025

provided full link path

3fc8c0c

github-actions bot added Next Action: Reviewers and removed Next Action: Author labels Jun 8, 2025

liferoad approved these changes Jun 9, 2025

View reviewed changes

liferoad merged commit df54bf4 into apache:master Jun 9, 2025
96 checks passed



		Instructions for building and testing Beam itself
		are in the [contribution guide](./CONTRIBUTING.md).


		## Python streaming pipelines

		Python [streaming pipeline execution](/documentation/sdks/python-streaming)

Update pypi documentation 30145 #34329

Update pypi documentation 30145 #34329

Uh oh!

Conversation

rakeshcusat commented Mar 18, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

github-actions bot commented Mar 18, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liferoad commented Mar 23, 2025

Uh oh!

rakeshcusat commented Apr 28, 2025

Uh oh!

liferoad commented Apr 28, 2025

Uh oh!

rakeshcusat commented May 5, 2025

Uh oh!

liferoad commented May 5, 2025

Uh oh!

rakeshcusat commented May 5, 2025

Uh oh!

codecov bot commented May 5, 2025

Codecov Report

Uh oh!

github-actions bot commented May 5, 2025

Uh oh!

liferoad commented May 5, 2025

Uh oh!

rakeshcusat commented May 6, 2025

Uh oh!

rakeshcusat commented May 6, 2025

Uh oh!

rakeshcusat commented May 6, 2025

Uh oh!

rakeshcusat commented May 9, 2025

Uh oh!

rakeshcusat commented May 10, 2025

Uh oh!

rakeshcusat commented May 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 23, 2025

Uh oh!

github-actions bot commented May 27, 2025

Uh oh!

damccorm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rakeshcusat commented Jun 8, 2025

Uh oh!

rakeshcusat commented Jun 9, 2025