[BEAM-11770] bump google-cloud-pubsub to 1.110.3(google_cloud_platform_libraries_bom ) #13994

masahitojp · 2021-02-16T12:29:32Z

bump google-cloud-pubsub to 1.110.3, because we follwed dependencies in google_cloud_platform_libraries_bom sets version.

The current master branch depends on 16.3.0 (#13990)

https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/16.3.0/artifact_details.html

^ It depends on com.google.cloud:google-cloud-pubsub:1.110.3
Since this is newer than the current master(1.108.6 ) , I think it's better to update

results of `sh sdks/java/build-tools/beam-linkage-check.sh origin/master beam-11770`

Success

https://gist.github.com/masahitojp/6a292e3d8ff5506383fca3a44805bafb

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2021-02-16T12:52:22Z

Codecov Report

Merging #13994 (df3fd10) into master (aaad864) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #13994      +/-   ##
==========================================
- Coverage   82.98%   82.97%   -0.01%     
==========================================
  Files         469      469              
  Lines       58298    58300       +2     
==========================================
- Hits        48379    48375       -4     
- Misses       9919     9925       +6

Impacted Files	Coverage Δ
...e_beam/portability/api/beam_runner_api_pb2_urns.py
...hon/apache_beam/examples/complete/game/__init__.py
.../apache_beam/portability/api/endpoints_pb2_grpc.py
...apache_beam/typehints/native_type_compatibility.py
...am/runners/interactive/options/capture_limiters.py
..._beam/runners/interactive/user_pipeline_tracker.py
sdks/python/apache_beam/coders/row_coder.py
...ks/python/apache_beam/runners/worker/operations.py
.../examples/snippets/transforms/elementwise/regex.py
...python/apache_beam/typehints/typehints_test_py3.py
... and 928 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aaad864...df3fd10. Read the comment docs.

masahitojp · 2021-02-16T13:33:49Z

Run Java_Examples_Dataflow PreCommit

masahitojp · 2021-02-16T13:33:59Z

Run Website PreCommit

masahitojp · 2021-02-16T14:34:06Z

R: @iemejia
Could you review this PR?

iemejia · 2021-02-16T19:59:44Z

I am not familiar with the consequences of dependency upgrades for Google related dependencies, so in the future for google related dependencies please choose as a reviewer like @kennknowles @chamikaramj or @suztomo

Can someone of you please help review this one or forward this to someone who can. Thanks!
Oh and thanks for your this and all the other recent contributions @masahitojp !

suztomo · 2021-02-16T22:11:47Z

Run Java Postcommit

suztomo · 2021-02-16T22:13:26Z

@masahitojp Would you try running the command below?

sh sdks/java/build-tools/beam-linkage-check.sh origin/master beam-11770

https://cwiki.apache.org/confluence/display/BEAM/Dependency+Upgrades#app-switcher (this has outdated command line parameter. It now takes 2 git branches.)

If there's no new conflict, it looks good to me (I'm not a committer though).

Note that there's back-and-forth (#13765 and #13972) of GCP dependencies in the last couple of weeks. They might interfere this PR; thank you for your patience.

masahitojp · 2021-02-17T11:44:56Z

Hi @suztomo

I tried the command.

sh sdks/java/build-tools/beam-linkage-check.sh origin/master beam-11770

but I got a below errors 😢

Failed to generate the baseline file. Check the build error above.
Checking linkage for origin/beam-11770
Previous HEAD position was 3930d12d47 Merge pull request #13981: [BEAM-11124] bump joda-time to 2.10.10
HEAD is now at 45ba4c5888 [BEAM-11770] bump google-cloud-pubsub to 1.111.0
Running linkage check (validate) for beam-sdks-java-core
using baseline build/linkagecheck/baseline-beam-sdks-java-core.xml
Error: Baseline file not found
Running linkage check (validate) for beam-sdks-java-io-google-cloud-platform
using baseline build/linkagecheck/baseline-beam-sdks-java-io-google-cloud-platform.xml
Error: Baseline file not found
Running linkage check (validate) for beam-runners-google-cloud-dataflow-java
 using baseline build/linkagecheck/baseline-beam-runners-google-cloud-dataflow-java.xml
Error: Baseline file not found
Running linkage check (validate) for beam-sdks-java-io-hadoop-format
 using baseline build/linkagecheck/baseline-beam-sdks-java-io-hadoop-format.xml
Error: Baseline file not found

In the previous jobs, it is displayed as "BUILD SUCCESSFUL ".
Do you know what is causing it? (Manjaro Linux + openjdk version "1.8.0_282")

suztomo · 2021-02-17T12:46:52Z

Check the build error above.

Do you see any error above the line?

…

On Wed, Feb 17, 2021 at 06:45 Masato Nakamura ***@***.***> wrote: Hi @suztomo <https://github.com/suztomo> I tried the command. sh sdks/java/build-tools/beam-linkage-check.sh origin/master beam-11770 but I got a below errors 😢 Failed to generate the baseline file. Check the build error above. Checking linkage for origin/beam-11770 Previous HEAD position was 3930d12 Merge pull request #13981: [BEAM-11124] bump joda-time to 2.10.10 HEAD is now at 45ba4c5888 [BEAM-11770] bump google-cloud-pubsub to 1.111.0 Running linkage check (validate) for beam-sdks-java-core using baseline build/linkagecheck/baseline-beam-sdks-java-core.xml Error: Baseline file not found Running linkage check (validate) for beam-sdks-java-io-google-cloud-platform using baseline build/linkagecheck/baseline-beam-sdks-java-io-google-cloud-platform.xml Error: Baseline file not found Running linkage check (validate) for beam-runners-google-cloud-dataflow-java using baseline build/linkagecheck/baseline-beam-runners-google-cloud-dataflow-java.xml Error: Baseline file not found Running linkage check (validate) for beam-sdks-java-io-hadoop-format using baseline build/linkagecheck/baseline-beam-sdks-java-io-hadoop-format.xml Error: Baseline file not found Do you know what is causing it? (openjdk version "1.8.0_282") — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13994 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAG7PFWCDCIDIFYKWD4NI3S7OT4TANCNFSM4XWLXX2Q> .

suztomo · 2021-02-17T16:20:13Z

I tried and got the same error. Digging further. https://gist.github.com/suztomo/b5a33361ad83c5e5c9054b393ac496c1

Created an issue: BEAM-11827.

suztomo · 2021-02-17T19:46:48Z

@masahitojp It seems that beam-linkage-check.sh has not been migrated properly when this repository moved build script from Gradle to Kotlin. (Your observation helped me to identify the problem. Thank you!)

I raised a pull request to fix the problem #14006. Give me and Beam committers some moment.

suztomo · 2021-02-17T21:29:14Z

@masahitojp I rebased your branch and ran the tool

https://gist.github.com/suztomo/0ba02d93aa0b2f5d9701c50f7d7fd81f It seems that proto-google-cloud-pubsub-v1 needs to have a newer version. Would you check it?

masahitojp · 2021-02-18T10:59:15Z

@suztomo Thank you for your great contribution!!

It seems that proto-google-cloud-pubsub-v1 needs to have a newer version. Would you check it?
I checked it. I can't say anything, but I think it's preferable to update it

beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Line 607 in b8dd9cb

    
           proto_google_cloud_pubsub_v1                : "com.google.api.grpc:proto-google-cloud-pubsub-v1", // google_cloud_platform_libraries_bom sets version

// google_cloud_platform_libraries_bom sets version

maybe　, In other words, I think it will affect this.....

Note that there's back-and-forth (#13765 and #13972) of GCP dependencies in the last couple of weeks.

I think it would be good to put it on hold until this work is finished. 😢

masahitojp · 2021-02-22T12:40:27Z

Run Java PreCommit

…dependencies in google_cloud_platform_libraries_bom sets version. https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/16.3.0/artifact_details.html

masahitojp · 2021-02-23T13:24:24Z

The current master branch depends on 16.3.0 (#13990)
In other words, there was no longer any affect on us.

Note that there's back-and-forth (#13765 and #13972) of GCP dependencies in the last couple of weeks.

#13990 is merged, and this PR is restarted, especially since it does not seem to be causing any problems in testing.

Now I've changed the purpose of this PR.

what

bump google-cloud-pubsub to 1.110.3, because we follwed dependencies in google_cloud_platform_libraries_bom sets version.

The current master branch depends on 16.3.0 (#13990)

https://storage.googleapis.com/cloud-opensource-java-dashboard/com.google.cloud/libraries-bom/16.3.0/artifact_details.html

^ It depends on com.google.cloud:google-cloud-pubsub:1.110.3
Since this is newer than the current master(1.108.6 ) , I think it's better to update

why(background)

The apache beam specifies the versions of the following packages

com.google.cloud:google-cloud-pubsub

But this package has dependencies with the following packages but this one is specified by google_cloud_platform_libraries_bom

com.google.api.grpc:proto-google-cloud-pubsub-v1
com.google.api.grpc:grpc-google-cloud-pubsub-v1

The dependencies of this package are difficult to understand for people who don't know about it, so there is a possibility that the dependencies will be broken unintentionally in the future. For this reason, I thought it would be better for me to unify them into google_cloud_platform_libraries_bom

^ It depends on com.google.cloud:google-cloud-pubsub:1.110.3
Since this is newer than the current master(1.108.6 ) , I think it's better to update

results of sh sdks/java/build-tools/beam-linkage-check.sh origin/master beam-11770

Success

https://gist.github.com/masahitojp/6a292e3d8ff5506383fca3a44805bafb
(Thanks to your support @suztomo )

Could you re-review this PR?
R: @TheNeuralBit @iemejia

suztomo · 2021-02-23T14:49:42Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

        google_cloud_dataflow_java_proto_library_all: "com.google.cloud.dataflow:google-cloud-dataflow-java-proto-library-all:0.5.160304",
        google_cloud_datastore_v1_proto_client      : "com.google.cloud.datastore:datastore-v1-proto-client:1.6.3",
-        google_cloud_pubsub                         : "com.google.cloud:google-cloud-pubsub:$google_cloud_pubsub_version",
+        google_cloud_pubsub                         : "com.google.cloud:google-cloud-pubsub", // google_cloud_platform_libraries_bom sets version


This is a nice enhancement. No need to maintain google-cloud-pubsub separately. Thank you.

+1, thank you

masahitojp force-pushed the beam-11770 branch from 45ba4c5 to 04f0e2c Compare February 21, 2021 00:36

masahitojp force-pushed the beam-11770 branch from 7110874 to df3fd10 Compare February 23, 2021 06:50

masahitojp changed the title ~~[BEAM-11770] bump google-cloud-pubsub to 1.111.0~~ [BEAM-11770] bump google-cloud-pubsub to 1.110.3(google_cloud_platform_libraries_bom ) Feb 23, 2021

suztomo approved these changes Feb 23, 2021

View reviewed changes

TheNeuralBit approved these changes Feb 23, 2021

View reviewed changes

TheNeuralBit merged commit 7b2c4dc into apache:master Feb 23, 2021

masahitojp deleted the beam-11770 branch February 23, 2021 16:00

[BEAM-11770] bump google-cloud-pubsub to 1.110.3(google_cloud_platform_libraries_bom ) #13994

[BEAM-11770] bump google-cloud-pubsub to 1.110.3(google_cloud_platform_libraries_bom ) #13994

Uh oh!

Conversation

masahitojp commented Feb 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

results of sh sdks/java/build-tools/beam-linkage-check.sh origin/master beam-11770

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

codecov bot commented Feb 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

masahitojp commented Feb 16, 2021

Uh oh!

masahitojp commented Feb 16, 2021

Uh oh!

masahitojp commented Feb 16, 2021

Uh oh!

iemejia commented Feb 16, 2021

Uh oh!

suztomo commented Feb 16, 2021

Uh oh!

suztomo commented Feb 16, 2021

Uh oh!

masahitojp commented Feb 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

suztomo commented Feb 17, 2021 via email

Uh oh!

suztomo commented Feb 17, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

suztomo commented Feb 17, 2021

Uh oh!

suztomo commented Feb 17, 2021

Uh oh!

masahitojp commented Feb 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

masahitojp commented Feb 22, 2021

Uh oh!

masahitojp commented Feb 23, 2021

what

why(background)

results of sh sdks/java/build-tools/beam-linkage-check.sh origin/master beam-11770

Uh oh!

suztomo Feb 23, 2021

Choose a reason for hiding this comment

Uh oh!

TheNeuralBit Feb 23, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

masahitojp commented Feb 16, 2021 •

edited

Loading

results of `sh sdks/java/build-tools/beam-linkage-check.sh origin/master beam-11770`

codecov bot commented Feb 16, 2021 •

edited

Loading

masahitojp commented Feb 17, 2021 •

edited

Loading

suztomo commented Feb 17, 2021 •

edited

Loading

masahitojp commented Feb 18, 2021 •

edited

Loading