Skip to content
This repository was archived by the owner on May 12, 2026. It is now read-only.

Fix: Not loosing the access token when calling UserCredentials#ToBuil… - #993

Merged
TimurSadykov merged 6 commits into
googleapis:mainfrom
qcastel:bugfix/992-fix-usercredentials-builder
Dec 2, 2022
Merged

TimurSadykov merged 6 commits into
googleapis:mainfrom
qcastel:bugfix/992-fix-usercredentials-builder

Conversation

@qcastel

@qcastel qcastel commented Sep 7, 2022

Copy link
Copy Markdown
Contributor

Fixes #992 ☕️

@qcastel
qcastel requested a review from a team September 7, 2022 13:05
@conventional-commit-lint-gcf

conventional-commit-lint-gcf Bot commented Sep 7, 2022 •

Copy link
Copy Markdown

🤖 I detect that the PR title and the commit message differ and there's only one commit. To use the PR title for the commit history, you can use Github's automerge feature with squashing, or use automerge label. Good luck human!

-- conventional-commit-lint bot
https://conventionalcommits.org/

@product-auto-label product-auto-label Bot added the size: s Pull request size is small. label Sep 7, 2022
@TimurSadykov TimurSadykov changed the title #992 Not loosing the access token when calling UserCredentials#ToBuil… #992 Fix: Not loosing the access token when calling UserCredentials#ToBuil… Sep 8, 2022
@TimurSadykov TimurSadykov changed the title #992 Fix: Not loosing the access token when calling UserCredentials#ToBuil… Fix: Not loosing the access token when calling UserCredentials#ToBuil… Sep 8, 2022

@TimurSadykov TimurSadykov left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! Few minor comments.

Comment thread oauth2_http/javatests/com/google/auth/oauth2/UserCredentialsTest.java Outdated
Comment thread oauth2_http/javatests/com/google/auth/oauth2/UserCredentialsTest.java Outdated
@qcastel
qcastel requested a review from TimurSadykov September 8, 2022 06:28
@qcastel

qcastel commented Sep 8, 2022

Copy link
Copy Markdown
Contributor Author

@TimurSadykov As I was making the test a little bit more generic than just checking the access token, I found out that two methods of the builders were missing.
The test is now covering that every attributes of the credentials are copied over correctly during the ToBuilder()

@qcastel

qcastel commented Oct 24, 2022

Copy link
Copy Markdown
Contributor Author

@TimurSadykov any news on this one?

@TimurSadykov TimurSadykov left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for a delay.

LGTM, i will merge once the last nit is fixed.

}

@Test
public void UserCredentials_ToBuilder_CopyEveryAttributes() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, every part has to stat with lowercase, like this and a typo fix: userCredentials_toBuilder_copyEveryAttribute

Comment thread oauth2_http/javatests/com/google/auth/oauth2/UserCredentialsTest.java Outdated

@TimurSadykov TimurSadykov left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@TimurSadykov
TimurSadykov merged commit 84afdb8 into googleapis:main Dec 2, 2022
@qcastel
qcastel deleted the bugfix/992-fix-usercredentials-builder branch December 2, 2022 09:08
dongjoon-hyun added a commit to apache/spark that referenced this pull request Aug 21, 2023
### What changes were proposed in this pull request?

This PR aims to upgrade gcs-connector to 2.2.17.

### Why are the changes needed?

To have the latest auth updates,

- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
  - GoogleCloudDataproc/hadoop-connectors#1041

```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```

- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
  - googleapis/google-auth-library-java#1100
  - googleapis/google-auth-library-java#993

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs and manual tests.

**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```

**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
    -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
    -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
    -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #42588 from dongjoon-hyun/SPARK-44898.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
valentinp17 pushed a commit to valentinp17/spark that referenced this pull request Aug 24, 2023
### What changes were proposed in this pull request?

This PR aims to upgrade gcs-connector to 2.2.17.

### Why are the changes needed?

To have the latest auth updates,

- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
  - GoogleCloudDataproc/hadoop-connectors#1041

```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```

- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
  - googleapis/google-auth-library-java#1100
  - googleapis/google-auth-library-java#993

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs and manual tests.

**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```

**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
    -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
    -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
    -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42588 from dongjoon-hyun/SPARK-44898.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
szehon-ho pushed a commit to szehon-ho/spark that referenced this pull request Feb 7, 2024
### What changes were proposed in this pull request?

This PR aims to upgrade gcs-connector to 2.2.17.

### Why are the changes needed?

To have the latest auth updates,

- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
  - GoogleCloudDataproc/hadoop-connectors#1041

```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```

- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
  - googleapis/google-auth-library-java#1100
  - googleapis/google-auth-library-java#993

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs and manual tests.

**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```

**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
    -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
    -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
    -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42588 from dongjoon-hyun/SPARK-44898.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
prabhjyotsingh pushed a commit to acceldata-io/spark3 that referenced this pull request Feb 8, 2025
### What changes were proposed in this pull request?

This PR aims to upgrade gcs-connector to 2.2.17.

### Why are the changes needed?

To have the latest auth updates,

- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
  - GoogleCloudDataproc/hadoop-connectors#1041

```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```

- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
  - googleapis/google-auth-library-java#1100
  - googleapis/google-auth-library-java#993

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs and manual tests.

**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```

**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
    -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
    -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
    -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42588 from dongjoon-hyun/SPARK-44898.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit 109e9cd)
prabhjyotsingh pushed a commit to acceldata-io/spark3 that referenced this pull request Feb 8, 2025
### What changes were proposed in this pull request?

This PR aims to upgrade gcs-connector to 2.2.17.

### Why are the changes needed?

To have the latest auth updates,

- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
  - GoogleCloudDataproc/hadoop-connectors#1041

```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```

- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
  - googleapis/google-auth-library-java#1100
  - googleapis/google-auth-library-java#993

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs and manual tests.

**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```

**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
    -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
    -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
    -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42588 from dongjoon-hyun/SPARK-44898.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit 109e9cd)
(cherry picked from commit 0cb12e9)
prabhjyotsingh pushed a commit to acceldata-io/spark3 that referenced this pull request Feb 8, 2025
### What changes were proposed in this pull request?

This PR aims to upgrade gcs-connector to 2.2.17.

### Why are the changes needed?

To have the latest auth updates,

- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
  - GoogleCloudDataproc/hadoop-connectors#1041

```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```

- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
  - googleapis/google-auth-library-java#1100
  - googleapis/google-auth-library-java#993

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs and manual tests.

**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```

**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
    -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
    -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
    -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42588 from dongjoon-hyun/SPARK-44898.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit 109e9cd)
(cherry picked from commit 0cb12e9)
shubhluck pushed a commit to acceldata-io/spark3 that referenced this pull request May 16, 2025
### What changes were proposed in this pull request?

This PR aims to upgrade gcs-connector to 2.2.17.

### Why are the changes needed?

To have the latest auth updates,

- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
  - GoogleCloudDataproc/hadoop-connectors#1041

```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```

- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
  - googleapis/google-auth-library-java#1100
  - googleapis/google-auth-library-java#993

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs and manual tests.

**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```

**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
    -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
    -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
    -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42588 from dongjoon-hyun/SPARK-44898.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit 109e9cd)
(cherry picked from commit 0cb12e9)
senthh pushed a commit to acceldata-io/spark3 that referenced this pull request May 26, 2025
### What changes were proposed in this pull request?

This PR aims to upgrade gcs-connector to 2.2.17.

### Why are the changes needed?

To have the latest auth updates,

- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
  - GoogleCloudDataproc/hadoop-connectors#1041

```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```

- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
  - googleapis/google-auth-library-java#1100
  - googleapis/google-auth-library-java#993

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs and manual tests.

**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```

**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
    -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
    -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
    -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42588 from dongjoon-hyun/SPARK-44898.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit 109e9cd)
(cherry picked from commit 0cb12e9)
shubhluck pushed a commit to acceldata-io/spark3 that referenced this pull request Sep 3, 2025
### What changes were proposed in this pull request?

This PR aims to upgrade gcs-connector to 2.2.17.

### Why are the changes needed?

To have the latest auth updates,

- https://github.com/GoogleCloudDataproc/hadoop-connectors/releases/tag/v2.2.17 (2023-08-15)
  - GoogleCloudDataproc/hadoop-connectors#1041

```xml
- <google.auth.version>1.12.1</google.auth.version>
+ <google.auth.version>1.14.0</google.auth.version>
- <google.cloud-storage.bom.version>2.23.0</google.cloud-storage.bom.version>
+ <google.cloud-storage.bom.version>2.25.0</google.cloud-storage.bom.version>
```

- https://github.com/googleapis/google-auth-library-java/releases/tag/v1.14.0 (2022-12-06)
  - googleapis/google-auth-library-java#1100
  - googleapis/google-auth-library-java#993

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs and manual tests.

**BUILD**
```
dev/make-distribution.sh -Phadoop-cloud
```

**TEST**
```
$ cd dist
$ export KEYFILE=~/.ssh/apache-spark-k8s-54ccbe6102d9.json
$ export EMAIL=$(jq -r '.client_email' < $KEYFILE)
$ export PRIVATE_KEY_ID=$(jq -r '.private_key_id' < $KEYFILE)
$ export PRIVATE_KEY="$(jq -r '.private_key' < $KEYFILE)"
$ bin/spark-shell \
    -c spark.hadoop.fs.gs.auth.service.account.email=$EMAIL \
    -c spark.hadoop.fs.gs.auth.service.account.private.key.id=$PRIVATE_KEY_ID \
    -c spark.hadoop.fs.gs.auth.service.account.private.key="$PRIVATE_KEY"
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/08/21 12:17:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id = local-1692645442153).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 4.0.0-SNAPSHOT
      /_/

Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_312)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.read.text("gs://apache-spark-bucket/README.md").count()
res0: Long = 124

scala> spark.read.orc("examples/src/main/resources/users.orc").write.mode("overwrite").orc("gs://apache-spark-bucket/users.orc")

scala> spark.read.orc("gs://apache-spark-bucket/users.orc").show()
+------+--------------+----------------+
|  name|favorite_color|favorite_numbers|
+------+--------------+----------------+
|Alyssa|          NULL|  [3, 9, 15, 20]|
|   Ben|           red|              []|
+------+--------------+----------------+
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42588 from dongjoon-hyun/SPARK-44898.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit 109e9cd)
(cherry picked from commit 0cb12e9)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

size: s Pull request size is small.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Access token lost in transit, when using UserCredentials ToBuilder()

2 participants