JVM Hangs on gRPC if you don't close Spanner Object

I ran across a path with the Java Spanner client that if you create several Spanner instances and do not explicitly close them the JVM will hang (it will not terminate it's process after code execution).

To reproduce the behavior create an object that wraps Spanner creation. Create the object several times in a loop. Then execute some additional code.

You can see the hanging wait on KQueueArrayWrapper.

Thanks.

We may creating non-daemon threads somewhere.

@ericandrewschmidt

@ericandrewschmidt Does this only happen if you create multiple instance of Spanner object?
Also are you creating multiple SpannerOptions or are you creating a single SpannerOptions and multiple Spanner instances from that?
It will be useful if you can put a short code snippet showing exactly how you are creating the spanner objects.
@garrettjonesgoogle Only explicit thread we create in spanner library is a daemon thread:
https://github.com/GoogleCloudPlatform/google-cloud-java/blob/master/google-cloud-spanner/src/main/java/com/google/cloud/spanner/spi/v1/WatchdogInterceptor.java#L106

@ericandrewschmidt

@ericandrewschmidt Ping!

I'll close this due to inactivity. Please reopen if you are still seeing this problem.

I experience similar thing.
Our application did:

SpannerOptions options = SpannerOptions.newBuilder().setProjectId(projectId).build();
Spanner spanner = options.getService();
dbClient = spanner.getDatabaseClient(DatabaseId.of(options.getProjectId(), instance, database));

In our application Spanner is not closed gracefully and the application hangs after dbClient read/write some requests. It has been hanging for days until reboot manually.

I am adding a fix to our application to fix this. But hanging without giving any error message isn't really helpful for debugging...

@TJ-Liu

@TJ-Liu which version of the client are you using?

@sduskis

I can confirm this behavior in all recent versions of the client library. I tested from 1.14.0 to 1.19.0, and they all show this behavior using this simple application:

  public static void main(String[] args) throws IOException {
    System.out.println("Starting application");
    SpannerOptions options = SpannerOptions.newBuilder().setProjectId("test-project")
        .setCredentials(GoogleCredentials.fromStream(new FileInputStream("key.json")))
        .build();
    Spanner spanner = options.getService();
    DatabaseClient client = spanner.getDatabaseClient(DatabaseId.of("test-project", "test-instance", "test-db"));
    try(ResultSet rs = client.singleUse().executeQuery(Statement.of("SELECT 1" ))) {
      while(rs.next()) {
        // do nothing
      }
    }
    System.out.println("Finished application");
  }

It seems it is caused by the default executor that is created here:

google-cloud-java/google-cloud-clients/google-cloud-core-grpc/src/main/java/com/google/cloud/grpc/GrpcTransportOptions.java

Line 57 in 07c3d62

ScheduledThreadPoolExecutor service = new ScheduledThreadPoolExecutor(8);

Changing that executor to create daemon threads solves the problem and will allow the above example program to shutdown automatically at the end of the main method.

@sduskis I would think that this is something that should be changed in google-cloud-core-grpc and not something that the Spanner library should try to work around. Or is there a reason why this class is not creating daemon threads?

Note to all above: Even though this problem should be fixed, it is still always a good idea to close a Spanner instance at the end of your application. Spanner#close() will close all connections and clean up all sessions that have been created on the backend Cloud Spanner servers. Failing to do so, will keep sessions living longer on the backend than is necessary. Spanner has a limit on the number of open sessions per node, and you could run into this limit if you fail to close your Spanner instances.

default executor used non-daemon threads

default executor used non-daemon threads (#5194)

chore(deps): update dependency com.google.cloud:libraries-bom to v26.…

feat: Adding CDC sample (#2263)

chore: cleanup release-please config (googleapis#2315)

docs: Add samples and tests for ingestion from Kafka sources (#2315)

Navigation Menu

Repository navigation

JVM Hangs on gRPC if you don't close Spanner Object #2315

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

JVM Hangs on gRPC if you don't close Spanner Object #2315

Description

Activity

garrettjonesgoogle commented on Aug 10, 2017

vkedia commented on Aug 22, 2017

vkedia commented on Sep 12, 2017

pongad commented on Jan 19, 2018

TJ-Liu commented on May 15, 2019

sduskis commented on May 16, 2019

olavloite commented on May 16, 2019

olavloite commented on May 16, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions