Skip to content

datastore emulator sometimes crashes, which causes test flakyness #3276

Closed
Closed
datastore emulator sometimes crashes, which causes test flakyness#3276
Assignees
chingor13
Labels
api: datastoreIssues related to the Datastore API.type: questionRequest for information or clarification. Not an issue.

Description

@ensonic
ensonic
opened on May 16, 2018

We're using bazel to build our software. At the Ci we noticed test flakyness. When I run the tests localy like:

bazel test --runs_per_test=10000 //src/java/com/example/foo:FooTest

I get about 5 failures that look like this:

SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@74287ea3
com.google.cloud.datastore.DatastoreException: I/O error
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:129)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.translate(HttpDatastoreRpc.java:114)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:182)
	at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:178)
	at com.google.cloud.datastore.DatastoreImpl$1.call(DatastoreImpl.java:174)
	at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
	at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
	at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
	at com.google.cloud.datastore.DatastoreImpl.runQuery(DatastoreImpl.java:173)
	at com.google.cloud.datastore.QueryResultsImpl.sendRequest(QueryResultsImpl.java:73)
	at com.google.cloud.datastore.QueryResultsImpl.<init>(QueryResultsImpl.java:57)
	at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:167)
	at com.google.cloud.datastore.DatastoreImpl.run(DatastoreImpl.java:158)
	at com.cloudrobotics.map.MapService.deleteOccupancyGrid(MapService.java:519)
	at com.cloudrobotics.map.MapService.deleteLayer(MapService.java:252)
	at com.cloudrobotics.map.MapService.deleteLayer(MapService.java:272)
	at com.cloudrobotics.map.v1alpha1.MapServiceGrpc$MethodHandlers.invoke(MapServiceGrpc.java:1755)
	at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:272)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:653)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:49)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener.halfClosed(ServerImpl.java:664)
	at io.grpc.inprocess.InProcessTransport$InProcessStream$InProcessClientStream.halfClose(InProcessTransport.java:621)
	at io.grpc.internal.ClientCallImpl.halfClose(ClientCallImpl.java:422)
	at io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:47)
	at io.grpc.ForwardingClientCall.halfClose(ForwardingClientCall.java:47)
	at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:271)
	at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:177)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:119)
	at com.cloudrobotics.map.v1alpha1.MapServiceGrpc$MapServiceBlockingStub.deleteLayer(MapServiceGrpc.java:1371)
	at com.cloudrobotics.map.MapServiceTest.testGetOccupancyGrid_afterDeleteLayer_throwsNotFound(MapServiceTest.java:472)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at com.google.testing.junit.runner.internal.junit4.CancellableRequestFactory$CancellableRunner.run(CancellableRequestFactory.java:89)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:138)
	at com.google.testing.junit.runner.junit4.JUnit4Runner.run(JUnit4Runner.java:112)
	at com.google.testing.junit.runner.BazelTestRunner.runTestsInSuite(BazelTestRunner.java:144)
	at com.google.testing.junit.runner.BazelTestRunner.main(BazelTestRunner.java:82)
Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
	at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:226)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:195)
	at com.google.datastore.v1.client.Datastore.runQuery(Datastore.java:119)
	at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.runQuery(HttpDatastoreRpc.java:180)
	... 57 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at sun.net.www.http.HttpClient.New(HttpClient.java:339)
	at sun.net.www.http.HttpClient.New(HttpClient.java:357)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
	at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1022)
	at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1020)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1019)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
	at sun.net.www.protocol.http.HttpURLConnection.access$100(HttpURLConnection.java:91)
	at sun.net.www.protocol.http.HttpURLConnection$8.run(HttpURLConnection.java:1283)
	at sun.net.www.protocol.http.HttpURLConnection$8.run(HttpURLConnection.java:1281)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:782)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1280)
	at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:77)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
	at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:183)
	... 59 more

Right now we're using a helper like this to mitigate it a bit
https://gist.github.com/ensonic/a1b64be415f537b59bcc951883b29596
but of course this won't help tests where datastore looses connection in the middle of the test.

Activity

  1. yihanzhen
    added
    type: questionRequest for information or clarification. Not an issue.
    api: datastoreIssues related to the Datastore API.
    on May 18, 2018

  2. chingor13 commented on Dec 5, 2018

    @chingor13
    chingor13
    on Dec 5, 2018
    Contributor

    The datastore emulator is not guaranteed to perfectly replicate the backend server, especially if you're trying to load test it - it looks like you're running the tests 10,000 times.

    This repository does not contain the actual datastore emulator, so if this is still happening, you can file an issue against the datastore emulator itself: https://issuetracker.google.com/issues/new?component=187197&template=0

  3. chingor13
    closed this as completedon Dec 5, 2018

  4. ensonic commented on Dec 8, 2018

    @ensonic
    ensonic
    on Dec 8, 2018
    Author

    The 10000 was not a load test, it was used to demonstrate that there is a bug. Anyway, this is just one of the reasons that we are migrating away from datastore,

  5. zaddok commented on Feb 6, 2019

    @zaddok
    zaddok
    on Feb 6, 2019

    Yes the emulator appears extremely buggy. If you trigger 10 concurrent go functions, and have them each insert 100 records. The emulator will freeze 50% of the time.

    (Our app calls web services and stores data, if the web service calls run concurrently, the emulator will break. I am guessing that the cause of the problem is concurrent attempts to write the same record)

  6. JustinBeckwith
    assigned
    chingor13
    on Feb 1, 2021
  7. suztomo
    added a commit that references this issue on Mar 9, 2026

    chore(deps): update dependency com.google.cloud:google-cloud-bigtable…

    fa1a6d8
  8. chingor13
    added 3 commits that reference this issue on Mar 24, 2026

    deps: update junit5 monorepo to v5.11.2 (#3276)

    685f46d

    chore(main): release 2.47.0 (#3261)

    25e9c85

    feat: add option for cancelling queries when closing client (#3276)

    526e1fb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

api: datastoreIssues related to the Datastore API.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions