datastore.it.ITDatastoreTest: testTransactionWithRead and testTransactionWithQuery fail if the test DB type is Firestore in Datastore mode #497
Description
These tests always timeout when testing with a Firestore in Datastore Mode DB. I think the reason they pass in continuous tests is because the type of DB they use (in project gcloud-devel) is "Cloud Datastore".
I believe the root cause of the timeouts is the transaction isolation and consistency behavior differences between the 2 Datastore DB types, documented here and here.
TL;DR
With Firestore in datastore mode when 2 concurrent transactions read or write the same data, the lock held by one transaction can delay the other transaction (hence the timeouts), but with Cloud Datastore the contention is reported as an error at commit.
Environment details
All environments
Steps to reproduce
- Create a Firestore in Datastore Mode DB (or use an existing DB, if its type is known to be "Firestore in Datastore Mode")
- Set the GOOGLE_CLOUD_PROJECT env var to the project ID that owns the DB
- Set the GOOGLE_APPLICATION_CREDENTIALS env var to the absolute path of the key file for a service account that has permission to write to the DB
- run the tests, e.g.
GOOGLE_CLOUD_PROJECT=some-project GOOGLE_APPLICATION_CREDENTIALS=/tmp/sa-key.json mvn -P enable-integration-tests verify
Stack trace
it.ITDatastoreTest.testTransactionWithQuery
org.junit.runners.model.TestTimedOutException: test timed out after 100 seconds
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:457)
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
at sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1332)
at sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:948)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:149)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:97)
at com.google.datastore.v1.client.Datastore.commit(Datastore.java:85)
at com.google.cloud.datastore.spi.v1.HttpDatastoreRpc.commit(HttpDatastoreRpc.java:162)
at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:564)
at com.google.cloud.datastore.DatastoreImpl$5.call(DatastoreImpl.java:561)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
at com.google.cloud.datastore.DatastoreImpl.commit(DatastoreImpl.java:560)
at com.google.cloud.datastore.DatastoreImpl.commitMutation(DatastoreImpl.java:553)
at com.google.cloud.datastore.DatastoreImpl.put(DatastoreImpl.java:513)
at com.google.cloud.datastore.DatastoreHelper.put(DatastoreHelper.java:53)
at com.google.cloud.datastore.DatastoreImpl.put(DatastoreImpl.java:488)
at com.google.cloud.datastore.it.ITDatastoreTest.testTransactionWithQuery(ITDatastoreTest.java:284)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
it.ITDatastoreTest.testTransactionWithRead
this test would also timeout if this code was updating ENTITY3 as stated in the comment.
java.lang.AssertionError: Expecting a failure
at org.junit.Assert.fail(Assert.java:89)
at com.google.cloud.datastore.it.ITDatastoreTest.testTransactionWithRead(ITDatastoreTest.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)