NDB: Sketch high level strategy #6908

chrisrossi · 2018-12-12T16:22:08Z

Diff base is #6888

This code is intentionally sketchy. It contains very partial implementations, non-conformant docstrings, and less than 100% test coverage. This code is not being proposed for merge but is offered as a sketch for discussion of general strategy and/or a guide for me to follow to produce a more complete, real implementation.

The main point of this was to provide the bare minimum to make the single system test pass. This test uses google.cloud.datastore to write an entity to Datastore and uses google.cloud.ndb to read it back out of Datastore as an instance of an NDB model class. I suggest looking at the test and thinking about if that is the correct user experience.

The salient points are:

google.cloud.ndb.Client as a parallel to google.cloud.datastore.Client, as a means of bootstrapping NDB and its connection to Datastore in a way familiar to GCP users. This is the entry point for establishing credentials, the working project/app, and the namespace, exactly like google.cloud.datastore.
Client.context() as a context manager for encapsulating code that uses NDB. Under the hood there is a thread local state which legacy NDB previously managed in GAE by attaching it to the running HTTP request, an fixture which is no longer available in GCP. Use of the context manager means most existing projects which use NDB can simply wrap their code in a context and their existing business code continues to work, as it can access any needed state from the running context.
_future.Future as a thin wrapper around Futures returned by gRPC. This class is meant to be subclassed and can wrap any number of gRPC Futures and compute an arbitrary result based on the results of those wrapped futures. As an example, there is key.GetByKeyFuture which wraps the return value of a call to Lookup.future() in Datastore's gRPC API and marsahall's an NDB entity from the returned Datastore protocol buffer. It's not doing any async on its own, here, just wrapping the raw gRPC async calls to provide Futures that yield arbitrarily computed results.

I think pretty much everything except the tasklets surface can be implemented in this way. I haven't fully worked out, yet, how this will integrate with the tasklets mini-framework but I think having a lot of the rest fleshed out will give us a good starting point.

…n is just going to fix anyway.

ndb/src/google/cloud/ndb/_api.py

+    state = _runstate.current()
+    request = datastore_pb2.LookupRequest(project_id=state.project)
+    key_pb = request.keys.add()
+    key_pb.CopyFrom(key._key.to_protobuf())


ndb/src/google/cloud/ndb/_future.py

+
+"""A Future class."""
+
+_NOT_COMPUTED = object()


ndb/src/google/cloud/ndb/_future.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""A Future class."""


ndb/src/google/cloud/ndb/_future.py

+
+    def get_result(self):
+        """Get the computed result for this future."""
+        if self._result is _NOT_COMPUTED:


ndb/src/google/cloud/ndb/_future.py

+    This class is meant to be subclassed with code to compute a result from the
+    results of the wrapped gRPC futures. :method:`_compute_result` should be
+    overriden to compute a result from the completed gRPC calls.
+    """


ndb/src/google/cloud/ndb/key.py

+
+    def _compute_result(self, result):
+        from google.cloud.ndb import model  # avoid circular import
+        return model._entity_from_protobuf(result.found[0].entity)


ndb/src/google/cloud/ndb/key.py

+class GetByKeyFuture(_future.Future):
+    """Future for looking up entities by Key."""
+
+    def _compute_result(self, result):
