Open
Conversation
Arvados-DCO-1.1-Signed-off-by: Joshua C. Randall <[email protected]>
…there to be nonzero BytesUsed Arvados-DCO-1.1-Signed-off-by: Joshua C. Randall <[email protected]>
…iner Arvados-DCO-1.1-Signed-off-by: Joshua C. Randall <[email protected]>
Arvados-DCO-1.1-Signed-off-by: Joshua C. Randall <[email protected]>
a42ccc4 to
57262cf
Compare
Member
|
@jrandall this is awesome! Aside from the regular code review, we need to figure out what our process is for integrating features we don't use ourselves. Looks like there are some tests, so that's good. I'm curious, do you run a few keepstore servers and have them talk to the Ceph nodes, or run a keepstore server on every compute node? |
Contributor
Author
|
We generally run a keepstore on every compute node (and also some dedicated ones for loading data in and out from external sites via keep-proxy/keep-web) |
Member
|
(see further discussion at https://dev.arvados.org/issues/13642) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements a librados (ceph) backend for keepstore.
Uses the go-ceph (https://github.com/ceph/go-ceph) rados bindings to the librados library.
Implements RadosVolume and TestableRadosVolume, and includes mock code for testing without access to an actual ceph cluster (although go-ceph is required even for the mock tests because some go-ceph structs are used directly by the mocks).
Runs the standard set of generic tests and some additional tests on context cancellation.
In addition to the mock implementation, a real implemenation can be used for the tests (although the context cancellation tests are skipped in that case as the required races cannot be guaranteed when running against a real-world system). This is specified in much the same way as with the Azure tests, by passing a test argument (
-test.rados-pool-volume <poolname>) and then setting other necessary-rados-*connection parameters as normal.A helper script (
run-rados-test-with-docker-ceph-demo.sh) is provided that spins up a "real" ceph cluster running in aceph/demodocker container (https://hub.docker.com/r/ceph/demo/) and runs the keepstore rados tests against it. It should work on ubuntu and debian, but I haven't tested portability against any other systems, and also note that Ceph OSDs (including those inceph/demo) are somewhat finicky about the capabilities of the underlying filesystem, so it may not work on all docker volume backends (see http://docs.ceph.com/docs/jewel/rados/configuration/filesystem-recommendations/#filesystem-background-info).In my hands, this implementation passes all tests using the mocks, using the
ceph/demodocker container, and also using an actual production ceph cluster on real hardware.This PR also includes one small change to the generic tests, which was to add a
PutRawcall to thetestStatustest involume_generic_test.gobefore callingStatusand then asserting that there should be non-zeroBytesUsed. My argument here is that it is not reasonable to require the backend to have nonzero bytes used if there are no objects in the backend store!