nit: comnpatibility -> compatibility

Shouldn't this configuration parameter have just "database_uri" and separate configuration parameters for each database that we want to test against? We might want to run same test suite against multiple versions of mongodb to observe functional differences across new major versions.

I'm not sure what the term 'reference instance' implies here. The suite should have a self-contained set of expectations.

That being said, being able to run in a comparison mode against two (or more?) URIs does seem like a good idea.

Yes I think we should make this generic to be able to test against any MongoDB compatible engine. We could do something like this.

bash
pytest --engine documentdb=mongodb://localhost:27017 --engine mongodb-7.0=mongodb://mongo:27017
Or config file
pytest --config engines.yaml

If we have tags based, IMO it suffice.

Can you please share an example why we want one test to depend on another?

Yes technically we can handle the required dependency as a setup for the test so we shouldn't really need dependent tests.

Should this always be presented at tag level? If we try to come up with general compatibility %, it always gets influenced by one or another tag which has large number of tests.

Also, under every test suite, should we have set of smoke tests identified by "smoke" tag which determines if given feature is even implemented in target database before deciding to run whole lot of tests against it?

We should provide both tag-level and overall metrics to avoid skewing. Overall compatibility should be weighted or normalized to prevent high-volume tags from dominating.

Yes, having smoke tagged tests would be valuable for quick feature detection and failing fast on unsupported features before running comprehensive test suites.

Something like that would need to coordinate closely with the infrastructure that manages driver and engine instances (i.e. github workflows, AWS, or Azure).

Not against the idea, but I think it'll be a while before the complexity is worth it.

Yes this is just a future prospect and not something we would focus on now.

'Runner' and 'Executor' are a little too similar. Can we call it the 'Orchestrator' or something similar?

I'm not sure combining these into one test suite has a lot of value. The MongoDB service tests are not actively maintained and the signal we'll get from them is noisy since they include things that we'll likely never support (like replication set management). Their licensing also puts some restrictions on how they can be run.

Agreed - additionally what are the licensing requirements for doing this?

I'm not really proposing to combine them. I was proposing that we run MongoDB service tests separately to measure compatibility percentage. We would need to engage legal before we actually do this to make sure we have the green light for doing this.

I'm a little skeptical about us trying to enable this from the framework layer. We might be better off focusing on a small, portable framework that makes it easy for the user to fit into whatever workflow they have.

This was meant to address future prospects. For now we will have the framework focus on the core correctness and compatibility testing.

It might be better to prioritize making this easy to run directly. i.e. something like a python package with a small requirements.txt that's easy to get working in any setup. Using docker might otherwise be more difficult to work with than common python environment management tools like conda or venv.

Even for our use it might be a little unwieldy since github workflows run in a docker container already and nested docker containers can be fickle.

I think supporting both approaches to run the tests would be useful. Contributors can clone the git repo and run the tests directly from the package. Other users wanting to run these tests against a remote cluster can leverage the docker image.

I think comparison between multiple engines should be an optional mode. The core responsibility should be to compare against defined expectations.

Yes that was the goal but wording in my RFC was confusing. See my comment below which addresses a similar concern.

Can you elaborate on how the communication will work here? Are the 'results' here just the pass/fail or are we capturing and transferring all of the output from each engine so that the analyzer can inspect it?

The final outcome for each test case is either passed or failed.

Not sure it's relevant at this stage of the design, but python's global interpreter lock might severely limit the actual amount of parallelism you get out of the runner with a simple threadpool. Multirocessing might be required for scale.

With the pytest approach this concern would be resolved since pytest-xdist uses multiprocessing for parallel test execution.

Can we elaborate more on how we want to model the directory structure? Presumably we won't have a single test_aggregate.py for all stages and aggregation operators.

I suggest to use the same structure of MongoDB documentation tree

The directory structure shown was just a high-level example. We'd definitely need more granular organization. For large features like find, we'd split by sub-functionality: basic queries, query operators, logical operators, projections, sorting, cursors, etc.

Is the idea to have one setup per file that initiates data for all test cases? Some tests might need their own specialized fixtures or a data generator function, and centralizing it in one place might be hard to maintain. Also, tests that write data might each need their own version of the same fixture.

Would it make more sense to model this with pytest fixtures instead?

i.e. a fixture that'll create a collection based on the test name and drop it afterward

@pytest.fixture
def collection(request):
    collection = my_db[request.node.name]
    yield collection
    collection.drop()

Used like this

@pytest.mark.collection
def test_something(collection):
    collection.find(...)
    assert(whatever())

A similar thing can be done for load_fixture or other resources, and we could have shared vs individual flavors for efficiency.

I think for the most test cases, it should only use minimum data required for the test case purpose.
For example:
// Test $add returns null if input has null.
db.test.drop()
db.test.insert({})
db.test.find({}, {res: {$add: [1, null]}})
// Test $add can use field expression
db.test.drop()
db.test.insert({a: null})
db.test.find({}, {res: {$add: [1, "$a"]}})

1. It's more readable what the specific case this test covers. If we have a complex document or many rows, we cannot rely on reviewer to track the coverage of the test file. In that case one command can cover multiple edge cases. IMO, if we have db.test.insert({a: null, b: 1}), b is a distraction for reviewer to understand the test case, let alone if we have a collection of real world data.
2. If one command covers multiple test cases, we can have buried behavior difference.
   For example:

> db.test.find()
{ "_id" : ObjectId("690eace782576799b86f4622"), "a" : 1 }
{ "_id" : ObjectId("690eacf382576799b86f4623"), "a" : "1" }
{ "_id" : ObjectId("690ead21fad8dbab68c7896c"), "a" : [ ] }
> db.test.find({}, {res: {$add: [1, "$a"]}})
Error: error: {
	"ok" : 0,
	"errmsg" : "Executor error during find command: test.test :: caused by :: $add only supports numeric or date types, not string",
	"code" : 14,
	"codeName" : "TypeMismatch"
}

We don't know the behavior of "a" being [].

The idea for using a 'collection' fixture was just to remove the boiler-plate logic of having to create a collection with a unique name (for safe parallelism) and make sure to drop it afterwards. We might end up having hundreds or thousands of test cases, so we ought to try to streamline as much as possible.

I agree that loading pre-defined collections might make it more difficult to see what the test is doing, but I'd prefer something like this to explicit drop/insert/drop statements

@pytest.mark.collection
def test_something(collection):
    collection.insert({...})
    assert(whatever())

Alternatively, something like this

@pytest.mark.documents([
     { "_id" : ObjectId("690eace782576799b86f4622"), "a" : 1 }
     { "_id" : ObjectId("690eacf382576799b86f4623"), "a" : "1" }
])
def my_test():
	//

Maybe even with some basic generator functions.

@pytest.mark.documents([
     repeat( 10, { "a" : randomString() } )
])

Yeah, I agree, the fixture will make it easier.

Yes using pytest fixtures make sense. We can also use the documents decorator for declarative test data for a test case that needs it.

I assume python is the primary lanuage for most test coverage. One exception is Python can not imitate "undefined" in JS which can be used in mongo shell, do we want to include that test coverage at initial stage? It requires JS executor. Or we should consider it language driver specific test case.

For which test do we need this?

Do we rely on contributor and reviewer to verify if test for a feature is comprehensive? Since the framework is also used to evaluate compatibility, I think we should have some system to validate the testing space for any feature supported in MongoDB API. Or better, some degree of automation that systematically generate test categories or even test cases based on predefined or contributor added rules. e.g. test category can be BASIC BSON TYPE TEST CASES, for any new feature that takes a input, we automatically generate test case for each supported bson type.
I can think of several levels of approach from full manual to fully automation (might be over-engineer):

1. A well structured README list all the aspect or categories each new feature should consider to add test.
2. A CI/CD test that use maintained rules to verify if a feature has all possible test categories. Like a operator should have a test for each data type input.
3. A maintained set rules that generate templates for each new feature, and contributor can add individual test cases in them. Contributor should first add rules for the new feature. Think of adding a Horizontal feature, assume we just added the support of Decimal128, almost all features should have a test for it. If we have this level of automation, contributor just need to add new data type to the list of supported bson types, then they know the minmum test cases to add. The template also ensures we maintain a good folder structure.
4. One step more, generate the test cases automatically. Contributor can still add more edge cases for the feature that special enough to not add as a new rule, but the automation covers the minimum.
   I sugget we have 3.

Test Runner also mentinoed it handles parallelism. It's not clear to me the differentiation of these two layers.

Disscussed this,

1. the parallelism means run multiple tests in parallel, not parallel command running needed by each test.
2. the runner only plan the execution, executor do the rest.

Can you show what log we generate and pass to Analyzer?

Discussed on this, we can use shared assertion functions to separate the raw log and assertion result type. Analyzer will only parse the formated assertion results. Raw logs are used for debugging.

I don't think real application data adds value to functional tests, like each price is double

Doesn't necessarily need to be real application data. We should use whatever data makes sense to appropriately test the desired behavior.

How can a test run have assertions fails on DocumentDB and another test fails on MongoDB, the expected outputs of a test should from either DocumentDB or MongoDB.

I think your question is how can the test fail on both MongoDB and DocumentDB when in reality we would have taken the test assertion from at least one of them. This can happen when there is a regression on the engine where the test was originally passing.

Agreed - additionally what are the licensing requirements for doing this?

Are there any existing frameworks and runners we can leverage to do this? Like pytest or some base framework that does the heavy lifting for us here? and we only need to provide libraries to extend the db specific stuff?

Yes I was considering pytest and it can actually handle the vast majority of this framework's desired functionality. I think we can go with that. We can keep the test result analyzer component which will do the post processing after we obtain the test results.

What does comparison mean here? Would appreciate more detail on this one - i presume a test generates a pass or a fail explicitly.

Also why is this comparison between DocumentDB and MongoDB? Isn't the test written to define a contract? the underlying presumption is that if the test passes, we meet the "spec" or the contract here. We separately could run this against Other database implementations such as MongoDB to ascertain the relative merits of the tests and the contract, but why is that coupled here?

I think the original wording in the RFC is confusing. What I meant was that the test cases would be written with specific asserts and that test could be run against any engine that supports the Mongo protocol. The comparison here just means comparing the result (pass or fail) of the test cases across different engines. The comparison doesn't really mean comparing the actual result of each query against the various engines under consideration and determining if they are the same or different.

So essentially the goal is to do exactly what you mentioned visridha

In short, we would do specification-based testing, not result comparison testing

Are we planning to have only 1 test for something like find? is that desirable? How would you split up stuff like aggregation pipelines?

And wouldn't this cause test files to be several thousands of lines? Is there a flow that allows for splitting these up more?

The directory structure shown was just a high-level example. We'd definitely need more granular organization. For large features like find, we'd split by sub-functionality: basic queries, query operators, logical operators, projections, sorting, cursors, etc.

nit: style consistency - are we doing methods as snake_cased or camelCased?

Good catch! We should follow Python conventions and use snake_case for all method names. I'll update the examples to be consistent throughout the RFC.

This should be a separate entity - also "Both" seems disingenuous - the test should describe a spec and should be targeted to an implementation. I would think this shouldn't be comparing against native Mongo as part of this spec. Imo We should set up a separate flow that compares across database implementations that may include MongoDB, or any other implementation as needed.

I think this question is similar to the previous one. I wasn't clear in the RFC that we will do specification based testing. To run the test against multiple engines we will simply provide an option when running tests (e.g., --engines=documentdb,mongodb,other-engine).