Add more *Async APIs for Collection and Cursor methods #386

menewman · 2022-04-08T21:51:48Z

Background

As Node moves away from supporting Fibers, inevitably Meteor is going to end up introduce asynchronous versions of Mongo collection and cursor methods that rely on async/await instead of on "synchronous" Fibers.

The Meteor team has an open PR that proposes to add async versions of each Mongo interface method:
meteor/meteor#11605

For example, that PR will introduce methods like updateAsync, findOneAsync, and fetchAsync.

Conveniently, meteor-protomongo already has some async methods following this format, which we've been using in the past to begin the migration away from Fibers. While we wait on the Meteor core changes to stabilize, we can go ahead and flesh out the versions here so we can start the migration faster.

Why not wait until the *Async methods ship in meteor/mongo?

That PR was being driven by Filipe, who is no longer with Meteor and (I assume) might not finalize the PR in the near future. It will likely be picked up again but I'm not sure about the timeline to finish the proposal and ship it.

In the meantime, since we're already monkey-patching, might as well monkey-patch more completely. The more we use *Async methods now, the easier it will be to migrate away from the synchronous versions and fibers in the future.

What happens after the core *Async methods do ship in meteor/mongo?

We should make sure to remove our monkey-patches in this lib so that we aren't overwriting the "core" versions.

Changes

For collections, I added findOneAsync. We already have async versions of the other core methods except for createIndex, which I don't think is quite as worrisome.

For cursors, I added fetchAsync, countAsync, forEachAsync, and mapAsync.

Finally, I split the extend interface into extendCollection and extendCursor -- I am intentionally breaking the API because I want to make it clear that the consumption needs to be updated for consumers. This is a 🔥 BREAKING 🔥 change and should be released as a major version bump. (We can handle the version/release bump process separately from this PR.)

Testing

I tested this by running npm pack and then installing the packed .tgz file into a local Meteor app, like meteor npm install packed-tar-file.tgz.

Within that app, I ran code that verified these methods, similar to this:

Meteor.users.find({
}).countAsync().then(x => x)

Albinutte

The changes make sense to me! I think it would be great to migrate this package to TypeScript (there's a chance that our TypeScript migration will be finished before Filipe's PR sees the light of day), but that's definitely out of the scope of this PR.

Approving with the caveat that I did not do any functional testing.

Should we introduce an ESLint rule in our apps that use this package to discourage the use of fiber-based cursor methods once we install this update?

Albinutte · 2022-04-11T06:20:31Z

src/mongo.js

 }

-module.exports = { extend };
+// $FlowFixMe: meteor/mongo doesn't have nice Flow types


We should migrate to TypeScript at some point 😏

You're right -- Flow seems especially useless for this package since we aren't even explicitly typing the inputs/outputs of these extended functions, and we are just wildly suppressing everything.

Buuuuuut I didn't want to bring that into scope here, so I left it alone and put more band-aids for now 😅

mpowaga

I did functional testing and it works great 👍 Requesting only one small change that should improve the installation experience.

mpowaga · 2022-04-11T15:27:38Z

src/mongo.js

+    throw new Error('Meteor object must exist');
+  }
+
+  const cursorPrototype = Object.getPrototypeOf(Meteor.users.find());


Since we now depend on accounts package it would be nice to check if Meteor.users exists and if it doesn't then throw a helpful message. Right now if you install ProtoMongo on a fresh Meteor project you will get an error like this:

W20220411-16:12:15.419(1)? (STDERR) /Users/michalpowaga/.meteor/packages/promise/.0.12.0.11feoh4.tacw++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/meteor-promise/promise_server.js:218 W20220411-16:12:15.432(1)? (STDERR) throw error; W20220411-16:12:15.432(1)? (STDERR) ^ W20220411-16:12:15.433(1)? (STDERR) W20220411-16:12:15.433(1)? (STDERR) TypeError: Cannot read property 'find' of undefined W20220411-16:12:15.433(1)? (STDERR) at Object.extendCursor (/Users/michalpowaga/Desktop/dev/protomongo-test/node_modules/@codesignal/meteor-protomongo/lib/mongo.js:13:112) W20220411-16:12:15.433(1)? (STDERR) at module (server/main.js:7:12) W20220411-16:12:15.433(1)? (STDERR) at fileEvaluate (packages/modules-runtime.js:336:7) W20220411-16:12:15.433(1)? (STDERR) at Module.require (packages/modules-runtime.js:238:14) W20220411-16:12:15.433(1)? (STDERR) at require (packages/modules-runtime.js:258:21) W20220411-16:12:15.433(1)? (STDERR) at /Users/michalpowaga/Desktop/dev/protomongo-test/.meteor/local/build/programs/server/app/app.js:70:15 W20220411-16:12:15.434(1)? (STDERR) at /Users/michalpowaga/Desktop/dev/protomongo-test/.meteor/local/build/programs/server/boot.js:401:38 W20220411-16:12:15.434(1)? (STDERR) at Array.forEach (<anonymous>) W20220411-16:12:15.434(1)? (STDERR) at /Users/michalpowaga/Desktop/dev/protomongo-test/.meteor/local/build/programs/server/boot.js:226:21 W20220411-16:12:15.434(1)? (STDERR) at /Users/michalpowaga/Desktop/dev/protomongo-test/.meteor/local/build/programs/server/boot.js:464:7 W20220411-16:12:15.434(1)? (STDERR) at Function.run (/Users/michalpowaga/Desktop/dev/protomongo-test/.meteor/local/build/programs/server/profile.js:280:14) W20220411-16:12:15.434(1)? (STDERR) at /Users/michalpowaga/Desktop/dev/protomongo-test/.meteor/local/build/programs/server/boot.js:463:13

I think that at minimum accounts-base package is required.

mpowaga · 2022-04-11T15:40:45Z

README.md

 npm install @codesignal/meteor-protomongo
 ```

 After the package is installed, add the following few lines in a file that's going to be loaded on startup:


[minor] Can you add a line saying that accounts-base package is required?

Probably most meteor projects already have accounts installed and/or this requirement can be communicated dynamically with an error message but it would be great to know of this prerequisite in the readme (and the reason behind it since it might be a little bit strange to require a package that seems to be unrelated if someone doesn't know the details 😄 ).

Do you think it would make sense to just ask the consumer to directly pass in the result of a find() instead, so that we can inject & remove that dependency?

The downside is that it exposes this "hack" to the consumer a little bit more. The upside is that it removes this dependency on accounts-base, which you're right is a little weird (I was trying to pick a collection I expect to exist in every Meteor instance, but I guess you're right that it's still a dep)

Update: Okay, I went ahead and made this change to ask the consumer to pass in a cursor, rather than assuming that Meteor.users is a valid collection. Thanks for pointing out the implicit dependency!

menewman · 2022-04-11T19:48:41Z

Should we introduce an ESLint rule in our apps that use this package to discourage the use of fiber-based cursor methods once we install this update?

I think that would be a great idea, yes!

…rsor. Fix cursor prototype override on the server.

menewman requested review from Albinutte and mpowaga April 8, 2022 21:52

menewman assigned Albinutte and mpowaga Apr 8, 2022

Albinutte approved these changes Apr 11, 2022

View reviewed changes

mpowaga suggested changes Apr 11, 2022

View reviewed changes

menewman force-pushed the async-apis branch from 9e80eb4 to 4336651 Compare April 11, 2022 19:46

Base automatically changed from housekeeping to master April 11, 2022 22:04

menewman added 3 commits April 11, 2022 15:04

Add interim *Async methods for Collection, Cursor

bc3b775

[Breaking] Separate the extend method into extendCollection, extendCu…

c84d898

…rsor. Fix cursor prototype override on the server.

Inject the cursor rather than depending on accounts-base

982a86c

menewman force-pushed the async-apis branch from 4336651 to 982a86c Compare April 11, 2022 22:05

menewman requested a review from mpowaga April 11, 2022 22:07

mpowaga approved these changes Apr 12, 2022

View reviewed changes

menewman merged commit 96f809a into master Apr 12, 2022

menewman deleted the async-apis branch April 12, 2022 14:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add more *Async APIs for Collection and Cursor methods #386

Add more *Async APIs for Collection and Cursor methods #386

Uh oh!

menewman commented Apr 8, 2022

Uh oh!

Albinutte left a comment

Uh oh!

Albinutte Apr 11, 2022

Uh oh!

menewman Apr 11, 2022

Uh oh!

mpowaga left a comment

Uh oh!

mpowaga Apr 11, 2022

Uh oh!

mpowaga Apr 11, 2022 •

edited

Loading

Uh oh!

menewman Apr 11, 2022

Uh oh!

menewman Apr 11, 2022

Uh oh!

menewman commented Apr 11, 2022

Uh oh!

Uh oh!

Add more *Async APIs for Collection and Cursor methods #386

Add more *Async APIs for Collection and Cursor methods #386

Uh oh!

Conversation

menewman commented Apr 8, 2022

Background

Why not wait until the *Async methods ship in meteor/mongo?

What happens after the core *Async methods do ship in meteor/mongo?

Changes

Testing

Uh oh!

Albinutte left a comment

Choose a reason for hiding this comment

Uh oh!

Albinutte Apr 11, 2022

Choose a reason for hiding this comment

Uh oh!

menewman Apr 11, 2022

Choose a reason for hiding this comment

Uh oh!

mpowaga left a comment

Choose a reason for hiding this comment

Uh oh!

mpowaga Apr 11, 2022

Choose a reason for hiding this comment

Uh oh!

mpowaga Apr 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

menewman Apr 11, 2022

Choose a reason for hiding this comment

Uh oh!

menewman Apr 11, 2022

Choose a reason for hiding this comment

Uh oh!

menewman commented Apr 11, 2022

Uh oh!

Uh oh!

mpowaga Apr 11, 2022 •

edited

Loading