Could you provide one example of what you mean by context in the client?

Here are a couple examples of context on the client:

• Tracker.currentComputation, which is used by Tracker.Dependency and in a few other places to know what the currently running computation is
• Most places that use Meteor.EnvironmentVariable. One place it is used is to track the currently running method stub.

If this is necessary, but is it? I already have a branch running without this top level Fiber.

Being able to run async code in the top-level of a module is more convenient in some cases and gives more control over when code is run. It isn't needed for Meteor apps to work, but is something we would lose from removing Fibers unless we implement an alternative. The ECMAScript proposal for top-level await is stage 4, so it should be part of the next version of ECMAScript, but we will have to implement it in our module system since we don't use native ECMAScript modules.

context on the client

I don't see why the changes in the server would affect client side, besides the API to keep the code compatible with server and client. Could you explain why?

Even if the code has a few async functions we can always await these calls.

If your concern is the server, we will use AsyncLocalStorage to keep the context that today we store inside the Fiber.current.

The ECMAScript proposal for top-level await is stage 4, so it should be part of the next version of ECMAScript, but we will have to implement it in our module system since we don't use native ECMAScript modules.

Yes, I'm following this proposal as well.

It would be nice to support it as we could provide better APIs but I don't see it as required but definitely a nice to have.

Even if the code has a few async functions we can always await these calls.

I'm not sure this is an option. On the client we don't have something like AsyncLocalStorage, so the execution context is always global. This works well when the code is synchronous, but not when it is async. If the execution context is reset after the promise resolves, Meteor would have to be very careful about other code running at the same time. For example, while running an async method stub, if a tracker computation or click event listener called collection.insert, it would cause problems if Meteor thought that insert happened in the method stub.

There is one option I've seen to having execution context with async code - we can use zones.js or a similar technique to it where it wraps all native api's that can cause code to run later, though there are a few downsides to doing something like this.

There are places that use tracker and minimongo where it might not be feasible or desirable to make it async. For example, blaze helpers, or most of the existing tracker integrations with the different view layers. Also, making tracker async might have interesting effects on when Tracker computations are re-run that might need to be considered.

It would be nice to support it as we could provide better APIs but I don't see it as required but definitely a nice to have.

I agree.

@zodern I concur with your remarks on minimongo / tracker impacts. Can it not be a tought to only address Meteor calls and it stubs and leave alone other Mongo calls on client side. That would make refactoring effort at least an order of magnitude lighter.

As this is all a lot of work, I was wondering if before building these new packages, a modern "skeleton" for these "async" packages could be defined:

• Using a better name for the fiber-free future instead of -async
• Published as NPM module ( maybe something like @meteor-x/mongo )
• Published as Atmosphere Package too, like mongo-x

We are going to decide details together along the way but yes, this is the idea, to provide a new skeleton Fiber-free and also provide a way for people with existent projects to write new code that doesn't depend on Fibers, first at least in the API level. So no migration would be necessary at least for the newly added code.

I'm not sure if it would be a good idea to also create npm packages in the same process as it could add more challenges that are not really necessary at this point.

I'm not sure if you actually run into these problems at all in practice, new Mongo.Collection doesn't have to do anything asynchronously at all, so this will be perfectly possible I guess?

Actually Mongo.Collection constructor does have some async calls inside it as you can see here and indirectly here.

Hi @JoHuang I was checking this package and it could be a nice alternative to Fibers at least to avoid async layers but it seems (by looking to the issues) that this package also has issues with other Node.js features.

Are there any large projects like Meteor using this package?

Hi @perbergland I'm not sure if I understood this sentence correctly:

That could be done by using the Promise.await() method to have the sync shims just forward to the async methods.

Are you saying that we could avoid this work by using Promise.await()? Promise.await() is implemented using Fibers so it would keep Fibers in Meteor and this is what we want to change in this work.

About the changes to async calls in general, yes, that is probably going to happen in the packages listed by you. It's important to understand that this was the chosen direction by TC39 so we don't have other options if we want to follow ECMAScript standards as close as possible. This is the way Javascript projects should write code nowadays, with async layers.

About the packages, it would be nice to also keep a list of Community packages that are going to be affected by these changes.

Are you saying that we could avoid this work by using Promise.await()? Promise.await() is implemented using Fibers so it would keep Fibers in Meteor and this is what we want to change in this work.

No, I was just trying to think of the progression of work here.

Let’s take Collection.findOne as an example. Its signature is

findOne(selector?: Selector<T> | ObjectID | string): U | undefined;

Phase one is to introduce its sibling in Mongo-async with an async signature:

findOne(selector?: Selector<T> | ObjectID | string): Promise<U | undefined>;

The implementation in phase 1 could be as easy as just invoking the sync method in the -async package.

The way I understand it, phase 2 would be to move the real implementation chunk by chunk into the -async package and then the original package implementation would become a shim on top of the async package where the sync method is just invoking

findOne(...args) {  
  return findOneAsync(...args).await();
}

I think we’re going to need to come up with good patterns here that all packages can use (and probably helper methods within Meteor to make it possible to standardize how deprecation warnings are emitted also from packages outside Meteor core).

Note:

The code in meteor/mongo is pretty complicated with write fences and what not, and may not lend itself to being converted in this way. It was more for getting the discussion to a level where we start talking in code examples. Using Typescript as the language for this is a natural choice, I think.

Yes, the general idea is exactly what you said, specially for the first step but there are some quirks as with Fibers we can pause the execution even inside constructors. This is hard with async functions as a constructor is always going to return a new instance of the desired class and not a Promise as async functions would.

Talking about mongo package specifically, our goal in the first step is to provide an alternative API where Fiber is not exposed at all, it means that where you will need to use await in the future you need to use await already in this new API so your new code will be ready for when Fiber is removed from internals and you don't need to worry about this code later.

This is important so people can continue to develop new code and this new code is not going to suffer rewrites in the future.

A few examples that I'm using while I'm making these changes to mongo package:

// api/LinksCollectionAsync.js
import { createAsyncCollection } from "meteor/mongo";

export const LinksCollectionAsync = createAsyncCollection("links");

// api/linksInsert.js
import { Meteor } from "meteor/meteor";
import { LinksCollectionAsync } from "./LinksCollectionAsync";

export const insertLink = async ({ title, url }) =>
  LinksCollectionAsync.insertAsync({
    title,
    url,
    createdAt: new Date()
  });

Meteor.methods({
  async insertAsync(arg) {
    await insertLink(arg);
  }
});

// server/main.js
import { Meteor } from "meteor/meteor";
import { LinksCollectionAsync } from "../api/LinksCollectionAsync";
import { insertLink } from "../api/linksInsert";

Meteor.publish("linksAsync", async () => LinksCollectionAsync.findAsync());

Meteor.startup(async () => {
  const linksCursor = await LinksCollectionAsync.findAsync();
  if (linksCursor.count()) {
    console.log(`No need to insert the first link`);
    console.log(`Links: `, linksCursor.fetch());
    return;
  }
  await insertLink({
    title: "Do the Tutorial",
    url: "https://www.meteor.com/tutorials/react/creating-an-app"
  });
  console.log(`Links: `, linksCursor.fetch());
});

The goal is also to be able to have the same collection with two instances, one that is async and the other instance that is sync so you don't need to migrate all the code from the same collection at the same time. You could migrate collection per collection and even partial usages of a specific collection.

I'll keep updating this discussion as I go further and before making final decisions I'm going to open a new discussion specially for this first step on mongo package. But I want to have working examples first so the discussion can be more rich and specific.

Everything that is being said in this discussion is being considered and will be considered in the future as well.

This is hard with async functions as a constructor is always going to return a new instance of the desired class and not a Promise as async functions would.

Is there code in the mongo package where this is actual a problem? Wouldn't this be simply wrapped in a factory function, like async createCursor = (...args) => new Cursor(...args) ?

Hi @jankapunkt, this is basically what I'm doing, of course, it is not that simple but the idea is the same.

Hi @red-meadow I understand your concern but what we are realizing is that we need to be as much aligned as possible with the Node.js ecosystem as a whole.

This is important to keep Meteor familiar for Node.js developers and vice-versa.

Supporting standard async and await is important in this direction, especially as Fibers (coroutines) is not the way Node.js decided to move forward with async execution.

Meteor is not relying a lot on Fibers at the API level so we believe the changes are not going to affect users a lot but in the end, it's going to be much easier to integrate Meteor code with Node.js code in general.

@filipenevola is there a plan to support the old fiber sync style API for a certain number of minor versions or will it be breaking, once the transition is complete?

The idea is in Phase 2 to have it as an option:

Phase 2 - Make Fibers usage in Meteor server application runtime optional

And only in Phase 4 it would be required not to use Fibers:

Phase 4 - Declare Meteor packages using Fibers in maintenance mode

What is your concern specifically?

The time between people finishing the code migration and Meteor releasing new versions?

We will still support the Fibers versions for essential fixes and improvements for a reasonable amount of time, except when these are very hard or impossible to do in the Fibers version.

@filipenevola it feels like there should be a Phase 5 in the list, which would be 'Dropping support for fibers'. Presumably to be done when meteor wants to update to node 16, assuming Phase 1.1 isnt completed.
From the current list it sounds like there are no plans to remove fibers from future versions, just to disable it on a per project basis

Hi @Julusian when Phase 3 is completed we will be able to run Meteor with Node.js 16.

Expected Result

Meteor will be compatible with async_hooks and WebAssembly and any other Node.js modules not only in application code but also in Meteor tools.

I'm not sure if we need an explicit Phase 5 as we state in Phase 4 that everything that uses Fibers is going to be in maintenance mode.

So in newer versions of Meteor supporting Node.js 16, for example, Fibers is not going to be there anymore.

In my experience 'maintenance mode' is generally when it will still be supported, with the only code changes being bug fixes. So it is unclear as to when fibers will stop working in an up to date version of meteor.

@filipenevola is there a plan to support the old fiber sync style API for a certain number of minor versions or will it be breaking, once the transition is complete?

Based on this comment, it sounds as though they also find it unclear in what version fibers will be not at all supported

I think that the approach that @filipenevola has chosen for MongoDB in #11605, i.e., creating *Async counterparts of the methods is the way to go. However, it's not making them as the defaults, but rather wrap the synchronous ones in Promises (1, 2). The actual transition can happen later and separately - API is the most important part.

Only the low-level API would be deprecated, so basically the hooks themselves. The main functionality of context tracking would still be exposed through AsyncLocalStorage and other similar APIs through the same internals. Deprecating the hooks will make it easier to change the internals later on, but the high-level functionality will still stay the same.

Generally speaking, a project like Meteor should only ever implement AsyncResource if needed and not hooks directly anyway. If it uses promises and doesn't do any internal queueing, it will probably not need to use anything at all.

who will lead the plan and bring another dev from Quave

For example, Meteor uses Profile which can work only with sync functions. For working with async, it should be adjusted.

You are right, we will probably have to re-implement Profile in a new way.

This work is going to happen inside this Phase:

Phase 3 - Make Fibers usage in Meteor tools optional

@matheusccastroo +1 for no.2

I currently am working on top level await. I've been focused on a different aspect: what limitations the legacy client add, and how to ensure a module's dependencies are loaded before the module is ran. With what I currently know, I think we can almost fully implement top level await in reify. I'm planning to start writing some code this week to check.

I'm putting you both in contact @zodern, so we can also join forces here! @matheusccastroo is from Quave and is helping us on the Fibers migration as well.

Hi, I'm missing the bi-weekly updates. Are they moved somewhere else? I was also looking at the overview doc and the Fibers Public Roadmap but they haven't updated for multiple weeks now.

To be more specific: We were planning to do the migration to 3.0 in Q1/2024. But since there is no beta yet we probably move it to Q2/2024. Do you have schedule for a beta release?

Here are some newer information: #12865

Great work, @denihs.

Is the API for callAsync/applyAsync considered stable now?

A blog post dedicated to this would be excellent. I could help with that, but we need the confirmation that it's the final API first.

Now, in beta zero, we can say we have a stable version. I think would be good to have this blog post about it

Are there any plans to continue to extend security support for Node 14 past this April? Will the eventual release date of Meteor 3 affect security support extensions?

Just went to give this a try. This is the first time I've tried updating to the new release.

meteor update --release 3.0-beta.4
MINISAT-out: Cannot enlarge memory arrays. Either (1) compile with -s TOTAL_MEMORY=X with X higher than the current value 67108864, (2) compile with ALLOW_MEMORY_GROWTH which adjusts the size at runtime but prevents some optimizations, or (3) set Module.TOTAL_MEMORY before the program runs.
MINISAT-err: Cannot enlarge memory arrays. Either (1) compile with -s TOTAL_MEMORY=X with X higher than the current value 67108864, (2) compile with ALLOW_MEMORY_GROWTH which adjusts the size at runtime but prevents some optimizations, or (3) set Module.TOTAL_MEMORY before the program runs.
abort() at Error                              
    at jsStackTrace (packages/logic-solver/minisat.js:3:18626)
    at stackTrace (packages/logic-solver/minisat.js:3:18809)
    at abort (packages/logic-solver/minisat.js:33:28956)
    at enlargeMemory (packages/logic-solver/minisat.js:3:19142)
    at Function.dynamicAlloc [as alloc] (packages/logic-solver/minisat.js:3:7927)
    at _sbrk (packages/logic-solver/minisat.js:3:58803)
    at Sd (packages/logic-solver/minisat.js:7:98389)
    at Ud (packages/logic-solver/minisat.js:7:109800)
    at gc (packages/logic-solver/minisat.js:7:41378)
    at pc (packages/logic-solver/minisat.js:7:46259)
    at _b (packages/logic-solver/minisat.js:7:31709)
    at $b (packages/logic-solver/minisat.js:7:35130)
    at Bc (packages/logic-solver/minisat.js:7:54870)
    at hd (packages/logic-solver/minisat.js:7:84479)
    at MiniSat.solveAssuming (packages/logic-solver/minisat_wrapper.js:83:18)
    at Logic.Solver.solve (packages/logic-solver/logic.js:1342:33)
    at Logic.Solver.solveAssuming (packages/logic-solver/logic.js:1375:21)
    at minMaxWS (packages/logic-solver/optimize.js:52:32)
    at packages/constraint-solver/solver.js:429:22
    at CS.Solver.minimize (packages/constraint-solver/solver.js:413:3)
    at CS.Solver.minimize (packages/constraint-solver/solver.js:407:5)
    at CS.Solver._getAnswer (packages/constraint-solver/solver.js:803:3)
    at Object.Logic.disablingAssertions (packages/logic-solver/logic.js:47:12)
    at Function.CS.PackagesResolver._resolveWithInput (packages/constraint-solver/constraint-solver.js:190:10)
    at CS.PackagesResolver.resolve (packages/constraint-solver/constraint-solver.js:151:14)
    at /tools/project-context.js:669:26
    at /tools/project-context.js:665:11
    at Object.enterJob (/tools/utils/buildmessage.js:387:12)
    at /tools/packaging/catalog/catalog.js:100:5
    at Object.capture (/tools/utils/buildmessage.js:282:5)
    at Object.catalog.runAndRetryWithRefreshIfHelpful (/tools/packaging/catalog/catalog.js:99:18)
    at ProjectContext._resolveConstraints (/tools/project-context.js:636:5)
    at /tools/project-context.js:402:9
    at Object.enterJob (/tools/utils/buildmessage.js:387:12)
    at /tools/cli/commands-packages.js:1739:5
    at Object.capture (/tools/utils/buildmessage.js:282:5)
    at Object.main.captureAndExit (/tools/cli/main.js:275:16)
    at maybeUpdateRelease (/tools/cli/commands-packages.js:1738:3)
    at Command.func (/tools/cli/commands-packages.js:1812:29)
    at /tools/cli/main.js:1527:15
If this abort() is unexpected, build with -s ASSERTIONS=1 which can give more information.

@wreiske There is a forum discussion with some suggestions for it: https://forums.meteor.com/t/trouble-updating-to-3-0-alpha-17-minisat/60728

Curious if you’re able to pinpoint a solution.

@wreiske @jamauro I think the root of the problem is a technical one in the dependancy resolution system; there is some sort of constraint satisfaction solver that tries to work iteratively and drowns itself with memory. That being said, the way you get around this is by

• delete the contents of your versions file
• comment out all of your dependencies in your packages file
• re-adding them all one by one
• (if needed) specify specific versions of packages to satisfy Meteor 3 constraints
• (if needed) clone broken packages onto your machine and repair manually

The process is a little tedious, but I only needed to do it once. It is also a great way of finding redundant/ outdated/ unused dependencies in your project.

Thank you! Looking forward to the eventual RC 🫡

@dallman2 the announcement is here

Is there a problem with the dependencies of mongo-decimal package. We need to change it to:

  api.use('[email protected]');
  api.use('[email protected]');

Thank you.

Same problem with less package:

  use: [
    "[email protected]",
    "[email protected]",
  ],

The strange thing is that if I copy the less package locally without any modifications, it works with the wrong dependencies.

It seems that all packages under packages/no-core are not handled during the distribution phase.

I've published a new [email protected] version I'll add a note to remember those packages in every release.

yep,

beta 7:

beta 6:

It is in the announcement for Beta 7

Ok, missed beta.7 - sorry for disturbance ;)

I've noticed there's also beta.8 on npm - should I test with it or stick with "official" beta.7?

Yeah, that was an issue we had. The npm version is off by one(npx meteor@beta / npx [email protected] will download the beta.7)
In the RCs, this will be solved. I'm sorry for the mistake and possible confusion; we are trying to improve our communication.

Ok, missed beta.7 - sorry for disturbance ;)

That was a valid question; Beta 7 was only officially launched yesterday.

✍️

I just realized the matching between official Meteor 3.x beta version and NPM CLI. I had to bump the version of CLI twice to fix a last issue with CLI and this got unsync. Good to know this for next versions planning.