coverage: 93.076%. remained the same
when pulling 936425b on myndzi:myndzi/better-sqlite3-support-bigint-mode
into c92bbb3 on knex:master.

@myndzi
Hi, I’ve taken a look at the work.
I had previously worked on #6320 as a PR for #6319, and it seems there may be some overlap in scope. I was wondering why a new PR was created that includes this work again.

@myndzi Hi, I’ve taken a look at the work. I had previously worked on #6320 as a PR for #6319, and it seems there may be some overlap in scope. I was wondering why a new PR was created that includes this work again.

Hi there. In brief, I was working on a larger set of changes that would address this need systematically (with respect to the types returned by queries), and add safety to better-sqlite3 specifically by rejecting queries that return numbers that are not safe (in other words: changing the default to always use safeIntegers, casting the values back to number, and throwing an error if it would be unsafe to do so).

I prepared individual pieces that could be reviewed and/or merged separately, but I was not sure whether the work in this PR would interact with the other PRs I was preparing, so I had it staged separately until everything was working together. (This PR is not ready for review, it was just holding one of those pieces)

I'm leaning towards coming back to that deeper work later and focusing on getting PRs merged so that I don't wind up generating a bunch of merge conflicts, which means that I can likely close this in favor of your PR (it wouldn't wind up changing to integrate with the safety change).

If you'd like, I'm happy to work with you to get your original PR merged instead of this one?

The main requests I'd have are to support per-query configuration of this option and to utilize the existing integration testing framework instead of making ad-hoc knex instances in the tests. You can see what I mean in the code here.

@myndzi
Thank you for the explanation. I’ve reviewed your comment and the PR code, and I’d be interested to hear your thoughts on the points and questions below.

1. Setting safeIntegers as the default and casting values back to number while intentionally blocking cases that would cause precision loss seems like a good approach. In this case, exposing an option to control whether bigint values are returned (e.g. options.safeIntegers: boolean) also sounds reasonable. Returning bigint when set to true, and returning number with precision checks when set to false, seems like it could address both cases where precision loss wasn’t previously an issue and cases where bigint support is needed.

2. Applying safeIntegers at the query level also makes sense. If point (1) is adopted, providing the bigint return option at the query options level would allow for more flexible usage.

3. Using the existing integration test framework, as shown in your PR, also seems beneficial. Centralized instance management and testing against the actual DB driver look like clear advantages. Since I have limited experience designing test code and would like to better understand Knex’s testing structure, I was wondering why integration tests were preferred over unit tests in this case.

If you’re able to help get this merged as PR #6032, I’d be happy to try incorporating the feedback you shared and iterating on the implementation.

@dong-jun-shin

Setting safeIntegers as the default and casting values back to number while intentionally blocking cases that would cause precision loss seems like a good approach. In this case, exposing an option to control whether bigint values are returned (e.g. options.safeIntegers: boolean) also sounds reasonable. Returning bigint when set to true, and returning number with precision checks when set to false, seems like it could address both cases where precision loss wasn’t previously an issue and cases where bigint support is needed.

For now, I'm shelving the full "casting behavior" thing. It's mergeable with a few more tests, but I'm not happy with the compromises I had to make; there is a clear point to refactor that would clean up those compromises, but it would touch more code than I want to change until we've knocked down the PR backlog.

I definitely agree with you that the extra safety is an improvement, but I want to be clear that it's not expected to be solved by this PR.

If you’re able to help get this merged as PR #6032, I’d be happy to try incorporating the feedback you shared and iterating on the implementation.

Let me come back to point 3 in a moment.

First, I should apologize for not having caught your PR in the first place. I'm still kinda in a mode where I expect all our PRs to be old and mostly abandoned. So I wound up creating a PR that does exactly what I would want from yours already 😅. This leaves me in an awkward position, because I'd rather merge yours and give you the contribution credit.

I'm very happy to spend some time helping you understand the decisions I made (which are by no means the final word or anything!) if you're interested. Especially if you're planning on contributing further (we're seeking all the help we can get!), there's a lot of value in getting folks up to speed on the codebase and how things work / what patterns are used and why.

With that said,

Using the existing integration test framework, as shown in your PR, also seems beneficial. Centralized instance management and testing against the actual DB driver look like clear advantages. Since I have limited experience designing test code and would like to better understand Knex’s testing structure, I was wondering why integration tests were preferred over unit tests in this case.

When I skimmed your PR, I didn't actually notice that the tests you added were in the unit directory. This is because what you've created are integration tests:

     knexInstance = knex({
        client: 'better-sqlite3',
        useNullAsDefault: true,
        connection: {
          filename: ':memory:',
          options: {
            defaultSafeIntegers: true,
          },
        },
      });

This is actively "connecting" to a real sqlite3 instance using the better-sqlite3 library and running queries through it.

Since you chose to do it this way, I believe you already understand why an integration test is correct here: the goal is to ensure that the code that was written actually has the desired effect, and the only way to do that is to actually exercise the interface between knex and the better-sqlite3 library.

While the way that integration tests are run (and databases are selected) could use some improvement, it will be easiest to implement that improvement later, after the tests in all the pending PRs have been merged. Keeping them "working the same way as the current tests work" makes that job easier in the future.

Next steps

• Let's get a PR merged as a stepwise improvement, simply allowing users to interact with the safeIntegers setting in better-sqlite3. The "safe bigint casting" behavior I described is out of scope for this improvement.
• It should support both a factory-level option and a query-level option
• If the user does not specify either option, it should not call safeIntegers at all. That is, the default behavior should be "the same behavior as before the PR", not "default to safeIntegers = false" or "default to safeIntegers = true" (so no !!flag)
• The tests, which are integration tests, should be in the integration2 directory (by my best estimation, this is the most-current testing skeleton for integration tests)

If you update your PR to satisfy these things, it'll be good to go. Feel free to ask me questions here (or join the discord) if you need something, I'm happy to help get it there.

@myndzi
Thank you very much for the detailed explanation of the background and your perspective. 🙂
I’m also glad to hear that you prefer contributing to the repo through my PR.

For now, I'm shelving the full "casting behavior" thing. It's mergeable with a few more tests, but I'm not happy with the compromises I had to make; there is a clear point to refactor that would clean up those compromises, but it would touch more code than I want to change until we've knocked down the PR backlog.

I frequently use Knex in the Node.js ecosystem, and if there are areas that need improvement, I’m more than willing to contribute as long as time allows.

When I skimmed your PR, I didn't actually notice that the tests you added were in the unit directory. This is because what you've created are integration tests
This is actively "connecting" to a real sqlite3 instance using the better-sqlite3 library and running queries through it.
Since you chose to do it this way, I believe you already understand why an integration test is correct here: the goal is to ensure that the code that was written actually has the desired effect, and the only way to do that is to actually exercise the interface between knex and the better-sqlite3 library.
While the way that integration tests are run (and databases are selected) could use some improvement, it will be easiest to implement that improvement later, after the tests in all the pending PRs have been merged. Keeping them "working the same way as the current tests work" makes that job easier in the future.

As you mentioned, the tests in my PR needed to verify interactions with the actual driver, so they should have been written as integration tests, and writing them as unit tests was not appropriate. Although it’s a bit embarrassing, I realize that my understanding of the tests I wrote was insufficient, and thanks to your detailed explanation, I now understand what I missed. Thank you. 🙏

Next steps

• Let's get a PR merged as a stepwise improvement, simply allowing users to interact with the safeIntegers setting in better-sqlite3. The "safe bigint casting" behavior I described is out of scope for this improvement.
• It should support both a factory-level option and a query-level option
• If the user does not specify either option, it should not call safeIntegers at all. That is, the default behavior should be "the same behavior as before the PR", not "default to safeIntegers = false" or "default to safeIntegers = true" (so no !!flag)
• The tests, which are integration tests, should be in the integration2 directory (by my best estimation, this is the most-current testing skeleton for integration tests)

If you update your PR to satisfy these things, it'll be good to go. Feel free to ask me questions here (or join the discord) if you need something, I'm happy to help get it there.

I strongly agree with the gradual improvement approach you suggested. I was planning to work on my PR (#6320) by incorporating the work from the current PR (#6328) and reflecting the next steps feedback, but it looks like it has already been merged… 😅

Should I proceed by working on this as a separate PR?

@dong-jun-shin yup, iterating in a separate PR would be great!