Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Support upload of large databases (> 2GB) #1383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
aibaars opened this issue Nov 21, 2022 · 3 comments · Fixed by #1465
Closed

Support upload of large databases (> 2GB) #1383

aibaars opened this issue Nov 21, 2022 · 3 comments · Fixed by #1465

Comments

@aibaars
Copy link
Collaborator

aibaars commented Nov 21, 2022 •
edited
Loading

Currently the codeql-action crashes when trying to upload a CodeQL database bundle that is larger than 2GB. This is because the codeql-action tries to load the entire zip file into a Buffer which is limited to 2GB in Javascript.

We should probably rewrite:

codeql-action/src/database-upload.ts

Lines 43 to 45 in 4fddc51

const payload = fs.readFileSync(
await bundleDb(config, language, codeql, language)
);

and supply the payload to the POST request as a streaming input.

Creating zip file at C:\a\_temp\codeql_databases\csharp.zip.
Error: File size (8048141762) is greater than 2 GB
RangeError [ERR_FS_FILE_TOO_LARGE]: File size (8048141762) is greater than 2 GB
    at new NodeError (node:internal/errors:371:5)
    at tryCreateBuffer (node:fs:419:13)
    at Object.readFileSync (node:fs:464:14)
--->at uploadDatabases (C:\a\_actions\github\codeql-action\v2\lib\database-upload.js:51:28)<---
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async run (C:\a\_actions\github\codeql-action\v2\lib\analyze-action.js:184:9)
    at async runWrapper (C:\a\_actions\github\codeql-action\v2\lib\analyze-action.js:235:9) {
  code: 'ERR_FS_FILE_TOO_LARGE'
@aeisenberg
Copy link
Contributor

What is the max db size that turboscan can handle?

@rneatherway
Copy link
Contributor

We believe that the limit is 5GB due to the need to use multipart uploads above that size. However, we haven't tested around those sizes yet.

@robertbrignull robertbrignull removed their assignment Jan 4, 2023
@robertbrignull robertbrignull mentioned this issue Jan 6, 2023
Merged
3 tasks
@robertbrignull
Copy link
Contributor

This particular issue should be fixed now. Though note that uploads of multi-gigabyte databases is still not fully supported, so it will currently fail for other reasons. We'll be working on improving this over time so larger databases can be uploaded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use a stream when uploading database contents github/codeql-action
4 participants
@rneatherway @aeisenberg @robertbrignull @aibaars