This simple server provides a single endpoint to which you can upload a CSV file. In turn, the contents of that CSV file will be added to a collection in a mongo database. This particular implementation was made for a specific use, but with minor modifications, it can work for almost any type of CSV and any type of document shape. It can quickly burn through hundreds of thousands of CSV records in seconds.
- Upload file
- Read stream by line-sized chunks.
- Parse the lines into objects.
- Convert the objects into client specified documents.
- Insert documents in batches of x documents (x is currently 500).
You will need the following environment variables:
MONGODB_URICOLLECTION_NAMEPROMISE_THROTTLEINSERT_BATCH_SIZE
Run npm install to load all the dependencies before running npm start.
Run the server and try the following command from the command-line:
curl -F "file=@./sample.csv" localhost:10123/submit