Seeding DB with DREAM Challenges #82

rrchai · 2021-06-24T12:53:17Z

@tschaffter
I pushed the codes and updated json files for seeding challenges. The followings properties still need to be tested if they work fine with current codes and latest schema.

start/endDate: they have been changed to optional in the latest schema, since some challenge has no start/end dates
organizerIds, dataProviderIds, grantIds: check if empty array, [] is accepted in the latest services, since they are required

tschaffter · 2021-06-24T21:48:05Z

@rrchai Thanks! I fixed the service issue that you reported about empty array (see Sage-Bionetworks/rocc-service#113). I'll now try to find an elegant way to fix the seeding using RxJS.

…et data to json

tschaffter · 2021-06-26T00:17:55Z

@rrchai Getting this error after your last commit:

POSThttp://localhost:4200/api/organizations?organizationId=applied-proteogenomics-organizational-learning-and-outcomes-network

EDIT: It's because organization ID can only be 60 char max (see schema)

tschaffter · 2021-06-26T00:23:12Z

Temporarily renaming this organization ID;

applied-proteogenomics-organizational-learning-and-outcomes-network => org-a
eunice-kennedy-shriver-national-institute-of-child-health-and-human-development => org-b

EDIT: No longer an issues after reverting @rrchai commit (see below).

tschaffter · 2021-06-26T00:28:31Z

@rrchai You replaced Challenge.organizationIds by Challenge.organizations, which is incorrect. i can't fix this because the value is now set to the name of the organization while we want the organization id.

EDIT: @rrchai I undid your last commit because it's introducing several issues. Let's discuss these issues next week so you can resubmit your changes without the issues.

tschaffter · 2021-06-27T02:18:55Z

Removed the following organizations from challenge.json because their ID is too long (> 60 char).

eunice-kennedy-shriver-national-institute-of-child-health-and-human-development
applied-proteogenomics-organizational-learning-and-outcomes-network

tschaffter · 2021-06-29T23:29:27Z

@rrchai Also remove the Person objects from the Challenge object, include the fake ids instead. Please use the default parameter names like organizerIds or grantIds instead of what I did previously for the grants (tmpGrantId). It will take care of managing the fake and real ids in the seeding script.

tschaffter · 2021-06-30T12:45:09Z

@rrchai You can generate "fake" MongoDB ObjectId with the instruction bson.objectid.ObjectId() from the Python Package PyMongo.

rrchai · 2021-06-30T15:34:59Z

@tschaffter I have updated files:

add organizerId to persons-fakeIds.json
add grantId to grants-fakeIds.json
update organizerIds and grantIds in challenge-fakeIds.json. Fake Ids in challenge objects match with the ids in person/grants objects.

tschaffter · 2021-06-30T16:53:59Z

@rrchai Can you remove the following files?

challenges-fakeIds.json
grants-fakeIds.json
organizations-fix.json
persons-fakeIds.json

And update the remaining files so they meet the required defined in my previous comment. Ultimately we should only have the following files whose content match the ROCC schemas either based on the schemas {schema} (e.g. Tag, Organization) or {schema}CreateRequest.

challenges.json
grants.json
organizations.json
persons.json
tags.json

tschaffter · 2021-07-01T00:59:18Z

@rrchai @vpchung FYI Here is a nice illustration on how Lodash can be used to manipulate objects. In this example, I use Lodash omit method to remove the property id from object in order to obtain {schema}CreateRequest objects that are ready to be posted. :)

tschaffter · 2021-07-01T01:03:07Z

@rrchai @vpchung This PR is ready for your review. The remaining issue "Error: bundle initial-es5 exceeded maximum budget." should be fixed soon.

@rrchai I can't assign you as a reviewer because you create this PR. Do you mind giving a 👍 to this comment if you approve this PR?

vpchung

Looks good! Just one design question about the future usage of Lodash.

vpchung · 2021-07-01T02:52:09Z

src/app/components/database-seed/database-seed.component.ts

-import { mergeMap, tap } from 'rxjs/operators';
+import { forkJoin, Observable, of } from 'rxjs';
+import { map, mapTo, mergeMap, switchMap, tap } from 'rxjs/operators';
+import { merge as _merge, omit as _omit } from 'lodash';


Do we plan on updating Lodash as newer versions come out? According to the creator, omit will be deprecated in v5 due to poor performance. We may need to rewrite the code if so.

We will update libraries every month to please the Dependabot God, unless the update of a given library is not trivial. I learned about the existence of @babel/plugin-proposal-object-rest-spread by following the link you shared. I don't find this approach suitable here as it depends on the order of the object properties. I decided to go with _.pick and specify the properties to keep (array). I wasn't sure about how _.pick handled an array that includes property names that are not in a given object, for example Challenge.endDate is optional. _.pick behaves nicely and acts has a filter and it will not complain if a given object does not have the property. I'm still - and always - open to alternatives that you would like to share.

I like the _.pick approach as well! That being said, it may become quite tedious if numerous properties are needed... do you think that could happen? If so, we could always write a util function that mimics _.omit? I came across this article here that showcases different alternatives and their performances.

Since we still are in the early stage of development, it's likely that object properties will be updated. It's also more likely that we will add or rename properties - i.e. which will need us to update the seeding code - than removing properties from {schema} object to get {schema}CreateResponse. Also one thing that I don't like about _.pick is because its behavior that I described earlier can lead to a point where the list of properties specified is out of sync with the properties of {schema}CreateResponse.

We can throw away the pickBy option as it it too slow and doesn’t provide any benefits.

Assuming that pick has similar performance, then let's use one of the alternative proposed in the article. I just updated the code to an an omit function based on ESRest + JavaScript delete. This implementation does not allow to remove flatten paths, which should be fine for now.

rrchai

Looks great and I tested it works! @tschaffter

rrchai added 3 commits June 24, 2021 08:39

codes for seeding challenges draft

710eddd

remove comma, change spaces

1a96653

comment out person response

ace4f20

tschaffter and others added 3 commits June 24, 2021 15:03

Rename organizationId to id in org JSON list

d400042

Trying to seed organizers (random error)

6ea929d

update challenges data based on latest code converting from spreadshe…

7e53c60

…et data to json

tschaffter mentioned this pull request Jun 26, 2021

Fix ECONNRESET errors that happens when seeding the DB #83

Closed

This was linked to issues Jun 26, 2021

Seed API service with example data #23

Closed

Add DREAM challenge data in JSON format #42

Closed

Fix ECONNRESET errors that happens when seeding the DB #83

Closed

tschaffter added 11 commits June 25, 2021 17:30

Temporarily fixing two org IDs that are too long

9d86ec2

Undo rong last commit

2e2ff09

Add forkJoinConcurrent.ts

0f500c1

Push organizations using forkJoinConcurrent

bd96106

Get id of created Challenge organizers

502f773

See challenge with organizers created dynamically

0db0a9c

Seed grants

ef28c1e

createGrants pipe now returns Grant[]

a2a260d

Tag and Organization pipes now return Tag[] and Organization[]

6935b2f

Seed challenges, working towards getting access to grants

0cd9ed1

Clean up before using dynamically created Grants

3b07da2

tschaffter added 4 commits June 26, 2021 19:59

Challenges are now created with correct Grant Ids

d4ac427

Cleanup

90bc5ce

Keep order of Challenge.grantIds

79c88d7

Fix lint checks

357d18f

tschaffter mentioned this pull request Jun 29, 2021

Fix Error: bundle initial-es5 exceeded maximum budget. #86

Closed

tschaffter changed the title ~~Seeding challenges data - tmp~~ Seeding DB with DREAM Challenges Jun 30, 2021

rrchai added 2 commits June 30, 2021 10:59

Add fakeIds to persons/grants/challenges-fakeIds.json

7b1a091

cupdatee date type; reformat empty array

51acca2

rrchai and others added 8 commits June 30, 2021 13:13

clean up old files and fix grant order

e8ccdfa

change grantId/organizerId to id

b8c5ff4

fix type of organizationIds in persons object & add 2012 persons

bd5283e

Seed persons

5629d2f

Seed challenges using new flat JSON files

c27f02a

Simplify request bodies with _omit

6cf31a2

Fix lint checks but max line length

f2d52c9

Fix max line length issues

e465206

tschaffter self-assigned this Jul 1, 2021

tschaffter marked this pull request as ready for review July 1, 2021 00:51

tschaffter linked an issue Jul 1, 2021 that may be closed by this pull request

Flatten DREAM data JSON files #84

Closed

tschaffter requested a review from vpchung July 1, 2021 00:59

vpchung approved these changes Jul 1, 2021

View reviewed changes

tschaffter added 3 commits June 30, 2021 20:40

Replace _omit by _pick

3c897c4

Specify additional object return type

fdd3b29

Add omit implementation based on ESRest + JavaScript delete

e92d6b0

rrchai commented Jul 1, 2021

View reviewed changes

Merge branch 'main' into seeding-challenge-data

69e17ab

tschaffter merged commit 0378823 into main Jul 1, 2021

tschaffter deleted the seeding-challenge-data branch July 1, 2021 20:25

Seeding DB with DREAM Challenges #82

Seeding DB with DREAM Challenges #82

Uh oh!

Conversation

rrchai commented Jun 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tschaffter commented Jun 24, 2021

Uh oh!

tschaffter commented Jun 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tschaffter commented Jun 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tschaffter commented Jun 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tschaffter commented Jun 27, 2021

Uh oh!

tschaffter commented Jun 29, 2021

Uh oh!

tschaffter commented Jun 30, 2021

Uh oh!

rrchai commented Jun 30, 2021

Uh oh!

tschaffter commented Jun 30, 2021

Uh oh!

tschaffter commented Jul 1, 2021

Uh oh!

tschaffter commented Jul 1, 2021

Uh oh!

vpchung left a comment

Choose a reason for hiding this comment

Uh oh!

vpchung Jul 1, 2021

Choose a reason for hiding this comment

Uh oh!

tschaffter Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vpchung Jul 1, 2021

Choose a reason for hiding this comment

Uh oh!

tschaffter Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rrchai left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rrchai commented Jun 24, 2021 •

edited

Loading

tschaffter commented Jun 26, 2021 •

edited

Loading

tschaffter commented Jun 26, 2021 •

edited

Loading

tschaffter commented Jun 26, 2021 •

edited

Loading

tschaffter Jul 1, 2021 •

edited

Loading

tschaffter Jul 1, 2021 •

edited

Loading