Thanks to visit codestin.com
Credit goes to github.com

Skip to content

PXD-3012: Add missing index to make submission faster #280

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
fantix merged 2 commits into from
Apr 29, 2019
Merged

PXD-3012: Add missing index to make submission faster #280

fantix merged 2 commits into from
Apr 29, 2019

Conversation

@fantix
Copy link
Contributor

@fantix fantix commented Apr 29, 2019 •
edited
Loading

See also: #217, uc-cdis/datamodelutils#10

Improvements

  • Added missing index to make submission faster

Dependency updates

  • datamodelutils bumped to 0.4.5

@PlanXCyborg
Copy link

PlanXCyborg commented Apr 29, 2019

Jenkins Build 1 : time taken 31 min
Check the https://jenkins.planx-pla.net/job/perf-sheepdog/job/PR-280/1/display/redirect

Test results:
Passed: 109, Failed: 0, Skipped: 0

Test Time (PR) Time (master) Diff
@ExportPerformanceTests @performance: Export all nodes of type: aliquot 0.227 0.253 -0.03
@ExportPerformanceTests @performance: Export all nodes of type: case 0.185 0.226 -0.04
@ExportPerformanceTests @performance: Export all nodes of type: experiment 0.445 0.28 0.16
@ExportPerformanceTests @performance: Export all nodes of type: program 0.078 0.086 -0.01
@ExportPerformanceTests @performance: Export all nodes of type: project 0.059 0.051 0.01
@ExportPerformanceTests @performance: Export all nodes of type: read_group_qc 0.086 0.092 -0.01
@ExportPerformanceTests @performance: Export all nodes of type: sample 0.323 0.542 -0.22
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: aliquot 0.146 0.183 -0.04
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: case 0.199 0.274 -0.07
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: experiment 0.168 0.246 -0.08
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: read_group 0.06 0.069 -0.01
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: read_group_qc 0.049 0.074 -0.03
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: sample 0.083 0.11 -0.03
@QueryPerformanceTests @performance: Executing bottomUp query # 0 0.586 0.51 0.08
@QueryPerformanceTests @performance: Executing bottomUp query # 1 0.556 0.487 0.07
@QueryPerformanceTests @performance: Executing bottomUp query # 2 0.538 0.486 0.05
@QueryPerformanceTests @performance: Executing bottomUp query # 3 0.435 0.382 0.05
@QueryPerformanceTests @performance: Executing bottomUp query # 4 0.327 0.298 0.03
@QueryPerformanceTests @performance: Executing bottomUp query # 5 0.296 0.225 0.07
@QueryPerformanceTests @performance: Executing bottomUp query # 6 0.158 0.14 0.02
@QueryPerformanceTests @performance: Executing bottomUp query # 7 0.079 0.07 0.01
@QueryPerformanceTests @performance: Executing topDown query # 0 0.066 0.043 0.02
@QueryPerformanceTests @performance: Executing topDown query # 1 0.08 0.071 0.01
@QueryPerformanceTests @performance: Executing topDown query # 2 0.102 0.086 0.02
@QueryPerformanceTests @performance: Executing topDown query # 3 0.177 0.249 -0.07
@QueryPerformanceTests @performance: Executing topDown query # 4 0.283 0.23 0.05
@QueryPerformanceTests @performance: Executing topDown query # 5 0.346 0.309 0.04
@QueryPerformanceTests @performance: Executing topDown query # 6 0.467 0.419 0.05
@QueryPerformanceTests @performance: Executing topDown query # 7 0.559 0.488 0.07
@SubmissionPerformanceTests @performance: Submission acknowledgement 10 0.214 0.35 -0.14
@SubmissionPerformanceTests @performance: Submission acknowledgement 100 1.549 2.325 -0.78
@SubmissionPerformanceTests @performance: Submission acknowledgement 1000 14.831 22.503 -7.67
@SubmissionPerformanceTests @performance: Submission aligned_reads_index 10 1.273 1.706 -0.43
@SubmissionPerformanceTests @performance: Submission aligned_reads_index 100 12.005 16.534 -4.53
@SubmissionPerformanceTests @performance: Submission aliquot 10 0.236 0.366 -0.13
@SubmissionPerformanceTests @performance: Submission aliquot 100 1.473 2.281 -0.81
@SubmissionPerformanceTests @performance: Submission aliquot 1000 15.46 22.814 -7.35
@SubmissionPerformanceTests @performance: Submission case 10 0.231 0.365 -0.13
@SubmissionPerformanceTests @performance: Submission case 100 1.436 2.35 -0.91
@SubmissionPerformanceTests @performance: Submission case 1000 15.308 23.659 -8.35
@SubmissionPerformanceTests @performance: Submission clinical_test 10 0.239 0.424 -0.19
@SubmissionPerformanceTests @performance: Submission clinical_test 100 1.697 2.569 -0.87
@SubmissionPerformanceTests @performance: Submission clinical_test 1000 17.568 27.734 -10.17
@SubmissionPerformanceTests @performance: Submission core_metadata_collection 10 0.962 0.753 0.21
@SubmissionPerformanceTests @performance: Submission core_metadata_collection 100 1.575 2.486 -0.91
@SubmissionPerformanceTests @performance: Submission core_metadata_collection 1000 15.385 23.725 -8.34
@SubmissionPerformanceTests @performance: Submission demographic 10 0.241 0.383 -0.14
@SubmissionPerformanceTests @performance: Submission demographic 100 1.733 2.733 -1.00
@SubmissionPerformanceTests @performance: Submission demographic 1000 16.786 27.738 -10.95
@SubmissionPerformanceTests @performance: Submission diagnosis 10 0.259 0.392 -0.13
@SubmissionPerformanceTests @performance: Submission diagnosis 100 1.807 2.815 -1.01
@SubmissionPerformanceTests @performance: Submission diagnosis 1000 19.048 29.139 -10.09
@SubmissionPerformanceTests @performance: Submission experiment 10 1.842 0.399 1.44
@SubmissionPerformanceTests @performance: Submission experiment 100 1.452 2.382 -0.93
@SubmissionPerformanceTests @performance: Submission experiment 1000 14.497 22.409 -7.91
@SubmissionPerformanceTests @performance: Submission experimental_metadata 10 1.314 1.812 -0.50
@SubmissionPerformanceTests @performance: Submission experimental_metadata 100 12.443 16.13 -3.69
@SubmissionPerformanceTests @performance: Submission exposure 10 0.234 0.371 -0.14
@SubmissionPerformanceTests @performance: Submission exposure 100 1.575 2.368 -0.79
@SubmissionPerformanceTests @performance: Submission exposure 1000 16.993 23.676 -6.68
@SubmissionPerformanceTests @performance: Submission family_history 10 0.226 0.351 -0.13
@SubmissionPerformanceTests @performance: Submission family_history 100 1.554 2.476 -0.92
@SubmissionPerformanceTests @performance: Submission family_history 1000 15.539 24.246 -8.71
@SubmissionPerformanceTests @performance: Submission keyword 10 0.214 0.353 -0.14
@SubmissionPerformanceTests @performance: Submission keyword 100 1.421 2.384 -0.96
@SubmissionPerformanceTests @performance: Submission keyword 1000 14.645 24.347 -9.70
@SubmissionPerformanceTests @performance: Submission publication 10 0.228 0.349 -0.12
@SubmissionPerformanceTests @performance: Submission publication 100 1.39 2.421 -1.03
@SubmissionPerformanceTests @performance: Submission publication 1000 15.323 21.918 -6.59
@SubmissionPerformanceTests @performance: Submission read_group 10 0.24 0.388 -0.15
@SubmissionPerformanceTests @performance: Submission read_group 100 1.692 2.565 -0.87
@SubmissionPerformanceTests @performance: Submission read_group 1000 17.577 27.326 -9.75
@SubmissionPerformanceTests @performance: Submission read_group_qc 10 0.439 0.515 -0.08
@SubmissionPerformanceTests @performance: Submission read_group_qc 100 2.811 4.2 -1.39
@SubmissionPerformanceTests @performance: Submission read_group_qc 1000 18.573 29.453 -10.88
@SubmissionPerformanceTests @performance: Submission sample 10 0.531 0.404 0.13
@SubmissionPerformanceTests @performance: Submission sample 100 1.655 2.733 -1.08
@SubmissionPerformanceTests @performance: Submission sample 1000 17.052 25.899 -8.85
@SubmissionPerformanceTests @performance: Submission slide 10 0.242 0.466 -0.22
@SubmissionPerformanceTests @performance: Submission slide 100 1.707 2.373 -0.67
@SubmissionPerformanceTests @performance: Submission slide 1000 16.646 24.755 -8.11
@SubmissionPerformanceTests @performance: Submission slide_count 10 0.229 0.443 -0.21
@SubmissionPerformanceTests @performance: Submission slide_count 100 1.551 2.499 -0.95
@SubmissionPerformanceTests @performance: Submission slide_count 1000 16.306 23.128 -6.82
@SubmissionPerformanceTests @performance: Submission slide_image 10 1.386 1.703 -0.32
@SubmissionPerformanceTests @performance: Submission slide_image 100 12.362 16.049 -3.69
@SubmissionPerformanceTests @performance: Submission submitted_aligned_reads 10 1.499 1.806 -0.31
@SubmissionPerformanceTests @performance: Submission submitted_aligned_reads 100 12.08 16.999 -4.92
@SubmissionPerformanceTests @performance: Submission submitted_copy_number 10 1.306 1.719 -0.41
@SubmissionPerformanceTests @performance: Submission submitted_copy_number 100 12.118 15.821 -3.70
@SubmissionPerformanceTests @performance: Submission submitted_methylation 10 1.925 1.79 0.13
@SubmissionPerformanceTests @performance: Submission submitted_methylation 100 12.088 16.281 -4.19
@SubmissionPerformanceTests @performance: Submission submitted_somatic_mutation 10 1.317 1.693 -0.38
@SubmissionPerformanceTests @performance: Submission submitted_somatic_mutation 100 12.354 17.043 -4.69
@SubmissionPerformanceTests @performance: Submission submitted_unaligned_reads 10 1.277 1.772 -0.50
@SubmissionPerformanceTests @performance: Submission submitted_unaligned_reads 100 11.866 16.324 -4.46
@SubmissionPerformanceTests @performance: Submission treatment 10 0.23 0.37 -0.14
@SubmissionPerformanceTests @performance: Submission treatment 100 1.804 2.413 -0.61
@SubmissionPerformanceTests @performance: Submission treatment 1000 15.75 23.32 -7.57

@PlanXCyborg
Copy link

PlanXCyborg commented Apr 29, 2019

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

@PlanXCyborg
Copy link

PlanXCyborg commented Apr 29, 2019

Jenkins Build 3 : time taken 32 min
Check the https://jenkins.planx-pla.net/job/perf-sheepdog/job/PR-280/3/display/redirect

Test results:
Passed: 109, Failed: 0, Skipped: 0

Test Time (PR) Time (master) Diff
@ExportPerformanceTests @performance: Export all nodes of type: experiment 0.302 0.28 0.02
@ExportPerformanceTests @performance: Export all nodes of type: program 0.135 0.086 0.05
@ExportPerformanceTests @performance: Export all nodes of type: project 0.058 0.051 0.01
@ExportPerformanceTests @performance: Export all nodes of type: read_group 0.43 0.399 0.03
@ExportPerformanceTests @performance: Export all nodes of type: read_group_qc 0.109 0.092 0.02
@ExportPerformanceTests @performance: Export all nodes of type: sample 0.386 0.542 -0.16
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: aliquot 0.167 0.183 -0.02
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: experiment 0.361 0.246 0.11
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: read_group 0.074 0.069 0.01
@ExportPerformanceTests @performance: Exporting a record by ID on nodes of type: sample 0.12 0.11 0.01
@QueryPerformanceTests @performance: Executing bottomUp query # 0 0.579 0.51 0.07
@QueryPerformanceTests @performance: Executing bottomUp query # 7 0.065 0.07 -0.01
@QueryPerformanceTests @performance: Executing topDown query # 1 0.083 0.071 0.01
@QueryPerformanceTests @performance: Executing topDown query # 3 0.151 0.249 -0.10
@QueryPerformanceTests @performance: Executing topDown query # 6 0.387 0.419 -0.03
@SubmissionPerformanceTests @performance: Submission acknowledgement 10 0.435 0.35 0.09
@SubmissionPerformanceTests @performance: Submission acknowledgement 1000 24.62 22.503 2.12
@SubmissionPerformanceTests @performance: Submission aligned_reads_index 10 2.195 1.706 0.49
@SubmissionPerformanceTests @performance: Submission aliquot 100 2.6 2.281 0.32
@SubmissionPerformanceTests @performance: Submission case 100 5.04 2.35 2.69
@SubmissionPerformanceTests @performance: Submission clinical_test 10 0.501 0.424 0.08
@SubmissionPerformanceTests @performance: Submission core_metadata_collection 10 0.82 0.753 0.07
@SubmissionPerformanceTests @performance: Submission core_metadata_collection 100 5.222 2.486 2.74
@SubmissionPerformanceTests @performance: Submission demographic 10 0.482 0.383 0.10
@SubmissionPerformanceTests @performance: Submission demographic 100 2.94 2.733 0.21
@SubmissionPerformanceTests @performance: Submission demographic 1000 23.656 27.738 -4.08
@SubmissionPerformanceTests @performance: Submission diagnosis 10 0.808 0.392 0.42
@SubmissionPerformanceTests @performance: Submission experiment 10 0.482 0.399 0.08
@SubmissionPerformanceTests @performance: Submission experiment 100 3.124 2.382 0.74
@SubmissionPerformanceTests @performance: Submission experimental_metadata 10 2.097 1.812 0.28
@SubmissionPerformanceTests @performance: Submission exposure 10 0.53 0.371 0.16
@SubmissionPerformanceTests @performance: Submission exposure 100 3.279 2.368 0.91
@SubmissionPerformanceTests @performance: Submission exposure 1000 25.574 23.676 1.90
@SubmissionPerformanceTests @performance: Submission family_history 10 0.433 0.351 0.08
@SubmissionPerformanceTests @performance: Submission family_history 1000 22.819 24.246 -1.43
@SubmissionPerformanceTests @performance: Submission keyword 10 0.423 0.353 0.07
@SubmissionPerformanceTests @performance: Submission keyword 100 2.125 2.384 -0.26
@SubmissionPerformanceTests @performance: Submission keyword 1000 22.636 24.347 -1.71
@SubmissionPerformanceTests @performance: Submission publication 10 0.383 0.349 0.03
@SubmissionPerformanceTests @performance: Submission publication 100 2.623 2.421 0.20
@SubmissionPerformanceTests @performance: Submission publication 1000 23.886 21.918 1.97
@SubmissionPerformanceTests @performance: Submission read_group 100 2.826 2.565 0.26
@SubmissionPerformanceTests @performance: Submission read_group_qc 100 4.548 4.2 0.35
@SubmissionPerformanceTests @performance: Submission read_group_qc 1000 27.07 29.453 -2.38
@SubmissionPerformanceTests @performance: Submission sample 10 0.564 0.404 0.16
@SubmissionPerformanceTests @performance: Submission sample 100 3.361 2.733 0.63
@SubmissionPerformanceTests @performance: Submission slide 10 0.379 0.466 -0.09
@SubmissionPerformanceTests @performance: Submission slide 100 4.568 2.373 2.19
@SubmissionPerformanceTests @performance: Submission slide_count 10 0.359 0.443 -0.08
@SubmissionPerformanceTests @performance: Submission slide_count 1000 24.834 23.128 1.71
@SubmissionPerformanceTests @performance: Submission slide_image 10 2.434 1.703 0.73
@SubmissionPerformanceTests @performance: Submission submitted_aligned_reads 10 1.613 1.806 -0.19
@SubmissionPerformanceTests @performance: Submission submitted_copy_number 10 2.554 1.719 0.83
@SubmissionPerformanceTests @performance: Submission submitted_copy_number 100 16.672 15.821 0.85
@SubmissionPerformanceTests @performance: Submission submitted_methylation 10 1.635 1.79 -0.15
@SubmissionPerformanceTests @performance: Submission submitted_somatic_mutation 10 2.493 1.693 0.80
@SubmissionPerformanceTests @performance: Submission treatment 10 0.578 0.37 0.21
@SubmissionPerformanceTests @performance: Submission treatment 100 2.535 2.413 0.12
@SubmissionPerformanceTests @performance: Submission treatment 1000 24.639 23.32 1.32

@fantix fantix merged commit 2425220 into master Apr 29, 2019
@fantix fantix deleted the fix/submission branch April 29, 2019 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@philloooo philloooo philloooo approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@fantix @PlanXCyborg @philloooo