Clickhousify samples endpoint #11393

onursumer · 2025-02-12T23:35:05Z

haynescd

Happy to discuss. I thought the new clickhouse work will utilize the clean arch approach. I can give you more details tomorrow

sonarqubecloud · 2025-02-26T18:27:19Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
1.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

haynescd

Looks good (needs a couple of changes) still looking at the derived sample table

haynescd · 2025-03-07T04:03:35Z

src/main/java/org/cbioportal/domain/sample/usecase/FetchMetaSamplesHeadersUseCase.java

+
+@Service
+@Profile("clickhouse")
+public class FetchMetaSamplesHeadersUseCase {


I would remove these use cases that add Http Headers and put this in the controller

haynescd · 2025-03-07T04:09:37Z

src/main/java/org/cbioportal/domain/sample/usecase/GetMetaSamplesHeadersUseCase.java

+
+@Service
+@Profile("clickhouse")
+public class GetMetaSamplesHeadersUseCase {


Same comment as above

haynescd · 2025-03-07T04:11:12Z

.../java/org/cbioportal/domain/sample/usecase/GetMetaSamplesOfPatientInStudyHeadersUseCase.java

+
+@Service
+@Profile("clickhouse")
+public class GetMetaSamplesOfPatientInStudyHeadersUseCase {


Same I would remove these http headers usse case

haynescd · 2025-03-10T13:31:42Z

src/main/resources/db-scripts/clickhouse/clickhouse.sql

+    sample_unique_id            String,
+    sample_unique_id_base64     String,
+    sample_stable_id            String,
+    patient_unique_id           String,
+    patient_unique_id_base64    String,
+    patient_stable_id           String,
+    cancer_study_identifier     LowCardinality(String),
+    internal_id                 Int,
+    -- fields below are needed for the DETAILED projection
+    sequenced                   Int,
+    copy_number_segment_present Int,
+    patient_internal_id         Int,
+    sample_type                 String,
+    cancer_study_id             Int,
+    type_of_cancer_id           String,
+    cancer_study_name           String,
+    cancer_study_description    String,
+    cancer_study_public         Int,
+    cancer_study_pmid           String,
+    cancer_study_citation       String,
+    cancer_study_groups         String,
+    cancer_study_status         Int,
+    cancer_study_import_date    DateTime,
+    reference_genome            String


Also, would be curious how much speed we get with the wide column vs just doing a simple join or using a dictionary. Don't want to just randomly add more columns to these tables if they aren't needed

https://github.com/cBioPortal/cbioportal/pull/11393/files#diff-90bb502239e23d89960abe06053613ae64223d06b81e263425d1c32026d5728fR86-R104

I think we should still keep this logic and the two additional columns (sequenced and copy_number_segment_present). The rest of the columns are straightforward to obtain with a simple join and won't probably cause much performance issue. I'll look into that.

Just tried moving everything, including the sequenced and copy_number_segment_present logic, to the SQL layer. It doesn't seem to degrade the performance significantly. The wide table is slightly faster but probably negligible.

Reduced the number of columns by 34f3b77.

For now keeping sequenced and copy_number_segment_present as well as sample_type and patient_internal_id, but it is also possible to derive them in SQL if we don't want to add any new columns to sample_derived at all.

alisman · 2025-04-22T14:55:02Z

src/main/java/org/cbioportal/domain/sample/Sample.java

+        );
+    }
+
+    public Sample(


We should have a comment here explaining the difference between these Sample overloads

alisman · 2025-04-22T14:59:31Z

...a/org/cbioportal/infrastructure/repository/clickhouse/sample/ClickhouseSampleRepository.java

+        List<String> sampleIds,
+        ProjectionType projection
+    ) {
+        return switch (projection) {


@onursumer where do pagination params go?

@alisman This method doesn't support pagination. Legacy implementation supports pagination only for certain methods where we get all samples. I kept it the same way for the clickhouse implementation.

cbioportal/src/main/java/org/cbioportal/infrastructure/repository/clickhouse/sample/ClickhouseSampleRepository.java

Lines 79 to 80 in fa1847f

Integer pageSize,

Integer pageNumber,

cbioportal/src/main/java/org/cbioportal/infrastructure/repository/clickhouse/sample/ClickhouseSampleRepository.java

Lines 110 to 111 in fa1847f

Integer pageSize,

Integer pageNumber,

cbioportal/src/main/java/org/cbioportal/infrastructure/repository/clickhouse/sample/ClickhouseSampleRepository.java

Lines 133 to 134 in fa1847f

Integer pageSize,

Integer pageNumber,

haynescd · 2025-04-24T14:57:14Z

src/main/resources/mappers/clickhouse/sample/ClickhouseSampleMapper.xml

+    <sql id="sampleColumnsSummary">
+        <include refid="sampleColumnsId" />,
+        sample_type as sampleType,
+        patient_internal_id as patientId
+    </sql>
+
+    <sql id="sampleColumnsDetailed">
+        <include refid="sampleColumnsSummary" />,
+        sequenced as sequenced,
+        copy_number_segment_present as copyNumberSegmentPresent,
+        patient_internal_id as "patient.internalId",
+        patient_stable_id as "patient.stableId",
+        cs.cancer_study_id as "patient.cancerStudyId",
+        cs.cancer_study_identifier as "patient.cancerStudyIdentifier",
+        cs.cancer_study_id as "patient.cancerStudy.cancerStudyId",
+        cs.cancer_study_identifier as "patient.cancerStudy.cancerStudyIdentifier",
+        cs.type_of_cancer_id as "patient.cancerStudy.typeOfCancerId",
+        cs.name as "patient.cancerStudy.name",
+        cs.description as "patient.cancerStudy.description",
+        cs.public as "patient.cancerStudy.publicStudy",
+        cs.pmid as "patient.cancerStudy.pmid",
+        cs.citation as "patient.cancerStudy.citation",
+        cs.groups as "patient.cancerStudy.groups",
+        cs.status as "patient.cancerStudy.status",
+        cs.import_date as "patient.cancerStudy.importDate",
+        rg.name as "patient.cancerStudy.referenceGenome"
+    </sql>


So, I would say this is a good abstraction but the extra abstraction makes it easier to get lost.

The balance is to not duplicate code, but also don't abstract so much

haynescd · 2025-04-24T14:57:38Z

src/main/resources/mappers/clickhouse/sample/ClickhouseSampleMapper.xml

+    <sql id="selectSample">
+        SELECT
+            <include refid="sampleColumnsId" />
+        FROM sample_derived
+    </sql>
+
+    <sql id="selectSummarySample">
+        SELECT
+            <include refid="sampleColumnsSummary" />
+        FROM sample_derived
+    </sql>
+
+    <sql id="selectDetailedSample">
+        SELECT
+            <include refid="sampleColumnsDetailed" />
+        <include refid="fromJoinedTable" />
+    </sql>
+
+    <sql id="selectMetaSample">
+        SELECT
+            COUNT(*) as totalCount
+        FROM sample_derived
+    </sql>


I would remove these, but keep the other abstractions

haynescd

Small change, but after that It is good

…lasses

…_derived

…d projection

sonarqubecloud · 2025-04-29T17:41:14Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
7.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

onursumer force-pushed the clickhousify-samples-endpoint branch 2 times, most recently from 84339f7 to 3b627d6 Compare February 13, 2025 00:28

dippindots assigned onursumer Feb 13, 2025

onursumer force-pushed the clickhousify-samples-endpoint branch from 3b627d6 to fb11408 Compare February 13, 2025 18:35

onursumer marked this pull request as ready for review February 13, 2025 19:27

onursumer requested review from alisman and haynescd February 13, 2025 19:27

onursumer added the needs tests label Feb 13, 2025

haynescd requested changes Feb 13, 2025

View reviewed changes

onursumer force-pushed the clickhousify-samples-endpoint branch 5 times, most recently from 374a048 to ae4e13a Compare February 24, 2025 21:00

dippindots requested a review from haynescd February 25, 2025 16:46

onursumer force-pushed the clickhousify-samples-endpoint branch 3 times, most recently from 2210ab2 to e329f81 Compare February 26, 2025 18:22

onursumer force-pushed the clickhousify-samples-endpoint branch 4 times, most recently from 8060e80 to f5cd65c Compare February 28, 2025 19:43

onursumer removed the needs tests label Feb 28, 2025

onursumer force-pushed the clickhousify-samples-endpoint branch 3 times, most recently from 6855908 to 5fdbe7e Compare March 3, 2025 20:37

haynescd requested changes Mar 7, 2025

View reviewed changes

onursumer force-pushed the clickhousify-samples-endpoint branch from 5fdbe7e to 342ca0d Compare March 7, 2025 18:14

haynescd reviewed Mar 10, 2025

View reviewed changes

dippindots requested a review from haynescd April 15, 2025 15:08

onursumer force-pushed the clickhousify-samples-endpoint branch from 1ac1cbc to 7b2dc5b Compare April 17, 2025 17:56

onursumer added api Clickhouse labels Apr 17, 2025

alisman reviewed Apr 22, 2025

View reviewed changes

dippindots approved these changes Apr 22, 2025

View reviewed changes

alisman reviewed Apr 22, 2025

View reviewed changes

onursumer force-pushed the clickhousify-samples-endpoint branch from 7b2dc5b to d0a8ece Compare April 22, 2025 18:35

haynescd reviewed Apr 24, 2025

View reviewed changes

haynescd requested changes Apr 24, 2025

View reviewed changes

onursumer force-pushed the clickhousify-samples-endpoint branch from d0a8ece to 8fc22ac Compare April 24, 2025 19:38

haynescd approved these changes Apr 24, 2025

View reviewed changes

onursumer force-pushed the clickhousify-samples-endpoint branch from 8fc22ac to e3ae3f3 Compare April 28, 2025 19:54

onursumer mentioned this pull request Apr 28, 2025

Enable validation for the new alteration enrichments and samples endpoints cBioPortal/cbioportal-frontend#5176

Merged

onursumer added 11 commits April 29, 2025 13:35

add sample column store controller, service, repository, and mapper c…

239a9ec

…lasses

add sample_type, sequenced, and copy_number_segment_present to sample…

282a7f6

…_derived

add fields required for detailed projection

88f4861

add meta samples mappers

4416473

add remaining endpoints

c80c96e

reorganize files in favor of the clickhouse only clean architecture

b36fc38

reduce number of sample_derived columns by using join for the detaile…

d5642b4

…d projection

replace sample service with individual use cases

de4b23e

use ArrayType Handler wherever possible

b66183c

add tests

681c074

add support for sample filter in hasPermission

cbf8bed

onursumer force-pushed the clickhousify-samples-endpoint branch from e3ae3f3 to cbf8bed Compare April 29, 2025 17:35

onursumer merged commit 14b4975 into cBioPortal:master Apr 30, 2025
19 of 22 checks passed

onursumer deleted the clickhousify-samples-endpoint branch April 30, 2025 16:54

Uh oh!

Clickhousify samples endpoint #11393

Clickhousify samples endpoint #11393

Uh oh!

Conversation

onursumer commented Feb 12, 2025

Uh oh!

haynescd left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Feb 26, 2025

Quality Gate passed

Uh oh!

haynescd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haynescd left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Apr 29, 2025

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants