Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 61385e9

Browse files
alexey-milovidovalexey-milovidov
committed
ClickBench/jd: explicit cdefs to keep load inside disk budget
The previous load relied on csvload_jd_'s auto-inference, which sampled the first 5000 rows for types and then ran csvscan to widen any byte columns to the full-file max. ClickBench has many sparse text columns whose 5000-row sample looked empty: they were typed as `byte`, then later widened to hundreds of chars × 100M rows. The splayed table grew past 500 GB during csvload and the loader hit a bus error. Skip csvcdefs/csvscan and write an explicit hits.cdefs: `varbyte` for every TEXT/VARCHAR/CHAR column, `int` (8-byte JINT) for every numeric column, and `edate`/`edatetime` for the date and timestamp columns. Switch to `int` rather than int1/int2/int4 because Jd leaves the latter as n,x char matrices and the `<>` predicate then fails on a shape-2 col vs a shape-0 scalar. Query adjustments forced by the new types: - Q23 swaps `min URL,min Title` (Jd has no varbyte aggregator) for `first URL,first Title` — semantically `ANY_VALUE`. - Q28 (`AVG(LENGTH(URL))`) joins Q29/Q43 in the `'null'` bucket. - Q25/Q27 add EventTime to the projection (Jd's `reads` rejects order-by columns that aren't in the select list). - Q5/Q6 use `# ~. ; }. jd '…'` so the unique scan skips the header row that Jd prepends to every result. - Q37-42 swap `EventDate range (15887,15917)` for the iso8601 string form `range ("2013-07-01","2013-07-31")` matching edate's literal grammar. All 43 queries execute on a 100k-row slice; disk usage is ~145 MB for that slice (≈145 GB extrapolated to 100M rows, comfortably inside the 500 GB cloud-init budget).
1 parent 6ac10cf commit 61385e9

3 files changed

Lines changed: 205 additions & 37 deletions

File tree

‎jd/README.md‎

Lines changed: 47 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -40,10 +40,37 @@ plus J operators for things Jd's query layer doesn't ship (`LIMIT`,
4040

4141
## Load
4242

43-
`./load` ingests `hits.csv` via Jd's built-in CSV loader
44-
(`csvprepare_jd_` + `csvload_jd_`). The loader writes per-column
45-
files to a dedicated database under `~/j9.6-user/temp/jd/csvload/`;
46-
that's the database `./query` opens.
43+
`./load` ingests `hits.csv` via Jd's CSV loader with an **explicit
44+
column schema** instead of `csvload_jd_`'s auto-inference. The
45+
default flow types every string column by sampling the first 5000
46+
rows and then runs `csvscan` to widen any column it inferred as
47+
`byte` to the full-file max width. ClickBench has very sparse text
48+
columns (e.g. `OpenstatServiceName`, `SocialNetwork`) that look
49+
empty in the 5000-row sample → typed as `byte`, then later widened
50+
to hundreds of chars × 100 M rows. With ~30 such columns the
51+
splayed table grew past 500 GB during the load and segfaulted.
52+
Declaring text columns as `varbyte` (variable-length, per-row
53+
offset + concatenated data) keeps storage proportional to actual
54+
string content. The script writes a hand-rolled `hits.cdefs` file
55+
into the csvload jdcsv folder, then calls `csvrd` directly,
56+
skipping `csvcdefs` (auto-type) and `csvscan` (byte-width
57+
widening).
58+
59+
Schema choices:
60+
61+
* `int` (8-byte signed) for every numeric column. Jd's `int1` /
62+
`int2` / `int4` leave per-row data as `n,x` char matrices, and
63+
the `<>` predicate then sees a shape-2 column vs a shape-0
64+
scalar, so we use the flat 8-byte JINT form everywhere.
65+
* `varbyte` for TEXT / VARCHAR / CHAR.
66+
* `edate` for `EventDate`, `edatetime` for the three TIMESTAMP
67+
columns. Both are 8-byte epoch-nanos and Jd's csv loader parses
68+
iso8601 from `iso8601-char` mode (CSV format is
69+
`YYYY-MM-DD` / `YYYY-MM-DD HH:MM:SS`).
70+
71+
The loader writes per-column files to a dedicated database under
72+
`~/j9.6-user/temp/jd/csvload/`; that's the database `./query`
73+
opens.
4774

4875
## Query
4976

@@ -62,15 +89,22 @@ places:
6289
* **`LIMIT n OFFSET m`** uses `n {. m }. jd '...'`.
6390
* **`COUNT(DISTINCT col)`** uses J's `# ~.` (count of unique items)
6491
after pulling the column with `jd 'reads col from t'`.
65-
* **Q29** (`REGEXP_REPLACE`) and **Q43** (`DATE_TRUNC('minute', ...)`)
66-
use facilities not in Jd's `reads` language; they currently return
67-
the literal `'null'` and the benchmark driver records them as
68-
missing. They could be expressed with a J-side computed column —
69-
contributions welcome.
70-
71-
`EventDate` literals (`'2013-07-01'`, etc.) in Q37–Q42 are encoded as
72-
days-since-epoch integers (the form Jd stores `EventDate` in after the
73-
CSV load): 2013-07-01 = day 15887, 2013-07-31 = day 15917.
92+
* **`min` / `avg` on `varbyte`**: Jd's aggregators are numeric-only,
93+
so Q23's `MIN(URL)` / `MIN(Title)` become `first URL` / `first Title`
94+
(any value from each group, semantically `ANY_VALUE`).
95+
* **Q28** (`AVG(LENGTH(URL))`), **Q29** (`REGEXP_REPLACE`), and
96+
**Q43** (`DATE_TRUNC('minute', ...)`) use facilities not in Jd's
97+
`reads` language; they currently return the literal `'null'` and
98+
the benchmark driver records them as missing. They could be
99+
expressed with a J-side computed column — contributions welcome.
100+
* **`order by` requires the column in `select`**: Jd's parser rejects
101+
`reads SearchPhrase from hits order by EventTime` because the order
102+
key isn't projected. Q25 / Q27 are rewritten to project
103+
`EventTime,SearchPhrase` (timing unaffected; only the printed output
104+
has one extra column).
105+
* **`COUNT(DISTINCT col)`**: outside `reads`, J's `# ~. ; }. jd '…'`
106+
(count of unique, after dropping the header row). The `}.` drops
107+
the header box so the unique scan only sees the data values.
74108

75109
## Performance notes
76110

‎jd/load‎

Lines changed: 146 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,25 +1,159 @@
11
#!/bin/bash
2-
# Load hits.csv into a Jd database via Jd's built-in CSV loader. The
3-
# loader creates / writes to a dedicated `csvload` database under
4-
# ~/j9.6-user/temp/jd/csvload — `query.ijs` opens that same DB.
2+
# Load hits.csv into a Jd database via Jd's CSV loader, using an
3+
# explicit column schema instead of csvcdefs' auto-inference.
54
#
6-
# ClickBench's hits.csv has no header row, so we use `csvload_jd_
7-
# 'hits';0` (treat the first row as data, Jd assigns sequential
8-
# default column names c1, c2, …) and then rename to the canonical
9-
# ClickBench schema via `csvrename_jd_`.
5+
# Why explicit: the high-level csvload_jd_ samples the first 5000
6+
# rows to pick types, then csvscan widens any column it inferred as
7+
# `byte` to the full-file max width. ClickBench has very sparse
8+
# text columns (OpenstatServiceName, SocialNetwork, …) that look
9+
# empty in the sample → typed as `byte`, then later scan widens
10+
# them to hundreds of chars × 100M rows. With 30 such columns the
11+
# splayed table grew past 500 GB during the load and segfaulted.
12+
# Declaring text columns as `varbyte` (variable-length, per-row
13+
# offset + concatenated data) keeps storage proportional to actual
14+
# string content.
1015
set -e
1116

1217
# Reset any prior csvload DB so we measure a clean load.
1318
rm -rf "$HOME/j9.6-user/temp/jd/csvload"
1419

1520
ijconsole </dev/null <<'JEOF'
1621
load 'data/jd/jd'
17-
csvprepare_jd_ 'hits';'hits.csv'
18-
csvload_jd_ 'hits';0 NB. 0 = no header row, default names c1..c105
1922
20-
newn=: ;:'WatchID JavaEnable Title GoodEvent EventTime EventDate CounterID ClientIP RegionID UserID CounterClass OS UserAgent URL Referer IsRefresh RefererCategoryID RefererRegionID URLCategoryID URLRegionID ResolutionWidth ResolutionHeight ResolutionDepth FlashMajor FlashMinor FlashMinor2 NetMajor NetMinor UserAgentMajor UserAgentMinor CookieEnable JavascriptEnable IsMobile MobilePhone MobilePhoneModel Params IPNetworkID TraficSourceID SearchEngineID SearchPhrase AdvEngineID IsArtifical WindowClientWidth WindowClientHeight ClientTimeZone ClientEventTime SilverlightVersion1 SilverlightVersion2 SilverlightVersion3 SilverlightVersion4 PageCharset CodeVersion IsLink IsDownload IsNotBounce FUniqID OriginalURL HID IsOldCounter IsEvent IsParameter DontCountHits WithHash HitColor LocalEventTime Age Sex Income Interests Robotness RemoteIP WindowName OpenerName HistoryLength BrowserLanguage BrowserCountry SocialNetwork SocialAction HTTPError SendTiming DNSTiming ConnectTiming ResponseStartTiming ResponseEndTiming FetchTiming SocialSourceNetworkID SocialSourcePage ParamPrice ParamOrderID ParamCurrency ParamCurrencyID OpenstatServiceName OpenstatCampaignID OpenstatAdID OpenstatSourceID UTMSource UTMMedium UTMCampaign UTMContent UTMTerm FromTag HasGCLID RefererHash URLHash CLID'
21-
oldn=: {."1 jd 'read from hits'
22-
csvrename_jd_ 'hits';oldn;<newn
23+
NB. Open (create on first use) the csvload DB and prep its jdcsv folder.
24+
NB. We replicate what csvprepare_jd_ does — admin + jdcsvfolder + write the
25+
NB. csvlink — then write our own cdefs file and call csvrd directly,
26+
NB. skipping csvcdefs (auto-type) and csvscan (byte-width widening).
27+
csvadmin_jd_ 'csvload'
28+
jdcsvfolder_jd_ ''
29+
30+
'hits.csv' fwrite CSVFOLDER,'hits.csvlink'
31+
32+
NB. Column schema. Types:
33+
NB. int — 8-byte signed; all SMALLINT / INTEGER / BIGINT columns.
34+
NB. (Jd's int1/int2/int4 leave per-row data as n,x char
35+
NB. matrices and the `<>` predicate then sees a shape-2 col
36+
NB. vs a shape-0 scalar, so we use the flat 8-byte JINT
37+
NB. form for every numeric column.)
38+
NB. varbyte — variable-length string; TEXT / VARCHAR / CHAR
39+
NB. edate — 8-byte epoch-nanos; DATE (EventDate)
40+
NB. edatetime — 8-byte epoch-nanos; TIMESTAMP (EventTime, ClientEventTime,
41+
NB. LocalEventTime). Iso8601-char parses the `YYYY-MM-DD HH:MM:SS`
42+
NB. form in the csv.
43+
cdefs =: 0 : 0
44+
1 WatchID int
45+
2 JavaEnable int
46+
3 Title varbyte
47+
4 GoodEvent int
48+
5 EventTime edatetime
49+
6 EventDate edate
50+
7 CounterID int
51+
8 ClientIP int
52+
9 RegionID int
53+
10 UserID int
54+
11 CounterClass int
55+
12 OS int
56+
13 UserAgent int
57+
14 URL varbyte
58+
15 Referer varbyte
59+
16 IsRefresh int
60+
17 RefererCategoryID int
61+
18 RefererRegionID int
62+
19 URLCategoryID int
63+
20 URLRegionID int
64+
21 ResolutionWidth int
65+
22 ResolutionHeight int
66+
23 ResolutionDepth int
67+
24 FlashMajor int
68+
25 FlashMinor int
69+
26 FlashMinor2 varbyte
70+
27 NetMajor int
71+
28 NetMinor int
72+
29 UserAgentMajor int
73+
30 UserAgentMinor varbyte
74+
31 CookieEnable int
75+
32 JavascriptEnable int
76+
33 IsMobile int
77+
34 MobilePhone int
78+
35 MobilePhoneModel varbyte
79+
36 Params varbyte
80+
37 IPNetworkID int
81+
38 TraficSourceID int
82+
39 SearchEngineID int
83+
40 SearchPhrase varbyte
84+
41 AdvEngineID int
85+
42 IsArtifical int
86+
43 WindowClientWidth int
87+
44 WindowClientHeight int
88+
45 ClientTimeZone int
89+
46 ClientEventTime edatetime
90+
47 SilverlightVersion1 int
91+
48 SilverlightVersion2 int
92+
49 SilverlightVersion3 int
93+
50 SilverlightVersion4 int
94+
51 PageCharset varbyte
95+
52 CodeVersion int
96+
53 IsLink int
97+
54 IsDownload int
98+
55 IsNotBounce int
99+
56 FUniqID int
100+
57 OriginalURL varbyte
101+
58 HID int
102+
59 IsOldCounter int
103+
60 IsEvent int
104+
61 IsParameter int
105+
62 DontCountHits int
106+
63 WithHash int
107+
64 HitColor varbyte
108+
65 LocalEventTime edatetime
109+
66 Age int
110+
67 Sex int
111+
68 Income int
112+
69 Interests int
113+
70 Robotness int
114+
71 RemoteIP int
115+
72 WindowName int
116+
73 OpenerName int
117+
74 HistoryLength int
118+
75 BrowserLanguage varbyte
119+
76 BrowserCountry varbyte
120+
77 SocialNetwork varbyte
121+
78 SocialAction varbyte
122+
79 HTTPError int
123+
80 SendTiming int
124+
81 DNSTiming int
125+
82 ConnectTiming int
126+
83 ResponseStartTiming int
127+
84 ResponseEndTiming int
128+
85 FetchTiming int
129+
86 SocialSourceNetworkID int
130+
87 SocialSourcePage varbyte
131+
88 ParamPrice int
132+
89 ParamOrderID varbyte
133+
90 ParamCurrency varbyte
134+
91 ParamCurrencyID int
135+
92 OpenstatServiceName varbyte
136+
93 OpenstatCampaignID varbyte
137+
94 OpenstatAdID varbyte
138+
95 OpenstatSourceID varbyte
139+
96 UTMSource varbyte
140+
97 UTMMedium varbyte
141+
98 UTMCampaign varbyte
142+
99 UTMContent varbyte
143+
100 UTMTerm varbyte
144+
101 FromTag varbyte
145+
102 HasGCLID int
146+
103 RefererHash int
147+
104 URLHash int
148+
105 CLID int
149+
options , LF " NO 0 iso8601-char
150+
)
151+
152+
cdefs fwrite CSVFOLDER,'hits.cdefs'
153+
154+
NB. Read csv into the `hits` table using our cdefs.
155+
jd 'csvrd hits.csvlink hits'
156+
jd 'csvreport /f hits'
23157
exit ''
24158
JEOF
25159

‎jd/queries.sql‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@ jd 'reads count jdindex from hits'
22
jd 'reads count jdindex from hits where AdvEngineID <> 0'
33
jd 'reads sum AdvEngineID,count jdindex,avg ResolutionWidth from hits'
44
jd 'reads avg UserID from hits'
5-
# ~. ; jd 'reads UserID from hits'
6-
# ~. ; jd 'reads SearchPhrase from hits'
5+
# ~. ; }. jd 'reads UserID from hits'
6+
# ~. ; }. jd 'reads SearchPhrase from hits'
77
jd 'reads min EventDate,max EventDate from hits'
88
10 {. jd 'reads c:count jdindex by AdvEngineID from hits where AdvEngineID <> 0 order by c desc'
99
10 {. jd 'reads u:count jdindex by RegionID from hits order by u desc'
@@ -20,12 +20,12 @@ jd 'reads min EventDate,max EventDate from hits'
2020
jd 'reads UserID from hits where UserID = 435090932899640449'
2121
jd 'reads count jdindex from hits where URL like ".*google.*"'
2222
10 {. jd 'reads min URL,c:count jdindex by SearchPhrase from hits where URL like ".*google.*" && SearchPhrase <> "" order by c desc'
23-
10 {. jd 'reads min URL,min Title,c:count jdindex,d:count jdindex by SearchPhrase from hits where Title like ".*Google.*" && URL unlike ".*\.google\..*" && SearchPhrase <> "" order by c desc'
23+
10 {. jd 'reads first URL,first Title,c:count jdindex,d:countunique UserID by SearchPhrase from hits where Title like ".*Google.*" && URL unlike ".*\.google\..*" && SearchPhrase <> "" order by c desc'
2424
10 {. jd 'reads * from hits where URL like ".*google.*" order by EventTime'
25-
10 {. jd 'reads SearchPhrase from hits where SearchPhrase <> "" order by EventTime'
25+
10 {. jd 'reads EventTime,SearchPhrase from hits where SearchPhrase <> "" order by EventTime'
2626
10 {. jd 'reads SearchPhrase from hits where SearchPhrase <> "" order by SearchPhrase'
27-
10 {. jd 'reads SearchPhrase from hits where SearchPhrase <> "" order by EventTime,SearchPhrase'
28-
25 {. jd 'reads l:avg URL,c:count jdindex by CounterID from hits where URL <> "" order by l desc'
27+
10 {. jd 'reads EventTime,SearchPhrase from hits where SearchPhrase <> "" order by EventTime,SearchPhrase'
28+
'null'
2929
'null'
3030
jd 'reads sum ResolutionWidth from hits'
3131
10 {. jd 'reads c:count jdindex,sum IsRefresh,avg ResolutionWidth by SearchEngineID,ClientIP from hits where SearchPhrase <> "" order by c desc'
@@ -34,10 +34,10 @@ jd 'reads sum ResolutionWidth from hits'
3434
10 {. jd 'reads c:count jdindex by URL from hits order by c desc'
3535
10 {. jd 'reads c:count jdindex by URL from hits order by c desc'
3636
10 {. jd 'reads c:count jdindex by ClientIP from hits order by c desc'
37-
10 {. jd 'reads c:count jdindex by URL from hits where CounterID=62 && EventDate range (15887,15917) && DontCountHits=0 && IsRefresh=0 && URL <> "" order by c desc'
38-
10 {. jd 'reads c:count jdindex by Title from hits where CounterID=62 && EventDate range (15887,15917) && DontCountHits=0 && IsRefresh=0 && Title <> "" order by c desc'
39-
10 {. (1000 }. jd 'reads c:count jdindex by URL from hits where CounterID=62 && EventDate range (15887,15917) && IsRefresh=0 && IsLink<>0 && IsDownload=0 order by c desc')
40-
10 {. (1000 }. jd 'reads c:count jdindex by TraficSourceID,SearchEngineID,AdvEngineID,Referer,URL from hits where CounterID=62 && EventDate range (15887,15917) && IsRefresh=0 order by c desc')
41-
10 {. (100 }. jd 'reads c:count jdindex by URLHash,EventDate from hits where CounterID=62 && EventDate range (15887,15917) && IsRefresh=0 && TraficSourceID in (-1,6) && RefererHash=3594120000172545465 order by c desc')
42-
10 {. (10000 }. jd 'reads c:count jdindex by WindowClientWidth,WindowClientHeight from hits where CounterID=62 && EventDate range (15887,15917) && IsRefresh=0 && DontCountHits=0 && URLHash=2868770270353813622 order by c desc')
37+
10 {. jd 'reads c:count jdindex by URL from hits where CounterID=62 && EventDate range ("2013-07-01","2013-07-31") && DontCountHits=0 && IsRefresh=0 && URL <> "" order by c desc'
38+
10 {. jd 'reads c:count jdindex by Title from hits where CounterID=62 && EventDate range ("2013-07-01","2013-07-31") && DontCountHits=0 && IsRefresh=0 && Title <> "" order by c desc'
39+
10 {. (1000 }. jd 'reads c:count jdindex by URL from hits where CounterID=62 && EventDate range ("2013-07-01","2013-07-31") && IsRefresh=0 && IsLink<>0 && IsDownload=0 order by c desc')
40+
10 {. (1000 }. jd 'reads c:count jdindex by TraficSourceID,SearchEngineID,AdvEngineID,Referer,URL from hits where CounterID=62 && EventDate range ("2013-07-01","2013-07-31") && IsRefresh=0 order by c desc')
41+
10 {. (100 }. jd 'reads c:count jdindex by URLHash,EventDate from hits where CounterID=62 && EventDate range ("2013-07-01","2013-07-31") && IsRefresh=0 && TraficSourceID in (-1,6) && RefererHash=3594120000172545465 order by c desc')
42+
10 {. (10000 }. jd 'reads c:count jdindex by WindowClientWidth,WindowClientHeight from hits where CounterID=62 && EventDate range ("2013-07-01","2013-07-31") && IsRefresh=0 && DontCountHits=0 && URLHash=2868770270353813622 order by c desc')
4343
'null'

0 commit comments

Comments
 (0)