Commits

Branch selector

 main

User selector

wbrown

Datepicker

All time

Commit History

Commits on Jul 9, 2025

• Merge pull request #63 from kurumuz/kurumuz.wasm-target

  Show description for 1571a6e
  wbrown

  authored
  1571a6e

  Copy full SHA for 1571a6e

Commits on Jul 7, 2025

• Merge pull request #62 from kurumuz/kurumuz.wasm-target

  Show description for 9e94b78
  wbrown

  authored
  9e94b78

  Copy full SHA for 9e94b78

Commits on Apr 23, 2025

• Merge pull request #55 from wbrown/wbrown.fix-streaming-add-md

  Show description for 7e0719a
  wbrown

  authored
  7e0719a

  Copy full SHA for 7e0719a

• Merge pull request #58 from wbrown/rwang.optimizeSplitter10072024

  Show description for 31e8561
  wbrown

  authored
  31e8561

  Copy full SHA for 31e8561

Commits on Nov 4, 2024

• Merge pull request #56 from wbrown/rwang.githubActions2.10182024

  Show description for fb3c65c
  wbrown

  authored
  fb3c65c

  Copy full SHA for fb3c65c

Commits on Oct 25, 2024

• Add search for `text` fields.

  wbrown

  committed
  3378de5

  Copy full SHA for 3378de5

Commits on Oct 24, 2024

• Splitter now returns batches of words rather than return individual words.

  wbrown

  committed
  eaa203d

  Copy full SHA for eaa203d

• Don't create a list of possible BPERanks, instead find the highest match.

  wbrown

  committed
  1f7a14e

  Copy full SHA for 1f7a14e

• Fix accumulator bug.

  wbrown

  committed
  62e8d40

  Copy full SHA for 62e8d40

Commits on Oct 22, 2024

• Correct offsets and tokens used in the index.

  wbrown

  committed
  b17155c

  Copy full SHA for b17155c

• Remove replace directive.

  wbrown

  committed
  31d63ac

  Copy full SHA for 31d63ac

• Changes in how we split words, and handle caches.

  wbrown

  committed
  d8a6a83

  Copy full SHA for d8a6a83

Commits on Oct 21, 2024

• Improve reporting of queues.

  wbrown

  committed
  f039c8b

  Copy full SHA for f039c8b

• Add appropriate exit codes.

  wbrown

  committed
  a641752

  Copy full SHA for a641752

• We should not be shuffling if the spec is none.

  wbrown

  committed
  3d07a1b

  Copy full SHA for 3d07a1b

• Change default context size from 2048 to 8192.

  wbrown

  committed
  fd24004

  Copy full SHA for fd24004

• Use a buffered reader on JSONL as well, and clean up some output.

  wbrown

  committed
  5a58781

  Copy full SHA for 5a58781

• Pre-read 1mb before putting reader onto channel.

  wbrown

  committed
  2beaf50

  Copy full SHA for 2beaf50

Commits on Oct 20, 2024

• Fix go.mod

  wbrown

  committed
  c2b1ae7

  Copy full SHA for c2b1ae7

• Increase channel depths to hopefully compensate for VAST.

  wbrown

  committed
  bfd4dc7

  Copy full SHA for bfd4dc7

• Merge pull request #54 from wbrown/wbrown.add-compressed-wiki

  Show description for f1784b0
  wbrown

  authored
  f1784b0

  Copy full SHA for f1784b0

• Fix `dataset_tokenizer` concurrency and finalize `StatusWatcher` implementation.

  wbrown

  committed
  309d6ef

  Copy full SHA for 309d6ef

Commits on Oct 18, 2024

• More go.mod fixes.

  wbrown

  committed
  4962aaa

  Copy full SHA for 4962aaa

• Revert more version updates.

  wbrown

  committed
  e42df30

  Copy full SHA for e42df30

• Revert go update

  wbrown

  committed
  0a023e1

  Copy full SHA for 0a023e1

• Update gpt_bpe dep

  wbrown

  committed
  2b6d480

  Copy full SHA for 2b6d480

• Update accumulator, get average number of tokens per doc

  wbrown

  committed
  c40bbba

  Copy full SHA for c40bbba

• Fix time reporting and change some params.

  wbrown

  committed
  9873176

  Copy full SHA for 9873176

• Handle EOF on input.

  wbrown

  committed
  4c53561

  Copy full SHA for 4c53561

• Change reporting mechanisms.

  wbrown

  committed
  005206f

  Copy full SHA for 005206f

• Fix streaming and handle `.md` files.

  wbrown

  committed
  fba60a1

  Copy full SHA for fba60a1

Commits on Oct 16, 2024

• Fix use of units.

  wbrown

  committed
  c7ac6aa

  Copy full SHA for c7ac6aa

• Use compressed `wiki.train.raw.xz`.

  wbrown

  committed
  9d5c6dc

  Copy full SHA for 9d5c6dc

• Comment out `replace` directive.

  wbrown

  committed
  0e8aee0

  Copy full SHA for 0e8aee0

• Some build fixes.

  wbrown

  committed
  fd89197

  Copy full SHA for fd89197

Pagination

Previous

Next