Commits

Branch selector

 main

User selector

All users

Datepicker

All time

Commit History

Commits on Jul 1, 2024

• [Statistics] Added generation stat analysis (execution times and number of generated tokens)

  reuank

  committed
  fbb54df

  Copy full SHA for fbb54df

Commits on Jun 28, 2024

• Adapt non CoT benchmark naming to CoT benchmarks

  reuank

  committed
  6129b86

  Copy full SHA for 6129b86

• Fix errors in non-CoT benchmarks

  reuank

  committed
  4b24470

  Copy full SHA for 4b24470

Commits on Jun 27, 2024

• [CLI] Show detailed testcase info only when necessary

  reuank

  committed
  565f835

  Copy full SHA for 565f835

Commits on Jun 23, 2024

• [CLI] Fix handling of multiple datasets/benchmarks

  reuank

  committed
  031c18a

  Copy full SHA for 031c18a

• Renamed ChoiceProb and RunsMatch

  reuank

  committed
  e5ccae7

  Copy full SHA for e5ccae7

• Remove unnecessary latex shortcodes option

  reuank

  committed
  f9c2f8d

  Copy full SHA for f9c2f8d

• Compare number of results when loading two runs

  reuank

  committed
  b7884b4

  Copy full SHA for b7884b4

• Added completion conversion test to debugging tools

  reuank

  committed
  8e9070b

  Copy full SHA for 8e9070b

• Refactor: renamed trace_analysis to evaluation, and added option for big plots

  reuank

  committed
  391a142

  Copy full SHA for 391a142

• [Stats] Added Label Probs Stat

  reuank

  committed
  06c3594

  Copy full SHA for 06c3594

• Changed non-CoT benchmark names

  reuank

  committed
  eb1eebd

  Copy full SHA for eb1eebd

Commits on Jun 21, 2024

• Handle broken server behavior (returning empty completions)

  reuank

  committed
  424bf15

  Copy full SHA for 424bf15

Commits on Jun 18, 2024

• Implemented CoT Nucleus Benchmark to test top_p

  reuank

  committed
  a5b6983

  Copy full SHA for a5b6983

Commits on Jun 11, 2024

• Added new CoT Variant to try to push label token probs

  reuank

  committed
  d14b602

  Copy full SHA for d14b602

• Improve wording of class part id retrieval function

  reuank

  committed
  4ba7658

  Copy full SHA for 4ba7658

• Fix registry groups

  reuank

  committed
  79f680f

  Copy full SHA for 79f680f

• Adding stack trace for exceptions in thinkbench

  reuank

  committed
  12f0400

  Copy full SHA for 12f0400

• Fix registry groups

  reuank

  committed
  2dbe0fd

  Copy full SHA for 2dbe0fd

• Added NonCoT Variant 1 to test if adding top tokens to the prompt boosts performance

  reuank

  committed
  760d756

  Copy full SHA for 760d756

• Implemented Top Token stat and added common RunStat class to all stats

  reuank

  committed
  66e015d

  Copy full SHA for 66e015d

• Implemented Choice Prob Stat

  reuank

  committed
  8f5c829

  Copy full SHA for 8f5c829

• Implemented label confusion matrix stat, moved matrix generation to new Plot class, renamed ResultLoader to TestCaseResultHelper and added matching functionality there

  reuank

  committed
  5cff205

  Copy full SHA for 5cff205

Commits on Jun 9, 2024

• Print class accuracies for all class ids if no class id is specified

  reuank

  committed
  8db0fca

  Copy full SHA for 8db0fca

• Save mismatch overview file to better debug mismatches between manual and automatic classifications

  reuank

  committed
  0487eaf

  Copy full SHA for 0487eaf

• Added debug output to automatic trace classifier, and edited some indicator arrays

  reuank

  committed
  388b94f

  Copy full SHA for 388b94f

Commits on Jun 8, 2024

• Refactored AutomaticTraceClassifier to be more modular

  reuank

  committed
  d838a9d

  Copy full SHA for d838a9d

• Renamed classify_traces

  reuank

  committed
  47146c5

  Copy full SHA for 47146c5

• Big refactor: Use typed dicts for classification results and loaded test case results, and use consistent variable names

  reuank

  committed
  db9984b

  Copy full SHA for db9984b

Commits on Jun 7, 2024

• Renamed and refactored the trace classifiers

  reuank

  committed
  8b46dae

  Copy full SHA for 8b46dae

• Corrected trace classes after weekly thesis presentation: classify whether label extraction worked instead of whether (non)cot runs match

  reuank

  committed
  a8377b8

  Copy full SHA for a8377b8

• Consistent writing: class instead of category, file_name instead of filename, " instead of ' (when possible)

  reuank

  committed
  7a0d07f

  Copy full SHA for 7a0d07f

• [CLI] [Trace Analysis] Added stats generation and file exists handling

  reuank

  committed
  955c8fe

  Copy full SHA for 955c8fe

Commits on Jun 4, 2024

• [Trace Analysis] Implemented confusion matrices for manual/automatic label comparison

  reuank

  committed
  5c07856

  Copy full SHA for 5c07856

• Implemented env loader, and replaced prints with logger info calls

  reuank

  committed
  f28c43a

  Copy full SHA for f28c43a

Pagination

Previous

Next