test-backend-ops: add support for specifying output format #14368

yeahdongcn · 2025-06-25T03:36:31Z

Make sure to read the contributing guidelines before submitting a PR

Testing Done

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops --help
Usage: ./build/bin/test-backend-ops [mode] [-o <op>] [-b <backend>] [-p <params regex>] [--output <console|sql>]
    valid modes:
      - test (default, compare with CPU backend for correctness)
      - grad (compare gradients from backpropagation with method of finite differences)
      - perf (performance evaluation)
    op names for -o are as given by ggml_op_desc() (e.g. ADD, MUL_MAT, etc)
    --output specifies output format (default: console)

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops perf -b CPU -o ADD
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 MUSA devices:
  Device 0: MTT S80, compute capability 2.1, VMM: yes
Testing 2 devices

Backend 1/2: MUSA0
  Skipping
Backend 2/2: CPU
  Device description: 12th Gen Intel(R) Core(TM) i5-12400
  Device memory: 31859 MB (31859 MB free)

  ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]):   ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]):                1572480 runs -     0.64 us/run -       48 kB/run -   71.97 GB/s
  ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1]):   ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1]):                13338 runs -    76.12 us/run -    24576 kB/run -  308.22 GB/s
  Backend CPU: OK

2/2 backends passed
OK

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops perf -b CPU -o ADD --output sql
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 MUSA devices:
  Device 0: MTT S80, compute capability 2.1, VMM: yes
CREATE TABLE IF NOT EXISTS test_backend_ops (
  test_time TEXT,
  build_commit TEXT,
  build_number INTEGER,
  backend_name TEXT,
  op_name TEXT,
  op_params TEXT,
  test_mode TEXT,
  supported INTEGER,
  passed INTEGER,
  error_message TEXT,
  time_us REAL,
  flops REAL,
  bandwidth_gb_s REAL,
  memory_kb INTEGER,
  n_runs INTEGER
);

INSERT INTO test_backend_ops (test_time, build_commit, build_number, backend_name, op_name, op_params, test_mode, supported, passed, error_message, time_us, flops, bandwidth_gb_s, memory_kb, n_runs) VALUES ('2025-06-27T05:54:54Z', '1d5f25c53', '5756', 'CPU', 'ADD', 'type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]', 'perf', '1', '1', '', '1.003772', '0.000000', '45.608051', '48', '1007370');
INSERT INTO test_backend_ops (test_time, build_commit, build_number, backend_name, op_name, op_params, test_mode, supported, passed, error_message, time_us, flops, bandwidth_gb_s, memory_kb, n_runs) VALUES ('2025-06-27T05:54:55Z', '1d5f25c53', '5756', 'CPU', 'ADD', 'type=f32,ne=[4096,1,1,1],nr=[1,512,1,1]', 'perf', '1', '1', '', '134.092504', '0.000000', '174.956746', '24576', '7524');

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops perf -b CPU -o ADD --output sql | sqlite3 add.sqlite
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 MUSA devices:
  Device 0: MTT S80, compute capability 2.1, VMM: yes

Edit (added build_commit ~~and build_number~~ for future use):

Copilot

Pull Request Overview

This PR adds support for specifying the output format (console or SQL) for the test-backend-ops tool and unifies logging via a new printer interface.

Introduces a printer interface with concrete implementations for console and SQL output.
Updates test evaluation functions and the CLI to use the printer for logging.
Adds new command-line option "--output" along with helper functions to parse the output format.

tests/test-backend-ops.cpp

JohannesGaessler

From my end I think these changes to test-backend-ops would be fine but keep in mind that it's an important piece of the project with many stakeholders.

tests/test-backend-ops.cpp

JohannesGaessler · 2025-06-25T12:47:47Z

tests/test-backend-ops.cpp

+        passed = false;
+
+        // Set test time
+        time_t t = time(NULL);


This was added to record the timestamp (the same as llama-bench), but I haven’t decided how to use it.

tests/test-backend-ops.cpp

yeahdongcn · 2025-06-26T04:42:19Z

From my end I think these changes to test-backend-ops would be fine but keep in mind that it's an important piece of the project with many stakeholders.

Thanks for pointing that out! I’ll add the recent contributors as reviewers.

tests/test-backend-ops.cpp

Signed-off-by: Xiaodong Ye <[email protected]>

tests/test-backend-ops.cpp

Signed-off-by: Xiaodong Ye <[email protected]>

am17an

LGTM but you can wait for @slaren for this input

tests/test-backend-ops.cpp

Signed-off-by: Xiaodong Ye <[email protected]>

slaren · 2025-07-03T10:55:53Z

@yeahdongcn Please allow maintaners to push to your branch, or merge the changes from slaren@5987a24

This is better and closer to what I meant, but I suspect that in practice it is going to be very hard to reuse these functions to create new printers because the abstraction is too fine-grained and at least would require keeping track of the current state in the printer, which is not great. Anyway, we can deal with that later.

yeahdongcn · 2025-07-03T12:51:46Z

@yeahdongcn Please allow maintaners to push to your branch, or merge the changes from slaren@5987a24

Done. Thanks for the updates!

This is better and closer to what I meant, but I suspect that in practice it is going to be very hard to reuse these functions to create new printers because the abstraction is too fine-grained and at least would require keeping track of the current state in the printer, which is not great. Anyway, we can deal with that later.

Got it — I understand your concern.

Signed-off-by: Xiaodong Ye <[email protected]>

slaren

There are some formatting issues, for instance there should an spa

tests/test-backend-ops.cpp

Signed-off-by: Xiaodong Ye <[email protected]>

github-actions bot added the testing Everything test related label Jun 25, 2025

yeahdongcn mentioned this pull request Jun 25, 2025

Add script to test op perf and compare #14354

Open

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from bfa7a43 to 359d792 Compare June 25, 2025 08:21

yeahdongcn requested a review from Copilot June 25, 2025 08:44

This comment was marked as outdated.

Sign in to view

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 359d792 to 34500f9 Compare June 25, 2025 09:16

yeahdongcn requested a review from Copilot June 25, 2025 09:19

Copilot AI reviewed Jun 25, 2025

View reviewed changes

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 34500f9 to 679a141 Compare June 25, 2025 09:46

yeahdongcn marked this pull request as ready for review June 25, 2025 09:47

JohannesGaessler reviewed Jun 25, 2025

View reviewed changes

yeahdongcn requested review from am17an, ggerganov and ngxson June 26, 2025 04:42

ggerganov requested a review from slaren June 26, 2025 08:06

yeahdongcn mentioned this pull request Jun 26, 2025

compare-commits.sh: support both llama-bench and test-backend-ops #14392

Open

slaren reviewed Jun 30, 2025

View reviewed changes

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

slaren mentioned this pull request Jul 1, 2025

ggml : add version function to get lib version ggml-org/ggml#1286

Merged

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 76ca4f6 to f4f5512 Compare July 2, 2025 02:10

yeahdongcn added 6 commits July 3, 2025 08:02

test-backend-ops: add support for specifying output format

b7bc322

Signed-off-by: Xiaodong Ye <[email protected]>

Address review comments

8ec8571

Signed-off-by: Xiaodong Ye <[email protected]>

Add build_commit and build_number in test_result

3cabda8

Signed-off-by: Xiaodong Ye <[email protected]>

Address review comments

2a5cd8e

Signed-off-by: Xiaodong Ye <[email protected]>

refactor

d3f68bd

Signed-off-by: Xiaodong Ye <[email protected]>

Get build commit from ggml_commit()

a4b376f

Signed-off-by: Xiaodong Ye <[email protected]>

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from f4f5512 to a4b376f Compare July 3, 2025 00:24

am17an reviewed Jul 3, 2025

View reviewed changes

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

yeahdongcn added 2 commits July 3, 2025 14:33

Merge errors into test_operation_info && address review comments

fb835fa

Signed-off-by: Xiaodong Ye <[email protected]>

Address review comments

bbca39f

Signed-off-by: Xiaodong Ye <[email protected]>

am17an approved these changes Jul 3, 2025

View reviewed changes

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

am17an reviewed Jul 3, 2025

View reviewed changes

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

Address review comments

a3252a0

Signed-off-by: Xiaodong Ye <[email protected]>

remove visitor nonsense

e8bc633

remove visitor comment

6db8a27

Signed-off-by: Xiaodong Ye <[email protected]>

slaren reviewed Jul 3, 2025

View reviewed changes

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

Address review comments

7359eed

Signed-off-by: Xiaodong Ye <[email protected]>

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 810ef25 to 7359eed Compare July 4, 2025 02:14

test-backend-ops: add support for specifying output format #14368

Are you sure you want to change the base?

test-backend-ops: add support for specifying output format #14368

Conversation

yeahdongcn commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing Done

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

JohannesGaessler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JohannesGaessler Jun 25, 2025

Choose a reason for hiding this comment

Uh oh!

yeahdongcn Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yeahdongcn commented Jun 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

am17an left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

slaren commented Jul 3, 2025

Uh oh!

yeahdongcn commented Jul 3, 2025

Uh oh!

slaren left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yeahdongcn commented Jun 25, 2025 •

edited

Loading

yeahdongcn Jun 26, 2025 •

edited

Loading