test-backend-ops: add support for specifying output format #14368

yeahdongcn · 2025-06-25T03:36:31Z

Make sure to read the contributing guidelines before submitting a PR

Testing Done

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops --help
Usage: ./build/bin/test-backend-ops [mode] [-o <op>] [-b <backend>] [-p <params regex>] [--output <console|sql>]
    valid modes:
      - test (default, compare with CPU backend for correctness)
      - grad (compare gradients from backpropagation with method of finite differences)
      - perf (performance evaluation)
    op names for -o are as given by ggml_op_desc() (e.g. ADD, MUL_MAT, etc)
    --output specifies output format (default: console)

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops perf -b CPU -o ADD
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 MUSA devices:
  Device 0: MTT S80, compute capability 2.1, VMM: yes
Testing 2 devices

Backend 1/2: MUSA0
  Skipping
Backend 2/2: CPU
  Device description: 12th Gen Intel(R) Core(TM) i5-12400
  Device memory: 31859 MB (31859 MB free)

  ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]):   ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]):                1572480 runs -     0.64 us/run -       48 kB/run -   71.97 GB/s
  ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1]):   ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1]):                13338 runs -    76.12 us/run -    24576 kB/run -  308.22 GB/s
  Backend CPU: OK

2/2 backends passed
OK

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops perf -b CPU -o ADD --output sql
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 MUSA devices:
  Device 0: MTT S80, compute capability 2.1, VMM: yes
CREATE TABLE IF NOT EXISTS test_backend_ops (
  test_time TEXT,
  build_commit TEXT,
  build_number INTEGER,
  backend_name TEXT,
  op_name TEXT,
  op_params TEXT,
  test_mode TEXT,
  supported INTEGER,
  passed INTEGER,
  error_message TEXT,
  time_us REAL,
  flops REAL,
  bandwidth_gb_s REAL,
  memory_kb INTEGER,
  n_runs INTEGER
);

INSERT INTO test_backend_ops (test_time, build_commit, build_number, backend_name, op_name, op_params, test_mode, supported, passed, error_message, time_us, flops, bandwidth_gb_s, memory_kb, n_runs) VALUES ('2025-06-27T05:54:54Z', '1d5f25c53', '5756', 'CPU', 'ADD', 'type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]', 'perf', '1', '1', '', '1.003772', '0.000000', '45.608051', '48', '1007370');
INSERT INTO test_backend_ops (test_time, build_commit, build_number, backend_name, op_name, op_params, test_mode, supported, passed, error_message, time_us, flops, bandwidth_gb_s, memory_kb, n_runs) VALUES ('2025-06-27T05:54:55Z', '1d5f25c53', '5756', 'CPU', 'ADD', 'type=f32,ne=[4096,1,1,1],nr=[1,512,1,1]', 'perf', '1', '1', '', '134.092504', '0.000000', '174.956746', '24576', '7524');

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops perf -b CPU -o ADD --output sql | sqlite3 add.sqlite
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 MUSA devices:
  Device 0: MTT S80, compute capability 2.1, VMM: yes

Edit (added build_commit and build_number for future use):

Copilot

Pull Request Overview

This PR adds support for specifying the output format (console or SQL) for the test-backend-ops tool and unifies logging via a new printer interface.

Introduces a printer interface with concrete implementations for console and SQL output.
Updates test evaluation functions and the CLI to use the printer for logging.
Adds new command-line option "--output" along with helper functions to parse the output format.

tests/test-backend-ops.cpp

JohannesGaessler

From my end I think these changes to test-backend-ops would be fine but keep in mind that it's an important piece of the project with many stakeholders.

tests/test-backend-ops.cpp

JohannesGaessler · 2025-06-25T12:47:47Z

tests/test-backend-ops.cpp

+        passed = false;
+
+        // Set test time
+        time_t t = time(NULL);


This was added to record the timestamp (the same as llama-bench), but I haven’t decided how to use it.

tests/test-backend-ops.cpp

yeahdongcn · 2025-06-26T04:42:19Z

From my end I think these changes to test-backend-ops would be fine but keep in mind that it's an important piece of the project with many stakeholders.

Thanks for pointing that out! I’ll add the recent contributors as reviewers.

tests/test-backend-ops.cpp

slaren · 2025-06-30T13:21:44Z

tests/test-backend-ops.cpp

+    // General purpose output methods
+    virtual void print_info(const char * format, ...) = 0;
+    virtual void print_error(const char * format, ...) = 0;
+    virtual void print_device_info(const char * format, ...) = 0;
+    virtual void print_test_summary(const char * format, ...) = 0;
+    virtual void print_status_ok() = 0;
+    virtual void print_status_fail() = 0;


I don't think this a good design, all of these functions do the same and could be replaced with a single print_message. If you want transfer the responsibility of formatting the output to a class, then the class needs to have the information to decide what to print, not an unstructured message. Ideally all of the functions would receive an object to print, and the class would determine how to format that object.

Sounds good. I’ll update the code accordingly.

Please check the latest commit. Thanks.

This doesn't really change anything. The ideal solution would be to remove all the messages entirely, and pass enough information to the printers so that they format the output in any way they want. That would require a significant refactor, and not simply replacing calls to printf with printer->print_message.

Please check if I understand this correctly: f4f5512
Thanks.

Signed-off-by: Xiaodong Ye <[email protected]>

am17an · 2025-07-03T03:48:59Z

tests/test-backend-ops.cpp

+        std::string type_str(type_name);
+
+        if (type_str.find("test_operation_info") != std::string::npos) {
+            handle_message(*static_cast<const test_operation_info*>(data));


This looks like a typical use-case for the visitor pattern, but idk if we should use it

Do you have any suggestions for improvement here?

Yes instead of the typeid->string dispatch, you can do a classic double dispatch, I mean indirectly you are following the same pattern

something like

struct msg { virtual void accept(message_visitor & v); = 0 struct message_visitor { virtual void visit(const test_operation_info&) = 0; virtual void visit(const device_info&) = 0; .... }; struct test_operation_info : public msg{ void accept(message_visitor& v) const override { v.visit(*this); } }; struct printer : message_visitor { void print_message(const printable_message& m) { m.accept(*this); } void visit(const test_operation_info&) override {} }; struct console_printer : public printer { /* override whatever you care about */ void visit(const test_operation_info& info) override { ... }; struct sql_printer: public printer { //same };

Then at the call site

std::unique_ptr<printer> p = create_printer(); test_operation_info msg; p->print_message(msg);

tests/test-backend-ops.cpp

Signed-off-by: Xiaodong Ye <[email protected]>

github-actions bot added the testing Everything test related label Jun 25, 2025

yeahdongcn mentioned this pull request Jun 25, 2025

Add script to test op perf and compare #14354

Open

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from bfa7a43 to 359d792 Compare June 25, 2025 08:21

yeahdongcn requested a review from Copilot June 25, 2025 08:44

This comment was marked as outdated.

Sign in to view

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 359d792 to 34500f9 Compare June 25, 2025 09:16

yeahdongcn requested a review from Copilot June 25, 2025 09:19

Copilot AI reviewed Jun 25, 2025

View reviewed changes

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 34500f9 to 679a141 Compare June 25, 2025 09:46

yeahdongcn marked this pull request as ready for review June 25, 2025 09:47

JohannesGaessler reviewed Jun 25, 2025

View reviewed changes

yeahdongcn requested review from am17an, ggerganov and ngxson June 26, 2025 04:42

ggerganov requested a review from slaren June 26, 2025 08:06

yeahdongcn mentioned this pull request Jun 26, 2025

compare-commits.sh: support both llama-bench and test-backend-ops #14392

Open

slaren reviewed Jun 30, 2025

View reviewed changes

slaren mentioned this pull request Jul 1, 2025

ggml : add version function to get lib version ggml-org/ggml#1286

Merged

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 76ca4f6 to f4f5512 Compare July 2, 2025 02:10

yeahdongcn added 6 commits July 3, 2025 08:02

test-backend-ops: add support for specifying output format

b7bc322

Signed-off-by: Xiaodong Ye <[email protected]>

Address review comments

8ec8571

Signed-off-by: Xiaodong Ye <[email protected]>

Add build_commit and build_number in test_result

3cabda8

Signed-off-by: Xiaodong Ye <[email protected]>

Address review comments

2a5cd8e

Signed-off-by: Xiaodong Ye <[email protected]>

refactor

d3f68bd

Signed-off-by: Xiaodong Ye <[email protected]>

Get build commit from ggml_commit()

a4b376f

Signed-off-by: Xiaodong Ye <[email protected]>

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from f4f5512 to a4b376f Compare July 3, 2025 00:24

am17an reviewed Jul 3, 2025

View reviewed changes

Merge errors into test_operation_info && address review comments

fb835fa

Signed-off-by: Xiaodong Ye <[email protected]>

test-backend-ops: add support for specifying output format #14368

Are you sure you want to change the base?

test-backend-ops: add support for specifying output format #14368

Conversation

yeahdongcn commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing Done

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

JohannesGaessler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yeahdongcn Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yeahdongcn commented Jun 26, 2025

Uh oh!

Uh oh!

slaren Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yeahdongcn commented Jun 25, 2025 •

edited

Loading

yeahdongcn Jun 26, 2025 •

edited

Loading

slaren Jun 30, 2025 •

edited

Loading