Ensure that API and SQL queries return identical results #421

PietrH · 2025-10-02T08:52:58Z

TODO:

Because all these queries are live, both over the api and the database (they have to be, or I'd have to mock all database returns, and then the test is kind of mute), this puts a lot of load on the API.

~~Perhaps these tests should only run on a single runner? Or only every so often?~~

~~Maybe they should run regardless of changes in the package, on a schedule instead?~~

These tests will only ever run on a machine with local database access, and thus can only be tested on the RStudio server. It will always be skipped on CI.

Run the tests on the RStudio Server
Blocked by I've lost access to the new RStudio Server #423
Investigate API - SQL mismatches
Switch etnservice over to natural sorting via stringr
Add schema for arrow so column classes are consistent for get_acoustic_detections(): create via datapackage.json? Some booleans are returned as characters over the API (entirely empty)
API returns wrong class for moratorium column in several functions:

expect_protocol_agnostic(get_cpod_projects())

 Error: api (`actual`) not identical to sql (`expected`).

`actual$moratorium` is a character vector ('0', '1', '1', '1', '0', ...)
`expected$moratorium` is a logical vector (FALSE, TRUE, TRUE, TRUE, FALSE, ...)

Can i just cast all function output based on datapackage.json instead of trying to work with R -> Arrow Schema -> R ?

test failures

Vector order

Seems to be related to the order in which it is returned. Strange that this differs! Switch etnservice to stringi natural sorting to avoid OS differences?

Functions that call stringr::str_sort(data$serial_number, numeric = TRUE) in etnservice do pass. That must be it.

fixed in inbo/etnservice v0.5.1

Boleans as strings

Some fields are interpreted as character, while they are logical. This happens over the API only, it's happening in the OpenCPU layer of the protocol.



> # get_acoustic_detections -------------------------------------------------
> 
> # get_acoustic_detections() has internal logic, and should be tested in more
> # detail.
> test_that("get_acoustic_detections() returns identical results on limit", {
+   expect_protocol_agnostic(get_acoustic_detections(limit = TRUE))
+ })
── Failure: get_acoustic_detections() returns identical results on limit ───────────────────────
api (`actual`) not identical to sql (`expected`).

`actual$qc_flag` is a character vector (NA, NA, NA, NA, NA, ...)
`expected$qc_flag` is a logical vector (NA, NA, NA, NA, NA, ...)
Backtrace:
    ▆
 1. └─expect_protocol_agnostic(get_acoustic_detections(limit = TRUE))
 2.   └─testthat::expect_identical(...) at tests/testthat/helper-expectations.R:38:3

Error:
! Test failed
Run `rlang::last_trace()` to see where the error occurred.
> test_that("get_acoustic_detections() returns identical results on big query", {
+   expect_protocol_agnostic(
+     get_acoustic_detections(animal_project_code = "2011_Warnow")
+   )
+ })
── Failure: get_acoustic_detections() returns identical results on big query ───────────────────
api (`actual`) not identical to sql (`expected`).

`actual$qc_flag` is a character vector (NA, NA, NA, NA, NA, ...)
`expected$qc_flag` is a logical vector (NA, NA, NA, NA, NA, ...)
Backtrace:
    ▆
 1. └─expect_protocol_agnostic(get_acoustic_detections(animal_project_code = "2011_Warnow"))
 2.   └─testthat::expect_identical(...) at tests/testthat/helper-expectations.R:38:3

Error:
! Test failed
Run `rlang::last_trace()` to see where the error occurred.
> test_that("get_acoustic_detections() returns identical results for multiple stations", {
+   expect_protocol_agnostic(
+     get_acoustic_detections(
+       acoustic_tag_id = "A69-1601-16130",
+       station_name = c("de-9", "de-10")
+     )
+   )
+ })
── Failure: get_acoustic_detections() returns identical results for multiple stations ──────────
api (`actual`) not identical to sql (`expected`).

`actual$qc_flag` is a character vector (NA, NA, NA, NA, NA, ...)
`expected$qc_flag` is a logical vector (NA, NA, NA, NA, NA, ...)
Backtrace:
    ▆
 1. └─expect_protocol_agnostic(...)
 2.   └─testthat::expect_identical(...) at tests/testthat/helper-expectations.R:38:3

Error:
! Test failed
Run `rlang::last_trace()` to see where the error occurred.
> expect_protocol_agnostic(get_cpod_projects())

 Error: api (`actual`) not identical to sql (`expected`).

`actual$moratorium` is a character vector ('0', '1', '1', '1', '0', ...)
`expected$moratorium` is a logical vector (FALSE, TRUE, TRUE, TRUE, FALSE, ...)
 

> expect_protocol_agnostic(get_animal_projects())

 Error: api (`actual`) not identical to sql (`expected`).

`actual$moratorium` is a character vector ('0', '0', '0', '0', '0', ...)
`expected$moratorium` is a logical vector (FALSE, FALSE, FALSE, FALSE, FALSE, ...)
 

> expect_protocol_agnostic(get_acoustic_projects())

 Error: api (`actual`) not identical to sql (`expected`).

`actual$moratorium` is a character vector ('0', '0', '0', '0', '0', ...)
`expected$moratorium` is a logical vector (FALSE, FALSE, FALSE, FALSE, FALSE, ...)
 

> expect_protocol_agnostic(get_animals())
> expect_protocol_agnostic(get_tags())

 Error: api (`actual`) not identical to sql (`expected`).

`actual$floating` is a character vector (NA, NA, NA, NA, NA, ...)
`expected$floating` is a logical vector (NA, NA, NA, NA, NA, ...)

PietrH · 2025-10-02T09:24:36Z

tests fail on windows, no real output message. This might be present on v2.3.1, something to look into.

Copilot

Pull Request Overview

This PR introduces comprehensive protocol-agnostic testing to ensure that API and SQL database queries return identical results. The implementation adds a test framework that compares function outputs when using either the OpenCPU API or local database connections, helping maintain data consistency across different access methods.

Added a new test file with 15 test cases covering listing functions and acoustic detection queries
Created a reusable expectation function that mocks protocol selection to test both API and database paths
Tests are designed to run only on machines with local database access, skipping automatically on CI

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
tests/testthat/test-protocol_agnostic.R	Implements comprehensive test cases for protocol-agnostic behavior of listing functions and acoustic detection queries
tests/testthat/helper-expectations.R	Creates the `expect_protocol_agnostic()` helper function that mocks protocol selection to compare API vs database results

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tests/testthat/helper-expectations.R

tests/testthat/test-protocol_agnostic.R

Co-authored-by: Copilot <[email protected]>

….com/inbo/etn into custom-expectation-api_sql-identical

Copilot

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

codecov · 2025-10-02T12:37:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.68%. Comparing base (d0909c2) to head (23c7354).

Additional details and impacted files

@@            Coverage Diff             @@
##           v2.3.1     #421      +/-   ##
==========================================
+ Coverage   84.55%   84.68%   +0.13%     
==========================================
  Files          26       26              
  Lines         725      725              
==========================================
+ Hits          613      614       +1     
+ Misses        112      111       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

PietrH · 2025-10-07T12:15:58Z

etnservice seems to return the columns as a boolean, not as 0 or 1. So this is either an OpenCPU or a client bug.

~~It's remarkable that it only happens when the serializer is feather.~~ Happens on others too.

PietrH added 4 commits November 6, 2024 11:46

Add custom expectation to compare API to SQL for given expression

35c5734

document custom expectation

68af68a

Merge branch 'v2.3.1' into custom-expectation-api_sql-identical

0365915

switch to local mocking and select_protocol helper

6ce4379

PietrH added 7 commits October 2, 2025 13:09

Improve function docs.

000fcc0

Rename function to refer to protocol

b3d7cf4

Make skipping more robust, don't depend on odbc.

4c39ff5

Document expectation.

2b32945

Stub testfile to see if functions are protocol agnostic.

7ea5b06

test get_acoustic_detections() on protocol independent results

85480b1

Clarify why I don't test all functions. Add some more checks.

f3129d2

PietrH self-assigned this Oct 2, 2025

PietrH added the blocked label Oct 2, 2025

PietrH requested a review from Copilot October 2, 2025 11:55

Copilot AI reviewed Oct 2, 2025

View reviewed changes

tests/testthat/helper-expectations.R Outdated Show resolved Hide resolved

tests/testthat/test-protocol_agnostic.R Outdated Show resolved Hide resolved

PietrH linked an issue Oct 2, 2025 that may be closed by this pull request

Test if API and local SQL queries result in same objects #330

Open

3 tasks

PietrH and others added 4 commits October 2, 2025 13:56

Update examples

5abb3c6

Co-authored-by: Copilot <[email protected]>

Style

45e3a42

Merge branch 'custom-expectation-api_sql-identical' of https://github…

f15697b

….com/inbo/etn into custom-expectation-api_sql-identical

Add tests for calls that previously had mismatches (see issue)

8b04976

PietrH requested a review from Copilot October 2, 2025 12:04

PietrH marked this pull request as ready for review October 2, 2025 12:04

Copilot AI reviewed Oct 2, 2025

View reviewed changes

Merge branch 'v2.3.1' into custom-expectation-api_sql-identical

8c0f868

PietrH marked this pull request as draft October 2, 2025 14:48

[email protected] added 2 commits October 7, 2025 11:47

Style

48de89d

Clarify label

23c7354

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Ensure that API and SQL queries return identical results #421

Ensure that API and SQL queries return identical results #421

Uh oh!

PietrH commented Oct 2, 2025 •

edited

Loading

Uh oh!

PietrH commented Oct 2, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

codecov bot commented Oct 2, 2025 •

edited

Loading

Uh oh!

PietrH commented Oct 7, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Ensure that API and SQL queries return identical results #421

Are you sure you want to change the base?

Ensure that API and SQL queries return identical results #421

Uh oh!

Conversation

PietrH commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO:

test failures

Vector order

Boleans as strings

Uh oh!

PietrH commented Oct 2, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

codecov bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

PietrH commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

PietrH commented Oct 2, 2025 •

edited

Loading

codecov bot commented Oct 2, 2025 •

edited

Loading

PietrH commented Oct 7, 2025 •

edited

Loading