Fixing ESS overvote discrepancy bug #2100
Conversation
|
Thanks for flagging @umbernhard and sorry for the extreme delay here! We'll pick this up and make sure it gets merged |
| J2,0001,BATCH1,1,0001013415,Round 1: 0.228946820159681463,AUDITED,"Choice 1-1, INVALID_WRITE_IN",Choice 1-1,,,"Choice 2-1, INVALID_WRITE_IN",Choice 2-1,,\r | ||
| J2,0001,BATCH1,3,0001013417,Round 1: 0.457121710197159606,AUDITED,Choice 1-1,Choice 1-1,,,Choice 2-1,Choice 2-1,,\r | ||
| J2,0001,BATCH2,3,0001000417,Round 1: 0.269793733438455805,AUDITED,"Choice 1-1, Choice 1-2",Overvote,Choice 1-1: -1; Choice 1-2: -1,,"Choice 2-1, Choice 2-3",Overvote,Choice 2-1: -1; Choice 2-3: -1,\r | ||
| J2,0001,BATCH2,3,0001000417,Round 1: 0.269793733438455805,AUDITED,"Choice 1-1, Choice 1-2",Overvote,Choice 1-2: -1,,"Choice 2-1, Choice 2-3",Overvote,,\r |
There was a problem hiding this comment.
I'm looking at this snapshot diff more closely and noticing something off.
The following
Audit result: Choice 1-1, Choice 1-2
CVR result: Overvote
is still resulting in a discrepancy of Choice 1-2: -1 (previously Choice 1-1: -1; Choice 1-2: -1).
In contrast, the following
Audit result: Choice 2-1, Choice 2-3
CVR result: Overvote
is expectedly resulting in no discrepancy (prevoiusly Choice 2-1: -1; Choice 2-3: -1).
I have a rough idea here what's going on and am prepping an edit to account for this. Because I can't push to this branch from a fork, I'll open a new PR.
There was a problem hiding this comment.
I believe the core issue is that, in this implementation, we only clear the discrepancy for a candidate if there's an "o" listed for that candidate in the reported object. But as far as I can tell, when a contest is overvoted, we may not have an "o" for every candidate. So rather than checking and clearing discrepancies on a per candidate basis, if we see any "o" listed, and the audit confirms an overvote, we should clear the discrepancy for all candidates.
Made that adjustment in #2161
There was a problem hiding this comment.
As far as I can tell, when a contest is overvoted, we do save an "o" for every candidate: https://github.com/votingworks/arlo/blob/main/server/api/cvrs.py#L885-L900
There was a problem hiding this comment.
Ah okay good to know! Sound like there's an issue with the test setup then
|
@umbernhard I know you flagged this related edge case in your PR description:
That comment plus our own investigation has gotten us thinking about overvote handling in ballot comparison audits, and we're chatting with Mark L to check our understanding. Based on that conversation, we're going to work on a more holistic fix, not just for ES&S overvotes but for overvotes in general. We're accordingly going to pause on this PR (as well as the PRs that @jonahkagan and I opened) and open something more holistic, definitely before the next big round of ballot comparison audits. Issue for tracking: #2163 |
|
That makes perfect sense to me! Sorry for not jumping into the conversation earlier, I was still trying to work out if the test you highlighted as problematic was related to this or its own thing. Happy to provide thoughts if it’s helpful, I’m genuinely unsure what the best way to proceed here is re: that edge case. |
|
No worries! We've got it from here and will just make sure to tag you on the eventual fix we land on |
Fixes #2099.
I'm pretty sure y'all can write a more elegant fix here, but this at least fixes the issues in my testing.
Another way to fix this might be to actually create the
uandonotation when the audit board marks an undervote/overvote. In some ways that's more elegant. There is a weird case here where non-ES&S CVR overvotes can still lead to discrepancies, i.e. if the reported votes are {A:1, B:1, C:0}, and the audited votes are {A:0, B:1, C:1}, I think Arlo will still output a discrepancy, even though both cases are overvotes.