change bjdata ndarray flag to detect negative size, as part of #3475#3479
change bjdata ndarray flag to detect negative size, as part of #3475#3479nlohmann merged 9 commits intonlohmann:developfrom
Conversation
|
My apologies for derailing the conversation in #3475. You have fixed an issue raised in #3475 but not the issue that started #3475. @nlohmann and I started talking about fuzzing in general, I discovered more failures, and then I shared one of those, which is the one you have now fixed.
I suggest you change the title of this PR to something more generic and we address all assertion failures discovered by fuzzing in one go. |
falbrechtskirchinger
left a comment
There was a problem hiding this comment.
I'd use 0x0100 instead of 256. I find it to be slightly clearer but maybe that's just my embedded background. 🤷♂️
Also please use bit operations to test and clear bit 8.
// unfortunately we need a cast here which makes it more verbose :-(
is_ndarray = static_cast<bool>(size_and_type.second & 0x0100)
size_and_type.second &= ~0x0100;|
After these changes, the only crashes that remain seem to be stack overflows. I've built the fuzzer with ASan to confirm. $ find fuzzer*/crashes/ -iname 'id*' -exec bash -c 'echo {}; ../fuzzer <{} |& grep -E "^SUMMARY"; echo' \;
fuzzer0/crashes/id:000000,sig:11,src:000034,time:600227,execs:7245472,op:havoc,rep:16
SUMMARY: AddressSanitizer: stack-overflow /var/tmp/portage/sys-libs/compiler-rt-sanitizers-14.0.3/work/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:26:3 in __asan_memset
fuzzer1/crashes/id:000000,sig:11,src:000007,time:602569,execs:7906177,op:havoc,rep:16
SUMMARY: AddressSanitizer: stack-overflow /var/tmp/portage/sys-libs/compiler-rt-sanitizers-14.0.3/work/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:26:3 in __asan_memset
fuzzer2/crashes/id:000000,sig:11,src:000023,time:600389,execs:7884746,op:havoc,rep:4
SUMMARY: AddressSanitizer: stack-overflow /var/tmp/portage/sys-libs/compiler-rt-sanitizers-14.0.3/work/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:26:3 in __asan_memset
fuzzer3/crashes/id:000000,sig:11,src:000006,time:602138,execs:8012670,op:havoc,rep:4
SUMMARY: AddressSanitizer: stack-overflow /var/tmp/portage/sys-libs/compiler-rt-sanitizers-14.0.3/work/compiler-rt/lib/asan/../sanitizer_common/sanitizer_stacktrace.h:53:45 in StackTrace
fuzzer4/crashes/id:000000,sig:11,src:000011+000001,time:602442,execs:7641057,op:splice,rep:2
SUMMARY: AddressSanitizer: stack-overflow /var/tmp/portage/sys-libs/compiler-rt-sanitizers-14.0.3/work/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:26:3 in __asan_memset
fuzzer5/crashes/id:000000,sig:11,src:000006,time:601260,execs:10537713,op:havoc,rep:16
SUMMARY: AddressSanitizer: stack-overflow /var/tmp/portage/sys-libs/compiler-rt-sanitizers-14.0.3/work/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:26:3 in __asan_memset
fuzzer6/crashes/id:000000,sig:11,src:000034+000053,time:601491,execs:7847427,op:splice,rep:16
SUMMARY: AddressSanitizer: stack-overflow /var/tmp/portage/sys-libs/compiler-rt-sanitizers-14.0.3/work/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:26:3 in __asan_memset
fuzzer7/crashes/id:000000,sig:11,src:000003,time:601378,execs:7127960,op:havoc,rep:8
SUMMARY: AddressSanitizer: stack-overflow /var/tmp/portage/sys-libs/compiler-rt-sanitizers-14.0.3/work/compiler-rt/lib/asan/asan_interceptors_memintrinsics.cpp:26:3 in __asan_memset
fuzzer8/crashes/id:000000,sig:11,src:000007,time:600565,execs:7814233,op:havoc,rep:8
SUMMARY: AddressSanitizer: stack-overflow /home/flo/projects/json/issue3475/include/nlohmann/detail/input/binary_reader.hpp:2417 in nlohmann::detail::binary_reader<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> > >, nlohmann::detail::iterator_input_adapter<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >, nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> > > > >::get_ubjson_array()
fuzzer9/crashes/id:000000,sig:11,src:000034+000031,time:601075,execs:7913481,op:splice,rep:16
SUMMARY: AddressSanitizer: stack-overflow /home/flo/projects/json/issue3475/include/nlohmann/detail/input/binary_reader.hpp:2417 in nlohmann::detail::binary_reader<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> > >, nlohmann::detail::iterator_input_adapter<__gnu_cxx::__normal_iterator<unsigned char const*, std::vector<unsigned char, std::allocator<unsigned char> > > >, nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> > > > >::get_ubjson_array() |
| exception_message(input_format, "invalid byte: 0x" + last_token, "type"), nullptr)); | ||
| } | ||
|
|
||
| if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(bjdtype[size_and_type.second]) )) |
There was a problem hiding this comment.
Test coverage has decreased because this SAX abort is untested. Can you please construct a test case for this scenario?
| SaxCountdown scp(10); | ||
| CHECK(!json::sax_parse(v, &scp, json::input_format_t::bjdata)); | ||
| } | ||
|
|
There was a problem hiding this comment.
From my testing, the correct counts seem to be 6 and 7.
Please also replace all occurrences of CHECK(! with CHECK_FALSE(.
There was a problem hiding this comment.
@falbrechtskirchinger, thanks. is there a way to print the event count? I previously try to manually count but sometime my count does not match the expected.
There was a problem hiding this comment.
Not that I know off, I just step through it with the debugger.
There was a problem hiding this comment.
Did I give you the wrong line, or what's going on here?
Line 2456 is being missed now. I'm going to compare it against the previous coverage run.
There was a problem hiding this comment.
No, it was definitely line 2445 before.
🤷♂️
There was a problem hiding this comment.
@falbrechtskirchinger, thanks. is there a way to print the event count? I previously try to manually count but sometime my count does not match the expected.
Do you mean something like https://github.com/doctest/doctest/blob/master/doc/markdown/logging.md#info?
There was a problem hiding this comment.
I guess we could add CHECK(m_event_count == -1) to ~SaxEventCountdown(). But that wouldn't solve things completely. A more advanced solution could add an enum to indicate in which event we expect the counter to reach -1 and check if that happened in the destructor. I'll add that to my to-do list. There was another PR that moved SaxEventCountdown into a separate header. Probably a good idea as well.
falbrechtskirchinger
left a comment
There was a problem hiding this comment.
Looks good to me.
|
Thanks for fixing! |
here is the fix to #3475. I took @falbrechtskirchinger's suggestion, and use size_and_type.second's bit 8 (all BJData nd UBJSON markers are ASCII letters) as an ND array indicator, this allows negative sizes to be thrown as
json.exception.out_of_rangeerror.This should fix the detected fuzzer errors. Let me know if you still see anything.