UBSAN error while parsing BJData (Null-dereference)

[nlohmann]

Description

OSS-Fuzz reports a runtime error: member access within null pointer of type 'std::__tree_end_node<std::__tree_node_base<void *> *>' when fuzzing with UBSAN.

• https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=47406
• https://oss-fuzz.com/testcase-detail/6432243825901568

Reproduction steps

• Compile with UBSAN
• Parse input
  clusterfuzz-testcase-minimized-parse_bjdata_fuzzer-6432243825901568.zip

Expected vs. actual results

Expected: parse error or valid JSON value returned

Actual: UBSAN runtime error

Minimal code example

No response

Error messages

[Environment] UBSAN_OPTIONS=print_stacktrace=1:silence_unsigned_overflow=1
+----------------------------------------Release Build Stacktrace----------------------------------------+
Command: /mnt/scratch0/clusterfuzz/resources/platform/linux/unshare -c -n /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds_json_3dc7f07cae4ab217c21b70d40f93a3acccc6b431/revisions/parse_bjdata_fuzzer -rss_limit_mb=2560 -timeout=60 -runs=100 /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/6cd12eba683055cf14980d433e5156c264b292204f76005b9c062f0af87a5aa0
Time ran: 0.025225162506103516
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 529278620
INFO: Loaded 1 modules   (5891 inline 8-bit counters): 5891 [0x5944d0, 0x595bd3),
INFO: Loaded 1 PC tables (5891 PCs): 5891 [0x530a68,0x547a98),
/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds_json_3dc7f07cae4ab217c21b70d40f93a3acccc6b431/revisions/parse_bjdata_fuzzer: Running 1 inputs 100 time(s) each.
Running: /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/6cd12eba683055cf14980d433e5156c264b292204f76005b9c062f0af87a5aa0
/usr/local/bin/../include/c++/v1/__tree:83:35: runtime error: member access within null pointer of type 'std::__tree_end_node<std::__tree_node_base<void *> *>'
    #0 0x4bddea in __tree_is_left_child<std::__1::__tree_node_base<void *> *> /usr/local/include/c++/v1/__tree:83:35
    #1 0x4bddea in void std::__1::__tree_balance_after_insert<std::__1::__tree_node_base<void*>*>(std::__1::__tree_node_base<void*>*, std::__1::__tree_node_base<void*>*) /usr/local/include/c++/v1/__tree:286:13
    #2 0x4bd672 in std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > > > >::__insert_node_at(std::__1::__tree_end_node<std::__1::__tree_node_base<void*>*>*, std::__1::__tree_node_base<void*>*&, std::__1::__tree_node_base<void*>*) /usr/local/include/c++/v1/__tree:2083:5
    #3 0x4bd3ba in std::__1::pair<std::__1::__tree_iterator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >, std::__1::__tree_node<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >, void*>*, long>, bool> std::__1::__tree<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >, std::__1::__map_value_compare<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, true>, std::__1::allocator<std::__1::__value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > > > >::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::piecewise_construct_t const&, std::__1::tuple<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, std::__1::tuple<> >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::piecewise_construct_t const&, std::__1::tuple<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&&, std::__1::tuple<>&&) /usr/local/include/c++/v1/__tree:2099:9
    #4 0x4bd2f5 in std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > > > >::operator[](std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /usr/local/include/c++/v1/map:1546:20
    #5 0x4bcb15 in nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >::key(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&) json/single_include/nlohmann/json.hpp:6234:62
    #6 0x4c9e13 in nlohmann::detail::binary_reader<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > >, nlohmann::detail::iterator_input_adapter<std::__1::__wrap_iter<unsigned char const*> >, nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > > >::get_ubjson_object() json/single_include/nlohmann/json.hpp:11062:21
    #7 0x4c7095 in nlohmann::detail::binary_reader<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > >, nlohmann::detail::iterator_input_adapter<std::__1::__wrap_iter<unsigned char const*> >, nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > > >::get_ubjson_value(int) json/single_include/nlohmann/json.hpp:10878:24
    #8 0x4b5a95 in parse_ubjson_internal json/single_include/nlohmann/json.hpp:10299:16
    #9 0x4b5a95 in nlohmann::detail::binary_reader<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > >, nlohmann::detail::iterator_input_adapter<std::__1::__wrap_iter<unsigned char const*> >, nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > > >::sax_parse(nlohmann::detail::input_format_t, nlohmann::detail::json_sax_dom_parser<nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >*, bool, nlohmann::detail::cbor_tag_handler_t) json/single_include/nlohmann/json.hpp:8597:26
    #10 0x4b3b51 in nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long, unsigned long, double, std::__1::allocator, nlohmann::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > >::from_bjdata<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >&>(std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >&, bool, bool) json/single_include/nlohmann/json.hpp:22496:93
    #11 0x4b34bc in LLVMFuzzerTestOneInput json/tests/src/fuzzer-parse_bjdata.cpp:40:19
    #12 0x43d5b3 in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
    #13 0x429242 in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned long) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
    #14 0x42ea8c in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
    #15 0x457682 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
    #16 0x7f2e00b290b2 in __libc_start_main /build/glibc-eX1tMB/glibc-2.31/csu/libc-start.c:308:16
    #17 0x407a6d in _start
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/local/bin/../include/c++/v1/__tree:83:35 in

Compiler and operating system

OSS-Fuzz

Library version

develop

Validation

• The bug also occurs if the latest version from the develop branch is used.
• I can successfully compile and run the unit tests.

[nlohmann]

FYI @fangq

[fangq]

tested this input file using the develop HEAD, but it returned a parse error. is this the right way to reproduce?

fangq$  g++ -g t3491.cpp -o t3491 -I include

fangq$ ./t3491 clusterfuzz-testcase-minimized-parse_bjdata_fuzzer-6432243825901568
terminate called after throwing an instance of 'nlohmann::detail::parse_error'
  what():  [json.exception.parse_error.110] parse error at byte 18: syntax error while parsing BJData value: unexpected end of input
Aborted (core dumped)

t3491.cpp

#include <nlohmann/json.hpp>
#include <iostream>
#include <fstream>

using json = nlohmann::json;

int main(int argc, char *argv[]) {
    if(argc<2)
       return 1;
    std::ifstream stream(argv[1], std::ios::binary);
    std::vector<uint8_t> contents((std::istreambuf_iterator<char>(stream)), std::istreambuf_iterator<char>());
    json j = json::from_bjdata(contents);
    std::cout << j << std::endl;
    return 0;
}

[falbrechtskirchinger]

Try building with -fsanitize=undefined.

Edit: And if nothing changes, try running the program with this environment variable: UBSAN_OPTIONS=print_stacktrace=1:silence_unsigned_overflow=1

[fangq]

@falbrechtskirchinger, perhaps I did not set up the environment properly for this - my work desktop runs Ubuntu 18.04. I tried both g++ and clang++6.0 with the suggested flag and env variable, but still it prints the end-of-input parse_error.

[falbrechtskirchinger]

I've not tried to reproduce these issues myself. I'm working on an improved fuzz testing setup. Once that is completed in the coming days, I'll take a look.

[nlohmann]

I analyzed the issue, and just like #3490 (comment) and #3492 (comment), it is caused by creating invalid SAX events.

These events are triggered:

<object>
    <key key="" />
    <array>
        <object size="3">
            <key key="_ArraySize_" />
            <array size="2">
                <number_integer val="32" />
                <number_integer val="0" />
            </array>
            <array size="0"> <!-- first error: expected a key here -->
            </array>
        </array>             <!-- second error: parsing an object, but closing an array -->
        <key key="" />
        <parse_error id="18" token="<end of file>" />

The parser must make sure that

1. Key events must only occur in objects.
2. Each value event in an object must be preceded by a key event.
3. End events must match the previous open event.

These mismatched events yield undefined behavior in the SAX-DOM parser. In #3498 I propose adding assertions to make this undefined behavior visible. That PR does not fix this issue, but makes it clear that is unrelated to any sanitizer.