There are several places where users may want to specify the type of
object IDs (sha1 or sha256) that should be used, for example, when
dealing with repositories, indexes, etc.
However, given that sha256 support remains disappointingly uncommon in
the wild, we should avoid hard API breaks when possible. Instead, update
these APIs to have an "extended" format (eg, `git_odb_open_ext`) that
provides an options structure with oid type information.
This allows callers who do care about sha256 to use it, and callers who
do not to avoid gratuitous API breakage.
libgit2 has two distinct requirements that were previously solved by
`git_buf`. We require:
1. A general purpose string class that provides a number of utility APIs
for manipulating data (eg, concatenating, truncating, etc).
2. A structure that we can use to return strings to callers that they
can take ownership of.
By using a single class (`git_buf`) for both of these purposes, we have
confused the API to the point that refactorings are difficult and
reasoning about correctness is also difficult.
Move the utility class `git_buf` to be called `git_str`: this represents
its general purpose, as an internal string buffer class. The name also
is an homage to Junio Hamano ("gitstr").
The public API remains `git_buf`, and has a much smaller footprint. It
is generally only used as an "out" param with strict requirements that
follow the documentation. (Exceptions exist for some legacy APIs to
avoid breaking callers unnecessarily.)
Utility functions exist to convert a user-specified `git_buf` to a
`git_str` so that we can call internal functions, then converting it
back again.
There's multiple headers included in our fuzzers that aren't required at
all. Furthermore, some of them are not available on Win32, causing
builds to fail. Remove them to fix this.
The `snprintf` function does not exist on Win32, it only has
`_snprintf_s` available. Let's just avoid any cross-platform hassle and
use our own `git_buf` functionality instead.
Use `p_unlink` instead of `unlink` to remove the generated packfiles in
our packfile fuzzer. Like this, we do not have to worry about using
proper includes that are known on all platforms, especially Win32.
By default, libgit2 allows up to 2^32 objects when downloading a
packfile from a remote. For each of these objects, libgit2 will allocate
up to two small structs, which in total adds up to quite a lot of
memory. As a result, our fuzzers might run out of memory rather quick in
case where they receive as input a packfile with such a huge count of
objects.
Limit the packfile object count to 10M objects. This is sufficiently big
to still work with most largish repos (linux.git has around 6M objects
as of now), but small enough to not cause the fuzzer to OOM.
The packfile_raw fuzzer is using some internal APIs from libgit2, which
makes it hard to compile it as part of the oss-fuzz project. As oss-fuzz
requires us to link against the C++ FuzzingEngine library, we cannot use
"-DBUILD_FUZZERS=ON" directly but instead have to first compile an
object from our fuzzers and then link against the C++ library. Compiling
the fuzzer objects thus requires an external invocation of CC, and we
certainly don't want to do further black magic by adding libgit2's
private source directory to the header include path.
To fix the issue, convert the code to not use any internal APIs. Besides
some headers which we have to add now, this also requires us to change
to the hashing function of the ODB. Note that this will change the
hashing result, as we have previously not prepended the object header to
the data that is to be hashed. But this shouldn't matter in practice, as
we don't care for the hash value anyway.
Our layout uses names like "examples" or "tests" which is why the "fuzz"
directory doesn't really fit in here. Rename the directory to be called
"fuzzers" instead. Furthermore, we rename the fuzzer "fuzz_packfile_raw"
to "packfile_raw_fuzzer", which is also in line with the already
existing fuzzer at google/oss-fuzz.
While at it, rename the "packfile_raw" fuzzer to instead just be called
"packfile" fuzzer.
