[cryptofuzz] Add OpenSSL 1.0.2 and 1.1.0 targets

[guidovranken]

There are a bunch of bugs in 1.0.2 and 1.10 that will cause crashes, like openssl/openssl#8980 and openssl/openssl#8972

I still have to create bug reports for the other ones.

You can either merge it now and let ClusterFuzz detect these bugs, or wait until they are all fixed, whichever you prefer.

CC @mattcaswell @kroeckx

I will remove these targets when they reach end-of-life later this year.

[jonathanmetzman]

I'll merge this once the travis failures are fixed.

[guidovranken]

Part of the EverCrypt build will not succeed, this is expected, hence we compile with make -j$(nproc) || true:

oss-fuzz/projects/cryptofuzz/build.sh

Line 108 in 5cfae06

make -j$(nproc) || true

Apparently Travis trips over this.. @s-zanella can the EverCrypt build script be changed to succeed entirely with sanitizers enabled?

[s-zanella]

@guidovranken I pushed a change to the PR I've opened in your fork that addresses this.

You can save a lot of time (and space in logs) by only compiling libcrypto.a in OpenSSL builds. Unfortunately, there is no recipe that works across all versions, but I believe make build_generated libcrypto.a works for OpenSSL 1.1.x and make build_crypto works for OpenSSL 1.0.2 (see openssl/openssl#4597).

[s-zanella]

This saves time by building only OpenSSL libcrypto.a

[s-zanella]

Suggested change

	make -j$(nproc)
	make -j$(nproc) build_generated libcrypto.a

[guidovranken]

When I apply this I get errors like

include/openssl/bio.h:667:27: warning: declaration of 'struct hostent' will not be visible outside of this function [-Wvisibility]
DEPRECATEDIN_1_1_0(struct hostent *BIO_gethostbyname(const char *name))
                          ^
include/openssl/bio.h:668:1: error: expected function body after function declarator
DEPRECATEDIN_1_1_0(int BIO_get_port(const char *str, unsigned short *port_ptr))
^
include/openssl/bio.h:680:55: warning: declaration of 'union BIO_sock_info_u' will not be visible outside of this function [-Wvisibility]
                  enum BIO_sock_info_type type, union BIO_sock_info_u *info);
                                                      ^
In file included from crypto/aes/aes_wrap.c:10:
In file included from crypto/include/internal/cryptlib.h:26:
include/openssl/bio.h:667:27: warning: declaration of 'struct hostent' will not be visible outside of this function [-Wvisibility]
DEPRECATEDIN_1_1_0(struct hostent *BIO_gethostbyname(const char *name))
                          ^
include/openssl/bio.h:668:1: error: expected function body after function declarator
DEPRECATEDIN_1_1_0(int BIO_get_port(const char *str, unsigned short *port_ptr))
^
include/openssl/bio.h:680:55: warning: declaration of 'union BIO_sock_info_u' will not be visible outside of this function [-Wvisibility]
                  enum BIO_sock_info_type type, union BIO_sock_info_u *info);
                                                      ^
In file included from crypto/aes/aes_ige.c:10:
In file included from crypto/include/internal/cryptlib.h:27:
include/openssl/err.h:249:1: error: expected function body after function declarator
DEPRECATEDIN_1_0_0(void ERR_remove_state(unsigned long pid))
^
In file included from crypto/aes/aes_wrap.c:10:
In file included from crypto/include/internal/cryptlib.h:27:
include/openssl/err.h:249:1: error: expected function body after function declarator
DEPRECATEDIN_1_0_0(void ERR_remove_state(unsigned long pid))
^
crypto/aes/aes_ige.c:47:5: error: use of undeclared identifier 'OPENSSL_FILE'; did you mean 'OPENSSL_die'?
    OPENSSL_assert(in && out && key && ivec);
    ^
include/openssl/crypto.h:332:60: note: expanded from macro 'OPENSSL_assert'
    (void)((e) ? 0 : (OPENSSL_die("assertion failed: " #e, OPENSSL_FILE, OPENSSL_LINE), 1))

[s-zanella]

Suggested change

	make -j$(nproc)
	make -j$(nproc) build_generated libcrypto.a

[s-zanella]

Suggested change

	make -j$(nproc)
	make -j$(nproc) build_crypto

[s-zanella]

Suggested change

	make -j$(nproc)
	make -j$(nproc) build_crypto

[guidovranken]

Thanks for the fixes @s-zanella . I tried your build speedups but I got errors so I'm leaving them out for now, will try again later.

@jonathanmetzman Travis now fails because "The job exceeded the maximum log length, and has been terminated.". Can't do anything about that now unless you want me to redirect output to /dev/null.

[s-zanella]

npro -> nproc

[s-zanella]

Thanks for the fixes @s-zanella . I tried your build speedups but I got errors so I'm leaving them out for now, will try again later.

Strange. What errors do you get?
I built and checked the builds of all fuzzers using the infra/helper.py script with these changes and didn't get any errors:
s-zanella@e257632

Overall build logs are much shorter. For instance, the log for --engine libfuzzer --sanitizer memory goes from 22.9k down to 14.3k lines. You can also silence the output of unzip with unzip -q to save some more space (~5k lines).

[jonathanmetzman]

@jonathanmetzman Travis now fails because "The job exceeded the maximum log length, and has been terminated.". Can't do anything about that now unless you want me to redirect output to /dev/null.

I'm not sure how to solve this in a general way. Maybe we should add -Wno-warnings to CFLAGS and CXXFLAGS on travis? You can try this yourself if you'd like, or adding -q to the unzip and apt-get commands (as well as removing the v argument to tar) should save us some unnecessary output.

If you don't want to go this trouble I can merge and see if it fails to build the old-fashioned way.

[jonathanmetzman]

btw, I checked that silencing these particular commands cause appropriate error messages to be printed on error. But if you feel like the verbose output provides value, feel to not silence them and I will merge this patch even though it fails travis (some projects will take too long on travis, so I don't expect every patch to pass on travis).

[guidovranken]

@jonathanmetzman all Travis builds pass now.

[jonathanmetzman]

LGTM