Got 12 improvements

[szmarczak]

This is a redo of #1521

Fixes #1668 (tested)
Fixes #1663 (tested)
Fixes #1538 (tested)

If this gets merged, we need to open an issue to fix the docs. I'll do them, as well as migration notes etc.

**Note**: By using the `Options` class we introduce a regression! In some cases where option normalization throws `error.options` is `undefined`. ~~This could be solved via **not throwing in the `Options` constructor**, but setting a property e.g. `lastKnownError` instead. That way `error.options` would be defined. It's just a matter what we do consider more friendly.~~ Ok found a better way - let's define `error.options` on the error thrown by the `Options` constructor. Later!

• Improved option normalization (new class: Options)
  • new Options(url | options, options?, defaults?)
    Note: Only the first url instance is used.
    Meaning: got('https://a', {url: 'https://b'}) will actually make a request to a.
    If you expect to replace the url option, then you need pass the initial url like this:
    got({url: 'https://a', ...otherOptions})
  • options.merge(newOptions) replaced got.mergeOptions & Request.normalizeArguments
    Note: It does NOT merge the url option - but it does merge the searchParameters option
  • Throws when an option does not exist
  • Dropped legacy URL support, use options.url instead (a URL instance)
  • Creates a brand new object for the native http options
  • No implicit timeout declaration, use {timeout: {request: 1000}} instead
  • No implicit retry declaration, use {retry: {limit: 3}} instead
  • dnsLookupIpVersion is now a number (4 or 6) or undefined
• Switched to get-stream (getting a buffer from a stream)
• Refactored core/index.ts
• Refactored as-promise/index.ts
  • Moved the onCancel handlers out of makeRequest in order to prevent them being called multiple times (.destroy() needs to be called only once)
  • The Request class now provides rawBody if request._noPipe is true (used internally by Promise API)
  • Switched from for/entries to classic for loop when iterating over afterResponse hooks in order to improve performance
    Why: .entries() sends a query to get enumerable properties + generates an array - but we don't need that since we already know the enumerable properties since it's an array and we don't need to generate a new one
  • Fixed Request options not being reused on retry (needs a test tested)
• Moved Response-related stuff to core/response.ts
• Moved error definitions to core/errors.ts - you now have to import them directly, meaning you cannot access them via got.RequestError
• New error type: RetryError which always triggers a new retry when thrown
• Moved waitForOpenFile to core/utils/wait-for-open-file.ts
• Added missing once types for Stream API
• Switched from symbols to privates
• redirectUrls and requestUrl now give URL instances
• The retry timeout in request._beforeError is now promisified
• Renamed request.aborted to request.isAborted
• Renamed the lookup option to dnsLookup
• The beforeRetry hook now accepts only one argument: error
  To retrieve:
  - the underlying request: error.request
  - the underlying options: error.options
  - current attempt number: error.request.retryCount + 1
• The beforeRedirect hook's first argument (options) is now a cloned instance of the Request options.
  This was done to make retrieving the original options possible. To retrieve the original options, use plainResponse.request.options.
• The redirect event now takes two arguments: updatedOptions and plainResponse
• The retry event now takes only one argument: error (reverted; reason: the current API is much nicer)
  To get the attempt count, use error.request.retryCount + 1
• The socketPath option has been removed. Use the unix: protocol instead.
• The retryWithMergedOptions function no longer returns a Promise. Keep in mind that this should be the last function being executed in an afterResponse hook.
• error.options is now enumerable
• defaults.handlers don't need a default handler ((options, next) => next(options)) now.
• Renamed options.https to options.httpsOptions
• Removed internal _isAboutToError function
• Renamed the searchParams option to searchParameters (reverted)
• You can no longer set options.agent to false. To do so, you need to define all the options.agent properties: http, https and http2.
• Refactored Got interface (instances, defaults, etc.)
• The init hook now accepts a second argument: self, which points to an Options instance
• The init hooks are ran only when passing an options object explicitly. It's no longer called on got('https://example.com') - in order to trigger init hooks, you need to call got('https://example.com', {})
• When passing a url option when paginating, it now needs to be an absolute URL - the prefixUrl option is always reset from now on. The same when retrying in an afterResponse hook.
• Fixed tests
  • Fix test/arguments.ts
  • Fix test/create.ts
  • Fix test/normalize-arguments.ts
  • Fix test/hooks.ts
• got.extend(...) will throw when passing some options that don't accept undefined - undefined no longer retains the old value, as setting undefined explicitly may reset the option
• In order to import Got in CommonJS now you have to do const {got} = require('got')
• Created a new documentation

To be fixed

• If agent fails, try again without #1654
• When transfer encoding is chunked, GOT sometimes does not download full content #1591 & Return error when content-length header does not match bytes transferred #1462
• default searchParams not properly merged when requesting searchParams is undefined #1610 (might be already fixed - needs a test)
• The types aren't compatible with CommonJS #1137
• Upper limit on the retry backoff #1599
• got.paginate does not call init hooks #1574 - needs a test
• statusCode for responses labeled undefined when promise has a statusCode of 200  #1569
• Add request.reusedSocket #1553
• "Parameters path and pathname are mutually exclusive" on retry with cache #1567 (might be already fixed - needs a test)
• Malformed response causing odd failure with no error #1527
• Ignore Got defaults when extending instances #1450 - needs a test
• Stream created from readable-stream module does not work #1503
• A body with a Readable can cause uncaughtException #1507
• Timeout not thrown with wrong https url, port, http2 enabled #1492
• beforeRequest doesn't merge options.json #1408
• How to correctly type an instance returned by got.extend #1117 (maybe, may be postponed to Got 12.x / 13)
• Document stream.options #1405 (docs)

Checklist

• I have read the documentation.
• I have included a pull request description of my changes.
• I have included some tests.
• If it's a new feature, I have included documentation updates in both the README and the types.

[szmarczak]

So I've benchmarked option normalization... and it's great <3 130k/s when normalizing URL. But when just cloning the options... 1M/s. That means we're bound by native JavaScript performance. Blazing fast.

Update: So we're down to 400-500k/s. But that's still better than the current 90-100k.

[szmarczak]

Oh, and one major thing: this PR uses TS default export.

[sindresorhus]

Fixes #1668 (needs a test)
Fixes #1663 (needs a test)
Fixes #1538 (needs a test)

Did tests get added for these?

[szmarczak]

Not yet, I can get them added in an hour. Currently I'm finishing my PR in ava/typescript.

[szmarczak]

Added the tests :)

[sindresorhus]

Created a new documentation

Can you open an issue for this?

[sindresorhus]

This is looking really good! Super nice work as always. 🙌

[sindresorhus]

Renamed the searchParams option to searchParameters

I don't think we should do this though. Can you revert that? While I generally prefer full words, I think it's more important here to use the same name as Fetch, and also, it's not worth doing a breaking change just for that.

[szmarczak]

Sure, will do this in a few hours.

[szmarczak]

7b3cbfd