Use retriable HTTP client in proxy mode

[mtojek]

Fixes: #882

This PR replaces the original HTTP client with go-retryablehttp client. It uses a custom retry policy to additionally check the content type header.

Testing:

EPR_LOG_LEVEL=debug go run . -feature-proxy-mode -proxy-to https://elastic.co # not EPR endpoint

Logs:

{"log.level":"debug","@timestamp":"2022-09-21T15:12:59.342+0200","log.origin":{"file.name":"proxymode/proxymode.go","file.line":116},"message":"Proxy /search request","request.uri":"https://elastic.co/search?prerelease=true","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-09-21T15:12:59.342+0200","log.origin":{"file.name":"proxymode/logger.go","file.line":29},"message":"performing request","method":"GET","url":"https://elastic.co/search?prerelease=true","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-09-21T15:13:00.285+0200","log.origin":{"file.name":"proxymode/logger.go","file.line":29},"message":"retrying request","request":"GET https://elastic.co/search?prerelease=true (status: 200)","timeout":1000000000,"remaining":4,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-09-21T15:13:01.484+0200","log.origin":{"file.name":"proxymode/logger.go","file.line":29},"message":"retrying request","request":"GET https://elastic.co/search?prerelease=true (status: 200)","timeout":2000000000,"remaining":3,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-09-21T15:13:03.697+0200","log.origin":{"file.name":"proxymode/logger.go","file.line":29},"message":"retrying request","request":"GET https://elastic.co/search?prerelease=true (status: 200)","timeout":4000000000,"remaining":2,"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-09-21T15:13:07.899+0200","log.origin":{"file.name":"proxymode/logger.go","file.line":29},"message":"retrying request","request":"GET https://elastic.co/search?prerelease=true (status: 200)","timeout":8000000000,"remaining":1,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-21T15:13:16.102+0200","log.origin":{"file.name":"package-registry/search.go","file.line":51},"message":"proxy mode: search failed","error":{"message":"can't proxy search request: GET https://elastic.co/search?prerelease=true giving up after 5 attempt(s): unexpected content type: text/html; charset=utf-8","stack_trace":"\ngh.neting.cc/elastic/package-registry/proxymode.(*ProxyMode).Search\n\t/Users/marcin.tojek/go/src/github.com/elastic/package-registry/proxymode/proxymode.go:119\nmain.searchHandlerWithProxyMode.func1\n\t/Users/marcin.tojek/go/src/github.com/elastic/package-registry/search.go:49\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/Cellar/go/1.19.1/libexec/src/net/http/server.go:2109\ngo.elastic.co/apm/module/apmhttp.(*handler).ServeHTTP\n\t/Users/marcin.tojek/go/pkg/mod/go.elastic.co/apm/module/apmhttp@v1.15.0/handler.go:74\ngh.neting.cc/felixge/httpsnoop.CaptureMetrics.func1\n\t/Users/marcin.tojek/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.3/capture_metrics.go:29\ngh.neting.cc/felixge/httpsnoop.(*Metrics).CaptureMetrics\n\t/Users/marcin.tojek/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.3/capture_metrics.go:84\ngh.neting.cc/felixge/httpsnoop.CaptureMetricsFn\n\t/Users/marcin.tojek/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.3/capture_metrics.go:39\ngh.neting.cc/felixge/httpsnoop.CaptureMetrics\n\t/Users/marcin.tojek/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.3/capture_metrics.go:28\ngh.neting.cc/elastic/package-registry/util.captureZapFieldsForRequest\n\t/Users/marcin.tojek/go/src/github.com/elastic/package-registry/util/logging.go:120\ngh.neting.cc/elastic/package-registry/util.logRequest\n\t/Users/marcin.tojek/go/src/github.com/elastic/package-registry/util/logging.go:114\ngh.neting.cc/elastic/package-registry/util.LoggingMiddleware.func1.1\n\t/Users/marcin.tojek/go/src/github.com/elastic/package-registry/util/logging.go:105\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/Cellar/go/1.19.1/libexec/src/net/http/server.go:2109\ngh.neting.cc/gorilla/mux.(*Router).ServeHTTP\n\t/Users/marcin.tojek/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/Cellar/go/1.19.1/libexec/src/net/http/server.go:2947\nnet/http.(*conn).serve\n\t/usr/local/Cellar/go/1.19.1/libexec/src/net/http/server.go:1991\nruntime.goexit\n\t/usr/local/Cellar/go/1.19.1/libexec/src/runtime/asm_amd64.s:1594"},"ecs.version":"1.6.0"}

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-21T13:25:13.874+0000

• Duration: 5 min 34 sec

Test stats 🧪

Test	Results
Failed	0
Passed	213
Skipped	0
Total	213

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[jsoriano]

👍