Error: can't proxy search request: invalid character '<' looking for beginning of value

[mtojek]

@mrodm spotted an issue in the elastic/integrations repository. It would be good to introduce a retry mechanism to prevent such problems - I suspect a temporary networking issue (502?)

Stack trace:

{"log.level":"error","@timestamp":"2022-09-19T16:39:07.032Z","log.origin":{"file.name":"package-registry/search.go","file.line":51},"message":"proxy mode: search failed","error":{"message":"can't proxy search request: invalid character '<' looking for beginning of value","stack_trace":"\ngh.neting.cc/elastic/package-registry/proxymode.(*ProxyMode).Search\n\t/package-registry/proxymode/proxymode.go:96\nmain.searchHandlerWithProxyMode.func1\n\t/package-registry/search.go:49\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2084\ngo.elastic.co/apm/module/apmhttp.(*handler).ServeHTTP\n\t/go/pkg/mod/go.elastic.co/apm/module/apmhttp@v1.15.0/handler.go:74\ngh.neting.cc/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:142\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2084\ngh.neting.cc/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:92\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2084\ngh.neting.cc/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:56\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2084\ngh.neting.cc/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerRequestSize.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:234\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2084\ngh.neting.cc/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerResponseSize.func1\n\t/go/pkg/mod/github.com/prometheus/client_golang@v1.13.0/prometheus/promhttp/instrument_server.go:284\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2084\ngh.neting.cc/elastic/package-registry/metrics.MetricsMiddleware.func1.1\n\t/package-registry/metrics/middleware.go:49\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2084\ngh.neting.cc/felixge/httpsnoop.CaptureMetrics.func1\n\t/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.3/capture_metrics.go:29\ngh.neting.cc/felixge/httpsnoop.(*Metrics).CaptureMetrics\n\t/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.3/capture_metrics.go:84\ngh.neting.cc/felixge/httpsnoop.CaptureMetricsFn\n\t/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.3/capture_metrics.go:39\ngh.neting.cc/felixge/httpsnoop.CaptureMetrics\n\t/go/pkg/mod/github.com/felixge/httpsnoop@v1.0.3/capture_metrics.go:28\ngh.neting.cc/elastic/package-registry/util.captureZapFieldsForRequest\n\t/package-registry/util/logging.go:120\ngh.neting.cc/elastic/package-registry/util.logRequest\n\t/package-registry/util/logging.go:114\ngh.neting.cc/elastic/package-registry/util.LoggingMiddleware.func1.1\n\t/package-registry/util/logging.go:105\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2084\ngh.neting.cc/gorilla/mux.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2916\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1966\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1571"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-19T16:39:07.032Z","log.logger":"http","message":"GET /search HTTP/1.1","source.address":"172.18.0.4","http.request.method":"GET","url.path":"/search","url.domain":"package-registry","http.response.code":500,"http.response.body.bytes":22,"event.duration":9007083907,"source.ip":"172.18.0.4","user_agent.original":"Kibana/8.0.0 node-fetch","url.query":"package=elastic_agent&experimental=true&kibana.version=8.0.0","url.port":8080,"ecs.version":"1.6.0"}