Regression in 2.1.0? #1029

jschniper · 2024-05-22T02:36:06Z

I'm receiving the following error with the latest version:

** (exit) exited in: GenServer.call(:hex_fetcher, {:await, {:tarball, "hexpm", "decimal", "2.1.1"}}, 120000)                                                                                                           
    ** (EXIT) time out                                                                                                                                                           
    (elixir 1.16.1) lib/gen_server.ex:1114: GenServer.call/3                                               
    (hex 2.1.0) lib/hex/scm.ex:147: Hex.SCM.update/1                                    
    (hex 2.1.0) lib/hex/scm.ex:246: Hex.SCM.checkout/1                                  
    (mix 1.16.1) lib/mix/dep/fetcher.ex:64: Mix.Dep.Fetcher.do_fetch/3                  
    (mix 1.16.1) lib/mix/dep/converger.ex:229: Mix.Dep.Converger.all/8                  
    (mix 1.16.1) lib/mix/dep/converger.ex:244: Mix.Dep.Converger.all/8                  
    (mix 1.16.1) lib/mix/dep/converger.ex:162: Mix.Dep.Converger.init_all/8             
    (mix 1.16.1) lib/mix/dep/converger.ex:146: Mix.Dep.Converger.all/4

The only thing that I can see about the decimal dependency that is at all special is it appeared in multiple places in the umbrella app and each of one those had an override: true option because of a library with an out of date version. I was able to remove the override: true and then upgrade that dependency and mix deps.get finished as expected.

I'll try to dig into this a bit more tomorrow and see if I can be of more help.

The text was updated successfully, but these errors were encountered:

jschniper · 2024-05-22T02:39:36Z

One more data point, I had similar problem with {:jose, "~> 1.11", override: true} and removing the override took care of it. That one only appeared once in the umbrella app.

ericmj · 2024-05-22T04:18:06Z

Can you give us a way to reproduce the error?

steve-pieris · 2024-05-22T04:29:07Z

We just started getting a similar looking error but for a different package:

** (exit) exited in: GenServer.call(:hex_fetcher, {:await, {:tarball, "hexpm", "cowboy", "2.9.0"}}, 120000)
    ** (EXIT) time out
    (elixir 1.12.3) lib/gen_server.ex:1024: GenServer.call/3
    (hex 2.1.0) lib/hex/scm.ex:147: Hex.SCM.update/1
    (hex 2.1.0) lib/hex/scm.ex:246: Hex.SCM.checkout/1
    (mix 1.12.3) lib/mix/dep/fetcher.ex:64: Mix.Dep.Fetcher.do_fetch/3
    (mix 1.12.3) lib/mix/dep/converger.ex:190: Mix.Dep.Converger.all/9
    (mix 1.12.3) lib/mix/dep/converger.ex:201: Mix.Dep.Converger.all/9
    (mix 1.12.3) lib/mix/dep/converger.ex:123: Mix.Dep.Converger.all/7
    (mix 1.12.3) lib/mix/dep/converger.ex:108: Mix.Dep.Converger.all/4

Ganarajr · 2024-05-22T04:54:56Z

Getting the same issue for a different package

Getting google_gax (Hex package)
** (exit) exited in: GenServer.call(:hex_fetcher, {:await, {:tarball, "hexpm", "google_gax", "0.3.1"}}, 120000)
    ** (EXIT) time out
    (elixir 1.14.1) lib/gen_server.ex:1038: GenServer.call/3
    (hex 2.1.0) lib/hex/scm.ex:147: Hex.SCM.update/1
    (hex 2.1.0) lib/hex/scm.ex:246: Hex.SCM.checkout/1
    (mix 1.14.1) lib/mix/dep/fetcher.ex:64: Mix.Dep.Fetcher.do_fetch/3
    (mix 1.14.1) lib/mix/dep/converger.ex:213: Mix.Dep.Converger.all/9
    (mix 1.14.1) lib/mix/dep/converger.ex:224: Mix.Dep.Converger.all/9
    (mix 1.14.1) lib/mix/dep/converger.ex:146: Mix.Dep.Converger.all/7
    (mix 1.14.1) lib/mix/dep/converger.ex:131: Mix.Dep.Converger.all/4

ericmj · 2024-05-22T06:13:35Z

It can probably happen for any package, but we need a way to reproduce it to fix the issue. Please share your mix project including the mix.exs and mix.lock files.

barrieloydall · 2024-05-22T06:13:51Z

Seeing the same issue with another dependency.

Only change between ci builds was the update from hex-2.0.6 to hex-2.1.0

We see the issue with html_entities, a dependency of floki, and floki is used by an in house library. Due to a breaking change some time ago we pinned the version of html_entities with override: true in our library that uses floki.

e.g. {:html_entities, "0.5.1", override: true},

I was able to avoid the error by additionally including the html_entities dependency definition in our main service application that used the library.

main_service (:html_entities, "0.5.1", override: true)
│ │──in_house_library (:html_entities, "0.5.1", override: true)
│    │──floki
│       │──html_entities

Seeing the following error:

* Getting html_entities (Hex package)
** (exit) exited in: GenServer.call(:hex_fetcher, {:await, {:tarball, "hexpm", "html_entities", "0.5.1"}}, 120000)
    ** (EXIT) time out
    (elixir 1.15.7) lib/gen_server.ex:1074: GenServer.call/3
    (hex 2.1.0) lib/hex/scm.ex:147: Hex.SCM.update/1
    (hex 2.1.0) lib/hex/scm.ex:246: Hex.SCM.checkout/1
    (mix 1.15.7) lib/mix/dep/fetcher.ex:64: Mix.Dep.Fetcher.do_fetch/3
    (mix 1.15.7) lib/mix/dep/converger.ex:229: Mix.Dep.Converger.all/8
    (mix 1.15.7) lib/mix/dep/converger.ex:244: Mix.Dep.Converger.all/8
    (mix 1.15.7) lib/mix/dep/converger.ex:162: Mix.Dep.Converger.init_all/8
    (mix 1.15.7) lib/mix/dep/converger.ex:146: Mix.Dep.Converger.all/4

Exited with code exit status 1

In the output for "Resolving Hex dependencies..." html_entities was no longer listed under the "Unchanged:" list.

ericmj · 2024-05-22T06:24:40Z

Please folks, you gotta help me out here. Just reporting the same thing does not help.

Please share a way for us to reproduce the error. The easiest way to do is by sharing your mix.exs file (all of them if it's an umbrella app) and the mix.lock file.

barrieloydall · 2024-05-22T06:29:15Z

Please folks, you gotta help me out here. Just reporting the same thing does not help.

Please share a way for us to reproduce the error. The easiest way to do is by sharing your mix.exs file (all of them if it's an umbrella app) and the mix.lock file.

hi @ericmj it can be a little difficult to share private mix files as they can contain other private dependencies and information. What details from the files would help specifically?

I will try and create a stripped back minimal version to help repo the issue.

steve-pieris · 2024-05-22T07:40:39Z

We also use private mix files (which possibly might be partly the cause of this timeout behaviour).
I managed to reproduce the issue in a stripped down project.

Using the same version as our existing app, Elixir v1.12.3 and phx 1.6.2 created a new umbrella project, ie:

mix archive.install hex phx_new 1.6.2
mix phx.new deps_timeout_test --umbrella
cd deps_timeout_test
mix deps.get

At this point everything works fine.

Then update apps/deps_timeout_test_web/mix.ex to include a private forked version of an esaml dep that we have customised, ie:
NOTE: i've replaced the fork and sha for privacy

      {:esaml, "~> 4.2.0",
        github: "<fork>/esaml", ref: "<gitsha>"}

then clear the deps and attempt to get again:
NOTE: the clear is important because our CI build server always start fresh with no fetched deps

mix deps.clean --all
mix deps.get

and the we get the error:

* Getting cowboy (Hex package)
** (exit) exited in: GenServer.call(:hex_fetcher, {:await, {:tarball, "hexpm", "cowboy", "2.12.0"}}, 120000)
    ** (EXIT) time out
    (elixir 1.12.3) lib/gen_server.ex:1024: GenServer.call/3
    (hex 2.1.0) lib/hex/scm.ex:147: Hex.SCM.update/1
    (hex 2.1.0) lib/hex/scm.ex:246: Hex.SCM.checkout/1
    (mix 1.12.3) lib/mix/dep/fetcher.ex:64: Mix.Dep.Fetcher.do_fetch/3
    (mix 1.12.3) lib/mix/dep/converger.ex:190: Mix.Dep.Converger.all/9
    (mix 1.12.3) lib/mix/dep/converger.ex:201: Mix.Dep.Converger.all/9
    (mix 1.12.3) lib/mix/dep/converger.ex:123: Mix.Dep.Converger.all/7
    (mix 1.12.3) lib/mix/dep/converger.ex:108: Mix.Dep.Converger.all/4

NOTE: When I inspect the esaml entry in the mix.lock file I do not see the cowboy dependency

If I update the apps/deps_timeout_test_web/mix.ex again to use the public esaml, ie:

{:esaml, "~> 4.2.0"}

and run

mix deps.clean --all
mix deps.get

I get this error:

Resolving Hex dependencies...
Resolution completed in 0.051s
Because "the lock" specifies cowboy 2.12.0 and esaml >= 4.0.0 and < 4.3.0 depends on cowboy 2.6.0, the lock is incompatible with esaml >= 4.0.0 and < 4.3.0.
And because your app depends on the lock, esaml >= 4.0.0 and < 4.3.0 is forbidden.
So, because your app depends on esaml ~> 4.2.0, version solving failed.

Our forked version of esaml has bumped cowboy in it's rebar.config to 2.9.0. But if there is a version mismatch I would expect an error instead of a timeout.

Ganarajr · 2024-05-22T07:46:36Z

We tried (#1019 (comment) for google_gax) which works, here is the updated mix.exs

{:google_gax, git: "https://github.com/googleapis/elixir-google-api.git", tag: "ba9bf3ac88905ef3009bd043aac5b32679aa1052", sparse: "clients/gax", override: true},

However, we now get the same error with poison

 Getting poison (Hex package)
** (exit) exited in: GenServer.call(:hex_fetcher, {:await, {:tarball, "hexpm", "poison", "3.1.0"}}, 120000)
    ** (EXIT) time out
    (elixir 1.14.1) lib/gen_server.ex:1038: GenServer.call/3
    (hex 2.1.0) lib/hex/scm.ex:147: Hex.SCM.update/1
    (hex 2.1.0) lib/hex/scm.ex:246: Hex.SCM.checkout/1
    (mix 1.14.1) lib/mix/dep/fetcher.ex:64: Mix.Dep.Fetcher.do_fetch/3
    (mix 1.14.1) lib/mix/dep/converger.ex:213: Mix.Dep.Converger.all/9
    (mix 1.14.1) lib/mix/dep/converger.ex:224: Mix.Dep.Converger.all/9
    (mix 1.14.1) lib/mix/dep/converger.ex:146: Mix.Dep.Converger.all/7
    (mix 1.14.1) lib/mix/dep/converger.ex:131: Mix.Dep.Converger.all/4

ericmj · 2024-05-22T08:23:40Z

hi @ericmj it can be a little difficult to share private mix files as they can contain other private dependencies and information. What details from the files would help specifically?

We need the deps entry from mix.exs and the full mix.lock file.

lud-wj · 2024-05-22T10:43:55Z

@ericmj I cannot share my mix.exs either because you would not be able to pull our private libraries.

But I ran a quick debug session and it seems that Hex.Parallel does not receive a {:run, ...} call for the tarball.

RUN {:registry, "hexpm", "singleton"} await=false
ENQUEUE {:registry, "hexpm", "singleton"}
RUN_TASK {:registry, "hexpm", "singleton"}
SEND REPLY {:registry, "hexpm", "singleton"}
Resolving Hex dependencies...
Resolution completed in 0.523s
Unchanged:
  [...]
* Getting singleton (Hex package)
AWAIT {:tarball, "hexpm", "singleton", "1.3.2"}
** (exit) exited in: GenServer.call(:hex_fetcher, {:await, {:tarball, "hexpm", "singleton", "1.3.2"}}, 5000)
    ** (EXIT) time out
    (elixir 1.16.2) lib/gen_server.ex:1114: GenServer.call/3

(timeout of 5000 is my change)

In my case we have a dependency that depends on singleton with override: true. To be clear the override flag is the direct dependency mix.exs file, not in ours, and it is not needed AFAIK because our direct dependency is the only one depending on singleton. That specific case does not seem to be a requirement for the bug though.

(not sure if this is helpful, I don't know how hex works)

stefanluptak · 2024-05-22T11:18:12Z

For those that have their CI pipelines failing because of this, replace this:
mix local.hex --force
with this:
mix local.hex 2.0.6 --force
in your CI scripts/workflows.

flambard · 2024-05-22T12:49:04Z

We are seeing this issue with one of our umbrella projects that has an Erlang application in it. We had problems with the dependencies of that application (hackney and jsx).

wojtekmach · 2024-05-22T12:56:54Z

For anyone running into this error, please consider narrowing this down to a minimal reproduction by removing dependencies and re-running and then sending us a mix.exs file. It might be enough to just have:

# mix.exs
defmodule Foo do
  def project do
    [app: :foo, version: "1.0.0", deps: deps()]
  end

  defp deps do
    [
      # ...
    ]
  end
end

If mix.lock matters, that is you can't reproduce this without it existing prior, please send it too. A Mix.install invocation would be perfectly fine too. Thank you.

stefanluptak · 2024-05-22T13:38:30Z

For anyone running into this error, please consider narrowing this down to a minimal reproduction by removing dependencies and re-running and then sending us a mix.exs file. It might be enough to just have:
# mix.exs
defmodule Foo do
  def project do
    [app: :foo, version: "1.0.0", deps: deps()]
  end

  defp deps do
    [
      # ...
    ]
  end
end
If mix.lock matters, that is you can't reproduce this without it existing prior, please send it too. A Mix.install invocation would be perfectly fine too. Thank you.

https://github.com/stefanluptak/hex_debug

Just clone it, and run mix deps.get from the root of the repo with hex 2.1.0

wojtekmach · 2024-05-22T13:44:04Z

@stefanluptak perfect, I was able to reproduce the issue.

jschniper · 2024-05-22T13:46:30Z

So a little more data on this one:

I've only been able to reproduce this with an umbrella app.
A fresh app leads to a different error: ** (Mix) The lock is missing for package decimal. This could be because another package has configured the application name for the dependency incorrectly. Verify with the maintainer of the parent application so the lock file has something to do with it.
You can get to the error mentioned above by installing the deps with 2.0.6 and then deleting the deps folders and running it with 2.1.0 now that you have a lock file.

Sample Mix files that are enough to trigger it:

defmodule A.MixProject do
  use Mix.Project

  def project do
    [
      app: :a,
      version: "0.1.0",
      build_path: "../../_build",
      config_path: "../../config/config.exs",
      deps_path: "../../deps",
      lockfile: "../../mix.lock",
      elixir: "~> 1.16",
      start_permanent: Mix.env() == :prod,
      deps: deps()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger],
      mod: {A.Application, []}
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      {:b, in_umbrella: true}
    ]
  end
end

defmodule B.MixProject do
  use Mix.Project

  def project do
    [
      app: :b,
      version: "0.1.0",
      build_path: "../../_build",
      config_path: "../../config/config.exs",
      deps_path: "../../deps",
      lockfile: "../../mix.lock",
      elixir: "~> 1.16",
      start_permanent: Mix.env() == :prod,
      deps: deps()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger],
      mod: {B.Application, []}
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      {:pg_ranges, "1.1.0"},
      {:decimal, "~> 2.0", override: true}
    ]
  end
end

CuriousCurmudgeon · 2024-05-22T14:26:53Z

We don't have a minimal reproduction, but we are seeing the same issue in a non-umbrella app. Explicitly using 2.0.6 fixed it for us as well.

SamHutchings · 2024-05-22T14:39:04Z

no minimal repro but adding another datapoint - we are seeing the same error when running mix do deps.get --only prod, deps.compile --skip-umbrella-children.

the package which times out is prometheus version 4.11.0. we also have a private repo for Oban pro. it is installed in an umbrella app mix file as {:prometheus, "~> 4.0", override: true},

Closes #1029.

ericmj · 2024-05-22T14:53:18Z

We don't have a minimal reproduction, but we are seeing the same issue in a non-umbrella app. Explicitly using 2.0.6 fixed it for us as well.

@CuriousCurmudgeon Do you have a non-minimal reproduction?

EDIT: nvm, I also found a way to reproduce without an umbrella.

Closes #1029.

ceolinrenato · 2024-05-22T16:58:12Z

Hey @ericmj thanks for the fix!

Is this going to be published as 2.1.1?

ericmj · 2024-05-22T17:27:57Z

Yes. 2.1.1 has been published.

ericmj added Kind:Bug Note:Needs more info labels May 22, 2024

chess4ever mentioned this issue May 22, 2024

Allow passing (mix) version for hex and rebar3 installation erlef/setup-beam#275

Open

ericmj added a commit that referenced this issue May 22, 2024

Only filter non Hex overrides from requests

d3d2399

Closes #1029.

ericmj added a commit that referenced this issue May 22, 2024

Only filter non Hex overrides from requests

007ccbb

Closes #1029.

ericmj mentioned this issue May 22, 2024

Only filter non Hex overrides from requests #1030

Merged

ericmj closed this as completed in #1030 May 22, 2024

ericmj added a commit that referenced this issue May 22, 2024

Only filter non Hex overrides from requests (#1030)

02862a4

Closes #1029.

VanHelmont mentioned this issue Aug 13, 2024

Private hex registry error on Hex > 2.1.0 #1048

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regression in 2.1.0? #1029

Regression in 2.1.0? #1029

jschniper commented May 22, 2024 •

edited

Loading

jschniper commented May 22, 2024

ericmj commented May 22, 2024

steve-pieris commented May 22, 2024

Ganarajr commented May 22, 2024 •

edited

Loading

ericmj commented May 22, 2024

barrieloydall commented May 22, 2024 •

edited

Loading

ericmj commented May 22, 2024

barrieloydall commented May 22, 2024 •

edited

Loading

steve-pieris commented May 22, 2024

Ganarajr commented May 22, 2024 •

edited

Loading

ericmj commented May 22, 2024

lud-wj commented May 22, 2024 •

edited

Loading

stefanluptak commented May 22, 2024

flambard commented May 22, 2024

wojtekmach commented May 22, 2024

stefanluptak commented May 22, 2024

wojtekmach commented May 22, 2024

jschniper commented May 22, 2024

CuriousCurmudgeon commented May 22, 2024

SamHutchings commented May 22, 2024 •

edited

Loading

ericmj commented May 22, 2024 •

edited

Loading

ceolinrenato commented May 22, 2024

ericmj commented May 22, 2024

Regression in 2.1.0? #1029

Regression in 2.1.0? #1029

Comments

jschniper commented May 22, 2024 • edited Loading

jschniper commented May 22, 2024

ericmj commented May 22, 2024

steve-pieris commented May 22, 2024

Ganarajr commented May 22, 2024 • edited Loading

ericmj commented May 22, 2024

barrieloydall commented May 22, 2024 • edited Loading

ericmj commented May 22, 2024

barrieloydall commented May 22, 2024 • edited Loading

steve-pieris commented May 22, 2024

Ganarajr commented May 22, 2024 • edited Loading

ericmj commented May 22, 2024

lud-wj commented May 22, 2024 • edited Loading

stefanluptak commented May 22, 2024

flambard commented May 22, 2024

wojtekmach commented May 22, 2024

stefanluptak commented May 22, 2024

wojtekmach commented May 22, 2024

jschniper commented May 22, 2024

CuriousCurmudgeon commented May 22, 2024

SamHutchings commented May 22, 2024 • edited Loading

ericmj commented May 22, 2024 • edited Loading

ceolinrenato commented May 22, 2024

ericmj commented May 22, 2024

jschniper commented May 22, 2024 •

edited

Loading

Ganarajr commented May 22, 2024 •

edited

Loading

barrieloydall commented May 22, 2024 •

edited

Loading

barrieloydall commented May 22, 2024 •

edited

Loading

Ganarajr commented May 22, 2024 •

edited

Loading

lud-wj commented May 22, 2024 •

edited

Loading

SamHutchings commented May 22, 2024 •

edited

Loading

ericmj commented May 22, 2024 •

edited

Loading