Dependency resolutions infinite loop?!

[kgizdov]

Hi,

I am trying the master branch of aura built with stack install -- aura on ArchLinux.

With the version from 27th July, I get pretty quickly into the dependency resolution falling into an infinite loop or stuck somehow. I've confirmed this today with the latest master.

I do this:

$ sudo ~/.local/bin/aura -Auxa
aura >>= Fetching package information...
aura >>= Comparing package versions...
aura >>= AUR Packages to upgrade:
chrome-gnome-shell-git :: 10.r0.ga8f2b1a-1 -> 10.1.r2.gf6d3cc3-1
chromium-widevine      :: 1:1.4.9.1088-2   -> 1:4.10.1146.0-1
google-chrome          :: 67.0.3396.99-1   -> 68.0.3440.75-1
python-keras           :: 2.2.0-7          -> 2.2.2-1
python2-keras          :: 2.2.0-7          -> 2.2.2-1
tpm2-tools             :: 3.1.0-2          -> 3.1.1-2
tpm2-tss               :: 2.0.0-2          -> 2.0.0-3
zotero                 :: 5.0.53-1         -> 5.0.54-1

aura >>= Saved package state.
aura >>= Determining dependencies...

and it never gets beyond this point. I also noticed that aura requests 1.0TB of virtual memory and 16 forks on a 2 core (4 thread) CPU, which to me seems unnecessary. Here's a shot from htop

If it helps, the current release aura version 1.4.0 returns an error:

$ sudo aura -Auxa
aura >>= Fetching package information...
aura >>= Comparing package versions...
aura >>= AUR Packages to upgrade:
chromium-widevine : 1:1.4.9.1088-2 => 1:4.10.1146.0-1
google-chrome : 67.0.3396.99-1 => 68.0.3440.75-1
python-keras : 2.2.0-7 => 2.2.2-1
python2-keras : 2.2.0-7 => 2.2.2-1
tpm2-tools : 3.1.0-2 => 3.1.1-2
tpm2-tss : 2.0.0-2 => 2.0.0-3
zotero : 5.0.53-1 => 5.0.54-1

aura >>= The following are not AUR packages:
python2-keras

aura >>= Determining dependencies...
aura >>= Dependency checking failed for these reasons:
The dependency `python2-keras-applications` could not be found. You may need to search for a package to satisfy it.

Although, python2-keras-applications is provided by pkgbase python-keras-applications. Something might be going wrong with this in particular. Maybe worth a look.

[fosskers]

Thanks, I'll take a look.

[kgizdov]

This turns out to be some sort of circular dependency issue. Check the AUR mailing list. My issue though is that aura 2.0 doesn't detect this and doesn't try to mitigate it. Therefore, indeed, it must be falling in an infinite loop. Could something be done to improve this?

EDIT: Looks like the packages were updated to correct the circular dependency issue. However, aura should still try and mitigate these situations imo.

[fosskers]

I also noticed that aura requests 1.0TB of virtual memory and 16 forks on a 2 core (4 thread) CPU, which to me seems unnecessary.

The Haskell runtime always requests 1.0TB of virtual memory - it has something to do with it having an infinite heap (or stack, can't remember which). Either way, it's nothing to worry about. Aura itself doesn't consume that much memory.

It's also funny that you mention the 16 forks, since before a recent change that number used to be much higher. Now it's something like:

• 1 thread per CPU for dependency resolution
• each of those uses 4 more (haskell) threads to concurrently do web and shell calls to obtain package information.

[fosskers]

However, aura should still try and mitigate these situations imo.

I'm looking into it.

[fosskers]

Tentatively solved. Aura can't yet tell where the cycle is, but it least it can detect it (thanks to the underlying algebraic-graphs library).

[fosskers]

The details of the cycle will now be displayed.

[kgizdov]

I made a fresh build today and the above mentioned issue is fixed, however, I see a new problem that may be related. Doing an upgrade this morning gave me:

$ sudo ~/.local/bin/aura -Auxa
aura >>= Fetching package information...
aura >>= Comparing package versions...
aura >>= AUR Packages to upgrade:
google-chrome :: 68.0.3440.75-1 -> 68.0.3440.84-1
tpm2-tools    :: 3.1.1-2        -> 3.1.1-3
tpm2-tss      :: 2.0.0-3        -> 2.0.0-4

aura >>= Saved package state.
aura >>= Determining dependencies...

aura >>= WARNING: The PKGBUILD of `tpm2-tools` contains blacklisted bash terms.
depends=('tpm2-tss>=2.0.0' 'curl');
aura >>= `curl` can be used to download arbitrary scripts that aren't tracked by this PKGBUILD.
aura >>= Do you wish to quit the build process? (recommended) [Y/n] n
aura >>= AUR Packages:
google-chrome
tpm2-tools
tpm2-tss
aura >>= Continue? [Y/n] 
aura >>= Building `google-chrome`...
Cloning into 'google-chrome'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 7 (delta 0), reused 5 (delta 0)
Unpacking objects: 100% (7/7), done.
==> Making package: google-chrome 68.0.3440.84-1 (Fri 03 Aug 2018 12:36:46 MSK)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Downloading google-chrome-stable_68.0.3440.84_amd64.deb...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 51.5M  100 51.5M    0     0   329k      0  0:02:40  0:02:40 --:--:--  305k
  -> Found eula_text.html
  -> Found google-chrome-stable.sh
==> Validating source files with md5sums...
    google-chrome-stable_68.0.3440.84_amd64.deb ... Passed
    eula_text.html ... Passed
    google-chrome-stable.sh ... Passed
==> Extracting sources...
  -> Extracting google-chrome-stable_68.0.3440.84_amd64.deb with bsdtar
==> Entering fakeroot environment...
==> Starting package()...
  -> Extracting the data.tar.xz...
  -> Moving stuff in place...
  -> Fixing Chrome icon resolution...
  -> Removing Debian Cron job and duplicate product logos...
==> Tidying install...
  -> Removing empty directories...
  -> Removing libtool files...
  -> Purging unwanted files...
  -> Removing static library files...
  -> Compressing man and info pages...
==> Checking for packaging issues...
==> Creating package "google-chrome"...
  -> Generating .PKGINFO file...
  -> Generating .BUILDINFO file...
  -> Adding install file...
  -> Generating .MTREE file...
  -> Compressing package...
==> Leaving fakeroot environment.
==> Finished making: google-chrome 68.0.3440.84-1 (Fri 03 Aug 2018 12:41:47 MSK)
aura >>= Building `tpm2-tss`...
Cloning into 'tpm2-tss'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 0), reused 4 (delta 0)
Unpacking objects: 100% (4/4), done.
==> Making package: tpm2-tss 2.0.0-4 (Fri 03 Aug 2018 12:41:52 MSK)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Missing dependencies:
  -> ibm-sw-tpm2
  -> uthash
==> ERROR: Could not resolve all dependencies.
aura >>= Well, building `tpm2-tss` failed.
aura >>= There was a makepkg failure.
aura >>= Would you like to continue anyway? [Y/n] 
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) google-chrome-68.0.3440.84-1

Total Installed Size:  185.33 MiB
Net Upgrade Size:       -0.15 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                     [##########################] 100%
(1/1) checking package integrity                   [##########################] 100%
(1/1) loading package files                        [##########################] 100%
(1/1) checking for file conflicts                  [##########################] 100%
(1/1) checking available disk space                [##########################] 100%
:: Processing package changes...
(1/1) upgrading google-chrome                      [##########################] 100%
==> NOTE: Custom flags should be put directly in: ~/.config/chrome-flags.conf
==> NOTE: The launcher is called: 'google-chrome-stable'
:: Running post-transaction hooks...
(1/3) Updating icon theme caches...
(2/3) Arming ConditionNeedsUpdate...
(3/3) Updating the desktop file MIME type cache...
aura >>= Building `tpm2-tools`...
Cloning into 'tpm2-tools'...
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 0), reused 4 (delta 0)
Unpacking objects: 100% (4/4), done.
==> Making package: tpm2-tools 3.1.1-3 (Fri 03 Aug 2018 12:44:49 MSK)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Downloading tpm2-tools-3.1.1.tar.gz...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   613    0   613    0     0    424      0 --:--:--  0:00:01 --:--:--   424
100  547k  100  547k    0     0  29103      0  0:00:19  0:00:19 --:--:-- 28051
  -> Downloading tpm2-tools-3.1.1.tar.gz.asc...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   617    0   617    0     0    687      0 --:--:-- --:--:-- --:--:--   687
100   801  100   801    0     0    448      0  0:00:01  0:00:01 --:--:--  1208
==> Validating source files with sha256sums...
    tpm2-tools-3.1.1.tar.gz ... Passed
    tpm2-tools-3.1.1.tar.gz.asc ... Skipped
==> Verifying source file signatures with gpg...
    tpm2-tools-3.1.1.tar.gz ... FAILED (unknown public key 49BCAE5443FFFC34)
==> ERROR: One or more PGP signatures could not be verified!
aura >>= Well, building `tpm2-tools` failed.
aura >>= There was a makepkg failure.
aura >>= Would you like to continue anyway? [Y/n] 
aura >>= Every package failed to build.

(ignore the failed signature)
There are several issues. Initially, the dependencies for tpm2-tss were found (in repo only!?) and the update continued, but when it came to actually making the package, the deps were not resolved. uthash is in Community and ibm-sw-tpm2 is in AUR. This is probably because I was not prompted to build/install them in the first place.
Which brings me to the second problem, google-chrome was built, but was only installed after the failed build of tpm2-tss, and not immediately after its build.

This all points me to maybe a deeper problem with how deps are handled. I'm guessing aura initially only checks if deps (for all requested packages) exist in a repo and then continues, but never checks if they are installed on the system (or at least it does not makes sure they are before the build). Then the entire state of the current install/upgrade is propagated as a whole, rather than handling it on a per package basis. That is probably why google-chrome was not immediately installed after it was built. This makes me worry that the reason a package is installed is not going to be correctly flagged by pacman - for example, a package installed as a dep should be labelled as such with its requesting package only; also an explicitly installed package should not be mixed up with other packages, even if it is a dependency of some other currently requested package; probably other issues as well. (Bonus issue - could it be that mixing deps from official repos and AUR is currently mishandled?)

I'm not aware if this is a design choice or unexpected behaviour, but it seems quite serious to me. Could you have another look?
Thanks.

[fosskers]

but never checks if they are installed on the system

It does indeed check to see if they're already installed.

a package installed as a dep should be labelled as such with its requesting package only

I do make sure to handle the "As Dep" status of each package.

Not sure why it failed, but I'll look into tpm2-tss specifically.

[kgizdov]

OK, I'm glad. It must be some fluke with the checkdepends.

Btw, is aura 2.0 no longer using the parallel downloads from powerpill and so on? I'm basically trying to switch to it on a daily basis. Quite like it so far, just want to make sure it's stable enough and reporting anything I see. Would help with code, but I can't haskell for the life of me :)

[fosskers]

Yup, I dropped support for powerpill. It has caused bugs in the past.

I'm basically trying to switch to it on a daily basis.

Cool! I'm going on vacation for a week tomorrow, so the current master is a great state to test things in. I just merged the improved PKGBUILD analysis, so please let me know if there's anything weird with that.