-filter-with-deps across mirrors

[rdicroce]

I was trying to make a snapshot of selected Debian Jessie packages, getting the latest version of each by mirroring the main Jessie repo plus updates, backports, and security. (Similar to the "mirroring with snapshots" tutorial.)

This exposes a problem when mirroring using -filter-with-deps, snapshotting, and then merging. Wireshark 1.x is in the main Jessie repo, but backports contains Wireshark 2.x. Both repos contain the wireshark-qt package, but the 2.x version has dependencies that the 1.x version does not. Said dependencies are in the main repo, but not the backports repo. So when I merged the snapshots together, 'aptly snapshot verify' failed because the new dependencies aren't in any of the source snapshots that were merged together. Attempting to install Wireshark from the resulting snapshot fails for the same reason.

I don't see any way to -filter-with-deps across multiple mirrors, so I can only see two workarounds:

1. Mirror the entirety of all of the repositories. This is not feasible due to the amount of bandwidth and disk space that would be consumed.
2. Add the missing dependencies to the filter configuration for each mirror. This works, but means you have to remember to verify your snapshots every time you update your mirrors, in order to find out if you're missing any dependencies. Then you need to update your mirror filters, rinse and repeat until verification succeeds.

It would be nice if aptly handled this situation better. Maybe users could define a "meta-mirror" that takes multiple URLs and presents a combined package list, which could then be filtered as a whole. This behavior would be more apt-like, since apt will search all the sources it has available when trying to locate a package.

[smira]

I understand this issue, but I'm not sure how we could implement it...

[rdicroce]

I haven't looked at aptly's code, so I don't know if this is feasbile, but allow me to expand on the "meta-mirror" idea. I suppose it might be better to call it a "mirror group".

A mirror group would function the same way as a regular mirror does now, except that when you create it, you can specify multiple URL + distribution + component(s) tuples. When the mirror group is updated, it pulls package lists from all of the configured locations and merges them together. The merge process is very dumb; it just appends the package lists together so that the resulting list contains all versions of all packages from all the locations. Then any filters are applied, with dependencies being followed (if desired). This could result in multiple versions of the same package being present after the filter, which is still fine as I believe that is legal for an Apt repo anyway. Finally, the filtered packages are downloaded from the appropriate mirrors.

At that point, the mirror group can be snapshotted and merged with other snapshots just like a normal mirror. Duplicate versions could be eliminated at this point using the existing options.

To make the above easier to follow, suppose we have the following:

• Two repositories we want to mirror (A and B).
• A package X where version 1.0 had no dependencies, but version 2.0 depends on package Y.
• X 1.0 and Y are in repository A, and X 2.0 is in repository B.
• A mirror group created with -filter="X" and -filter-with-deps

In that case, when the mirror group is updated it would:

1. Fetch package lists for both repositories.
2. Append them together. The combined list contains X 1.0, X 2.0, Y, and whatever other packages are in those repos.
3. Apply the filter against the combined list. Since -filter-with-deps is true and the filter is just for X, the filtered list contains X 1.0, X 2.0, and Y.
4. Download X 1.0 and Y from A, and download X 2.0 from B.

After this point, everything is the same as it is now. We snapshot the mirror group, then merge it with some other snapshots, perhaps using the default behavior or -latest to resolve duplicate package versions.

On the downside, this results in a snapshot that may contain unneeded packages, but on the upside, it will definitely pass verification. The unneeded packages problem could be solved using a smarter package list merge process, using the same options as snapshot merge already has (i.e. first repo wins, latest version wins, or include all versions).

[wwentland]

This would be nice to have

[rtennill]

Have you tried -latest=true option to snapshot merging? I was running into an issue with differing versions of libc when merging main and security and using this option fixed it.

It would apply to all packages in the snapshot which could be undesirable.

[rfarley3]

Currently dealing with this. Another concrete example: bwm-ng is on xenial universe, but its dependency libstatgrab10 is on xenial (main). If I build mirrors of both universe and main with -filter-with-deps -filter='bwm-ng', then libstatgrab10 will not be in my mirror/snapshot/published repo and bwm-ng can not be apt installed from my repo. A bit rough to say this, but effectively, aptly silently fails. After looking around, if you set -dep-verbose-resolve, then you can parse the stdout of mirror update for lines that start with Unsatis to get a list of packages and detect the failure.

In my situation, I'm mirroring a filter (list of direct dependencies from ansible plays) of the ubuntu xenial repos and putting their snapshots into a single published repo. My current work around is as previous suggested (manually modify the filter as I find unsat deps). You can take the union of the Unsatisfied packages with output from aptly mirror show -with-packages to make your new filter. For simplicity, I used the same (but increasingly larger) filter across all mirrors. Keep editing and updating all your mirrors until there are no new packages found. I then published all their snapshots as a single endpoint.

[rfarley3]

68 primary deps become 689 over 3 iterations. ugly example:

    aptly_filter = [ some_initial_package_names, ]
    unsat = []
    for mirror in mirrors:
        mirror_create(mirror, aptly_filter)
        unsat.extend(mirror_update(mirror))
    aptly_filter = get_pkgs_from_all_mirrors(mirrors, unsat)
    prev_rdeps_cnt = 0
    curr_rdeps_cnt = len(aptly_filter)
    while prev_rdeps_cnt < curr_rdeps_cnt:
        prev_rdeps_cnt = curr_rdeps_cnt
        unsat = []
        for mirror in mirrors:
            mirror_edit(mirror, aptly_filter)
            unsat.extend(mirror_update(mirror))
        aptly_filter = get_pkgs_from_all_mirrors(mirrors, unsat)
        curr_rdeps_cnt = len(aptly_filter)

def get_pkgs_from_all_mirrors(mirrors, unsat):
    # merges mirror show pkgs with mirror update unsat pkgs


def mirror_update(mirror):
    cmd = [
        'aptly',
        'mirror',
        'update',
        mirror['name'],
    ]
    c_out = delegate_run(cmd)
    # If this mirror is missing a filter's dep, it won't fail
    # but that means get_pkgs_from_all_mirrors will not see it
    # with dependencyVerboseResolve, we can parse it from stdout, looks like:
    # Unsatisfied dependency: libtcl8.6 (>= 8.6.0) [amd64]
    lines = [s.strip() for s in c_out.splitlines() if s.startswith('Unsatis')]
    pkgs = []
    for pkg_line in lines:
        pkg = pkg_line.split(':')[1].strip().split(' ')[0]
        pkgs.append(pkg)
    return pkgs

[abh23]

Just hit this issue. I am creating 2 mirrors (trusty-main and trusty-updates). While creating trusty-updates, I am filtering 'openjdk-7-jre' along with its dependencies. I can see 'Unresolved dependencies' errors while doing 'aptly mirror update trusty-updates'

I believe it is actually happening since few of the dependencies are in trusty-main and there is no way trusty-updates mirror could resolve it.

anyone able to find a workaround to this issue yet?

[sehe]

Me too. I added my comment to #562 first #562 (comment)

Quoting a possible idea:

There should really be a way to allow recursively inspecting dependencies from other components (so a dependencyFollowAllComponents similar to the existing dependencyFollowAllVariants)