mpi metapackage

[minrk]

Current proposal (updated 2017-12-12):

• this PR adds mpi metapackage, which will have a variant for each mpi provider (only mpich variant in this PR, conda-forge.yml matrix will add openmpi)
• features will not be used anywhere
• in mpi consumers, mpi provider will be indicated by build string (e.g mpich_0)
• mutual exclusivity will be achieved by making MPI providers depend directly on mpi metapackage with appropriate build string, e.g. mpi 1.0 mpich (and marking previous builds as broken)

TODO after merging this PR:

• add conda-forge.yml in mpi metapackage for mpich, openmpi variants add the build matrix mpi-feedstock#1
• make mpich, openmpi packages depend on mpi metapackage:
  • mpi metapackage mpich-feedstock#11
  • mpi metapackage openmpi-feedstock#11
• mark previous builds of mpich, openmpi as broken so they don't get pulled in as a solution to avoiding mutual exclusivity (requires conda-forge admin intervention)
• begin rollout of variants
  • add mpi variants mpi4py-feedstock#10

Sample for mpi provider (not much to do without features):

dependencies:
  run:
    - mpi 1.0 mpich

Sample for mpi consumer:

# recipe/meta.yaml
{% set mpi = os.environ.get('MPI_VARIANT', 'mpich') %}
{% set mpi_version = {'mpich': '3.2.*', 'openmpi': '3.0.*'}[mpi] %}
build:
  number: {{ build }}
  string: {{ mpi }}_{{ build }}
requirements:
  build:
    - {{ mpi }} {{ mpi_version }}
  run:
    - {{ mpi }} {{ mpi_version }}

# conda-forge.yml
matrix:
  - [[MPI_VARIANT, 'openmpi']]
  - [[MPI_VARIANT, 'mpich']]
travis:
  ...

Original description below:

NOTE: the current state of this discussion suggests that there would be no metapackage, in which case this PR should not be merged.

copy of blas metapackage. Only mpich for now, while we wait for openmpi.

Following the example of #643, I'll add the build matrix on the feedstock (after openmpi is packaged).

Still to decide: MPI default preference (openmpi or mpich)? I really don't have a preference, but currently mpich has fortran working, while openmpi seems to need to disable it on OS X currently,
so that would push my vote toward mpich.

cc @dalcinl for input on defaults. Not sure who else to ping.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/mpi) and found it was in an excellent condition.

[minrk]

Also cc @pelson, @jakirkham if this seems like the right way to support mpi via features. blas looks like a pretty similar situation, though I haven't seen any packages that use blas other than the default.

My understanding, based on looking at openblas, blas, and numpy/scikit-learn:

• track_features goes in the metapackage
• mpi-providing packages (mpich, openmpi) do not define features
• packages that link mpi (mpi4py, petsc, etc.) should depend on mpi metapackage and define the features.

[msarahan]

Great thought here. Here's my understanding.

mpi-providing packages should define features (in the build section), but should not track features. You're essentially saying "this package is part of this system of packages" more than you're saying "this package provides this feature"

mpi4py and others should install an mpi metapackage that activates whatever feature you want.

What is missing from this story is a way to choose which mpi metapackage you want. I am not sure that there is presently a solid way to do that, but there is discussion at https://gist.github.com/mcg1969/20667002c79f702e3c7003c81b5f24c8

[minrk]

@msarahan that makes sense to me in principle, but the blas packages don't currently do that. Does that mean that the openblas feedstock be defining blas_openblas feature in its build config?

[msarahan]

IMO, yes. This implies one blas implementation per environment (or by extrapolation, one mpi implementation per environment), and also implies that for full support, you'd need builds of something like mpi4py for all available mpi providers.

@jakirkham, I think you did the OpenBLAS implementation here. Do you remember your thoughts on including or excluding blas_openblas from your recipe?

[minrk]

@msarahan I tried adding the mpi_mpich feature to the mpich package for testing, but discovered that you cannot install a package with a feature without also installing a package that tracks that feature. The result being that mpich can't be installed on its own without also installing the mpi metapackage unless mpich itself also tracks the mpi_mpich feature. And if mpich both provides and tracks the mpi_mpich feature, it seems like the mpi metapackage doesn't accomplish much. Though I admit I've never had the best grasp of features.

[msarahan]

That is totally accurate, and features are squirrelly. Until the notion of variants is solidified, I agree that the mpi metapackage does not accomplish much. I think you want the notion of "provides" - and we do too, but it's not here yet.

[jakirkham]

I'd recommend reading this, @minrk. Maybe that will help inform this and its usage a bit.

[jakirkham]

Also take a look at the numpy recipe to get a feeling of how something like this should be used.

[jakirkham]

I think we got a second set of + by accident.

[minrk]

indeed. fixed.

[minrk]

What do you think about the following proposal:

• mpi implementations both define and track their mpi features
• mpi metapackage tracks mpi features, same as blas
• downstream packages that are sensitive to the mpi implementation track features
• downstream packages that require mpi but don't care about implementation depend on mpi (e.g. nothing linked at build time), but don't track features.

Does that make sense? If so, this PR should be ready as-is, and I can open PRs to openmpi and mpich to add the features.

The alternative is to follow the blas example exactly (differences in bold):

• mpi implementations neither define nor track features
• mpi metapackage tracks mpi features, same as blas
• downstream packages that are sensitive to the mpi implementation define features ref: numpy
• downstream packages that require mpi but don't care about implementation depend on mpi (e.g. nothing linked at build time), but don't track features.

Neither case changes this metapackage, so if the idea of an mpi metapackage makes sense, I think this package ought to be ready as-is, unless I'm missing something.

The next step that I'm not super familiar with is the best way to add MPI flavors to the build matrix, given the conda-smithy tooling. How do we most easily add iteration over feature flavors to feedstocks?

[minrk]

I answered my last question immediately after asking it: conda-forge.yml has a matrix field. Are the editable fields of conda-forge.yml documented anywhere?

[jakirkham]

No. 😞 When it comes to these, @pelson is the expert.

[dalcinl]

@minrk I still don't fully understand why we need an mpi metapackage. As the MPI API is not ABI-compatible, every library out there using MPI has to link against a specific MPI implementation, and you cannot switch it at runtime.

In your previous comment, you mention: "downstream packages that require mpi but don't care about implementation depend on mpi (e.g. nothing linked at build time), but don't track features." It is very hard to have a package using MPI with nothing linked at build time. It is teoretically possible, you could lookpup the MPI shared library, dlopen and probe some symbols, then distinguish between MPICH and Open MPI, and handle each differently. But to be honest, this would be quite involved, and I doubt anyone will ever write such a messy approach that can easily break because of ABI changes in major releases.

If you look at my original MPI recipes in https://bitbucket.org/petsc/conda-recipes, the MPI recipes both define and trach their own MPI features. Everything was working just fine. If you run conda install -c mpi4py mpi4py mpich it works, conda install -c mpi4py mpi4py openmpi also works. BTW, conda install -c mpi4py mpi4py also works, an prefers MPICH, maybe because "mpich" sorts first wrt "openmpi".

Maybe the whole point of an mpi metapackage is to have a clean way of conda install picking up the default MPI implementations (mpich in our case) w/o relying in pkg name sort order? Even though, if we want to properly support both MPICH and Open MPI in conda forge, we have to build EVERY MPI-based package (e.g. mpi4py/petsc/petsc4py/slepc/slepc4py) for both MPI flavors.

[minrk]

I'm fine with using features, as well. I think the only benefit of the mpi metapackage is it allows packages to depend on any provider of an mpi runtime by not tracking features, such as a package that's not linked to mpi, but will use mpiexec, not caring where it came from. There may be no such packages, of course.

I'd just like a policy decision for how packages should express which mpi they use, then we can get started on the build matrix for these things.

[tadeu]

Could it use the variant name for the version?
So the user could do things like:

conda install mpi4py mpi=openmpi
conda install mpi4py mpi=mpich
conda install mpi4py mpi=msmpi60

In this case, the recipe wouldn't need to manually set the build string, as the version goes into the package filename.

One great advantage of using version is that it would not be possible to install two different MPI flavors at the same time. Also, making packages differ only by build string seems weird.

[tadeu]

Also, about the run requirement, sometimes MPI is installed in the system, so could this metapackage not depend on an MPI implementation package?

[dalcinl]

@tadeu I'm not sure about abusing the version to select the backend MPI implementation. Your example should just be conda install mpi4py mpich or conda install mpi4py openmpi. Please look again to @minrk comments and my comments, users should never have to install the mpi metapackage but a concrete mpi provider.

[tadeu]

@dalcinl

But in this case, what would happen with the following command?
conda install mpi4py mpich openmpi

Also, I'm thinking about the case when the user already has MPI installed in the system. For example, on Windows, the "correct" way seems to be the user to install the Microsoft MPI packages in the system, totally outside of conda, so the user would only need to choose the correct metapackage version.

[dalcinl]

@tadeu conda install mpich openmpi should just fail, but this would require conda to honor the requeriments/conflicts section of the recipe meta.yaml file of each concrete MPI package.

On windows, users should just conda install mpi4py or even conda install mpi4py msmpi, in the first case dependencies should trigger the installation of the concrete MPI package msmpi. This concrete package would just run a post-install script to check that MS-MPI v7.1 is actually installed in the system, otherwise error and print info in the console requesting users to manually install MS-MPI (or we can download the installers automatically and start the installation, I do not remember how that would work for non-admin accounts).

[tadeu]

@dalcinl AFAIK, requeriments/conflicts is undocumented and is currently not working :/

The problem on Windows is that the user could have MS-MPI v6.0 (the mpi4py package from Anaconda distribution uses this), MS-MPI v7.1 or even MS HPC, and all would need different features and different versions of mpi4py & company. A metapackage that only tracks the feature seems better in this scenario (the "alternative" proposal from @minrk comment: #1501 (comment)), and this is consistent with the blas package and the vc package on Windows.

[minrk]

Picking this up again: I'd like to propose the following, based on @dalcinl's points:

• mpi implementations both define and track their mpi features, which should be mpi_packagename
• packages that depend on mpi depend directly on an mpi-providing package, and track the appropriate feature
• no metapackage

So the mpich package looks like this:

features:
  - mpi_{{name}}
track_features:
  - mpi_{{name}}

and mpi4py would look like this:

{% set mpi = os.environ.get('MPI_VARIANT', 'mpich') %}

track_features:
  - mpi_{{mpi}}

requires:
  build:
    - {{mpi}}
  run:
    - {{mpi}}

Another option is to follow @dalcinl's example exactly, and:

• no features or tracking in mpi-providing packages
• define features in mpi-requiring packages

for example:

{% set mpi = os.environ.get('MPI_VARIANT', 'mpich') %}

features:
  - mpi_{{mpi}}

requires:
  build:
    - {{mpi}}
  run:
    - {{mpi}}

(which is the same as the above, but declaring instead of tracking the mpi_mpich feature in the downstream packages).

The same thing should work for msmpi, however it turns out msmpi packages should be defined.

[ocefpaf]

@isuruf can you take a look at this one?

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/mpi) and found some lint.

Here's what I've got...

For recipes/mpi:

• The "test" section was expected to be a dictionary, but got a NoneType.
• The recipe must have some tests.
• The "test" section was expected to be a dictionary, but got a NoneType.

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/mpi) and found it was in an excellent condition.

[minrk]

From conda-forge meeting discussion, the proposal was changed to: basically the same as before, but use build strings directly instead of setting any features:

1. nothing to do with features in any package
2. mpi metapackage sets build string for each variant
3. mpi providers depend on metapackage with buildstring, e.g. - mpi 1.0 mpich, rather than relying on features for mutual exclusivity
4. downstream packages include mpi variant in build string manually, e.g.

   {% set mpi = os.environ['MPI_VARIANT'] %}
   build:
     number: {{ build }}
     string: {{ mpi }}_{{ build }}

5. downstream packages depend directly on mpi provider, not mpi metapackage
6. users just install packages, but can pin metapackage if they so choose

BTW, this proposal means that downstream packages can begin adopting mpi variants right away, since there aren't features to wait for in the upstream packages. Only the mutual-exclusivity provided by the metapackage is something to wait for, but that's independent of changes to the packages.

[minrk]

I've updated the description of this PR with the todo items after merge. I think this is good to go now.

[dalcinl]

@minrk Is there anything left? This PR has been sitting here for more than a month, what should we do to get it merged?

[minrk]

I believe there is nothing left to do. It is awaiting approval (and potentially requests for further changes). The description at the top has several TODO items, all of which are to be done after merge of this PR.

[ocefpaf]

Let's go for this. I guess we dropped the ball here. We can improve things as we go.

[minrk]

Yay!

[leofang]

Dear @minrk, I begin looking into maintaining MPI-related packages for our own Conda ecosystem. I see that the current practice is summarized in your #1501 (comment) (thanks for documenting it btw!), but I get confused: what's the advantage of having the metapackage mpi, then?

Taking mpi4py as an example, if I understand it correctly the current dependency graph looks like this:

mpi4py → mpich   ↘ 
                   mpi 
mpi4py → openmpi ↗

But it seems mpi can be removed altogether. Am I right? Thanks!

[minrk]

the mpi metapackage is really only there for mutual exclusivity. Without it, installing mpi4py with mpich and then later installing mpi4py with openmpi would result in all three of mpi4py, openmpi, and mpich installed at the same time, which isn't good because e.g. openmpi and mpich both provide mpiexec and friends.

[leofang]

Thank you @minrk for the explanation!

[minrk]

Since more recent conda has the (undocumented, I think) run_constrained field, we can actually achieve mutual exclusivity without a metapackage now, but that didn't exist when the current scheme was put together.

But it should be considered an implementation detail (no package or user should interact directly with the mpi metapackage other than mpi providers themselves).

[rainwoodman]

Hi, bringing this up again -- could you explain about this commit[1]? I couldn't quite understand why it doesn't break mpi4py: the build string is omitted, but the discussion here suggests a build string with the mpi impl is required.

[1] conda-forge/mpi4py-feedstock@25a6265#diff-e178b687b10a71a3348107ae3154e44c

[minrk]

Putting the mpi provider in the build string accomplished two things:

1. allow uploading multiple variants. This was made obsolete by conda-build 3 including a hash of dependencies in the build string, so variants no longer need explicit custom build strings. The default build strings work.
2. allow users specifying a variant at install time. This is not generally needed, since conda install mpich mpi4py accomplishes the same thing and is clearer and simpler than conda install mpi4py=*=*mpich*. Putting the provider in the build string is optional and mainly useful in cases where a nompi variant is desired.

[rainwoodman]

Thanks for the explanation!

…

On Mon, Jul 22, 2019, 1:29 AM Min RK ***@***.***> wrote: Putting the mpi provider in the build string accomplished two things: 1. allow uploading multiple variants. This was made obsolete by conda-build 3 including a hash of dependencies in the build string, so variants no longer need explicit custom build strings. The default build strings work. 2. allow users specifying a variant at install time. This is not generally needed, since conda install mpich mpi4py accomplishes the same thing and is clearer and simpler than conda install mpi4py=*=*mpich*. Putting the provider in the build string is optional and mainly useful in cases where a nompi variant is desired. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1501?email_source=notifications&email_token=AABBWTHTRFTUYIBIVHWYPLTQAVVU3A5CNFSM4CO2YIIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2PFUCI#issuecomment-513694217>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABBWTCTO3W4UNX37PLO333QAVVU3ANCNFSM4CO2YIIA> .