Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: simple platform specific dependencies #111

Merged
merged 14 commits into from
Jun 20, 2023
20 changes: 20 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 6 additions & 3 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ native-tls = ["reqwest/native-tls", "rattler_repodata_gateway/native-tls", "ratt
rustls-tls = ["reqwest/rustls-tls", "rattler_repodata_gateway/rustls-tls", "rattler/rustls-tls"]

[dependencies]
ariadne = "0.3.0"
anyhow = "1.0.70"
clap = { version = "4.2.4", default-features = false, features = ["derive", "usage", "wrap_help", "std", "color", "error-context"] }
clap_complete = "4.2.1"
Expand All @@ -23,7 +24,7 @@ dirs = "5.0.1"
dunce = "1.0.4"
futures = "0.3.28"
indicatif = "0.17.3"
insta = "1.29.0"
insta = { version = "1.29.0", features=["yaml"] }
is_executable = "1.0.1"
itertools = "0.10.5"
minijinja = { version = "0.32.0" }
Expand All @@ -43,12 +44,14 @@ rattler_networking = { version = "0.3.0", default-features = false, git = "https
#rattler_virtual_packages = { version = "0.2.0", default-features = false, path="../rattler/crates/rattler_virtual_packages" }
reqwest = { version = "0.11.16", default-features = false }
serde = "1.0.163"
serde_with = "3.0.0"
serde_with = { version = "3.0.0", features = ["indexmap"] }
shlex = "1.1.0"
tempfile = "3.5.0"
tokio = { version = "1.27.0", features = ["macros", "rt-multi-thread"] }
toml_edit = { version = "0.19.8", features = ["serde"] }
toml_edit = { version = "0.19.10", features = ["serde"] }
serde_spanned = "0.6.2"
tracing = "0.1.37"
indexmap = { version = "1.9.3", features = ["serde"] }

[profile.release-lto]
inherits = "release"
Expand Down
77 changes: 49 additions & 28 deletions docs/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,55 @@ For instance, to check the platform compatibility of the python package, you can

Incorporating the appropriate platform configurations in your project ensures its broad usability and accessibility across various environments.

## The `dependencies` part

As pixi is a package manager we obviously provide a way to specify dependencies.
Dependencies are specified using a "version" or "version range" which for Conda is a "MatchSpec"

This is a conda specific way to specify dependencies, to avoid failing to write a good explanation I'll link you to some introductory reads:
- [Conda build documentation](https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html#id6)
- [Excellent stackoverflow answer](https://stackoverflow.com/a/57734390/13258625)
- [Conda's python implementation](https://github.com/conda/conda/blob/main/conda/models/match_spec.py)
- [Rattler's rust implementation (ours)](https://github.com/mamba-org/rattler/blob/main/crates/rattler_conda_types/src/match_spec/mod.rs)

Here are some examples:
```toml
[dependencies]
python = "3.*"
python = "3.7.*"
python = "3.7.10.*"
python = "3.8.2 h8356626_7_cpython"
python = ">3.8.2"
python = "<3.8.2"
python = ">=3.8.2"
python = "<=3.8.2"
python = ">=3.8,<3.9"
python = "3.11"
python = "3.10.9|3.11.*"
```

**Gotcha**: `python = "3"` resolves to `3.0.0` which is not a possible version.
To get the latest version of something always append with `.*` so that would be `python = "3.*"`

### Dependencies per platform

You can also specify a dependency specific to a certain platform:

```toml
[dependencies]
python = "3.11"

[target.osx-arm64.dependencies]
python = "3.10"
```

In the case above, we specify a specific dependency to OSX.
This overwrites the generic python dependency specified in the dependencies block.

### Resolution order
As a rule the target specific dependencies take precedence over generic ones
in the order that they are specified, so should multiple targets match the last specification is used.

## The `commands` part
In addition to managing dependencies, `pixi` aims to provide a user-friendly interface that simplifies the execution of repetitive, complex commands.
The commands section in your `pixi` configuration serves this purpose.
Expand All @@ -147,31 +196,3 @@ The `depends_on` will run the specified command in there to be run before the co
So in the example `build` will be run before `test`.
`depends_on` can be a string or a list of strings e.g.: `depends_on="build"` or `depends_on=["build", "anything"]`

## The `dependencies` part
As pixi is a package manager we obviously provide a way to specify dependencies.
Dependencies are specified using a "version" or "version range" which for Conda is a "MatchSpec"

This is a conda specific way to specify dependencies, to avoid failing to write a good explanation I'll link you to some excellent reads:
- [Conda build documentation](https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html#id6)
- [Excelent stackoverflow answer](https://stackoverflow.com/a/57734390/13258625)
- [Conda's python implementation](https://github.com/conda/conda/blob/main/conda/models/match_spec.py)
- [Rattler's rust implementation(ours)](https://github.com/mamba-org/rattler/blob/main/crates/rattler_conda_types/src/match_spec/mod.rs)

Here are some examples:
```toml
[dependencies]
python = "3.*"
python = "3.7.*"
python = "3.7.10.*"
python = "3.8.2 h8356626_7_cpython"
python = ">3.8.2"
python = "<3.8.2"
python = ">=3.8.2"
python = "<=3.8.2"
python = ">=3.8,<3.9"
python = "3.11"
python = "3.10.9|3.11.*"
```

**Gotcha**: `python = "3"` resolves to `3.0.0` which is not a possible version.
To get the latest version of something always append with `.*` so that would be `python = "3.*"`
2 changes: 1 addition & 1 deletion src/cli/add.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,14 +51,14 @@ pub async fn execute(args: Args) -> anyhow::Result<()> {
.collect::<anyhow::Result<HashMap<String, NamelessMatchSpec>>>()?;

// Get the current specs
let current_specs = project.dependencies()?;

// Fetch the repodata for the project
let sparse_repo_data = project.fetch_sparse_repodata().await?;

// Determine the best version per platform
let mut best_versions = HashMap::new();
for platform in project.platforms() {
let current_specs = project.dependencies(*platform)?;
// Solve the environment with the new specs added
let solved_versions = match determine_best_version(
&new_specs,
Expand Down
48 changes: 33 additions & 15 deletions src/environment.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
use crate::report_error::ReportError;
use crate::{
consts,
prefix::Prefix,
Expand All @@ -8,6 +9,7 @@ use crate::{
Project,
};
use anyhow::Context;
use ariadne::{ Label, Report, ReportKind, Source};
use futures::future::ready;
use futures::{stream, FutureExt, StreamExt, TryFutureExt, TryStreamExt};
use indicatif::ProgressBar;
Expand Down Expand Up @@ -41,7 +43,18 @@ pub async fn get_up_to_date_prefix(project: &Project) -> anyhow::Result<Prefix>
// Make sure the project supports the current platform
let platform = Platform::current();
if !project.platforms().contains(&platform) {
anyhow::bail!("the project is not configured for your current platform. Add '{}' to the 'platforms' key in project's {} to include it", platform, consts::PROJECT_MANIFEST)
let span = project.manifest.project.platforms.span();
let report = Report::build(ReportKind::Error, consts::PROJECT_MANIFEST, span.start)
.with_message("the project is not configured for your current platform")
.with_label(Label::new((consts::PROJECT_MANIFEST, span)).with_message(format!("add '{platform}' here")))
.with_help(format!("The project needs to be configured to support your platform ({platform})."))
.finish();

return Err(ReportError {
source: (consts::PROJECT_MANIFEST, Source::from(&project.source)),
report,
}
.into());
}

// Make sure the system requirements are met
Expand Down Expand Up @@ -126,11 +139,14 @@ pub fn lock_file_up_to_date(project: &Project, lock_file: &CondaLock) -> anyhow:
return Ok(false);
}

// Check if all dependencies exist in the lock-file.
let dependencies = project.dependencies()?.into_iter().collect::<VecDeque<_>>();

// For each platform,
for platform in platforms.iter().cloned() {
// Check if all dependencies exist in the lock-file.
let dependencies = project
.dependencies(platform)?
.into_iter()
.collect::<VecDeque<_>>();

// Construct a queue of dependencies that we wanna find in the lock file
let mut queue = dependencies.clone();

Expand Down Expand Up @@ -258,10 +274,6 @@ pub async fn update_lock_file(
repodata: Option<Vec<SparseRepoData>>,
) -> anyhow::Result<CondaLock> {
let platforms = project.platforms();
let dependencies = project.dependencies()?;

// Extract the package names from the dependencies
let package_names = dependencies.keys().collect_vec();

// Get the repodata for the project
let sparse_repo_data = if let Some(sparse_repo_data) = repodata {
Expand All @@ -276,14 +288,20 @@ pub async fn update_lock_file(
.iter()
.map(|channel| conda_lock::Channel::from(channel.base_url().to_string()));

let match_specs = dependencies
.iter()
.map(|(name, constraint)| MatchSpec::from_nameless(constraint.clone(), Some(name.clone())))
.collect_vec();

let mut builder =
LockFileBuilder::new(channels, platforms.iter().cloned(), match_specs.clone());
// Empty match-specs because these differ per platform
let mut builder = LockFileBuilder::new(channels, platforms.iter().cloned(), vec![]);
for platform in platforms.iter().cloned() {
let dependencies = project.dependencies(platform)?;
let match_specs = dependencies
.iter()
.map(|(name, constraint)| {
MatchSpec::from_nameless(constraint.clone(), Some(name.clone()))
})
.collect_vec();

// Extract the package names from the dependencies
let package_names = dependencies.keys().collect_vec();

// Get the repodata for the current platform and for NoArch
let platform_sparse_repo_data = sparse_repo_data.iter().filter(|sparse| {
sparse.subdir() == platform.as_str() || sparse.subdir() == Platform::NoArch.as_str()
Expand Down
8 changes: 7 additions & 1 deletion src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,20 @@ mod prefix;
mod progress;
mod project;
mod repodata;
mod report_error;
mod virtual_packages;

pub use project::Project;

#[tokio::main]
pub async fn main() {
if let Err(err) = cli::execute().await {
eprintln!("{}: {:?}", style("error").bold().red(), err);
match err.downcast::<report_error::ReportError>() {
Ok(report) => report.eprint(),
Err(err) => {
eprintln!("{}: {:?}", style("error").bold().red(), err);
}
}
std::process::exit(1);
}
}
Loading