Composer/Packagist: support platform compatibility during lookup

[rarkins]

As described in #1357 (comment), we need to support restricting updates according to the platform version, e.g. php <= 5.6.

Because our lookups are done independently of composer, we need to reproduce the logic that composer uses to determine if a newer version is compatible.

Steps:

1. When parsing the composer.json file, determine if it includes a php field under platform

2. Pass the php restriction to the packagist datasource lookup

3. Packagist datasource should return only compatible versions

[rarkins]

@swashata do you know if there’s a standardised field in packaging responses that indicates each version’s compatibility?

[swashata]

I have already checked and it seems there isn't a standardized field. Actually if we are to publish a package that requires some particular version of PHP, we put it under require field, just like dependencies.

A good example is this https://packagist.org/packages/paragonie/random_compat.json For different versions of the package, there are "php": ">=5.2.0" under require. If it is not there, in my experience, composer just thinks the version fulfills all platform requirements.

I will do some more digging and will let you know.

[swashata]

Also do note that when defining platform (in the app), we don't specify PHP versions like "php": "<=5.6". Rather we just specify the exact version.

{
  "config": {
    "platform": {
      "php": "5.6.0"
    }
  }
}

It is upto the libraries that app uses to make sure they "require" the PHP version (here one can specify range) they need.

[rarkins]

https://stackoverflow.com/questions/26277151/force-composer-to-require-php-version-between-version-x-and-version-y

[darthf1]

In my composer.json, i have the following:

{
    "require": {
        "php": "7.3"
    }
}

Maybe this can be used?

[nonsenz]

hi,
i am wondering if there is any idea how and when to address this? we would really love to use renovate, but i can't see how it is usable in a php project if composer.json platform is ignored and --ignore-platform-reqs is activated? if somebody uses renovate in a php project: how are you doing this?

[bytestream]

@rarkins what are you willing to accept to see this fixed?

This is causing me an issue where:

My composer:

"require": {"endroid/qr-code": "^3.7"},
"config": { "platform": { "php": "7.2.0" } },

Endroid composer:

"require": {"symfony/property-access": "^3.4||^4.0||^5.0"}

The Pin Dependencies PR pins endroid/qr-code to 3.7.7, but then updates symfony/property-access from 4.4.5 to 5.0.7 which later results in:

symfony/property-access v5.0.7 requires php ^7.2.5 -> your PHP version (7.3.16) overridden by "config.platform.php" version (7.2.0) does not satisfy that requirement.

I'm not sure why renovate is updating symfony/property-access? I thought I could bypass the issue by running npm run renovate inside a docker container that's running PHP 7.2.0 but that hasn't seemed to help at all

[bytestream]

Renovate 192.205.0 now has the option to specify composerIgnorePlatformReqs: false (#5937)

The lack of config.platform support can now be bypassed by building your own Docker image that contains your minimum PHP requirements and setting the above config option.

[bytestream]

@rarkins I think this should be simpler than expected to implement.

1. When parsing the composer.json file, look for config.platform and extract all values (it's an object of {"name": "version", "name": "version"} - see https://getcomposer.org/doc/01-basic-usage.md#platform-packages
2. Before running composer update / composer install loop through extracted platform packages and run composer config platform.<name><value> - see https://getcomposer.org/doc/03-cli.md#config

Example commands:
composer config platform.php 7.2.0
composer config platform.ext-gd 2.1.0

If someone points me in the right direction then I can submit a PR but this feels like it's achievable very quickly.

At a complete guess (I don't know TypeScript) so it will take me significantly longer than someone who does:

    // extract.ts
   if (composerJson.platform) {
     res.compatibility = composerJson.platform;
   }

    // artifacts.ts
    if (config.compatibility) {
      for (const [depName, version] of Object.entries(config.compatibility)) {
        args = 'config platform.' + depName + ' ' + version;
        
        logger.debug({ cmd, args }, 'composer command');
        await exec(`${cmd} ${args}`, execOptions);
      }
    }

[viceice]

@bytestream Why do we need to set the config options to composer? I think extracting the compatibility for php would be enough, so we can use the right php version.

Does composer have problems with different php patch releases, because our php docker images currently only allows to build latest patch for supported php minor versions. we are using https://deb.sury.org/ as php source

[bytestream]

@viceice

Why do we need to set the config options to composer?

I don't fully understand how renovate bot works but it ignores the config.platform.php defintion in composer.json. #2355 (comment) mentioned that it should be passed to packagist but then seemed to run into a road block as the API doesn't have that kind of support.

composer config platform.php 7.2.0 is equalivent to it being set in composer.json. It's also equivalent to running php7.2.0 composer.phar install (running composer under PHP 7.2.0), which is what I suggested #2355 (comment). That isn't sustainable though if you have different platform constraints in different repositories...

Does composer have problems with different php patch releases

No.

I don't fully understand what you mean with this question.

[viceice]

@bytestream My questions was simply can we use php 7.2.6 if the composer platform config says 7.2.3? So using a higher php patch release to update the lockfile.

[bytestream]

Yes, anything greater than the specified platform version. Sent via mobile. Please excuse the spelling, brevity and punctuation.

…

On Tue, 9 Jun 2020, 17:03 Michael Kriese, ***@***.***> wrote: @bytestream <https://github.com/bytestream> My questions was simply can we use php 7.2.6 if the composer platform config says 7.2.3? So using a higher php patch release to update the lockfile. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2355 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANUT3J3LLK7AZDHK25OE4LRVZMLLANCNFSM4FOIKJXQ> .

[bytestream]

So how do you think this should be implemented @viceice ? I'm keen to get movement because my previous fix is not feasible when using renovate bot on multiple repositories with different platform requirements.

[viceice]

first we need to add before here:

renovate/lib/manager/composer/extract.ts

Line 185 in aed8d75

return res;

    if (composerJson.platform) {
      res.compatibility = composerJson.platform;
    }

then we need to add tagConstraint and tagScheme to here:

renovate/lib/manager/composer/artifacts.ts

Lines 112 to 114 in aed8d75

	docker: {
	image: 'renovate/composer',
	},

image needs to be changed to renovate/php
preCommands needs to contain dynamic composer install (for this to work we maybe need to add the installer scipt to the php image before.

like in bundler

renovate/lib/manager/bundler/artifacts.ts

Lines 134 to 138 in aed8d75

	docker: {
	image: 'renovate/ruby',
	tagScheme: 'ruby',
	tagConstraint: await getRubyConstraint(updateArtifact),
	preCommands,

[internalsystemerror]

Thanks for the merge! In order to ensure that renovate runs using a specific PHP version, you have several options:

• it will use the highest available from require.php in composer.json
• or it can be overriden using config.platform.php in composer.json
• or it can be overriden using contraints.php in renovate.json

We are currently using the second option to specify a specific version as we use a wide constrain for the first. The PR referenced will ensure that updates to the require.php in composer.json are proposed as with other updates.

[WyriHaximus]

Thanks for the merge! In order to ensure that renovate runs using a specific PHP version, you have several options:

• it will use the highest available from require.php in composer.json
• or it can be overriden using config.platform.php in composer.json
• or it can be overriden using contraints.php in renovate.json

We are currently using the second option to specify a specific version as we use a wide constrain for the first. The PR referenced will ensure that updates to the require.php in composer.json are proposed as with other updates.

How are you using the second? Because currently I'm attempting to do that and it still seems to take the highest available version from the first when resolving the possible package version for a bump.

[internalsystemerror]

How are you using the second? Because currently I'm attempting to do that and it still seems to take the highest available version from the first when resolving the possible package version for a bump.

Are you specifying the full version?

Here is an example of how we're using it: https://github.com/laminas/laminas-servicemanager/blob/f22969b8b0ea2d695aa832c8895be29622b9bd94/composer.json#L25

"config": {
    "platform": {
        "php": "8.0.99"
    }
}

This is treated by packagist as essentially 8.0.* as it is sent over as is. Renovate similarly only uses the 8.0 part when looking up the renovate-php docker tag. If you intend to use this organisation wide, and don't want to update all your composer.json files then I would recommend the 3rd option, setting constraints.php in a shared renovate.json. We couldn't as each repo has its own PHP requirements.

[WyriHaximus]

How are you using the second? Because currently I'm attempting to do that and it still seems to take the highest available version from the first when resolving the possible package version for a bump.

Are you specifying the full version?

Yup using the full version, and got it in every composer.json already. Thank you for answering, this confirms to me I have that set up right. Wanted to make sure I hadn't miss anything. But we're working on parts of the things I've ran into over at #18715

[github-actions]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[AlexSkrypnyk]

After reading several issues and trying different variants, I'm leaving the below comment as a summary of my findings.

As of https://github.com/renovatebot/renovate/releases/tag/34.154.4, this is how it works for Drupal projects.

About platform version constraints

RenovateBot assesses 3 places to determine the PHP platform requirements:

1. The require.php version value in composer.json.
2. The config.platform version value in composer.json.
3. The constraints.php version value in renovate.json.

The purpose of require.php is to set the minimum PHP language requirements
for a package. For example, the minimum version required for Drupal 10 is 8.0.2
or above, which can be specified as >=8.

The purpose of config.platform is to set the PHP language requirements for the
specific instance of the package running in the current environment. For
example, while the minimum version required for Drupal 10 is 8.0.2 or above, the
actual PHP version on the hosting provider could be 8.1.0. The value of this
field should provide your exact version of PHP with all 3 parts of the version.

The purpose of constraints.php is to set the PHP language requirements for the
RenovateBot run, in case the above two values are not set or need to be
overridden.

Which versions to specify in a Drupal site?

It is recommended to specify require.php as a range constraint (e.g. >=8.1)
to ensure it is used as the minimum requirement for project dependencies.

It is also recommended to specify config.platform as a specific version (e.g.
8.1.19) constraint to ensure that only the package versions supported by your
current environment are used.

Once the PHP version of your environment is updated, the config.platform
value should also be updated to the specific version.

It is not recommended to use constraints.php in renovate.json, as every
project may use a different version of PHP. This value should only be used if
your Drupal project does not have require.php or config.platform provided
(in which case those values should be provided instead).

Caveats

If config.platform does not have a value with all 3 parts of the version (e.g., 8.1), renovate will choke on Command failed: install-tool php 8.1. Composer does not validate config.platform version validity to have all 3 components and Renovate does not convert 2-component version to a 3-component version (I'm not sure if it should or not).

Working example

I have setup a Drupal-related repo (and been supporting it) to provide zero-config Renovate setup for Drupal projects (GitHub app and self-hosted in GitHub and CircleCI) https://github.com/drevops/renovate-drupal-example

[rarkins]

Thanks @AlexSkrypnyk for this great summary. Could you create a bug report issue with reproduction repo for the 2-part config.platform value? We should fix it.

Also, does 8.1 in config.php mean 8.1.0, or 8.1.*?

[Wirone]

I believe 8.1 could mean 8.1.* since patch versions don't introduce breaking changes (while minor versions do).

[herndlm]

Be careful, the docs seem to differ:
https://getcomposer.org/doc/06-config.md#platform

This also means if {"php":"7.4"} is specified; no packages will be used that define 7.4.1 as minimum.

[rarkins]

Yes, that would imply that 7.4 should be treated as 7.4.0

[jayay]

The purpose of require.php is to set the minimum PHP language requirements
for a package.

However, the highest possible PHP version from require.php is being taken. For example, if my composer.json specifies

"php": ">=7.2 <8.2.0"

I obviously intend to have my project being able to run on PHP 7.2, but Renovate attempts to update all dependencies to versions only compatible with PHP 8.1. An example is phpunit from version 8 to 10.

I tried different notations and combinations, such as ^7.2 || ^8.1 (which leaves out 8.0), but with the same result. Specifying an exact version seems to work, but is not an option for many use cases.

Is this intended behavior? If so, why? It really doesn't make any sense to me.

[Wirone]

@jayay did you try ^7.2 || ^8.0 <8.2.0?

[rarkins]

I'm going to close this as completed. For further discussion, please create a Discussion, not Issue, and ideally include a reproduction repo.