Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add the build ID in TargetDescription #2644

Merged
merged 1 commit into from
Aug 19, 2023

Conversation

rtbo
Copy link
Contributor

@rtbo rtbo commented May 21, 2023

Allow 3rd party programs (e.g. Meson) to locate artifacts in the cache.
Following @Geod24 comment in #2642.

@WebFreak001 @Geod24 This would be enough for Meson to locate Dub's artifacts.
If you prefer to use this proposal, feel free to revert #2642.

@github-actions
Copy link

github-actions bot commented May 21, 2023

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 14

Total warnings: 0

Build statistics:

 statistics (-before, +after)
 executable size=5280256 bin/dub
-rough build time=77s
+rough build time=76s
Full build output
DUB version 1.32.1, built on Jul 23 2023
LDC - the LLVM D compiler (1.33.0):
  based on DMD v2.103.1 and LLVM 15.0.7
  built with LDC - the LLVM D compiler (1.33.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: icelake-server
  http://dlang.org - http://wiki.dlang.org/LDC

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    avr        - Atmel AVR Microcontroller
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    hexagon    - Hexagon
    lanai      - Lanai
    mips       - MIPS (32-bit big endian)
    mips64     - MIPS (64-bit big endian)
    mips64el   - MIPS (64-bit little endian)
    mipsel     - MIPS (32-bit little endian)
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc32le    - PowerPC 32 LE
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    riscv32    - 32-bit RISC-V
    riscv64    - 64-bit RISC-V
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    ve         - VE
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.34.0+commit.14.gf621c89c: building configuration [application]
source/dub/dependency.d(917,18): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals`
source/dub/dependency.d(920,30): Deprecation: scope variable `this` assigned to non-scope parameter `a` calling `doCmp`
source/dub/dependency.d(921,27): Deprecation: scope variable `this` assigned to non-scope parameter `b` calling `doCmp`
source/dub/dependency.d(939,26): Deprecation: scope variable `this` assigned to non-scope parameter `oth` calling `opEquals`
source/dub/internal/configy/Exceptions.d(130,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter
source/dub/internal/configy/Exceptions.d(134,34): Deprecation: reference to local variable `buffer` assigned to non-scope anonymous parameter
source/dub/internal/configy/Exceptions.d(248,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(250,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(283,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(286,27): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(323,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(325,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(332,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
source/dub/internal/configy/Exceptions.d(335,31): Deprecation: `@safe` function `formatMessage` calling `formattedWrite`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/write.d(537,34):        which calls `std.format.spec.FormatSpec!char.FormatSpec.writeUpToNextSpec!(void delegate(in char[]) @safe).writeUpToNextSpec`
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        which would be `@system` because of:
/opt/hostedtoolcache/dc/ldc2-1.33.0/x64/ldc2-1.33.0-linux-x86_64/bin/../import/std/format/spec.d(258,33):        scope variable `this` assigned to non-scope parameter `e` calling `put`
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5280256 bin/dub
STAT:rough build time=76s

@rtbo
Copy link
Contributor Author

rtbo commented May 21, 2023

Instead of cacheBuildId, I could use cacheArtifactPath with the full path to the built artifact in the cache.
It would avoid Meson to take any assumption about the internal cache structure of Dub
i.e.

$dubhome/cache/$pkg/$version[/+$subpkg]/build/$buildId/$targetFileName

Even with the build Id available, if this structure changes, Meson will be lost again.

@PetarKirov
Copy link
Member

A full path would be better, as Meson shouldn't depend on implementation details of Dub. As @Geod24 mentioned, dub describe is what other tools should use and if it doesn't contain all necessary information we should simply update it to fix that.

@rtbo rtbo marked this pull request as draft May 21, 2023 22:30
@rtbo
Copy link
Contributor Author

rtbo commented May 22, 2023

There seem to be a bug in ProjectGenerator.configurePackages that prevent this PR to work as intended.
For example, when I run dub describe eventcore, the target taggedalgebraic get versions from target eventcore, despite having no dependency towards eventcore:

				"versions": [
					"EventcoreEpollDriver",
					"Have_taggedalgebraic"
				],

However when I run dub build eventcore, taggedalgebraic is not built (because static libs) and when I run dub build taggedalgebraic, it is built without the EventcoreEpollDriver version.

It means that dub describe is currently describing targets that can't actually be built.

This come down to this piece of code, which looks wrong to me. I don't understand why dependencies need versions from their parents.

		// 1. downwards inherits versions, debugVersions, and inheritable build settings
		static void configureDependencies(const scope ref TargetInfo ti, TargetInfo[string] targets, size_t level = 0)
		{
			// do not use `visited` here as dependencies must inherit
			// configurations from *all* of their parents
			logDebug("%sConfigure dependencies of %s, deps:%(%s, %)", ' '.repeat(2 * level), ti.pack.name, ti.dependencies);
			foreach (depname; ti.dependencies)
			{
				auto pti = &targets[depname];
				mergeFromDependent(ti.buildSettings, pti.buildSettings);
				configureDependencies(*pti, targets, level + 1);
			}
		}

		configureDependencies(*roottarget, targets);

@WebFreak001
Copy link
Member

This would be enough for Meson to locate Dub's artifacts.
If you prefer to use this proposal, feel free to revert #2642.

actually I think having both would be beneficial, as dub describe will only tell you about a single configuration, but you might want to know what everything in the folder is.

@rtbo
Copy link
Contributor Author

rtbo commented May 25, 2023

When I completely remove the "downwards inherits versions etc." section from Dub's code, I still make valid builds.
Do you know the purpose of this (dependencies receiving versions from their parents) ?

Right now dub build and dub describe are not fully compatible with each other.

@rikkimax
Copy link
Contributor

Do you know the purpose of this (dependencies receiving versions from their parents) ?

It is used quite heavily, see bindbc projects that use it to specify the version of the library it needs to load.

Very useful.

@rtbo rtbo force-pushed the buildid_in_describe branch from 2092047 to d226782 Compare May 26, 2023 21:26
@rtbo rtbo marked this pull request as ready for review May 26, 2023 21:29
@rtbo
Copy link
Contributor Author

rtbo commented May 26, 2023

It is used quite heavily, see bindbc projects that use it to specify the version of the library it needs to load.

Ah yes, quite obvious.
Anyway this PR is ready. It will be fully useful if the --deep flag I've proposed is merged.

Allow 3rd party programs (e.g. Meson) to locate artifacts in the cache
@rtbo rtbo force-pushed the buildid_in_describe branch from 5c3268f to bbf33c0 Compare August 18, 2023 12:58
@rtbo
Copy link
Contributor Author

rtbo commented Aug 18, 2023

Hi, can this be merged before next Dub release?
The change seems to have consensus.
I'd like to use it to fix Meson support for Dub dependencies.

@WebFreak001 WebFreak001 merged commit fda37f3 into dlang:master Aug 19, 2023
@rtbo rtbo deleted the buildid_in_describe branch August 20, 2023 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants