Provide 'rename' capability for installing executables and libraries

[keith-packard]

I'm building a multilib library, which means the library is built multiple times, once per multilib target. For installation, each library ends up in a target-specific directory using their 'plain' name. For compilation, I'm creating per-target names for the library. That means the compiled library file name doesn't match the install library file name, so I'm hoping to have something like the 'rename' option that other install methods support to let me set the install filename for each target to match the 'plain' name.

I actually have a kludge which is working for now -- the per-target name I'm using has the target as a directory prefix, which makes the install 'work', but causes meson to emit dire warnings about abuse of the system.

Here's the system I'm creating, if you would like to see what it does.

https://salsa.debian.org/electronics-team/newlib-nano

[nirbheek]

Renaming of build targets can't be done reliably for many reasons. For instance, the soname would be wrong, on macOS the install_name will be wrong, and on Windows the import library embeds the name of the DLL, and so on. It should be fine for static libraries, though.

We should try to find some other way to solve your problem, because it seems to be a workflow that should work for all targets.

I looked at your build files, and it seems the list of multilib targets is dynamic, and multiple library targets are defined with different c_args: to get different outputs.

This means the library targets must be placed in subdirs dynamically, which is what happens when you define static_library('x86/libc', ...), but this is bad because it breaks the layout contract (which is why it is deprecated). I noticed that you only use this trick for libraries that are going to be installed, which means you only need this a few times, and that gives me an idea.

subdir() only requires the meson.build file to exist when it is called. Instead of defining a target called x86/foo, you could have a python script that takes a subdir name and a string, and creates a meson.build file in that subdir of the current source dir with the string as the contents. Then, call that script with the args ['x86', 'static_library('foo', ...)'] and call subdir('x86').

Alternatively you could do multilib at a higher level, by only building for one target in the build files, and use separate build directories for each target. The configuration time should be trivial, and since you can't share object files between multilib targets anyway, it's not any less efficient.

PS: Instead of passing pic: false to each target, you can also set b_staticpic=false.
in default_options: in project().

[keith-packard]

Thanks for taking a look at my little issue.

I did start by wrapping meson in a script that generated separate build directories; that 'works', but it's quite clunky to use; getting debian packaging wrapped around that would require a fairly sophisticated script, which starts to feel like autotools all over again. Having all of the magic contained in meson configuration files seems so clean and it very nearly works already (well, actually it does work, except for the dire warnings).

As for using different temporary names; we have both names available at link time, so (at least on linux), we could pass -soname= so that the file contained the correct information for run-time. I wonder if there are similar flags for OS X and Windows?

Oh, I tried adding 'b_staticpic=false' to my project default_options and it didn't work -- the compiler was still getting -fPIC on the command line.

[nirbheek]

As for using different temporary names; we have both names available at link time, so (at least on linux), we could pass -soname= so that the file contained the correct information for run-time. I wonder if there are similar flags for OS X and Windows?

The problem happens when other things in your project link to that library, then you need to rewrite the DT_NAME/install_name and RPATHs for everything (on Linux/BSD/macOS), and afaik there's no way to change the contents of an implib, but we could write a manual rewriter. So overall, it's all doable, but it's a question of whether it's worth the greatly increased complexity.

Thinking about this more, I think the general case for this is to build the same target multiple times with different c_args: but with the same output file name, and install them into separate dirs. This is already sort of needed for Windows, and we could implement both with the same infrastructure.

See: #3304 and #3819. The original idea was to be able to be able to map each library_type: (static/shared) to different c_args:, but that can be generalized to a 'build target specification' which would let you specify multiple different c_args: + install_dir: pairs for the same library type, which would create multiple versions of the same library and install each into the specified install dirs.

Build the same library as static and shared with different c_args::

tspec = {
  'static': {'c_args': ['-DFOO_STATIC'], 'target_type': 'static_library'},
  'shared': {'c_args': [], 'target_type': 'shared_library'},
}
library('foo', sources, target_spec: tspec)

Build multiple static libraries for different multilib archs

tspec = {
  '32' : {'c_args': ['-m32'], 'install_dir': 'lib32'},
  '64' : {'c_args': ['-m64'], 'install_dir': 'lib64'},
  'x32' : {'c_args': ['-blah'], 'install_dir': 'lib'},
}
static_library('foo', sources, target_spec: tspec)

tspec would be able to contain all kwargs that library() or static_library() can accept. The key used for the target spec must be a unique key for that target, and will be used as part of the internal directory structure in the builddir (inside the private dir, which is not part of the layout contract).

Thinking about the implementation details (resolving library paths, RPATHs, etc) it all seems easy to implement. The return value of *library() would also be a dict in this case, with the same keys as the build specification. Same for other build targets.

As plus, this resolves the question we've had about how to pass kwargs to functions since it doesn't require new syntax in the parser.

Oh, I tried adding 'b_staticpic=false' to my project default_options and it didn't work -- the compiler was still getting -fPIC on the command line.

I will investigate this. This use-case was exactly why that builtin-option was added and we have tests for this.

[jpakkane]

Trying to build two architectures (-m32 and -m64) at the same time in the same project is fraught with danger. Usually it makes more sense to have two build directories, one set up to 32 bit and one to 64 bit and compile and install them both separately.

[keith-packard]

Agreed that it's generally better to use separate build directories, but in this case I'm actually detecting which multilib configurations are available right in the build system; I don't know beforehand what those are. The existing autotools build system for this project works this way; I'd like to replicate that within meson rather than needing external kludges, of course.

[nirbheek]

Thinking more, I realized that another reason for having this mechanism to build the same library twice with different c_args: is the static_c_args: feature needed for Windows. The mechanism I listed above (different c_args: for shared and static) works, but it breaks down when you need to link a helper static library that exports symbols into the main library. In that case you will need to build that helper static library with and without the -DFOO_STATIC argument.

So I think we need to implement a mechanism that allows this anyway. Then, whether or not people decide to take the risk of passing 'interesting' compiler args like -m32 and -m64 which will break if you use dependencies: is up to them.

[xclaesse]

I really like this target_spec proposal, but we need to make sure we think about every cases.

Example:

tspec = {
  '32' : {'c_args': ['-m32'], 'install_dir': 'lib32'},
  '64' : {'c_args': ['-m64'], 'install_dir': 'lib64'},
  'x32' : {'c_args': ['-blah'], 'install_dir': 'lib'},
}
lib = static_library('foo', sources, target_spec: tspec)

I guess lib would be a dict:

{
  '32' : StaticLibraryHolder,
  '64' : StaticLibraryHolder,
  'x32' : StaticLibraryHolder,
}

I don't think we should allow doing exe = executable('app', app_sources, link_with : lib) that would make stuff too complicated. But we could allow:

exe_tspec = {
  '32' : {'c_args': ['-m32'], link_with : lib['32']},
  '64' : {'c_args': ['-m64'], link_with : lib['64']},
  'x32' : {'c_args': ['-blah'], , link_with : lib['x32']},
}
exe = executable('app', app_sources, target_spec : exe_tspec)

exe would also be a dict, and we could do run_command(exe['32'], ...).

Now how would it work with both_librariens() or library() and default_library=both

tspec = {
  '32' : {'c_args': ['-m32'], 'install_dir': 'lib32'},
  '64' : {'c_args': ['-m64'], 'install_dir': 'lib64'},
  'x32' : {'c_args': ['-blah'], 'install_dir': 'lib'},
}
lib = both_libraries('foo', sources, target_spec: tspec)

Following the same logic, lib would be a dict:

{
  '32' : BothLibrariesHolder,
  '64' : BothLibrariesHolder,
  'x32' : BothLibrariesHolder,
}

And if we want different c_args for static and shared, we could do like that:

tspec = {
  '32' : [{'c_args': ['-m32', '-DSTATIC_COMPILATION'], 'install_dir': 'lib32', target_type : 'static_library'},
           {'c_args': ['-m32'], 'install_dir': 'lib32', target_type : 'shared_library'}]
  '64' : {'c_args': ['-m64'], 'install_dir': 'lib64'},
  'x32' : {'c_args': ['-blah'], 'install_dir': 'lib'},
}
lib = both_libraries('foo', sources, target_spec: tspec)

So if the value in tspec dict is a list, it picks the one that has target_type that match the target it currently build. Otherwise it applies to both libraries. But target_type values should be unique in that list, so that's more a dict case. So what about this tspec:

tspec = {
  '32' : {
    'static_library' : {'c_args': ['-m32', '-DSTATIC_COMPILATION'], 'install_dir': 'lib32'},
    'shared_library' : {'c_args': ['-m32'], 'install_dir': 'lib32'},
  }
  '64' : {
    'static_library' : {'c_args': ['-m64', '-DSTATIC_COMPILATION'], 'install_dir': 'lib64'},
    'shared_library' : {'c_args': ['-m64'], 'install_dir': 'lib64'},
  }
  'x32' : {
    'static_library' : {'c_args': [''-blah'', '-DSTATIC_COMPILATION'], 'install_dir': 'lib'},
    'shared_library' : {'c_args': [''-blah''], 'install_dir': 'lib'},
  }
}

It's getting complicated structure, but it covers all cases. Also it means if we want different c_args for shared and static, we are forced to have library() returning a dict, which means modifications everywhere in existing projects. I personally think we need also that kind of similar syntax:

tspec = {
  'static_library' : {'c_args': ['-DSTATIC_COMPILATION']},
}
lib = both_libraries('foo', sources, target_spec : tspec)

Where lib would be a BothLibrariesHolder and not a dict.

[xclaesse]

To summary, I see 3 different formats possible for tspec, and possibly a mix of them:

tspec1 = {'<custom key>' : {'<target type>' : {'<kwarg>' : <value>, ...}, ...}, ...}
tspec2 = {'<custom key>' : {'<kwarg>' : <value>, ...}, ...}
tspec3 = {'<target type>' : {'<kwarg>' : <value>, ...}, ...}

We could probably check the values of keys in the dict to determine which case we have, but stuff are getting a bit 'magical' IMHO.

[xclaesse]

otoh, we are not force to fit everything in a single dict. It would make a lot of sense to split this into 2 separate features:

multilib_args = {
  '32' : {'c_args': ['-m32'], 'install_dir': 'lib32', name_prefix : 'lib1'},
  '64' : {'c_args': ['-m64'], 'install_dir': 'lib64'},
  'x32' : {'c_args': ['-blah'], 'install_dir': 'lib'},
}
tspec = {
  'static_library' : {'c_args': ['-DSTATIC_COMPILATION'], name_prefix : 'lib2'},
}
lib = library('foo', sources, multilib : multilib_args, target_spec : tspec, c_args : ['-DFOO'], name_prefix : 'lib3')

lib would be a dict only of multilib kwarg is present, but not if only target_spec is specified. We still need to define in which order args are overriden, or appended. In the above example what would be the value of c_args and name_prefix for a 32bits static library?

[rilian-la-te]

Is tspec approach will allow us to build a same library with different dependencies, but for same arch?

[jpakkane]

I don't like the fact that we are putting many different target specifications into one target invocation. It is not readable and very confusing. For example it is hard to tell which compiler flags are in effect, basically you need to replicate the detection logic inside your head.

I also particularly don't like the fact that the output of library changes. Thus far we have had an extremely straightforward model in Meson: one target produces one and exactly one output. This is already streched a bit by both_libraries but it's still manageable. Returning dicts makes things harder to implement, reason about and generally just plain confusing.

[xclaesse]

@jpakkane I expected that answer, and I mostly agree tbh. I think the multilib part is far fetched, that's why in my last comment, I made a proposal that split multilib: and target_spec: into separate kwargs.

This bug here is about the multilib part, the most controversial because it would require library() to return a dict.

Bugs #3304 and #4133 are about passing different args for static and shared libraries when using library() and I think it's less controversial to add a kwarg like this one for that:

tspec = {
  'static_library' : {'c_args': ['-DSTATIC_COMPILATION'], name_suffix : 'lib'},
}

But that's orthogonal to this bug, since I think we shouldn't have one kwarg for both use cases.

[xclaesse]

Back to initial problem; multilib can be implemented like that, a bit verbose but not that bad IMHO:

multilib_spec = {
  '32' : {'c_args': ['-m32'], 'install_dir': 'lib32'},
  '64' : {'c_args': ['-m64'], 'install_dir': 'lib64'},
  'x32' : {'c_args': ['-blah'], 'install_dir': 'lib'},
}
lib = {}
foreach arch, spec : multilib_spec
  lib += {arch : library('foo', sources, c_args : spec['c_args'], install_dir : spec['install_dir'])}
endforeach

The only problem is it's going to create a target id collision. To fix that collision, Target.get_id() needs to somehow take into account the install_dir kwarg to create a unique id. We can use a small hash of that string. Target.get_id() would return something like <subdir>@@foo@e22b69bd@sta

Or we could add a new kwarg to let user explicitly uniquify the id, the above example could become: lib += {arch : library('foo', sources, c_args : spec['c_args'], install_dir : spec['install_dir'], uniquify_id : arch)} and Target.get_id() would return something like <subdir>@@foo@32@sta.

[mstorsjo]

(I removed a comment I accidentally posted on the wrong thread.)

[le-jzr]

I need the 'rename' capability as well, but for a very different reason.
The project I'm trying to convert consists of many different executable and library subdirectories, all of which share some extra logic. Duplicating the logic in every individual bottom-level meson file would be insane, as right now most of those consist of a single line file list.

Since meson doesn't support user-defined subroutines, the only sensible solution is to have the executable() and library() calls in a shared parent meson script. Currently, I'm including the path into the target name (which emits a lot of warnings, but works). Ideally, I could either put the shared meson code in a named subroutine, or I could underscorify the binary names and then rename them on install. As it is, I can do neither.

[dcbaker]

I actually ran into a use for this as well (I think my use is closer to @le-jzr's), in xts, tests are installed in a nested structure, with many subdirectories containing binaries that are invoked as subprocesses, and have overlapping names (extensions added for clarity):

feature1/
  feature1.exe
  tests/
      test1.exe
      test2.exe
feature2/
   feature2.exe
   tests/
       test1.exe
       test2.exe

Due to our target name duplication restriction I have to have an install script that renames them, if I could simply have something like:

executable(
  'feature-test1',
  ...
   install_as : 'test1',
   install_dir : unique_dir_path,
)

That would be convenient.

[danielcjacobs]

I also would appreciate this feature for another reason.

I'm trying to use a cmake project as a subproject, but this project uses LIBRARY_OUTPUT_NAME, for which there is no equivalent meson flag. This means I can't use this subproject as a fallback for a dependency (if the subproject was installed to the system via cmake) without some weird hackiness.