Update for Version 0.2.0 #1

mettke · 2018-02-08T14:55:02Z

Changes

Added script for release builds and release profile
Added json and clap dependency
Added further package metadata
Added license and empty readme
Added format file
Added cli parsing
Added readelf stub
Added checks for required programs
Package verify is now ignoring libraries contained inside the package itself
Added data structure to separate input, processing, and output
Added json for output
Added nested parallel execution to include fetching files for packages
Listing libraries for files, files for libraries and packages for libraries
Added ability to check only selected packages
Added ability to ignore libraries

Usage

$ aurebuildcheck-rs
aurebuildcheck-rs 0.2.0
Matthias Krüger <matthias.krueger@famsik.de>, Marc Mettke <marc@itmettke.de>
Archlinux package checker to identify packages which may need a rebuild

USAGE:
    aurebuildcheck-rs [FLAGS] <SUBCOMMAND>

FLAGS:
        --group_by_containing_package    groups output by packages containg libraries
        --group_by_file                  groups output by files missing libraries
        --group_by_library               groups output by librarires required in files
    -h, --help                           Prints help information
    -j, --output_json                    Uses json for the list of missing libraries
    -q, --quite                          Hides all messages [aliases: s, silent]
    -c, --show_candidates                Prints a list of packages containing the missing library
    -V, --version                        Prints version information

SUBCOMMANDS:
    help       Prints this message or the help of the given subcommand(s)
    ldd        Checks packages using ldd
    readelf    Checks packages using readelf

$ aurebuildcheck-rs ldd --help
aurebuildcheck-rs-ldd
Checks packages using ldd

USAGE:
    aurebuildcheck-rs ldd [FLAGS] [OPTIONS] <packages>...

FLAGS:
    -a, --all_packages    Checks all installed packages marked as local
    -h, --help            Prints help information
    -V, --version         Prints version information

OPTIONS:
    -i, --ignore_libs <ignore libraries>...    List of libraries to ignore (eg lib1,lib2)

ARGS:
    <packages>...    List of packages to check (eg package1,package2)

Output

These results are rigged. Don't try to archive them yourself by running the binary, 'cause the missing libraries are included in the package itself and will be filtered out. I have removed that functionality for this output to be able to showcase something

$ aurebuildcheck-rs ldd android-studio
elf file "/opt/android-studio/jre/jre/lib/amd64/libattach.so" is missing:
        libjvm.so

(...)

library "libjvm.so" is required by:
        /opt/android-studio/jre/jre/lib/amd64/libattach.so
        /opt/android-studio/jre/jre/lib/amd64/libawt.so
        (...)

$ aurebuildcheck-rs -c ldd android-studio
elf file "/opt/android-studio/jre/jre/lib/amd64/libattach.so" is missing:
        libjvm.so

(...)

library "libjvm.so" is required by:
        /opt/android-studio/jre/jre/lib/amd64/libattach.so
        /opt/android-studio/jre/jre/lib/amd64/libawt.so
        (...)

library "libjvm.so" is packaged in:
        extra/jre7-openjdk-headless
        extra/jre8-openjdk-headless
        (...)

$ aurebuildcheck-rs -j ldd android-studio
[{"package_name":"android-studio","file_dependencies":[{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libattach.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libawt.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libawt_headless.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libawt_xawt.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libfontmanager.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libjava.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libjawt.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libjpeg.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/liblcms.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libmanagement.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libnet.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libnio.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libsctp.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libunpack.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libverify.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/jre/lib/amd64/libzip.so","library_dependencies":["libjvm.so"]},{"file_name":"/opt/android-studio/jre/lib/amd64/libjawt.so","library_dependencies":["libawt_xawt.so"]}],"library_requirements":[{"library_name":"libjvm.so","files_requiring":["/opt/android-studio/jre/jre/lib/amd64/libattach.so","/opt/android-studio/jre/jre/lib/amd64/libawt.so","/opt/android-studio/jre/jre/lib/amd64/libawt_headless.so","/opt/android-studio/jre/jre/lib/amd64/libawt_xawt.so","/opt/android-studio/jre/jre/lib/amd64/libfontmanager.so","/opt/android-studio/jre/jre/lib/amd64/libjava.so","/opt/android-studio/jre/jre/lib/amd64/libjawt.so","/opt/android-studio/jre/jre/lib/amd64/libjpeg.so","/opt/android-studio/jre/jre/lib/amd64/liblcms.so","/opt/android-studio/jre/jre/lib/amd64/libmanagement.so","/opt/android-studio/jre/jre/lib/amd64/libnet.so","/opt/android-studio/jre/jre/lib/amd64/libnio.so","/opt/android-studio/jre/jre/lib/amd64/libsctp.so","/opt/android-studio/jre/jre/lib/amd64/libunpack.so","/opt/android-studio/jre/jre/lib/amd64/libverify.so","/opt/android-studio/jre/jre/lib/amd64/libzip.so"]},{"library_name":"libawt_xawt.so","files_requiring":["/opt/android-studio/jre/lib/amd64/libjawt.so"]}]}]

TODO

Performance tests - was I able to completely ruin the performance with my changes?
Execution of "all packages" - I'm currently not able to try to run on all packages
Testing Error Handling
Implement readelf
Exit Code (non 0) if libraries are missing

Further changes

Do you think it would make sense to be able to take enviromental LD_LIBRARY_PATH into account?

I will look into it a bit more when I implement readelf. But sounds reasonable to add it

Would it make sense to ignore libraries on a larger scale via regex?
Something like --ignore "foo.so." or --ignore bar.so.3.

That is a great idea. I will definitely take a look at regex crates for rust

* Added script for release builds and release profile * Added json and clap dependency * Added further package metadata * Added license and empty readme * Added format file * Added cli parsing * Added readelf stub * Added checks for required programs * Package verify is now ignoring libraries contained inside the package itself * Added data structure to seperate input, processing and output * Added json for output * Added nested parallel execution to include fetching files for packages * Listing libraries for files, files for libraries and packages for libraries * Added ability to check only selected packages * Added ability to ignore libraries

* Added enum for differnt output types * Improved Error Handling at command execution * Replaced library vector with hashset for easier dedup * Added further verbose output * Restructured json output * moved check if library is ignored or in package from cmd.rs to process.rs

matthiaskrgr

Wow this is huge, thanks a lot!

I just quickly skipped through the code at this point, I will try this out later today. :)

matthiaskrgr · 2018-02-09T13:24:23Z

build_release.sh

@@ -0,0 +1,6 @@
+#!/bin/sh
+cargo rustc --release


I am curious, what is the difference between cargo build --release and cargo rustc --release?

There is none :D - I was trying out cargo rustc --release -- -C prefer-dynamic but realized that dynamic linking is not a good idea. Will change it back with the next commit

matthiaskrgr · 2018-02-09T13:28:04Z

Cargo.toml

+
+[profile.release]
+lto = true
+panic = 'abort'


I would like to keep codegen-units=1
It will slow down building in release mode a bit but it will speed up linking a lot and will allow for better optimizations.
rust-lang/rust#48025

I saw your commit for rpm - will do (it wasn't set on the arch branch yet)

Edit: done

Ah yeah. The machine I made the rust prototype on runs fedora so I just used rpm to be able to do some quick "does it run" tests. Looks like I forgot to merge the arch branch, but whatever.

matthiaskrgr · 2018-02-09T13:32:56Z

src/cli.rs

+        .arg(
+            Arg::with_name("group by library")
+                .long("group_by_library")
+                .help("groups output by librarires required in files"),


typo: librarires

matthiaskrgr · 2018-02-09T13:33:48Z

src/cli.rs

+        .arg(
+            Arg::with_name("group by containing package")
+                .long("group_by_containing_package")
+                .help("groups output by packages containg libraries"),


typo: containg

matthiaskrgr · 2018-02-09T13:38:16Z

src/cmd.rs

+    for line in output.lines() {
+        if line.ends_with("=> not found") {
+            let mut library_name = String::from(line.trim());
+            let new_length = library_name.len() - 13;


Is the -13 is length of "=> not found\n"?
I would like some kind of clarifying comment where the number is coming from.

I hope in the future we can just write "hello world".len() and it resolve that to integer at compile time.

Nevermind, it seems that "foo".len() works already! :D
https://godbolt.org/g/gvcivA

okay so I can instead use " => not found".len() (note the extra space so it gets to 13)

that's definitely better for clarity sake

matthiaskrgr · 2018-02-09T13:49:35Z

src/cmd.rs

+            dependency.library_dependencies.insert(library_name);
+        }
+    }
+    if dependency.library_dependencies.len() > 0 {


maybe !dependency.library_dependencies.is_empty() is bit more idiomatic, what do you think?

matthiaskrgr · 2018-02-09T13:49:53Z

src/cmd.rs

+pub fn verify_files_via_readelf<'a>(file: &str) -> Result<Option<FileDependency>, Error<'a>> {
+    let mut dependency = FileDependency::default();
+    dependency.file_name = String::from(file);
+    if dependency.library_dependencies.len() > 0 {


matthiaskrgr · 2018-02-09T13:51:41Z

src/main.rs

@@ -1,5 +1,13 @@
+//! A program for checking ArchLinux packages for missing libraries.
+//!
+//! If a package is missing a library it may means, that is necessary to rebuild that given package.


typo: it may means -> it may mean

* Changed build_release.sh from rustc to build * Added codegen-units=1 * Corrected a few linguistic mistakes * Small code readability fixes * Added non 0 exit code if a package is missing libraries

mettke · 2018-02-09T14:45:54Z

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Marc Mettke


Ups that must be corrected

matthiaskrgr · 2018-02-10T01:56:32Z

Hmm, is the "--all_packages" flag working?

I tried aurebuildcheck_rs ldd --all_packages but it seems it expects "aurebuildcheck-rs ldd ... --all_packages" which is strange.

matthiaskrgr · 2018-02-10T02:00:33Z

Nevermind, I see it (missing underscore):
You used Arg::with_name("all packages") but

.required_unless("all_packages")
 .conflicts_with("all_packages"),

matthiaskrgr · 2018-02-10T02:08:57Z

Porting to rpm? And maybe a way to identify whether to use pacman or rpm (merge both)

I think you can drop that. RPM was just me making sure the script works on a fedora machine, this was not intended for real world use case. :)

matthiaskrgr · 2018-02-10T02:11:17Z

I would like to keep the "just run it and it checks everything" functionality.
Could we make it so that if the program is launched with no args, one of ldd or readelf is picked and all foreign packages are checked by default?

mettke · 2018-02-10T05:15:11Z

I would like to keep the "just run it and it checks everything" functionality.
Could we make it so that if the program is launched with no args, one of ldd or readelf is picked and all foreign packages are checked by default?

Yes, no problem. Even tho I find it to be a bit counter-intuitive know. I thought about it for the bash script, 'cause it is required to be backward compatible but this is a new program with (maybe) new rules.

My problem with it is, that the user probably won't expect it to run all packages. Currently, he has to specify via subcommand whether he wants to use ldd and readelf (or maybe other in the future?). There is no indication that the program will select something for him if he doesn't. But I will leave it up to you, these are my proposal:

Way 1

The current way, the user has to specify exactly what he wants to do or nothing happens:

# Run all
aurebuildcheck_rs ldd -a
# Run for multiple packages
aurebuildcheck_rs ldd android-studio

Way 2

The alternative would be to always prefer something. If there is only the subcommand, it will run all packages automatically. If there is nothing it will run all packages with ldd automatically:

# Resolves to "aurebuildcheck_rs ldd -a"
aurebuildcheck_rs ldd
# Resolves to "aurebuildcheck_rs ldd -a"
aurebuildcheck_rs

mettke · 2018-02-10T05:17:48Z

Nevermind, I see it (missing underscore):

But did it work as expected?

# Conflicts: # Cargo.toml

mettke · 2018-02-10T05:41:14Z

Btw I have added an aur package for this binary. Currently, it points to my repo, but I will change that when this merge and the version tags are added. I can add you as Co-Maintainer if you want (just need your username)

https://aur.archlinux.org/packages/aurebuildcheck-rs-git/

mettke · 2018-02-10T06:42:57Z

I have pushed my changed to the arch branch and created a new PR. That way you will be able to add changes as well

Closing in favor of #2

matthiaskrgr and others added 2 commits February 5, 2018 17:46

rpm -> pacman

1838579

mettke mentioned this pull request Feb 8, 2018

Suggestion: Single package verification, lib ignore, readelf extension matthiaskrgr/aurebuildcheck#2

Closed

matthiaskrgr reviewed Feb 9, 2018

View reviewed changes

Small Corrections

ec90d62

* Changed build_release.sh from rustc to build * Added codegen-units=1 * Corrected a few linguistic mistakes * Small code readability fixes * Added non 0 exit code if a package is missing libraries

mettke commented Feb 9, 2018

View reviewed changes

LICENSE

@@ -0,0 +1,21 @@

MIT License

Copyright (c) 2017 Marc Mettke

Copy link

Collaborator Author

mettke Feb 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ups that must be corrected

mettke changed the base branch from arch to master February 10, 2018 05:16

Merge remote-tracking branch 'origin/master'

eae7e65

# Conflicts: # Cargo.toml

mettke closed this Feb 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update for Version 0.2.0 #1

Update for Version 0.2.0 #1

mettke commented Feb 8, 2018 •

edited

Loading

matthiaskrgr left a comment

matthiaskrgr Feb 9, 2018

mettke Feb 9, 2018

matthiaskrgr Feb 9, 2018

mettke Feb 9, 2018 •

edited

Loading

matthiaskrgr Feb 10, 2018

matthiaskrgr Feb 9, 2018

matthiaskrgr Feb 9, 2018

matthiaskrgr Feb 9, 2018

matthiaskrgr Feb 9, 2018

mettke Feb 9, 2018

matthiaskrgr Feb 9, 2018

mettke Feb 9, 2018

matthiaskrgr Feb 9, 2018

matthiaskrgr Feb 9, 2018

mettke Feb 9, 2018

matthiaskrgr commented Feb 10, 2018

matthiaskrgr commented Feb 10, 2018

matthiaskrgr commented Feb 10, 2018 •

edited

Loading

matthiaskrgr commented Feb 10, 2018

mettke commented Feb 10, 2018

mettke commented Feb 10, 2018

mettke commented Feb 10, 2018 •

edited

Loading

mettke commented Feb 10, 2018

Update for Version 0.2.0 #1

Update for Version 0.2.0 #1

Conversation

mettke commented Feb 8, 2018 • edited Loading

Changes

Usage

Output

TODO

Further changes

matthiaskrgr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mettke Feb 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthiaskrgr commented Feb 10, 2018

matthiaskrgr commented Feb 10, 2018

matthiaskrgr commented Feb 10, 2018 • edited Loading

matthiaskrgr commented Feb 10, 2018

mettke commented Feb 10, 2018

Way 1

Way 2

mettke commented Feb 10, 2018

mettke commented Feb 10, 2018 • edited Loading

mettke commented Feb 10, 2018

mettke commented Feb 8, 2018 •

edited

Loading

mettke Feb 9, 2018 •

edited

Loading

matthiaskrgr commented Feb 10, 2018 •

edited

Loading

mettke commented Feb 10, 2018 •

edited

Loading