Building over existing repositories #159
Conversation
|
@jendo42 after merging the PR from PS2SDK, the |
|
I understand what you are trying to do here (and this is something that I wanted to improve too but I haven't dedicated time to it yet), but we need to find another way of doing this. I understand that the main motivation of this PR is to avoid removing and fetching again all repos when you run it locally, however with the implementation you have provided the first time we are cloning the whole repo, and then checkout to a concrete sha. You can see other places in Cheers! |
There was a problem hiding this comment.
You can check times here:
Currently: 8m:30seg
https://github.com/ps2dev/ps2sdk-ports/actions/runs/9807369913
With your PR: 10m:32seg
https://github.com/ps2dev/ps2sdk-ports/actions/runs/9798432303?pr=159
|
Ah, yes I see. Firstly I had this with --depth=1 but there was at least one dependency that relays on the |
jendo42
force-pushed
the
devel
branch
3 times, most recently
from
19cdf85
to
6694b65
Compare
|
Good improvements! Cheers |
Signed-off-by: Michal Jenikovsky <jendo@jmsystems.sk>
|
|
||
| ## | ||
| ## Build cmake projects | ||
| ## | ||
| PROC_NR=1 build zlib -DUNIX:BOOL=ON # Forcing to compile with -j1 because there is a race condition in zlib | ||
| build zlib -DUNIX:BOOL=ON -DZLIB_BUILD_EXAMPLES=OFF |
There was a problem hiding this comment.
I'm not sure about this change.....
Even in PSP they required to use that trick with the latest available zlib version.
https://github.com/pspdev/psp-packages/blob/master/zlib/PSPBUILD#L25
There was a problem hiding this comment.
This is my output, with -j 16 but I have active ccache.
[ 93%] Building C object CMakeFiles/zlibstatic.dir/zutil.c.o
[100%] Linking C static library libz.a
[100%] Linking C static library libz.a
[100%] Built target zlib
[100%] Built target zlibstatic
Without ccache and with examples turned on:
Scanning dependencies of target zlibstatic
Scanning dependencies of target zlib
[ 5%] Building C object CMakeFiles/zlibstatic.dir/adler32.c.o
[ 5%] Building C object CMakeFiles/zlib.dir/adler32.c.o
[ 19%] Building C object CMakeFiles/zlibstatic.dir/deflate.c.o
[ 19%] Building C object CMakeFiles/zlib.dir/crc32.c.o
[ 19%] Building C object CMakeFiles/zlib.dir/compress.c.o
[ 19%] Building C object CMakeFiles/zlibstatic.dir/crc32.c.o
[ 19%] Building C object CMakeFiles/zlibstatic.dir/compress.c.o
[ 44%] Building C object CMakeFiles/zlibstatic.dir/gzclose.c.o
[ 44%] Building C object CMakeFiles/zlib.dir/gzwrite.c.o
[ 44%] Building C object CMakeFiles/zlib.dir/gzread.c.o
[ 44%] Building C object CMakeFiles/zlib.dir/gzlib.c.o
[ 44%] Building C object CMakeFiles/zlib.dir/deflate.c.o
[ 44%] Building C object CMakeFiles/zlibstatic.dir/gzwrite.c.o
[ 44%] Building C object CMakeFiles/zlib.dir/gzclose.c.o
[ 44%] Building C object CMakeFiles/zlibstatic.dir/gzlib.c.o
[ 44%] Building C object CMakeFiles/zlibstatic.dir/gzread.c.o
[ 50%] Building C object CMakeFiles/zlibstatic.dir/inflate.c.o
[ 50%] Building C object CMakeFiles/zlib.dir/inflate.c.o
[ 55%] Building C object CMakeFiles/zlib.dir/infback.c.o
[ 55%] Building C object CMakeFiles/zlibstatic.dir/infback.c.o
[ 61%] Building C object CMakeFiles/zlibstatic.dir/inftrees.c.o
[ 61%] Building C object CMakeFiles/zlib.dir/inftrees.c.o
[ 66%] Building C object CMakeFiles/zlibstatic.dir/inffast.c.o
[ 66%] Building C object CMakeFiles/zlib.dir/inffast.c.o
[ 75%] Building C object CMakeFiles/zlibstatic.dir/trees.c.o
[ 83%] Building C object CMakeFiles/zlib.dir/uncompr.c.o
[ 83%] Building C object CMakeFiles/zlib.dir/trees.c.o
[ 83%] Building C object CMakeFiles/zlibstatic.dir/uncompr.c.o
[ 83%] Building C object CMakeFiles/zlib.dir/zutil.c.o
[ 83%] Building C object CMakeFiles/zlibstatic.dir/zutil.c.o
[ 86%] Linking C static library libz.a
[ 88%] Linking C static library libz.a
[ 88%] Built target zlib
[ 88%] Built target zlibstatic
Scanning dependencies of target minigzip
Scanning dependencies of target example
[ 94%] Building C object CMakeFiles/example.dir/test/example.c.o
[ 94%] Building C object CMakeFiles/minigzip.dir/test/minigzip.c.o
[100%] Linking C executable minigzip
[100%] Linking C executable example
[100%] Built target example
[100%] Built target minigzip
Install the project...
Even I downgraded version to v1.2.12:
Scanning dependencies of target zlibstatic
Scanning dependencies of target zlib
[ 5%] Building C object CMakeFiles/zlibstatic.dir/adler32.o
[ 5%] Building C object CMakeFiles/zlibstatic.dir/compress.o
[ 11%] Building C object CMakeFiles/zlibstatic.dir/crc32.o
[ 11%] Building C object CMakeFiles/zlibstatic.dir/deflate.o
[ 13%] Building C object CMakeFiles/zlibstatic.dir/gzclose.o
[ 16%] Building C object CMakeFiles/zlibstatic.dir/gzread.o
[ 22%] Building C object CMakeFiles/zlibstatic.dir/gzwrite.o
[ 22%] Building C object CMakeFiles/zlibstatic.dir/gzlib.o
[ 33%] Building C object CMakeFiles/zlib.dir/compress.o
[ 38%] Building C object CMakeFiles/zlib.dir/deflate.o
[ 44%] Building C object CMakeFiles/zlib.dir/adler32.o
[ 44%] Building C object CMakeFiles/zlib.dir/crc32.o
[ 44%] Building C object CMakeFiles/zlib.dir/gzclose.o
[ 44%] Building C object CMakeFiles/zlibstatic.dir/inflate.o
[ 44%] Building C object CMakeFiles/zlib.dir/gzlib.o
[ 44%] Building C object CMakeFiles/zlib.dir/gzread.o
[ 50%] Building C object CMakeFiles/zlib.dir/gzwrite.o
[ 50%] Building C object CMakeFiles/zlibstatic.dir/infback.o
[ 52%] Building C object CMakeFiles/zlib.dir/inflate.o
[ 61%] Building C object CMakeFiles/zlibstatic.dir/inftrees.o
[ 61%] Building C object CMakeFiles/zlibstatic.dir/inffast.o
[ 61%] Building C object CMakeFiles/zlib.dir/infback.o
[ 63%] Building C object CMakeFiles/zlibstatic.dir/trees.o
[ 66%] Building C object CMakeFiles/zlibstatic.dir/uncompr.o
[ 72%] Building C object CMakeFiles/zlibstatic.dir/zutil.o
[ 72%] Building C object CMakeFiles/zlib.dir/inftrees.o
[ 75%] Building C object CMakeFiles/zlib.dir/inffast.o
[ 77%] Building C object CMakeFiles/zlib.dir/trees.o
[ 80%] Building C object CMakeFiles/zlib.dir/uncompr.o
[ 83%] Building C object CMakeFiles/zlib.dir/zutil.o
[ 86%] Linking C static library libz.a
[ 86%] Built target zlibstatic
[ 88%] Linking C static library libz.a
[ 88%] Built target zlib
Scanning dependencies of target minigzip
Scanning dependencies of target example
[ 94%] Building C object CMakeFiles/example.dir/test/example.o
[ 94%] Building C object CMakeFiles/minigzip.dir/test/minigzip.o
[ 97%] Linking C executable minigzip
[100%] Linking C executable example
[100%] Built target example
[100%] Built target minigzip
Install the project...
I definitely saw some strange issues before but now it just somehow works for me. 😅 May be I can rebase this patch on top so it can be easly reverted later?
It was something with the linking of the examples, so I turned it off.
There was a problem hiding this comment.
In your log you can clearly see how it appears twice:
[ 86%] Linking C static library libz.a
[ 88%] Linking C static library libz.a
I’m pretty sure that if you compile with make VERBOSE=1 you will see that the link of the “libz.a“ apppears twice too
There was a problem hiding this comment.
Anyway… let’s merge your changes, and we will see if this issue is no longer happening, if it fails we clearly know how to “fix it”
There was a problem hiding this comment.
You have changed file permissions 100755 → 100644 and now it fails compiling
There was a problem hiding this comment.
omggg I fixed that once
There was a problem hiding this comment.
Fix compilation and be sure that zlib static library is not compiled twice, you can search in the logs by Linking C static library libz.a and you will see it appears twice, this is why we required to do the -j1 because otherwise it was having race conditions when building, especially in high-end computers.
|
Yeah I think I saw something like that with the zlib on my setup but I was not sure if it's intended or not and it somehow compiled for me. |
Sources can be rebuild and installed from already populated repositories on disk which should speed-up developement cycle. Introduced fetch.sh which does force-checkout of the source GIT repo. Also revisited "clean" and "sample" targets to more automated ones that dives into subprojects and call the "clean" / "sample" target. This change introduced warning of targets redefinition but this is intended because some of the dependencies have extra parameters to the build. Signed-off-by: Michal Jenikovsky <jendo@jmsystems.sk>
Signed-off-by: Michal Jenikovsky <jendo@jmsystems.sk>
Signed-off-by: Michal Jenikovsky <jendo@jmsystems.sk>
- fixing issue with parallel build Signed-off-by: Michal Jenikovsky <jendo@jmsystems.sk>
Signed-off-by: Michal Jenikovsky <jendo@jmsystems.sk>
Tried to push the build performance by calling the repo fetch in parallel. Signed-off-by: Michal Jenikovsky <jendo@jmsystems.sk>
|
@jendo42 could you create another PR putting back the -j1 for zlib? It is already failing |
|
Yes no problem, thanks for test. I will take a look later. |
The original did not work when the build directory for sub-dependency already existed and I did not want to remove all the packages, so I changed the script that it now could download the repos when they are already exist.
I have also updated libpng and zlib to try if the zlib parallel build works. It worked for me, but I suggest more testing of that.
Also check #641