Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] CI broken (windows and linux) #4091

Closed
t-bltg opened this issue Feb 3, 2022 · 9 comments
Closed

[BUG] CI broken (windows and linux) #4091

t-bltg opened this issue Feb 3, 2022 · 9 comments
Labels
bug CI continuous integration

Comments

@t-bltg
Copy link
Member

t-bltg commented Feb 3, 2022 •
edited
Loading

linux
https://github.com/JuliaPlots/Plots.jl/runs/5048581857?check_suite_focus=true

[ Info: Testing plot: gr:1:Lines
corrupted size vs. prev_size
signal (6): Aborted
[ Info: Testing plot: gr:1:Lines
free(): invalid next size (normal)

signal (6): Aborted

windows
https://github.com/JuliaPlots/Plots.jl/runs/5048581915?check_suite_focus=true

 [ Info: Testing plot: gr:1:Lines

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x7ffbce572131 -- RtlAllocateHeap at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
in expression starting at D:\a\Plots.jl\Plots.jl\test\runtests.jl:215
RtlAllocateHeap at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
RtlAllocateHeap at C:\Windows\SYSTEM32\ntdll.dll (unknown line)
malloc at C:\Windows\System32\msvcrt.dll (unknown line)
ERROR: Package Plots errored during testing (exit code: 3221226356)

This also segfaults locally. @jheinen, if you have any idea helping us narrowing the root cause, that would be very much appreciated. It looks like a bug introduced by a package update (GR not involved here) within the 4 last days since latest commit 2a7d2d7 to master has passed. Ping @BeastyBlacksmith.
@t-bltg t-bltg added bug CI continuous integration labels Feb 3, 2022
@t-bltg t-bltg changed the title [BUG] CI: GR broken windows and linux [BUG] CI: GR broken (windows and linux) Feb 3, 2022
@t-bltg t-bltg added the priority label Feb 3, 2022
@t-bltg t-bltg changed the title [BUG] CI: GR broken (windows and linux) [BUG] CI broken (windows and linux) Feb 3, 2022
@johnnychen94 johnnychen94 mentioned this issue Feb 3, 2022
Closed
@juliohm
Copy link

juliohm commented Feb 3, 2022

I am having similar issue in my visual tests with ReferenceTests.jl. They used to work fine but all tests with plots are now stuck with malloc errors.

@t-bltg
Copy link
Member Author

t-bltg commented Feb 3, 2022 •
edited
Loading

Thanks for helping out.

Here is a crude diff of pkg versions on log files between working and failing runs (1.6, ubuntu).

--- OK.txt	2022-02-03 22:34:16.425319574 +0100
+++ NOK.txt	2022-02-03 22:34:37.525377794 +0100
@@ -1,10 +1,7 @@
      Testing Plots
 ┌ Warning: Could not use exact versions of packages in manifest, re-resolving
 └ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src
-ERROR: Unexpected end of data : jl_rUlkOOhYGLXA~
-┌ Warning: failed to extract archive downloaded from https://pkg.julialang.org/package/670881a/2e6084db6cccab11fe0bc3e4130bd3d117092ed9
-└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src
-      Status `/tmp/jl_PAxAAP/Project.toml`
+      Status `/tmp/jl_x9m43o/Project.toml`
   [5ae59095] Colors v0.12.8
   [d38c429a] Contour v0.5.7
   [31c24e10] Distributions v0.23.8
@@ -27,7 +24,7 @@
   [995b91a9] PlotUtils v1.1.3
   [f0f68f2c] PlotlyJS v0.18.8
   [91a5bcdd] Plots v1.25.7 `~/work/Plots.jl/Plots.jl`
-  [ce6b1742] RDatasets v0.7.6
+  [ce6b1742] RDatasets v0.7.7
   [3cdcf5f2] RecipesBase v1.2.1
   [01d81517] RecipesPipeline v0.5.0
   [189a3867] Reexport v0.2.0
@@ -55,19 +52,19 @@
   [10745b16] Statistics `@stdlib/Statistics`
   [8dfed614] Test `@stdlib/Test`
   [cf7118a7] UUIDs `@stdlib/UUIDs`
-      Status `/tmp/jl_PAxAAP/Manifest.toml`
+      Status `/tmp/jl_x9m43o/Manifest.toml`
   [621f4979] AbstractFFTs v1.1.0
   [79e6a3ab] Adapt v3.3.3
   [dce04be8] ArgCheck v2.1.0
   [7d9fca2a] Arpack v0.4.0
-  [4fba245c] ArrayInterface v4.0.1
+  [4fba245c] ArrayInterface v4.0.2
   [bf4720bc] AssetRegistry v0.1.0
   [13072b0f] AxisAlgorithms v1.0.1
   [39de3d68] AxisArrays v0.4.4
   [9e28174c] BinDeps v1.0.2
   [ad839575] Blink v0.12.5
   [fa961155] CEnum v0.4.1
-  [336ed68f] CSV v0.9.11
+  [336ed68f] CSV v0.10.2
   [159f3aea] Cairo v1.0.5
   [49dc2e85] Calculus v0.5.1
   [aafaddc9] CatIndices v0.2.2
@@ -147,7 +144,7 @@
   [92d709cd] IrrationalConstants v0.1.1
   [c8e1da08] IterTools v1.4.0
   [82899510] IteratorInterfaceExtensions v1.0.0
-  [692b3bcd] JLLWrappers v1.4.0
+  [692b3bcd] JLLWrappers v1.4.1
   [97c1335a] JSExpr v0.5.3
   [682c06a0] JSON v0.21.2
   [5ab0869b] KernelDensity v0.5.1
@@ -177,7 +174,7 @@
   [bac558e1] OrderedCollections v1.4.1
   [90014a1f] PDMats v0.10.1
   [8314cec4] PGFPlotsX v1.4.1
-  [f57f5aa1] PNGFiles v0.3.12
+  [f57f5aa1] PNGFiles v0.3.13
   [5432bcbf] PaddedViews v0.5.11
   [d96e819e] Parameters v0.12.3
   [69de0a69] Parsers v2.2.1
@@ -197,7 +194,7 @@
   [dca85d43] QuartzImageIO v0.7.3
   [94ee1d12] Quaternions v0.4.2
   [df47a6cb] RData v0.8.3
-  [ce6b1742] RDatasets v0.7.6
+  [ce6b1742] RDatasets v0.7.7
   [b3c3ace0] RangeArrays v0.3.2
   [c84ed2f1] Ratios v0.4.2
   [3cdcf5f2] RecipesBase v1.2.1
@@ -254,7 +251,7 @@
   [a3f928ae] Fontconfig_jll v2.13.93+0
   [d7e528f0] FreeType2_jll v2.10.4+0
   [559328eb] FriBidi_jll v1.0.10+0
-  [0656b61e] GLFW_jll v3.3.5+1
+  [0656b61e] GLFW_jll v3.3.6+0
   [d2c73de3] GR_jll v0.63.1+0
   [77ec8976] GTK3_jll v3.24.31+0
   [78b55507] Gettext_jll v0.21.0+0

@t-bltg
Copy link
Member Author

t-bltg commented Feb 3, 2022 •
edited
Loading

Since it's crashing somewhere along image comparisons, could it be PNGFiles, @Drvi or @IanButterworth?

@Drvi
Copy link

Drvi commented Feb 4, 2022

Locally, I tried pinning PNGFiles@0.3.12 and running the tests, still getting a segfault. Perhaps a bad interaction between the new JLLWrappers and GR_jll?

(@v1.7) pkg> activate .
  Activating project at `~/.julia/dev/Plots`

shell> grep PNGFiles -A4 Manifest.toml
[[deps.PNGFiles]]
deps = ["Base64", "CEnum", "ImageCore", "IndirectArrays", "OffsetArrays", "libpng_jll"]
git-tree-sha1 = "6d105d40e30b635cfed9d52ec29cf456e27d38f8"
uuid = "f57f5aa1-a3ce-4bc8-8ab9-96f992907883"
version = "0.3.12"

(Plots) pkg> test
     Testing Plots
...
Test Summary: | Pass  Total
Utils         |   23     23
[ Info: Testing plot: gr:1:Lines
malloc(): corrupted top size

signal (6): Aborted
in expression starting at /home/drvi/.julia/dev/Plots/test/runtests.jl:215

julia> versioninfo()
Julia Version 1.7.1
Commit ac5cc99908 (2021-12-22 19:35 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, ivybridge)

@t-bltg
Copy link
Member Author

t-bltg commented Feb 4, 2022

Thanks for eliminating that possibility.

@t-bltg
Copy link
Member Author

t-bltg commented Feb 4, 2022 •
edited
Loading

Backtracing by hand:

Segfaults at https://github.com/JuliaPlots/VisualRegressionTests.jl/blob/d18e703cca0162989096342c06da6acb1be292f9/src/imgcomp.jl#L7

Which in turn segfaults at https://github.com/JuliaIO/FileIO.jl/blob/afbe547afc003a8b7413065c829f084e2d0b9151/src/loadsave.jl#L219

Which segfaults at https://github.com/JuliaIO/ImageIO.jl/blob/d63b43b77404069f8709c6fa003dfe627c31976b/src/ImageIO.jl#L57

Which segfaults at https://github.com/JuliaIO/PNGFiles.jl/blob/63701b1e902a7ab8bc22a62d1ec09e371d0deb4e/src/io.jl#L266

@Drvi, this seems related to PNGFiles. I've inspected the png files written in /tmp by Plots, and they open up fine using my linux viewer.

EDIT: Crash occurs spuriously when loading tmp images or reference images, maybe indicating something related to GC ?

Ping @johnnychen94 which might know something here ?

@t-bltg t-bltg mentioned this issue Feb 4, 2022
Closed
@t-bltg
Copy link
Member Author

t-bltg commented Feb 4, 2022 •
edited
Loading

Finally got a full backtrace (don't mind the line numbers, they are off):

munmap_chunk(): invalid pointer

signal (6): Aborted
in expression starting at [...]/Plots.jl/test/runtests.jl:259
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7fca567fe3ed)
unknown function (ip: 0x7fca5680647b)
unknown function (ip: 0x7fca568066cb)
inflateEnd at [...]/julia-1.7.1/bin/../lib/julia/libz.so.1 (unknown line)
png_destroy_read_struct at [...]/julia-1.7.1/.julia/artifacts/ddfc455343aff48d27c1b39d7fcb07e0d9242b50/lib/libpng16.so (unknown line)
png_destroy_read_struct at [...]/julia-1.7.1/.julia/packages/PNGFiles/UbDS0/gen/libpng/libpng_api.jl:586 [inlined]
_load! at [...]/julia-1.7.1/.julia/packages/PNGFiles/UbDS0/src/io.jl:265
unknown function (ip: 0x7fc9a0252173)
_jl_invoke at /buildworker/worker/package_linux64/build/srmunmap_chunk(): invalid pointer
c/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:757
#invokelatest#2 at ./essentials.jl:716 [inlined]
invokelatest at ./essentials.jl:714 [inlined]
#_load#6 at [...]/julia-1.7.1/.julia/packages/PNGFiles/UbDS0/src/io.jl:250
_load##kw at [...]/julia-1.7.1/.julia/packages/PNGFiles/UbDS0/src/io.jl:106 [inlined]
#load#2 at [...]/julia-1.7.1/.julia/packages/PNGFiles/UbDS0/src/io.jl:54
load at [...]/julia-1.7.1/.julia/packages/PNGFiles/UbDS0/src/io.jl:48
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:757
#invokelatest#2 at ./essentials.jl:716 [inlined]
invokelatest at ./essentials.jl:714 [inlined]
#load#3 at [...]/julia-1.7.1/.julia/packages/ImageIO/IUnoW/src/ImageIO.jl:57
load at [...]/julia-1.7.1/.julia/packages/ImageIO/IUnoW/src/ImageIO.jl:57
unknown function (ip: 0x7fc9a02459cd)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
jl_f__call_latest at /buildworker/worker/package_linux64/build/src/builtins.c:757
#invokelatest#2 at ./essentials.jl:716 [inlined]
invokelatest at ./essentials.jl:714 [inlined]
#action#33 at [...]/julia-1.7.1/.julia/packages/FileIO/QkYgA/src/loadsave.jl:219
action at [...]/julia-1.7.1/.julia/packages/FileIO/QkYgA/src/loadsave.jl:197 [inlined]
#action#32 at [...]/julia-1.7.1/.julia/packages/FileIO/QkYgA/src/loadsave.jl:185
action at [...]/julia-1.7.1/.julia/packages/FileIO/QkYgA/src/loadsave.jl:185 [inlined]
#load#14 at [...]/julia-1.7.1/.julia/packages/FileIO/QkYgA/src/loadsave.jl:113
load at [...]/julia-1.7.1/.julia/packages/FileIO/QkYgA/src/loadsave.jl:110 [inlined]
#compare_images#6 at [...]/VisualRegressionTests.jl/src/imgcomp.jl:17
compare_images##kw at [...]/VisualRegressionTests.jl/src/imgcomp.jl:5 [inlined]
#test_images#7 at [...]/VisualRegressionTests.jl/src/imgcomp.jl:45
test_images##kw at [...]/VisualRegressionTests.jl/src/imgcomp.jl:44

@Drvi
Copy link

Drvi commented Feb 4, 2022

@t-bltg Good investigation. Do you happen to have a small reproducer handy?

@t-bltg
Copy link
Member Author

t-bltg commented Feb 4, 2022

I'll open an issue with a mwe in PNGFiles in a minute.

This was referenced Feb 4, 2022
Closed
Merged
@t-bltg t-bltg closed this as completed Feb 6, 2022
@t-bltg t-bltg removed the priority label Feb 6, 2022
@t-bltg t-bltg mentioned this issue Feb 6, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug CI continuous integration
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Drvi @juliohm @t-bltg