Unexpected hangs/performance issues

[hongxuchen]

As of 2df6437, when running img2sixel $FILE -o /tmp/test.six, it may cause some unexpected hangs (may be because infinite loop, haven't looked into them) or there are some performance issues (specially for some crafted PSD files).

The files in this directory on my machine took more than 10s to execute.

[saitoha]

@hongxuchen Thanks for your reporting.
Now I've checked h001. This is huge size(65527 * 19980) of GIF image.
It takes some seconds to encode it because SIXEL encoder have to process them by 6 lines at a time, but it seems there is not a performance bug.
I'll check other POCs one by one. It will take time to examine all of them.

[hongxuchen]

I think the problem is that the actual size of h001 is only 205 bytes and libsixel should quickly detect that 65527 * 19980 is incorrect.

[saitoha]

Indeed, it must detect actual size. thanks.

[saitoha]

current status report

• h001: fixed (e3a4c0e, hotfix-issue-76-h001 branch)
• h002: fixed (3c071b9, hotfix-issue-76-h002 branch)
• h003: not reproduced on hotfix-issue-76-h001 branch
• h004: gif animation loop is a designed behavior of img2sixel. We can disable it with --loop-control=disable option. (the default setting is --loop-control=auto)
• h005: same as h004
• h006: same as h004
• h007: same as h004
• h008: same as h004
• h009: same as h004

[saitoha]

• h010: fixed (d7b2600, hotfix-issue-76-h010 branch)
• h011: same as h004
• h012: same as h004
• h013: same as h004
• h014: same as h004
• h015: same as h004
• h016: same as h004
• h017: same as h004
• h018: same as h004
• h019: not reproduced on hotfix-issue-76-h010 branch
• h020: same as h004
• h021: same as h004
• h022: same as h004
• h023: same as h004
• h024: same as h004
• h025: same as h004
• h026: same as h004
• h027: same as h004
• h028: same as h004

[saitoha]

• h029: can't reproduce it on my environment

$ uname -a
Darwin mbp4 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64

without libpng:

$ ./configure --without-jpeg --without-png && make
$ converters/img2sixel -V | grep libpng
  libpng: no
$ curl -O -sL -G -d raw=true 'https://github.com/ntu-sec/pocs/blob/master/libsixel-2df6437/hangs/h029' 
$ time converters/img2sixel h029 
stb_image error


real	0m0.062s
user	0m0.018s
sys	0m0.026s

with libpng:

$ ./configure --without-jpeg --with-png && make
$ converters/img2sixel -V | grep libpng
  libpng: yes
$ curl -O -sL -G -d raw=true 'https://github.com/ntu-sec/pocs/blob/master/libsixel-2df6437/hangs/h029' 
$ time converters/img2sixel h029 
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng error: [01][00][00][00]: invalid chunk type
Abort trap: 6

real	0m0.060s
user	0m0.017s
sys	0m0.024s

$ otool -L converters/.libs/img2sixel 
converters/.libs/img2sixel:
	/usr/local/lib/libsixel.1.dylib (compatibility version 2.0.0, current version 2.6.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
	/usr/local/opt/libpng/lib/libpng16.16.dylib (compatibility version 51.0.0, current version 51.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)

[hongxuchen]

I think the problem may lie in the libpng16 library.
I tried a fresh machine (also Ubuntu 18.04) that with and without libpng (from deb package); when compiled with libpng support, it still took quite long time to finish and kept warning repeated error messages like:

...
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
...

$ ldd img2sixel
        linux-vdso.so.1 (0x00007ffcef8eb000)
        libsixel.so.1 => /home/exp/FOT/libsixel/install/lib/libsixel.so.1 (0x00007f01a1505000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f01a1167000)
        libpng16.so.16 => /usr/lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f01a0f35000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f01a0d18000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f01a0927000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f01a1782000)

When without libpng support, the result is exactly the same on your machine.

[saitoha]

OK, for now I step back from the issue of libpng.

• h029: libpng issue
• h030: same as h004
• h031: same as h004
• h032: same as h004
• h033: same as h004
• h034: same as h004
• h035: same as h004
• h036: same as h004
• h037: same as h004
• h038: same as h004
• h039: same as h004
• h040: same as h004
• h041: libpng issue
• h042: same as h004
• h043: same as h004
• h044: fixed on hotfix-issue-76-h010 branch
• h045: same as h004
• h046: same as h004
• h047: same as h004
• h048: same as h004
• h049: same as h004
• h050: fixed on head of hotfix-issue-76-h010 branch (197d025)

[saitoha]

On v1.8.5, h077, h081, h087, h091 and h093 still cause hungs.

$ seq 93 | xargs printf %03d\\n | while read i; do echo -e "$i"; gtimeout 10 converters/img2sixel "https://github.com/ntu-sec/pocs/blob/master/libsixel-2df6437/hangs/h${i}?raw=true" -l disable > /dev/null; test $? = 124 && echo -e "\033[31mhung\033[m";done
001
002
stb_image error
outofmem
003
004
005
006
007
runtime error
missing valid block terminator (unknown code 3c).
008
009
010
stb_image error
corrupt
011
runtime error
missing valid block terminator (unknown code 3c).
012
013
runtime error
missing valid block terminator (unknown code 3c).
014
015
016
017
018
runtime error
missing valid block terminator (unknown code 3c).
019
stb_image error
corrupt
020
021
022
023
024
025
026
027
028
029
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng error
PNG unsigned integer out of range
030
031
032
033
034
035
036
037
038
039
040
041
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng error
PNG unsigned integer out of range
042
043
044
stb_image error
corrupt
045
046
047
048
049
050
stb_image error
bad file
051
libpng warning: gAMA: CRC error
libpng warning: gAMA: invalid
libpng error
PNG unsigned integer out of range
052
stb_image error
outofmem
053
054
055
056
libpng warning: cHRM: CRC error
libpng warning: bKGD: CRC error
libpng warning: bKGD: invalid
libpng error
[01][00][00][00]: invalid chunk type
057
stb_image error
outofmem
058
059
stb_image error
outofmem
060
stb_image error
bad file
061
stb_image error
bad file
062
stb_image error
corrupt
063
libpng warning: gAMA: CRC error
libpng warning: gAMA: invalid
libpng error
PNG unsigned integer out of range
064
stb_image error
corrupt
065
stb_image error
corrupt
066
stb_image error
corrupt
067
stb_image error
bad file
068
stb_image error
bad file
069
070
stb_image error
bad file
071
072
stb_image error
corrupt
073
stb_image error
bad file
074
runtime error
missing valid block terminator (unknown code 3c).
075
stb_image error
bad file
076
stb_image error
corrupt
077
hung
078
079
080
stb_image error
bad file
081
hung
082
libpng warning: gAMA: CRC error
libpng warning: gAMA: invalid
libpng error
PNG unsigned integer out of range
083
stb_image error
bad file
084
stb_image error
corrupt
085
stb_image error
bad file
086
libpng warning: cHRM: CRC error
libpng warning: tIME: CRC error
libpng warning: tIME: invalid
libpng error
PNG unsigned integer out of range
087
hung
088
stb_image error
bad file
089
stb_image error
outofmem
090
runtime error: bad allocation error
load_gif: sixel_allocator_malloc() failed. size=0.
091
hung
092
stb_image error
outofmem
093
hung

[saitoha]

h077, h081, h087, h091 and h093 are just large size files.
I've confirmed they are processed within 20secs on my laptop PC.

for i in 077 081 087 091 093; do echo h$i ; time converters/img2sixel "https://github.com/ntu-sec/pocs/blob/master/libsixel-2df6437/hangs/h${i}?raw=true" -l disable 2> /dev/null 1>/dev/null; test $? = 124 && echo -e "\033[31mhung detected\033[m${i}";done
h077

real	0m13.950s
user	0m12.193s
sys	0m0.703s
h081

real	0m10.775s
user	0m8.748s
sys	0m0.546s
h087

real	0m14.526s
user	0m12.388s
sys	0m0.713s
h091

real	0m13.231s
user	0m11.559s
sys	0m0.514s
h093

real	0m16.966s
user	0m13.426s
sys	0m0.623s