Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nginx - php-fpm | Function not implemented #468

Closed
dzygann opened this issue Mar 22, 2022 · 20 comments
Closed

nginx - php-fpm | Function not implemented #468

dzygann opened this issue Mar 22, 2022 · 20 comments

Comments

@dzygann
Copy link

dzygann commented Mar 22, 2022

Hi,

I'm using nginx in combination with php-fpm. My OS is ubuntu20.04.

nginx graminized standalone is working fine and php-fpm graminized standalone is working, too.
Also if php-fpm is graminized with a not graminized nginx is working.

Unfortunately, it doesn't work when both applications are graminized.

We could see following alert messages, which are suspicious:

    15  2022/03/21 21:34:12 [alert] 1#0: sendmsg() failed (38: Function not implemented)
    18  2022/03/21 21:34:12 [debug] 1#0: pass channel s:2 pid:4 fd:10 to s2022/03/21 21:34:12 [alert] 4#0: prctl(PR_SET_DUMPABLE) failed (38: Function not implemented)
    23  2022/03/21 21:34:12 [debug] 1#0: pass channel s:3 pid:5 fd:12 to s2022/03/21 21:34:12 [alert] 5#0: prctl(PR_SET_DUMPA

Do you have an idea what I'm doing wrong?

I get the following error.log from nginx, when I run gramine-sgx nginx :

    1  2022/03/21 21:34:12 [debug] 1#0: bind() 0.0.0.0:80 #5 
     2  2022/03/21 21:34:12 [debug] 1#0: bind() 0.0.0.0:443 #6 
     3  2022/03/21 21:34:12 [notice] 1#0: using the "epoll" event method
     4  2022/03/21 21:34:12 [debug] 1#0: counter: 0000000004DD9080, 1
     5  2022/03/21 21:34:12 [notice] 1#0: nginx/1.18.0
     6  2022/03/21 21:34:12 [notice] 1#0: OS: Linux 3.10.0
     7  2022/03/21 21:34:12 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 900:65536
     8  2022/03/21 21:34:12 [debug] 1#0: write: 7, 00000000060C3A90, 2, 0
     9  2022/03/21 21:34:12 [debug] 1#0: setproctitle: "nginx: master "
    10  2022/03/21 21:34:12 [notice] 1#0: start worker processes
    11  2022/03/21 21:34:12 [debug] 1#0: channel 3:7
    12  2022/03/21 21:34:12 [debug] 2#0: add cleanup: 000000000582022/03/21 21:34:12 [debug] 1#0: channel 8:9
    13  2022/03/21 21:34:12 [notice] 1#0: start worker process 3
    14  2022/03/21 21:34:12 [debug] 1#0: pass channel s:1 pid:3 fd:8 to s:0 pid:2 fd:3
    15  2022/03/21 21:34:12 [alert] 1#0: sendmsg() failed (38: Function not implemented)
    16  2022/03/21 21:34:12 [debug] 1#0: channel 10:11
    17  2022/03/21 21:34:12 [notice] 1#0: start worker process 4
    18  2022/03/21 21:34:12 [debug] 1#0: pass channel s:2 pid:4 fd:10 to s2022/03/21 21:34:12 [alert] 4#0: prctl(PR_SET_DUMPABLE) failed (38: Function not implemented)
    19  2022/03/21 21:34:12 [debug] 4#0: notify eventfd: 13
    20  2022/03/21 21:34:12 [debug] 4#0: testing the EPOLLRDHUP flag: fail
    21  2022/03/21 21:34:12 [debug] 4#0: malloc: 002022/03/21 21:34:12 [debug] 1#0: channel 12:13
    22  2022/03/21 21:34:12 [notice] 1#0: start worker process 5
    23  2022/03/21 21:34:12 [debug] 1#0: pass channel s:3 pid:5 fd:12 to s2022/03/21 21:34:12 [alert] 5#0: prctl(PR_SET_DUMPA
2022/03/21 21:34:42 [debug] 2#0: epoll: fd:5 ev:0001 d:000000000580F230

We did also a strace while we requested the php-fpm without gramine-sgx:

[{EPOLLIN, {u32=701177872, u64=139801886662672}}], 512, -1) = 1
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 66) = 66
[pid   355] accept4(5, {sa_family=AF_INET, sin_port=htons(35910), sin_addr=inet_addr("172.18.0.1")}, [112->16], SOCK_NONBLOCK) = 14
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 76) = 76
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 69) = 69
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 78) = 78
[pid   355] epoll_ctl(8, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=701178544, u64=139801886663344}}) = 0
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 48) = 48
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] epoll_wait(8, [{EPOLLIN, {u32=701178544, u64=139801886663344}}], 512, 60000) = 1
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 64) = 64
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 68) = 68
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 60) = 60
[pid   355] recvfrom(14, "GET /ping HTTP/1.1\r\nHost: localh"..., 1024, 0, NULL, NULL) = 77
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 80) = 80
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 64) = 64
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 78) = 78
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 56) = 56
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 52) = 52
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 53) = 53
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 80) = 80
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 71) = 71
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 69) = 69
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 77) = 77
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 65) = 65
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 55) = 55
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 69) = 69
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 57) = 57
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 64) = 64
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 64) = 64
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 77) = 77
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 55) = 55
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 55) = 55
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 59) = 59
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 55) = 55
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 55) = 55
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 78) = 78
[pid   355] epoll_ctl(8, EPOLL_CTL_MOD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=701178544, u64=139801886663344}}) = 0
[pid   355] getsockname(14, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("172.18.0.3")}, [112->16]) = 0
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 63) = 63
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 94) = 94
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 71) = 71
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 73) = 73
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 71) = 71
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 73) = 73
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 72) = 72
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 63) = 63
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 63) = 63
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 71) = 71
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 63) = 63
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 72) = 72
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 87) = 87
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 66) = 66
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 81) = 81
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 73) = 73
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 62) = 62
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 76) = 76
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 59) = 59
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 76) = 76
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 66) = 66
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 82) = 82
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 65) = 65
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 64) = 64
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 85) = 85
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 68) = 68
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 79) = 79
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 63) = 63
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 68) = 68
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 79) = 79
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 60) = 60
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 71) = 71
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 78) = 78
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 74) = 74
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 62) = 62
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 76) = 76
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 76) = 76
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 84) = 84
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 72) = 72
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 73) = 73
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 58) = 58
[pid   355] socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 15
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 55) = 55
[pid   355] ioctl(15, FIONBIO, [1])     = 0
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 78) = 78
[pid   355] epoll_ctl(8, EPOLL_CTL_ADD, 15, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=701178768, u64=139801886663568}}) = 0
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] connect(15, {sa_family=AF_INET, sin_port=htons(9000), sin_addr=inet_addr("172.18.0.2")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 64) = 64
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 79) = 79
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 83) = 83
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 65) = 65
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 50) = 50
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 48) = 48
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] epoll_wait(8, [{EPOLLOUT, {u32=701178544, u64=139801886663344}}, {EPOLLOUT, {u32=701178768, u64=139801886663568}}], 512, 60000) = 2
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 65) = 65
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 89) = 89
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 73) = 73
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 65) = 65
[pid   355] getsockopt(15, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 66) = 66
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 72) = 72
[pid   355] writev(15, [{iov_base="\1\1\0\1\0\10\0\0\0\1\0\0\0\0\0\0\1\4\0\1\1\330\0\0\17\25SCRIPT"..., iov_len=512}], 1) = 512
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 57) = 57
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 73) = 73
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 69) = 69
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 50) = 50
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 48) = 48
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] epoll_wait(8, [{EPOLLIN|EPOLLOUT, {u32=701178768, u64=139801886663568}}], 512, 60000) = 1
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 68) = 68
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 60) = 60
[pid   355] recvfrom(15, "\1\6\0\1\0\225\3\0Content-type: text/plain"..., 4096, 0, NULL, NULL) = 176
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 62) = 62
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 100) = 100
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 100) = 100
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 123) = 123
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 63) = 63
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 310) = 310
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 127) = 127
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 71) = 71
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 69) = 69
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 60) = 60
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 55) = 55
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 130) = 130
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 63) = 63
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 57) = 57
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 76) = 76
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 60) = 60
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 130) = 130
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 86) = 86
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 66) = 66
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 50) = 50
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 48) = 48
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] epoll_wait(8, [{EPOLLIN|EPOLLOUT|EPOLLRDHUP, {u32=701178768, u64=139801886663568}}], 512, 60000) = 1
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 70) = 70
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 69) = 69
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 60) = 60
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 58) = 58
[pid   355] readv(15, [{iov_base="", iov_len=3920}], 1) = 0
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 57) = 57
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 130) = 130
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 54) = 54
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 68) = 68
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 68) = 68
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 68) = 68
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 63) = 63
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 69) = 69
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 66) = 66
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 65) = 65
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 80) = 80
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 85) = 85
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 52) = 52
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 127) = 127
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 125) = 125
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 125) = 125
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 125) = 125
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 71) = 71
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 67) = 67
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 65) = 65
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 86) = 86
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 75) = 75
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 72) = 72
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 68) = 68
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 55) = 55
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 73) = 73
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 73) = 73
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 69) = 69
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 61) = 61
[pid   355] close(15)                   = 0
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 64) = 64
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 66) = 66
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 65) = 65
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 85) = 85
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 52) = 52
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 127) = 127
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 125) = 125
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 125) = 125
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 125) = 125
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 125) = 125
[pid   355] write(4, "2022/03/21 21:08:00 [debug] 355#"..., 71) = 71

And a second log with gramine-sgx nginx:

[P7:T7:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P4:T4:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P3:T3:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P4:T4:nginx] trace: ---- shim_gettimeofday(0x60c3730, 0) = 0x0
[P4:T4:nginx] trace: ---- shim_clock_gettime(6, 0x60c36c0) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f00, 0x48) ...
[P2:T2:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x40) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x40
[P4:T4:nginx] trace: ---- shim_accept4(5, 0x60c36d0, 0x60c3674, 2048) ...
[P6:T6:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P4:T4:nginx] trace: ---- return from shim_accept4(...) = 0x7
[P6:T6:nginx] trace: ---- shim_gettimeofday(0x60c3730, 0) = 0x0
[P6:T6:nginx] trace: ---- shim_clock_gettime(6, 0x60c36c0) = 0x0
[P2:T2:nginx] trace: ---- shim_gettimeofday(0x60c3730, 0) = 0x0
[P3:T3:nginx] trace: ---- shim_gettimeofday(0x60c3730, 0) = 0x0
[P7:T7:nginx] trace: ---- shim_gettimeofday(0x60c3730, 0) = 0x0
[P5:T5:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P3:T3:nginx] trace: ---- shim_clock_gettime(6, 0x60c36c0) = 0x0
[P7:T7:nginx] trace: ---- shim_clock_gettime(6, 0x60c36c0) = 0x0
[P7:T7:nginx] trace: ---- shim_write(4, 0x60c2f00, 0x48) ...
[P7:T7:nginx] trace: ---- return from shim_write(...) = 0x48
[P7:T7:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x40) ...
[P7:T7:nginx] trace: ---- return from shim_write(...) = 0x40
[P7:T7:nginx] trace: ---- shim_accept4(5, 0x60c36d0, 0x60c3674, 2048) ...
[P7:T7:nginx] trace: ---- return from shim_accept4(...) = -11
[P7:T7:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x5b) ...
[P7:T7:nginx] trace: ---- return from shim_write(...) = 0x5b
[P7:T7:nginx] trace: ---- shim_write(4, 0x60c2f90, 0x34) ...
[P7:T7:nginx] trace: ---- return from shim_write(...) = 0x34
[P7:T7:nginx] trace: ---- shim_write(4, 0x60c2fd0, 0x2e) ...
[P7:T7:nginx] trace: ---- return from shim_write(...) = 0x2e
[P7:T7:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x31) ...
[P7:T7:nginx] trace: ---- return from shim_write(...) = 0x31
[P7:T7:nginx] trace: ---- shim_epoll_wait(18, 0x57e6040, 512, -1) ...
[P6:T6:nginx] trace: ---- shim_write(4, 0x60c2f00, 0x48) ...
[P6:T6:nginx] trace: ---- return from shim_write(...) = 0x48
[P6:T6:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x40) ...
[P6:T6:nginx] trace: ---- return from shim_write(...) = 0x40
[P6:T6:nginx] trace: ---- shim_accept4(5, 0x60c36d0, 0x60c3674, 2048) ...
[P6:T6:nginx] trace: ---- return from shim_accept4(...) = -11
[P6:T6:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x5b) ...
[P6:T6:nginx] trace: ---- return from shim_write(...) = 0x5b
[P6:T6:nginx] trace: ---- shim_write(4, 0x60c2f90, 0x34) ...
[P6:T6:nginx] trace: ---- return from shim_write(...) = 0x34
[P6:T6:nginx] trace: ---- shim_write(4, 0x60c2fd0, 0x2e) ...
[P6:T6:nginx] trace: ---- return from shim_write(...) = 0x2e
[P6:T6:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x31) ...
[P6:T6:nginx] trace: ---- return from shim_write(...) = 0x31
[P6:T6:nginx] trace: ---- shim_epoll_wait(16, 0x57e6040, 512, -1) ...
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d00, 0x4a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d80, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P5:T5:nginx] trace: ---- shim_gettimeofday(0x60c3730, 0) = 0x0
[P3:T3:nginx] trace: ---- shim_write(4, 0x60c2f00, 0x48) ...
[P3:T3:nginx] trace: ---- return from shim_write(...) = 0x48
[P3:T3:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x40) ...
[P3:T3:nginx] trace: ---- return from shim_write(...) = 0x40
[P3:T3:nginx] trace: ---- shim_accept4(5, 0x60c36d0, 0x60c3674, 2048) ...
[P3:T3:nginx] trace: ---- return from shim_accept4(...) = -11
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ce0, 0x4c) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4c
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d00, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c70, 0x4b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4b
[P4:T4:nginx] debug: epoll: added 7 (0x1b853378) to epoll handle 0x1b853048
[P4:T4:nginx] trace: ---- shim_epoll_ctl(12, ADD, 7, {.events=EPOLLIN|EPOLLRDHUP|EPOLLET, .data=0x580f4d0}) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f90, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2fd0, 0x2e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x2e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_epoll_wait(12, 0x57e6040, 512, 60000) ...
[P3:T3:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x5b) ...
[P3:T3:nginx] trace: ---- return from shim_write(...) = 0x5b
[P4:T4:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P4:T4:nginx] trace: ---- shim_gettimeofday(0x60c3730, 0) = 0x0
[P4:T4:nginx] trace: ---- shim_clock_gettime(6, 0x60c36c0) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f00, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2eb0, 0x3e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2df0, 0x42) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x42
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e60, 0x3a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3a
[P4:T4:nginx] trace: ---- shim_recvfrom(7, 0x73a9de0, 0x400, 0, 0, 0) = 0x4d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e50, 0x3a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e90, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x4e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e40, 0x3e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e40, 0x4c) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4c
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e10, 0x36) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x36
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e10, 0x32) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x32
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e10, 0x33) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x33
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d10, 0x4e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2dc0, 0x45) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x45
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2dc0, 0x43) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x43
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2dc0, 0x4b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2dc0, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2dc0, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d60, 0x46) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x46
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c30, 0x37) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x37
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c30, 0x3e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c80, 0x3e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c80, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2cd0, 0x4b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2cd0, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ce0, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P3:T3:nginx] trace: ---- shim_write(4, 0x60c2f90, 0x34) ...
[P3:T3:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ce0, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ce0, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ce0, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ce0, 0x39) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x39
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ce0, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ce0, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2be0, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2b50, 0x4b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4b
[P4:T4:nginx] debug: epoll: modified 7 (0x1b853378) on epoll handle 0x1b853048
[P4:T4:nginx] trace: ---- shim_epoll_ctl(12, MOD, 7, {.events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, .data=0x580f4d0}) = 0x0
[P4:T4:nginx] trace: ---- shim_getsockname(7, 0x60c2fe0, 0x60c2fbc) = 0x0
[P4:T4:nginx] trace: ---- shim_getsockname(7, 0x60c3060, 0x60c303c) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x5c) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x5c
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x45) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x45
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x47) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x47
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x49) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x49
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x45) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x45
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x47) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x47
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x46) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x46
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x45) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x45
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x49) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x49
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x46) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x46
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x55) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x55
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x40) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x40
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x4f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x47) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x47
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3c) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3c
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x4a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x39) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x39
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x4a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x40) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x40
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x50) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x50
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x53) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x53
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x4a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x38) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x38
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x43) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x43
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x4a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x3a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x45) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x45
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2880, 0x41) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x41
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x4c) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4c
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2890, 0x3c) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3c
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x4a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x4a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x52) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x52
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28b0, 0x46) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x46
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2a50, 0x47) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x47
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2980, 0x38) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x38
[P4:T4:nginx] trace: ---- shim_socket(INET, STREAM, 0) = 0x9
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c29d0, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_ioctl(9, FIONBIO, 0x60c3284) ...
[P4:T4:nginx] trace: ---- return from shim_ioctl(...) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2980, 0x4b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4b
[P4:T4:nginx] debug: epoll: added 9 (0x1b853268) to epoll handle 0x1b853048
[P4:T4:nginx] trace: ---- shim_epoll_ctl(12, ADD, 9, {.events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, .data=0x580f5b0}) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c29c0, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_connect(9, {family=INET,ip=172.18.0.2,port=htons(9000)}, 16) ...
[P3:T3:nginx] trace: ---- shim_write(4, 0x60c2fd0, 0x2e) ...
[P3:T3:nginx] trace: ---- return from shim_write(...) = 0x2e
[P3:T3:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x31) ...
[P3:T3:nginx] trace: ---- return from shim_write(...) = 0x31
[P3:T3:nginx] trace: ---- shim_epoll_wait(10, 0x57e6040, 512, -1) ...
[P4:T4:nginx] trace: ---- return from shim_connect(...) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c29d0, 0x2e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x2e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2a50, 0x3d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c29c0, 0x4d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2a10, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_getsockopt(9, 1, 4, 0x60c32c0, 0x60c32c4) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c29b0, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28e0, 0x40) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x40
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28e0, 0x46) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x46
[P5:T5:nginx] trace: ---- shim_clock_gettime(6, 0x60c36c0) = 0x0
[P5:T5:nginx] trace: ---- shim_write(4, 0x60c2f00, 0x48) ...
[P5:T5:nginx] trace: ---- return from shim_write(...) = 0x48
[P5:T5:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x40) ...
[P5:T5:nginx] trace: ---- return from shim_write(...) = 0x40
[P5:T5:nginx] trace: ---- shim_accept4(5, 0x60c36d0, 0x60c3674, 2048) ...
[P5:T5:nginx] trace: ---- return from shim_accept4(...) = -11
[P5:T5:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x5b) ...
[P5:T5:nginx] trace: ---- return from shim_write(...) = 0x5b
[P5:T5:nginx] trace: ---- shim_write(4, 0x60c2f90, 0x34) ...
[P5:T5:nginx] trace: ---- return from shim_write(...) = 0x34
[P5:T5:nginx] trace: ---- shim_write(4, 0x60c2fd0, 0x2e) ...
[P5:T5:nginx] trace: ---- return from shim_write(...) = 0x2e
[P5:T5:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x31) ...
[P5:T5:nginx] trace: ---- return from shim_write(...) = 0x31
[P5:T5:nginx] trace: ---- shim_epoll_wait(14, 0x57e6040, 512, -1) ...
[P4:T4:nginx] trace: ---- shim_writev(9, 0x60c2d90, 1) = 0x1f8
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2400, 0x37) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x37
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c28e0, 0x47) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x47
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c29d0, 0x4c) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4c
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c60, 0x51) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x51
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c20, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f90, 0x30) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x30
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2fd0, 0x2e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x2e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_epoll_wait(12, 0x57e6040, 512, 60000) ...
[P4:T4:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P4:T4:nginx] trace: ---- shim_gettimeofday(0x60c3730, 0) = 0x0
[P4:T4:nginx] trace: ---- shim_clock_gettime(6, 0x60c36c0) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f00, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ee0, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e60, 0x57) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x57
[P4:T4:nginx] trace: ---- shim_recvfrom(7, 0x60c3777, 0x1, 2, 0, 0) = -11
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e60, 0x64) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x64
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f90, 0x30) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x30
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2fd0, 0x2e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x2e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_epoll_wait(12, 0x57e6040, 512, 59991) ...
[P2:T2:nginx] trace: ---- shim_clock_gettime(6, 0x60c36c0) = 0x0
[P2:T2:nginx] trace: ---- shim_write(4, 0x60c2f00, 0x48) ...
[P2:T2:nginx] trace: ---- return from shim_write(...) = 0x48
[P2:T2:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x40) ...
[P2:T2:nginx] trace: ---- return from shim_write(...) = 0x40
[P2:T2:nginx] trace: ---- shim_accept4(5, 0x60c36d0, 0x60c3674, 2048) ...
[P2:T2:nginx] trace: ---- return from shim_accept4(...) = -11
[P2:T2:nginx] trace: ---- shim_write(4, 0x60c2d90, 0x5b) ...
[P2:T2:nginx] trace: ---- return from shim_write(...) = 0x5b
[P2:T2:nginx] trace: ---- shim_write(4, 0x60c2f90, 0x34) ...
[P2:T2:nginx] trace: ---- return from shim_write(...) = 0x34
[P2:T2:nginx] trace: ---- shim_write(4, 0x60c2fd0, 0x2e) ...
[P2:T2:nginx] trace: ---- return from shim_write(...) = 0x2e
[P2:T2:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x31) ...
[P2:T2:nginx] trace: ---- return from shim_write(...) = 0x31
[P2:T2:nginx] trace: ---- shim_epoll_wait(8, 0x57e6040, 512, -1) ...
[P4:T4:nginx] trace: ---- return from shim_epoll_wait(...) = 0x0
[P4:T4:nginx] trace: ---- shim_gettimeofday(0x60c3730, 0) = 0x0
[P4:T4:nginx] trace: ---- shim_clock_gettime(6, 0x60c36c0) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f90, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f60, 0x46) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x46
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f20, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ee0, 0x41) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x41
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e90, 0x3a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3a
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e40, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e90, 0xfc) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0xfc
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e40, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e20, 0x42) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x42
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e40, 0x46) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x46
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e00, 0x47) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x47
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2de0, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_close(9) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2de0, 0x52) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x52
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2db0, 0x49) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x49
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d10, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2bc0, 0x37) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x37
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2bc0, 0x3e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c60, 0x45) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x45
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c60, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c70, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c70, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c70, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c70, 0x34) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x34
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c70, 0x39) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x39
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c70, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c70, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c50, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c50, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c50, 0x36) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x36
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2b50, 0x4f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2a00, 0x42) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x42
[P4:T4:nginx] trace: ---- shim_openat(AT_FDCWD, "/entrypoint/html/50x.html", O_RDONLY|0x800, 0000) = -2
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2b50, 0xea) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0xea
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2bf0, 0x56) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x56
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2bc0, 0x4d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c27e0, 0xbe) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0xbe
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2710, 0x7d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x7d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2730, 0x45) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x45
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2b00, 0x44) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x44
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2a80, 0x43) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x43
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c27d0, 0x57) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x57
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2650, 0x7d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x7d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2650, 0x7d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x7d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2650, 0x7c) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x7c
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2670, 0x45) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x45
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2680, 0x3e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3e
[P4:T4:nginx] trace: ---- shim_writev(7, 0x60c2b30, 3) = 0x134
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c21a0, 0x37) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x37
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2680, 0x47) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x47
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2a70, 0x45) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x45
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2bb0, 0x54) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x54
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2b70, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2da0, 0x55) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x55
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d20, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2cc0, 0x37) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x37
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2be0, 0x35) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x35
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c80, 0x46) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x46
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2c80, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2cf0, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d20, 0x3e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d20, 0x40) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x40
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2cf0, 0x30) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x30
[P4:T4:nginx] trace: ---- shim_setsockopt(7, 6, 1, 0x60c35d4, 4) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d00, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ce0, 0x4c) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4c
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2d20, 0x40) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x40
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f60, 0x3f) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3f
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f60, 0x49) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x49
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e80, 0x42) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x42
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ec0, 0x3a) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3a
[P4:T4:nginx] trace: ---- shim_recvfrom(7, 0x73a9de0, 0x400, 0, 0, 0) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2eb0, 0x39) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x39
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x4e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x4e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ee0, 0x3d) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3d
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e80, 0x46) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x46
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2e80, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_close(7) = 0x0
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ea0, 0x3b) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x3b
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2ea0, 0x48) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x48
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2fd0, 0x2e) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x2e
[P4:T4:nginx] trace: ---- shim_write(4, 0x60c2f10, 0x31) ...
[P4:T4:nginx] trace: ---- return from shim_write(...) = 0x31
[P4:T4:nginx] trace: ---- shim_epoll_wait(12, 0x57e6040, 512, -1) ...

This is our manifest.template

# Nginx manifest example

loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "/entrypoint/sbin/nginx"
loader.argv0_override = "/entrypoint/sbin/nginx"
loader.log_level = "error"
loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}:/usr/local/lib:/usr/lib:/usr/{{ arch_libdir }}"

fs.mount.lib.type = "chroot"
fs.mount.lib.path = "/lib"
fs.mount.lib.uri = "file:{{ gramine.runtimedir() }}"

fs.mount.lib2.type = "chroot"
fs.mount.lib2.path = "{{ arch_libdir }}"
fs.mount.lib2.uri = "file:{{ arch_libdir }}"

fs.mount.lib3.type = "chroot"
fs.mount.lib3.path = "/usr{{ arch_libdir }}"
fs.mount.lib3.uri = "file:/usr{{ arch_libdir }}"

fs.mount.lib4.type = "chroot"
fs.mount.lib4.path = "/usr/local/lib"
fs.mount.lib4.uri = "file:/usr/local/lib"

fs.mount.etc.type = "chroot"
fs.mount.etc.path = "/etc"
fs.mount.etc.uri = "file:/etc"


## mount all sub- and temporary folders of /entrypoint

# client stores requests here
fs.mount.tmp.type = "tmpfs"
fs.mount.tmp.path = "/entrypoint/client_body_temp" 
fs.mount.tmp.uri = "file:/entrypoint/client_body_temp"

# client stores requests from fast-cgi server here
fs.mount.cgi.type = "tmpfs"
fs.mount.cgi.path = "/entrypoint/fastcgi_temp" 
fs.mount.cgi.uri = "file:/entrypoint/fastcgi_temp"

fs.mount.conf.type = "chroot"
fs.mount.conf.path = "/entrypoint/conf" 
fs.mount.conf.uri = "file:/entrypoint/conf"

fs.mount.sbin.type = "chroot"
fs.mount.sbin.path = "/entrypoint/sbin" 
fs.mount.sbin.uri = "file:/entrypoint/sbin"

fs.mount.html.type = "chroot"
fs.mount.html.path = "/entrypoint/html" 
fs.mount.html.uri = "file:/entrypoint/html"

fs.mount.logs.type = "chroot"
fs.mount.logs.path = "/entrypoint/logs" 
fs.mount.logs.uri = "file:/entrypoint/logs"

sgx.debug = false
sgx.nonpie_binary = true
sgx.enclave_size = "512M"
sgx.thread_num = 8
sys.enable_sigterm_injection = true
sys.insecure__allow_eventfd = true


# Nginx benefits from Exitless. Uncomment the below line to use it.
sgx.rpc_thread_num = 8

sgx.trusted_files = [
  "file:{{ gramine.libos }}",
  "file:{{ gramine.runtimedir() }}/",
  "file:{{ arch_libdir }}/",
  "file:/usr/{{ arch_libdir }}/",
  "file:/entrypoint/sbin/nginx",
  "file:/entrypoint/conf/",
  "file:/etc/nsswitch.conf",
  "file:/etc/hosts",
  "file:/etc/group",
  "file:/etc/passwd",
]

sgx.allowed_files = [

"file:/entrypoint/logs",
"file:/entrypoint/html/",
]

Thanks in advance!

@dimakuv
Copy link

dimakuv commented Apr 4, 2022

15 2022/03/21 21:34:12 [alert] 1#0: sendmsg() failed (38: Function not implemented)

This particular line in the log doesn't look good. I guess this is the root cause.

I wouldn't pay attention to the prctl(PR_SET_DUMPABLE) log line. This doesn't seem important -- Nginx tries to dump some memory or registers on the crash to aid with debugging, but this is not the root cause of your issue.

What is this combination of nginx and php-fpm? Does nginx call php-fpm binaries via fork + execve at some point? Your strace/Gramine logs don't show the actual logs of this interaction between Nginx and PHP-FPM.

Also, what does it mean exactly "it doesn't work when both applications are graminized"? What happens exactly? Nginx doesn't respond to client requests? The Nginx process fails? The child PHP-FPM process fails?

@aep
Copy link

aep commented Apr 5, 2022

nginx uses SCM_RIGHTS to pass an FD, which isnt implemented in gramine, probably for nginx worker comunication?
so please try setting workers to 1

/// ./src/os/unix/ngx_channel.c
33     } else {
34         msg.msg_control = (caddr_t) &cmsg;
// ./LibOS/shim/src/sys/shim_socket.c
1264     if (msg->msg_controllen != 0) {
1265         log_warning("\"struct msghdr\" ancillary data is not supported");
1266         return -ENOSYS;

// edit

I now roughly understand how the gramine implementation of sockets works now and this is not going to be a fun ride.
It looks like this was written for windows, treating sockets as streams instead of files.

I guess we could implement socket passing by short circuiting it in the PAL? Assuming the heap is shared, it's just a matter of passing around the handle with some locking. If it's not shared, we'd have to use an indirection on the host, which i think contradcits the whole reason nginx does that in first place.

@aep
Copy link

aep commented Apr 5, 2022

worker 1 fixes this, but the underlying issue with fcgi is actually #489

the workaround is

worker_processes 1;

events {
    use select;
}

@dimakuv
Copy link

dimakuv commented Apr 5, 2022

@aep We are actively working on the Big Sockets Rewrite -- this is one of the few parts of Gramine that wasn't overwritten in the past 5 years. So there are still many bugs and unimplemented functionality in the sockets emulation in Gramine. But this should be fixed rather soon (in ~1 month time).

The SCM_RIGHTS is indeed missing functionality, it is not yet implemented in Gramine. I once tried to implement it, but it was ugly and hacky as hell. So we're actually taking a slower and more robust/correct approach of fixing the sockets first, then fixing our serialization/deserialization logic (aka checkpoint-restore), and then we'll have all the pieces to implement a correct SCM_RIGHTS passing of objects from one process to another.

Using the select mechanism instead of epoll is a workaround, but no more than that -- select is pretty slow by itself, and even slower in Gramine. So you're trading performance for some stability gains. This is Ok to unblock some Nginx workloads, but in general I recommend to keep an eye on Gramine's Sockets Rewrite.

@aep
Copy link

aep commented Apr 5, 2022

aight, let me know how I can help.
not much sense in me poking around if you're rewriting the socket stack anyway

@boryspoplawski
Copy link
Contributor

You can try #499 to see if it helps

aep added a commit to enclaive/enclaive-docker-nginx-sgx that referenced this issue Apr 7, 2022
nginx in enclave:

    - do not fork workers due to gramineproject/gramine#468 (also generally dont)
    - use select instead of epoll due to gramineproject/gramine#489
    - link pcre static due to undetermined dynloader bug

php in enclave:

    - patch away sockopt SO_LISTENQLEN and TCP_INFO due to gramineproject/gramine#503
    - do not fork workers
    - manually force linking libm static, because the ld loader implements glibc versions incorrectly
    - remove apparmor from AC

wordpress in enclave:

    - installer loads fine but needs mysql server to proceed
aep added a commit to enclaive/enclaive-docker-nginx-sgx that referenced this issue Apr 8, 2022
nginx in enclave:

    - do not fork workers due to gramineproject/gramine#468 (also generally dont)
    - use select instead of epoll due to gramineproject/gramine#489
    - link pcre static due to undetermined dynloader bug

php in enclave:

    - patch away sockopt SO_LISTENQLEN and TCP_INFO due to gramineproject/gramine#503
    - do not fork workers
    - manually force linking libm static, because the ld loader implements glibc versions incorrectly
    - remove apparmor from AC

wordpress in enclave:

    - installer loads fine but needs mysql server to proceed
aep added a commit to enclaive/enclaive-docker-nginx-sgx that referenced this issue Apr 8, 2022
nginx in enclave:

    - do not fork workers due to gramineproject/gramine#468 (also generally dont)
    - use select instead of epoll due to gramineproject/gramine#489
    - link pcre static due to undetermined dynloader bug

php in enclave:

    - patch away sockopt SO_LISTENQLEN and TCP_INFO due to gramineproject/gramine#503
    - do not fork workers
    - manually force linking libm static, because the ld loader implements glibc versions incorrectly
    - remove apparmor from AC

wordpress in enclave:

    - installer loads fine but needs mysql server to proceed
aep added a commit to enclaive/enclaive-docker-nginx-sgx that referenced this issue Apr 8, 2022
nginx in enclave:

    - do not fork workers due to gramineproject/gramine#468 (also generally dont)
    - use select instead of epoll due to gramineproject/gramine#489
    - link pcre static due to undetermined dynloader bug

php in enclave:

    - patch away sockopt SO_LISTENQLEN and TCP_INFO due to gramineproject/gramine#503
    - do not fork workers
    - manually force linking libm static, because the ld loader implements glibc versions incorrectly
    - remove apparmor from AC

wordpress in enclave:

    - installer loads fine but needs mysql server to proceed
@boryspoplawski
Copy link
Contributor

What is the status of this? What errors you are seeing?
You can check new sockets version here: #579
Btw, there are no plans for implementing SCM_RIGHTS

@aep
Copy link

aep commented May 18, 2022

yeah i don't think nginx will ever work in gramine.
i think we should close this?
epoll is tracked in the other issue.

@boryspoplawski
Copy link
Contributor

yeah i don't think nginx will ever work in gramine.

but it does work, at least in most configurations (and this seems to be the only one that used SCM_RIGHTS)

epoll is tracked in the other issue.

which issue? I thought we fixed all problems with epoll

@yanzhichao
Copy link

yanzhichao commented Jun 17, 2022

I try to use nginx to proxy to backend server which listen with https . But it does not work, the responce will be 502 Bad Gateway ,config and error is as fellow, did I miss some thing ?

What is puzzling, it work well when the backend server which listen with http

nginx.conf

worker_processes 1; #must

error_log  /dev/stderr  info; #must mean do not change

pid        logs/nginx.pid;

events {
    worker_connections  768; #must
}

http {
    include            mime.types;
    default_type       application/octet-stream;
    sendfile           on;
    keepalive_timeout  65;

    server {
        listen 80;      #must
        server_name localhost;
        rewrite ^(.*)$ https://${server_name}$1 permanent; #must
    }

    # a single HTTP/HTTPS server
    server {
        listen  4439 ssl;
        ssl_certificate            /workplace/app/certs/server.crt;  #must
        ssl_certificate_key        /workplace/app/certs/server.key;  #must
        ssl_session_cache          shared:SSL:10m;
        ssl_session_timeout        10m;
        ssl_protocols              TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers                HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers  on;

        location / {
            root   html;
            index  index.html index.htm;
        }

        #for use to download ca
        location /download/ca.crt {
               alias   /workplace/app/ca/ca.crt;
       }

       # an example for config proxy to backend
       location /jxbapi/ {
            rewrite /jxbapi/(.*) /$1 break;
            proxy_set_header  X-real-ip $remote_addr;
            proxy_read_timeout 300;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

            proxy_ssl_verify on;
            proxy_ssl_session_reuse on;
            proxy_ssl_trusted_certificate /workplace/app/server-ca/server-ca.crt; #must
            proxy_pass https://10.10.11.209:8443
    }

        access_log /dev/stdout;  #must
    }
}
daemon off;

the error logs

[P2:T7:nginx] debug: epoll: added 3 (0xfad67288) to epoll handle 0xfad67048
[P2:T7:nginx] debug: epoll: modified 3 (0xfad67288) on epoll handle 0xfad67048
[P2:T7:nginx] debug: epoll: added 11 (0xfad67168) to epoll handle 0xfad67048
2022/06/17 03:14:52 [alert] 7#0: *5 connect() failed (11: Resource temporarily unavailable) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /jxbapi/jxbs/api/v1/proxy/init/serverStatus HTTP/1.1", upstream: "https://10.10.11.209:8443/jxbs/api/v1/proxy/init/serverStatus", host: "127.0.0.1:4439"
127.0.0.1 - - [17/Jun/2022:03:14:52 +0000] "GET /jxbapi/jxbs/api/v1/proxy/init/serverStatus HTTP/1.1" 502 157 "-" "curl/7.68.0"
2022/06/17 03:14:52 [info] 7#0: *5 client 127.0.0.1 closed keepalive connection

@boryspoplawski
Copy link
Contributor

What is puzzling, it work well when the backend server which listen with http

Maybe you did not add all necessary files to protected, trusted or allowed files in manifest file?

@boryspoplawski
Copy link
Contributor

Closing this as: #579 was merged, so all issues should be fixed (except for missing support SCM_RIGHTS, but we don't have plans for implementing it in the near future and workarounds exist).

@yanzhichao
Copy link

yanzhichao commented Jun 18, 2022

What is puzzling, it work well when the backend server which listen with http

Maybe you did not add all necessary files to protected, trusted or allowed files in manifest file?

It will not be, beacase I run the app in container, so I scan all the file, and add to manifest as trusted.

Maybe have some other problem, the trace log when I requested is as fellow:

trace log

[P2:T6:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P2:T6:nginx] trace: ---- shim_gettimeofday(0xb0fdf6c0, 0) = 0x0
[P2:T6:nginx] trace: ---- shim_clock_gettime(6, 0xb0fdf710) = 0x0
[P2:T6:nginx] trace: ---- shim_accept4(8, 0xb0fdf6d0, 0xb0fdf6cc, 2048) ...
[P2:T6:nginx] trace: ---- return from shim_accept4(...) = 0x3
[P2:T6:nginx] debug: epoll: added 3 (0xf23f9168) to epoll handle 0xf23f9048
[P2:T6:nginx] trace: ---- shim_epoll_ctl(10, ADD, 3, {.events=EPOLLIN|EPOLLRDHUP|EPOLLET, .data=0xd7838741}) = 0x0
[P2:T6:nginx] trace: ---- shim_epoll_wait(10, 0xd7811ad0, 512, 60000) ...
[P2:T6:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P2:T6:nginx] trace: ---- shim_gettimeofday(0xb0fdf6c0, 0) = 0x0
[P2:T6:nginx] trace: ---- shim_clock_gettime(6, 0xb0fdf710) = 0x0
[P2:T6:nginx] trace: ---- shim_recvfrom(3, 0xb0fdf6d0, 0x1, 2, 0, 0) = 0x1
[P2:T6:nginx] trace: ---- shim_setsockopt(3, 6, 1, 0xb0fdf6b4, 4) = 0x0
[P2:T6:nginx] trace: ---- shim_read(3, 0xd788ea03, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = 0x1
[P2:T6:nginx] trace: ---- shim_read(3, 0xd788ea04, 0x4144) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = 0x204
[P2:T6:nginx] trace: ---- shim_time(0) = 0x62adb847
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_time(0) = 0x62adb847
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_time(0) = 0x62adb847
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_time(0) = 0x62adb847
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_time(0) = 0x62adb847
[P2:T6:nginx] trace: ---- shim_write(3, 0xd7809ab0, 0x346) ...
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0x346
[P2:T6:nginx] trace: ---- shim_read(3, 0xd788ea03, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = -11
[P2:T6:nginx] trace: ---- shim_epoll_wait(10, 0xd7811ad0, 512, 59996) ...
[P2:T6:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P2:T6:nginx] trace: ---- shim_gettimeofday(0xb0fdf6c0, 0) = 0x0
[P2:T6:nginx] trace: ---- shim_clock_gettime(6, 0xb0fdf710) = 0x0
[P2:T6:nginx] trace: ---- shim_read(3, 0xd788ea03, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = 0x5d
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_time(0) = 0x62adb847
[P2:T6:nginx] trace: ---- shim_write(3, 0xd7809ab0, 0x33) ...
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0x33
[P2:T6:nginx] trace: ---- shim_read(3, 0xd78894a3, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = -11
[P2:T6:nginx] trace: ---- shim_epoll_wait(10, 0xd7811ad0, 512, 59995) ...
[P2:T6:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P2:T6:nginx] trace: ---- shim_gettimeofday(0xb0fdf6c0, 0) = 0x0
[P2:T6:nginx] trace: ---- shim_clock_gettime(6, 0xb0fdf710) = 0x0
[P2:T6:nginx] trace: ---- shim_read(3, 0xd78894a3, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = 0x87
[P2:T6:nginx] trace: ---- shim_read(3, 0xd78894a3, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = -11
[P2:T6:nginx] trace: ---- shim_write(4, 0xb0fdecb0, 0xc8) ...
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0xc8
2022/06/18 11:34:31 [notice] 6#0: *7 "/jxbapi/(.*)" matches "/jxbapi/jxbs/api/v1/user/info", client: 127.0.0.1, server: , request: "GET /jxbapi/jxbs/api/v1/user/info HTTP/1.1", host: "127.0.0.1:4439"
[P2:T6:nginx] trace: ---- shim_write(4, 0xb0fded00, 0xc4) ...
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0xc4
2022/06/18 11:34:31 [notice] 6#0: *7 rewritten data: "/jxbs/api/v1/user/info", args: "", client: 127.0.0.1, server: , request: "GET /jxbapi/jxbs/api/v1/user/info HTTP/1.1", host: "127.0.0.1:4439"
[P2:T6:nginx] debug: epoll: modified 3 (0xf23f9168) on epoll handle 0xf23f9048
[P2:T6:nginx] trace: ---- shim_epoll_ctl(10, MOD, 3, {.events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, .data=0xd7838741}) = 0x0
[P2:T6:nginx] trace: ---- shim_socket(INET, STREAM, 0) = 0xb
[P2:T6:nginx] trace: ---- shim_ioctl(11, FIONBIO, 0xb0fdf264) ...
[P2:T6:nginx] trace: ---- return from shim_ioctl(...) = 0x0
[P2:T6:nginx] debug: epoll: added 11 (0xf23f9288) to epoll handle 0xf23f9048
[P2:T6:nginx] trace: ---- shim_epoll_ctl(10, ADD, 11, {.events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, .data=0xd7838831}) = 0x0
[P2:T6:nginx] trace: ---- shim_connect(11, {family=INET,ip=10.10.11.209,port=htons(8443)}, 16) ...
[P2:T6:nginx] trace: ---- return from shim_connect(...) = 0x0
[P2:T6:nginx] trace: ---- shim_getsockopt(11, 1, 4, 0xb0fdf260, 0xb0fdf264) = 0x0
[P2:T6:nginx] trace: ---- shim_setsockopt(11, 6, 1, 0xb0fdf264, 4) = 0x0
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_time(0) = 0x62adb847
[P2:T6:nginx] trace: ---- shim_write(11, 0xd7890290, 0x15d) ...
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0x15d
[P2:T6:nginx] trace: ---- shim_read(11, 0xd789a723, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = -11
[P2:T6:nginx] trace: ---- shim_epoll_wait(10, 0xd7811ad0, 512, 60000) ...
[P2:T6:nginx] trace: ---- return from shim_epoll_wait(...) = 0x2
[P2:T6:nginx] trace: ---- shim_gettimeofday(0xb0fdf6c0, 0) = 0x0
[P2:T6:nginx] trace: ---- shim_clock_gettime(6, 0xb0fdf710) = 0x0
[P2:T6:nginx] trace: ---- shim_recvfrom(3, 0xb0fdf737, 0x1, 2, 0, 0) = -11
[P2:T6:nginx] trace: ---- shim_read(11, 0xd789a723, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = -11
[P2:T6:nginx] trace: ---- shim_epoll_wait(10, 0xd7811ad0, 512, 60000) ...
[P2:T6:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P2:T6:nginx] trace: ---- shim_gettimeofday(0xb0fdf6c0, 0) = 0x0
[P2:T6:nginx] trace: ---- shim_clock_gettime(6, 0xb0fdf710) = 0x0
[P2:T6:nginx] trace: ---- shim_read(11, 0xd789a723, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = 0x8f
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_getpid() = 0x6
[P2:T6:nginx] trace: ---- shim_time(0) = 0x62adb847
[P2:T6:nginx] trace: ---- shim_write(11, 0xd7890290, 0x33) ...
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0x33
[P2:T6:nginx] trace: ---- shim_getsockopt(11, 1, 4, 0xb0fdf6f0, 0xb0fdf6f4) = 0x0
[P2:T6:nginx] trace: ---- shim_write(4, 0xb0fdede0, 0x122) ...
2022/06/18 11:34:31 [alert] 6#0: *7 connect() failed (11: Resource temporarily unavailable) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /jxbapi/jxbs/api/v1/user/info HTTP/1.1", upstream: "https://10.10.11.209:8443/jxbs/api/v1/user/info", host: "127.0.0.1:4439"
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0x122
[P2:T6:nginx] trace: ---- shim_write(11, 0xd788ad33, 0x1f) ...
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0x1f
[P2:T6:nginx] trace: ---- shim_close(11) = 0x0
[P2:T6:nginx] trace: ---- shim_write(3, 0xd788d4b3, 0x157) ...
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0x157
[P2:T6:nginx] trace: ---- shim_write(5, 0xd7898800, 0x72) ...
127.0.0.1 - - [18/Jun/2022:11:34:31 +0000] "GET /jxbapi/jxbs/api/v1/user/info HTTP/1.1" 502 157 "-" "curl/7.68.0"
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0x72
[P2:T6:nginx] trace: ---- shim_epoll_wait(10, 0xd7811ad0, 512, 65000) ...
[P2:T6:nginx] trace: ---- return from shim_epoll_wait(...) = 0x1
[P2:T6:nginx] trace: ---- shim_gettimeofday(0xb0fdf6c0, 0) = 0x0
[P2:T6:nginx] trace: ---- shim_clock_gettime(6, 0xb0fdf710) = 0x0
[P2:T6:nginx] trace: ---- shim_read(3, 0xd78894a3, 0x4145) ...
[P2:T6:nginx] trace: ---- return from shim_read(...) = 0x1f
[P2:T6:nginx] trace: ---- shim_write(4, 0xb0fdee60, 0x50) ...
2022/06/18 11:34:31 [info] 6#0: *7 client 127.0.0.1 closed keepalive connection
[P2:T6:nginx] trace: ---- return from shim_write(...) = 0x50
[P2:T6:nginx] trace: ---- shim_close(3) = 0x0
[P2:T6:nginx] trace: ---- shim_epoll_wait(10, 0xd7811ad0, 512, -1) ...

@aep
Copy link

aep commented Jun 18, 2022

EAGAIN is possibly due to epoll. TLS has different timing than naked http. just disable it.

#468 (comment)

worker 1 fixes this, but the underlying issue with fcgi is actually #489

the workaround is

worker_processes 1;

events {
    use select;
}

@yanzhichao
Copy link

yanzhichao commented Jun 18, 2022

I swich to select, but still not work

nginx.conf

worker_processes 1; #must

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
error_log  /dev/stderr info; #must mean do not change

pid        logs/nginx.pid;

events {
#  worker_connections  768; #must
  use select;
}

http {
  include            mime.types;
  default_type       application/octet-stream;
  sendfile           on;
  keepalive_timeout  65;

  server {
    listen 880;      #must
    server_name localhost;
    rewrite ^(.*)$ https://${server_name}$1 permanent; #must
  }

  # a single HTTP/HTTPS server
  server {
    listen  4438 ssl;
    ssl_certificate            /workplace/app/certs/server.crt;  #must
    ssl_certificate_key        /workplace/app/certs/server.key;  #must
    ssl_session_cache          shared:SSL:10m;
    ssl_session_timeout        10m;
    ssl_protocols              TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers                HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;

    location / {
      root   html;
      index  index.html;
    }

    #for use to download ca
    location /download/ca.crt {
      alias   /workplace/app/ca/ca.crt;
    }

    # an example for config proxy to backend
    location /jxbapi/ {
      rewrite /jxbapi/(.*) /$1 break;
      proxy_set_header  X-real-ip $remote_addr;
      proxy_read_timeout 300;
      # proxy_set_header Host $host;
      # proxy_set_header Host localhost;
       proxy_ssl_server_name on;
      # proxy_ssl_name localhost;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      # proxy_ssl_certificate /etc/nginx/cert.pem;
      # proxy_ssl_certificate_key /etc/nginx/cert.key;
     # proxy_ssl_verify on;
      proxy_ssl_session_reuse on;
     # proxy_ssl_trusted_certificate /workplace/app/server-ca/server-ca.crt; #must
      proxy_pass https://10.10.11.209:8443;

    }

    access_log /dev/stdout;  #must
  }
}

daemon off;

error log

[P1:T1:nginx] warning: Unsupported system call rseq
2022/06/18 13:16:48 [notice] 1#0: using the "select" event method
2022/06/18 13:16:48 [notice] 1#0: nginx/1.16.1
2022/06/18 13:16:48 [notice] 1#0: built by gcc 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1) 
2022/06/18 13:16:48 [notice] 1#0: OS: Linux 3.10.0
2022/06/18 13:16:48 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 900:65536
[P1:T1:nginx] debug: Creating pipe: pipe.srv:4fc3b206ee6533f3d325f087405913fb4cc052ce595a1b4f6d54664bd19f71f1
2022/06/18 13:16:48 [notice] 1#0: start worker processes
[P1:T1:nginx] debug: Installed async event at 1655558208652090
[P1:T1:nginx] debug: Creating pipe: pipe.srv:5fb24e54206d4c8f50424daca0151ee33a56ce7a9e5d9ea2de032bd2728b70e0

[P2:T7:nginx] debug: ipc_get_id_owner: got a response: 0
[P2:T7:nginx] debug: Shim process initialized
2022/06/18 13:16:48 [notice] 1#0: start worker process 7
2022/06/18 13:16:56 [alert] 7#0: *1 connect() failed (11: Resource temporarily unavailable) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /jxbapi/jxbs/api/v1/user/info HTTP/1.1", upstream: "https://10.10.11.209:8443/jxbs/api/v1/user/info", host: "127.0.0.1:4438"
127.0.0.1 - - [18/Jun/2022:13:16:56 +0000] "GET /jxbapi/jxbs/api/v1/user/info HTTP/1.1" 502 157 "-" "curl/7.68.0"
2022/06/18 13:16:56 [info] 7#0: *1 client 127.0.0.1 closed keepalive connection

@boryspoplawski
Copy link
Contributor

@yanzhichao The first log seems to be ok, I don't see anything failing in there... Actually, there is getsockopt(x, SOL_SOCKET, SO_ERROR), but no way to know what it returned. Hm, maybe it returned -EAGAIN that was reported. Are you sure you are running the sockets PR (#579) or the current master as I asked before? This shouldn't be an issue now.

@aep That was fixed over two months ago, why would you think it's related?

@yanzhichao
Copy link

I have try the last commit which Completely rewrite networking code

the error is same, still not work @boryspoplawski

@boryspoplawski
Copy link
Contributor

@yanzhichao Could you provide log trace of the failing run?

@yanzhichao
Copy link

sorry, I rebuild the gramine use latest master again, it work now. I may have just confused the version.

ok, summary:

  • v1.1 , when do Reverse proxy, it will timeout.

  • v1.2 , when do Reverse Proxy which to https, it will occure [alert] 7#0: *1 connect() failed (11: Resource temporarily unavailable) while connecting to upstream, client: 127.0.0.1, server: , request: "GET /jxbapi/jxbs/api/v1/user/info HTTP/1.1", upstream: "https://10.10.11.209:8443/jxbs/api/v1/user/info", host: "127.0.0.1:4438" 127.0.0.1 - - [18/Jun/2022:13:16:56 +0000] "GET /jxbapi/jxbs/api/v1/user/info HTTP/1.1" 502 157 "-" "curl/7.68.0" 2022/06/18 13:16:56 [info] 7#0: *1 client 127.0.0.1 closed keepalive connection , while can reverse proxy to http successful

  • lastest master banch(which after Completely rewrite networking code), fix all the issue

@boryspoplawski
Copy link
Contributor

Good to hear

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants