Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle EOF to prevent file descriptor leak #37

Merged
merged 1 commit into from
May 10, 2019

Conversation

georgethebeatle
Copy link
Contributor

@georgethebeatle georgethebeatle commented May 10, 2019

Currently the EOF that the server gets when a client connection is
closed is being ignored, which means that the goroutine that is
processing the client connection will never exit. If it never exits
it will never close the underlying unix socket and this would lead
to a file descriptor leak.

Signed-off-by: Georgi Sabev georgethebeatle@gmail.com

Reprodction steps:

# create a long running task
ctr image pull docker.io/library/alpine:latest
ctr run --rm docker.io/library/alpine:latest test sleep 9999

# count number of open sockets
ls -la /proc/$(pidof containerd-shim)/fd | grep socket | wc -l
2

# restart containerd to make it reconnect to the shim 
kill $(pidof containerd)
containerd & 

# observe that socket number increased. It increases by 1 with every containerd restart
ls -la /proc/$(pidof containerd-shim)/fd | grep socket | wc -l
3

Currently the EOF that the server gets when a client connection is
closed is being ignored, which means that the goroutine that is
processing the client connection will never exit. If it never exits
it will never close the underlying unix socket and this would lead
to a file descriptor leak.

Signed-off-by: Georgi Sabev <georgethebeatle@gmail.com>
@crosbymichael
Copy link
Member

LGTM

Copy link
Member

@estesp estesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@estesp estesp merged commit 69eadd1 into containerd:master May 10, 2019
@@ -452,6 +452,11 @@ func (c *serverConn) run(sctx context.Context) {
if err != nil && err != io.EOF {
logrus.WithError(err).Error("error receiving message")
}
if err == io.EOF || err == io.ErrUnexpectedEOF {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The err == io.ErrUnexpectedEO won't work unless you ignore it above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would not have worked if there was a return statement in the if above, but there isn't. It would still be better to not log the ErrUnecpectedEOF though so we have created a new pr: #38. Pls, take a look.

thaJeztah added a commit to thaJeztah/docker that referenced this pull request Jun 14, 2019
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7

> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this pull request Jun 17, 2019
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7

> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: d5669ec1c6eedcd5dd8b0ecd615638934561daa4
Component: engine
thaJeztah added a commit to thaJeztah/docker that referenced this pull request Aug 26, 2019
full diff: containerd/ttrpc@699c4e4...1fb3814

changes included:

- containerd/ttrpc#37 Handle EOF to prevent file descriptor leak
- containerd/ttrpc#38 Improve connection error handling
- containerd/ttrpc#40 Support headers
- containerd/ttrpc#41 Add client and server unary interceptors
- containerd/ttrpc#42 Refactor close handling for ttrpc clients
- containerd/ttrpc#43 metadata as KeyValue type

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
thaJeztah added a commit to thaJeztah/docker that referenced this pull request Aug 28, 2019
full diff: containerd/ttrpc@699c4e4...92c8520

changes:

- containerd/ttrpc#37 Handle EOF to prevent file descriptor leak
- containerd/ttrpc#38 Improve connection error handling
- containerd/ttrpc#40 Support headers
- containerd/ttrpc#41 Add client and server unary interceptors
- containerd/ttrpc#43 metadata as KeyValue type
- containerd/ttrpc#42 Refactor close handling for ttrpc clients
- containerd/ttrpc#44 Fix method full name generation
- containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up)
- containerd/ttrpc#49 Handle ok status

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this pull request Sep 3, 2019
full diff: containerd/ttrpc@699c4e4...92c8520

changes:

- containerd/ttrpc#37 Handle EOF to prevent file descriptor leak
- containerd/ttrpc#38 Improve connection error handling
- containerd/ttrpc#40 Support headers
- containerd/ttrpc#41 Add client and server unary interceptors
- containerd/ttrpc#43 metadata as KeyValue type
- containerd/ttrpc#42 Refactor close handling for ttrpc clients
- containerd/ttrpc#44 Fix method full name generation
- containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up)
- containerd/ttrpc#49 Handle ok status

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 8769255d1bb9c469d4f2966e7e9869a9f126f9e9
Component: engine
thaJeztah added a commit to thaJeztah/docker that referenced this pull request Sep 12, 2019
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7

> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d5669ec)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
thaJeztah added a commit to thaJeztah/docker that referenced this pull request Sep 12, 2019
full diff: containerd/ttrpc@699c4e4...92c8520

changes:

- containerd/ttrpc#37 Handle EOF to prevent file descriptor leak
- containerd/ttrpc#38 Improve connection error handling
- containerd/ttrpc#40 Support headers
- containerd/ttrpc#41 Add client and server unary interceptors
- containerd/ttrpc#43 metadata as KeyValue type
- containerd/ttrpc#42 Refactor close handling for ttrpc clients
- containerd/ttrpc#44 Fix method full name generation
- containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up)
- containerd/ttrpc#49 Handle ok status

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8769255)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this pull request Sep 12, 2019
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7

> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d5669ec1c6eedcd5dd8b0ecd615638934561daa4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 768923199f89246ff51039ae030e4b492f8d4555
Component: engine
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this pull request Sep 23, 2019
full diff: containerd/ttrpc@699c4e4...92c8520

changes:

- containerd/ttrpc#37 Handle EOF to prevent file descriptor leak
- containerd/ttrpc#38 Improve connection error handling
- containerd/ttrpc#40 Support headers
- containerd/ttrpc#41 Add client and server unary interceptors
- containerd/ttrpc#43 metadata as KeyValue type
- containerd/ttrpc#42 Refactor close handling for ttrpc clients
- containerd/ttrpc#44 Fix method full name generation
- containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up)
- containerd/ttrpc#49 Handle ok status

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8769255d1bb9c469d4f2966e7e9869a9f126f9e9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 525e8ed3febff46d07cb01961601824b5f8b301b
Component: engine
thaJeztah added a commit to thaJeztah/docker that referenced this pull request Sep 27, 2019
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7

> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d5669ec)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this pull request Sep 27, 2019
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7

> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby/moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d5669ec1c6eedcd5dd8b0ecd615638934561daa4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 8c7928adaa83264947b6e296eeb068b99843822e
Component: engine
burnMyDread pushed a commit to burnMyDread/moby that referenced this pull request Oct 21, 2019
From the release notes: https://github.com/containerd/containerd/releases/tag/v1.2.7

> Welcome to the v1.2.7 release of containerd!
>
> The seventh patch release for containerd 1.2 introduces OCI image
> descriptor annotation support and contains fixes for containerd shim logs,
> container stop/deletion, cri plugin and selinux.
>
> It also contains several important bug fixes for goroutine and file
> descriptor leakage in containerd and containerd shims.
>
> Notable Updates
>
> - Support annotations in the OCI image descriptor, and filtering image by annotations. containerd/containerd#3254
> - Support context timeout in ttrpc which can help avoid containerd hangs when a shim is unresponsive. containerd/ttrpc#31
> - Fix a bug that containerd shim leaks goroutine and file descriptor after containerd restarts. containerd/ttrpc#37
> - Fix a bug that a container can't be deleted if first deletion attempt is canceled or timeout. containerd/containerd#3264
> - Fix a bug that containerd leaks file descriptor when using v2 containerd shims, e.g. containerd-shim-runc-v1. containerd/containerd#3273
> - Fix a bug that a container with lingering processes can't terminate when it shares pid namespace with another container. moby#38978
> - Fix a bug that containerd can't read shim logs after restart. containerd/containerd#3282
> - Fix a bug that shim_debug option is not honored for existing containerd shims after containerd restarts. containerd/containerd#3283
> - cri: Fix a bug that a container can't be stopped when the exit event is not successfully published by the containerd shim. containerd/containerd#3125, containerd/containerd#3177
> - cri: Fix a bug that exec process is not cleaned up if grpc context is canceled or timeout. contaienrd/cri#1159
> - Fix a selinux keyring labeling issue by updating runc to v1.0.0-rc.8 and selinux library to v1.2.2. opencontainers/selinux#50
> - Update ttrpc to f82148331ad2181edea8f3f649a1f7add6c3f9c2. containerd/containerd#3316
> - Update cri to 49ca74043390bc2eeea7a45a46005fbec58a3f88. containerd/containerd#3330

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: zach <Zachary.Joyner@linux.com>
burnMyDread pushed a commit to burnMyDread/moby that referenced this pull request Oct 21, 2019
full diff: containerd/ttrpc@699c4e4...92c8520

changes:

- containerd/ttrpc#37 Handle EOF to prevent file descriptor leak
- containerd/ttrpc#38 Improve connection error handling
- containerd/ttrpc#40 Support headers
- containerd/ttrpc#41 Add client and server unary interceptors
- containerd/ttrpc#43 metadata as KeyValue type
- containerd/ttrpc#42 Refactor close handling for ttrpc clients
- containerd/ttrpc#44 Fix method full name generation
- containerd/ttrpc#46 Client.Call(): do not return error if no Status is set (gRPC v1.23 and up)
- containerd/ttrpc#49 Handle ok status

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: zach <Zachary.Joyner@linux.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants