proposal: os: make the internal lockedfile package public #33974

azr · 2019-08-30T12:50:35Z

An internal filelock pkg can be found here:
https://pkg.go.dev/cmd/go/internal/lockedfile/internal/filelock

A few projects are implementing file locking [1] but they do not seem to be maintained and I think they are not as nice as the internal filelock pkg I mentioned before. As a result some projects are doing their own version of it [2].

I suggest we make the filelock pkg public as I think this could be beneficial to the mass.

I'd be glad to do it given some pointers, like where to put it ?

Thanks !

[1] File lock projects:

https://github.com/gofrs/flock
https://github.com/MichaelS11/go-file-lock
https://github.com/juju/fslock

[2] Projects that implement their own file locking:

terraform

boltdb

mvdan · 2019-08-30T15:09:42Z

Why not use the exported version at https://godoc.org/github.com/rogpeppe/go-internal? :)

azr · 2019-09-03T13:27:28Z

Hey @mvdan, 🙂 here are a few reasons why I didn't go with it:

the filelock of go-internal is in an internal pkg
importing go-internal makes my go-deps bigger than necessary ( edit: this not such a big issue )
it feels wrong, if I find a bug, then I have to make a pull request to go and then wait for it to get merged and then wait for go-internals to update. It sounds like it has all the annoying effects of having these pkgs not internal anyway.

mvdan · 2019-09-03T13:41:49Z

the filelock of go-internal is in an internal pkg

I'm not sure why that is, perhaps @rogpeppe or @myitcv can answer. Have you looked at https://godoc.org/github.com/rogpeppe/go-internal/lockedfile? That seems like a public, higher-level version of it.

That aside, sure, there are some upsides to having this in the standard library. But I think the downsides are generally greater:

Once a package is public, it can never have backwards incompatible changes
The standard library can grow, but never shrink in size
Once the number of users greatly outgrows just cmd/go, the maintainers will probably have extra work responding to issues, reviewing patches, and fixing bugs that might not even affect cmd/go

I think a far better proposal would be to include this in one of the official external repos, like x/sync. Once that's worked well for a while, then it can be part of the public standard library. This is the path that context took, for example.

azr · 2019-09-03T14:20:16Z

lockedfile fits right to our usage and lockedfile doesn't have the filelock.ErrNotSupported. But yes lockedfile could work here.

Yes, I agree this should probably be in an x/ pkg 🙂, I only suggested we make the filelock pkg public and agree this would be a bit quick to put in the stdlib.

ianlancetaylor · 2019-09-03T20:31:08Z

CC @bcmills

bcmills · 2019-09-03T21:25:59Z

At this point we've been using lockedfile for a full release cycle and it seems mostly fine, modulo what appears to be an AIX kernel bug (#32817). I'd be fine with making it public.

Then the question is where to put it: x/sys, x/exp, x/sync, or someplace else entirely?

azr · 2019-09-04T08:56:50Z

It's right at the intersection of sync and sys.
I think lockedfile should be in x/sync/lockedfile
I think that filelock is a bit lower level and could fit in x/sys/filelock ( if we ever make it public ). But these two are may be a bit redundant.

edit: regroup them both under x/exp/syssync ?

bcmills · 2019-09-04T12:55:07Z

@azr, filelock should remain internal — it's much too subtle to use on its own. (In particular, if you want your program to remain portable, you have to be very careful to stick to a narrow subset of the possible modes of use.)

If folks discover other interesting use-cases for filelock, then we can probably add more ergonomic packages to address those use-cases while still keeping filelock internal.

rsc · 2019-09-25T17:29:49Z

Given that the author of filelock says it is not ready to be exported, it seems like this is a likely decline.

Leaving open for a week for final comments.

bcmills · 2019-09-25T18:39:09Z

To be clear: I support making the lockedfile package itself public — I just want to ensure that the lockedfile/internal/filelock package remains internal to it.

azr · 2019-09-26T09:46:45Z

I just want to ensure that the lockedfile/internal/filelock package remains internal to it.

That's fine by me 🙂.

I think I'll put it in x/sync/lockedfile soonish; (unless you think another place is better ).

rsc · 2019-10-02T17:15:19Z

I misunderstood. Thanks for the clarification, @bcmills. Removing the FinalCommentPeriod label.

rsc · 2019-10-02T17:18:55Z

I don't think we are ready to add lockedfile to the standard library. It is a peculiar set of functions that does not really line up with the usual things in either os or io. Maybe it would be OK in x/sync but maybe it would be better to start in x/exp to understand if the API needs any adjustments.

If we are going to add a new public package (even in x/exp), I think the next step would be for someone to write a proper proposal doc laying out the API and explaining the decisions, get comments, and so on.

@azr, do you want to do that?

azr · 2019-10-03T09:35:13Z

@rsc gotcha ! Yes I want to ! Where should such a proposal doc be written ? Is there a template for that ?

bcmills · 2019-10-03T12:55:52Z

@azr, see https://github.com/golang/proposal#design-documents

rsc · 2019-10-09T17:53:18Z

Putting proposal on hold for design doc.

azr · 2019-10-15T18:41:55Z

Hello there; I created the proposal ( first ever ! ) here golang/proposal#21; I hope it's good enough please feel free to give feedback 🙂 🙂 !

Updates golang/go#33974

ianlancetaylor · 2021-01-08T20:26:46Z

It's in the "incoming" list. We move proposals from "incoming" to "active" as we have bandwidth to handle them. See https://go.googlesource.com/proposal/+/refs/heads/master/README.md .

arl · 2021-01-08T20:30:39Z

@ianlancetaylor I had missed that bit of information. Thank you

tamilmani1989 · 2021-10-15T23:00:42Z

any update on making lockedfile package public?

kensipe · 2022-02-07T22:34:33Z

Some the details seem buried... The proposal process completed and landed the proposal on Mar 10, 2020: https://github.com/golang/proposal/blob/master/design/33974-add-public-lockedfile-pkg.md

The proposal looks good. It is unclear how / who to involve to move past proposal. Any insight @azr ? Happy to help if there is a need.

ianlancetaylor · 2022-02-07T23:37:08Z

@kensipe This proposal has still not been accepted. I'll put it on the list for faster consideration.

zpavlinovic · 2022-03-31T18:44:47Z

vuln/cmd/govulncheck could also make use of this library.

jastBytes · 2022-08-18T11:41:25Z

Is it going to be public anytime soon?

The logic here is racy since multiple endtoend tests can create the file at around the same time and allocate the same base port. This then leads to flaky failing endtoend tests since not all ports can be used. Even with the listening check later on, this is still racy because the listen check might run at a later point still trying to allocate the same port for the same service in different endtoend test. This copies the Go filelock implementation, but only for unix systems since that's the only supported platform for Vitess. Go has a proposal open to make filelock available, but that is still pending in golang/go#33974. Once that is resolved, we can remove our copy of the implementation. Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>

* Fix race in determining start port for cluster The logic here is racy since multiple endtoend tests can create the file at around the same time and allocate the same base port. This then leads to flaky failing endtoend tests since not all ports can be used. Even with the listening check later on, this is still racy because the listen check might run at a later point still trying to allocate the same port for the same service in different endtoend test. This copies the Go filelock implementation, but only for unix systems since that's the only supported platform for Vitess. Go has a proposal open to make filelock available, but that is still pending in golang/go#33974. Once that is resolved, we can remove our copy of the implementation. Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> * Change range of dynamic ports to not collide with tests Next to the previous commit which fixes the race base port computation, we also need to avoid collisions between ephemeral outgoing ports to 127.0.0.1 and the listening ones. The problem is that we set 1024 as the first ephemeral port, but that means we can collide and trigger flaky tests with all the tests that allocate a server port as well. Given we start allocating server ports now starting at 6700 for endtoend tests and we allocate in blocks of 200, we have room for 75+ concurrent endtoend tests which should totally suffice. We still have a range of 40+ ephemeral ports which should still be sufficient as well. Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com> Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>

Running two instances of the server which share the same directory (e.g. default configuration `/tmp/nats/jetstream') will corrupt each other. We mitigate by creating an empty file called LOCK in the directory and then acquire a posix `flock(2)` on it; a common pattern. Note this is not inherently cross-platform without golang/go#33974 Signed-off-by: Jason Volk <jason@zemos.net>

gopherbot added this to the Proposal milestone Aug 30, 2019

gopherbot added the Proposal label Aug 30, 2019

smasher164 changed the title ~~Proposal: make the internal filelock pkg public~~ proposal: make the internal filelock package public Aug 30, 2019

griesemer mentioned this issue Sep 3, 2019

proposal: review meeting minutes #33502

Open

azr mentioned this issue Sep 5, 2019

circle-ci: add tests for windows and macos & test using go 1.13 hashicorp/packer#8078

Merged

rsc added the Proposal-FinalCommentPeriod label Sep 25, 2019

azr changed the title ~~proposal: make the internal filelock package public~~ proposal: make the internal lockedfile package public Sep 26, 2019

rsc removed the Proposal-FinalCommentPeriod label Oct 2, 2019

rsc added the Proposal-Hold label Oct 9, 2019

azr mentioned this issue Oct 15, 2019

design: add proposal doc for 33974 golang/proposal#21

Closed

azr added a commit to azr/proposal that referenced this issue Oct 16, 2019

design: add proposal doc for 33974

6898178

Updates golang/go#33974

azr added a commit to azr/proposal that referenced this issue Oct 16, 2019

design: add proposal doc for 33974

0c9e80a

Updates golang/go#33974

azr added a commit to azr/proposal that referenced this issue Oct 16, 2019

design: add proposal doc for 33974

4207027

Updates golang/go#33974

bcmills mentioned this issue Oct 29, 2021

x/vulndb: build broken on aix-ppc64 since 2021-10-18 due to "undefined: unix.Flock" in github.com/go-git/go-billy #49218

Closed

nicksieger mentioned this issue Dec 3, 2021

Error: writing tilt api configs: open /home/$USER/.tilt-dev/config.lock: file exists tilt-dev/tilt#4814

Closed

klihub mentioned this issue Jan 4, 2022

resmgr: write a PID file upon successful startup. intel/cri-resource-manager#756

Merged

binderclip mentioned this issue Jan 12, 2022

Go 用文件实现一个跨进程的 Mutex binderclip/clip#6

Closed

kensipe mentioned this issue Feb 10, 2022

Platform-specific file locking for kubeconfig kubernetes/kubernetes#107354

Closed

This was referenced Mar 26, 2022

Create a proper Lock/Unlock rwxrob/fs#2

Closed

Loader should use standard file-level locking kubernetes/client-go#1073

Closed

sverdlov93 mentioned this issue Mar 29, 2022

Fixing config file lock mechanism jfrog/jfrog-cli-core#352

Closed

3 tasks

FredrikAppelros mentioned this issue Jun 15, 2022

Allow bundle persistence in cloud native environments open-policy-agent/opa#4782

Closed

amustaque97 mentioned this issue Jul 19, 2022

Corruption on simultanous calls to set config token datreeio/datree#725

Closed

rsc changed the title ~~proposal: make the internal lockedfile package public~~ proposal: os: make the internal lockedfile package public Aug 10, 2022

uhppoted mentioned this issue Nov 15, 2022

Handling of lockfile on crash/restart uhppoted/uhppoted-tunnel#7

Closed

dbussink mentioned this issue Nov 16, 2022

Fix two additional flaky test sources in endtoend tests vitessio/vitess#11743

Merged

3 tasks

mvdan mentioned this issue Jan 7, 2023

random magic value using modified cmd/link alternative burrowers/garble#628

Merged

nicks mentioned this issue Feb 2, 2023

v4.16.2: docker context list gets "unexpected end of JSON input" docker/for-win#13180

Closed

3 tasks

agouin mentioned this issue Feb 21, 2023

migrate to flock for config lock cosmos/relayer#1104

Merged

nicks mentioned this issue Feb 22, 2023

context: adjust the file write logic to avoid corrupt context meta.json files docker/cli#4042

Merged

bcmills mentioned this issue Jul 12, 2023

proposal: cmd/go: add an option to sequentially run tests #61318

Open

thallgren mentioned this issue Sep 4, 2023

Add process wide concurrency protection to Telepresence cache files. telepresenceio/telepresence#3335

Closed

bcmills mentioned this issue Dec 12, 2023

x/sys/unix: missing some AIX syscalls, notably Flock #64669

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

proposal: os: make the internal lockedfile package public #33974

proposal: os: make the internal lockedfile package public #33974

azr commented Aug 30, 2019 •

edited by dmitshur

Loading

mvdan commented Aug 30, 2019 •

edited

Loading

azr commented Sep 3, 2019 •

edited

Loading

mvdan commented Sep 3, 2019

azr commented Sep 3, 2019

ianlancetaylor commented Sep 3, 2019

bcmills commented Sep 3, 2019

azr commented Sep 4, 2019 •

edited

Loading

bcmills commented Sep 4, 2019

rsc commented Sep 25, 2019

bcmills commented Sep 25, 2019 •

edited

Loading

azr commented Sep 26, 2019

rsc commented Oct 2, 2019

rsc commented Oct 2, 2019

azr commented Oct 3, 2019

bcmills commented Oct 3, 2019

rsc commented Oct 9, 2019

azr commented Oct 15, 2019

ianlancetaylor commented Jan 8, 2021

arl commented Jan 8, 2021

tamilmani1989 commented Oct 15, 2021

kensipe commented Feb 7, 2022

ianlancetaylor commented Feb 7, 2022

zpavlinovic commented Mar 31, 2022

jastBytes commented Aug 18, 2022

proposal: os: make the internal lockedfile package public #33974

proposal: os: make the internal lockedfile package public #33974

Comments

azr commented Aug 30, 2019 • edited by dmitshur Loading

mvdan commented Aug 30, 2019 • edited Loading

azr commented Sep 3, 2019 • edited Loading

mvdan commented Sep 3, 2019

azr commented Sep 3, 2019

ianlancetaylor commented Sep 3, 2019

bcmills commented Sep 3, 2019

azr commented Sep 4, 2019 • edited Loading

bcmills commented Sep 4, 2019

rsc commented Sep 25, 2019

bcmills commented Sep 25, 2019 • edited Loading

azr commented Sep 26, 2019

rsc commented Oct 2, 2019

rsc commented Oct 2, 2019

azr commented Oct 3, 2019

bcmills commented Oct 3, 2019

rsc commented Oct 9, 2019

azr commented Oct 15, 2019

ianlancetaylor commented Jan 8, 2021

arl commented Jan 8, 2021

tamilmani1989 commented Oct 15, 2021

kensipe commented Feb 7, 2022

ianlancetaylor commented Feb 7, 2022

zpavlinovic commented Mar 31, 2022

jastBytes commented Aug 18, 2022

azr commented Aug 30, 2019 •

edited by dmitshur

Loading

mvdan commented Aug 30, 2019 •

edited

Loading

azr commented Sep 3, 2019 •

edited

Loading

azr commented Sep 4, 2019 •

edited

Loading

bcmills commented Sep 25, 2019 •

edited

Loading