Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature Request: Operating system interface for file server #3720

Closed
ss098 opened this issue Sep 12, 2020 · 8 comments · Fixed by #4909
Closed

Feature Request: Operating system interface for file server #3720

ss098 opened this issue Sep 12, 2020 · 8 comments · Fixed by #4909
Labels
feature ⚙️ New feature or request
Milestone
v2.6.0

Comments

@ss098
Copy link

ss098 commented Sep 12, 2020 •
edited
Loading

Currently, File server module only support file system.

I think the File server module could support more data sources, such as S3 or elsewhere.

This could be done by implementing a Certmagic storage interface, or roughly an OS compatible interface.
@ss098 ss098 changed the title Feature Request: Operating system interface for file servers Feature Request: Operating system interface for file server Sep 12, 2020
@francislavoie francislavoie added discussion 💬 The right solution needs to be found feature ⚙️ New feature or request labels Sep 12, 2020
@francislavoie
Copy link
Member

francislavoie commented Sep 12, 2020 •
edited
Loading

Did a bit of googling, https://github.com/spf13/afero/ looks like a pretty solid option for a filesystem abstraction layer.

The S3 support doesn't look super certain though, from what I'm reading, but this is the most recent iteration: https://github.com/fclairamb/afero-s3, discussion here: spf13/afero#90 (comment)

If you'd like to give a shot at implementing this (maybe separate from the existing file_server handler; if it pans out well and ends up backwards compatible, we could potentially swap it in replacing the existing one), that would be cool.

I can already think of a few neat features we could get from using afero, e.g. optional cache-to-memory-on-read, so repeat requests to the same file would hit memory instead of disk (this is super easy, it's basically 3 lines to set up with afero, see the readme).

Also, note that there's also the file matcher that reads from disk to make decisions whether a handler should apply - this would also need to use the configured filesystem backend

@mholt
Copy link
Member

mholt commented Sep 12, 2020 •
edited
Loading

I think we should wait and see if the Go team continues to develop a mature FS abstraction: https://go.googlesource.com/proposal/+/master/design/draft-iofs.md

@francislavoie
Copy link
Member

This has landed in Go 1.16! https://golang.org/doc/go1.16#fs

We'll need to wait until we bump Caddy's minimum Go version to 1.16. We just bumped to minimum of 1.15, so if we go by that pattern of supporting the past two versions of Go, then that might be 6 months from now after 1.17 is released. TBD.

@francislavoie
Copy link
Member

Now that we've bumped to minimum of Go 1.16, this is now a refactor we could do!

This isn't a top priority, so if some enterprising individual wants to give us a hand, this would be a good issue to pick up!

@francislavoie francislavoie added the help wanted 🆘 Extra attention is needed label Aug 23, 2021
mholt added a commit that referenced this issue Jul 25, 2022
@mholt
fileserver: Support virtual file systems (close #3720)
ca27a0a 
This change replaces the hard-coded use of os.Open() and os.Stat() with
the use of the new (Go 1.16) io/fs APIs, enabling virtual file systems.
It introduces a new module namespace, caddy.fs, for such file systems.

Also improve documentation for the file server. I realized it was one of
the first modules written for Caddy 2, and the docs hadn't really been
updated since!
@mholt mholt mentioned this issue Jul 25, 2022
Merged
@mholt
Copy link
Member

mholt commented Jul 25, 2022

@ss098 You will like this: #4909

@mholt mholt removed help wanted 🆘 Extra attention is needed discussion 💬 The right solution needs to be found labels Jul 25, 2022
@mholt mholt added this to the v2.6.0 milestone Jul 25, 2022
@ss098
Copy link
Author

ss098 commented Jul 25, 2022

@mholt I'm excited about the implementation of this feature!!!

@hairyhenderson
Copy link
Collaborator

FYI - https://github.com/hairyhenderson/go-fsimpl provides some useful (IMO) filesystems for io/fs 😉

@mholt
Copy link
Member

mholt commented Jul 27, 2022

@hairyhenderson That's very cool!

mholt added a commit that referenced this issue Jul 30, 2022
@mholt
fileserver: Support virtual file systems (#4909)
6668271 
* fileserver: Support virtual file systems (close #3720)

This change replaces the hard-coded use of os.Open() and os.Stat() with
the use of the new (Go 1.16) io/fs APIs, enabling virtual file systems.
It introduces a new module namespace, caddy.fs, for such file systems.

Also improve documentation for the file server. I realized it was one of
the first modules written for Caddy 2, and the docs hadn't really been
updated since!

* Virtualize FS for file matcher; minor tweaks

* Fix tests and rename dirFS -> osFS

(Since we do not use a root directory, it is dynamic.)
@francislavoie francislavoie modified the milestones: v2.6.0, v2.6.0-beta.1 Aug 21, 2022
@mholt mholt modified the milestones: v2.6.0-beta.1, v2.6.0 Sep 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature ⚙️ New feature or request
Projects
None yet
Milestone
v2.6.0
Development

Successfully merging a pull request may close this issue.

fileserver: Support virtual file systems caddyserver/caddy
4 participants
@mholt @hairyhenderson @francislavoie @ss098