Skip to content
This repository has been archived by the owner on Jan 8, 2024. It is now read-only.

Support for Caddyfile #5

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Support for Caddyfile #5

wants to merge 4 commits into from

Conversation

rigon
Copy link

@rigon rigon commented Jan 3, 2022

No description provided.

@vrongmeal
Copy link
Owner

Hey @rigon
This looks pretty good. I’ll try it out once and merge it today! Thanks for the PR

@mholt
Copy link

mholt commented Jan 4, 2022

Wow, that's awesome. Quite the PR! 😄

@joeworkman
Copy link

I would be onboard with testing this! I am completely new to Caddy and I need this plugin to replace my existing setup on my Apache server.

@vrongmeal
Copy link
Owner

Hey @joeworkman ! Definitely go for it :)

@joeworkman
Copy link

I am a super n00b with Caddy. How would I take the source to compile it into a custom Caddy build? I have used xcaddy. Although I learned that I need to get it installed on my linux server in order to get a custom build working on that server.

I am happy to test if you can help me figure out how to build it.

@mholt
Copy link

mholt commented Jan 5, 2022

You don't need to install it on your Linux server in order to use a custom build there... Where did you learn that?

You can compile caddy from any machine, for any machine. Instructions are on this page: https://caddyserver.com/docs/build

@joeworkman
Copy link

I thought that I would need to use xcaddy to build this on my linux server. When I built a custom build on my Mac earlier today, it would not run on my server (you helped me with that).

Its 1am now though. I'll check back in the morning.

@francislavoie
Copy link
Collaborator

I thought that I would need to use xcaddy to build this on my linux server. When I built a custom build on my Mac earlier today, it would not run on my server (you helped me with that).

You can do what's called a cross-build on any machine, to build for a different OS/architecture. You just need to set the GOOS and GOARCH environment variables to the appropriate one for the target machine you're building for. (It's also how the https://caddyserver.com/download page works, you choose the OS/arch from the selection and the build server will make you a build for that platform)

Assuming your Linux server is x86 (not ARM) and 64-bit, then you would do:

GOOS=linux GOARCH=amd64 xcaddy build --with ...

Caddy works on almost every platform combination. There's a list of all the OSes and architectures Go supports here: https://gist.github.com/asukakenji/f15ba7e588ac42795f421b48b8aede63

@@ -102,8 +104,122 @@ $ xcaddy build v2.1.1 \
}
}
```
As an handler within a route.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
As an handler within a route.
As a handler within a route.

Comment on lines +168 to +174
"match": [
{
"path": [
"/update"
]
}
]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest moving the matcher to above the handle, where it's more easily noticed. It would make it more apparent that the route is meant for a request to one specific path.

}
```

## Caddyfile
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we also need Caddyfile docs for the global option.

For more control use the following syntax (bear in mind, this options are different from v1):

git [<matcher>] [<repo>] [<path>] {
repo|url <repo>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I see the value in supporting both repo and url at the same time. I think we should just pick one. I don't think the "transition from v1 to v2" matters. Any choice here locks us in forever basically.

Caddy v2 is a complete rewrite compared to Caddy v1 so it's the perfect opportunity for making these kinds of decisions.

branch <branch>
auth_user <username>
auth_secret <password>
single_branch true|false
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For boolean options, usually we just make an option with no value that implies "turn on this thing" or "turn off this thing" as the opposite of the default. If the default for single_branch is false (in JSON), then the option could be named use_single_branch and drop the true|false value.

Comment on lines +200 to +201
command_after <command>
command_async true|false
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. Technically more than one command could be run when configured via JSON. There might be a better way to do this to also support multiple commands. Maybe something like:

command_after echo "hello world sync"
command_after echo "hello world async" {
	async
}
command_after {
	command echo "hello world async long way"
	async
}

So each time a command is defined, it would append it to the list, and each one could have the async flag set individually.

For implementation, basically you do a d.RemainingArgs() to get the command on the same-line, then for nesting := d.Nesting(); d.NextBlock(nesting); loop to get the options within. Throw an error if a command is defined twice (both via args and the block).

Comment on lines +196 to +197
service_type <service type>
service_interval <service interval>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs some explanation, I'm not sure what service types are acceptable as a reader. Is this ssh vs http?

@joeworkman
Copy link

I would love to test this today. How can I use xcaddy to build with the version from this PR?

@mholt
Copy link

mholt commented Jan 10, 2022

I think this should work:

$ xcaddy build --with github.com/vrongmeal/caddygit=github.com/rigon/caddygit

@joeworkman
Copy link

@mholt Thanks. That got me close. That errored out saying that I needed a local directory. I clone the other repo and an built it like this...

xcaddy build --with github.com/vrongmeal/caddygit=./caddygit

@joeworkman
Copy link

joeworkman commented Jan 10, 2022

I build caddy like this on my local Mac. Server is running Ubuntu.

GOOS=linux GOARCH=amd64 xcaddy build --with github.com/caddy-dns/cloudflare --with github.com/vrongmeal/caddygit/module/git=./caddygit

Here is my Caddyfile

{
    email email@example.com
    acme_dns cloudflare API_KEY_HERE
    order git before file_server
}
aspecthq.com {
    redir https://www.aspecthq.com{uri}
}
www.aspecthq.com {
    git /update "https://gitlab.com/joeworkman/aspecthq.com" /var/www/aspecthq.com
    root * /var/www/aspecthq.com
    log {
        output file /var/log/caddy/aspecthq.com/access.log {
            roll_size 3MiB
            roll_keep 5
            roll_keep_for 48h
        }
        format console
    }
    encode gzip zstd
    php_fastcgi unix//run/php/php8.1-fpm.sock
    file_server
    push
}

When I try to run caddy, I am getting the following error that the git directive is not registered.

adapting config using caddyfile: parsing caddyfile tokens for 'order': /etc/caddy/Caddyfile:13 - Error during parsing: git is not a registered directive

@mholt
Copy link

mholt commented Jan 10, 2022

Make sure you run the binary you built, not another version.

@joeworkman
Copy link

Yes. I have verified that it's the server being used.

CleanShot 2022-01-10 at 15 09 17

@mholt
Copy link

mholt commented Jan 10, 2022

I don't know what that is, but it looks like there are two there that look the same.

@joeworkman
Copy link

That is from the update-alternatives command from the Caddy docs...

https://caddyserver.com/docs/build#package-support-files-for-custom-builds-for-debianubunturaspbian

It does mean that /usr/bin/caddy is pointing to my version that is compile with Cloudflare and Git modules.

@joeworkman
Copy link

Taking my server out of the mix...

  • I built caddy just as above for my Mac
  • I saved the above Caddyfile to my Mac
  • Then I started the newly compiled caddy on my Mac pointing to that config file.
caddy run --config Caddyfile

Here is the error that it produces.

2022/01/10 23:40:37.377	INFO	using provided configuration	{"config_file": "Caddyfile", "config_adapter": ""}
run: adapting config using caddyfile: parsing caddyfile tokens for 'order': Caddyfile:4 - Error during parsing: git is not a registered directive

@mholt
Copy link

mholt commented Jan 10, 2022

That's odd. And that's the only Caddy binary on your Mac? I'd feel more sure with a direct path like ./caddy instead of PATH usage, i.e. caddy.

If that's not it, I'm not sure... never seen that happen before. And you have verified the output of caddy list-modules?

@joeworkman
Copy link

It's not my only caddy binary. However, if I run it with ./ it's the same results.

@joeworkman
Copy link

The list-modules command does not list git.

admin.api.load
admin.api.metrics
admin.api.reverse_proxy
caddy.adapters.caddyfile
caddy.config_loaders.http
caddy.listeners.tls
caddy.logging.encoders.console
caddy.logging.encoders.filter
caddy.logging.encoders.filter.delete
caddy.logging.encoders.filter.ip_mask
caddy.logging.encoders.filter.replace
caddy.logging.encoders.json
caddy.logging.encoders.single_field
caddy.logging.writers.discard
caddy.logging.writers.file
caddy.logging.writers.net
caddy.logging.writers.stderr
caddy.logging.writers.stdout
caddy.storage.file_system
http
http.authentication.hashes.bcrypt
http.authentication.hashes.scrypt
http.authentication.providers.http_basic
http.encoders.gzip
http.encoders.zstd
http.handlers.acme_server
http.handlers.authentication
http.handlers.encode
http.handlers.error
http.handlers.file_server
http.handlers.headers
http.handlers.map
http.handlers.metrics
http.handlers.push
http.handlers.request_body
http.handlers.reverse_proxy
http.handlers.rewrite
http.handlers.static_response
http.handlers.subroute
http.handlers.templates
http.handlers.vars
http.matchers.expression
http.matchers.file
http.matchers.header
http.matchers.header_regexp
http.matchers.host
http.matchers.method
http.matchers.not
http.matchers.path
http.matchers.path_regexp
http.matchers.protocol
http.matchers.query
http.matchers.remote_ip
http.matchers.vars
http.matchers.vars_regexp
http.precompressed.br
http.precompressed.gzip
http.precompressed.zstd
http.reverse_proxy.selection_policies.cookie
http.reverse_proxy.selection_policies.first
http.reverse_proxy.selection_policies.header
http.reverse_proxy.selection_policies.ip_hash
http.reverse_proxy.selection_policies.least_conn
http.reverse_proxy.selection_policies.random
http.reverse_proxy.selection_policies.random_choose
http.reverse_proxy.selection_policies.round_robin
http.reverse_proxy.selection_policies.uri_hash
http.reverse_proxy.transport.fastcgi
http.reverse_proxy.transport.http
pki
tls
tls.certificates.automate
tls.certificates.load_files
tls.certificates.load_folders
tls.certificates.load_pem
tls.certificates.load_storage
tls.handshake_match.remote_ip
tls.handshake_match.sni
tls.issuance.acme
tls.issuance.internal
tls.issuance.zerossl
tls.stek.distributed
tls.stek.standard

  Standard modules: 83

dns.providers.cloudflare

  Non-standard modules: 1

  Unknown modules: 0

Is this command correct?

xcaddy build --with github.com/caddy-dns/cloudflare --with github.com/vrongmeal/caddygit=./caddygit 

@mholt
Copy link

mholt commented Jan 10, 2022

Ah... no, it's not. After looking closely at the structure of this repo, it turns out the module registration is NOT in the root of this repository. My bad for assuming.

The module is registered in github.com/vrongmeal/caddygit/module/git: https://github.com/vrongmeal/caddygit/blob/main/module/git/app.go

That's a bit unusual, so I didn't think to check for it.

@joeworkman
Copy link

joeworkman commented Jan 11, 2022

OK. This command works...

xcaddy build --with github.com/vrongmeal/caddygit/module/git

But this does not...

xcaddy build  --with github.com/vrongmeal/caddygit/module/git=./caddygit 

I need to reference the local folder since it contains this PR.

@mohammed90
Copy link

mohammed90 commented Jan 11, 2022

Try:

xcaddy build  --with github.com/vrongmeal/caddygit/module/git=./caddygit/module/git

Edit: never mind...

github.com/vrongmeal/caddygit/module/git: module /Users/mohammed/caddygit/module/git: reading ../caddygit/module/git/go.mod: open /Users/mohammed/caddygit/module/git/go.mod: no such file or directory

The replace directive works on Go modules, not on packages, which is where the issue. We need to import github.com/vrongmeal/caddygit/module/git but replace github.com/vrongmeal/caddygit, which is tricky with xcaddy. Just a sec...

@mohammed90
Copy link

Got it. Try:

xcaddy build --with github.com/vrongmeal/caddygit/module/git  --with github.com/vrongmeal/caddygit=./caddygit

This will underscore-import github.com/vrongmeal/caddygit/module/git and github.com/vrongmeal/caddygit (the latter of which is harmless), but replaces github.com/vrongmeal/caddygit with the local directory.

@mohammed90
Copy link

I build caddy like this on my local Mac. Server is running Ubuntu.

GOOS=linux GOARCH=amd64 xcaddy build --with github.com/caddy-dns/cloudflare --with github.com/vrongmeal/caddygit/module/git=./caddygit

Here is my Caddyfile

{
    email email@example.com
    acme_dns cloudflare API_KEY_HERE
    order git before file_server
}
aspecthq.com {
    redir https://www.aspecthq.com{uri}
}
www.aspecthq.com {
    git /update "https://gitlab.com/joeworkman/aspecthq.com" /var/www/aspecthq.com
    root * /var/www/aspecthq.com
    log {
        output file /var/log/caddy/aspecthq.com/access.log {
            roll_size 3MiB
            roll_keep 5
            roll_keep_for 48h
        }
        format console
    }
    encode gzip zstd
    php_fastcgi unix//run/php/php8.1-fpm.sock
    file_server
    push
}

When I try to run caddy, I am getting the following error that the git directive is not registered.

adapting config using caddyfile: parsing caddyfile tokens for 'order': /etc/caddy/Caddyfile:13 - Error during parsing: git is not a registered directive

Adapter result (removing cloudflare):

{
	"logging": {
		"logs": {
			"default": {
				"exclude": [
					"http.log.access.log0"
				]
			},
			"log0": {
				"writer": {
					"filename": "/var/log/caddy/aspecthq.com/access.log",
					"output": "file",
					"roll_keep": 5,
					"roll_keep_days": 2,
					"roll_size_mb": 3
				},
				"encoder": {
					"format": "console"
				},
				"include": [
					"http.log.access.log0"
				]
			}
		}
	},
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						":443"
					],
					"routes": [
						{
							"match": [
								{
									"host": [
										"www.aspecthq.com"
									]
								}
							],
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "vars",
													"root": "/var/www/aspecthq.com"
												},
												{
													"encodings": {
														"gzip": {},
														"zstd": {}
													},
													"handler": "encode",
													"prefer": [
														"gzip",
														"zstd"
													]
												},
												{
													"handler": "push"
												}
											]
										},
										{
											"handle": [
												{
													"handler": "static_response",
													"headers": {
														"Location": [
															"{http.request.uri.path}/"
														]
													},
													"status_code": 308
												}
											],
											"match": [
												{
													"file": {
														"try_files": [
															"{http.request.uri.path}/index.php"
														]
													},
													"not": [
														{
															"path": [
																"*/"
															]
														}
													]
												}
											]
										},
										{
											"handle": [
												{
													"handler": "rewrite",
													"uri": "{http.matchers.file.relative}"
												}
											],
											"match": [
												{
													"file": {
														"split_path": [
															".php"
														],
														"try_files": [
															"{http.request.uri.path}",
															"{http.request.uri.path}/index.php",
															"index.php"
														]
													}
												}
											]
										},
										{
											"handle": [
												{
													"handler": "reverse_proxy",
													"transport": {
														"protocol": "fastcgi",
														"split_path": [
															".php"
														]
													},
													"upstreams": [
														{
															"dial": "unix//run/php/php8.1-fpm.sock"
														}
													]
												}
											],
											"match": [
												{
													"path": [
														"*.php"
													]
												}
											]
										},
										{
											"handle": [
												{
													"commands_after": [
														{}
													],
													"handler": "git",
													"hook": null,
													"repo": {
														"path": "/var/www/aspecthq.com",
														"url": "https://gitlab.com/joeworkman/aspecthq.com"
													}
												}
											],
											"match": [
												{
													"path": [
														"/update"
													]
												}
											]
										},
										{
											"handle": [
												{
													"handler": "file_server",
													"hide": [
														"./Caddyfile"
													]
												}
											]
										}
									]
								}
							],
							"terminal": true
						},
						{
							"match": [
								{
									"host": [
										"aspecthq.com"
									]
								}
							],
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"handler": "static_response",
													"headers": {
														"Location": [
															"https://www.aspecthq.com{http.request.uri}"
														]
													},
													"status_code": 302
												}
											]
										}
									]
								}
							],
							"terminal": true
						}
					],
					"logs": {
						"logger_names": {
							"www.aspecthq.com": "log0"
						},
						"skip_hosts": [
							"aspecthq.com"
						]
					}
				}
			}
		},
		"tls": {
			"automation": {
				"policies": [
					{
						"subjects": [
							"www.aspecthq.com",
							"aspecthq.com"
						],
						"issuers": [
							{
								"email": "email@example.com",
								"module": "acme"
							},
							{
								"email": "email@example.com",
								"module": "zerossl"
							}
						]
					}
				]
			}
		}
	}
}

@joeworkman
Copy link

That seems to have worked! I never would have guessed to try that.

@mholt
Copy link

mholt commented Jan 11, 2022

Thanks @mohammed90.

@joeworkman It's a nuance of how Go modules does replacements, I guess.

@vrongmeal Most Caddy modules have their registrations in the root of the repo, that does simplify some things. Please consider that, if you would :)

@joeworkman
Copy link

So should this module not be using github.com/vrongmeal/caddygit/module/git? I assume that changing it to be github.com/vrongmeal/caddygit/ would break some backward compatibility? Or not since it's built into static binaries anyways.

@joeworkman
Copy link

joeworkman commented Jan 11, 2022

I have been trying to figure out the configuration here. I tried to look through the code of the plugin to figure out what is going on. I am completely new to Caddy and Go so it's a bit of overload. Here is what I have so far...

www.aspecthq.com {
    root * /var/www/aspecthq.com
    encode gzip zstd
    php_fastcgi unix//run/php/php8.1-fpm.sock
    file_server
    push
    git /update {
        url https://gitlab.com/joeworkman/aspecthq.com
        path /var/www/aspecthq.com
        branch master
    }
}

I thought that this may mean that if you go to URL https://aspecthq.com/update that it would simply trigger a git pull. That is not happening. This site is currently on GitLab but can easily move it to Github if I need to.

@joeworkman
Copy link

joeworkman commented Jan 11, 2022

I see this error in syslog

Jan 10 21:15:03 **** caddy[120153]: {"level":"error","ts":1641867303.8984256,"logger":"http.handlers.git","msg":"repository not setup","error":"cannot setup repository: worktree contains unstaged changes","path":"/var/www/aspecthq.com"}

It says that there are unstaged changes. However, there aren't any changes in /var/www/aspecthq.com

@joeworkman
Copy link

@vrongmeal @rigon Can one of you help me with the config? I will be writing a full tutorial on my setup to help others and will be happy to help update the docs.

@joeworkman
Copy link

joeworkman commented Jan 11, 2022

I will put my money where my mouth is guys... $100 to the guy/gal who helps me get this working. 😃

@mohammed90
Copy link

mohammed90 commented Jan 11, 2022

I will put my money where my mouth is guys... $100 to the guy who helps me get this working. 😃

Run:

git -C /var/www/aspecthq.com status

It will most likely say something like:

On branch master
Your branch is ahead of 'origin/master' by 6 commits.
  (use "git push" to publish your local commits)

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   ...

You should handle those unstaged/uncommitted changes. How to handle them depends on your goal, but there's one way here.

@joeworkman
Copy link

working tree is clean. I have one commit to pull down. I did that on purpose in order to test the plugin.

git -C /var/www/aspecthq.com status
On branch master
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)

nothing to commit, working tree clean

I am pretty sure that the issue is with the config (or the plugin)

@greenpau
Copy link

I will put my money where my mouth is guys... $100 to the guy/gal who helps me get this working.

@joeworkman . I will take it! 👍 What do you want to get done? :-)

@joeworkman
Copy link

@greenpau I am just trying to get this plugin working. I have my caddy server built with Cloudflare and the code from this PR. This PR adds Caddyfile support for this git plugin.

I simply cannot get the plugin to work. I don't know if it's my config or the plugin itself. I am attempting to get the web hook to work. I would expect that when I make a request to the /update url (config above) that the plugin would then do it's thing and do a git pull on the repo in order to update the site.

I am all ears...

@greenpau
Copy link

@joeworkman , get the $100 ready ;-) will do it in a few hours.

@joeworkman
Copy link

I am a man of my word! Let's git-r-done... (see what I did there?) LOL

@greenpau
Copy link

I am a man of my word! Let's git-r-done... (see what I did there?) LOL

@joeworkman , please track this issue.

@greenpau
Copy link

@joeworkman, I also agreed to it because I love Foundation! 👍

@joeworkman
Copy link

Sweet! I may not be able to play around with it until the morning.

I'm glad that you love Foundation. Guess what the new website will be running on? 😃

@francislavoie
Copy link
Collaborator

francislavoie commented Jan 12, 2022

@joeworkman, please track this issue.

So... it looks like you decided to completely write a new plugin from scratch? Huh?

@greenpau
Copy link

So... it looks like you decided to completely write a new plugin from scratch? Huh?

@francislavoie , for the thing to work I needed to both app and handler. I have a template to do it already. Unfortunately doing it here might take more time than I have.

@greenpau
Copy link

@joeworkman , just to be clear. The github.com/greenpau/caddy-git does not have all the fancy features, but does what you want. What I just created will get you started, but this plugin (github.com/vrongmeal/caddygit) could be a better option for the advanced features. Just embrace the variety/diversity of plugins for now 😄

P.S It took me a while to get Caddyfile format supported.

@mholt
Copy link

mholt commented Jan 12, 2022

Wow, impressive, Paul! Maybe, do you think we could combine the functionality back into this plugin? So there's only 1 git plugin (less confusing, less divergence)?

@greenpau
Copy link

do you think we could combine the functionality back into this plugin?

@mholt , i tried doing that initially, but after about 30 minutes I realized that would be a 75% rewrite. The other 25% are the git specific functions. The issue here is that caddy app/plugin code is intertwined with the functional (git) code. Thus, to scale and be more readable, it needs to be decoupled. One thing to do something for sport for an hour, another spending days going through such a rewrite. Plus maintainer availability, i.e. reviewing 1000+ lines of code, etc.

@greenpau
Copy link

In re: caddy app/plugin code is intertwined with the functional (git) code, Here are some principles I use when working with plugins.

  1. All references to github.com/caddyserver/caddy/ must be in in root directory only (and perhaps /cmd), e.g. caddyfile.go, app.go, plugin.go
  2. The nested directories SHOULD NOT contain references to github.com/caddyserver/caddy/
  3. If using Gin or other Go-based frameworks, rry maintaining the code supporting the plugin in a separate repo (i don't follow it, but really should).

@joeworkman
Copy link

I just wanted to follow up. @greenpau's new plugin is completely working for me now. As promised, I sent him the $100 last night. 💸 😄

@Magicking
Copy link

Happy NY, any update on seeing this merged?
Ty

@vrongmeal
Copy link
Owner

Happy NY, any update on seeing this merged? Ty

Hey! I don’t plan on working on this. I encourage you to use the git plugin by @greenpau : https://github.com/greenpau/caddy-git which is more actively maintained.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants