Any events on webhook send twice

[matrozov]

• Gitea version (or commit ref): 1.11.0+dev-331-g055f6d229
• Git version: not needed
• Operating system: Docker container from gitea/gitea
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
• Log gist:

Description

I deployed gitea in my kubernetes from the docker image of gitea/gitea. I linked the webhook to the created repository and noticed that all events are fired twice (push, create / delete branch and tags, etc.). The pull-request event was triggered as many as 4 times per action, but there is a separate issue for this #9181. In the images below, I pushed a new brunch and received two push and brunch creation events each. Webhook has been configured in both "All Events" and "Push events" mode. In both cases, push events arrived twice, and in "All events" mode all events came twice.

Screenshots

[lunny]

Have you add webhooks on both repository webhook setting and orgnization webhook setting?

[matrozov]

In the beginning, I used only the settings in the repository. Then, for the test, webhook was also added to the global settings, but this did not affect the double sending in any way. Duplicate notifications were from the start. Subsequent experiments with deleting, adding webhook, repositories and other types of events did not change the situation.

[lunny]

I cannot reproduce this on my local instance. Could you reproduce this on try.gitea.io ?

[matrozov]

Not. I tried, but on try.gitea.io I received only neighboring bugs #9184, #9181 and #9180. I will try to deploy gitea on a separate host again today. Perhaps something is tied to the initial settings.

[matrozov]

I received confirmation that the duplication of webhook depends on the installation configuration. I deployed on the local machine about 10 times and got different results depending on whether I redefined the parameters through ENV. In my product version in kubernetes, where I initially received an error, I also deployed with parameters predefined via ENV.

I attach my docker-compose.yaml file, as well as the resulting app.ini. At the installation stage, I only specify ssh_domain and administrator authorization settings. As far as I could understand, if you do not pass EMV when deploying, then gitea does not duplicate webhook. I did not have enough time to figure out more specifically which parameter in the deployment leads to such an error. I hope you can repeat my experiment.

test3.zip

After the deployment and installation, I did:

• added ssh key to user profile
• created a new repository
• added a webhook to this repository with all the default parameters (specified only url)
• cloned the repository locally (via ssh)
• did push

[stklcode]

I experience the same issue. Every webhook (types "Slack" and "Gitea" in use, all added at repository level) is triggered twice when pushing from Git client (SSH or HTTPS). As far as I can tell, it happens on any repository.
Same JSON body (except for the timestamp +/- 1 second), same event-header, different delivery-header.

I'm running two separate instances. Both same version and about the same configuration, updated regularily since 1.1 or 1.2. Sadly I cannot remember when it started.

Faulty instance:
Version: 1.10.1
Platform: Linux, amd64 (Debian Stretch, Kernel 3.16)
DB: MySQL (MariaDB 10.1)
Deployment: Direct setup (Downloaded from https://dl.gitea.io)
Configuration:

app.ini

APP_NAME = git @ mydomain.example.com
RUN_USER = git
RUN_MODE = prod

[database]
DB_TYPE  = mysql
HOST     = /var/run/mysqld/mysqld.sock
NAME     = gitea
USER     = gitea
PASSWD   = ***
SSL_MODE = disable
PATH     = data/gitea.db

[repository]
ROOT = /opt/gitea/repo

[server]
SSH_DOMAIN             = git.mydomain.example.com
PROTOCOL               = unix
HTTP_ADDR              = /opt/gitea/gitea.sock
HTTP_PORT              = 3000
UNIX_SOCKET_PERMISSION = 666
ROOT_URL               = https://git.mydomain.example.com/
DISABLE_SSH            = false
DISABLE_HTTP_GIT       = true
SSH_PORT               = 2222
SSH_LISTEN_PORT        = 2222
START_SSH_SERVER       = true
OFFLINE_MODE           = true
LANDING_PAGE           = explore

[mailer]
ENABLED = true
HOST    = mail.example.com:587
FROM    = "Git" <git@example.com>
USER    = git@example.com
PASSWD  = ***

[service]
REGISTER_EMAIL_CONFIRM = false
ENABLE_NOTIFY_MAIL     = true
DISABLE_REGISTRATION   = true
ENABLE_CAPTCHA         = false
REQUIRE_SIGNIN_VIEW    = false

[picture]
DISABLE_GRAVATAR        = true
ENABLE_FEDERATED_AVATAR = false

[session]
PROVIDER      = file
COOKIE_SECURE = true
COOKIE_NAME   = git_session

[log]
MODE      = file
LEVEL     = Info
ROOT_PATH = /opt/gitea/log

[security]
INSTALL_LOCK       = true
SECRET_KEY         = ***
INTERNAL_TOKEN     = ***
IMPORT_LOCAL_PATHS = false

[other]
SHOW_FOOTER_BRANDING           = false
SHOW_FOOTER_VERSION            = false
SHOW_FOOTER_TEMPLATE_LOAD_TIME = false

[oauth2]
JWT_SECRET = ***

Working instance:
Version: 1.10.1
Platform: Linux, amd64 (Debian Buster, Kernel 4.19)
Deployment: Docker Stack (official Image gitea/gitea:1.10, only USER_UID, USER_GID env)
DB: MySQL (MariaDB 10.3)
Configuration:

app.ini

APP_NAME = git @ otherdomain.example.com
RUN_MODE = prod

[repository]
ROOT = /data/git/repositories

[repository.upload]
TEMP_PATH = /data/gitea/uploads

[server]
APP_DATA_PATH   = /data/gitea
SSH_DOMAIN      = otherdomain.example.com
HTTP_PORT       = 3000
ROOT_URL        = https://otherdomain.example.com/
DISABLE_SSH     = false
SSH_PORT        = 2222
SSH_LISTEN_PORT = 22

[database]
DB_TYPE  = mysql
HOST     = /var/run/mysqld/mysqld.sock
NAME     = gitea
USER     = gitea
PASSWD   = ***

[session]
PROVIDER_CONFIG = /data/gitea/sessions

[mailer]
ENABLED = false

[picture]
AVATAR_UPLOAD_PATH      = /data/gitea/avatars
DISABLE_GRAVATAR        = true
ENABLE_FEDERATED_AVATAR = false

[log]
MODE      = file
LEVEL     = Warn
ROOT_PATH = /data/gitea/log

[security]
INSTALL_LOCK   = true
SECRET_KEY     = ***
INTERNAL_TOKEN = ***

[other]
SHOW_FOOTER_BRANDING           = false
SHOW_FOOTER_VERSION            = false
SHOW_FOOTER_TEMPLATE_LOAD_TIME = false

[oauth2]
JWT_SECRET = ***

[stklcode]

I've done some additional testing on the first of the above instances.

Two repositories (real world, no test dummies), both 1 Slack webhook and 1 Gitea webhook. Same configuration, same owner, same commit (dummy file on new temp-branch).

One repository (the slightly older one) behaves correcly, the other triggers twice. Both logs (Trace level) attached:
https://gist.github.com/stklcode/0e09e79033173413d55b9f800916a7e3

I cannot debug in that instance, as it is live and prod, but I had a look into the code.
DeliverHooks() iterates over undelivered tasks twice. For some reason the hooks appear in both selects, i.e. they are either places placed twice in the DB before or not marked as delivered.
deliver.go in master:

gitea/modules/webhook/deliver.go

Line 152 in 2b0e4e0

func DeliverHooks(ctx context.Context) {

webhooks.go in v1.10:

gitea/models/webhook.go

Line 930 in f0f48e0

func DeliverHooks() {

[lunny]

I take a look at two implementation on master and v1.10. It's the same.

[stklcode]

I've updated from 1.10.1 to 1.10.2 today. No change.

However by chance I've found the origin of the bug, because I have migrated my Gitea to a different directory structure and had to re-synchronize all hook scripts.

Apparently there are 2 identical hook files for pre- and post-receive phase, one named gitea (the correct one) and one named like the phase.

<repo>/hooks/
 |- pre-receive.d/
 |-- gitea
 |-- pre-receive
 |- post-receive.d/
 |-- gitea
 |-- post-receive

(not for update)

Last modification of the "wrong" scripts is somewhere back in 2017, the gitea scripts are modified today (I've triggered the "synchronize ... hooks for all repositories" task).

So the hook is really triggered twice externally.

[lunny]

pre-receive.d/gitea is managed by gitea internal. pre-receive.d/pre-receive could be changed on gitea UI if you have enough permission.

[stklcode]

I know. Content was exactly the same for all affected repos and I‘m pretty sure I never configured this and none of the collaborators knew the binary/config paths... 🤔

[lunny]

If the content in pre-receive.d/pre-receive is the same as which in pre-receive.d/gitea.

[theAkito]

However by chance I've found the origin of the bug, because I have migrated my Gitea to a different directory structure and had to re-synchronize all hook scripts.

Same here and same issue.
This bug is 100% confirmed.

My Drone is going crazy and begging for mercy...

[zeripath]

Then you need to remove these unnecessary duplicated hooks.

Do you have any idea how they ended up being there? When did you start using Gitea?

[theAkito]

@zeripath

Then you need to remove these unnecessary duplicated hooks.

How do I do that automatically? I certainly can't do it manually as this would take hours. What file's content do I have to compare to which other file's one? Is the naming and pathing consistent? Is maybe another method more useful?

In my situation I had to do something like this, urgently, which could've caused additional issues, but I can at least push now and the double hooks are a smaller problem than the one described in this thread.
(Problem started after Gitea migration with overhaul of the directory structure, as described by @stklcode.)

When did you start using Gitea?

I think it was shortly after 1.0, don't remember really.

[zeripath]

Ok, so it's a longstanding migration issue from that point in time. Sorry about that - ok here's how you remove those double hooks:

cd gitea-repositories
for i in */*.git/hooks; do
     if test -e "$i/pre-receive.d/pre-receive"; then
        diff "$i/pre-receive.d/pre-receive" "$i/pre-receive.d/gitea" > /dev/null && rm "$i/pre-receive.d/pre-receive"
    fi
done

And similarly for update and post-receive.

(Or something like that. Please be aware I'm writing bash on my phone in bed - stick echos in front of those rms before you run it.)

[theAkito]

@zeripath

Thanks for your help. From what I understand, this issue needs a general fix i.e. pull request, anyway?

[zeripath]

I don't think this needs a PR - there's no current bug and the situation cannot occur in new repositories or on new installs. This is a migration issue from early Gitea.

We just need to delete the duplicate hooks. Unless we can see that the problem comes back...

(Likely some migration script failed for you a number of years ago, which one I don't know.)

[theAkito]

(Likely some migration script failed for you a number of years ago, which one I don't know.)

I never really "migrated" in the classic sense, except recently, when the sleeping error woke up.
When I checked the contents of the hooks, I saw some old directories I deprecated over a year ago. Since then I had Resynchronize pre-receive, update and post-receive hooks of all repositories. run a couple of times. Apparently, it doesn't seem to have helped, at least with the remains of an older version of the installation.

So are all Gitea native hooks now saved as /pre-receive.d/gitea? Then this issue should not occur on newer installations, as you said, though maybe there should be an additional step in the FAQ/Troubleshooting section that explains this situation and its one-time fix to users of an older version that are maybe unaware of the issue.

[zeripath]

You're correct that they're all saved as pre-receive.d/gitea etc.

The code is here:

gitea/modules/repository/hooks.go

Lines 26 to 70 in d92781b

	// createDelegateHooks creates all the hooks scripts for the repo
	func createDelegateHooks(repoPath string) (err error) {
	var (
	hookNames = []string{"pre-receive", "update", "post-receive"}
	hookTpls = []string{
	fmt.Sprintf("#!/usr/bin/env %s\ndata=$(cat)\nexitcodes=\"\"\nhookname=$(basename $0)\nGIT_DIR=${GIT_DIR:-$(dirname $0)}\n\nfor hook in ${GIT_DIR}/hooks/${hookname}.d/*; do\ntest -x \"${hook}\" || continue\necho \"${data}\" | \"${hook}\"\nexitcodes=\"${exitcodes} $?\"\ndone\n\nfor i in ${exitcodes}; do\n[ ${i} -eq 0 ] || exit ${i}\ndone\n", setting.ScriptType),
	fmt.Sprintf("#!/usr/bin/env %s\nexitcodes=\"\"\nhookname=$(basename $0)\nGIT_DIR=${GIT_DIR:-$(dirname $0)}\n\nfor hook in ${GIT_DIR}/hooks/${hookname}.d/*; do\ntest -x \"${hook}\" || continue\n\"${hook}\" $1 $2 $3\nexitcodes=\"${exitcodes} $?\"\ndone\n\nfor i in ${exitcodes}; do\n[ ${i} -eq 0 ] || exit ${i}\ndone\n", setting.ScriptType),
	fmt.Sprintf("#!/usr/bin/env %s\ndata=$(cat)\nexitcodes=\"\"\nhookname=$(basename $0)\nGIT_DIR=${GIT_DIR:-$(dirname $0)}\n\nfor hook in ${GIT_DIR}/hooks/${hookname}.d/*; do\ntest -x \"${hook}\" || continue\necho \"${data}\" | \"${hook}\"\nexitcodes=\"${exitcodes} $?\"\ndone\n\nfor i in ${exitcodes}; do\n[ ${i} -eq 0 ] || exit ${i}\ndone\n", setting.ScriptType),
	}
	giteaHookTpls = []string{
	fmt.Sprintf("#!/usr/bin/env %s\n\"%s\" hook --config='%s' pre-receive\n", setting.ScriptType, setting.AppPath, setting.CustomConf),
	fmt.Sprintf("#!/usr/bin/env %s\n\"%s\" hook --config='%s' update $1 $2 $3\n", setting.ScriptType, setting.AppPath, setting.CustomConf),
	fmt.Sprintf("#!/usr/bin/env %s\n\"%s\" hook --config='%s' post-receive\n", setting.ScriptType, setting.AppPath, setting.CustomConf),
	}
	)
	hookDir := filepath.Join(repoPath, "hooks")
	for i, hookName := range hookNames {
	oldHookPath := filepath.Join(hookDir, hookName)
	newHookPath := filepath.Join(hookDir, hookName+".d", "gitea")
	if err := os.MkdirAll(filepath.Join(hookDir, hookName+".d"), os.ModePerm); err != nil {
	return fmt.Errorf("create hooks dir '%s': %v", filepath.Join(hookDir, hookName+".d"), err)
	}
	// WARNING: This will override all old server-side hooks
	if err = os.Remove(oldHookPath); err != nil && !os.IsNotExist(err) {
	return fmt.Errorf("unable to pre-remove old hook file '%s' prior to rewriting: %v ", oldHookPath, err)
	}
	if err = ioutil.WriteFile(oldHookPath, []byte(hookTpls[i]), 0777); err != nil {
	return fmt.Errorf("write old hook file '%s': %v", oldHookPath, err)
	}
	if err = os.Remove(newHookPath); err != nil && !os.IsNotExist(err) {
	return fmt.Errorf("unable to pre-remove new hook file '%s' prior to rewriting: %v", newHookPath, err)
	}
	if err = ioutil.WriteFile(newHookPath, []byte(giteaHookTpls[i]), 0777); err != nil {
	return fmt.Errorf("write new hook file '%s': %v", newHookPath, err)
	}
	}
	return nil
	}

Prior to that it was here:

gitea/models/repo.go

Lines 919 to 963 in 27c6b8f

	// createDelegateHooks creates all the hooks scripts for the repo
	func createDelegateHooks(repoPath string) (err error) {
	var (
	hookNames = []string{"pre-receive", "update", "post-receive"}
	hookTpls = []string{
	fmt.Sprintf("#!/usr/bin/env %s\ndata=$(cat)\nexitcodes=\"\"\nhookname=$(basename $0)\nGIT_DIR=${GIT_DIR:-$(dirname $0)}\n\nfor hook in ${GIT_DIR}/hooks/${hookname}.d/*; do\ntest -x \"${hook}\" || continue\necho \"${data}\" | \"${hook}\"\nexitcodes=\"${exitcodes} $?\"\ndone\n\nfor i in ${exitcodes}; do\n[ ${i} -eq 0 ] || exit ${i}\ndone\n", setting.ScriptType),
	fmt.Sprintf("#!/usr/bin/env %s\nexitcodes=\"\"\nhookname=$(basename $0)\nGIT_DIR=${GIT_DIR:-$(dirname $0)}\n\nfor hook in ${GIT_DIR}/hooks/${hookname}.d/*; do\ntest -x \"${hook}\" || continue\n\"${hook}\" $1 $2 $3\nexitcodes=\"${exitcodes} $?\"\ndone\n\nfor i in ${exitcodes}; do\n[ ${i} -eq 0 ] || exit ${i}\ndone\n", setting.ScriptType),
	fmt.Sprintf("#!/usr/bin/env %s\ndata=$(cat)\nexitcodes=\"\"\nhookname=$(basename $0)\nGIT_DIR=${GIT_DIR:-$(dirname $0)}\n\nfor hook in ${GIT_DIR}/hooks/${hookname}.d/*; do\ntest -x \"${hook}\" || continue\necho \"${data}\" | \"${hook}\"\nexitcodes=\"${exitcodes} $?\"\ndone\n\nfor i in ${exitcodes}; do\n[ ${i} -eq 0 ] || exit ${i}\ndone\n", setting.ScriptType),
	}
	giteaHookTpls = []string{
	fmt.Sprintf("#!/usr/bin/env %s\n\"%s\" hook --config='%s' pre-receive\n", setting.ScriptType, setting.AppPath, setting.CustomConf),
	fmt.Sprintf("#!/usr/bin/env %s\n\"%s\" hook --config='%s' update $1 $2 $3\n", setting.ScriptType, setting.AppPath, setting.CustomConf),
	fmt.Sprintf("#!/usr/bin/env %s\n\"%s\" hook --config='%s' post-receive\n", setting.ScriptType, setting.AppPath, setting.CustomConf),
	}
	)
	hookDir := filepath.Join(repoPath, "hooks")
	for i, hookName := range hookNames {
	oldHookPath := filepath.Join(hookDir, hookName)
	newHookPath := filepath.Join(hookDir, hookName+".d", "gitea")
	if err := os.MkdirAll(filepath.Join(hookDir, hookName+".d"), os.ModePerm); err != nil {
	return fmt.Errorf("create hooks dir '%s': %v", filepath.Join(hookDir, hookName+".d"), err)
	}
	// WARNING: This will override all old server-side hooks
	if err = os.Remove(oldHookPath); err != nil && !os.IsNotExist(err) {
	return fmt.Errorf("unable to pre-remove old hook file '%s' prior to rewriting: %v ", oldHookPath, err)
	}
	if err = ioutil.WriteFile(oldHookPath, []byte(hookTpls[i]), 0777); err != nil {
	return fmt.Errorf("write old hook file '%s': %v", oldHookPath, err)
	}
	if err = os.Remove(newHookPath); err != nil && !os.IsNotExist(err) {
	return fmt.Errorf("unable to pre-remove new hook file '%s' prior to rewriting: %v", newHookPath, err)
	}
	if err = ioutil.WriteFile(newHookPath, []byte(giteaHookTpls[i]), 0777); err != nil {
	return fmt.Errorf("write new hook file '%s': %v", newHookPath, err)
	}
	}
	return nil
	}

With the name being gitea since at #1006 when the directories were created. So I don't know where you picked up the pre-receive etc copies as they shouldn't have been put there by Gitea.

[theAkito]

So I don't know where you picked up the pre-receive etc copies as they shouldn't have been put there by Gitea.

🤷‍♀

[zeripath]

Possibly there was some broken migration script?! Anyway have you managed to get rid of the things?

[theAkito]

@zeripath

I ran #6636 (comment) and your #9183 (comment) but it did not change anything, as the scripts don't seem to find anything. I need to investigate that, as I assume, the solution should be pretty much like what was provided here.

Well, I fixed it.

I manually ls'd the hook directories and noticed a strange finding:
Each repository had a hook named gitea (correct, as expected) and a duplicate hook called gitean. I don't remember if this is my fault or if this is directly related to this migration type of issue.
My quick fix:

cd gitea-repositories
find . -name "gitean" -exec rm {} \;

I suppose these issues can be closed now?