failed to mount entry &vault.MountEntry{Path:"secret/"

[samprakos]

This issue #648 is happening to me (0.3.0). Does this issue have a known solution? I would really rather not clear all the vault data and restart. I am using Consul as a backend and getting the below message over and over. I swapped out 3 aws instances with Consul and Vault on them with 3 new ones...then shut the old ones down and I believe this started happening when a Vault instance on one of the new servers tried to take leadership. Consul reacted fine to all of this.

Error = failed to mount entry &vault.MountEntry{Path:"secret/", Type:"generic", Description:"generic secret storage", UUID:"b0ebbea3-c775-8cd4-2cf7-c16f91b5e7f9", Config:vault.MountConfig{DefaultLeaseTTL:0, MaxLeaseTTL:0}, Options:map[string]string{}, Tainted:false}: cannot mount under existing mount 'secret/'

[jefferai]

Assuming it's the same underlying cause as #648, the fix is the same too -- you can either upgrade to master now, or it should be fixed when 0.4 comes out.

If possible I'd recommend upgrading to master, if for no other reason than if you do not have the same underlying cause we can work on fixing the problem before 0.4 comes out.

[samprakos]

With the latest code on master, it did not solve the issue...the logs look a little different...here is a sample:

==> Vault server configuration:

     Log Level: info
         Mlock: supported: true, enabled: false
       Backend: consul (HA available)

Advertise Address: http://10.1.8.10:8200
Listener 1: tcp (addr: ":8200", tls: "disabled")

==> Vault server started! Log data will stream in below:

2015/10/30 18:03:14 [INFO] core: vault is unsealed
2015/10/30 18:03:14 [INFO] core: entering standby mode
2015/10/30 18:03:14 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:14 [INFO] core: post-unseal setup starting
2015/10/30 18:03:14 [INFO] core: mounted backend of type generic at logical/b0ebbea3-c775-8cd4-2cf7-c16f91b5e7f9/
2015/10/30 18:03:14 [INFO] core: mounted backend of type system at sys/
2015/10/30 18:03:14 [INFO] core: mounted backend of type transit at logical/38385df5-a602-eb0a-70af-09effb633861/
2015/10/30 18:03:14 [INFO] core: mounted backend of type cubbyhole at logical/70e84f87-f6a3-f474-37cc-8f880168aeb3/
2015/10/30 18:03:14 [INFO] rollback: starting rollback manager
2015/10/30 18:03:15 [INFO] expire: restored 6 leases
2015/10/30 18:03:15 [ERR] core: failed to create audit entry file/: sanity check failed; unable to open given path for writing
2015/10/30 18:03:15 [ERR] core: post-unseal setup failed: failed to setup audit table
2015/10/30 18:03:15 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:15 [INFO] core: post-unseal setup starting
2015/10/30 18:03:15 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:15 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:15 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:15 [INFO] core: post-unseal setup starting
2015/10/30 18:03:16 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:16 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:16 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:16 [INFO] core: post-unseal setup starting
2015/10/30 18:03:17 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:17 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:17 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:17 [INFO] core: post-unseal setup starting
2015/10/30 18:03:17 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:17 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:17 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:17 [INFO] core: post-unseal setup starting
2015/10/30 18:03:18 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:18 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:18 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:18 [INFO] core: post-unseal setup starting
2015/10/30 18:03:19 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:19 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:19 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:19 [INFO] core: post-unseal setup starting
2015/10/30 18:03:19 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:19 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:19 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:20 [INFO] core: post-unseal setup starting
2015/10/30 18:03:20 [INFO] expire: revoked 'secret/omaha-web/session-auth-key/9f5bc26a-d4f4-f38a-d415-e67c2b32d210'
2015/10/30 18:03:20 [INFO] expire: revoked 'secret/omaha-web/session-encrypt-key/0f5ec503-b41e-4e3a-7cea-484255b6f300'
2015/10/30 18:03:20 [INFO] expire: revoked 'secret/omaha-web/session-auth-key/cea8956a-c2d6-1e9b-1c43-77e76f8ded91'
2015/10/30 18:03:20 [INFO] expire: revoked 'secret/omaha-web/session-auth-key/51c9617c-04fb-a973-28fa-0c054f65ad3c'
2015/10/30 18:03:20 [INFO] expire: revoked 'secret/omaha-web/session-encrypt-key/e8b66296-edf2-8556-f43e-f8fa6409ce8d'
2015/10/30 18:03:20 [INFO] expire: revoked 'secret/omaha-web/session-encrypt-key/9532bef0-466e-d63c-bbb2-8ff0834990b5'
2015/10/30 18:03:20 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:20 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:20 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:20 [INFO] core: post-unseal setup starting
2015/10/30 18:03:21 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:21 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:21 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:21 [INFO] core: post-unseal setup starting
2015/10/30 18:03:22 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:22 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:22 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:22 [INFO] core: post-unseal setup starting
2015/10/30 18:03:22 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:22 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:22 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:22 [INFO] core: post-unseal setup starting
2015/10/30 18:03:23 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:23 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:23 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:23 [INFO] core: post-unseal setup starting
2015/10/30 18:03:24 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:24 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:24 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:24 [INFO] core: post-unseal setup starting
2015/10/30 18:03:24 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:24 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:24 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:24 [INFO] core: post-unseal setup starting
2015/10/30 18:03:25 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:25 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:25 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:25 [INFO] core: post-unseal setup starting
2015/10/30 18:03:26 [ERR] core: failed to mount entry secret/: cannot mount under existing mount 'secret/'
2015/10/30 18:03:26 [ERR] core: post-unseal setup failed: failed to setup mount table
2015/10/30 18:03:26 [INFO] core: acquired lock, enabling active operation
2015/10/30 18:03:26 [INFO] core: post-unseal setup starting

[jefferai]

I think this is a different problem.

What seems to be happening is that, because it cannot open the designated file for writing, the audit backend is failing to start. This is causing the post-unseal setup phase of the node becoming active to fail, however, when this happens it's not cleaning up after itself. Then it realizes it's master and tries to perform the post-unseal setup again.

In the short term you can probably solve this by fixing access to your audit file.

[samprakos]

I wasn't aware that I had auditing enabled. When I attempt to auth at the command line I get "Error validating token: http: read on closed response body". Is there another way to disable auditing?

[jefferai]

Does it get into this bad state every time you start it?

[jefferai]

I just pushed a change to master which will cause the file path to be printed in the error message.

[samprakos]

No...but I launched new AMIs so this is a fresh aws instance.

So, on this log: 2015/10/30 18:03:15 [ERR] core: failed to create audit entry file/: sanity check failed; unable to open given path for writing

What...LOL just got your message ok...will try

[jefferai]

Audit config is stored in the storage backend, so a fresh AWS instance won't help if you're using the same storage.

[samprakos]

Great...so I got this log: 2015/10/30 18:37:21 [ERR] core: failed to create audit entry file/: sanity check failed; unable to open /var/log/vault_audit.log for writing

Then went and created that file w/ proper permissions, restarted, unsealed...and all seems fixed!! Thanks :)

[jefferai]

Sure. I'm going to leave this open because I want to look into the underlying issue, but feel free to unsubscribe if you like.

[msabramo]

I also ran into this with vault 0.3.1, which doesn't print the path of the audit log it is trying to write.

I was too lazy to pull from git master and install on our servers, so I did some strace to find where it was writing to.

CORP\marca@consul02:/var/log$ sudo strace -e file -ff sudo -u vault /usr/local/bin/vault server -config /etc/vault.d/vault.hcl

and in my case it was trying to write audit.log in the current directory, which is weird, because it depends on what directory you happen to start vault in. I actually had to modify the upstart file for vault to always force it to use --chdir /var/log so that the audit.log would be written to /var/log/audit.log.

The other curious thing is that I don't remember turning on the audit feature -- maybe I did and forgot, but the poster above said the same so I wonder if it's getting turned on by default perhaps?

[msabramo]

It might be useful to get a new version of vault released with the recent fixes?

It might also be handy if there was some way to disable the auth backend when it's causing problems.

vault audit-list did not work when I was having the problem because the server would never start up properly.

Maybe if opening the log file fails, vault can print a message in the log that advises running vault with a new command line option -- e.g.: vault server -disable-audit; or it can be made a config setting. Having some way to turn it off seems handy though.

[jefferai]

and in my case it was trying to write audit.log in the current directory

Probably then when it was configured it was given a relative path instead of a full path.

The other curious thing is that I don't remember turning on the audit feature -- maybe I did and forgot, but the poster above said the same so I wonder if it's getting turned on by default perhaps?

No, like other kinds of backends, audit backends are only enabled via administrative commands.

It might be useful to get a new version of vault released with the recent fixes?

It will be out soon enough. Although, if you're too lazy to pull from git master and install, won't you be too lazy to get the new release and install? 👅

Maybe if opening the log file fails, vault can print a message in the log that advises running vault with a new command line option -- e.g.: vault server -disable-audit; or it can be made a config setting. Having some way to turn it off seems handy though.

The problem there is it exposes a relatively easy way for an attacker to turn off audit logging; if Vault is up and operative enough to properly handle an administrative token, it should already be logging things into the audit log.

So far, the new behavior of printing the file that Vault is attempting to use seems to be a decent enough workaround (and in 0.3(.1?) it does check that it can write to that file before allowing the backend to be enabled).

[msabramo]

No, like other kinds of backends, audit backends are only enabled via administrative commands.

Hmmm. I don't remember doing this, but that of course means nothing as I do lots of things and don't remember them 😄 Especially since I set up Vault a few months ago. Wish I had a way to audit the enabling of audit logs 😄 or maybe just a pill to be less scatter-brained 😄

too lazy to pull from git master and install, won't you be too lazy to get the new release and install?

Yeah, fair point. Though I think I could better sell writing an ansible playbook that updates Vault to the latest released version; folks might wince if I wrote the playbook to be able to pull from git (although that could be pretty cool for situations like these...). I'm using the kbrebanov.vault ansible role and it doesn't seem to work particularly well -- i.e.: I think it failed to update Vault and I had to manually download and unpack the file to upgrade Vault -- any recos for a better role/playbook? Heck since Vault is written in Golang, it should be pretty trivial to do with just a silly little shell script.

decent enough workaround

Yeah, it very well may be enough to have these things. Obviously, there is a tension between making Vault easy to troubleshoot/maintain and security, which is obvious a paramount concern for Vault.

Thanks again @jefferai for all the pointers which helped me get back up and running!

[jefferai]

Unfortunately, I don't know much about the various playbooks/scripts/etc. for using Vault with puppet, chef, or ansible. But, hopefully you'll find something better that works for you.

FWIW, when I was using Ansible with Vault, I just had a simple play that fetched the new binary from where I had stashed it and pushed it onto each server. (I had to stage it and then use cp -f because otherwise I got complaints about the file being busy.)