Decrypting_Error: Service key not available

[kristian-lesko]

Hello,

I am having trouble configuring this plugin. I filled out the auth/kerberos/config path with the base64-encoded keytab file content and the service_account entry (in the format HTTP/full.hostname@REALM.COM). The plugin seems to be properly mounted, but when trying to authenticate against this backend (using both the example Python script from README and a custom curl command call yields the same result), I receive the following error from the API:

{u'errors': [u'[Root cause: Decrypting_Error] Decrypting_Error: Error decrypting encpart of service ticket provided: [KRB Error: (45) KRB_AP_ERR_NOKEY Service key not available - Could not get key from keytab: Matching key not found in keytab. Looking for [HTTP/full-hostname.keytab@EXAMPLE.COM] realm: EXAMPLE.COM kvno: 1 etype: 18]']}

However, when I klist -kt the keytab used, the principal that I put into the service_account config entry is there.

Has anybody seen such an issue, or do you have any idea how to resolve this? Thanks a lot.

[ah-]

Hi,

What keys are exactly in the keytab? In particular including encryption type and kvno? We've seen gokrb5 being a bit picky around keytabs.

Does that match up with the kvno/etype in the error message?

[kristian-lesko]

Hi,

the original keytab I was testing the plugin with contained about 6 instances of exactly the same key (the one for the principal I was using in the service_account); I have also tried generating a new keytab (using ktutil) with only one entry, but I got the same error. Both the KVNO and etype match what is displayed in the error message.

Should the service_account entry of auth/kerberos/config also contain the realm part of the principal (i.e. HTTP/full.hostname@REALM.COM) or not (i.e. HTTP/full.hostname)? Thank you.

[ah-]

I think the service_account should be the underlying account that you authenticated with instead of the SPN. Do you have one and could you try using that?

[kristian-lesko]

I've tried this as well (using an user account after kinit that tries to authenticate against the backend) but with the same result (Matching key not found in keytab).

Moreover, I think that the config should contain the service account pertaining to the keytab (as that's what the gokrb5 code checks if I understand it correctly); I wouldn't probably want to replace the service account in the plugin config each time another user wants to authenticate...

[ah-]

It should be the service account that was used to generate the server keytab.

So let's debug this. Stack trace is roughly:

1. https://github.com/wintoncode/vault-plugin-auth-kerberos/blob/master/path_login.go#L112
   ok, creds, err := spnegoKrb5Authenticate(*kt, config.ServiceAccount, authorization, req.Connection.RemoteAddr)
2. https://github.com/wintoncode/vault-plugin-auth-kerberos/blob/master/vendor/gopkg.in/jcmturner/gokrb5.v3/service/APExchange.go#L18

   func ValidateAPREQ(APReq messages.APReq, kt keytab.Keytab, sa string, cAddr string) (bool, credentials.Credentials, error) {
   var creds credentials.Credentials
   err := APReq.Ticket.DecryptEncPart(kt, sa)```
   

3. https://github.com/wintoncode/vault-plugin-auth-kerberos/blob/master/vendor/gopkg.in/jcmturner/gokrb5.v3/messages/Ticket.go

   func (t *Ticket) DecryptEncPart(keytab keytab.Keytab, sa string) error {
   ...
   key, err := keytab.GetEncryptionKey(upn, t.Realm, t.EncPart.KVNO, t.EncPart.EType)```
   

4. https://github.com/wintoncode/vault-plugin-auth-kerberos/blob/master/vendor/gopkg.in/jcmturner/gokrb5.v3/keytab/keytab.go#L48

   // GetEncryptionKey returns the EncryptionKey from the Keytab for the newest entry with the required kvno, etype and matching principal.
   func (kt *Keytab) GetEncryptionKey(nameString []string, realm string, kvno, etype int) (types.EncryptionKey, error) {
       var key types.EncryptionKey
       var t time.Time
       for _, k := range kt.Entries {
           if k.Principal.Realm == realm && len(k.Principal.Components) == len(nameString) && int(k.Key.KeyType) == etype && (int(k.KVNO) == kvno || kvno == 0) && k.Timestamp.After(t) {
               p := true
               for i, n := range k.Principal.Components {
                   if nameString[i] != n {
                       p = false
                       break
                   }
               }
               if p {
                   key = k.Key
               }
           }
       }
       if len(key.KeyValue) < 1 {
           return key, fmt.Errorf("Matching key not found in keytab. Looking for %v realm: %v kvno: %v etype: %v", nameString, realm, kvno, etype)
       }
       return key, nil
   }
   
   

Can you see anything that would not match? Maybe some dns name? Otherwise the next thing I'd recommend is adding some logging and check what's in the keytab.

[ah-]

Or rather, I'd try and build a minimal gokrb5 example that reads your keytab and prints out all the keys it has.

[kristian-lesko]

Hello,

thanks for the advice regarding re-compiling the gokrb5 module for debug output; it ultimately led me to the solution.

The issue was that when using a principal of the service/hostname format (e.g., HTTP/hostname1), the gokrb5 module parses the principal name from the keytab entry into two parts (k.Principal.Components: [HTTP, hostname1]), but the nameString variable passed from the path_login module of the plugin keeps the principal name in a single string ([HTTP/hostname1]); the comparison then returns False because the principals are different, no key is matched from the keytab, and the error is triggered.

I'll try to send a patch to also support this use case; many thanks for help with debugging this, @ah-!

[ah-]

Did you get it to work? It sounds like this is something we should fix then, by similarly splitting the string in path_login?

Do you want to send a PR?

[kristian-lesko]

Yes, I managed to make it work (and actually retrieve a token from Vault using Kerberos, which is great). I manually generated a keytab with no service prefix in principal (i.e. only hostname@REALM.COM) but will need to support the splitting of the principal name (because we usually use keytabs with this structure and it wouldn't really be feasible for us to only use a hostname/realm format); I will try to prepare a PR with the fix here shortly.

[pault28]

I am actually having this same problem. So what's the working approach pending when there is a fix @kristian-lesko ?

Vault token: {'errors': ['[Root cause: Decrypting_Error] Decrypting_Error: Error decrypting encpart of service ticket provided: [KRB Error: (45) KRB_AP_ERR_NOKEY Service key not available - Could not get key from keytab: Matching key not found in keytab. Looking for [vault] realm: REALM kvno: 9 etype: 23]']}

[kristian-lesko]

Hello @pault28,

until the fix is merged, I have patched the code manually and re-built the package with this fix.

[pault28]

Understood @kristian-lesko . One thing though it seems you managed to get it working with the current setup based:

Yes, I managed to make it work (and actually retrieve a token from Vault using Kerberos, which is great). I manually generated a keytab with no service prefix in principal (i.e. only hostname@REALM.COM) but will need to support the splitting of the principal name

Just wondering if you could shed more light on the keytab creation step and how you then authenticate with python. Thanks.

[kristian-lesko]

Hello,

I'm using FreeIPA for access management, which includes keytab creation (using the ipa-getkeytab command). FreeIPA, however, only grants certificates with in the format service/hostname, which was the root cause of this problem.

With the patch linked above, the authentication works; without the patch, you can manually create a keytab with just the hostname format (e.g., `hostname1@REALM.COM) using the ktutil command.

[ah-]

I'll have a look at how to integrate the fix best later. Until then what @kristian-lesko says is best, try it with a keytab with just one part.

Although according to your log it's already just vault, so it might be something else. I'd double-check the kvno values, we've had issues with gokrb5 being picky about them.

[pault28]

Thanks both. What I am struggling with is not knowing what's going on. I create keytab with:

ktpass -princ vault.domainname.com@REALM -mapuser domainname\SERVICE_ACCOUNT -pass ******* -out http.keytab -ptype KRB5_NT_PRINCIPAL
# This is what I then convert to base64 before writing to vault kerberos config.

Then on another node I log on as the service account and run the python code below:
klist on the client I am testing from gives:

(test) SERVICE_ACCOUNT@CLIENT_SERVER:~$ klist
Ticket cache: FILE:/tmp/krb5cc_159401033
Default principal: SERVICE_ACCOUNT@REALM

Valid starting     Expires            Service principal
20/04/18 07:42:24  20/04/18 17:42:24  krbtgt/domain@REALM
        renew until 20/04/18 21:30:25

python testing;

# I use service = "krbtgt/domain@REALM" for the service as that's the only one that doesn't throw:
# {'errors': ['SPNEGO OID of MechToken is not of type KRB5']}
service = "krbtgt/domain@REALM"
rc, vc = kerberos.authGSSClientInit(service=service, mech_oid=kerberos.GSS_MECH_OID_SPNEGO)
kerberos.authGSSClientStep(vc, "")
kerberos_token = kerberos.authGSSClientResponse(vc)
# vault is bound to 127.0.0.1:8200 so I hit nginx in the next call
r = requests.post("https://vault/v1/auth/kerberos/login", json={'authorization': 'Negotiate ' + kerberos_token}, verify=False)
print('Vault token:', r.json()['auth']['client_token']) 
# that fails. Printing out just r.josn() gives:
print(r.json())
{'errors': ['[Root cause: Decrypting_Error] Decrypting_Error: Error decrypting encpart of service ticket provided: 
[KRB Error: (45) KRB_AP_ERR_NOKEY Service key not available - 
Could not get key from keytab: Matching key not found in keytab. Looking for [SERVICE_ACCOUNT] realm: REALM kvno: 3 etype: 18]']}

[ah-]

Does your SERVICE_ACCOUNT contain a /?

If not, could you please double-check that your kvno and etype in the error message match with what's actually in the keytab?

[pault28]

@kristian-lesko @ah- I got this to work in the end. Thanks both for your help.

In all honestly I am not 100% sure of all the steps that made it to work but I will share them here.

1. Used ktpass to bind service to an AD account. I didn't do this but command is something in the line of

ktpass -out c:\temp\myappserver.keytab
-princ HTTP/myappserver.austin.ibm.com@WSSEC.AUSTIN.IBM.COM
-mapUser myappserv 
-mapOp set 
-pass was1edu
-crypto DES-CBC-MD5 
-pType KRB5_NT_PRINCIPAL
+DesOnly

2. Use ktutil as described in this plugin readme. Only thing I changed was the encryption type and kvno. As a matter of the fact in my case the only encryption type that worked is "RC4-HMAC". This page was helpful as it contains step for validating the generated keytab works.

$ ktutil
ktutil:  addent -password -p your_service_account@REALM.COM -e RC4-HMAC -k 1
Password for your_service_account@REALM.COM:
ktutil:  list -e
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    1            your_service_account@REALM.COM (aes256-cts-hmac-sha1-96)
ktutil:  wkt vault.keytab

3. Test from a different machine with python:

import requests
import kerberos
import hvac
service = "HTTP/vault.DOMAIN@REALM"
rc, vc = kerberos.authGSSClientInit(service=service, mech_oid=kerberos.GSS_MECH_OID_SPNEGO)
kerberos.authGSSClientStep(vc, "")
kerberos_token = kerberos.authGSSClientResponse(vc)
r = requests.post("https://vault/v1/auth/kerberos/login", json={'authorization': 'Negotiate ' + kerberos_token}, verify=False)
token = r.json()['auth']['client_token']
print('Vault token:', token)

client = hvac.Client(url="https://vault", token=token, verify=False)
client.write('secret/foo', baz='bar', lease='1h')
client.write('secret/foo2', baz='bar', lease='1h')
print(client.read('secret/Devops'))
print(client.read('secret/foo'))
print(client.read('secret/foo2'))
client.delete('secret/foo')

[ah-]

Oh great, thanks for reporting back! So this seems like it was caused by different encryption schemes. I'm not quite sure how it picks the encryption method, maybe that's a server side setting, or you can control it during the gss steps?

[pault28]

.. or you can control it during the gss steps?

You might be able to control it via the gss steps, and yes it is a server side thing too. Googling showed we needed to enable aes256 in AD. The steps to doing this wasn't clear so stick what worked for now.

[ah-]

This should all be fixed now so closing, feel free to open a new ticket if you find anything else.