[KeyVault][Certificates] - Get certificate AND private key?

[julie-ng]

Use Case

As a developer, I want to secure communication between services with client/server certificates. I want to authenticate via client certificate to a server using a custom httpsAgent per Node.js documentation

But... how to get both the public certificate AND private key from key vault?

This is standard, not a special library and common practice to require:

• public certificate, i.e. -----BEGIN CERTIFICATE----- part
• private key, i.e. -----BEGIN PRIVATE KEY----- part

Problem: Azure SDK does not return private key

The @azure/keyvault-certificates SDK shows how to get the certificate. But how do you get the private key?

The standard https Node.js library expects this, as copied from official node.js docs

const options = {
  hostname: 'encrypted.google.com',
  port: 443,
  path: '/',
  method: 'GET',
  key: fs.readFileSync('test/fixtures/keys/agent2-key.pem'),
  cert: fs.readFileSync('test/fixtures/keys/agent2-cert.pem')
};
options.agent = new https.Agent(options);

The Azure SDK returns something in this format:

{
  keyId: 'https://xxxxx.vault.azure.net/keys/my-cert-name/xxxxxxxxx',
  secretId: 'https://xxxxx.vault.azure.net/secrets/my-cert-name/xxxxxxxxx',
  name: 'my-cert-name',
  cer: <Buffer 30 82 04 ba 30 82 02 a2 02 01 01 30 0d 06 09 2a 86 48 86 f7 0d 01 01 05 05 00 30 36 31 12 30 10 06 03 55 04 03 0c 09 6c 6f 63 61 6c 68 6f 73 74 31 20 ... 1164 more bytes>,
  policy: {
    lifetimeActions: [ [Object] ],
    contentType: 'application/x-pem-file',
    enhancedKeyUsage: [],
    keyUsage: [],
    validityInMonths: 13,
    subject: 'CN=Alice',
    subjectAlternativeNames: undefined,
    enabled: true,
    keyType: 'RSA',
    keySize: 4096,
    reuseKey: false,
    keyCurveName: undefined,
    exportable: true,
    issuerName: 'Unknown',
    certificateType: undefined,
    certificateTransparency: undefined
  },
  properties: {
    createdOn: 2020-03-04T10:11:20.000Z,
    updatedOn: 2020-03-04T10:11:20.000Z,
    expiresOn: 2021-03-04T09:58:56.000Z,
    id: 'https://xxxxx.vault.azure.net/certificates/my-cert-name/xxxxxxxxx',
    name: 'my-cert-name',
    enabled: true,
    notBefore: 2020-03-04T09:58:56.000Z,
    recoveryLevel: 'Purgeable',
    vaultUrl: 'https://xxxxx.vault.azure.net',
    version: '6ed0b80fa4fc45ac8e9b361ed040d6cc',
    tags: undefined,
    x509Thumbprint: <Buffer e8 43 01 68 ac 42 ba fb b8 a6 0a cc d6 84 6f 4a a2 b7 68 cc>
  }
}

Problem: Public Certificate is (technically) incomplete?

When the cer buffer is converted to a string, it is missing the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----parts.

Why?

Ask

How is a developer supposed to create a custom https agent with client certificate stored in Key Vault using Key Vault certificates?

At this point, I am questioning the utility (or lack thereof) in using Key Vault certificate abstraction. I am thinking it's better just to use secrets, pure strings.... as suggested here https://social.msdn.microsoft.com/Forums/azure/en-US/c9d515e1-22c4-4e7b-84ea-70fb38a86b1e/azure-key-vault-nodejs-certificate?forum=AzureKeyVault

[xirzec]

@sadasant can you take this one?

[sadasant]

@julie-ng I'm gathering information to give you the best response I can. I'll be back as soon as possible 🌞

[sadasant]

Hello @julie-ng,

Thank you so much for getting in touch with us! We're here to help.

When a certificate is retrieved from the KeyVault CertificateClient, one of the properties of the returned abstraction is called cer. This property indeed contains the x509 certificate. The property's value is similar to what would be obtained out of PowerShell's Export-Certificate command. By convention, this format doesn't include the certificate's private key.

Here I'm going to share some ways of retrieving your certificate's full information. Then I'll mention what further actions we'll be taking to address your feedback.

The long route to obtain the public certificate in PEM format

From the buffer you receive in the cer property of a KeyVaultCertificate, you can build a PEM formatted certificate by encoding it to base64, then providing the PEM's header and footer, as the following example shows:

// Let's authenticate our CertificateClient
const credential = await new ClientSecretCredential(
  process.env.AZURE_TENANT_ID,
  process.env.AZURE_CLIENT_ID,
  process.env.AZURE_CLIENT_SECRET
);
const keyVaultName = process.env.KEYVAULT_NAME;
const keyVaultUrl = `https://${keyVaultName}.vault.azure.net`;
const client = new CertificateClient(keyVaultUrl, credential);

// Creating the certificate
const certificateName = "MyCertificate";
const createPoller = await client.beginCreateCertificate(certificateName, {
  issuerName: "Self",
  subject: "cn=MyCert"
});
const keyVaultCertificate = await createPoller.pollUntilDone();

// Making the PEM formatted public certificate
const { cer } = keyVaultCertificate;
const base64CER = Buffer.from(cer).toString("base64");
const PEMFormattedCER = [
  "-----BEGIN CERTIFICATE-----",
  base64CER,
  "-----END CERTIFICATE-----"
].join("\n");

// We could write this into a .pem file
fs.writeFileSync("MyCertificatePublic.pem", PEMFormattedCer);

There are better ways to do this, but first, let's get some context.

What happened with the private key of my certificate?

Azure's KeyVault's design makes sharp distinctions between Keys, Secrets and Certificates. The KeyVault service's Certificates features were designed making use of it's Keys and Secrets capabilities. Let's evaluate the composition of a KeyVault Certificate:

When a Key Vault certificate is created, an addressable key and secret are also created with the same name. The Key Vault key allows key operations and the Key Vault secret allows retrieval of the certificate value as a secret. A Key Vault certificate also contains public x509 certificate metadata.
Source: Composition of a Certificate.

The KeyVault service stores both the public and the private parts of your certificate in a KeyVault secret, along with any other secret you might have created in that same KeyVault instance.

With this separation comes considerable control of the level of access you can give to people in your organization regarding your certificates. The access control can be specified through the policy you pass in when creating a certificate.

The policy used to create the certificate must indicate that the key is exportable. If the policy indicates non-exportable, then the private key isn't a part of the value when retrieved as a secret.
Source: Exportable or Non-exportable key.

You can also dive in the security features of Azure and how they operate in conjunction with KeyVault through the docs section: Secure access to a key vault.

How to obtain the private key

Knowing that the private key is stored in a KeyVault Secret, with the public certificate included, we can retrieve it by using the KeyVault-Secrets client, as follows:

// Using the same credential object we used before,
// and the same keyVaultUrl,
// let's create a SecretClient
const secretClient = new SecretClient(keyVaultUrl, credential);

// Assuming you've already created a KeyVault certificate,
// and that certificateName contains the name of your certificate
const certificateSecret = await secretClient.getSecret(certificateName);

// Here we can find both the private key and the public certificate, in PKCS 12 format:
const PKCS12Certificate = certificateSecret.value!;

// You can write this into a file:
fs.writeFileSync("myCertificate.p12", PKCS12Certificate);

Note that, by default, the contentType of the certificates is PKCS 12. Though specifying the content type of your certificate will make it trivial to get it's secret key in PEM format, let's explore how to retrieve a PEM secret key from a PKCS 12 certificate first.

Using openssl, you can retrieve the public certificate in PEM format by using the following command:

openssl pkcs12 -in myCertificate.p12 -out myCertificate.crt.pem -clcerts -nokeys

You can also use openssl to retrieve the private key, as follows:

openssl pkcs12 -in myCertificate.p12 -out myCertificate.key.pem -nocerts -nodes

Note that in both cases, openssl will ask you for the password used to create the certificate. If you have used my sample code so far, the certificate created didn't specify a password, so you can either leave it blank, or append -passin 'pass:' to the end of each command.

How to obtain the private key directly in PEM format

If you will be using PEM formatted certificates in an everyday basis, you can tell Azure's KeyVault service to create and manage your certificates in PEM format by providing the contentType property at the moment of creating the certificates.
The following example shows how to create and retrieve the public and the private parts of a PEM formatted certificate using the KeyVault clients:

// Creating the certificate
const certificateName = "MyCertificate";
const createPoller = await client.beginCreateCertificate(certificateName, {
  issuerName: "Self",
  subject: "cn=MyCert",
  contentType: "application/x-pem-file" // Here you specify you want to work with PEM certificates.
});
const keyVaultCertificate = await createPoller.pollUntilDone();

// Getting the PEM formatted private key and public certificate:
const certificateSecret = await secretClient.getSecret(certificateName);
const PEMPair = certificateSecret.value!;

console.log(PEMPair);

Keep in mind that, in this format, your public certificate will be in the same blob of content as your private key. You can use the PEM headers to extract them accordingly.

Shouldn't the KeyVault Certificates Client make this easier?

Your issue has highlighted the importance of providing an easier way to retrieve the full certificate. Internally, we have talked about this during the past months, but we haven't added it to our roadmap yet. We are listening, so please stay tuned to the changes that we will be making in these clients. Keep in mind that some other changes might come first.

What we can do right away

I've made an initial pull request that adds a couple of integration tests that show how to retrieve the private key and public certificate out of both a PKCS 12 and a PEM formatted KeyVault Certificate here: #7650

I'll be using this PR to bring my teams attention to this issue, so that we can define paths of improvement for the TypeScript client and the clients that we've made for other languages.

---

@julie-ng, does this help? Please don't hesitate to report any other issue you might have already experienced, or any that might come up in the future. Every interaction with our customers help us deliver the best product we can.

[julie-ng]

Wow, thanks @sadasant for the very thorough response. I did not know I could fetch the secret, just by using the different client.

Shouldn't the KeyVault Certificates Client make this easier?

Ideally as a developer, I would want to just pull the certs out of Key Vault and pass them in memory (no need to write them to files) to https.Agent.

I think this is where we need to improve. Azure has its own way of thinking that doesn't necessarily align with how everyone else in the OSS world does things. In this case, the https.Agent just needs a cert and key.

Next Steps

I want to test fetching the secret as you wrote above - and then putting it into a function. Will let you know how that goes.

I also plan on throwing this use-case together into an blog article, which could be used for docs. Will share if I get to it tomorrow :)

[sadasant]

@julie-ng sounds good!

[julie-ng]

@sadasant thanks for the thorough changes. The customer use case with the Azure Function turned out to be a Networking issue or at least out of SDK scope.

So closing for now.

[swisman]

Dear @sadasant,

I'm trying to retrieve the private key following your instructions. Unfortunately, I get the following error message when running e.g. openssl pkcs12 -in myCertificatesSecret.p12 -out myCertificate.crt.pem -clcerts -nokeys:

1944:error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:crypto\asn1\tasn_dec.c:1149:
1944:error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error:crypto\asn1\tasn_dec.c:309:Type=PKCS12

Same error when trying to extract the private key out of "myCertificatesSecret.p12". The file "myCertificatesSecret.p12" seems to be not in correct PKCS12 format.

Please note: I've downloaded the certificates corresponding Key Vault Secret using the Azure CLI command
az keyvault secret download --vault-name myKeyVault -n myCertificate -f myCertificatesSecret.p12

Do you have any idea what I am missing? Thanks for your assistance!

P.S.: When downloading the certificate as pfx manually from Azure Key Vault, I can extract the private key with the openssl commands provided.

EDIT
Wrong encoding when downloading the key vault secret was causing my issue. Needs to be set to base64, then everything works as described above:
az keyvault secret download --vault-name myKeyVault -n myCertificate -f myCertificatesSecret.pfx --encoding base64
If this is not something specific to my certificate what I assume (I imported an ordinary pfx file to an Azure Key Vault certificate), then this piece of information would be worth mentioning in the docs somewhere.

[sadasant]

Hello @swisman , thank you for the feedback! I’ll be opening an issue based on your comment. Give me some time to process things on my side.

[sadasant]

@swisman I’ve made an issue to help us keep track of your feedback. We’ll come back with an update next month. Thank you for taking the time to report this to us!

The new issue is: #15462

[almiavicas]

If you are a developer and bump into this, there is a gist that will help you a lot. Check this https://gist.github.com/erikbern/756b1d8df2d1487497d29b90e81f8068?permalink_comment_id=4503917#gistcomment-4503917