The other day I was dealing with a client who had a strange issue when we got on a call. To preface this, they were running a PowerShell script in their Azure tenant going through multiple subscription which had key vaults. Essentially the script was something like, for each key vault in each subscription, get all the keys and secrets and check their expiration dates, if their expiration dates fall within x number of days, send email for a warning. Pretty simple.

When the script ran – which ran as an Azure Automation Runbook, it actually would fail on a specific vault. Ok. So, the next step was to see the logs and see where the exception was being thrown from and/or what was causing it. After looking further, we saw that in the affected vault it had trouble trying to add a found key to an array for which it was going to be processed by the next PowerShell step. Ok. Well, why? What’s the key that was affected? So, on this call, the client showed me in their Azure Portal, in their key vault, that the only object they had was a certificate they generated and that’s all.

Where is the key?

Ok, here’s the strange thing that I didn’t know about but would find out later. The client then showed me that if he ran Get-AzKeyVaultKey -VaultName “Affected Key Vault” locally on his machine, it would show a key there. Ok, so why does it show a key there but not in Azure Portal? Was the key deleted and perhaps in the deleted keys section of the portal? Nope. Are you sure this is the correct key vault? Yep.

Hmmm I said.

Then looking at the properties of that magical key, I see that the content type is set to something called: application/x-pkcs12. What is that? I don’t know. So naturally I google it and at this point I am just scoping and checking out multiple site’s to give me an idea of what that is. Sites like: Deploying certificates with application/x-pkcs12 · Issue #18579 · MicrosoftDocs/azure-docs (github.com). After looking at a couple sites I see this is certificate related. Ok. So, what does a magical key have to do with a certificate.

Then it hit me!

The key vault only had 1 certificate. So, I need to look more into how certificates work in Azure Key Vault and so I go here: About Azure Key Vault Certificates – Azure Key Vault | Microsoft Learn and what do you think I find when reading this doc? Tell you what, I’ll show you! The Microsoft doc said: “When a Key Vault certificate is created, an addressable key and secret are also created with the same name. The Key Vault key allows key operations, and the Key Vault secret allows retrieval of the certificate value as a secret.”

Oh, ok. I had to test this out in my own tenant to verify. I deleted all keys and only kept one secret from something else. In all, I had no keys, 1 secret and no certificates. First step was to create a certificate. Now, in the Azure Portal, I saw I still had no keys, the same old secret from something else, and the newly generated certificate. But the doc said it would create a key and secret along with the certificate right?

Absolutely! However, it was only visible if queried from PowerShell. And that’s exactly what happened.

Here’s the proof. Below are the images so you can see how there are no keys, 1 secret from a previous job, and the newly created certificate.

After I checked these out, I went to PowerShell to verify if there was a key there too or not. I ran Get-AzKeyVaultKey -VaultName Test-Vault-Lenny24 locally on my machine and got this:

Interesting. Azure Portal said I had no keys but in PowerShell it says I do. Next, I wanted to check the secrets too since the doc said if you create a certificate, it will create both a key AND a secret in the backend. I ran Get-AzKeyVaultSecret -VaultName Test-Vault-Lenny24 locally on the machine and got this:

Ah! Now instead of the 1 secret I originally had, now I have 2! So, do you remember that magical key the client was seeing earlier? That was due to the certificate they generated. Unfortunately, the script they were using was having a problem with grabbing that imaginary key and placing it into an array for further processing. However, they were able to write some more code that would ignore that key vault. If I recall correctly, they mentioned that it was probably something they didn’t need at all.

Conclusion:

There are so many tiny things like this that make you go mad! But overall, this was an excellent learning opportunity to see how generated certificates in Azure Key Vault worked from the Azure Portal perspective and of course, the PowerShell perspective as well.