Correct workflow for key provisioning a device in manufacturing

[johnnywyatt]

I am using a ESP32 WROVER based custom piece of hardware. The primary AWS functions of the device are to publish to the shadow and gets OTA updates. After having many of the same issues and frustrations as discussed in #1535, I started with the OTA demo, got that working, then added in MQTT shadow support.

To generate the creds and private key we are using a custom version of the AWS console script that was provided in the aws_config_quick_start directory to create the Thing, get the creds, generate the creds partition, and write it to the device along with the other flash partitions that are needed.

Here is the current partition table

# Name,     Type,   SubType,    Offset,     Size,       Flags
nvs,        data,   nvs,        0x9000,     0x5000,
otadata,    data,   ota,        0xe000,     0x2000,
ota_0,      0,      ota_0,      0x10000,    0x7F0000,
ota_1,      0,      ota_1,      0x800000,   0x7F0000,
eeprom,     data,   153,        0xFF0000,   0x1000,
creds,      data,   0x40,       0xFF1000,   0x1000,
storage,    data,   nvs,        0xFF2000,   0xE000

To store the creds and private key we created a partition called "creds". On startup I read this from flash, then call vAlternateKeyProvisioning() to provision the device. This seems to work very well but I have E (17558) PKCS11: failed nvs get file size 4354 0 printed out in the logs. When digging into this, it seems that we may not be provisioning the device in the correct way. I noticed that the "storage" partition is getting written but am not sure exactly what the intended design was. There seems to be a lot of documentation around developer key provisioning, but very little around what the final design should look like.

1. Is there a better way?
2. What documentation did I miss that might have led me down a better path?

I am also reading through #2961.

[lundinc2]

Hello @johnnywyatt,

Here is a white paper that you may find useful.

Can you expand a bit on your use case and requirements? Are you wanting to provision the device in the factory over the serial port or are you wanting to flash the certificates into the creds NVS partition?

The partition used by PKCS #11 is derived from this location in the PAL, you can modify this string to creds if you want to change the partition.

Thanks,

Carl

[johnnywyatt]

Thanks @lundinc2.
Currently we are writing the creds partition with an image that we are creating from the creds generated from running the AWS setup python script. I think the answer to your question is yes in this case. We are writing that using the Espressif programming tools in the factory, though our creds partition is not NVS, just ASCII encoded. I did see the storage partition called out in the code, but wasn't sure about the workflow and what format the creds should be in. Should we overhaul the approach or is this workable in a manufacturing setting?

[lundinc2]

Hi @johnnywyatt,

I suggest the use of a Secure Element or other type of HSM to protect sensitive data, like your device credentials. As well as following the best practices outlined here.

I also suggest that you reach out to Espressif about potential solutions. This link seems to be a good place to start https://www.espressif.com/en/products/services/manufacturing-services.

I hope this is helpful.

Thanks,

Carl

[shubhamkulkarni97]

Hi @johnnywyatt,

• As @lundinc2 correctly pointed, please refer to core_pkcs11_config.h header file for storage_partition, namespace, labels for device keys and certificates.
• You can use NVS Partition Generator Utility for generation of binary file based on key value pairs provided in .CSV file. In case of AFR, key should be specific label for device key/cert and value should be key/cert file.
• When device credentials are stored in NVS partition, it recommended to enable NVS encryption. Please find more information about NVS encryption here.

Hope these pointers help!

Thanks,
Shubham

[johnnywyatt]

Thanks @lundinc2 and @shubhamkulkarni97. I'm reading through the provided whitepaper and wish I had found this sooner. Out of curiosity, where should I have been looking to find the link for this document? I feel like I searched high and low for this type of documentation and never came across this. Where did I overlook it?

Update: My co-worker just noticed that this is dated January 2021....this could explain a lot.

[lundinc2]

Hi @johnnywyatt seems that the update resolve this :)

To find documents like this, there are some related resource links here https://docs.aws.amazon.com/iot/index.html that direct you to AWS Whitepapers and Case Studies. You can search there, but your search engine should likely be able to find these documents as well.

Thanks,

Carl

[lundinc2]

I am going to consider this as resolved, and close the issue. Please re-open it with any further questions.

Thanks,

Carl