An utility to benchmark speed of operations of a PKCS#11 implementation.
On Nov 30th 2023, This repo has been rebased for quality reasons. Pay attention when you are fetching or pulling from it. We apologize for the inconvenience.
You will need the following:
- a recent/decent C++ compiler that supports C++17
- the Boost library (a recent copy, above 1.60)
- the Botan library (a recent copy, above 2.17.1)
On RedHat/Oracle, you can generally find Boost on RPMs, from the EPEL repository. Check out this link to add EPEL to your yum repositories. Note that on EPEL, boost is suffixed with a version number, like boost169.
Botan has to be compiled from scratch. Check out online instructions to compile and install it.
To boostrap the autotools environment, you will need the following packages deployed on your system:
Once you have it all, you can bootstrap, configure and compile. You will need to adapt the flags in the instruction below to adapt to your environment.
In the example below, boost library is deployed within /usr/lib64/boost169, and botan has its pkg-config configuration deployed in /usr/local/lib/pkgconfig
$ ./bootstrap.sh
$ ./configure -C --with-boost-libdir=/usr/lib64/boost169 CXXFLAGS=-I/usr/include/boost169 PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
$ make
Note that to execute p11perftest, you may have to adjust LD_LIBRARY_PATH to include the path to where Botan is deployed (typically /usr/local/lib )
In case session keys cannot be used (which is the default behaviour), keys must be created directly on the cryptographic token. In order to execute the test, you will need to create the following keys upfront:
| key name | description |
|---|---|
rsa-2048 |
a 2048 bits RSA key, with CKA_SIGN, CKA_ENCRYPT, CKA_DECRYPT, CKA_WRAP, CKA_UNWRAP |
rsa-3072 |
a 3072 bits RSA key, with CKA_SIGN, CKA_ENCRYPT, CKA_DECRYPT, CKA_WRAP, CKA_UNWRAP |
rsa-4096 |
a 4096 bits RSA key, with CKA_SIGN, CKA_ENCRYPT, CKA_DECRYPT, CKA_WRAP, CKA_UNWRAP |
ecdsa-secp256r1 |
a secp256r1 ECDSA key, with CKA_SIGN |
ecdsa-secp384r1 |
a secp384r1 ECDSA key, with CKA_SIGN |
ecdsa-secp521r1 |
a secp521r1 ECDSA key, with CKA_SIGN |
ecdh-secp256r1 |
a secp256r1 ECDH key, with CKA_DERIVE |
ecdh-secp384r1 |
a secp384r1 ECDH key, with CKA_DERIVE |
ecdh-secp521r1 |
a secp521r1 ECDH key, with CKA_DERIVE |
des-128 |
a 2DES key, with CKA_ENCRYPT |
des-192 |
a 3DES key, with CKA_ENCRYPT |
aes-128 |
a 128 bits AES key, with CKA_ENCRYPT |
aes-192 |
a 192 bits AES key, with CKA_ENCRYPT |
aes-256 |
a 256 bits AES key, with CKA_ENCRYPT |
hmac-sha1 |
a 160 bits generic secret key, with CKA_SIGN |
hmac-sha256 |
a 256 bits generic secret key, with CKA_SIGN |
hmac-sha512 |
a 512 bits generic secret key, with CKA_SIGN |
xorder-128 |
a 128 bits generic secret key, with CKA_DERIVE |
rand-128 |
a 128 bits AES key (not used during testing), presence yet needed |
There is a script at scripts/createkeys.sh to create these keys, using the PKCS#11 toolkit.
There is also a python script, at scripts/generatekeys.py. It requires Python 3, and you will need to deploy the dependent libraries using pip:
$ pip install -r requirements.txt| name | measured operation | allowed vectors | involved mechanisms |
|---|---|---|---|
aescbc |
AES encryption, in CBC mode | 16*n, n>1 | CKM_AES_CBC |
aesecb |
AES encryption, in ECB mode | 16*n, n>1 | CKM_AES_ECB |
aesgcm |
AES encryption, in GCM mode, IV=12 bytes, no AAD | 1+ | CKM_AES_GCM |
descbc |
3DES encryption, in CBC mode | 8*n, n>1 | CKM_DES3_CBC |
desecb |
AES encryption, in ECB mode | 8*n, n>1 | CKM_DES3_ECB |
ecdh |
Elliptic curve based Diffie Hellman key derivation | keysize dependent | CKM_ECDH1_DERIVE |
ecdsa |
ECDSA digital signature (hashing in software) | 1+ | CKM_ECDSA |
hmac |
HMAC generation | 1+ CKM_SHA_1_HMAC, CKM_SHA256_HMAC, ... |
|
jwe |
JWE decryption (RFC7516), using RSA OAEP and AES GCM | 1+ | CKM_RSA_PKCS_OAEP and CKM_AES_GCM |
oaep |
RSA OAEP decryption | keysize dependent | CKM_RSA_PKCS_OAEP with C_Decrypt() |
oaepunw |
RSA OAEP unwrapping ( a generic secret key) | keysize dependent | CKM_RSA_PKCS_OAEP with C_Unwrap() |
rand |
Generate random numbers | 1+ | C_GenerateRandom() |
xorder |
Key derivation based on exclusive OR | 1+ | CKM_XOR_BASE_AND_DATA |
p11perftest can be invoked with only three arguments:
- a path to a PKCS#11 library
- a slot index
- a password
Here is the full list of supported arguments:
-h [ --help ], print help message-l [ --library ] arg, PKCS#11 library path-s [ --slot ] arg, slot index to use-p [ --password ] arg, password for token in slot-t [ --threads ] arg (=1), number of concurrent threads-i [ --iterations ] arg (=200), number of iterations--skip arg (=0), number of iterations to skip before recording for statistics (in addition to iterations)-j [ --json ], output results as JSON-o [ --jsonfile ] arg, JSON output file name-c [ --coverage ] arg (=rsa,ecdsa,ecdh,hmac,des,aes,xorder,rand,jwe,oaep,oaepunw), coverage of test cases-v [ --vectors ] arg (=8,16,64,256,1024,4096), test vectors to use-k [ --keysizes ] arg (=rsa2048,rsa3072,rsa4096,ecnistp256,ecnistp384,ecnistp521,hmac160,hmac256,hmac512,des128,des192,aes128,aes192,aes256), key sizes or curves to use-f [ --flavour ] arg (=generic), PKCS#11 implementation flavour. Possible values:generic,luna,utimaco,entrust,marvell-n [ --nogenerate ], do not attempt to generate session keys; instead, use pre-existing keys on token
Some arguments allow to specify more than one value. To do so, just separate values with a comma , and without space between values.
Some tokens tend to show a different performance for the first call of an API, compared to the subsequent ones. The parameter --skip allows to skip any number of iterations, i.e. these are executed but not accounted for in statistics.
By default, coverage for des includes ECB and CBC mode; coverage for aes includes ECB, CBC and GCM modes; coverage for jwe includes RSA-OAEP and RSA-OAEP-SHA256; coverage for oaep includes OAEP decryption with SHA1 and OAEP with SHA256, and oaepunw includes OAEP key unwrapping with SHA1 and with SHA256. It is possible to narrow down to specific modes:
- for AES,
aesecb,aescbc, oraesgcminstead ofaes - for DES,
desecbordescbcfordes - for JWE,
jweoaepsha1for RSA-OAEP orjweoaepsha256for RSA-OAEP-SHA256 - for OAEP decryption,
oaepsha1for OAEP with SHA1 oroaepsha256for OAEP with SHA256 - for OAEP unwrapping,
oaepunwsha1for OAEP with SHA1 oroaepunwsha256for OAEP with SHA256
Some tests need more than one key type to operate. If specific key sizes are chosen, it is important to include all keys needed by the algorithms. Forgetting to give one of the key sizes lead to skip the test case, even if specified on the command line.
- for JWE, both RSA and AES key sizes must be specified
- for OAEP, both RSA and AES key sizes must be specified
All environment variables below can be used instead of command line options. When both are present, command line option takes precedence.
PKCS11LIB: path to a PKCS#11 library. Equivalent to-l [ --library ] arg.PKCS11SLOT: valid PKCS#11 slot. Equivalent to-s [ --slot ] arg.PKCS11PASSWORD: token password. Equivalent to-p [ --password ] arg. Note that at this point,p11perftestdoes not support the syntaxes from pkcs11-tools - accessing public objects and pkcs11-tools - fetching password from a subprocess yet.
JSON output files (when -j and/or -o options are specified) can be turned into Excel spreadsheets, using scripts/json2xlsx.py script. To run that package, you must first deploy the dependencies, using the requirements.txt file. Once completed, the script can be executed. It takes two arguments: the source JSON file, and a file name for the target spreadsheet.
$ pip install -r requirements.txt
$ scripts/json2xlsx myresults.json myresults.xlsx
Using the spreadsheet produced at previous step, graphs can be created using gengraph.py from the scripts directory. Just provide the spreadhseet as argument, and graphs will be created automatically.
There are two possibilities for the graphs that are generated:
- The effect of number of threads on latency and throughput, for fixed vector sizes (this is the default). Usage:
gengraphs.py FILE [threads]where the optional switchthreadsis redundant. - The effect of vector size on latency and throughput, for fixed numbers of threads. Usage
gengraphs.py FILE size [--reglines], where the optional switch --reglines will draw lines of best fit for latency and throughput.
There is a further option to compare two data sets using the --comparison switch. Run python gengraphs.py -h for usage.
p11perftest originally created by Eric Devolder.
Additional contributions to gengraphs.py by Marcel Armour.
Copyright (c) 2018 Mastercard
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
- https://www.unine.ch/files/live/sites/physique/files/TP/G%C3%A9n%C3%A9ral/Intro%20calcul%20d'erreur.pdf
- https://stats.libretexts.org/Bookshelves/Introductory_Statistics/Book%3A_Introductory_Statistics_(Shafer_and_Zhang)/06%3A_Sampling_Distributions/6.01%3A_The_Mean_and_Standard_Deviation_of_the_Sample_Mean
- https://stats.libretexts.org/Bookshelves/Introductory_Statistics/Book%3A_Introductory_Statistics_(Shafer_and_Zhang)/06%3A_Sampling_Distributions/6.02%3A_The_Sampling_Distribution_of_the_Sample_Mean