Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

example and docs for hash API #624

Merged
merged 9 commits into from
Oct 10, 2024
+575 −292
Merged

example and docs for hash API #624

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
af23a40
hash rust example
yshekel Oct 9, 2024
c9a9123
add CUDA option to hash test
yshekel Oct 9, 2024
eede005
add readme to example
yshekel Oct 9, 2024
cfd44e7
fix examples as worksapce for shared compilation
yshekel Oct 9, 2024
0cf1717
rust binding hash docs
yshekel Oct 9, 2024
497e8dc
minor
yshekel Oct 9, 2024
6730a1a
fixed bad link
yshekel Oct 9, 2024
8ffa24e
try to reduce compile time for rust examples
yshekel Oct 9, 2024
a23a5ec
code review fixes
yshekel Oct 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/examples.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
- name: Rust examples
working-directory: ./examples/rust
if: needs.check-changed-files.outputs.rust == 'true' || needs.check-changed-files.outputs.examples == 'true'
run: |
run: |
# loop over all directories in the current directory
for dir in $(find . -mindepth 1 -maxdepth 1 -type d); do
if [ -d "$dir" ]; then
Expand Down
22 changes: 0 additions & 22 deletions docs/docs/icicle/golang-bindings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,25 +124,3 @@ func main() {
```

Replace `/path/to/shared/libs` with the actual path where the shared libraries are located on your system.

## Supported curves, fields and operations

### Supported curves and operations

| Operation\Curve | bn254 | bls12_377 | bls12_381 | bw6-761 | grumpkin |
| --------------- | :---: | :-------: | :-------: | :-----: | :------: |
| MSM | ✅ | ✅ | ✅ | ✅ | ✅ |
| G2 | ✅ | ✅ | ✅ | ✅ | ❌ |
| NTT | ✅ | ✅ | ✅ | ✅ | ❌ |
| ECNTT | ✅ | ✅ | ✅ | ✅ | ❌ |
| VecOps | ✅ | ✅ | ✅ | ✅ | ✅ |
| Polynomials | ✅ | ✅ | ✅ | ✅ | ❌ |

### Supported fields and operations

| Operation\Field | babybear |
| --------------- | :------: |
| VecOps | ✅ |
| Polynomials | ✅ |
| NTT | ✅ |
| Extension Field | ✅ |
52 changes: 30 additions & 22 deletions docs/docs/icicle/libraries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ ICICLE is composed of two main logical parts:

## ICICLE device

The ICICLE device library serves as an abstraction layer for interacting with various hardware devices. It provides a comprehensive interface for tasks such as setting the active device, querying device-specific information like free and total memory, determining the number of available devices, and managing memory allocation. Additionally, it offers functionality for copying data to and from devices, managing task queues (streams) for efficient device utilization, and abstracting the complexities of device management away from the user.
The ICICLE device library serves as an abstraction layer for interacting with various hardware devices. It provides a comprehensive interface for tasks such as setting the active device, querying device-specific information like free and total memory, determining the number of available devices, and managing memory allocation. Additionally, it offers functionality for copying data to and from devices, managing task queues (streams) for efficient device utilization, and abstracting the complexities of device management away from the user.

See programmers guide for more details. [C++](./programmers_guide/cpp#device-management), [Rust](./programmers_guide/rust#device-management), [Go](./programmers_guide/go)

Expand All @@ -24,36 +24,44 @@ Each library has a corresponding crate. See [programmers guide](./programmers_gu

#### Supported curves and operations

| Operation\Curve | [bn254](https://neuromancer.sk/std/bn/bn254) | [bls12-377](https://neuromancer.sk/std/bls/BLS12-377) | [bls12-381](https://neuromancer.sk/std/bls/BLS12-381) | [bw6-761](https://eprint.iacr.org/2020/351) | grumpkin |
| --------------------------------------------------- | :------------------------------------------: | :---------------------------------------------------: | :---------------------------------------------------: | :-----------------------------------------: | :------: |
| [MSM](./primitives/msm) | ✅ | ✅ | ✅ | ✅ | ✅ |
| G2 MSM | ✅ | ✅ | ✅ | ✅ | ❌ |
| [NTT](./primitives/ntt) | ✅ | ✅ | ✅ | ✅ | ❌ |
| ECNTT | ✅ | ✅ | ✅ | ✅ | ❌ |
| [Vector operations](./primitives/vec_ops) | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Polynomials](./polynomials/overview) | ✅ | ✅ | ✅ | ✅ | ❌ |
| [Poseidon](primitives/poseidon) | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Merkle Tree](primitives/poseidon#the-tree-builder) | ✅ | ✅ | ✅ | ✅ | ✅ |
| Operation\Curve | [bn254](https://neuromancer.sk/std/bn/bn254) | [bls12-377](https://neuromancer.sk/std/bls/BLS12-377) | [bls12-381](https://neuromancer.sk/std/bls/BLS12-381) | [bw6-761](https://eprint.iacr.org/2020/351) | grumpkin |
| ----------------------------------------- | :------------------------------------------: | :---------------------------------------------------: | :---------------------------------------------------: | :-----------------------------------------: | :------: |
| [MSM](./primitives/msm) | ✅ | ✅ | ✅ | ✅ | ✅ |
| G2 MSM | ✅ | ✅ | ✅ | ✅ | ❌ |
| [NTT](./primitives/ntt) | ✅ | ✅ | ✅ | ✅ | ❌ |
| ECNTT | ✅ | ✅ | ✅ | ✅ | ❌ |
| [Vector operations](./primitives/vec_ops) | ✅ | ✅ | ✅ | ✅ | ✅ |
| [Polynomials](./polynomials/overview) | ✅ | ✅ | ✅ | ✅ | ❌ |
| [Poseidon](primitives/poseidon) | ✅ | ✅ | ✅ | ✅ | ✅ |

#### Supported fields and operations

| Operation\Field | [babybear](https://eprint.iacr.org/2023/824.pdf) | [Stark252](https://docs.starknet.io/documentation/architecture_and_concepts/Cryptography/p-value/) |
| ----------------------------------------- | :----------------------------------------------: | :------------------------------------------------------------------------------------------------: |
| [Vector operations](./primitives/vec_ops) | ✅ | ✅ |
| [Polynomials](./polynomials/overview) | ✅ | ✅ |
| [NTT](primitives/ntt) | ✅ | ✅ |
| Extension Field | ✅ | ❌ |
| Operation\Field | [babybear](https://eprint.iacr.org/2023/824.pdf) | [Stark252](https://docs.starknet.io/documentation/architecture_and_concepts/Cryptography/p-value/) | m31 |
| ----------------------------------------- | :----------------------------------------------: | :------------------------------------------------------------------------------------------------: | :---: |
| [Vector operations](./primitives/vec_ops) | ✅ | ✅ | ✅ |
| [Polynomials](./polynomials/overview) | ✅ | ✅ | ❌ |
| [NTT](primitives/ntt) | ✅ | ✅ | ❌ |
| Extension Field | ✅ | ❌ | ✅ |
| [Poseidon](primitives/poseidon) | ✅ | ✅ | ✅ |

### Misc

| Operation | Description |
| ----------- | :------------------------------------------: |
| Keccak | supporting 256b and 512b digest |
| SHA3 | supporting 256b and 512b digest |
| Blake2s | digest is 256b |
| Merkle-Tree | works with any combination of hash functions |




#### Supported hashes

| Hash | Sizes |
| ------ | :------: |
| Keccak | 256, 512 |

## Backend
Each backend must implement the device API interface.
Each backend may implement
- One or more ICICLE library. For example implement only bn254 curve.
- One or more ICICLE library. For example implement only bn254 curve.
- One or more APIs in this library. For example MSM only.

See [CUDA backend](./install_cuda_backend.md) and [Build Your Own Backend](./build_your_own_backend.md) for more info about implementing a backend.
137 changes: 137 additions & 0 deletions docs/docs/icicle/primitives/hash.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# ICICLE Hashing Logic

## Overview

ICICLE’s hashing system is designed to be flexible, efficient, and optimized for both general-purpose and cryptographic operations. Hash functions are essential in operations such as generating commitments, constructing Merkle trees, executing the Sumcheck protocol, and more.

ICICLE provides an easy-to-use interface for hashing on both CPU and GPU, with transparent backend selection. You can choose between several hash algorithms such as Keccak-256, Keccak-512, SHA3-256, SHA3-512, Blake2s, Poseidon and more, which are optimized for processing both general data and cryptographic field elements or elliptic curve points.

## Hashing Logic

Hashing in ICICLE involves creating a hasher instance for the desired algorithm, configuring the hash function if needed, and then processing the data. Data can be provided as strings, arrays, or field elements, and the output is collected in a buffer that automatically adapts to the size of the hashed data.

## Batch Hashing

For scenarios where large datasets need to be hashed efficiently, ICICLE supports batch hashing. The batch size is automatically derived from the output size, making it adaptable and optimized for parallel computation on the GPU (when using the CUDA backend). This is useful for Merkle-trees and more.

## Supported Hash Algorithms

ICICLE supports the following hash functions:

1. **Keccak-256**
2. **Keccak-512**
3. **SHA3-256**
4. **SHA3-512**
5. **Blake2s**
6. **Poseidon**

:::info
Additional hash functions might be added in the future. Stay tuned!
:::

### Keccak and SHA3

[Keccak](https://keccak.team/files/Keccak-implementation-3.2.pdf) is a cryptographic hash function designed by Guido Bertoni, Joan Daemen, Michaël Peeters, and Gilles Van Assche. It was selected as the winner of the NIST hash function competition, becoming the basis for the [SHA-3 standard](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf).

Keccak can take input messages of any length and produce a fixed-size hash. It uses the sponge construction, which absorbs the input data and squeezes out the final hash value. The permutation function, operating on a state array, applies iterative rounds of operations to derive the hash.

### Blake2s

[Blake2s](https://www.rfc-editor.org/rfc/rfc7693.txt) is an optimized cryptographic hash function that provides high performance while ensuring strong security. Blake2s is ideal for hashing small data (such as field elements), especially when speed is crucial. It produces a 256-bit (32-byte) output and is often used in cryptographic protocols.


### Poseidon

[Poseidon](https://eprint.iacr.org/2019/458) is a cryptographic hash function designed specifically for field elements. It is highly optimized for zero-knowledge proofs (ZKPs) and is commonly used in ZK-SNARK systems. Poseidon’s main strength lies in its arithmetization-friendly design, meaning it can be efficiently expressed as arithmetic constraints within a ZK-SNARK circuit.

Traditional hash functions, such as SHA-2, are difficult to represent within ZK circuits because they involve complex bitwise operations that don’t translate efficiently into arithmetic operations. Poseidon, however, is specifically designed to minimize the number of constraints required in these circuits, making it significantly more efficient for use in ZK-SNARKs and other cryptographic protocols that require hashing over field elements.


## Using Hash API

### 1. Creating a Hasher Object

First, you need to create a hasher object for the specific hash function you want to use:

```cpp
#include "icicle/hash/keccak.h"
#include "icicle/hash/blake2s.h"
#include "icicle/hash/poseidon.h"

// Create hasher instances for different algorithms
auto keccak256 = Keccak256::create();
auto keccak512 = Keccak512::create();
auto sha3_256 = Sha3_256::create();
auto sha3_512 = Sha3_512::create();
auto blake2s = Blake2s::create();
// Poseidon requires specifying the field type and arity (supported 3,5,9,12)
auto poseidon = Poseidon::create<scalar_t>(arity);
```

### 2. Hashing Data

Once you have a hasher object, you can hash any input data by passing the input, its size, a configuration, and an output buffer:

```cpp
/**
* @brief Perform a hash operation.
*
* This function delegates the hash operation to the backend.
*
* @param input Pointer to the input data as bytes.
* @param size The number of bytes to hash. If 0, the default chunk size is used.
* @param config Configuration options for the hash operation.
* @param output Pointer to the output data as bytes.
* @return An error code of type eIcicleError indicating success or failure.
*/
eIcicleError hash(const std::byte* input, uint64_t size, const HashConfig& config, std::byte* output) const;

/**
* @brief Perform a hash operation using typed data.
*
* Converts input and output types to `std::byte` pointers and delegates the call to the backend.
*
* @tparam PREIMAGE The type of the input data.
* @tparam IMAGE The type of the output data.
* @param input Pointer to the input data.
* @param size The number of elements of type `PREIMAGE` to hash.
* @param config Configuration options for the hash operation.
* @param output Pointer to the output data.
* @return An error code of type eIcicleError indicating success or failure.
*/
template <typename PREIMAGE, typename IMAGE>
eIcicleError hash(const PREIMAGE* input, uint64_t size, const HashConfig& config, IMAGE* output) const;
```

Example Usage:

```cpp
// Using the Blake2s hasher
const std::string input = "Hello, I am Blake2s!";
const uint64_t output_size = 32; // Blake2s outputs 32 bytes
auto output = std::make_unique<std::byte[]>(output_size);
auto config = default_hash_config();

eIcicleErr err = blake2s.hash(input.data(), input.size(), config, output.get());

// Alternatively, use another hasher (e.g., Keccak256, SHA3-512)
```

### 3. Batch Hashing

To perform batch hashing, set the `config.batch` field to indicate the number of batches. This allows for multiple inputs to be hashed in parallel:

```cpp
auto config = default_hash_config();
config.batch = 2;

const std::string input = "0123456789abcdef"; // This is a batch of "01234567" and "89abcdef"
auto output = std::make_unique<std::byte[]>(32 * config.batch); // Allocate output for 2 batches

eIcicleErr err = keccak256.hash(input.data(), input.size() / config.batch, config, output.get());
```

### Supported Bindings

- [Rust](../rust-bindings/hash)
- Go bindings soon
Loading
Loading