Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEBT: Performance and Memory Allocation Optimization #3

Merged
merged 2 commits into from
Oct 26, 2024
Merged

DEBT: Performance and Memory Allocation Optimization #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6d3db50
Updated CHANGELOG
mprimeaux Oct 26, 2024
4210751
Updated CHANGELOG
mprimeaux Oct 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions CHANGELOG/CHANGELOG-1.x.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
### Security

---
## [1.4.0] - 2024-OCT-26

### Added
- **FEATURE:**: Added concurrent benchmark tests.
### Changed
- **DEBT:** Maintained Safety with Linter Suppression: Added `// nolint:gosec` with justification for safe conversions.
- **DEBT:** Refactored Slice Initialization: Initialized `idRunes` with zero length and pre-allocated capacity, using append to build the slice.
- **DEBT:** Ensured Comprehensive Testing: Reviewed and updated tests to handle all edge cases and ensure no runtime errors.
### Deprecated
### Removed
- **FEATURE:** Removed Unicode support for custom dictionaries.
### Fixed
- **DEFECT:** Fixed Operator Precedence: Changed `bits.Len(uint(alphabetLen - 1))` to `bits.Len(uint(alphabetLen) - 1)` to ensure safe conversion.
### Security

---
## [1.3.0] - 2024-OCT-26

Expand All @@ -35,7 +51,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Deprecated
### Removed
### Fixed
- **DFECT:** Fixed version compare links in CHANGELOG.
- **DEFECT:** Fixed version compare links in CHANGELOG.
### Security

---
Expand All @@ -49,7 +65,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
### Security

[Unreleased]: https://github.com/scriptures-social/platform/compare/v1.3.0...HEAD
[Unreleased]: https://github.com/scriptures-social/platform/compare/v1.4.0...HEAD
[1.4.0]: https://github.com/sixafter/nanoid/compare/v1.3.0...v1.4.0
[1.3.0]: https://github.com/sixafter/nanoid/compare/v1.2.0...v1.3.0
[1.2.0]: https://github.com/sixafter/nanoid/compare/v1.0.0...v1.2.0
[1.0.0]: https://github.com/sixafter/nanoid/compare/a6a1eb74b61e518fd0216a17dfe5c9b4c432e6e8...v1.0.0
Expand Down
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ test: ## Execute unit tests

.PHONY: bench
bench: ## Execute benchmark tests
$(GO_TEST) -bench=.
$(GO_TEST) -bench=. -benchmem ./...

.PHONY: clean
clean: ## Remove previous build
Expand Down
89 changes: 65 additions & 24 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,18 @@ A simple, fast, and efficient Go implementation of [NanoID](https://github.com/a

## Features

- **Secure**: Uses `crypto/rand` for cryptographically secure random number generation.
- **Fast**: Optimized for performance with efficient algorithms.
- **Thread-Safe**: Safe for concurrent use in multi-threaded applications.
- **Customizable**: Specify custom ID lengths and alphabets.
- **Easy to Use**: Simple API with sensible defaults.
* **Stateless Design**: Each function operates independently without relying on global state or caches, eliminating the need for synchronization primitives like mutexes. This design ensures predictable behavior and simplifies usage in various contexts.
* **Cryptographically Secure**: Utilizes Go's crypto/rand package for generating cryptographically secure random numbers. This guarantees that the generated IDs are both unpredictable and suitable for security-sensitive applications.
* **High Performance**: Optimized algorithms and efficient memory management techniques ensure rapid ID generation. Whether you're generating a few IDs or millions, the library maintains consistent speed and responsiveness.
* **Memory Efficient**: Implements sync.Pool to reuse byte slices, minimizing memory allocations and reducing garbage collection overhead. This approach significantly enhances performance, especially in high-throughput scenarios.
* **Thread-Safe**: Designed for safe concurrent use in multi-threaded applications. Multiple goroutines can generate IDs simultaneously without causing race conditions or requiring additional synchronization.
* **Customizable**: Offers flexibility to specify custom ID lengths and alphabets. Whether you need short, compact IDs or longer, more complex ones, the library can accommodate your specific requirements.
* **User-Friendly API**: Provides a simple and intuitive API with sensible defaults, making integration straightforward. Developers can start generating IDs with minimal configuration and customize as needed.
* **Zero External Dependencies**: Relies solely on Go's standard library, ensuring ease of use, compatibility, and minimal footprint within your projects.
* **Comprehensive Testing**: Includes a robust suite of unit tests and concurrency tests to ensure reliability, correctness, and thread safety. This commitment to quality guarantees consistent performance across different use cases.
* **Detailed Documentation**: Accompanied by clear and thorough documentation, including examples and usage guidelines. New users can quickly understand how to implement and customize the library to fit their needs.
* **Efficient Error Handling**: Employs predefined errors to avoid unnecessary allocations, enhancing both performance and clarity in error management.
* **Optimized for Low Allocations**: Carefully structured to minimize heap allocations, reducing memory overhead and improving cache locality. This optimization is crucial for applications where performance and resource usage are critical.

## Installation

Expand Down Expand Up @@ -61,27 +68,14 @@ fmt.Println("Generated Nano ID of size 32:", id)
Generate a Nano ID using a custom alphabet:

```go
customAlphabet := "abcdef123456"
id, err := nanoid.NewCustom(16, customAlphabet)
alphabet := "abcdef123456"
id, err := nanoid.NewCustom(16, alphabet)
if err != nil {
log.Fatal(err)
}
fmt.Println("Generated Nano ID with custom alphabet:", id)
```

### Generate a Nano ID with Unicode Alphabet

Generate a Nano ID using a Unicode alphabet:

```go
unicodeAlphabet := "あいうえお漢字🙂🚀"
id, err := nanoid.NewCustom(10, unicodeAlphabet)
if err != nil {
log.Fatal(err)
}
fmt.Println("Generated Nano ID with Unicode alphabet:", id)
```

### Generate a Nano ID with Custom Random Source

Generate a Nano ID using a custom random source that implements io.Reader:
Expand Down Expand Up @@ -148,17 +142,64 @@ func main() {
* `DefaultAlphabet`: The default alphabet used for ID generation: `-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz`
* `DefaultSize`: The default size of the generated ID: `21`

## Unicode Support

This implementation fully supports custom alphabets containing Unicode characters, including emojis and characters from various languages. By using []rune internally, it correctly handles multi-byte Unicode characters.

## Performance

The package is optimized for performance and low memory consumption:
* **Efficient Random Byte Consumption**: Uses bitwise operations to extract random bits efficiently.
* **Avoids `math/big`**: Does not use `math/big`, relying on built-in integer types for calculations.
* **Minimized System Calls**: Reads random bytes in batches to reduce the number of system calls.

## Execute Benchmarks:

Run the benchmarks using the go test command with the `bench` make target:

```shell
make bench
```

### Interpreting Results:

Sample output might look like this:

```shell
go test -bench=. -benchmem ./...
goos: darwin
goarch: arm64
pkg: github.com/sixafter/nanoid
cpu: Apple M3 Max
BenchmarkNew-16 6329498 189.2 ns/op 40 B/op 3 allocs/op
BenchmarkNewSize/Size10-16 11600679 102.4 ns/op 24 B/op 2 allocs/op
BenchmarkNewSize/Size21-16 6384469 186.7 ns/op 40 B/op 3 allocs/op
BenchmarkNewSize/Size50-16 2680179 448.2 ns/op 104 B/op 6 allocs/op
BenchmarkNewSize/Size100-16 1387914 863.3 ns/op 192 B/op 11 allocs/op
BenchmarkNewCustom/Size10_CustomASCIIAlphabet-16 9306187 128.8 ns/op 24 B/op 2 allocs/op
BenchmarkNewCustom/Size21_CustomASCIIAlphabet-16 5062975 239.4 ns/op 40 B/op 3 allocs/op
BenchmarkNewCustom/Size50_CustomASCIIAlphabet-16 2322037 515.3 ns/op 101 B/op 5 allocs/op
BenchmarkNewCustom/Size100_CustomASCIIAlphabet-16 1235755 972.0 ns/op 182 B/op 9 allocs/op
BenchmarkNew_Concurrent/Concurrency1-16 2368245 513.1 ns/op 40 B/op 3 allocs/op
BenchmarkNew_Concurrent/Concurrency2-16 1940826 609.5 ns/op 40 B/op 3 allocs/op
BenchmarkNew_Concurrent/Concurrency4-16 1986049 585.6 ns/op 40 B/op 3 allocs/op
BenchmarkNew_Concurrent/Concurrency8-16 1999959 602.2 ns/op 40 B/op 3 allocs/op
BenchmarkNew_Concurrent/Concurrency16-16 2018793 595.6 ns/op 40 B/op 3 allocs/op
BenchmarkNewCustom_Concurrent/Concurrency1-16 1960315 611.7 ns/op 40 B/op 3 allocs/op
BenchmarkNewCustom_Concurrent/Concurrency2-16 1790460 673.7 ns/op 40 B/op 3 allocs/op
BenchmarkNewCustom_Concurrent/Concurrency4-16 1766841 670.7 ns/op 40 B/op 3 allocs/op
BenchmarkNewCustom_Concurrent/Concurrency8-16 1768189 677.4 ns/op 40 B/op 3 allocs/op
BenchmarkNewCustom_Concurrent/Concurrency16-16 1765303 689.5 ns/op 40 B/op 3 allocs/op
PASS
ok github.com/sixafter/nanoid 33.279s
```

* `ns/op` (Nanoseconds per Operation):
* Indicates the average time taken per operation.
* Lower values signify better CPU performance.
* `B/op` (Bytes Allocated per Operation):
* Shows the average number of bytes allocated per operation.
* `0 B/op` indicates no heap allocations, which is optimal.
* `allocs/op` (Allocations per Operation):
* Represents the average number of memory allocations per operation.
* `0 allocs/op` is ideal as it indicates no heap allocations.

## Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.
Expand Down
111 changes: 61 additions & 50 deletions nanoid.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,107 +2,118 @@
//
// This source code is licensed under the MIT License found in the
// LICENSE file in the root directory of this source tree.

// nanoid.go
package nanoid

import (
"crypto/rand"
"errors"
"io"
"math/bits"
"strings"
"sync"
)

// Constants for default settings.
const (
DefaultAlphabet = "-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz"
DefaultSize = 21
MaxUintSize = 1024 // Adjust as needed
)

// Predefined errors to avoid allocations on each call.
var (
ErrInvalidSize = errors.New("size must be greater than zero")
ErrSizeExceedsMaxUint = errors.New("size exceeds maximum allowed value")
ErrEmptyAlphabet = errors.New("alphabet must not be empty")
ErrRandomSourceNoData = errors.New("random source returned no data")
)

// New generates a NanoID with the default size and alphabet using crypto/rand as the random source.
// Byte pool to reuse byte slices and minimize allocations.
var bytePool = sync.Pool{
New: func() interface{} {
b := make([]byte, MaxUintSize) // Non-zero length and capacity

return &b
},
}

// New generates a Nano ID with the default size and alphabet using crypto/rand as the random source.
func New() (string, error) {
return NewSize(DefaultSize)
}

// NewSize generates a NanoID with a specified size and the default alphabet using crypto/rand as the random source.
// NewSize generates a Nano ID with a specified size and the default alphabet using crypto/rand as the random source.
func NewSize(size int) (string, error) {
return NewCustom(size, DefaultAlphabet)
}

// NewCustom generates a NanoID with a specified size and custom alphabet using crypto/rand as the random source.
// NewCustom generates a Nano ID with a specified size and custom ASCII alphabet using crypto/rand as the random source.
func NewCustom(size int, alphabet string) (string, error) {
return NewCustomReader(size, alphabet, cryptoRandReader)
}

// NewCustomReader generates a NanoID with a specified size, custom alphabet, and custom random source.
func NewCustomReader(size int, alphabet string, rnd io.Reader) (string, error) {
if rnd == nil {
return "", errors.New("random source cannot be nil")
}
if size <= 0 {
return "", errors.New("size must be greater than zero")
return "", ErrInvalidSize
}

// Convert alphabet to []rune to support Unicode characters
alphabetRunes := []rune(alphabet)
alphabetLen := len(alphabetRunes)
if alphabetLen == 0 {
return "", errors.New("alphabet must not be empty")
if size > MaxUintSize {
return "", ErrSizeExceedsMaxUint
}

// Handle special case when alphabet length is 1
if alphabetLen == 1 {
return strings.Repeat(string(alphabetRunes[0]), size), nil
if len(alphabet) == 0 {
return "", ErrEmptyAlphabet
}

// Calculate the number of bits needed to represent the alphabet indices
bitsPerChar := bits.Len(uint(alphabetLen - 1))
return generateASCIIID(size, alphabet)
}

// generateASCIIID generates an ID using a byte-based (ASCII) alphabet.
func generateASCIIID(size int, alphabet string) (string, error) {
//nolint:gosec // G115: conversion from int to uint is safe due to prior bounds checking
bitsPerChar := bits.Len(uint(len(alphabet) - 1))
if bitsPerChar == 0 {
bitsPerChar = 1
}

idRunes := make([]rune, size)
// Acquire a pointer to a byte slice from the pool
bufPtr, ok := bytePool.Get().(*[]byte)
if !ok {
panic("bytePool.Get() did not return a *[]byte")
}
buf := *bufPtr
buf = buf[:size] // Slice to desired size

defer func() {
// Reset the slice back to MaxUintSize before putting it back
*bufPtr = (*bufPtr)[:MaxUintSize]
bytePool.Put(bufPtr)
}()

var bitBuffer uint64
var bitsInBuffer int
i := 0

for i < size {
// If we don't have enough bits, read more random bytes
for i := 0; i < size; {
if bitsInBuffer < bitsPerChar {
var b [8]byte // Read up to 8 bytes at once for efficiency
n, err := rnd.Read(b[:])
var b [8]byte
n, err := rand.Read(b[:])
if err != nil {
return "", err
}
if n == 0 {
return "", errors.New("random source returned no data")
return "", ErrRandomSourceNoData
}
// Append the new random bytes to the bit buffer
for j := 0; j < n; j++ {
bitBuffer |= uint64(b[j]) << bitsInBuffer
bitsInBuffer += 8
}
}

// Extract bitsPerChar bits to get the index
idx := int(bitBuffer & ((1 << bitsPerChar) - 1))
mask := uint64((1 << bitsPerChar) - 1)
idx := bitBuffer & mask
bitBuffer >>= bitsPerChar
bitsInBuffer -= bitsPerChar

// Use the index if it's within the alphabet range
if idx < alphabetLen {
idRunes[i] = alphabetRunes[idx]
//nolint:gosec // G115: conversion from int to uint is safe due to prior bounds checking
if int(idx) < len(alphabet) {
buf[i] = alphabet[idx]
i++
}
// Else discard and continue
}

return string(idRunes), nil
}

// cryptoRandReader is a wrapper around crypto/rand.Reader to match io.Reader interface.
var cryptoRandReader io.Reader = cryptoRandReaderType{}

type cryptoRandReaderType struct{}

func (cryptoRandReaderType) Read(p []byte) (int, error) {
return rand.Read(p)
return string(buf), nil
}
Loading
Loading