v0.4.0

Highlights

• Revamped the build pipeline to use maturin
• Reduced allocations (speedup for small safetensors files with many small tensors)

What's Changed

• Back on dev. by @Narsil in #313
• Clarify specification around endianness and header padding by @akx in #306
• Adding new target for m1 by @Narsil in #317
• Fixing macos 13 release script (by prebuilding). by @Narsil in #315
• Cache dir all the time. by @Narsil in #319
• Supporting whisper conversion. by @Narsil in #320
• add tensorshare to featured projects by @chainyo in #322
• Fix Bench GH action. by @Narsil in #323
• Yaml is nice. by @Narsil in #325
• Fixing big endian. by @Narsil in #327
• Fixing boolean + numpy > 1.20 by @Narsil in #326
• Add NumPy for all backends dependencies by @chainyo in #332
• impl View for TensorView by @coreylowman in #329
• Reworked the release script entirely by @Narsil in #334
• Preparing small patch release (Big Endian fix). by @Narsil in #335
• Temporary revert of the breaking change (keep it for 0.4.0). by @Narsil in #336
• Don't release rust versions on RC by @Narsil in #337
• Fixing release script. by @Narsil in #338
• Python 3.8 for arm by @Narsil in #339
• Win64. by @Narsil in #342
• win64 by @Narsil in #343
• Trying out maturin. by @Narsil in #344
• Reduce memory allocations by @cospectrum in #350
• [docs] Convert weights by @stevhliu in #345
• Update types for load_model filename by @kevinhu in #356
• Fixing release script for windows aarch64 and 3.12 by @Narsil in #348
• Minor docstring edit to flax.py by @paulbricman in #359
• Stop recreating the hashmap all the time. by @Narsil in #363
• fix: add py.typed by @b-kamphorst in #365
• Preparing a new release (0.4.0). by @Narsil in #366

New Contributors

• @akx made their first contribution in #306
• @chainyo made their first contribution in #322
• @coreylowman made their first contribution in #329
• @cospectrum made their first contribution in #350
• @stevhliu made their first contribution in #345
• @kevinhu made their first contribution in #356
• @paulbricman made their first contribution in #359
• @b-kamphorst made their first contribution in #365

Full Changelog: v0.3.2...v0.4.0

v0.4.0rc1

Highlights

• Revamped entire build pipeline
• Fixed a bug on small files containing many small tensors allocating too much

What's Changed

• Back on dev. by @Narsil in #313
• Clarify specification around endianness and header padding by @akx in #306
• Adding new target for m1 by @Narsil in #317
• Fixing macos 13 release script (by prebuilding). by @Narsil in #315
• Cache dir all the time. by @Narsil in #319
• Supporting whisper conversion. by @Narsil in #320
• add tensorshare to featured projects by @chainyo in #322
• Fix Bench GH action. by @Narsil in #323
• Yaml is nice. by @Narsil in #325
• Fixing big endian. by @Narsil in #327
• Fixing boolean + numpy > 1.20 by @Narsil in #326
• Add NumPy for all backends dependencies by @chainyo in #332
• impl View for TensorView by @coreylowman in #329
• Reworked the release script entirely by @Narsil in #334
• Preparing small patch release (Big Endian fix). by @Narsil in #335
• Temporary revert of the breaking change (keep it for 0.4.0). by @Narsil in #336
• Don't release rust versions on RC by @Narsil in #337
• Fixing release script. by @Narsil in #338
• Python 3.8 for arm by @Narsil in #339
• Win64. by @Narsil in #342
• win64 by @Narsil in #343
• Trying out maturin. by @Narsil in #344
• Reduce memory allocations by @cospectrum in #350
• [docs] Convert weights by @stevhliu in #345
• Update types for load_model filename by @kevinhu in #356
• Fixing release script for windows aarch64 and 3.12 by @Narsil in #348
• Minor docstring edit to flax.py by @paulbricman in #359
• Stop recreating the hashmap all the time. by @Narsil in #363
• fix: add py.typed by @b-kamphorst in #365
• Preparing a new release (0.4.0). by @Narsil in #366

New Contributors

• @akx made their first contribution in #306
• @chainyo made their first contribution in #322
• @coreylowman made their first contribution in #329
• @cospectrum made their first contribution in #350
• @stevhliu made their first contribution in #345
• @kevinhu made their first contribution in #356
• @paulbricman made their first contribution in #359
• @b-kamphorst made their first contribution in #365

Full Changelog: v0.3.2...v0.4.0rc1

v0.3.3

What's Changed

Mostly fixes for big endian machines.

• Back on dev. by @Narsil in #313
• Clarify specification around endianness and header padding by @akx in #306
• Adding new target for m1 by @Narsil in #317
• Fixing macos 13 release script (by prebuilding). by @Narsil in #315
• Cache dir all the time. by @Narsil in #319
• Supporting whisper conversion. by @Narsil in #320
• add tensorshare to featured projects by @chainyo in #322
• Fix Bench GH action. by @Narsil in #323
• Yaml is nice. by @Narsil in #325
• Fixing big endian. by @Narsil in #327
• Fixing boolean + numpy > 1.20 by @Narsil in #326
• Add NumPy for all backends dependencies by @chainyo in #332
• impl View for TensorView by @coreylowman in #329
• Reworked the release script entirely by @Narsil in #334
• Preparing small patch release (Big Endian fix). by @Narsil in #335
• Temporary revert of the breaking change (keep it for 0.4.0). by @Narsil in #336
• Don't release rust versions on RC by @Narsil in #337
• Fixing release script. by @Narsil in #338

New Contributors

• @akx made their first contribution in #306
• @chainyo made their first contribution in #322
• @coreylowman made their first contribution in #329

Full Changelog: v0.3.2...v0.3.3

v0.3.3rc1

What's Changed

• Back on dev. by @Narsil in #313
• Clarify specification around endianness and header padding by @akx in #306
• Adding new target for m1 by @Narsil in #317
• Fixing macos 13 release script (by prebuilding). by @Narsil in #315
• Cache dir all the time. by @Narsil in #319
• Supporting whisper conversion. by @Narsil in #320
• add tensorshare to featured projects by @chainyo in #322
• Fix Bench GH action. by @Narsil in #323
• Yaml is nice. by @Narsil in #325
• Fixing big endian. by @Narsil in #327
• Fixing boolean + numpy > 1.20 by @Narsil in #326
• Add NumPy for all backends dependencies by @chainyo in #332
• impl View for TensorView by @coreylowman in #329
• Reworked the release script entirely by @Narsil in #334
• Preparing small patch release (Big Endian fix). by @Narsil in #335
• Temporary revert of the breaking change (keep it for 0.4.0). by @Narsil in #336
• Don't release rust versions on RC by @Narsil in #337

New Contributors

• @akx made their first contribution in #306
• @chainyo made their first contribution in #322
• @coreylowman made their first contribution in #329

Full Changelog: v0.3.2...v0.3.3.rc1

v0.3.2.rc1

What's Changed

• Prepare 0.3.2 by @Narsil in #239
• Fixing conda workflow. by @Narsil in #240
• Adding a banner to the README. by @Narsil in #244
• [doc] Add a doc page about (remote) Metadata parsing by @julien-c in #243
• [doc] Improve metadata parsing doc by @mishig25 in #245
• There should be .dev0 while on main by @mishig25 in #246
• [doc] Add logo img to index doc page by @mishig25 in #247
• [doc] featured projects by @mishig25 in #249
• Update to actions/checkout@v3 by @mfuntowicz in #251
• Added method len on SafeTensors by @mfuntowicz in #252
• [docs] Safetensors format by @mishig25 in #248
• Fixing the spec with mention about the buffer which cannot be non-contiguous. by @Narsil in #253
• Update metadata_parsing.mdx by @mishig25 in #261
• Add safejax in "Featured Projects" by @alvarobartt in #260
• Adding error checking, convertion can fail on schedulers. by @Narsil in #266
• Add BERTopic to Featured Projects by @davanstrien in #267
• Fix typos in documents by @kiszk in #257
• Adding issue templates. by @Narsil in #268
• Remove ISSUE links. by @Narsil in #269
• [doc build] Use secrets by @mishig25 in #270
• Make tensor identifier a bit more robust, and transformers compatible by @thomasw21 in #264
• Add a reference to tch-rs by @LaurentMazare in #274
• fix syntax error in import statement of safetensors pytorch api docs code example by @kaczmarj in #276
• Missing files. by @Narsil in #282
• Fix wrong docstring for SafeTensors' names method (#250) by @dirac3000 in #284
• Fixing uniform empty tensor handling by @Narsil in #283
• Superseed #279 by @Narsil in #285
• Fix failing doc build workflows by @mishig25 in #287
• Revert dev change by @mishig25 in #288
• Fix numpy.save(-file) docstring. by @amitani in #290
• Adding more control to the chosen weights saved on disk. by @Narsil in #292
• Autodoc for save_model and load_model by @wouterzwerink in #293
• Fix link to shared tensor page by @wouterzwerink in #294
• Support musicgen conversion. by @Narsil in #296
• Passing the token around. by @Narsil in #302
• Adding slice get_dtype() method to get the dtype directly on slices. by @Narsil in #303
• fix: typo in attacks/README.md by @bliutech in #305
• feat: Add filter for shared storage non overlapping tensors by @Narsil in #309
• feat: Supporting big endian architecture. by @Narsil in #310
• 0.3.2.rc1 by @Narsil in #311

New Contributors

• @mfuntowicz made their first contribution in #251
• @davanstrien made their first contribution in #267
• @kiszk made their first contribution in #257
• @LaurentMazare made their first contribution in #274
• @kaczmarj made their first contribution in #276
• @dirac3000 made their first contribution in #284
• @amitani made their first contribution in #290
• @wouterzwerink made their first contribution in #293
• @bliutech made their first contribution in #305

Full Changelog: v0.3.1rc1...v0.3.2.rc1

v0.3.2

What's Changed

• Remove too many keywords. by @Narsil in #187
• Adding memmap example to read from file. by @Narsil in #189
• Making sure the safetensors existing PR is up-to-date. by @Narsil in #192
• Nicer error when failed to find tensor in Rust. by @Narsil in #191
• Adding commit description to convertion PR. by @Narsil in #198
• Python 2.0 added a warning on ByteStorage, so using UntypedStorage by @Narsil in #196
• Adding a minimal version for torch. by @Narsil in #204
• Minor improvements for README by @nickkolok in #201
• More badges. by @Narsil in #210
• Remove unwrap in prod code. by @Narsil in #209
• Upload codecov for main branch. by @Narsil in #211
• Upload codecov for main branch. by @Narsil in #212
• Fixing codecov rust tag. by @Narsil in #213
• Increase coverage on the error paths. by @Narsil in #214
• Deactivating codecov commenting. by @Narsil in #217
• Zero width shape are OK (0-rank tensors). They are simply a scalar. by @Narsil in #208
• Adding simple proptest. by @Narsil in #216
• Disallowing zero-width tensor. by @Narsil in #205
• Disallow erroneous buffer. by @Narsil in #206
• Overspecify a bit the format. by @Narsil in #215
• Adding arithmetic overflow checks (unchecked in release builds). by @Narsil in #207
• Add a safety warning in the convertion script. by @Narsil in #219
• Fixing empty shape test. by @Narsil in #218
• Remove the breaking change of disallowing zero sized tensors. by @Narsil in #221
• Adding last missing test for validate. by @Narsil in #225
• 0-sized tensors allowed. by @Narsil in #226
• Specify python versions. by @Narsil in #227
• Minor checks + proptest in python. by @Narsil in #228
• Committing Cargo.lock. by @Narsil in #233
• Fixing the logic to determine if PR is up-to-date. by @Narsil in #230
• Fixing alignment by @Narsil in #235
• Adding save_model and load_model to help with shared tensors with PyTorch. by @Narsil in #236
• Fixing shared detection logic. by @Narsil in #232
• New release candidate. by @Narsil in #224
• Prepare 0.3.2 by @Narsil in #239
• Fixing conda workflow. by @Narsil in #240
• Adding a banner to the README. by @Narsil in #244
• [doc] Add a doc page about (remote) Metadata parsing by @julien-c in #243
• [doc] Improve metadata parsing doc by @mishig25 in #245
• There should be .dev0 while on main by @mishig25 in #246
• [doc] Add logo img to index doc page by @mishig25 in #247
• [doc] featured projects by @mishig25 in #249
• Update to actions/checkout@v3 by @mfuntowicz in #251
• Added method len on SafeTensors by @mfuntowicz in #252
• [docs] Safetensors format by @mishig25 in #248
• Fixing the spec with mention about the buffer which cannot be non-contiguous. by @Narsil in #253
• Update metadata_parsing.mdx by @mishig25 in #261
• Add safejax in "Featured Projects" by @alvarobartt in #260
• Adding error checking, convertion can fail on schedulers. by @Narsil in #266
• Add BERTopic to Featured Projects by @davanstrien in #267
• Fix typos in documents by @kiszk in #257
• Adding issue templates. by @Narsil in #268
• Remove ISSUE links. by @Narsil in #269
• [doc build] Use secrets by @mishig25 in #270
• Make tensor identifier a bit more robust, and transformers compatible by @thomasw21 in #264
• Add a reference to tch-rs by @LaurentMazare in #274
• fix syntax error in import statement of safetensors pytorch api docs code example by @kaczmarj in #276
• Missing files. by @Narsil in #282
• Fix wrong docstring for SafeTensors' names method (#250) by @dirac3000 in #284
• Fixing uniform empty tensor handling by @Narsil in #283
• Superseed #279 by @Narsil in #285
• Fix failing doc build workflows by @mishig25 in #287
• Revert dev change by @mishig25 in #288
• Fix numpy.save(-file) docstring. by @amitani in #290
• Adding more control to the chosen weights saved on disk. by @Narsil in #292
• Autodoc for save_model and load_model by @wouterzwerink in #293
• Fix link to shared tensor page by @wouterzwerink in #294
• Support musicgen conversion. by @Narsil in #296
• Passing the token around. by @Narsil in #302
• Adding slice get_dtype() method to get the dtype directly on slices. by @Narsil in #303
• fix: typo in attacks/README.md by @bliutech in #305
• feat: Add filter for shared storage non overlapping tensors by @Narsil in #309
• feat: Supporting big endian architecture. by @Narsil in #310
• 0.3.2.rc1 by @Narsil in #311
• Fixes release scripts - Release ready version by @Narsil in #312

New Contributors

• @nickkolok made their first contribution in #201
• @mfuntowicz made their first contribution in #251
• @davanstrien made their first contribution in #267
• @kiszk made their first contribution in #257
• @LaurentMazare made their first contribution in #274
• @kaczmarj made their first contribution in #276
• @dirac3000 made their first contribution in #284
• @amitani made their first contribution in #290
• @wouterzwerink made their first contribution in #293
• @bliutech made their first contribution in #305

Full Changelog: v0.3.0...v0.3.2

v0.3.1

Notable changes

• Fix warning for PyTorch 2.0
• Fix alignment
• Overall hardened code + test suite

What's Changed

• Remove too many keywords. by @Narsil in #187
• Adding memmap example to read from file. by @Narsil in #189
• Making sure the safetensors existing PR is up-to-date. by @Narsil in #192
• Nicer error when failed to find tensor in Rust. by @Narsil in #191
• Adding commit description to convertion PR. by @Narsil in #198
• Python 2.0 added a warning on ByteStorage, so using UntypedStorage by @Narsil in #196
• Adding a minimal version for torch. by @Narsil in #204
• Minor improvements for README by @nickkolok in #201
• More badges. by @Narsil in #210
• Remove unwrap in prod code. by @Narsil in #209
• Upload codecov for main branch. by @Narsil in #211
• Upload codecov for main branch. by @Narsil in #212
• Fixing codecov rust tag. by @Narsil in #213
• Increase coverage on the error paths. by @Narsil in #214
• Deactivating codecov commenting. by @Narsil in #217
• Zero width shape are OK (0-rank tensors). They are simply a scalar. by @Narsil in #208
• Adding simple proptest. by @Narsil in #216
• Disallowing zero-width tensor. by @Narsil in #205
• Disallow erroneous buffer. by @Narsil in #206
• Overspecify a bit the format. by @Narsil in #215
• Adding arithmetic overflow checks (unchecked in release builds). by @Narsil in #207
• Add a safety warning in the convertion script. by @Narsil in #219
• Fixing empty shape test. by @Narsil in #218
• Remove the breaking change of disallowing zero sized tensors. by @Narsil in #221
• Adding last missing test for validate. by @Narsil in #225
• 0-sized tensors allowed. by @Narsil in #226
• Specify python versions. by @Narsil in #227
• Minor checks + proptest in python. by @Narsil in #228
• Committing Cargo.lock. by @Narsil in #233
• Fixing the logic to determine if PR is up-to-date. by @Narsil in #230
• Fixing alignment by @Narsil in #235
• Adding save_model and load_model to help with shared tensors with PyTorch. by @Narsil in #236
• Fixing shared detection logic. by @Narsil in #232
• New release candidate. by @Narsil in #224

New Contributors

• @nickkolok made their first contribution in #201

Full Changelog: v0.3.0...v0.3.1

v0.3.1rc1

What's Changed

• Remove too many keywords. by @Narsil in #187
• Adding memmap example to read from file. by @Narsil in #189
• Making sure the safetensors existing PR is up-to-date. by @Narsil in #192
• Nicer error when failed to find tensor in Rust. by @Narsil in #191
• Adding commit description to convertion PR. by @Narsil in #198
• Python 2.0 added a warning on ByteStorage, so using UntypedStorage by @Narsil in #196
• Adding a minimal version for torch. by @Narsil in #204
• Minor improvements for README by @nickkolok in #201
• More badges. by @Narsil in #210
• Remove unwrap in prod code. by @Narsil in #209
• Upload codecov for main branch. by @Narsil in #211
• Upload codecov for main branch. by @Narsil in #212
• Fixing codecov rust tag. by @Narsil in #213
• Increase coverage on the error paths. by @Narsil in #214
• Deactivating codecov commenting. by @Narsil in #217
• Zero width shape are OK (0-rank tensors). They are simply a scalar. by @Narsil in #208
• Adding simple proptest. by @Narsil in #216
• Disallowing zero-width tensor. by @Narsil in #205
• Disallow erroneous buffer. by @Narsil in #206
• Overspecify a bit the format. by @Narsil in #215
• Adding arithmetic overflow checks (unchecked in release builds). by @Narsil in #207
• Add a safety warning in the convertion script. by @Narsil in #219
• Fixing empty shape test. by @Narsil in #218
• Remove the breaking change of disallowing zero sized tensors. by @Narsil in #221
• Adding last missing test for validate. by @Narsil in #225
• 0-sized tensors allowed. by @Narsil in #226
• Specify python versions. by @Narsil in #227
• Minor checks + proptest in python. by @Narsil in #228
• Committing Cargo.lock. by @Narsil in #233
• Fixing the logic to determine if PR is up-to-date. by @Narsil in #230
• Fixing alignment by @Narsil in #235
• Adding save_model and load_model to help with shared tensors with PyTorch. by @Narsil in #236
• Fixing shared detection logic. by @Narsil in #232
• New release candidate. by @Narsil in #224

New Contributors

• @nickkolok made their first contribution in #201

Full Changelog: v0.3.0...v0.3.1rc1

v0.3.0

Everything in this new release should be backward compatible.

The only thing which might not, is loading a file and saving it again might change it's layout.
Content will be the same, but the order and positions of tensors might be changed to improve the alignment of buffers which can enable much
faster loads on lower level languages. ( #148)

Major changes

• Fixing the fact that the context manager wasn't properly cleaning up after itself. by @Narsil in #166
• [Major Change] Enforcing tensor alignment by @Narsil in #148
• Faster GPU Loads ?? by @Narsil in #184
• Removed the need for SAFETENSORS_FAST_GPU=1 with a safe alternative that loads just as fast.

What's Changed

• Improve type hinting by @thomasw21 in #159
• New unreleased version. by @Narsil in #161
• Update README.md by @thomasw21 in #163
• Enabling 3.8 support for M1. by @Narsil in #162
• Added JSONSchema of the header. by @KOLANICH in #151
• Adding custom error type for users to properly catch. by @Narsil in #165
• Fixing the fact that the context manager wasn't properly cleaning up after itself. by @Narsil in #166
• Minor pydoc fixes for torch bindings by @jbn in #167
• Better error messages on outdated context manager. by @Narsil in #168
• Clippy fix. by @Narsil in #171
• Update README.md by @Narsil in #170
• Fixing error message for FileNotFound. by @Narsil in #175
• Add PaddlePaddle backend by @JunnYu in #176
• [Docs] add paddlepaddle docs by @JunnYu in #177
• [Major Change] Enforcing tensor alignment by @Narsil in #148
• README.md proposed clarity by @fdwr in #181
• Better Rust API (tch-rs) by @Narsil in #179
• Faster GPU Loads ?? by @Narsil in #184
• Preparing for release 0.3rc1. by @Narsil in #185
• Version 0.3.0 by @Narsil in #186

New Contributors

• @jbn made their first contribution in #167
• @JunnYu made their first contribution in #176
• @fdwr made their first contribution in #181

Full Changelog: v0.2.8...v0.3.0

v0.3.0rc1

Everything in this new release should be backward compatible.

The only thing which might not, is loading a file and saving it again might change it's layout.
Content will be the same, but the order and positions of tensors might be changed to improve the alignment of buffers which can enable much
faster loads on lower level languages. ( #148)

Major changes

• Fixing the fact that the context manager wasn't properly cleaning up after itself. by @Narsil in #166
• [Major Change] Enforcing tensor alignment by @Narsil in #148
• Faster GPU Loads ?? by @Narsil in #184
• Removed the need for SAFETENSORS_FAST_GPU=1 with a safe alternative that loads just as fast.

What's Changed

• Improve type hinting by @thomasw21 in #159
• New unreleased version. by @Narsil in #161
• Update README.md by @thomasw21 in #163
• Enabling 3.8 support for M1. by @Narsil in #162
• Added JSONSchema of the header. by @KOLANICH in #151
• Adding custom error type for users to properly catch. by @Narsil in #165
• Fixing the fact that the context manager wasn't properly cleaning up after itself. by @Narsil in #166
• Minor pydoc fixes for torch bindings by @jbn in #167
• Better error messages on outdated context manager. by @Narsil in #168
• Clippy fix. by @Narsil in #171
• Update README.md by @Narsil in #170
• Fixing error message for FileNotFound. by @Narsil in #175
• Add PaddlePaddle backend by @JunnYu in #176
• [Docs] add paddlepaddle docs by @JunnYu in #177
• [Major Change] Enforcing tensor alignment by @Narsil in #148
• README.md proposed clarity by @fdwr in #181
• Better Rust API (tch-rs) by @Narsil in #179
• Faster GPU Loads ?? by @Narsil in #184
• Preparing for release 0.3rc1. by @Narsil in #185

New Contributors

• @jbn made their first contribution in #167
• @JunnYu made their first contribution in #176
• @fdwr made their first contribution in #181

Full Changelog: python-v0.2.8...v0.3.0rc1