Prepare for stability (1.0 release)

[matklad]

It feels like the ecosystem is big enough that the last breaking release, 1.0 is due. I don't think the current APIs are good enough for 1.0 release though, so I expect we need at least one more 0.x release with substantional breakage. This issue collects all know future-compatibility problems, but isn't intended to be full stabilization check-list just yet.

• Fix soundness (BorshDeserialize can cause UB by copying zero sized objects with no safe Copy impl #19 and then audit the code)
• Hide schema behind an schema cargo feature and future-proof it's API
• rename schema cargo feature flag to unstable__schema
• Remove mandatory dependency on hashbrown #45
• Hide maybe_std module
• Introduce #[doc(hidden)] pub __rt #[doc(hidden)] pub __private module with macro runtime for use by derives
• add top-level io module.
  • I am not sure defining our own no-std IO is the right approach, but that's a big design discusison out of scope for the issue.
• hide derive behind an optional feature-flag
• deprecate BorshDeserialize::try_from_slice in favor of borsh::from_slice top-level function ( 😡 a lot of churn with this one)
• clarify bounds on Rc/Arc impls we use something significantly funkier than what serde is doing
• remove const-generic feature (it does nothing these days, and was only left to avoid a breaking change)
• unsplit borsh-derive-internal
• cleanup derive syntax to follow serde -- borsh(skip) rather bosh_skip
• maybe forbid collections of ZSTs? Zero Sized Types should not enable DoS #52
• move docs from readme into lib.rs
• setup merge queue for CI
• Introduce snapshot testing for all the serialized results #152
• Future proof schema API #46
• (not for merge, discussion comments) enum discriminant de- and serialization behavior  #148
• cut 1.0.0-alpha.1 release
• test alpha release on nearcore and capture migration guide steps, create a draft PR (only merge after 1.0.0 release)
• test alpha release on near-sdk-rs and capture migration guide steps, create a draft PR (only merge after 1.0.0 release)
  • update near-abi-rs to work with alpha release

[iho]

unsplit borsh-derive-internal

error: `proc-macro` crate types currently cannot export any items other than functions tagged with `#[proc_macro]`, `#[proc_macro_derive]`, or `#[proc_macro_attribute]`

I got this error because we export macroses and some functions. To do that we need to have crates.

[frol]

@iho Re: borsh-derive-internal

The question is whether we need to export those methods at all. I see no need in that, and serde went that path as well and 5 years ago it gave them 7% better compile time: serde-rs/serde@3859f58

[iho]

I am going to work on

• unsplit borsh-derive-internal
• cleanup derive syntax to follow serde -- borsh(skip) rather bosh_skip

[frol]

@iho Re: unsplit borsh-derive-internal

I checked serde-derive-internals crate dependents and realized that they actually have quite a few dependencies there, so borsh-derive-internals might be not that useless after all, so before we merge the "unsplit" work, let's measure what we actually gain from it. If we see <5s compilation time improvement, I vote to keep it as is (two separate crates)

[dj8yfo]

taking the following sub-issue for trial and error and resolution

• Hide maybe_std module

[dj8yfo]

taking the following sub-issue for trial and error and resolution

• extract hashbrown dependency under a feature (as per clarification)

[dj8yfo]

the following is covered in scope of #155

• remove const-generic feature (it does nothing these days, and was only left to avoid a breaking change)

[dj8yfo]

taking the following sub-issue for trial, error and resolution:

• Introduce snapshot testing for all the serialized results #152

[dj8yfo]

The following is (partially) covered in scope of #155 . nostd_io in integration test.
The completeness of coverage remains unclear atm.

• add top-level io module

[dj8yfo]

taking following sub-issue in scope of #46

• Hide schema behind an schema cargo feature

[dj8yfo]

taking this sub-issue:

• hide derive behind an optional feature-flag

[frol]

@matklad Could you, please, remind what you had in mind when added this item?

• Introduce #[doc(hidden)] pub __rt module with macro runtime for use by derives

It seems that the only macro runtime things are “maybestd” and the public traits, aren’t they? Given that we already renamed and hid “maybestd” to “__maybestd”, it seems that there is no further action items there worth introducing the module.

[dj8yfo]

taking this issue

• clarify bounds on Rc/Arc impls we use something significantly funkier than what serde is doing

[matklad]

It seems that the only macro runtime things are “maybestd” and the public traits, aren’t they? Given that we already renamed and hid “maybestd” to “__maybestd”, it seems that there is no further action items there worth introducing the module.

Yup, if it’s just __maybestd, no further action is required. I would still maybe prefer to call this __runtime module, so that, if in the future we need to use any new name from macro, we can use a normal name and put it into __runtime module, rather than adding a different __ name to the top-level. Looking at serde and anyhow __private is probably a more standard name here than __runtime:

• https://github.com/serde-rs/serde/blob/master/serde/src/lib.rs#L319
• https://github.com/dtolnay/anyhow/blob/master/src/lib.rs#L640

[dj8yfo]

taking this sub-issue

• Introduce #[doc(hidden)] pub __private module with macro runtime for use by derives

[iho]

error: cannot find derive macro `BorshSerialize` in this scope
error[E0433]: failed to resolve: could not find `BorshDeserialize` in `borsh`

If you got one of those two errors, but your import is correct it means that you forgot to enable derive feature for borsh crate in your Cargo.toml.

[dj8yfo]

taking following sub-issue:

• test alpha release on near-sdk-rs and capture migration guide steps

[dj8yfo]

near-contract-standards v4.1.1 
├── near-sdk v4.1.1 
│   ├── borsh v1.0.0-alpha.2
│   ├── near-abi v0.3.0
│   │   ├── borsh v0.9.3
│   ├── near-crypto v0.17.0
│   │   ├── borsh v0.10.3
│   │   ├── near-account-id v0.17.0
│   │   │   ├── borsh v0.10.3 (*)
│   │   ├── near-config-utils v0.17.0
│   │   ├── near-stdx v0.17.0
│   ├── near-primitives v0.17.0
│   │   ├── borsh v0.10.3 (*)
│   │   ├── near-crypto v0.17.0 (*)
│   │   ├── near-fmt v0.17.0
│   │   │   └── near-primitives-core v0.17.0
│   │   │       ├── borsh v0.10.3 (*)
│   │   │       ├── near-account-id v0.17.0 (*)
│   │   ├── near-primitives-core v0.17.0 (*)
│   │   ├── near-rpc-error-macro v0.17.0 (proc-macro)
│   │   ├── near-stdx v0.17.0
│   │   ├── near-vm-errors v0.17.0
│   │   │   ├── borsh v0.10.3 (*)
│   │   │   ├── near-account-id v0.17.0 (*)
│   │   │   ├── near-rpc-error-macro v0.17.0 (proc-macro) (*)
│   ├── near-primitives-core v0.17.0 (*)
│   ├── near-sdk-macros v4.1.1 (proc-macro) 
│   ├── near-sys v0.2.0 
│   ├── near-vm-logic v0.17.0
│   │   ├── borsh v0.10.3 (*)
│   │   ├── near-account-id v0.17.0 (*)
│   │   ├── near-crypto v0.17.0 (*)
│   │   ├── near-fmt v0.17.0 (*)
│   │   ├── near-o11y v0.17.0
│   │   │   ├── near-crypto v0.17.0 (*)
│   │   │   ├── near-primitives-core v0.17.0 (*)
│   │   ├── near-primitives v0.17.0 (*)
│   │   ├── near-primitives-core v0.17.0 (*)
│   │   ├── near-stdx v0.17.0
│   │   ├── near-vm-errors v0.17.0 (*)
│   │   └── zeropool-bn v0.5.11
│   │       ├── borsh v0.9.3 (*)

near-sdk v4.1.1  (*)

near-sdk-macros v4.1.1 (proc-macro)  (*)

near-sys v0.2.0 

as dependencies are alined as follows: borsh -> nearcore -> near-sdk-rs,
instead of borsh -> near-sdk-rs -> nearcore
taking the following sub-issue first

• test alpha release on nearcore and capture migration guide steps, create a draft PR (only merge after 1.0.0 release)

[dj8yfo]

taking following sub-issue for resolution:

• test alpha release on near-sdk-rs and capture migration guide steps, create a draft PR (only merge after 1.0.0 release)

[matklad]

So the alpha is out and the API is mostly settled? I think I can allocate some time to take a look at what we've arrived at. No promises, so don't block on me, but I'll try to do API review before Monday!

[dj8yfo]

So the alpha is out and the API is mostly settled?

I hope so.
@matklad But don't hesitate to post a review, if you find any minor or serious flaws.

[matklad]

Notes from review (haven't look at the code for a while, so take as a weak advice as most!):

borsh-rs/borsh/Cargo.toml

Line 30 in 1953864

borsh-derive = { path = "../borsh-derive", version = "1.0.0-alpha.3", optional = true }

You want version = "=1.0.0-alpha.3" version most likely, to ensure that derives and the main crate are in sync.

borsh-rs/borsh/Cargo.toml

Line 31 in 1953864

hashbrown = { version = ">=0.11,<0.14", optional = true }

probably makes sense to just switch to hashbrown=0.14. Or maybe even completely get rid of it? There's probably some historical reason why we need it, but, fundamentally, we shouldn't be neededing.

borsh-rs/borsh/src/lib.rs

Lines 9 to 11 in 1953864

	* **std** -
	When enabled, this will cause `borsh` to use the standard library. Currently,
	disabling this feature will result in building the crate in `no_std` environment.

This should probably say something about nostd_io traits as that's the main unusual thing you get with no std borsh.

https://github.com/near/borsh-rs/blob/master/borsh/src/lib.rs#L39

It's a bit surprising that that's not the default. Probably makes sense to explain why we need this feature:

If this feature is not enabled, it is possible that two different byte slices could deserialize into the same object

borsh-rs/borsh/src/lib.rs

Lines 49 to 55 in 1953864

	* **hash_collections** -
	This is a feature alias, set up in `build.rs` to be equivalent to (**std** OR **hashbrown**).
	Gates implementation of [BorshSerialize](crate::ser::BorshSerialize), [BorshDeserialize](crate::de::BorshDeserialize)
	and [BorshSchema](crate::schema::BorshSchema)
	for [HashMap](std::collections::HashMap)/[HashSet](std::collections::HashSet).
	* **derive_schema** -
	This is a feature alias, set up in `build.rs` to be equivalent to (**derive** AND **schema**).

This is implementation detail, so it shouldn't go into user-facing docs. Additionally, if having feature aliases is the only reason to have build.rs, I'd probably recomend removing those --- build.rs slows down compilation somewhat, because build script needs to get compiled, linked and run even before the crate could start compiling.

borsh-rs/borsh/src/lib.rs

Line 79 in 1953864

pub mod schema_helpers;

_helpers is an odd name for publicly available API.

borsh-rs/borsh/src/lib.rs

Line 96 in 1953864

"feature \"schema\" depends on \"derive\" feature in its implementation; enable it too.."

Duplicate full stop.

borsh-rs/borsh/src/schema_helpers.rs

Lines 27 to 28 in 1953864

	let mut res = schema.try_to_vec()?;
	res.extend(value.try_to_vec()?);

Could save an allocation of intermediate vector here.

borsh-rs/borsh/src/schema.rs

Line 48 in 1953864

variants: Vec<(VariantName, Declaration)>,

Here and in structs, we don't actually enforce that names are unique. So potentially someone can hand-craft schemas with duplicate names and cause some mischief. Not sure if we need to worry about this. Similarly, we don't enforce that all declarations have corresponding definitions.

borsh-rs/borsh/src/schema.rs

Line 144 in 1953864

fn add_definition(

Feels like an odd method for a trait. It makes no sense to override this, so it probably should be a free-standing function somewhere. Also, I belive implementatin can be simplifid to

let prev = defitnions.insert(declaration, definiton);
assert!(prev.is_none())

borsh-rs/borsh/src/schema.rs

Line 162 in 1953864

fn schema_container() -> BorshSchemaContainer {

Similarly how we add top-level borsh::from_slice, this shouldn't be a trait method, just a top-level borsh::shema_of::<T> function.

borsh-rs/borsh/src/ser/mod.rs

Line 58 in 1953864

fn try_to_vec(&self) -> Result<Vec<u8>> {

Given borsh::to_vec we want to depricate or remove this method.

borsh-rs/borsh/src/ser/mod.rs

Line 151 in 1953864

assert!(

we return InvalidData when trying to serialize too long slices. So we probably shold InvalidData here as well.

borsh-rs/borsh/src/ser/mod.rs

Lines 281 to 285 in 1953864

	if size_of::<T>() == 0 {
	return Err(Error::new(
	ErrorKind::InvalidData,
	"Vectors of zero-sized types are not allowed due to deny-of-service concerns on deserialization.",
	));

I am not sure, but perhaps Rust is advanced enough that its possible to leveate this to a compile-time error via some const-fn trickery?

borsh-rs/borsh/src/ser/mod.rs

Line 371 in 1953864

vec.sort_by(|(a, _), (b, _)| a.partial_cmp(b).unwrap());

Similarly, might want to InvalidData rather than unwrap here?

borsh-rs/borsh/src/ser/mod.rs

Line 390 in 1953864

let mut vec = self.iter().collect::<Vec<_>>();

this should be consistent with Vec wrt handling zero-sized types

borsh-rs/borsh/src/de/mod.rs

Line 82 in 1953864

pub trait EnumExt: BorshDeserialize {

does this want to be in the __private module perhaps?

[dj8yfo]

vec.sort_by(|(a, _), (b, _)| a.partial_cmp(b).unwrap());

I've also noticed this one. I've skipped it so far for 2 reasons:

1. One needs to constructs an Eq + PartialOrd<Self> type in order to be able to trigger this panic, which feels kind of artificial.
2. I haven't found a fallible sort helper in standard lib.

Edit: probably the best fix for this is restricting the bound on type K to Ord, as K: PartialOrd + partial_cmp(b).unwrap() is equivalent to using subset of K, which is Ord.

[dj8yfo]

taking this subissue

• rename schema cargo feature flag to unstable__schema

[maxammann]

I marked the 1.0.0 as fixed in Rustsec: rustsec/advisory-db#1794

[frol]

@matklad Could you, please, take one more last look here? We are about to cut 1.0.0!

[matklad]

Sorry, was taking the previous week off of GitHub, but I trust you all to do a good job here!

Thanks for persevering, 1.0 for borsh is quite important!