Use Maps for storage

[ethanfrey]

This pulls out a lot of the ground-work from #108 without the actual indexes.

Shows another approach instead of Buckets. Code is basically done. Time for feedback and polish

• Refine API
• More unit test
• Proper README
• Use in some contract

I will add a refinement of the Range queries in a follow up PR and add the docs for them and contract integrations there.

[ethanfrey]

This is heavily WIP, but I would like a first review on the public API.
I will be improving some of the internals, and adding more functions, but the public API is ready for review.

Example:

// note these don't need 2 functions to define (like Buckets), but one compile-time `const`
const PEOPLE: Map<&[u8], Data> = Map::new(&[b"people", b"_pk"]);
const ALLOWANCE: Map<(&[u8], &[u8]), u64> = Map::new(&[b"allow", b"_pk"]);

fn demo() -> StdResult<()> {
    // usage for simple keys - either caching the path, or just generating it on the fly
    let john = PEOPLE.key(b"john");
    john.save(&mut store, input)?;
    let data = PEOPLE.load(&store, b"john")?;

    // auto-generate usage for complex (composite) keys
    let allow = ALLOWANCE.key((b"owner", b"spender"));
    allow.save(&mut store, &1234)?;
    let amount = allow.load(&store)?;
    
    // update, using keys inline
    let add_ten = |a: Option<u64>| -> StdResult<_> { Ok(a.unwrap_or_default() + 10) };
    ALLOWANCE.update(&mut store, (b"owner", b"spender"), add_ten).unwrap();

    // with iterator feature, we allow range queries
    // when the key is &[u8] and prefix (), we special-case it and provide it directly
    let people_iter = PEOPLE.range(&store, None, None, Order::Ascending);
    let allow_iter = ALLOWANCE.prefix(b"owner").range(&store, None, None, Order::Ascending);
}

Compare usage with Bucket (ignoring the long definitions for now)

    // current cosmwasm-storage usage is (with hand-written functions):
    let data = people_read(&store).load(b"john")?;

    // now we can use helpers and just:
    let data = PEOPLE.load(&store, b"john")?;

[maurolacy]

Hi, will review asap.

[ethanfrey]

This should be ready to merge from my side. I will add more improvements in future PRs - like adjusting the Range interface, adding secondary indexes, and maybe performance/code improvements.

I would like to merge this in piece by piece as soon as it is high enough quality with no obvious trouble in what is implemented.

I would love feedback on the README, which should give a nice overview of how to use the code.

[ethanfrey]

Merging now, more coming soon.

Feel free to comment on this and ping me if wished. Or just raise issues. I will address some points in future PRs.

[maurolacy]

Looks good (though relatively complex) to me.

A general comment: It strikes me that what you're implementing is basically tantamount to a file system. A namespace defines a particular fs, prefixes are equivalent to directories, and keys are files; which contain (structured) data.

This is out of my area of expertise, as I've never had to deal with an implementation at this level. But I can't help but wonder if taking a look at actual fs implementations, design decisions, tradeoffs, etc. would be of any help.

By example: Maybe making the "prefixes" (directories) a different / specific data structure, containing data of its contents, could be of help, and be really useful for (sub)listing / selection / sorting.

Update: Will continue reviewing and analysing during the day, posting single comments if I find something else.

[maurolacy]

I think this is a really good idea / approach in general. That is, to separate a working impl from new ideas / concepts, to try and test them first.

[maurolacy]

Probably not so efficient, but this could be resolved with multiple calls to to_length_prefixed().

[ethanfrey]

Exactly, I want to make this more efficient. I will trim all these (internal) functions down at the end to one or two that do what we need.

[ethanfrey]

I'm going to make a PR using much of this cleanup, and improve naming, but I want to get the Range PR in first. Then do the improvements. Can you please take a quick look at the API there?

A number of devs are hitting issues as the last released version of much of the code doesn't match the last wasmd release, so I want to get a 0.3.0 tag out soon, which for me requires finalizing these interfaces (even if I add more later, not to break them)

[maurolacy]

I'm going to make a PR using much of this cleanup, and improve naming, but I want to get the Range PR in first. Then do the improvements. Can you please take a quick look at the API there?

I'm on it right now.

Update: Sorry to block you, by the way. In any case, just merge stuff and I can always review it later if you want, and add my comments.

[ethanfrey]

Sure. I do appreciate your reviews and they push me to make cleaner code and better APIs.