Question about MultiIndex

[durkmurder]

Hey, I am struggling to make MultiIndex work the way I need, maybe it's not possible at all at this stage. Hope to get some advice.

Basically I have a primary key which is a tuple (Addr, String) and additional index collection which is a String. I need to be able to query by secondary index(by collection) and do filtering and pagination by primary key.
I can do this with two maps: Primary: Map<(Addr, String), Value> + SecondaryIndex: Map<(String, (Addr, String)), Empty> and then I can do something like this:

SecondaryIndex
.prefix(collection)
.keys(...)
.filter(|(_, (addr, s)| addr == filter_addr)
.take(1)
.map(|(_, (addr, s) Primary.prefix(addr).range(Bound(s), None).map(// do something with result)

This is probably doable but not very efficient and hard to maintain I would like to do something:

get_indexed_map()
.idx.collection.prefix(collection)
// here I would like to prefix primary key with filter_addr and then call range to do pagination
.range(
deps.storage,
Bound(s),
None,
Order::Ascending,

pub struct ExtraIndexes<'a> {
    // pk goes to second tuple element
    pub collection: MultiIndex<'a, (String, Vec<u8>), Value>,
}

get_indexed_map() -> IndexedMap<'a, (Addr, String), Value, ExtraIndexes<'a>>

Any advice is appreciated.

[maurolacy]

That should work. I recommend you to follow the docs at https://docs.cosmwasm.com/tutorials/storage/indexes/

If you post more specific questions / code snippets, we could help you better.

[durkmurder]

I have checked those docs, but it doesn't give me an answer.
I will give more concrete example:

Sample:

pub struct OwnedTokensIndexes<'a> {
    // pk goes to second tuple element
    pub collection: MultiIndex<'a, (String, Vec<u8>), OwnedTokensInfo>,
}

impl<'a> IndexList<OwnedTokensInfo> for OwnedTokensIndexes<'a> {
    fn get_indexes(&'_ self) -> Box<dyn Iterator<Item = &'_ dyn Index<OwnedTokensInfo>> + '_> {
        let v: Vec<&dyn Index<OwnedTokensInfo>> = vec![&self.collection];
        Box::new(v.into_iter())
    }
}

/// Mapping that represents (Owner, TokenId) -> OwnedTokensInfo
/// It used to quickly query tokens by owner without counting editions.
pub fn owned_tokens<'a>() -> IndexedMap<'a, (Addr, String), OwnedTokensInfo, OwnedTokensIndexes<'a>>
{
    let indexes = OwnedTokensIndexes {
        collection: MultiIndex::new(
            |d: &OwnedTokensInfo, k: Vec<u8>| (d.collection.clone(), k),
            "owned_tokens",
            "owned_tokens__collection",
        ),
    };
    IndexedMap::new("owned_tokens", indexes)
}

So here my value is indexed by (Owner, TokenId) and Collection. I would like to benefit from that:
I can write:

let start_after = Some("some_token_id");
owned_tokens().prefix(owner_addr).range(
            deps.storage,
            start_after.map(Bound::exclusive),
            None,
            Order::Ascending,
),

This will work as expected, I will first filter by address using prefix(owner_addr) and then use range to paginate over second element of tuple.
How should I approach this if I am looking up index first?

owned_tokens().idx.collection.prefix(collection).range(
            deps.storage,
            None, // ? what should go here? is is range over (Owner, TokenId) ? How to construct one ?
            None,
            Order::Ascending,
),

How do I prefix and then use range to get the same functionality as I was getting without using multi index.

Bonus Question:

How to parse Vec as primary key to the actual type? I've noticed when tuples are used it contains extra bytes in the beginning.

owned_tokens().idx.collection.prefix(collection).keys(
            deps.storage,
            None,
            None,
            Order::Ascending,
).map (| v | // v is Vec<u8> how to parse it?)

[maurolacy]

None, // ? what should go here? is is range over (Owner, TokenId) ? How to construct one ?

The primary key (that is, the suffix of the full key, after you prefixed it). In raw format, because the 2nd element of the index is a Vec<u8>.

To construct one, and assuming this is a start_after pagination param that comes as a String, you can use start_after.into_bytes().

If it comes as an Addr (because you validated it or so), you can do start_after.to_string().into_bytes() or similar.

If this is a pagination parameter, it's usually enough, when iterating in the client, to use the last received value (while it's non-null) and send it as the start_after param. Then you use it as the start of range with an exclusive bound.

[maurolacy]

).map (| v | // v is Vec how to parse it?)

Use from_utf8 to re-convert to an owned String, and return it.

[maurolacy]

There are examples of this in in the cw-plus contracts.

We're are also working on a version of range with full key deserialization. So that you can directly get an Addr there, instead of a Vec<u8>. This is already implemented for Map, and we plan to extend support to indexes soon.

Stay tuned.

[durkmurder]

).map (| v | // v is Vec how to parse it?)

Use from_utf8 to re-convert to an owned String, and return it.

but it's not a string, my primary key is (Addr, String) so Vec<u8> will have something like [0, 1, bytes for addr, bytes for string] at least 2 extra bytes at the beginning.

[durkmurder]

None, // ? what should go here? is is range over (Owner, TokenId) ? How to construct one ?

The primary key (that is, the suffix of the full key, after you prefixed it). In raw format, because the 2nd element of the index is a Vec<u8>.

To construct one, and assuming this is a start_after pagination param that comes as a String, you can use start_after.into_bytes().

If it comes as an Addr (because you validated it or so), you can do start_after.to_string().into_bytes() or similar.

If this is a pagination parameter, it's usually enough, when iterating in the client, to use the last received value (while it's non-null) and send it as the start_after param. Then you use it as the start of range with an exclusive bound.

owned_tokens().idx.collection.prefix(collection) at this point I can call keys or range and both can accept start and end.
I assume that start should be serialized primary key. Since my primary key is a tuple I would like to prefix using first element(addr) and range over second element(token_id), is that possible? In other words I need filtering by address and pagination by token_id

[maurolacy]

).map (| v | // v is Vec how to parse it?)

Use from_utf8 to re-convert to an owned String, and return it.

but it's not a string, my primary key is (Addr, String) so Vec<u8> will have something like [0, 1, bytes for addr, bytes for string] at least 2 extra bytes at the beginning.

Ah, I see, you're right, your primary key is composite. You'll have to deserialize it manually. This is relatively non-trivial, as the first element is length prefixed.

I'll post a code snippet in a follow-up.

[maurolacy]

None, // ? what should go here? is is range over (Owner, TokenId) ? How to construct one ?

The primary key (that is, the suffix of the full key, after you prefixed it). In raw format, because the 2nd element of the index is a Vec<u8>.
To construct one, and assuming this is a start_after pagination param that comes as a String, you can use start_after.into_bytes().
If it comes as an Addr (because you validated it or so), you can do start_after.to_string().into_bytes() or similar.
If this is a pagination parameter, it's usually enough, when iterating in the client, to use the last received value (while it's non-null) and send it as the start_after param. Then you use it as the start of range with an exclusive bound.

owned_tokens().idx.collection.prefix(collection) at this point I can call keys or range and both can accept start and end. I assume that start should be serialized primary key. Since my primary key is a tuple I would like to prefix using first element(addr) and range over second element(token_id), is that possible? In other words I need filtering by address and pagination by token_id

No. range here is over the remaining elements of the index, namely, the full primary key, which is a tuple. You can range over it, but will have to filter manually afterwards.

A way to build the bound is Bound::exclusive((owner, token_id).joined_key()). There are other ways, but this is the most concise.

Update: If you want to keep owner fixed, you can do something like:

owned_tokens().idx.collection.prefix(collection).keys(
            deps.storage,
            Some(Bound::exclusive((owner, U64Key::new(token_id)).joined_key())),
            Some(Bound::inclusive((owner, U64Key::new(u64::MAX)).joined_key())),
            Order::Ascending,
)

Assuming token_id is a u64 here. That will work (as long as you just want to keep owner fixed).

Finally, for pagination, you could either take(limit) afterwards, or use U64Key::new(token_id + limit) (taking advantage of the fact token_id is numeric.

Of course, if token_id is non-numeric, that trick will not work. You'll have to build a "max value" for your string token ids, and use that (a string of b"\xff"s of the max length of your token ids).

That's why using take(limit) is simpler / more robust; and you just use None for end range (But, you have to manually filter potential non-owners...)

[maurolacy]

To deserialize the composite:

// `value` has the original value. It has to be mutable (to use this example. You can always use `split_at` and work with slices).

// helper to deserialize the length
fn parse_length(value: &[u8]) -> StdResult<usize> {
    Ok(u16::from_be_bytes(
        value
            .try_into()
            .map_err(|_| StdError::generic_err("Could not read 2 byte length"))?,
    )
    .into())
}

let mut tu = value.split_off(2);
let t_len = parse_length(&value)?;
let u = tu.split_off(t_len);

// Now `tu` is the `Addr`, and `u` is the `String`
let addr = Addr::unchecked(from_utf8(tu)?);
let string = from_utf8(u)?;

[maurolacy]

Makes sense? As I said previously, we're working on ways to make all this (much?) simpler to use.

[durkmurder]

Thanks a lot for responses, now it makes sense. I had suspicions that it's the case but needed to verify it. Any chance that you could introduce prefix after querying the index? Would really help a lot + I am really waiting for deserialization updates(I've seen PR in progress).

[maurolacy]

We recently added prefix_range to Map, which in fact, allows you to fix a prefix and iterate over the rest. See #446.

But it will not work / it's still not supported here (i.e. after a prefix). We will consider adding it, and also for keys. So that you could in principle do something like:

owned_tokens().idx.collection.prefix(collection).prefix_keys(
            deps.storage,
            Some(PrefixBound::inclusive(owner)),
            Some(PrefixBound::inclusive(owner)),
            Order::Ascending,
)

I.e. give me all the keys that start with owner.

[maurolacy]

In any case, take a look at the code above. It will work, and properly support pagination / continuation.

I mentioned some stuff about fixed vs. variable length addresses, that I now deleted, because that's only an issue in case you want to continue iterating past owner, and have variable length strings. If you precisely want to keep owner fixed, code above should work, and not miss elements.

[durkmurder]

In any case, take a look at the code above. It will work, and properly support pagination / continuation.

I mentioned some stuff about fixed vs. variable length addresses, that I now deleted, because that's only an issue in case you want to continue iterating past owner, and have variable length strings.

I am trying to utilize U64Key after your suggestions, is there a function to deserialize Vec into U64Key? I need it to access key in range().map()

[maurolacy]

Just use let id = u64::from_be_bytes(value.as_slice().try_into()...) and you'll recover the number. Then U64Key::new(id), and you have it.

[durkmurder]

Just use let id = u64::from_be_bytes(value.as_slice().try_into()...) and you'll recover the number. Then U64Key::new(id), and you have it.

Thanks.
I am doing the same with Endian::from_be_bytes, seems does the trick.

[maurolacy]

Thanks. I am doing the same with Endian::from_be_bytes, seems does the trick.

Yeah, it's just a wrapper around it. That I'm planning to deprecate soon; along with IntKey. See #472 . 😸