Having trouble implementing OneOrMany

[twiclo]

I'm making a simple api backend and I would like the insert endpoint to be able to take a single object or an array objects (without having to just wrap everything in an array)

I believe that I need a struct that I can implement OneOrMany on so I created a wrapper struct around my "NewDbObject" type which is a type on a trait. My models implement this trait.

#[serde_as]
#[derive(Deserialize)]
pub struct MaybeMany<'a, NewDbObject: serde::Deserialize<'a>> {
  #[serde_as(deserialize_as = "OneOrMany<_, PreferMany>")]
  items: Vec<NewDbObject>
}

This is the error I'm getting

error[E0392]: parameter 'a is never used
--> src/utils.rs:84:22
|
84 | pub struct MaybeMany<'a, NewDbObject: serde::Deserialize<'a>> {
| ^^ unused parameter
|
= help: consider removing 'a, referring to it in a field, or using a marker such as PhantomData

Ideally I could just have my insert function on my trait take a OneOrMany<T>:

#[async_trait]
pub trait DbObject: Send + Serialize + 'static {
  type NewDbObject: Send;

  async fn insert(db: web::Data<MySqlPool>, input: OneOrMany<Self::NewDbObject>) -> Result<TableId, sqlx::Error> where Self: Sized;
}

Thank you for any help

[jonasbb]

I'm confused how your code looks like. Maybe you could provide a larger example. It would be good to see where you declare MaybeMany, where you use it, and how the text you want to deserialize looks like.
Given how you wrote MaybeMany it looks like it is inside a impl DbObject scope.

The OneOrMany type is only useful during deserialization. The idea is that your code can use Vec<T> everywhere. On the struct you annotate the Vec-field with OneOrMany to allow it to be deserialized from different representations. OneOrMany is not useful anywhere besides the serde_as annotation.

[twiclo]

The entry point is main.rs the endpoint localhost/enclosures/id is set up to take a json payload from the caller

#[actix_rt::main]
async fn main() -> Result<(), std::io::Error> {
  dotenv::dotenv().ok();
  let db_url = std::env::var("DATABASE_URL").expect("DATABASE_URL must be set");

  let pool = MySqlPoolOptions::new()
    .max_connections(10)
    .connect(&db_url).await.unwrap();

  HttpServer::new(move || {
    App::new()
      .data(pool.clone())
      .service(
        web::resource("/enclosures")
          .route(web::get().to(utils::get::<Enclosure>))
          .route(web::post().to(utils::add::<Enclosure>))
      )

The payload gets sent to utils::add:

pub async fn add<Model: DbObject>(db: web::Data<MySqlPool>, input: web::Json<Model::NewDbObject>) -> Result<HttpResponse, Error> {
  Ok(Model::insert(db, input.into_inner())
    .await
    .map(|db_object| HttpResponse::Created().json(db_object))
    .map_err(|_| HttpResponse::InternalServerError())?)
}

Here is the trait that NewDbObject is declared on and the MaybeMany struct declaration

#[async_trait]
pub trait DbObject: Send + Serialize + 'static {
  type NewDbObject: Send;

  async fn insert(db: web::Data<MySqlPool>, input: Self::NewDbObject) -> Result<TableId, sqlx::Error> where Self: Sized;
}

#[serde_as]
#[derive(Deserialize)]
pub struct MaybeMany<'a, NewDbObject: serde::Deserialize<'a>> {
  #[serde_as(deserialize_as = "OneOrMany<_, PreferMany>")]
  items: Vec<NewDbObject>
}

It would be good to see where you declare MaybeMany, where you use it, and how the text you want to deserialize looks like.

Declaration shown above. I haven't tried using it yet. The text I'd like to deserialize is either going to be:

{
    "A": "b",
    "C": "d"
}

or

[
    {
        "A": "b",
        "C": "d"
    },
    {
        "A": "e",
        "C": "f"
    }
]

The idea is that your code can use Vec everywhere.

This is what I would like to do once I get the declaration building. I want the endpoint caller to be able to pass an array of JSONs or just one. I want it all shoved into a vec regardless

[jonasbb]

It seems that your MaybeMany struct is not actually dependent on the trait. You are only using NewDbObject as a generic here. Given the two samples inputs the correct way to define MaybeMany is like this.

#[serde_as]
#[derive(Deserialize)]
#[serde(bound(deserialize = "T: Deserialize<'de>"))]
pub struct MaybeMany<T>(
  #[serde_as(deserialize_as = "OneOrMany<_>")]
  pub Vec<T>
);

#[derive(Debug, PartialEq, Deserialize)]
#[serde(rename_all = "SCREAMING_SNAKE_CASE")]
pub struct Data {
    a: String,
    c: String,
}

// Test that deserializing works

let single = serde_json::json!({
    "A": 'b',
    "C": 'd'
});

let multi = serde_json::json!([
    {
        "A": 'b',
        "C": 'd'
    },
    {
        "A": 'e',
        "C": 'f'
    }
]);

let s: MaybeMany<Data> = serde_json::from_value(single)?;
assert_eq!(vec![Data{a: "b".to_string(), c: "d".to_string()}], s.0);
let m: MaybeMany<Data> = serde_json::from_value(multi)?;
assert_eq!(
    vec![
        Data{a: "b".to_string(), c: "d".to_string()},
        Data{a: "e".to_string(), c: "f".to_string()},
    ],
    m.0
);

In that case you should be able to define your add function roughly like this. The MaybeMany is part of the input type. When you are using it, you access the only member of the newtype struct, i.e., .into_inner().0.

pub async fn add<Model: DbObject>(db: web::Data<MySqlPool>, input: web::Json<MaybeMany<Model::NewDbObject>>) -> Result<HttpResponse, Error> {
  Ok(Model::insert(db, input.into_inner().0)
    .await
    .map(|db_object| HttpResponse::Created().json(db_object))
    .map_err(|_| HttpResponse::InternalServerError())?)
}

You can either keep the insert function to only insert a single element, in which case you would need to adapt your add. Alternatively, the insert function could now take an input: Vec<Self::NewDbObject>.

Does that help you solve your problem?

[twiclo]

That was what I needed. Thank you

[twiclo]

I just finished implementing this and I still need to wrap the payload in square brackets:

curl -v -H "Content-Type: application/json" -X POST -d '[{"site_id": 1, "category": "drop", "name": "testName", "location": {"lat": 10.0, "lng": 11.0}}]' localhost:8081/enclosures

I get a 400 without the square brackets. Here's the code:

#[async_trait]
pub trait DbObject: Send + Serialize + 'static {
  type NewDbObject: Send;

  async fn insert(db: web::Data<MySqlPool>, inputs: Vec<Self::NewDbObject>) -> Result<Vec<TableId>, sqlx::Error> where Self: Sized;
}

#[serde_as]
#[derive(Deserialize)]
#[serde(bound(deserialize = "NewDbObject: Deserialize<'de>"))]
pub struct MaybeMany<NewDbObject> {
  #[serde_as(deserialize_as = "OneOrMany<_, PreferMany>")]
  items: Vec<NewDbObject>
}

async fn insert(db: web::Data<MySqlPool>, inputs: Vec<NewEnclosure>) -> Result<Vec<TableId>, sqlx::Error> {
  let mut res: Vec<TableId> = Vec::new();
  for input in inputs {
    let cat_id = sqlx::query!("select category_id from enc_cat where category = ?", input.category)
      .fetch_one(db.get_ref()).await?.category_id;
    let enc_id = sqlx::query!("INSERT INTO enclosure (site_id, category_id, lat, lng) values (?, ?, ?, ?)",
        input.site_id,
        cat_id,
        input.location.lat,
        input.location.lng)
      .execute(db.get_ref()).await?
      .last_insert_id();
    res.push(TableId { id: enc_id });
  }
  Ok(res)
}

[jonasbb]

Why do you use a field items in your MaybeMany struct? I don't see that in the JSON and my earlier post also defines MaybeMany differently. So it seems to me like you are not using MaybeMany right now. Did you change the input on the add to something like input: web::Json<MaybeMany<Model::NewDbObject>>?

[twiclo]

I don't think I can just put this macro on my NewEnclosure struct because I assume OneOrMany needs a vec to deserialize into. If I just put it on NewEnclosure there'd be no vec. So instead I'm making a vec of NewEnclosures.

Here's the add function:

pub async fn add<Model: DbObject>(db: web::Data<MySqlPool>, input: web::Json<MaybeMany<Model::NewDbObject>>) -> Result<HttpResponse, Error> {
  Ok(Model::insert(db, input.into_inner().items)
    .await
    .map(|db_object| HttpResponse::Created().json(db_object))
    .map_err(|_| HttpResponse::InternalServerError())?)
}

Here's the NewEnclosure struct if it helps:

#[derive(Deserialize, Debug)]
pub struct NewEnclosure {
  pub site_id: u64,
  pub category: String,
  pub name: String,
  pub location: Coordinate
}

[jonasbb]

put this macro on my NewEnclosure struct

Which macro? #[serde_as], a derive-macro?

Could you still answer this questions? Why do you use a field items in your MaybeMany struct? items does not appear in your JSON.

It would be really helpful if you could produce a self-contained example in which I can just execute cargo run and see the problems you have for myself.

[twiclo]

I believe I was answering your question. I put items on there because a vec needs a key for every item. I believe I need this wrapper vec because I believe OneOrMany needs a vec to put many NewDbObjects in to. If I can just put this derive macro (serde_as) on my NewDbObject, in this case NewEnclosure, and get a vec of enclosures then I would do that but I believe OneOrMany would be trying to give me a NewEnclosure that had many vecs of every item on a NewEnclosure

Here's the struct in question

pub struct NewEnclosure {
  pub site_id: u64,
  pub category: String,
  pub name: String,
  pub location: Coordinate
}

I would love if I could just get this out of OneOrMany

Vec<NewEnclosure>

But I believe that if I put the derive-macro on NewEnclosure I would get this:

pub struct NewEnclosure {
  pub site_id: Vec<u64>,
  pub category: Vec<String>,
  pub name: Vec<String>,
  pub location: Vec<Coordinate>
}

[jonasbb]

because a vec needs a key for every item

That is not true. You can deserialize a [1,2,3] into a Vec<u32> perfectly fine.

But I believe that if I put the derive-macro on NewEnclosure I would get this:

No, this is never what the #[serde_as] attribute does. It only transforms #[serde_as] attributes on fields such that serde understands them.

Regardless, why do you want to apply #[serde_as] suddenly to NewEnclosure? That was never discussed so far. What you did is correct, namely define the MaybeMany<T> type and use that. Could you try to use the solution I posted here? It doesn't use items. Defining a field called items would require the JSON to also have one, but so far no of your examples show this.

If you deserialize a MaybeMany<NewEnclosure> and access its inner type, you get a Vec<NewEnclosure>.

let mm: MaybeMany<NewEnclosure> = todo!();
let enclosures: Vec<NewEnclosure> = mm.0;

[twiclo]

Ideally I would get rid of MaybeMany. I only created it because I thought I would need a wrapper vec around NewEnclosure.

Defining a field called items would require the JSON to also have one
I successfully pushed json with this curl command and it doesn't contain items

curl -v -H "Content-Type: application/json" -X POST -d '[{"site_id": 1, "category": "drop", "name": "testName", "location": {"lat": 10.0, "lng": 11.0}}]' localhost:8081/enclosures

If you deserialize a MaybeMany and access its inner type, you get a Vec.
I have been doing this and it has been working. The problem is that I need to wrap my json in [] in order for it to work. I would hope that if only one NewEnclosure was provided that it wouldn't require the []

[twiclo]

I feel like we're not on the same page so let me start over.

I have a trait called DbObject. There's a type on that trait called NewDbObject. There's also a function insert that takes the json payload, deserializes it, adds it to the DB, then responds with the ID of the newly created row.

We're working specifically with Enclsoure and NewEnclosure here:

#[derive(Serialize, Debug)]
pub struct Enclosure {
  pub enclosure_id: u64,
  pub site_id: u64,
  pub category: String,
  pub name: String,
  pub location: Coordinate
}
#[derive(Deserialize, Debug)]
pub struct NewEnclosure {
  pub site_id: u64,
  pub category: String,
  pub name: String,
  pub location: Coordinate
}

This works just fine when attempting to create only one NewEnclosure. Notice the lack of square brackets

curl -v -H "Content-Type: application/json" -X POST -d
'{
  "site_id": 1,
  "category": "drop",
  "name": "testName",
  "location": {"lat": 10.0, "lng": 11.0}
}'
localhost:8081/enclosures

I would like to have it so I could either only provide one NewEnclosure OR provide an array of NewEnclosures

curl -v -H "Content-Type: application/json" -X POST -d
'[
  {
    "site_id": 1,
    "category": "drop",
    "name": "testName",
    "location": {"lat": 10.0, "lng": 11.0}
  },
  {
    "site_id": 1,
    "category": "drop",
    "name": "TEST2",
    "location": {"lat": 10.0, "lng": 11.0}
  }
]'
localhost:8081/enclosures

Ideally I could do one or the other. If I was only to pass one NewEnclosure I would like to not have to provide the square brackets in my payload.

Forgetting everything else I've said about MaybeMany and whatever, how would you implement OneOrMany to do this?

[jonasbb]

// The struct we want to deserialize
#[derive(Deserialize, Debug)]
pub struct NewEnclosure {
    pub site_id: u64,
    pub category: String,
    pub name: String,
    pub location: Coordinate
}

#[derive(Deserialize, Debug)]
pub struct Coordinate {
    pub lat: f32,
    pub lng: f32,
}

// Two sample JSON inputs which need to be deserializable
let json_single = serde_json::json!({
    "site_id": 1,
    "category": "drop",
    "name": "testName",
    "location": {"lat": 10.0, "lng": 11.0}
});
let json_multi = serde_json::json!([
    {
        "site_id": 1,
        "category": "drop",
        "name": "testName",
        "location": {"lat": 10.0, "lng": 11.0}
    },
    {
        "site_id": 1,
        "category": "drop",
        "name": "TEST2",
        "location": {"lat": 10.0, "lng": 11.0}
    }
]);

// In the code base we want to use `Vec<NewEnclosure>` where ever possible
// And we know we can use `OneOrMany` to achieve that.
// But we need a struct where we can apply the `#[serde_as]` attribute, such that we can use `OneOrMany`.
#[serde_as]
#[derive(Deserialize)]
// The automatically derived trait bounds won't work, so we need to override them
#[serde(bound(deserialize = "T: Deserialize<'de>"))]
pub struct Wrap<T>(
    #[serde_as(deserialize_as = "OneOrMany<_>")]
    pub Vec<T>
);

// Now check that deserialization works with both sample inputs
let wrap_single: Wrap<NewEnclosure> = serde_json::from_value(json_single)?;
let single: Vec<NewEnclosure> = wrap_single.0;
println!("single: {:?}", single);
assert_eq!(1, single.len());

let wrap_multi: Wrap<NewEnclosure> = serde_json::from_value(json_multi)?;
let multi: Vec<NewEnclosure> = wrap_multi.0;
println!("multi: {:?}", multi);
assert_eq!(2, multi.len());

Where we tell the JSON code which type we want, we write Wrap<NewEnclosure>. You can simply access the field of the newtype to get the Vec<NewEnclosure> which is contained inside the Wrap newtype.

How you use the Wrap struct in your codebase now I'm not quite sure. I think changing the function definition of your add function, which seems to be responsible for the POST requests should be enough.
Something like this might work:

pub async fn add<Model: DbObject>(db: web::Data<MySqlPool>, input: web::Json<Wrap<NewEnclosure>>) -> Result<HttpResponse, Error> {...}

but you add function seems to rely on the DbObject trait, so likely more something like this:

pub async fn add<Model: DbObject>(db: web::Data<MySqlPool>, input: web::Json<Wrap<DbObject::NewDbObject>>) -> Result<HttpResponse, Error> {...}

[twiclo]

Okay I've gotten the struct made and its on my insert function:

#[async_trait]
pub trait DbObject: Send + Serialize + 'static {
  type NewDbObject: Send;

  async fn insert(db: web::Data<MySqlPool>, inputs: Vec<Self::NewDbObject>) -> Result<Vec<TableId>, sqlx::Error> where Self: Sized
}

#[serde_as]
#[derive(Deserialize)]
#[serde(bound(deserialize = "NewDbObject: Deserialize<'de>"))]
pub struct MaybeMany<NewDbObject> (
  #[serde_as(deserialize_as = "OneOrMany<_, PreferMany>")]
  Vec::<NewDbObject>

I'm having trouble getting it through the add function:

pub async fn add<Model: DbObject>(db: web::Data<MySqlPool>, input: web::Json<MaybeMany<Model::NewDbObject>>) -> Result<HttpResponse, Error> {
  let x: Vec<Model::NewDbObject> = input.0;
  Ok(Model::insert(db, x)
    .await
    .map(|db_object| HttpResponse::Created().json(db_object))
    .map_err(|_| HttpResponse::InternalServerError())?)
}

error[E0308]: mismatched types
  --> src/utils.rs:47:35
   |
47 |     let x: Vec<Model::NewDbObject> = input.0;
   |            -----------------------   ^^^^^^^ expected struct `Vec`, found struct `MaybeMany`
   |            |
   |            expected due to this
   |
   = note: expected struct `Vec<<Model as DbObject>::NewDbObject>`
              found struct `MaybeMany<<Model as DbObject>::NewDbObject>`

[twiclo]

Oh, doing input.0.0 fixed this. Odd

[twiclo]

Answer: #308 (reply in thread)