-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Discussion] Dynamic Query Builder Design #291
Comments
I guess that would result in usage like the following to perform a multi-row insert: // E.g.
let records: Vec<(Uuid, i32)> = vec![...];
let records_iter = records.iter();
let query = QueryBuilder::new("INSERT INTO users VALUES ")
.comma_sep(move |builder| {
if let Some(record) = records_iter.next() {
builder.push("(");
builder.push_bind(record.0);
builder.push(", ");
builder.push_bind(record.1);
builder.push(")");
true
} else {
false
}
})
.build();
let result = query.execute(&conn).await?;
assert_eq!(result, 2); This would be a decent starting point, though the inner building would be a bit painful for large structs. |
Prisma has a working SQL builder: https://github.com/prisma/quaint |
Note that this interface would be aimed at being used by quaint. Not at replacing Quaint. We want to reach for low-level while still doing what we can with our db-specific knowledge ( placeholders, ID quoting, etc. ). |
Maybe instead we have |
Any progress on this? I feel like this is very much a necessary feature for real world applications. |
Any update on this? |
Wouldn't that by virtue of the builder also clone the arguments? It seems like that would be a bit heavy. Alternatively, if the design had a placeholder type param, we could clone the query "plan" and later bind the args. |
Hmm, no, the arguments would be wrapped in |
do you use push_bind to repopulate the builder args?
…On Fri, Sep 11, 2020, 6:03 PM Austin Bonander ***@***.***> wrote:
Wouldn't that by virtue of the builder also clone the arguments?
Hmm, no, the arguments would be wrapped in Option internally and it would
probably panic if called more than once. Really though, the only reason to
take &mut self here is so that it can be called at the end of a method
chain.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#291 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABRMQUOK7SFG4UQF7UBXABLSFKNETANCNFSM4MUDDMWA>
.
|
I wouldn't mind trying to implement a basic query builder similar to the one you had suggested @abonander . Should I give it a shot? |
@nkconnor @elertan please do! We unfortunately don't have much time to dedicate to SQLx these days but we're trying to get back on it. |
Hi, anyone can advise me about how to make a massive database ingestion (> 200k rows) with SQLX, there is a bulk operation builder or somethig like that?, thanks |
@shiftrtech from my experience wrapping bunch of separate inserts into a transaction gives roughly the same speed |
Can you give me some basic example? |
let mut tx = db.begin().await?;
for row in your_data {
sqlx::query!("INSERT ...").execute(&mut tx).await?;
}
tx.commit().await?; |
@abonander @elertan Hey, did you guys started working on this? If not I can take this up. |
By all means go at it, I honestly kind of left this in the dust for quite a while, but would be great to have this feature implemented. |
@elertan Awesome.. will start on it right away and will keep this thread updated. |
Is there anything that can be used by now? All I can find are some incomplete snippets... I really like sqlx, but I'm at a point where I need to construct a filter based on user input (the filters for a list endpoint) and I don't see a way of doing this except writing my own query builder which assembles the sql string and the arguments. Is there a way that I'm just overseeing? |
@trusch I ended up just building the filter on the db level. As you can see in the following example some parameters are optional and you can check if they're exist on postgres level. async fn fetch_items(
db: &PgPool,
start: DateTime<FixedOffset>,
finish: DateTime<FixedOffset>,
company: Option<i32>,
purpose: Option<&str>,
from_id: Option<i32>,
limit: i64,
) -> Result<Vec<Transaction>> {
Ok(sqlx::query_as!(
Transaction,
r#"
SELECT
-- total number of rows before limit applied
count(*) OVER() as full_count,
t.id,
c.id as company_id,
c.name as company_name,
t.type as kind,
CAST(t.amount as DOUBLE PRECISION),
CAST(t.aws_amount as DOUBLE PRECISION),
CAST(t.current_balance as DOUBLE PRECISION),
t.user_id,
-- hotfix for a sqlx bug - it doesn't see the field as nullable
u.full_name as "user_name?",
t.extra,
t.created,
t.charge_for,
t.software
FROM billing_transaction t
JOIN app_company c on c.id = t.account_id
LEFT JOIN user_user u on u.id = t.user_id
WHERE t.created >= $1
AND t.created < $2
AND ($3::int IS null OR t.account_id = $3)
AND ($4::text IS null OR t.extra -> 'computer' -> 'type' ->> 'purpose' = $4::text)
AND ($5::int IS null OR t.id < $4::int)
ORDER BY t.created DESC
LIMIT $6
"#,
start,
finish,
company,
purpose,
from_id,
limit,
)
.fetch_all(db)
.await?)
} |
@imbolc Thanks for the hint! Unfortunatly my usecase is even more dynamic since I want to filter on a jsonb field and the structure of it is only known at runtime. So I really get something like a tuple of fieldname, operation and value and need to translate that to something like |
Dynamic behavior like that is simply not something that SQLx tries to target. Even with this proposal we aren't going to be adding a SQL builder that lets you fluently build up a query. This is a minimal proposal to make it easier for a library to wrap SQLx to provide such an API. The library ormx may work for you. You can add optional clauses to your query. Another library choice could be any SQL builder. Something like https://github.com/perdumonocle/sql-builder that you then pass the resulting SQL into SQLx. |
How about use a static query to deal with dynamic user input? // assume age is an optional variable from user input
// age: Option<i32>
let users = sqlx::query_as!(User, r#"
SELECT * From "User" WHERE (age = $1 OR 0 = $2 );
"#,
age.unwrap_or_default(),
age.unwrap_or_default()
).fetch_all(&ctx.db).await.ok().unwrap_or_default(); pros:
cons:
|
I can second https://github.com/perdumonocle/sql-builder . It does what's being asked here and integrates nicely with sqlx. |
Is it sql injections free? |
I disagree. SqlBuilder is nice and simple, but it lacks some required APIs to be able to build any query with it. It does include preventive measures such as 'quote' against sql injection and the like. I recently stumbled upon SeaQuery which is a little more complicated but is more flexible than SqlBuilder is. |
I recently too start using SeaQuery and really like it, and the best thing is that you can construct a SeaQuery connection from a sqlx pool, so you can with easy have a shared sqlx pool for static queries and using the same pool for SeaQuery too. |
Hey I hope I don't derail the focus on this issue and I am happy to create this somewhere else, but have you guys ever thought about a a declarative api as well, in addition to the "fluent" query builder proposal ? |
Anyone given this a try? Looks simple |
Can someone help me understand how If I have this: fn get_all(pool: &PgPool, sort_by = Option<String>) -> ... {
let mut query: QueryBuilder<Postgres> = QueryBuilder::new(
r#"SELECT
id,
name
FROM thing_users"#
)
if let Some(s) = sort_by {
query.push("ORDER BY ").push_bind(s);
};
}
query.build_query_as::<User>().fetch_all(pool).await The I think I'm fundamentally misunderstanding how |
If
with If you're trying to order by the column
Of course, if |
We have received a lot of requests, and have a significant need ourselves, for a dynamic query builder in SQLx that can generate and execute valid SQL with the appropriate bind parameters for a given database backend at runtime.
This is to do things like:
where
clauses based on user inputINSERT INTO foo(...) VALUES(...)
statements with a variable number of records in theVALUES
expression, taking into account the SQL flavor's max number of bind parameters per query (which we can build a more specialized API for on top of this)MVP
I'm thinking the minimum viable product for this would be something like the following:
Thoughts to the Future
quote::quote!()
INSERT INTO
statementsNon-Goals
cc @mehcode @izik1 @danielakhterov
The text was updated successfully, but these errors were encountered: