-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
sql: Initial support for PostgREST #69010
Comments
do you know why postgrest needs SET ROLE? |
https://postgrest.org/en/stable/auth.html does a better job explaining this than I will be able to. There is overlap here with Also, to be specific, PostgREST actually uses |
Needed for cockroachdb#69010. This commit implements the `information_schema._pg_index_position` builtin function. Given an index's OID and an underlying-table column number, `information_schema._pg_index_position` return the column's position in the index (or NULL if not there). The function is implemented as a user-defined function in Postgres here: https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql Release note: None. Release justification: None, waiting for v22.1.
…a._pg_truetypmod Needed for cockroachdb#69010. This commit adds implementations for the `information_schema._pg_truetypid` and `information_schema._pg_truetypmod` builtin functions. These functions return the "true" type ID and modifier, disregarding indirection introduced by domain types. The builtins are implemented as user-defined functions in Postgres here: https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql Release justification: None, waiting for v22.1.
Needed for cockroachdb#69010. This commit adds implementations for the `information_schema._pg_char_max_length` builtin function. This functions returns the maximum character length of a type with the provided ID and modifier. The builtin is implemented as a user-defined function in Postgres here: https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql Release justification: None, waiting for v22.1.
Needed for cockroachdb#69010. This commit implements the `information_schema._pg_index_position` builtin function. Given an index's OID and an underlying-table column number, `information_schema._pg_index_position` return the column's position in the index (or NULL if not there). The function is implemented as a user-defined function in Postgres here: https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql Release note: None. Release justification: None, waiting for v22.1.
I've knocked out each of the missing introspection builtins. They were all reasonably straightforward, in part due to recent investments in role-based access control, and will land in v22.1. With that added PG compatibility, we're now able to run startup queries like the following from in PostgREST:
From what I can tell, that leaves WITH RECURSIVE UNION as the remaining startup blocker. @RaduBerinde has estimated that will take 2-4 weeks of work to land. Things aren't as far along with the remaining runtime blockers. I've identified 4 cases where our JSON support is insufficient to run queries through PostgREST. The concerning part about these is that they aren't surface-level deficiencies. From what I can tell, our SQL type-system is not currently powerful enough to implement builtins like I'd like to get @otan's eyes on this at some point because he'll be able to classify these limitations and map them to existing initiatives. Until then, my takeaway is that we need to keep investing in our type system and JSON support. More and more tools are interested in tight JSON integration, and we have some real gaps in this area. These gaps will be considered hard blockers for these tools, and they're not the kinds of blockers that can be worked around in the tools. They need to be solved in CockroachDB itself. |
There have been a few other issues that have come up around our divergence in how we handle
just providing these for context - perhaps the issues that you discuss above should be included as part of this theme. |
Needed for cockroachdb#69010. This commit implements the `pg_is_other_temp_schema` builtin function. `pg_is_other_temp_schema` returns true if the given OID is the OID of another session's temporary schema. This can be useful, for example, to exclude other sessions' temporary tables from a catalog display. Release note (sql change): The pg_is_other_temp_schema builtin function is now supported, which returns whether the given OID is the OID of another session's temporary schema. Release justification: None, waiting for v22.1.
By the way, Supabase sits on top of PostgREST, so for all intents and purposes, these are blockers for Supabase support as well. |
Needed for cockroachdb#69010. This commit implements the `pg_has_role` builtin function. `pg_has_role` returns whether the user has privileges for a specified role or not. Allowable privilege types are MEMBER and USAGE. MEMBER denotes direct or indirect membership in the role (that is, the right to do SET ROLE), while USAGE denotes whether the privileges of the role are immediately available without doing SET ROLE. `pg_has_role` was the last remaining unimplemented "access privilege inquiry functions", and was omitted from 94c25be because our role-based access control system was not mature enough to support it at the time. The commit also makes a small modification to `pg_catalog.pg_roles` and `pg_catalog.pg_authid` to reflect that fact that all users and roles inherit the privileges of roles they are members of. Release note (sql change): The pg_has_role builtin function is now supported, which returns whether a given user has privileges for a specified role or not. Release justification: None, waiting for v22.1.
Needed for cockroachdb#69010. This commit implements the `pg_is_other_temp_schema` builtin function. `pg_is_other_temp_schema` returns true if the given OID is the OID of another session's temporary schema. This can be useful, for example, to exclude other sessions' temporary tables from a catalog display. Release note (sql change): The pg_is_other_temp_schema builtin function is now supported, which returns whether the given OID is the OID of another session's temporary schema. Release justification: None, waiting for v22.1.
Needed for cockroachdb#69010. This commit implements the `information_schema._pg_index_position` builtin function. Given an index's OID and an underlying-table column number, `information_schema._pg_index_position` return the column's position in the index (or NULL if not there). The function is implemented as a user-defined function in Postgres here: https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql Release note (sql change): The `information_schema._pg_index_position` builtin function is now supported, which improves compatibility with PostgreSQL. Release justification: None, waiting for v22.1.
69913: sql: implement information_schema.{_pg_truetypid, _pg_truetypmod, _pg_char_max_length} r=nvanbenschoten a=nvanbenschoten Needed for #69010. This PR adds implementations for the following three builtin functions `information_schema._pg_truetypid` `information_schema._pg_truetypmod` `information_schema._pg_char_max_length` The first two functions return the "true" type ID and modifier, disregarding indirection introduced by domain types. The third returns the maximum character length of a type with the provided ID and modifier. The builtins are implemented as user-defined functions in Postgres here: https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql Combined with #69909 and #69911, this PR unlocks these two gnarly introspection queries in PostgREST: - https://github.com/PostgREST/postgrest/blob/b05898d17f8e33c8c82fc1d05a30eb3044999668/src/PostgREST/DbStructure.hs#L538 - https://github.com/PostgREST/postgrest/blob/b05898d17f8e33c8c82fc1d05a30eb3044999668/src/PostgREST/DbStructure.hs#L709 Release justification: None, waiting for v22.1. Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Needed for cockroachdb#69010. This commit implements the `pg_is_other_temp_schema` builtin function. `pg_is_other_temp_schema` returns true if the given OID is the OID of another session's temporary schema. This can be useful, for example, to exclude other sessions' temporary tables from a catalog display. Release note (sql change): The pg_is_other_temp_schema builtin function is now supported, which returns whether the given OID is the OID of another session's temporary schema. Release justification: None, waiting for v22.1.
69909: sql: implement pg_my_temp_schema and pg_is_other_temp_schema r=nvanbenschoten a=nvanbenschoten Needed for #69010. This PR adds a real implementation for the `pg_my_temp_schema` builtin function. `pg_my_temp_schema` returns the OID of session's temporary schema, or 0 if the session has not yet created a temporary schema. The PR then implements the `pg_is_other_temp_schema` builtin function. `pg_is_other_temp_schema` returns true if the given OID is the OID of another session's temporary schema. This can be useful, for example, to exclude other sessions' temporary tables from a catalog display. Release note (sql change): The pg_my_temp_schema builtin function now properly returns the OID of the active session's temporary schema, if one exists. Release note (sql change): The pg_is_other_temp_schema builtin function is now supported, which returns whether the given OID is the OID of another session's temporary schema. Release justification: None, waiting for v22.1. Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Needed for cockroachdb#69010. This commit implements the `information_schema._pg_index_position` builtin function. Given an index's OID and an underlying-table column number, `information_schema._pg_index_position` return the column's position in the index (or NULL if not there). The function is implemented as a user-defined function in Postgres here: https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql Release note (sql change): The `information_schema._pg_index_position` builtin function is now supported, which improves compatibility with PostgreSQL. Release justification: None, waiting for v22.1.
69911: sql: implement information_schema._pg_index_position r=nvanbenschoten a=nvanbenschoten Needed for #69010. This commit implements the `information_schema._pg_index_position` builtin function. Given an index's OID and an underlying-table column number, `information_schema._pg_index_position` return the column's position in the index (or NULL if not there). The function is implemented as a user-defined function in Postgres here: https://github.com/postgres/postgres/blob/master/src/backend/catalog/information_schema.sql Release note (sql change): The `information_schema._pg_index_position` builtin function is now supported, which improves compatibility with PostgreSQL. Release justification: None, waiting for v22.1. Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
Needed for cockroachdb#69010. This commit implements the `pg_has_role` builtin function. `pg_has_role` returns whether the user has privileges for a specified role or not. Allowable privilege types are MEMBER and USAGE. MEMBER denotes direct or indirect membership in the role (that is, the right to do SET ROLE), while USAGE denotes whether the privileges of the role are immediately available without doing SET ROLE. `pg_has_role` was the last remaining unimplemented "access privilege inquiry functions", and was omitted from 94c25be because our role-based access control system was not mature enough to support it at the time. The commit also makes a small modification to `pg_catalog.pg_roles` and `pg_catalog.pg_authid` to reflect that fact that all users and roles inherit the privileges of roles they are members of. Release note (sql change): The pg_has_role builtin function is now supported, which returns whether a given user has privileges for a specified role or not. Release justification: None, waiting for v22.1.
Needed for cockroachdb#69010. This commit implements the `pg_has_role` builtin function. `pg_has_role` returns whether the user has privileges for a specified role or not. Allowable privilege types are MEMBER and USAGE. MEMBER denotes direct or indirect membership in the role (that is, the right to do SET ROLE), while USAGE denotes whether the privileges of the role are immediately available without doing SET ROLE. `pg_has_role` was the last remaining unimplemented "access privilege inquiry functions", and was omitted from 94c25be because our role-based access control system was not mature enough to support it at the time. The commit also makes a small modification to `pg_catalog.pg_roles` and `pg_catalog.pg_authid` to reflect that fact that all users and roles inherit the privileges of roles they are members of. Release note (sql change): The pg_has_role builtin function is now supported, which returns whether a given user has privileges for a specified role or not. Release justification: None, waiting for v22.1.
69939: sql: implement pg_has_role r=nvanbenschoten a=nvanbenschoten Needed for #69010. Related to #22734. This commit implements the `pg_has_role` builtin function. `pg_has_role` returns whether the user has privileges for a specified role or not. Allowable privilege types are MEMBER and USAGE. MEMBER denotes direct or indirect membership in the role (that is, the right to do SET ROLE), while USAGE denotes whether the privileges of the role are immediately available without doing SET ROLE. `pg_has_role` was the last remaining unimplemented "access privilege inquiry functions", and was omitted from 94c25be because our role-based access control system was not mature enough to support it at the time. The commit also makes a small modification to `pg_catalog.pg_roles` and `pg_catalog.pg_authid` to reflect that fact that all users and roles inherit the privileges of roles they are members of. Release note (sql change): The pg_has_role builtin function is now supported, which returns whether a given user has privileges for a specified role or not. Release justification: None, waiting for v22.1. Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
We've landed a few more changes on master since the last progress update here. @jordanlewis has also made some very nice progress on the two issues that were blocking INSERTs/UPDATEs with #70115 and #70100. I've rebased the prototype branch from earlier onto master and cherry-picked those two PRs on top of it. It looks like that was enough to unblock INSERTs and UPDATEs! # INSERT
➜ curl -s 'http://localhost:3000/promo_codes?code=like.%torest' | jq
[]
➜ curl -s -X POST 'http://localhost:3000/promo_codes' \
-H "Content-Type: application/json" \
-d '{"code": "timetorest", "description": "newly created", "rules": {"details": "lol, this is json"}}' | jq
➜ curl -s 'http://localhost:3000/promo_codes?code=like.%torest' | jq
[
{
"code": "timetorest",
"creation_time": null,
"description": "newly created",
"expiration_time": null,
"rules": {
"details": "lol, this is json"
}
}
]
# UPDATE
➜ curl -s -X PATCH 'http://localhost:3000/promo_codes?code=like.%torest' \
-H "Content-Type: application/json" \
-d '{"description": "created and updated"}' | jq
➜ curl -s 'http://localhost:3000/promo_codes?code=like.%torest' | jq
[
{
"code": "timetorest",
"creation_time": null,
"description": "created and updated",
"expiration_time": null,
"rules": {
"details": "lol, this is json"
}
}
] I did find one new issue that blocks Bulk inserts also work. As do both forms of UPSERTs: ➜ curl -s -X POST 'http://localhost:3000/promo_codes' \
-H "Content-Type: application/json" \
-H "Prefer: resolution=merge-duplicates" \
-d '{"code": "timetorest", "description": "created, updated, upserted"}' | jq
➜ curl -s 'http://localhost:3000/promo_codes?code=like.%torest' | jq
[
{
"code": "timetorest",
"creation_time": null,
"description": "created, updated, upserted",
"expiration_time": null,
"rules": {
"details": "lol, this is json"
}
}
]
➜ curl -s -X PUT 'http://localhost:3000/promo_codes?code=eq.timetorest' \
-H "Content-Type: application/json" \
-d '{"code": "timetorest", "description": "created, updated, upserted 2x"}' | jq
➜ curl -s 'http://localhost:3000/promo_codes?code=like.%torest' | jq
[
{
"code": "timetorest",
"creation_time": null,
"description": "created, updated, upserted 2x",
"expiration_time": null,
"rules": {
"details": "lol, this is json"
}
}
] Also, one thing I meant to show before is that JOINs work and they're pretty convenient to use: ➜ curl -s 'http://localhost:3000/rides?limit=1&select=*' | jq
[
{
"city": "amsterdam",
"end_address": "66037 Belinda Plaza Apt. 93",
"end_time": "2018-12-14T08:04:05",
"id": "ab020c49-ba5e-4800-8000-00000000014e",
"revenue": 77,
"rider_id": "c28f5c28-f5c2-4000-8000-000000000026",
"start_address": "1905 Christopher Locks Apt. 77",
"start_time": "2018-12-13T03:04:05",
"vehicle_city": "amsterdam",
"vehicle_id": "aaaaaaaa-aaaa-4800-8000-00000000000a"
}
]
➜ curl -s 'http://localhost:3000/rides?limit=1&select=*,vehicles(*),users(*)' | jq
[
{
"city": "amsterdam",
"end_address": "66037 Belinda Plaza Apt. 93",
"end_time": "2018-12-14T08:04:05",
"id": "ab020c49-ba5e-4800-8000-00000000014e",
"revenue": 77,
"rider_id": "c28f5c28-f5c2-4000-8000-000000000026",
"start_address": "1905 Christopher Locks Apt. 77",
"start_time": "2018-12-13T03:04:05",
"users": {
"address": "14729 Karen Radial",
"city": "amsterdam",
"credit_card": "5844236997",
"id": "c28f5c28-f5c2-4000-8000-000000000026",
"name": "Maria Weber"
},
"vehicle_city": "amsterdam",
"vehicle_id": "aaaaaaaa-aaaa-4800-8000-00000000000a",
"vehicles": {
"city": "amsterdam",
"creation_time": "2019-01-02T03:04:05",
"current_location": "62609 Stephanie Route",
"ext": {
"color": "red"
},
"id": "aaaaaaaa-aaaa-4800-8000-00000000000a",
"owner_id": "c28f5c28-f5c2-4000-8000-000000000026",
"status": "in_use",
"type": "scooter"
}
}
] So we're actually pretty close here. I think the key is that PostgREST isn't trying to perform schema changes or make any system catalog modifications itself. It's actually entirely read-only outside of user-performed mutations, which is a pretty respectable stance to take. I wish more tools embraced that idea. What this means for Cockroach is that it's not pushing quite as hard on advanced features like user-defined types or user-defined functions as other similar tools do (although you can certainly use your own if you want). What PostgREST does push pretty hard on is roles and permissions, because it delegates all authorization to the database. This is pretty critical if you intend to actually expose a service like this on the open web because it means that your database's access control policy is the only thing keeping your data safe. I say this mostly to point out that the recent work by @RichardJCai and @rafiss in this area has made a big impact for PostgREST support, even if we didn't realize it beforehand. I imagine the blocker list would have been 50% longer if we did this investigation 6 months ago. |
So cool 😍 |
70115: sql: add json{,b}_populate_record{,set} r=jordanlewis a=jordanlewis Updates #69010 Updates #70037 These generator builtins permit type-safe transformation of JSON to table data. They're added for compatibility with PostgreSQL. Release note (sql change): add the json_populate_record, jsonb_populate_record functions, json_populate_recordset, and jsonb_populate_recordset functions, which transform JSON into row tuples based on the labels in a record type. 70289: ci: add `bazel Nightlies / SQLite logic tests` job for CI r=rail a=rickystewart Release note: None Co-authored-by: Jordan Lewis <jordanthelewis@gmail.com> Co-authored-by: Ricky Stewart <ricky@cockroachlabs.com>
|
I think this was needed to call stored procedures. See |
D'oh, I probably should have checked that first. Oh well, presumably some other cases will need |
With 22.2 we will have initial support for PostgREST which enables basic CRUD operations on REST APIs. |
Hello! ContextMac M1 Pro - MacOS 14.2.1 postgREST version 12.0.2 cockroach version:
install & run cockroach
Setup rolescreate role web_anon nologin;
grant select on all tables in schema public to web_anon;
create role authenticator login;
grant web_anon to authenticator; Install, configure & run postgRESTbrew install postgrest
echo 'db-uri = "postgres://authenticator@localhost:26257/movr"' >> tutorial.conf
echo 'db-schema = "public"' >> tutorial.conf
echo 'db-anon-role = "web_anon"' >> tutorial.conf
postgrest tutorial.conf GET querycurl -s 'http://localhost:3000/vehicles?status=eq.in_use&select=city,ext&limit=2' This returns the right result, but add an extra POST querycurl -s -X POST 'http://localhost:3000/promo_codes' \
-H "Content-Type: application/json" \
-d '{"code": "timetorest", "description": "newly created", "rules": {"details": "lol, this is json"}}' This returns |
@alexsala You will want to try PostgREST 10.0. Newer versions do not work. |
Hi @dikshant, thanks for letting me know. Will rollback to v10 and see if it all works. |
Add support for PostgREST. PostgREST is a standalone web server that turns your PostgreSQL database directly into a RESTful API. The structural constraints and permissions in the database determine the API endpoints and operations.
@nvanbenschoten did an early investigation and identified the following items needed for support:
Done:
T_unknown
wire decoding: sql/pgwire: support decoding unknown (OID 705) in text/binary format #70027literal -> tuple[]
parsing: sql: support string literal -> tuple[] parsing #70030GRANT CONNECT ON DATABASE <db> TO PUBLIC
.Hard blockers:
None
Soft blockers (workarounds available or only blocking limited functionality):
gz#9915
Jira issue: CRDB-9405
The text was updated successfully, but these errors were encountered: