JSON/REST transcoding

[cwe1ss]

Hi, first, thanks for all the work you put into this library! I'm really looking forward to using it!!

Are there any plans to support the REST-like HTTP/JSON transcoding (https://cloud.google.com/endpoints/docs/grpc/transcoding ) via the http service annotations?

Having that together with OpenAPI/Swagger is a HUGE help during development when you can just call the API via JSON objects.

---

2020/01/14 UPDATE: #167 (comment)

---

2020/04/24 UPDATE:

• Introducing gRPC HTTP API blog post
• Microsoft.AspNetCore.Grpc.HttpApi is added to NuGet.org

---

2020/12/04 UPDATE:

• Create JSON Web APIs from gRPC is on docs.microsoft.com
• Microsoft.AspNetCore.Grpc.Swagger is added to NuGet.org

---

2022/02/16 UPDATE:

Thanks for all the interest in this feature. I'm happy to say that the .NET team is investing in JSON/REST transcoding. The aim is to publish this feature along with .NET 7.

Details:

• .NET 7 or later will be required
• Will continue to be a NuGet package
• I estimate a preview will ship in .NET 7 preview 3 or preview 4.
• A final version will ship with .NET 7 in November.

Development work is tracked at [EPIC] gRPC JSON transcoding.

---

2022/11/11 UPDATE:

.NET 7 has shipped with gRPC JSON transcoding.

Documentation is available at: https://learn.microsoft.com/aspnet/core/grpc/json-transcoding

[JamesNK]

No plans to do it for 3.0. Beyond 3.0, it depends on how many people ask for it.

[JunTaoLuo]

We're keeping this issue open to solicit feedback and see how much demand there exists for this feature.

[jtattermusch]

FTR, regardless of whether this gets implemented or not, there are multiple proxies that currently offer this functionality:
https://github.com/grpc-ecosystem/grpc-gateway
https://www.envoyproxy.io/docs/envoy/latest/configuration/http_filters/grpc_json_transcoder_filter

[govis]

We need HTTP/JSON to gRPC transcoding so that same plumbing/service can be used to support REST and gRPC clients

[raz-canva]

This would be super helpful to be provided out of the box. As opposed to setting up grpc gateway

[Jonathan34]

Also interested. We use grpc gateway in golang but need the same functionality in dotnet as we are a big .net consumer.

[bobrot]

Also interested in this. While there are 3rd party tools that can do this, a simpler implementation is always better.

[Jonathan34]

if i had to start again, I'd probably use an envoy proxy instead similar to that: https://blog.envoyproxy.io/envoy-and-grpc-web-a-fresh-new-alternative-to-rest-6504ce7eb880
so at least I can have the same solution and transcoding for all the languages

[VladimirLuzhin]

That would be very cool since would allow smooth transition from JSON API to GRPC

[JamesNK]

We have an experimental project that allows JSON/REST with gRPC here: https://github.com/aspnet/AspLabs/tree/master/src/GrpcHttpApi

The README.md at the link has details about how to use it and its status as a project.

[ElderJames]

Looking forward to this feature.

[dashkan]

Great job!
I think this is a must have for dotnet core framework and tooling. Why proxy gRPC calls when you don't have to. Much more elegant IMO.

[TomDeVree]

This is very helpful. Should it be possible to use Swagger (for example Swashbuckle) with this setup? For now it seems it cannot find any operations.

[JamesNK]

There is currently no way to generate Swagger. I think a Swashbuckle extension would need to be created that lets Swashbuckle understand gRPC endpoints and protobuf schemas.

[MedAnd]

Thoughts on support for Azure Functions?

cc @jeffhollan

[Otto404]

impressive. I hope to find these functionalities in production soon

[kollerdroid]

Great project. I think GrpcHttpApi should be part of dotnet 5.
Waiting for AuthContext implementation... ;-)

[Dchoksi-brierley]

The gRPC HTTP API is a great solution - almost every .net application that is adding on gRPC will benefit in maintaining existing external REST interfaces via this solution.
We tried it within our team and it works seamlessly on .net core 3.0. However our application is being implemented using .net core 3.1 - any plans on moving the gRPC HTTP API forward?

[JamesNK]

Does it not work on 3.1?

[alejandrobonillamondragon]

Hi there,
I followed the instructions from
https://www.nuget.org/packages/Microsoft.AspNetCore.Grpc.HttpApi... Installed the pre-release version of Microsoft.AspNetCore.Grpc.HttpApi and Microsoft.AspNetCore.Grpc.Swagger (0.1.0-alpha.20254.1) from NuGet official library. However, when opening swagger UI it says "No operations defined in spec!" and the endpoints are not reachable through HTTP (getting 404)... I'm using .NET Core 3.1, I think a similar issue was already reported here: #745. Do you know if Grpc.Swagger works for 3.1?

This is the Startup.cs

using GrpcGateway.Server;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.OpenApi.Models;

namespace GrpcGateway
{
public class Startup
{
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}

    public IConfiguration Configuration { get; }

    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddGrpcHttpApi();

        services.AddSwaggerGen(c =>
        {
            c.SwaggerDoc("v1", new OpenApiInfo { Title = "gRPC HTTP API Example", Version = "v1" });
        });
        services.AddGrpcSwagger();

        //services.AddControllers();
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseHttpsRedirection();
        app.UseDefaultFiles();
        app.UseStaticFiles();

        app.UseSwagger();
        app.UseSwaggerUI(c =>
        {
            c.SwaggerEndpoint("/swagger/v1/swagger.json", "gRPC HTTP API Example V1");
        });

        app.UseRouting();

        //app.UseAuthorization();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapGrpcService<GreeterService>();
        });
    }
}

}

[Dchoksi-brierley]

Thanks for the details Alejandro.. James - please let us know if you have any info for us or need any details about the issue. Thanks so much for responding so quickly - much appreciated!

[JamesNK]

It should work in 3.0. I haven't got time to debug this, but feel free to copy the source code for those packages to your machine and step through the code to find out why you get that message.

[Dchoksi-brierley]

Thanks again for responding back.
We'll debug into the source code - fingers crossed. We were not sure if there were known issues with 3.1 ("Beyond 3.0, it depends on how many people ask for it")... In any case, we'll share what we find - in case it helps any one else.

[JamesNK]

Works fine in 3.1 - https://github.com/aspnet/AspLabs/tree/master/src/GrpcHttpApi/sample

[Dchoksi-brierley]

Many thanks!!
We were able to make progress - adding a reference to Grpc.AspNetCore.Server and Grpc.Tools made the difference. Those two packages were not required to generate the server stubs in the latest version of Grpc.AspNetCore but looks like they were needed for Microsoft.AspNetCore.Grpc.HttpApi.

[alejandrobonillamondragon]

It works in 3.1. In case this helps someone else. In my case the issue was in the proto files, the Proto folder needs to be placed in the solution's root folder and create a service reference in the project using the main .proto file (the one who imports annotations.proto) marking it to generate the server class. I was placing those files in the project folder and generating the server stub directly from it (without the service reference), in that way it doesn't work.

[ziaulhasanhamim]

Anything on response trailers?

[JamesNK]

An HTTP/1.x request was sent to an HTTP/2 only endpoint.

I had a similar thing OOB @vCillusion - I created a new gRPC service - but it's Kestrel options that's biting you. The dotnet new template has Kestrel options added to appsettings.json which disables HTTP/1 by default. Maybe you've got a similar thing.

I updated the docs to talk about protocol - https://github.com/dotnet/AspNetCore.Docs/blob/8d08eeb1440523d746b635d6bd308264a7151e07/aspnetcore/grpc/httpapi.md#http-protocol

Anything on response trailers?

Response trailers don't have an equivalent in REST. They're discarded.

[rars]

@JamesNK is there any plan to provide the annotations.proto and http.proto in a supporting NuGet package or similar to avoid users having to manually copy these proto files into their projects to set up?

[JamesNK]

I looked into it a couple of months ago and can't figure out a way to make it work using out-of-the-box NuGet features. The package would require some custom msbuild scripts to copy the files to the correct location.

If someone gets it working, then I'd consider accepting a contribution of a NuGet package that automates this.

[rars]

@JamesNK have you considered something like the example here: https://github.com/rars/JsonTranscodingDemo
Using GeneratePathProperty="true" on the PackageReference and then AdditionalImportDirs="$(PkgJsonTranscoding_Protos)\content" on the relevant Protobuf elements that need to import google/api/annotations.proto.

[JamesNK]

Update:

.NET 7 has shipped with gRPC JSON transcoding.

Documentation is available at: https://learn.microsoft.com/aspnet/core/grpc/json-transcoding

[JamesNK]

@JamesNK have you considered something like the example here: rars/JsonTranscodingDemo Using GeneratePathProperty="true" on the PackageReference and then AdditionalImportDirs="$(PkgJsonTranscoding_Protos)\content" on the relevant Protobuf elements that need to import google/api/annotations.proto.

@rars I think I found a way to make this idea work without the developer having to do anything by including some msbuild with the package. No need to add GeneratePathProperty="true" or AdditionalImportDirs="$(PkgJsonTranscoding_Protos)\content" to csproj.

See dotnet/aspnetcore#44999

[alexhiggins732]

Update:

.NET 7 has shipped with gRPC JSON transcoding.

Documentation is available at: https://learn.microsoft.com/aspnet/core/grpc/json-transcoding

@JamesNK Great to see this made it into .NET 7 which will open GRPC to clients that currently can't support it. Given that only server-side streaming is currently supported what do you think about adding support for client-side streaming or bidirectional streaming using websockets or SignalR. Since the middleware already supports routing JSON requests to GRPC I think it might be a logical next step to expose this additional functionality to these clients. Thoughts?

[JamesNK]

The purpose of gRPC JSON transcoding is to support RESTful APIs over any HTTP version from gRPC services. RESTful APIs don't have client or bidirectional streaming endpoints because the JavaScript fetch API doesn't support those kinds of requests.

I doubt that something like that will be added.

[alexhiggins732]

Obviously, tradition JSON Rest APIs don't support bidirectional. My question pertained to clients that support web sockets. The server could act as a proxy. The server would have web socket connection to the client and a GRPC streaming connection to the GRPC service. The Server would then tunnel web socket messages from the client mapping them to GRPC endpoints and in reverse take GRPC messages from the GRPC service and tunnel them back to the client in JSON over the web socket.

[saquibmian]

Personally I find that client-streaming is almost never a good idea and that HTTP/2 is more than sufficient for the use cases where I would typically reach for client-streaming. HTTP/2 connection re-use is efficient and request pipelining further alleviates client-side pressure. Client-side streams in practice usually are finicky. They are not reliable over LTE/5G, necessitating complex stream-resumption logic. Even in a reliable connection, there are many failure points from client to server that can easily interrupt the stream. Aside from this, there are other application-level failures that can occur, such as an auth token expiring mid-stream.
(aside: I feel the same with server-streaming, with the caveat that short-lived streams are acceptable).

There is no cross-platform browser API for client streaming, until fetch-streams lands, but even then my arguments above would deter me from client-streaming as an architectural decision. Web Sockets don't encode a protocol, and anything JSON Transcoding does to support Web Sockets would be non-platform neutral (e.g., SignalR has bindings for ASP.NET Core and the browser, but that's it; it's also a totally different abstraction from HTTP, being primarily an RPC protocol).
Finally, the main advantage behind JSON Transcoding is that the web APIs do not betray their gRPC-implementation; a protocol that just shuffles gRPC messages across the wire is inherently problematic. For one, gRPC calls rely on HTTP Trailers for status codes; this is problematic for encoding client-streaming calls over HTTP/1 or Web Sockets, as we would effectively need to build an application-level protocol to support this.

[catkins-dmg]

@JamesNK The update to .NET 7 and gRPC JSON transcoding was smooth and I'm happy with the fixes that have come along with it. Although, it is sad to see that query params still do not fully work. They now work to the extent that they show up in the Swagger UI and can be filled in on the test request. However, when that request is actually executed the query parameters do not make it to the API implementation, they are always empty.

It is disingenuous that query parameters are presented as an option on the documentation page when they don't work. https://learn.microsoft.com/en-us/aspnet/core/grpc/json-transcoding-binding?view=aspnetcore-7.0#http-rules

Example:
rpc QueryParamTest (QueryParamTestRequest) returns (QueryParamTestResponse) { option (google.api.http) = { get: "/test/{required_param}", }; }
message QueryParamTestRequest { string required_param = 1; string query_param = 2; } message QueryParamTestResponse{ string message = 1; }

[yogingang]

When I run it in aspnet 7, I always get the following error.
It seems that the versions of the modules do not match.
It's not something I can configure.

[agomezb]

Hi Guys, there is some way to customize the JSON format message produced by an exception? When using Json Transcoding.

[gkinsman]

Hi there! I hope this is the right place for this question, but happy to create another issue if necessary.

I have a reproduction of the issue I'm hitting here: https://github.com/gkinsman/GrpcTranscodingRepro.git

We have quite a few proto files, so instead of adding them one by one we use the wildcard syntax: Include="**/*.proto".

When combining this with the annotated/http files though, we get a stack of duplicate type definitions:

Error CS0433 : The type 'AnnotationsReflection' exists in both 'Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' and 'Google.Api.CommonProtos, Version=2.5.0.0, Culture=neutral, PublicKeyToken=3ec5ea7f18953e47'
1>------- Finished building project: Api. Succeeded: False. Errors: 1. Warnings: 0

I think this is because the google protobuf files are being compiled along with our own. Including the proto's explicitly works, so the repro works if using this method instead.

I suspect there's some combination of Protobuf msbuild tasks to get this to work, but I can't work it out having spent a few hours trying things and inspecting the docs.

Any ideas would be greatly appreciated!

[runenilsenoe]

Hi there! I hope this is the right place for this question, but happy to create another issue if necessary.

I have a reproduction of the issue I'm hitting here: https://github.com/gkinsman/GrpcTranscodingRepro.git

We have quite a few proto files, so instead of adding them one by one we use the wildcard syntax: Include="**/*.proto".

When combining this with the annotated/http files though, we get a stack of duplicate type definitions:

Error CS0433 : The type 'AnnotationsReflection' exists in both 'Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' and 'Google.Api.CommonProtos, Version=2.5.0.0, Culture=neutral, PublicKeyToken=3ec5ea7f18953e47'
1>------- Finished building project: Api. Succeeded: False. Errors: 1. Warnings: 0

I think this is because the google protobuf files are being compiled along with our own. Including the proto's explicitly works, so the repro works if using this method instead.

I suspect there's some combination of Protobuf msbuild tasks to get this to work, but I can't work it out having spent a few hours trying things and inspecting the docs.

Any ideas would be greatly appreciated!

Hey @gkinsman , there might be a better solution with the new transcoding packages in c#7 but our solution to this was to use reference a shared contract, to avoid multiple files with the same content within the same proto space.

https://github.com/runenilsenoe/GrpcTranscodingRepro

here's a working example

[gkinsman]

Hi there! I hope this is the right place for this question, but happy to create another issue if necessary.
I have a reproduction of the issue I'm hitting here: https://github.com/gkinsman/GrpcTranscodingRepro.git
We have quite a few proto files, so instead of adding them one by one we use the wildcard syntax: Include="**/*.proto".
When combining this with the annotated/http files though, we get a stack of duplicate type definitions:

Error CS0433 : The type 'AnnotationsReflection' exists in both 'Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' and 'Google.Api.CommonProtos, Version=2.5.0.0, Culture=neutral, PublicKeyToken=3ec5ea7f18953e47'
1>------- Finished building project: Api. Succeeded: False. Errors: 1. Warnings: 0

I think this is because the google protobuf files are being compiled along with our own. Including the proto's explicitly works, so the repro works if using this method instead.
I suspect there's some combination of Protobuf msbuild tasks to get this to work, but I can't work it out having spent a few hours trying things and inspecting the docs.
Any ideas would be greatly appreciated!

Hey @gkinsman , there might be a better solution with the new transcoding packages in c#7 but our solution to this was to use reference a shared contract, to avoid multiple files with the same content within the same proto space.

https://github.com/runenilsenoe/GrpcTranscodingRepro

here's a working example

Edit:
Managed to get it working thanks to your changes @runenilsenoe!

It looks like the AdditionalImportDirs property works to import the additional protos but doesn't compile them - it only imports them for protoc. So I added a ProtoHack project with just the google protos in the root under google/api/ and imported that into our contracts and it all works. I look forward to when this isn't necessary with .net 8!

Thanks again!

---

Thanks for your response! I made a mistake in my repro and had copied the protos in both Api and Contracts, but we only want them in the Contracts lib which I think is what you suggested. I've updated the repro now to reflect that.

I'm not quite sure how your sample works with the files folder though - where is that in the file system?

It now builds with warnings but I get this exception when running:

Unhandled exception. System.InvalidOperationException: Error binding gRPC service 'GreeterService'.
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.InvalidOperationException: The stored extension value has a type of 'Google.Api.HttpRule, Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. This a different from the requested type of 'Google.Api.HttpRule, Google.Api.CommonProtos, Version=2.5.0.0, Culture=neutral, PublicKeyToken=3ec5ea7f18953e47'.

Thanks!

[runenilsenoe]

@gkinsman

Good question, not sure why it did not get included in the commit. But when the proto root is set there is no need to reference the Google file for code gen. you just need them present for The other files to ref. so i only care about the files folder which contains my other shared proto file

I added them to the repro now

[JamesNK]

The stored extension value has a type of 'Google.Api.HttpRule, Contracts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. This a different from the requested type of 'Google.Api.HttpRule, Google.Api.CommonProtos, Version=2.5.0.0, Culture=neutral, PublicKeyToken=3ec5ea7f18953e47'.

This exception indicates you're compiling your own version of HttpRule. This happens because you include http.proto and annotations.proto with <Proto /> to Grpc.Tools.

You need those files in your project so other protos can reference them, but they shouldn't generate code. Instead, you should reference the Google.Api.CommonProtos so HttpRule from that assembly is used.

I recognize this feels like an annoying detail. You need to do this for now, but a PR has just merged to make it work from .NET 8 or later: dotnet/aspnetcore#48194

[gkinsman]

Thanks for the response @JamesNK. I've found that having these protos in the same project as our protos causes them to be compiled for some reason, even though they're outside the proto root:

<Protobuf ProtoRoot="protos" Include="**/*.proto" GrpcServices="Server" AdditionalImportDirs="..\protohack" />

With folders /protos and /protohack in the same project.

Moving them to a folder upstairs next to the other projects keeps them out of the compilation it seems and does the trick.

[probablyhank]

I updated the docs to talk about protocol - https://github.com/dotnet/AspNetCore.Docs/blob/8d08eeb1440523d746b635d6bd308264a7151e07/aspnetcore/grpc/httpapi.md#http-protocol

@JamesNK FYI this breaks server reflection in Postman. Any settings I could try adjusting to get that working again?

[sgsm74]

An HTTP/1.x request was sent to an HTTP/2 only endpoint.

Do you find any solution for this?

[JamesNK]

I updated the docs to talk about protocol - dotnet/AspNetCore.Docs@8d08eeb/aspnetcore/grpc/httpapi.md#http-protocol

@JamesNK FYI this breaks server reflection in Postman. Any settings I could try adjusting to get that working again?

Did you set the port to support HTTP/1.1 and HTTP/2 without TLS? The docs say that TLS is required if you do this.

[JamesNK]

I'm closing this issue. The feature is done. And I don't want people to throw all there questions or bugs in one issue.