Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No service server interface created #38

Closed
EvanDarwin opened this issue Feb 14, 2019 · 21 comments
Closed

No service server interface created #38

EvanDarwin opened this issue Feb 14, 2019 · 21 comments

Comments

@EvanDarwin
Copy link

I'm running the following commands, but my outputted .d.ts files have no server interface?

$ grpc_tools_node_protoc --grpc_out=./src/proto --plugin="protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin" -I ./proto --js_out="import_style=commonjs,binary:./src/proto/" ./proto/*.proto
$ protoc --plugin="protoc-gen-ts=./node_modules/.bin/protoc-gen-ts" -I ./proto --ts_out="./src/proto/" ./proto/*.proto

Currently running:

protoc - 3.6.1
grpc - 1.18.0
grpc-tools - 1.6.6
grpc_tools_node_protoc_ts - 2.5.0
ts-protoc-gen - 0.9.0

protobuf definition:

syntax = "proto3";

///// SERVICES
service AuditService {
    rpc CreateAuditSource (CreateAuditSourceRequest) returns (CreateAuditSourceResponse) {}
}

///// TYPES
message TerminalAuditData {
    uint32 employee_id = 1;
    uint32 location_id = 2;
    string fingerprint = 3;
    uint32 ip = 4;
    string user_agent = 5;
}

///// RESPONSES
message CreateAuditSourceResponse {
    bool success = 1;
    uint32 id = 2;
}

///// REQUESTS
message CreateAuditSourceRequest {
    uint32 employee_id = 1;
    uint32 location_id = 2;
    uint32 auth_token_id = 3;
    TerminalAuditData terminal = 4;
}

output TS file:

// file: audit.proto

import * as jspb from "google-protobuf";

export class TerminalAuditData extends jspb.Message {
  getEmployeeId(): number;
  setEmployeeId(value: number): void;

  getLocationId(): number;
  setLocationId(value: number): void;

  getFingerprint(): string;
  setFingerprint(value: string): void;

  getIp(): number;
  setIp(value: number): void;

  getUserAgent(): string;
  setUserAgent(value: string): void;

  serializeBinary(): Uint8Array;
  toObject(includeInstance?: boolean): TerminalAuditData.AsObject;
  static toObject(includeInstance: boolean, msg: TerminalAuditData): TerminalAuditData.AsObject;
  static extensions: {[key: number]: jspb.ExtensionFieldInfo<jspb.Message>};
  static extensionsBinary: {[key: number]: jspb.ExtensionFieldBinaryInfo<jspb.Message>};
  static serializeBinaryToWriter(message: TerminalAuditData, writer: jspb.BinaryWriter): void;
  static deserializeBinary(bytes: Uint8Array): TerminalAuditData;
  static deserializeBinaryFromReader(message: TerminalAuditData, reader: jspb.BinaryReader): TerminalAuditData;
}

export namespace TerminalAuditData {
  export type AsObject = {
    employeeId: number,
    locationId: number,
    fingerprint: string,
    ip: number,
    userAgent: string,
  }
}

export class CreateAuditSourceResponse extends jspb.Message {
  getSuccess(): boolean;
  setSuccess(value: boolean): void;

  getId(): number;
  setId(value: number): void;

  serializeBinary(): Uint8Array;
  toObject(includeInstance?: boolean): CreateAuditSourceResponse.AsObject;
  static toObject(includeInstance: boolean, msg: CreateAuditSourceResponse): CreateAuditSourceResponse.AsObject;
  static extensions: {[key: number]: jspb.ExtensionFieldInfo<jspb.Message>};
  static extensionsBinary: {[key: number]: jspb.ExtensionFieldBinaryInfo<jspb.Message>};
  static serializeBinaryToWriter(message: CreateAuditSourceResponse, writer: jspb.BinaryWriter): void;
  static deserializeBinary(bytes: Uint8Array): CreateAuditSourceResponse;
  static deserializeBinaryFromReader(message: CreateAuditSourceResponse, reader: jspb.BinaryReader): CreateAuditSourceResponse;
}

export namespace CreateAuditSourceResponse {
  export type AsObject = {
    success: boolean,
    id: number,
  }
}

export class CreateAuditSourceRequest extends jspb.Message {
  getEmployeeId(): number;
  setEmployeeId(value: number): void;

  getLocationId(): number;
  setLocationId(value: number): void;

  getAuthTokenId(): number;
  setAuthTokenId(value: number): void;

  hasTerminal(): boolean;
  clearTerminal(): void;
  getTerminal(): TerminalAuditData | undefined;
  setTerminal(value?: TerminalAuditData): void;

  serializeBinary(): Uint8Array;
  toObject(includeInstance?: boolean): CreateAuditSourceRequest.AsObject;
  static toObject(includeInstance: boolean, msg: CreateAuditSourceRequest): CreateAuditSourceRequest.AsObject;
  static extensions: {[key: number]: jspb.ExtensionFieldInfo<jspb.Message>};
  static extensionsBinary: {[key: number]: jspb.ExtensionFieldBinaryInfo<jspb.Message>};
  static serializeBinaryToWriter(message: CreateAuditSourceRequest, writer: jspb.BinaryWriter): void;
  static deserializeBinary(bytes: Uint8Array): CreateAuditSourceRequest;
  static deserializeBinaryFromReader(message: CreateAuditSourceRequest, reader: jspb.BinaryReader): CreateAuditSourceRequest;
}

export namespace CreateAuditSourceRequest {
  export type AsObject = {
    employeeId: number,
    locationId: number,
    authTokenId: number,
    terminal?: TerminalAuditData.AsObject,
  }
}
@agreatfool
Copy link
Owner

I guess the server interface you asked shall be in file audit_grpc_pb.d.ts:

// package: 
// file: audit.proto

/* tslint:disable */

import * as grpc from "grpc";
import * as audit_pb from "./audit_pb";

interface IAuditServiceService extends grpc.ServiceDefinition<grpc.UntypedServiceImplementation> {
    createAuditSource: IAuditServiceService_ICreateAuditSource;
}

interface IAuditServiceService_ICreateAuditSource extends grpc.MethodDefinition<audit_pb.CreateAuditSourceRequest, audit_pb.CreateAuditSourceResponse> {
    path: string; // "/.AuditService/CreateAuditSource"
    requestStream: boolean; // false
    responseStream: boolean; // false
    requestSerialize: grpc.serialize<audit_pb.CreateAuditSourceRequest>;
    requestDeserialize: grpc.deserialize<audit_pb.CreateAuditSourceRequest>;
    responseSerialize: grpc.serialize<audit_pb.CreateAuditSourceResponse>;
    responseDeserialize: grpc.deserialize<audit_pb.CreateAuditSourceResponse>;
}

export const AuditServiceService: IAuditServiceService;

export interface IAuditServiceServer {
    createAuditSource: grpc.handleUnaryCall<audit_pb.CreateAuditSourceRequest, audit_pb.CreateAuditSourceResponse>;
}

export interface IAuditServiceClient {
    createAuditSource(request: audit_pb.CreateAuditSourceRequest, callback: (error: grpc.ServiceError | null, response: audit_pb.CreateAuditSourceResponse) => void): grpc.ClientUnaryCall;
    createAuditSource(request: audit_pb.CreateAuditSourceRequest, metadata: grpc.Metadata, callback: (error: grpc.ServiceError | null, response: audit_pb.CreateAuditSourceResponse) => void): grpc.ClientUnaryCall;
    createAuditSource(request: audit_pb.CreateAuditSourceRequest, metadata: grpc.Metadata, options: Partial<grpc.CallOptions>, callback: (error: grpc.ServiceError | null, response: audit_pb.CreateAuditSourceResponse) => void): grpc.ClientUnaryCall;
}

export class AuditServiceClient extends grpc.Client implements IAuditServiceClient {
    constructor(address: string, credentials: grpc.ChannelCredentials, options?: object);
    public createAuditSource(request: audit_pb.CreateAuditSourceRequest, callback: (error: grpc.ServiceError | null, response: audit_pb.CreateAuditSourceResponse) => void): grpc.ClientUnaryCall;
    public createAuditSource(request: audit_pb.CreateAuditSourceRequest, metadata: grpc.Metadata, callback: (error: grpc.ServiceError | null, response: audit_pb.CreateAuditSourceResponse) => void): grpc.ClientUnaryCall;
    public createAuditSource(request: audit_pb.CreateAuditSourceRequest, metadata: grpc.Metadata, options: Partial<grpc.CallOptions>, callback: (error: grpc.ServiceError | null, response: audit_pb.CreateAuditSourceResponse) => void): grpc.ClientUnaryCall;
}

@EvanDarwin
Copy link
Author

@agreatfool It only seems to be generating the *_grpc_pb.js files and not the .d.ts ones. Anything I can do to debug/fix?

@agreatfool
Copy link
Owner

agreatfool commented Feb 14, 2019

I saw you use protoc command in ts generating.

Try grpc_tools_node_protoc instead, just like the sample in the doc:

grpc_tools_node_protoc \
--plugin=protoc-gen-ts=../bin/protoc-gen-ts \
--ts_out=${PROTO_DEST} \
-I ./proto \
proto/*.proto

@EvanDarwin
Copy link
Author

@agreatfool This is the command I'm using:

./node_modules/.bin/grpc_tools_node_protoc \
    --plugin="protoc-gen-ts=./node_modules/.bin/protoc-gen-ts" \
    -I $PROTO_SRC_DIR --ts_out=$PROTO_OUT_DIR \
    $PROTO_FILES

It's not generating any *_grpc_pb files at all for me now, or JS files. Should I be running both commands or just a single one?

@agreatfool
Copy link
Owner

Sample bash command you given:

$ grpc_tools_node_protoc --grpc_out=./src/proto --plugin="protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin" -I ./proto --js_out="import_style=commonjs,binary:./src/proto/" ./proto/*.proto
$ protoc --plugin="protoc-gen-ts=./node_modules/.bin/protoc-gen-ts" -I ./proto --ts_out="./src/proto/" ./proto/*.proto

=> try this:

$ grpc_tools_node_protoc --grpc_out=./src/proto --plugin="protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin" -I ./proto --js_out="import_style=commonjs,binary:./src/proto/" ./proto/*.proto
$ grpc_tools_node_protoc --plugin="protoc-gen-ts=./node_modules/.bin/protoc-gen-ts" -I ./proto --ts_out="./src/proto/" ./proto/*.proto

protoc => grpc_tools_node_protoc

@EvanDarwin
Copy link
Author

./node_modules/.bin/grpc_tools_node_protoc --grpc_out=$PROTO_OUT_DIR \
    --plugin="protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin" \
    -I $PROTO_SRC_DIR --js_out="import_style=commonjs,binary:${PROTO_OUT_DIR}" \
    $PROTO_FILES
./node_modules/.bin/grpc_tools_node_protoc \
    --plugin="protoc-gen-ts=./node_modules/.bin/protoc-gen-ts" \
    -I $PROTO_SRC_DIR --ts_out=${PROTO_OUT_DIR} \
    $PROTO_FILES

I believe these commands are the same. Now I am getting the _grpc_pb.js files generated, but still no .d.ts.

@EvanDarwin
Copy link
Author

It looks the files are generating now that I'm using grpc@1.9.0. I think I just missed it in the README. Thanks @agreatfool

@agreatfool
Copy link
Owner

Cool~

@EvanDarwin
Copy link
Author

EvanDarwin commented Feb 15, 2019

Very odd... now it's not working again. I'm also having some trouble compiling grpc 1.9.x, I keep getting weird file not found errors and other things.

$ uname -a
Linux protobuf-dev 4.15.0-1026-gcp #27-Ubuntu SMP Thu Dec 6 18:27:01 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
Node v11.10.0
npm v6.7.0
Yarn 1.13.0

It works in one project, but not a project that's essentially a clone. I tried installing my dependencies with a clean node_modules in both npm and yarn, neither worked properly. This is what the package.json file looks like for the project that's not generating the grpc_pb.d.ts files:

{
  "name": "[redacted]",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "compile": "./bin/compile.sh"
  },
  "dependencies": {
    "typescript": "^3.3.3"
  },
  "devDependencies": {
    "@types/google-protobuf": "^3.2.7",
    "@types/node": "^11.9.4",
    "google-protobuf": "^3.6.0",
    "grpc": "^1.9.0",
    "grpc-tools": "^1.6.6",
    "grpc_tools_node_protoc_ts": "^2.5.0",
    "nodemon": "^1.18.10",
    "protobufjs": "^6.8.8",
    "ts-node": "^8.0.2",
    "ts-protoc-gen": "^0.9.0"
  }
}

@EvanDarwin EvanDarwin reopened this Feb 15, 2019
@agreatfool
Copy link
Owner

That's a bit hard to help.

I think you could clone grpc_tools_node_protoc_ts, use the env versions I provided in README doc. And have a try. If it's all OK. Then switch tool version as you want one by one.

Figure out issues by isolation.

@EvanDarwin
Copy link
Author

EvanDarwin commented Feb 20, 2019

I've been trying to isolate the issue by using the examples/ directory in this project repo, since it has a known working package-lock.json file. I'm using Docker to build an image that runs a simple script to install and build the example.

Here is my Dockerfile:

FROM node:11
ADD run.sh /
RUN chmod +x /run.sh
ENTRYPOINT ["/run.sh"]

And the run.sh script:

#!/bin/bash

git clone https://github.com/agreatfool/grpc_tools_node_protoc_ts.git /tmp/grpc
cd /tmp/grpc/examples
npm i -g node-pre-gyp && npm i -g grpc-tools@1.6.6 && npm i && npm run build

I wasn't able to install the dependencies properly on any version of Node 8-11. Could this be because of the older version of grpc-tools that's being used?

This is the error output consistent for all versions:

# docker build . -t protoc ; docker run --rm -it protoc
Sending build context to Docker daemon  4.096kB
Step 1/4 : FROM node:8
 ---> 4f01e5319662
Step 2/4 : ADD run.sh /
 ---> 866c46415f69
Step 3/4 : RUN chmod +x /run.sh
 ---> Running in d49988583bf6
Removing intermediate container d49988583bf6
 ---> 780f9726ee31
Step 4/4 : ENTRYPOINT ["/run.sh"]
 ---> Running in ec30194890a9
Removing intermediate container ec30194890a9
 ---> df471edd5375
Successfully built df471edd5375
Successfully tagged protoc:latest
Cloning into '/tmp/grpc'...
remote: Enumerating objects: 58, done.
remote: Counting objects: 100% (58/58), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 898 (delta 27), reused 41 (delta 16), pack-reused 840
Receiving objects: 100% (898/898), 264.44 KiB | 0 bytes/s, done.
Resolving deltas: 100% (536/536), done.
/usr/local/bin/node-pre-gyp -> /usr/local/lib/node_modules/node-pre-gyp/bin/node-pre-gyp
+ node-pre-gyp@0.12.0
added 66 packages from 25 contributors in 3.748s
/usr/local/bin/grpc_tools_node_protoc -> /usr/local/lib/node_modules/grpc-tools/bin/protoc.js
/usr/local/bin/grpc_tools_node_protoc_plugin -> /usr/local/lib/node_modules/grpc-tools/bin/protoc_plugin.js

> grpc-tools@1.6.6 install /usr/local/lib/node_modules/grpc-tools
> node-pre-gyp install

node-pre-gyp ERR! install error 
node-pre-gyp ERR! stack Error: EPERM: operation not permitted, utime '/usr/local/lib/node_modules/grpc-tools/bin'
node-pre-gyp ERR! System Linux 4.15.0-1027-gcp
node-pre-gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/grpc-tools/node_modules/.bin/node-pre-gyp" "install"
node-pre-gyp ERR! cwd /usr/local/lib/node_modules/grpc-tools
node-pre-gyp ERR! node -v v8.15.0
node-pre-gyp ERR! node-pre-gyp -v v0.6.38
node-pre-gyp ERR! not ok 
EPERM: operation not permitted, utime '/usr/local/lib/node_modules/grpc-tools/bin'
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! grpc-tools@1.6.6 install: `node-pre-gyp install`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the grpc-tools@1.6.6 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-02-20T04_00_34_292Z-debug.log

@agreatfool
Copy link
Owner

agreatfool commented Feb 20, 2019

Several issues:

npm permission
If npm run in docker, option --unsafe-perm has to be appended, since docker run all commands in root, but npm not satisfied.

Read more: grpc/grpc-node#604

grpc missing
Command protoc is required and not installed.

typescript missing
Command tsc is required and not installed.


try this:

FROM node:8.4.0

RUN apt-get update && \
    apt-get -y install git unzip build-essential autoconf libtool

RUN git clone https://github.com/google/protobuf.git && \
    cd protobuf && \
    ./autogen.sh && \
    ./configure && \
    make && \
    make install && \
    ldconfig && \
    make clean && \
    cd .. && \
    rm -r protobuf

RUN git clone https://github.com/agreatfool/grpc_tools_node_protoc_ts.git grpc_protoc && \
    cd grpc_protoc && \
    npm i -g grpc-tools@1.6.6 --unsafe-perm && \
    npm i -g typescript --unsafe-perm && \
    npm i --unsafe-perm && \
    ./bash/build.sh

@EvanDarwin
Copy link
Author

EvanDarwin commented Feb 22, 2019

Dockerfile:

FROM node:8.4.0

RUN apt-get update && \
    apt-get -y install git unzip build-essential autoconf libtool

RUN git clone https://github.com/google/protobuf.git && \
    cd protobuf && \
    ./autogen.sh && \
    ./configure && \
    make -j8 && \
    make install && \
    ldconfig && \
    make clean && \
    cd .. && \
    rm -r protobuf

RUN git clone https://github.com/agreatfool/grpc_tools_node_protoc_ts.git grpc_protoc && \
    cd grpc_protoc/examples && \
    npm i -g  grpc-tools@1.6.6 typescript@3.3 --unsafe-perm && \
    npm i --unsafe-perm ts-protoc-gen && \
    npm i --unsafe-perm && \
    ./bash/build.sh

Error Output

module.js:491
    throw err;
    ^

Error: Cannot find module 'google-protobuf/google/protobuf/compiler/plugin_pb'
    at Function.Module._resolveFilename (module.js:489:15)
    at Function.Module._load (module.js:439:25)
    at Module.require (module.js:517:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/grpc_protoc/build/index.js:11:21)
    at Module._compile (module.js:573:30)
    at Object.Module._extensions..js (module.js:584:10)
    at Module.load (module.js:507:32)
    at tryModuleLoad (module.js:470:12)
    at Function.Module._load (module.js:462:3)
--ts_out: protoc-gen-ts: Plugin failed with status code 1.
/usr/local/lib/node_modules/grpc-tools/bin/protoc.js:41
    throw error;
    ^

Error: Command failed: /usr/local/lib/node_modules/grpc-tools/bin/protoc --plugin=protoc-gen-grpc=/usr/local/lib/node_modules/grpc-tools/bin/grpc_node_plugin --plugin=protoc-gen-ts=../bin/protoc-gen-ts --ts_out=./src/proto -I ./proto proto/book.proto
module.js:491
    throw err;
    ^

Error: Cannot find module 'google-protobuf/google/protobuf/compiler/plugin_pb'
    at Function.Module._resolveFilename (module.js:489:15)
    at Function.Module._load (module.js:439:25)
    at Module.require (module.js:517:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/grpc_protoc/build/index.js:11:21)
    at Module._compile (module.js:573:30)
    at Object.Module._extensions..js (module.js:584:10)
    at Module.load (module.js:507:32)
    at tryModuleLoad (module.js:470:12)
    at Function.Module._load (module.js:462:3)
--ts_out: protoc-gen-ts: Plugin failed with status code 1.

    at ChildProcess.exithandler (child_process.js:270:12)
    at emitTwo (events.js:125:13)
    at ChildProcess.emit (events.js:213:7)
    at maybeClose (internal/child_process.js:927:16)
    at Socket.stream.socket.on (internal/child_process.js:348:11)
    at emitOne (events.js:115:13)
    at Socket.emit (events.js:210:7)
    at Pipe._handle.close [as _onclose] (net.js:545:12)

@agreatfool
Copy link
Owner

I did some research and found the reason.

I just separated the building process into three parts:

First

FROM node:8.4.0

RUN apt-get update && \
    apt-get -y install git unzip build-essential autoconf libtool

RUN git clone https://github.com/google/protobuf.git && \
    cd protobuf && \
    ./autogen.sh && \
    ./configure && \
    make -j8 && \
    make install && \
    ldconfig && \
    make clean && \
    cd .. && \
    rm -r protobuf
$ docker build -t node_protobuf:0.1 .
...
make[1]: Leaving directory '/protobuf/src'
Removing intermediate container fc421c4bc68c
 ---> 4fefe33499e4
Successfully built 4fefe33499e4
Successfully tagged node_protobuf:0.1

Second

FROM node_protobuf:0.1

RUN git clone https://github.com/agreatfool/grpc_tools_node_protoc_ts.git grpc_protoc && \
    cd grpc_protoc/examples && \
    npm i -g grpc-tools@1.6.6 --unsafe-perm && \
    npm i -g --unsafe-perm typescript@3.3 && \
    npm i --unsafe-perm
$ docker build -t node_protoc_built:0.1 .
...
npm WARN example@1.0.0 No repository field.

added 211 packages in 451.898s
npm info ok
Removing intermediate container b3bcff295074
 ---> 29f2fa49da4b
Successfully built 29f2fa49da4b
Successfully tagged node_protoc_built:0.1

Third

$ docker run --name protoc --rm -i -t node_protoc_built:0.1 /bin/bash
root@25806b55bea6:/# ls
bin  boot  dev	etc  grpc_protoc  home	lib  lib64  media  mnt	opt  proc  root  run  sbin  srv  sys  tmp  usr	var
root@e3cf4ba4661b:/# cd grpc_protoc
root@e3cf4ba4661b:/# ls
LICENSE  README.md  bash  bin  build  examples	package-lock.json  package.json  src  tsconfig.json  tslint.json

As you can see, there is no node_modules in code directory. Thus, tsc failed.

@agreatfool
Copy link
Owner

I didn't notice you have changed the dir to 'examples', so please ignore previous comment.

@agreatfool
Copy link
Owner

I got the issue:

Since the tsc plugin command used in examples/bash/build.sh is a relative path:

grpc_tools_node_protoc \
--plugin=protoc-gen-ts=../bin/protoc-gen -ts \
--ts_out=${PROTO_DEST} \
-I ./proto \
proto/*.proto

Notice: ../bin/protoc-gen

And the outside plugin main npm packages not installed, so error encountered.


Update:

FROM node:8.4.0

RUN apt-get update && \
    apt-get -y install git unzip build-essential autoconf libtool

RUN git clone https://github.com/google/protobuf.git && \
    cd protobuf && \
    ./autogen.sh && \
    ./configure && \
    make -j8 && \
    make install && \
    ldconfig && \
    make clean && \
    cd .. && \
    rm -r protobuf

RUN git clone https://github.com/agreatfool/grpc_tools_node_protoc_ts.git grpc_protoc && \
    cd grpc_protoc && \
    npm i -g grpc-tools@1.6.6 --unsafe-perm && \
    npm i -g --unsafe-perm typescript@3.3 && \
    npm i --unsafe-perm && \
    cd ./examples && \
    npm i --unsafe-perm && \
    ./bash/build.sh

I did't test the combined Dockerfile (I just tested with separated piece), you could have a try.

@EvanDarwin
Copy link
Author

@agreatfool Well I'm glad we could get a baseline docker image to work. But I'm still having issues when I install it on a fresh Linux host (I've given up on Windows at this point). I was able to run a docker image running Node 11, so I'm not sure where these issues are coming from now:

$ sudo npm i -g grpc-tools@1.6.6 grpc@1.9.0 --unsafe-perm --compile-from-source ; npm i
/usr/bin/grpc_tools_node_protoc -> /usr/lib/node_modules/grpc-tools/bin/protoc.js
/usr/bin/grpc_tools_node_protoc_plugin -> /usr/lib/node_modules/grpc-tools/bin/protoc_plugin.js

> grpc-tools@1.6.6 install /usr/lib/node_modules/grpc-tools
> node-pre-gyp install

[grpc-tools] Success: "/usr/lib/node_modules/grpc-tools/bin/grpc_tools.node" is installed via remote

> grpc@1.9.0 install /usr/lib/node_modules/grpc
> node-pre-gyp install --fallback-to-build --library=static_library

node-pre-gyp ERR! Tried to download(403): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.0/node-v67-linux-x64-glibc.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for grpc@1.9.0 and node@11.10.0 (node-v67 ABI, glibc) (falling back to source compile with node-gyp) 
node-pre-gyp ERR! Tried to download(undefined): https://storage.googleapis.com/grpc-precompiled-binaries/node/grpc/v1.9.0/node-v67-linux-x64-glibc.tar.gz 
node-pre-gyp ERR! Pre-built binaries not found for grpc@1.9.0 and node@11.10.0 (node-v67 ABI, glibc) (falling back to source compile with node-gyp) 
make: Entering directory '/usr/lib/node_modules/grpc/build'
make: Entering directory '/usr/lib/node_modules/grpc/build'
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/backoff/backoff.o
rm: cannot remove './Release/.deps/Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o.d.raw': No such file or directory
grpc.target.mk:394: recipe for target 'Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o' failed
make: *** [Release/obj.target/grpc/deps/grpc/src/core/lib/surface/init.o] Error 1
make: Leaving directory '/usr/lib/node_modules/grpc/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:197:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:254:12)
gyp ERR! System Linux 4.15.0-1027-gcp
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--library=static_library" "--module=/usr/lib/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc/grpc_node.node" "--module_name=grpc_node" "--module_path=/usr/lib/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc"
gyp ERR! cwd /usr/lib/node_modules/grpc
gyp ERR! node -v v11.10.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
node-pre-gyp ERR! build error 
node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --library=static_library --module=/usr/lib/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc/grpc_node.node --module_name=grpc_node --module_path=/usr/lib/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/usr/lib/node_modules/grpc/node_modules/node-pre-gyp/lib/util/compile.js:83:29)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:197:13)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:984:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:265:5)
node-pre-gyp ERR! System Linux 4.15.0-1027-gcp
node-pre-gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/grpc/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build" "--library=static_library"
node-pre-gyp ERR! cwd /usr/lib/node_modules/grpc
node-pre-gyp ERR! node -v v11.10.0
node-pre-gyp ERR! node-pre-gyp -v v0.6.39
node-pre-gyp ERR! not ok 
Failed to execute '/usr/bin/node /usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --fallback-to-build --library=static_library --module=/usr/lib/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc/grpc_node.node --module_name=grpc_node --module_path=/usr/lib/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc' (1)
  CXX(target) Release/obj.target/grpc/deps/grpc/src/core/lib/channel/channel_args.onstall script
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! grpc@1.9.0 install: `node-pre-gyp install --fallback-to-build --library=static_library`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the grpc@1.9.0 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/evan/.npm/_logs/2019-02-24T03_43_30_701Z-debug.log
../deps/grpc/src/core/lib/channel/channel_args.cc:499:1: fatal error: opening dependency file ./Release/.deps/Release/obj.target/grpc/deps/grpc/src/core/lib/channel/channel_args.o.d.raw: No such file or directory
 }
 ^
compilation terminated.
grpc.target.mk:394: recipe for target 'Release/obj.target/grpc/deps/grpc/src/core/lib/channel/channel_args.o' failed
make: *** [Release/obj.target/grpc/deps/grpc/src/core/lib/channel/channel_args.o] Error 1
make: Leaving directory '/usr/lib/node_modules/grpc/build'
gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:262:23)
gyp ERR! stack     at ChildProcess.emit (events.js:197:13)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:254:12)
gyp ERR! System Linux 4.15.0-1027-gcp
gyp ERR! command "/usr/bin/node" "/usr/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--library=static_library" "--module=/usr/lib/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc/grpc_node.node" "--module_name=grpc_node" "--module_path=/usr/lib/node_modules/grpc/src/node/extension_binary/node-v67-linux-x64-glibc"
gyp ERR! cwd /usr/lib/node_modules/grpc
gyp ERR! node -v v11.10.0
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok 
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/62/ce'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/be/44'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/e5/b2'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/63/88'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/1f/c2'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/5e/f3'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/58/c9'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/25/03'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/f9/cf'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/15/95'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/58/3d'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/ab/39'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/5f/58'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/be/e5'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/f9/1b'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/ab/83'
Unhandled rejection Error: EACCES: permission denied, mkdir '/home/evan/.npm/_cacache/index-v5/15/7b'

npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://npm.community>

@EvanDarwin
Copy link
Author

EvanDarwin commented Feb 24, 2019

OK, when I use --unsafe-perm for everything, I'm able to generate my prototypes (and their services), but I'm encountering another issue:

tsconfig.json:

{
  "compilerOptions": {
    "strict": true,
    "allowSyntheticDefaultImports": true,
    "allowUnreachableCode": false,
    "allowUnusedLabels": false,
    "noImplicitThis": true,
    "noImplicitReturns": true,
    "noImplicitAny": false,
    "noUnusedLocals": true,
    "noFallthroughCasesInSwitch": true,
    "target": "es2016",
    "lib": [
      "es5",
      "es6",
      "es7",
      "esnext"
    ],
    "rootDir": "lib/",
    "outDir": "dist/",
    "declaration": true,
    "declarationDir": "dist/"
  },
  "exclude": [
    "dist/",
    "lib/**/*.d.ts"
  ]
}

package.json:

{
  "private": true,
  "index": "./dist/index.js",
  "types": "./dist/index.ts",
  "scripts": {
    "compile": "./bin/compile.sh",
    "postinstall": "tsc -p ."
  },
  "dependencies": {
    "@grpc/grpc-js": "^0.3.5",
    "google-protobuf": "~3.5.0",
    "grpc_tools_node_protoc_ts": "^2.5.0",
    "protobufjs": "^6.8.8"
  },
  "devDependencies": {
    "@types/google-protobuf": "^3.2.7",
    "@types/node": "^11.9.4",
    "grpc-tools": "^1.6.6",
    "shx": "^0.3.2",
    "ts-node": "^8.0.2",
    "typescript": "^3.3.3"
  }
}

tsc error:

lib/proto/customer/customer_grpc_pb.d.ts:6:23 - error TS2307: Cannot find module 'grpc'.

6 import * as grpc from "grpc";
                        ~~~~~~

I'll investigate more and see if I can figure out what the permissions issue is, and how we can update the readme to make this easier for new users.

@EvanDarwin
Copy link
Author

So funny enough, getting a new PC solved this issue for me. I think it may have something to do with working on the project in both Windows and Linux (WSL) and directory case sensitivity.

It does not work on Windows, but I'm able to quickly switch to bash, remove node_modules and reinstall it in Linux, and then run the compiler script.

bin/compile.sh

#!/bin/bash
PROTO_SRC_DIR="./proto/"
PROTO_OUT_DIR="./lib/"
PROTO_FILES=$(find ${PROTO_SRC_DIR} -type f -name '*.proto' -printf ' %P')

bin_not_found() {
    echo "Failed to find binary at: $1" >&2
    exit 1
}

PROTO_GRPC_TS_BIN="./node_modules/.bin/grpc_tools_node_protoc"
PROTO_GRPC_TS_PLUGIN_BIN="./node_modules/.bin/grpc_tools_node_protoc_plugin"
PROTOC_GEN_TS_BIN="./node_modules/grpc_tools_node_protoc_ts/bin/protoc-gen-ts"

if ! [[ -x "$(command -v ${PROTO_GRPC_TS_BIN})" ]]; then bin_not_found "$PROTO_GRPC_TS_BIN"; fi
if ! [[ -x "$(command -v ${PROTO_GRPC_TS_PLUGIN_BIN})" ]]; then bin_not_found "$PROTO_GRPC_TS_PLUGIN_BIN"; fi
if ! [[ -x "$(command -v ${PROTOC_GEN_TS_BIN})" ]]; then bin_not_found "$PROTOC_GEN_TS_BIN"; fi

rm -r ${PROTO_OUT_DIR}/*
mkdir -p ${PROTO_OUT_DIR}

${PROTO_GRPC_TS_BIN} --grpc_out=${PROTO_OUT_DIR} \
    --plugin="protoc-gen-grpc=${PROTO_GRPC_TS_PLUGIN_BIN}" \
    -I ${PROTO_SRC_DIR} --js_out="import_style=commonjs,binary:${PROTO_OUT_DIR}" \
    ${PROTO_FILES}

${PROTO_GRPC_TS_BIN} \
    --plugin="protoc-gen-ts=$PROTOC_GEN_TS_BIN" \
    -I ${PROTO_SRC_DIR} --ts_out=${PROTO_OUT_DIR} \
    ${PROTO_FILES}

package.json dependencies:

Dependency Version
google-protobuf ~3.5.0
grpc_tools_node_protoc_ts ^2.5.0
protobufjs ^6.8.8
grpc-tools ^1.6.6

@yz-ark7
Copy link

yz-ark7 commented Apr 20, 2020

I have met the same problem. After some investigation, I've figured out that .bin/protoc-gen-ts was linked by two plugins: this one and the "ts-protoc-gen": "^0.9.0". There's a race condition depending on the platform.

After uninstalling the other plugin, the problem solved.

@agreatfool
Copy link
Owner

@yz-ark7 very useful information, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants