Running Hydra on AWS Elasticache

[waglik]

I am trying to run Hydra app with AWS Elasticache but unfortunately I got

ReplyError: ERR unknown command 'config'

error. It is caused by the fact that config command is restricted on AWS ElastiCache. Is there a way to start hydra without calling config command?

[cjus]

@waglik where do you see that error? Not sure I follow. Please provide more information on how you're running your application that uses Hydra and perhaps an example of your config file.

[waglik]

@cjus I've created simple hydra application and then wrapped it into Docker image. When I run it locally on Redis running on local host it is fine. But when I run Docker image on AWS ESC it fails when starting. Reason is that AWS ElastiCache (this is my Redis instance) does not allow CONFIG command.

here is my config file

{ "environment": "development", "hydra": { "serviceName": "facebook-service", "serviceIP": "", "servicePort": 0, "serviceType": "facebook", "serviceDescription": "Fetch facebook comments", "plugins": { "logger": { "logRequests": false } }, "redis": { "url": "<url to AWS ElastiCache>", "port": 6379, "db": 15 } } }

Error happens when npm start is called. My guess is that it is happening here :

config.init('./config/config.json')

[cjus]

@waglik one thing you could try is backing the config into the service - not ideal - but good for seeing whether this is the only issue.

Just create a JS object for your config and pass it into the hydra.init() call. Are you using hydra or hydraExpress? If hydraExpress then just pass the JavaScript object representing your config to hydraExpress.init.
https://github.com/cjus/hello-service/blob/master/hello-service.js#L19

[cjus]

@waglik also, are you also using AWS Elastic Container Service? ECS?

[waglik]

@cjus I am using just hydra (not hydraExpress). I've added couple of loggers so I am pretty confident that this fails in the init method. If I remove config.init() it works fine.

Yes I am using ECS. I also checked connection with telnet so this is not an issue.

Stack tracke :

events.js:160
      throw er; // Unhandled 'error' event
      ^
ReplyError: ERR unknown command 'config'
    at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:193:12)
    at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:303:14)

npm info lifecycle facebook-service@0.0.2~start: Failed to exec start script
npm ERR! Linux 4.4.51-40.58.amzn1.x86_64
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
npm ERR! node v6.3.1
npm ERR! npm  v3.10.3
npm ERR! code ELIFECYCLE

[cjus]

@waglik Are you saying that if you don't use config.init() to load a config file then you don't have an issue?

Does your redis url looking something like this? test-dev-redis1.p42rev.0001.usw1.cache.amazonaws.com and do you have a password on it?

You can try to format your config this way:

{
  redis: {
    url: 'redis://test-dev-redis1.p42rev.0001.usw1.cache.amazonaws.com:6379/15'
  }
}

Let me know if that makes a difference.

[waglik]

@cjus I've added some more logging. Code looks like that :

console.log('here');
config.init('./config/config.json')
  .then(() => {
    console.log('here too 2');
    config.version = version;
    config.hydra.serviceVersion = version;
    /**
    * Initialize hydra
    */
    return hydra.init(config);
  })
  .then(() => {console.log('xxx');hydra.registerService();console.log('here 3')})
  .then(serviceInfo => {
    console.log('here 4');
   ....

Output is :

here
here too 2

events.js:160
      throw er; // Unhandled 'error' event
      ^
ReplyError: ERR unknown command 'config'
    at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:193:12)
    at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:303:14)

I'm using xyz-cluster.rmz72x.0001.euc1.cache.amazonaws.com cluster link and in config.json :

"redis": {
      "url": "redis://xyz-cluster.rmz72x.0001.euc1.cache.amazonaws.com:6379/15"
    }
  }

[emadum]

I'm not sure why it is trying to execute a CONFIG command. I couldn't find any calls to it in the node-redis module source, and we're not calling it directly from Hydra or Hydra Express. I can't see how it could have anything to do with loading config.json. @waglik, your code and config look OK to me. We're also using AWS Elasticache and haven't run into this issue.

As a test to help narrow this down, could you try connecting to redis directly after config.init? E.g. something like:

const redis = require('redis');
const config = require('fwsp-config');
config.init('./config/config.json')
  .then(() => {
    let client = redis.createClient(config.hydra.redis);
    client.time(redis.print);
  });

That should print out two values, e.g.

$ node testredis
Reply: 1493046174,987530

[cjus]

@waglik @emadum I think the issue might be that hydra.init requires a config JSON and not a config object. It's necessary to call hydra.init this way:

hydra.init(config.getObject())

Also see: https://github.com/flywheelsports/fwsp-config/blob/master/index.js#L18

[waglik]

@cjus just tried and gives same result. Beside same docker image works locally against Redis running on my localhost. So I am confident that issue is with AWS Redis and not with the code itself. From what I found AWS Redis does not allow calling CONFIG command. Is hydra calling that command inside init() method?

[emadum]

@waglik did you see my comment above?

[waglik]

@emadum no, sorry.

I've made changes you suggested and I got :

here
/usr/local/bin/node[21]: ../src/pipe_wrap.cc:268:static void node::PipeWrap::Connect(const v8::FunctionCallbackInfo<v8::Value>&): Assertion `args[1]->IsString()' failed.
 1: node::Abort() [node]
 2: node::Assert(char const* const (*) [4]) [node]
 3: node::PipeWrap::Connect(v8::FunctionCallbackInfo<v8::Value> const&) [node]
 4: v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) [node]
 5: 0x9d5b0b [node]
 6: 0x9d60b1 [node]
 7: 0x23900f10961b
Aborted

code :

console.log('here');

config.init('./config/config.json')
  .then(() => {
    let client = redis.createClient(config.hydra.redis);
    client.time(redis.print);
  });

[emadum]

@waglik what version of node are you using?

[cjus]

@waglik Hydra requires Node 6.2.1 or greater.

[waglik]

@emadum @cjus Docker file says "FROM node:6.3" , it's the one generated by Hydra.

[cjus]

@waglik Would you be interested in doing a skype call? I'm at cjustini34
If so I'd like to find a time when we can step through the issue you're experiencing. I'm in New York.

[waglik]

@cjus I really appreciate that. I've sent you skype invite, let me know when is a good time. I am in EET time zone most of the time.

[cjus]

@waglik accepted. I should be able to do a call during your evenings.

[cjus]

@waglik I'm still interested in a skype call if you have time ;-)

[waglik]

@cjus I am sorry I missed the notifications. I will ping you today. thanks!

[cjus]

An update on this ticket. @waglik and I had a skype call to discuss and I'm in the process of setting up an AWS ECS cluster for testing this issue.

[itsJess1ca]

Hi I'm looking into using hydra, it will be running on AWS - I just noticed this issue and wanted to check in and see if this is still an ongoing problem? :)
Excited to use Hydra, it looks great!

[cjus]

@j3ddesign I never did get to test this. If you try it let us know what you find and we'll offer support. Recommend you sign up for our slack channel where others are on-hand to weigh in.