The forms-flow-analytics component crashes

[zmezei-proxifile]

Bug Report

Please provide information about your setup

• formsflow.ai Version: v4.0.2, git commit ID: 1735fde
• Browser: (irrelevant)
• Browser version: (irrelevant)

Steps to Reproduce

1. Set up a local Keycloak with the default config
2. Set up forms-flow-forms with the following setup:

sed "s/{your-ip-address}/localhost/g" sample.env > .env

3. Set up forms-flow-analytics with the following setup:

sed "s/{your-ip-address}/localhost/g" sample.env > .env
echo -e "\n# Tuning parameters\nSQLALCHEMY_POOL_SIZE=32\nSQLALCHEMY_MAX_OVERFLOW=16" >> .env

Observed Behaviour

The command to set up the forms-flow-analytics component's database (docker-compose -f docker-compose-linux.yml run --rm server create_db) doesn't run successfully, and throws this error:

Successfully built b515be528431
Successfully tagged forms-flow-analytics_server:latest
WARNING: Image for service server was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating forms-flow-analytics_postgres_1 ... done
Creating forms-flow-analytics_redis_1    ... done
Creating forms-flow-analytics_server_run ... done
ERROR: 139

The adhoc_worker process suffers from the same issue, and logs these messages:

adhoc_worker_1  | 2021-09-27 22:19:56,189 INFO success: worker_healthcheck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
adhoc_worker_1  | 2021-09-27 22:20:08,232 INFO exited: worker_healthcheck (terminated by SIGSEGV (core dumped); not expected)
adhoc_worker_1  | 2021-09-27 22:20:08,233 INFO spawned: 'worker_healthcheck' with pid 50
adhoc_worker_1  | 2021-09-27 22:20:08,313 INFO exited: worker-2 (terminated by SIGSEGV (core dumped); not expected)
adhoc_worker_1  | 2021-09-27 22:20:08,421 INFO exited: worker-3 (terminated by SIGSEGV (core dumped); not expected)
adhoc_worker_1  | 2021-09-27 22:20:08,422 INFO exited: worker-0 (terminated by SIGSEGV (core dumped); not expected)
adhoc_worker_1  | 2021-09-27 22:20:09,433 INFO success: worker_healthcheck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
adhoc_worker_1  | 2021-09-27 22:20:09,435 INFO spawned: 'worker-0' with pid 56
adhoc_worker_1  | 2021-09-27 22:20:09,437 INFO spawned: 'worker-2' with pid 57
adhoc_worker_1  | 2021-09-27 22:20:09,439 INFO spawned: 'worker-3' with pid 58
adhoc_worker_1  | 2021-09-27 22:20:09,440 INFO exited: worker-1 (terminated by SIGSEGV (core dumped); not expected)

Expected Behaviour

These services should run properly.

[kurianbenoy-aot]

@zmezei-proxifile what is the operating system you are using? Is your system meeting per-requisities of the system to have greater than 16GB RAM?

I was able to get the redash while running the same commands you mentioned to setup the redash logs. Yet a colleague of mine faced same issues when bring forms-flow-analytics component up in their system. We are looking more into it.

[zmezei-proxifile]

@kurianbenoy-aot - there are two systems I've tried to test on:

• my own machine has 64 GB of RAM and is running Archlinux, docker version 20.10.8 and compose version 1.29.2
• the server machine has 16 GB of RAM and is running Amazon Linux 2, docker version 20.10.7 and compose version 1.29.2

I've tried to increase the memory from 16 GB to 32 GB on the server machine, but it didn't help:

[ec2-user@ip-10-3-129-191 forms-flow-analytics]$ free -g
              total        used        free      shared  buff/cache   available
Mem:             31           1          16           0          13          29
Swap:             0           0           0
[ec2-user@ip-10-3-129-191 forms-flow-analytics]$ docker-compose -f docker-compose-linux.yml run --rm server create_db
Creating forms-flow-analytics_server_run ... done
ERROR: 139

Please note that we have a different server that we set up earlier, and that one runs this component properly on 32 GB of RAM, so I don't think the memory is the issue here. I suspect maybe some upstream dependencies have changed? Unfortunately I'm not familiar enough with redash to have a better guess.

[kurianbenoy-aot]

@zmezei-proxifile it looks like an environmental issue reported by Redash team.

getredash/redash#5593

They closed mentioning This seems like an environmental issue. Closing since we cannot reproduce. For further help/discussion, I'd suggest the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

A temporary fix which worked for my colleague was by changing line 5 in the docker-compose file as image: redash/redash:latest

[zizzencs]

I can confirm I was able to build the DB using the official redash/redash:latest image:

[ec2-user@ip-10-3-129-191 forms-flow-analytics]$ docker-compose -f docker-compose-linux.yml run --rm server create_db
Creating network "forms-flow-analytics_default" with the default driver
Creating forms-flow-analytics_redis_1    ... done
Creating forms-flow-analytics_postgres_1 ... done
Creating forms-flow-analytics_server_run ... done
[2021-09-28 08:09:36,803][PID:1][INFO][alembic.runtime.migration] Context impl PostgresqlImpl.
[2021-09-28 08:09:36,804][PID:1][INFO][alembic.runtime.migration] Will assume transactional DDL.
[2021-09-28 08:09:36,819][PID:1][INFO][alembic.runtime.migration] Running stamp_revision  -> e5c7a4e2df4d

However with this image the scheduler can't start properly:

[ec2-user@ip-10-3-129-191 forms-flow-analytics]$ docker ps
CONTAINER ID   IMAGE                               COMMAND                  CREATED         STATUS                            PORTS                                                 NAMES
a97f2bae2141   redash/redash:latest                "/app/bin/docker-ent…"   4 minutes ago   Up 2 minutes                      5000/tcp                                              forms-flow-analytics_worker_1
ed4bd14f6477   redash/redash:latest                "/app/bin/docker-ent…"   4 minutes ago   Restarting (127) 36 seconds ago                                                         forms-flow-analytics_scheduler_1
b3ab8b16076d   redash/redash:latest                "/app/bin/docker-ent…"   4 minutes ago   Up 2 minutes                      0.0.0.0:7000->5000/tcp, :::7000->5000/tcp             forms-flow-analytics_server_1
ee8a29af1a6b   redash/redash:latest                "/app/bin/docker-ent…"   4 minutes ago   Up 2 minutes                      5000/tcp                                              forms-flow-analytics_adhoc_worker_1
f995bff9f1cd   postgres:9.6-alpine                 "docker-entrypoint.s…"   4 minutes ago   Up 2 minutes                      0.0.0.0:7432->5432/tcp, :::7432->5432/tcp             forms-flow-analytics_postgres_1
33b7c137ee2d   redis:3-alpine                      "docker-entrypoint.s…"   5 minutes ago   Up 2 minutes                      6379/tcp                                              forms-flow-analytics_redis_1
9326716869a3   forms-flow-forms_forms-flow-forms   "node main"              10 hours ago    Up About an hour                  0.0.0.0:3001->3001/tcp, :::3001->3001/tcp             forms-flow-forms_forms-flow-forms_1
572c6c1b1f5c   mongo                               "docker-entrypoint.s…"   10 hours ago    Up About an hour                  0.0.0.0:27018->27017/tcp, :::27018->27017/tcp         forms-flow-forms_forms-flow-forms-db_1
6de26fa27621   jboss/keycloak:14.0.0               "/opt/jboss/tools/do…"   12 hours ago    Up About an hour                  0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 8443/tcp   keycloak
12e2fbded76e   postgres:latest                     "docker-entrypoint.s…"   12 hours ago    Up About an hour                  0.0.0.0:5431->5431/tcp, :::5431->5431/tcp, 5432/tcp   keycloak_db
[ec2-user@ip-10-3-129-191 forms-flow-analytics]$ docker logs ed4bd14f6477
/app/bin/docker-entrypoint: line 124: exec: dev_scheduler: not found
/app/bin/docker-entrypoint: line 124: exec: dev_scheduler: not found

This can be resolved by changing the entrypoint from dev_scheduler to scheduler - I guess this Redash version only supports the latter one.

I hope this can be fixed permanently soon, as I suspect we'll get some CORS errors with the official Redash image as the documentation mentions.

[kurianbenoy-aot]

You can continue with the rest of formsflow.ai setup, without using analytics. In the upcoming version, we are planning to upgrade to latest version of Redash v10(stable), as soon as it's released(which is expected in a week or two).

I hope this can be fixed permanently soon, as I suspect we'll get some CORS errors with the official Redash image as the documentation mentions.

Yes you will run into CORS issues when connecting analytics component with formsflow.ai. We will see what we can do and let you know.

[kurianbenoy-aot]

@zmezei-proxifile we have patched the forms-flow-analytics components issues temporarily in the branch: https://github.com/AOT-Technologies/redash.git#redash-139errorfix

You can fix it with docker-compose update of line5 here by changing with below line:

build: https://github.com/AOT-Technologies/redash.git#redash-139errorfix

We will be merging to master branch in a day or two as well.

[kurianbenoy-aot]

Merged to master branch as well