Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send events through postgres notify #2449

Merged
merged 21 commits into from
Oct 24, 2019
Merged

Conversation

saneery
Copy link
Contributor

@saneery saneery commented Jul 26, 2019

#2354

Realtime events are sending by Registry, but when indexer runs separately from webapp, on different instances, events aren't delivered.

So I updated publisher to send events through Postgres notification and added Listener that listen to events from Postgres and send them to subscribed processes

To test you can run indexer and webapp separately and make sure that realtime updates are delivered in the main page.

$ mix cmd --app indexer iex -S mix
$ mix cmd --app block_scout_web mix phx.server

@saneery saneery added the wip label Jul 26, 2019
@saneery saneery force-pushed the realtime-pubsub-by-postges-notify branch from 77cd8f3 to e6df337 Compare July 26, 2019 14:42
@coveralls
Copy link

coveralls commented Jul 26, 2019

Pull Request Test Coverage Report for Build 3bb441bb-ca24-467d-a26e-7e2f2216da9f

  • 7 of 22 (31.82%) changed or added relevant lines in 4 files are covered.
  • 4 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-0.2%) to 76.597%

Changes Missing Coverage Covered Lines Changed/Added Lines %
apps/explorer/lib/explorer/chain/events/db_sender.ex 0 6 0.0%
apps/explorer/lib/explorer/chain/events/listener.ex 0 9 0.0%
Files with Coverage Reduction New Missed Lines %
apps/block_scout_web/lib/block_scout_web/controllers/chain/market_history_chart_controller.ex 2 71.43%
apps/indexer/lib/indexer/fetcher/replaced_transaction.ex 2 90.91%
Totals Coverage Status
Change from base Build 2afb100c-ab3c-44d0-9d7d-8b6d646b7cfe: -0.2%
Covered Lines: 5289
Relevant Lines: 6905

💛 - Coveralls

@saneery saneery force-pushed the realtime-pubsub-by-postges-notify branch 2 times, most recently from 89bef55 to cc0f237 Compare July 29, 2019 13:19
@saneery saneery force-pushed the realtime-pubsub-by-postges-notify branch from cc0f237 to 19d84ab Compare July 29, 2019 13:42
@saneery saneery added ready for review This PR is ready for reviews. and removed wip labels Aug 1, 2019
Copy link
Member

@vbaranov vbaranov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@saneery

  • please update the corresponding chapter with the new env vars
  • please add those env var(s) to the description of PR
  • can we make sending events through postgres notify configurable via env var and if it is not set - use the current websocket architecture?

@saneery
Copy link
Contributor Author

saneery commented Aug 15, 2019

@vbaranov

  • please update the corresponding chapter with the new env vars

I didn't add new vars

  • can we make sending events through postgres notify configurable via env var and if it is not set - use the current websocket architecture?

Yes, I think it's possible

@vbaranov
Copy link
Member

I didn't add new vars

Ah, indeed, sorry. I mistakenly thought, that HEALTHY_BLOCKS_PERIOD introduced in this PR.

@jcortejoso
Copy link

jcortejoso commented Sep 25, 2019

Hi! Is there any plan to merge/review this PR? I have been using Blockscout from this commit and so far so good.

@jcortejoso
Copy link

I have been testing this PR and I think it breaks the option to configure the database with the env variable DATABASE_URL. When running from this branch and configuring database access with DATABASE_URL I get the next error:

╰─ mix phx.server
2019-10-01T22:10:35.339 application=que [info] [Que] Booting Que
2019-10-01T22:10:35.376 application=que [info] [Que] Booting Server Supervisor for Workers
2019-10-01T22:10:35.496 [info] Application explorer exited: Explorer.Application.start(:normal, []) returned an error: shutdown: failed to start child: Explorer.Chain.Events.Listener
    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: {:error, {%KeyError{key: :database, message: nil, term: [types: nil, types: Postgrex.DefaultTypes, hostname: "localhost", username: "jcortejoso", sync_connect: true, migration_timestamps: [type: :utc_datetime_usec], url: "postgres://postgres:1234@pucela.local:5432/blockscout?ssl=false", pool_size: 20, ssl: false, prepare: :unnamed, timeout: 60000]}, [{Keyword, :fetch!, 2, [file: 'lib/keyword.ex', line: 393]}, {Postgrex.Protocol, :startup, 2, [file: 'lib/postgrex/protocol.ex', line: 676]}, {Postgrex.Protocol, :handshake, 2, [file: 'lib/postgrex/protocol.ex', line: 579]}, {Postgrex.Notifications, :connect, 2, [file: 'lib/postgrex/notifications.ex', line: 153]}, {Postgrex.Notifications, :init, 1, [file: 'lib/postgrex/notifications.ex', line: 143]}, {Connection, :init_it, 6, [file: 'lib/connection.ex', line: 424]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}
            (explorer) lib/explorer/chain/events/listener.ex:15: Explorer.Chain.Events.Listener.init/1
            (stdlib) gen_server.erl:374: :gen_server.init_it/2
            (stdlib) gen_server.erl:342: :gen_server.init_it/6
            (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3
** (Mix) Could not start application explorer: Explorer.Application.start(:normal, []) returned an error: shutdown: failed to start child: Explorer.Chain.Events.Listener
    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: {:error, {%KeyError{key: :database, message: nil, term: [types: nil, types: Postgrex.DefaultTypes, hostname: "localhost", username: "jcortejoso", sync_connect: true, migration_timestamps: [type: :utc_datetime_usec], url: "postgres://postgres:1234@pucela.local:5432/blockscout?ssl=false", pool_size: 20, ssl: false, prepare: :unnamed, timeout: 60000]}, [{Keyword, :fetch!, 2, [file: 'lib/keyword.ex', line: 393]}, {Postgrex.Protocol, :startup, 2, [file: 'lib/postgrex/protocol.ex', line: 676]}, {Postgrex.Protocol, :handshake, 2, [file: 'lib/postgrex/protocol.ex', line: 579]}, {Postgrex.Notifications, :connect, 2, [file: 'lib/postgrex/notifications.ex', line: 153]}, {Postgrex.Notifications, :init, 1, [file: 'lib/postgrex/notifications.ex', line: 143]}, {Connection, :init_it, 6, [file: 'lib/connection.ex', line: 424]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}}
            (explorer) lib/explorer/chain/events/listener.ex:15: Explorer.Chain.Events.Listener.init/1
            (stdlib) gen_server.erl:374: :gen_server.init_it/2
            (stdlib) gen_server.erl:342: :gen_server.init_it/6
            (stdlib) proc_lib.erl:249: :proc_lib.init_p_do_apply/3

Configuration using config files work well.

@ayrat555
Copy link
Contributor

ayrat555 commented Oct 2, 2019

@jcortejoso I'll finish this. Thanks for testing

@ayrat555 ayrat555 self-assigned this Oct 2, 2019
@jcortejoso
Copy link

@ayrat555 thanks. I added this to apps/explorer/config/prod.exs and seems to work now:

diff --git a/apps/explorer/config/prod.exs b/apps/explorer/config/prod.exs
index 0957ffa1f..1e040434a 100644
--- a/apps/explorer/config/prod.exs
+++ b/apps/explorer/config/prod.exs
@@ -2,7 +2,12 @@ use Mix.Config

 # Configures the database
 config :explorer, Explorer.Repo,
-  url: System.get_env("DATABASE_URL"),
+  url: System.get_env("DATABASE_URL") || "postgresql://postgres:1234@localhost:5432/blockscout",
+  username: System.get_env("DATABASE_USER") || "postgres",
+  password: System.get_env("DATABASE_PASSWORD") || "1234",
+  database: System.get_env("DATABASE_DB") || "blockscout",
+  hostname: System.get_env("DATABASE_HOSTNAME") || "localhost",
+  port: System.get_env("DATABASE_PORT") || "5432",
   pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10"),
   ssl: String.equivalent?(System.get_env("ECTO_USE_SSL") || "true", "true"),
   prepare: :unnamed,

Best!

@ayrat555
Copy link
Contributor

ayrat555 commented Oct 9, 2019

this kind of variables should be configured with prod.secret.exs file

@ayrat555 ayrat555 requested review from vbaranov and pasqu4le and removed request for pasqu4le October 10, 2019 07:08
@pasqu4le
Copy link
Contributor

@ayrat555 there is no changelog entry

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready for review This PR is ready for reviews.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants