Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error flushing outbound message: I/O error: stopped by peer: code 0 #415

Closed
sandreae opened this issue Jun 21, 2023 · 3 comments
Closed

Error flushing outbound message: I/O error: stopped by peer: code 0 #415

sandreae opened this issue Jun 21, 2023 · 3 comments

Comments

@sandreae
Copy link
Member

After fixing the #407 here #413 then I now get this new error/behaviour:

Node A

   Compiling aquadoggo v0.4.0 (/home/sandreae/Code/aquadoggo/aquadoggo)
   Compiling aquadoggo_cli v0.2.0 (/home/sandreae/Code/aquadoggo/aquadoggo_cli)
    Finished dev [unoptimized + debuginfo] target(s) in 2m 00s
     Running `target/debug/aquadoggo`
[2023-06-21T04:25:41Z DEBUG aquadoggo::schema::schema_provider] Initialised schema provider:
    - schema_definition_v1
    - venue_0020ca1d2b85567443b2db42bba286c07fc64b1a46f2d3e67ba819550b09e18eccd0
    - event_0020c3560ffcfa50f14ab01740cd8f832e0fb35035eeb8cd4aca124bac2d42a47a2a
    - band_00207c0dec3322d78efc419432c4a215b3635cb17b3cf20674ff47ec3aed0fa3f24e
    - schema_field_definition_v1
[2023-06-21T04:25:41Z INFO  aquadoggo::manager] Start materializer service
[2023-06-21T04:25:41Z INFO  aquadoggo::materializer::worker] Register reduce worker with pool size 16
[2023-06-21T04:25:41Z INFO  aquadoggo::materializer::worker] Register dependency worker with pool size 16
[2023-06-21T04:25:41Z INFO  aquadoggo::materializer::worker] Register schema worker with pool size 16
[2023-06-21T04:25:41Z DEBUG aquadoggo::materializer::service] Dispatch 0 pending tasks from last runtime
[2023-06-21T04:25:41Z DEBUG aquadoggo::materializer::service] Materialiser service is ready
[2023-06-21T04:25:41Z INFO  aquadoggo::manager] Start http service
[2023-06-21T04:25:41Z DEBUG aquadoggo::graphql::schema] Subscribing GraphQL manager to schema provider
[2023-06-21T04:25:41Z DEBUG aquadoggo::graphql::schema] Finished building initial GraphQL schema
[2023-06-21T04:25:41Z DEBUG aquadoggo::http::service] HTTP service is ready
[2023-06-21T04:25:41Z INFO  aquadoggo::manager] Start network service
[2023-06-21T04:25:41Z INFO  aquadoggo::network::swarm] Local peer id: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk
[2023-06-21T04:25:41Z DEBUG aquadoggo::network::behaviour] AutoNAT network behaviour enabled
[2023-06-21T04:25:41Z DEBUG aquadoggo::network::behaviour] mDNS network behaviour enabled
[2023-06-21T04:25:41Z INFO  aquadoggo::manager] Start replication service
[2023-06-21T04:25:41Z INFO  libp2p_mdns::behaviour::iface] creating instance on iface 192.168.40.110
[2023-06-21T04:25:41Z INFO  libp2p_mdns::behaviour::iface] creating instance on iface 172.17.0.1
[2023-06-21T04:25:41Z DEBUG libp2p_quic::transport] New listen address: /ip4/127.0.0.1/udp/2022/quic-v1
[2023-06-21T04:25:41Z DEBUG libp2p_swarm] Listener ListenerId(17562011302729291341); New address: "/ip4/127.0.0.1/udp/2022/quic-v1"
[2023-06-21T04:25:41Z DEBUG aquadoggo::network::service] Listening on /ip4/127.0.0.1/udp/2022/quic-v1
[2023-06-21T04:25:41Z DEBUG libp2p_quic::transport] New listen address: /ip4/192.168.40.110/udp/2022/quic-v1
[2023-06-21T04:25:41Z DEBUG libp2p_swarm] Listener ListenerId(17562011302729291341); New address: "/ip4/192.168.40.110/udp/2022/quic-v1"
[2023-06-21T04:25:41Z DEBUG aquadoggo::network::service] Listening on /ip4/192.168.40.110/udp/2022/quic-v1
[2023-06-21T04:25:41Z DEBUG libp2p_quic::transport] New listen address: /ip4/172.17.0.1/udp/2022/quic-v1
[2023-06-21T04:25:41Z DEBUG libp2p_swarm] Listener ListenerId(17562011302729291341); New address: "/ip4/172.17.0.1/udp/2022/quic-v1"
[2023-06-21T04:25:41Z DEBUG aquadoggo::network::service] Listening on /ip4/172.17.0.1/udp/2022/quic-v1
[2023-06-21T04:25:53Z INFO  libp2p_mdns::behaviour] discovered: 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd /ip4/172.17.0.1/udp/2024/quic-v1
[2023-06-21T04:25:53Z INFO  libp2p_mdns::behaviour] discovered: 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd /ip4/192.168.40.110/udp/2024/quic-v1
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::service] mDNS discovered a new peer: 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::service] Dial success: skip remaining addresses for: 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd
[2023-06-21T04:25:53Z DEBUG libp2p_swarm] Connection established: PeerId("12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd") Listener { local_addr: "/ip4/0.0.0.0/udp/2022/quic-v1", send_back_addr: "/ip4/192.168.40.110/udp/2024/quic-v1" }; Total (peer): 1.
[2023-06-21T04:25:53Z INFO  aquadoggo::replication::service] Connected to peer: 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd (2)
[2023-06-21T04:25:53Z INFO  aquadoggo::replication::manager] Initiate outbound replication session with peer 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd (2)
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::manager] Drop pending outbound session and process inbound session request with duplicate id 0
[2023-06-21T04:25:53Z INFO  aquadoggo::replication::manager] Initiate outbound replication session with peer 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd (2)
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(0) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(1) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(2) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(5) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(6) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(7) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(8) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(9) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(10) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(11) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(12) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(13) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(14) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd (2)
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(0) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(1) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(2) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(3) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(2) on LogId(3) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(3) on LogId(3) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(4) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(2) on LogId(4) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(3) on LogId(4) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(4) on LogId(4) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(5) on LogId(4) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(5) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(6) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(7) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(8) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(9) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(10) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(11) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(12) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(13) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(14) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(15) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(2) on LogId(15) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::strategies::log_height] Prepare message containing entry at SeqNum(1) on LogId(16) for <PublicKey 53fc96>
[2023-06-21T04:25:53Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd (2)
[2023-06-21T04:25:53Z WARN  aquadoggo::network::peers::handler] Error flushing outbound message: I/O error: stopped by peer: code 0
[2023-06-21T04:25:56Z DEBUG libp2p_autonat::behaviour::as_client] Outbound dial-back request aborted: No qualified server.
[2023-06-21T04:25:56Z INFO  aquadoggo::replication::manager] Initiate outbound replication session with peer 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd (2)
[2023-06-21T04:25:56Z WARN  aquadoggo::network::peers::handler] Error flushing outbound message: I/O error: stopped by peer: code 0
[2023-06-21T04:25:58Z DEBUG libp2p_swarm] Connection attempt to unknown peer failed with Transport([("/ip4/172.17.0.1/udp/2024/quic-v1", Other(Custom { kind: Other, error: HandshakeTimedOut }))])
[2023-06-21T04:25:58Z WARN  aquadoggo::network::service] Outgoing connection error occurred: Failed to negotiate transport protocol(s): [(/ip4/172.17.0.1/udp/2024/quic-v1: : Handshake with the remote timed out.)]
[2023-06-21T04:25:58Z DEBUG libp2p_swarm] Incoming connection failed: Transport(Other(Custom { kind: Other, error: HandshakeTimedOut }))
[2023-06-21T04:25:58Z WARN  aquadoggo::network::service] Incoming connection error occurred with /ip4/0.0.0.0/udp/2022/quic-v1 and /ip4/192.168.40.110/udp/2024/quic-v1: Listen error: Failed to negotiate transport protocol(s)
[2023-06-21T04:26:41Z DEBUG libp2p_swarm] Connection closed with error IO(Custom { kind: Other, error: Connection(ConnectionError(Reset)) }): Connected { endpoint: Listener { local_addr: "/ip4/0.0.0.0/udp/2022/quic-v1", send_back_addr: "/ip4/192.168.40.110/udp/2024/quic-v1" }, peer_id: PeerId("12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd") }; Total (peer): 0.
[2023-06-21T04:26:41Z WARN  aquadoggo::network::service] Connection error occurred with peer 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd: reset by peer
[2023-06-21T04:26:41Z INFO  aquadoggo::replication::service] Disconnected from peer: 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd (2)
[2023-06-21T04:26:41Z DEBUG aquadoggo::replication::service] Remove peer: 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd (2)
^C[2023-06-21T04:26:44Z INFO  aquadoggo::manager] Received shutdown signal

Node B

    Finished dev [unoptimized + debuginfo] target(s) in 1.90s
     Running `target/debug/aquadoggo --http-port 2023 --quic-port 2024 --data-dir=/home/sandreae/test`
[2023-06-21T04:25:51Z INFO  aquadoggo::network::config] Created new network key pair and saved it to "/home/sandreae/test/private-key"
[2023-06-21T04:25:51Z DEBUG aquadoggo::schema::schema_provider] Initialised schema provider:
    - schema_definition_v1
    - schema_field_definition_v1
[2023-06-21T04:25:52Z INFO  aquadoggo::manager] Start materializer service
[2023-06-21T04:25:53Z INFO  aquadoggo::materializer::worker] Register reduce worker with pool size 16
[2023-06-21T04:25:53Z INFO  aquadoggo::materializer::worker] Register dependency worker with pool size 16
[2023-06-21T04:25:53Z INFO  aquadoggo::materializer::worker] Register schema worker with pool size 16
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::service] Dispatch 0 pending tasks from last runtime
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::service] Materialiser service is ready
[2023-06-21T04:25:53Z INFO  aquadoggo::manager] Start http service
[2023-06-21T04:25:53Z DEBUG aquadoggo::graphql::schema] Subscribing GraphQL manager to schema provider
[2023-06-21T04:25:53Z DEBUG aquadoggo::graphql::schema] Finished building initial GraphQL schema
[2023-06-21T04:25:53Z DEBUG aquadoggo::http::service] HTTP service is ready
[2023-06-21T04:25:53Z INFO  aquadoggo::manager] Start network service
[2023-06-21T04:25:53Z INFO  aquadoggo::network::swarm] Local peer id: 12D3KooWPYoBAiqWfFxrqNv6waU8hbyoN3nBAqbAXtGY93PP17Zd
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::behaviour] AutoNAT network behaviour enabled
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::behaviour] mDNS network behaviour enabled
[2023-06-21T04:25:53Z INFO  aquadoggo::manager] Start replication service
[2023-06-21T04:25:53Z INFO  libp2p_mdns::behaviour::iface] creating instance on iface 192.168.40.110
[2023-06-21T04:25:53Z INFO  libp2p_mdns::behaviour::iface] creating instance on iface 172.17.0.1
[2023-06-21T04:25:53Z DEBUG libp2p_quic::transport] New listen address: /ip4/127.0.0.1/udp/2024/quic-v1
[2023-06-21T04:25:53Z DEBUG libp2p_swarm] Listener ListenerId(4814561773630677650); New address: "/ip4/127.0.0.1/udp/2024/quic-v1"
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::service] Listening on /ip4/127.0.0.1/udp/2024/quic-v1
[2023-06-21T04:25:53Z INFO  libp2p_mdns::behaviour] discovered: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk /ip4/192.168.40.110/udp/2022/quic-v1
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::service] mDNS discovered a new peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::service] Dial success: skip remaining addresses for: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk
[2023-06-21T04:25:53Z INFO  libp2p_mdns::behaviour] discovered: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk /ip4/172.17.0.1/udp/2022/quic-v1
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::service] mDNS discovered a new peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::service] Dial success: skip remaining addresses for: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk
[2023-06-21T04:25:53Z DEBUG libp2p_quic::transport] New listen address: /ip4/192.168.40.110/udp/2024/quic-v1
[2023-06-21T04:25:53Z DEBUG libp2p_swarm] Listener ListenerId(4814561773630677650); New address: "/ip4/192.168.40.110/udp/2024/quic-v1"
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::service] Listening on /ip4/192.168.40.110/udp/2024/quic-v1
[2023-06-21T04:25:53Z DEBUG libp2p_quic::transport] New listen address: /ip4/172.17.0.1/udp/2024/quic-v1
[2023-06-21T04:25:53Z DEBUG libp2p_swarm] Listener ListenerId(4814561773630677650); New address: "/ip4/172.17.0.1/udp/2024/quic-v1"
[2023-06-21T04:25:53Z DEBUG aquadoggo::network::service] Listening on /ip4/172.17.0.1/udp/2024/quic-v1
[2023-06-21T04:25:53Z DEBUG libp2p_swarm] Connection established: PeerId("12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk") Dialer { address: "/ip4/192.168.40.110/udp/2022/quic-v1", role_override: Dialer }; Total (peer): 1.
[2023-06-21T04:25:53Z INFO  aquadoggo::replication::service] Connected to peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z INFO  aquadoggo::replication::manager] Initiate outbound replication session with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z DEBUG aquadoggo::replication::manager] Ignore inbound request and keep pending outbound session with duplicate id 0
[2023-06-21T04:25:53Z INFO  aquadoggo::replication::manager] Accept inbound replication session with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::network::peers::handler] Error decoding inbound message: CBOR error: invalid type: integer `100`, expected p2panda replication message
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::worker] Sending materializer reduce task with input <TaskInput 002099dd2c7a77ce59fddb48bcfd18baefc73888849436d471d4aba40aa8f027e13a/-> to the task queue.
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::reduce] Working on <TaskInput 002099dd2c7a77ce59fddb48bcfd18baefc73888849436d471d4aba40aa8f027e13a/->
[2023-06-21T04:25:53Z INFO  aquadoggo::materializer::tasks::reduce] Created <Document 27e13a>
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::reduce] Dispatch dependency task for view with id: 002099dd2c7a77ce59fddb48bcfd18baefc73888849436d471d4aba40aa8f027e13a
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::worker] Sending materializer dependency task with input <TaskInput -/002099dd2c7a77ce59fddb48bcfd18baefc73888849436d471d4aba40aa8f027e13a> to the task queue.
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/002099dd2c7a77ce59fddb48bcfd18baefc73888849436d471d4aba40aa8f027e13a>
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::dependency] Document retrieved from storage with view id: 002099dd2c7a77ce59fddb48bcfd18baefc73888849436d471d4aba40aa8f027e13a
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 1 reduce tasks
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::worker] Sending materializer schema task with input <TaskInput -/002099dd2c7a77ce59fddb48bcfd18baefc73888849436d471d4aba40aa8f027e13a> to the task queue.
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::schema] Working on <TaskInput -/002099dd2c7a77ce59fddb48bcfd18baefc73888849436d471d4aba40aa8f027e13a>
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::worker] Silently failing worker schema with task <QueueItem 0 w. <TaskInput -/002099dd2c7a77ce59fddb48bcfd18baefc73888849436d471d4aba40aa8f027e13a>>: Related schema definition not given (yet)
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::worker] Sending materializer reduce task with input <TaskInput 00209f6b09271e1a71ae118d1db23247e3f9365a067d05a7c600f6294efe2853915a/-> to the task queue.
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::reduce] Working on <TaskInput 00209f6b09271e1a71ae118d1db23247e3f9365a067d05a7c600f6294efe2853915a/->
[2023-06-21T04:25:53Z INFO  aquadoggo::materializer::tasks::reduce] Created <Document 53915a>
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::reduce] Dispatch dependency task for view with id: 00209f6b09271e1a71ae118d1db23247e3f9365a067d05a7c600f6294efe2853915a
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::worker] Sending materializer dependency task with input <TaskInput -/00209f6b09271e1a71ae118d1db23247e3f9365a067d05a7c600f6294efe2853915a> to the task queue.
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/00209f6b09271e1a71ae118d1db23247e3f9365a067d05a7c600f6294efe2853915a>
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::dependency] Document retrieved from storage with view id: 00209f6b09271e1a71ae118d1db23247e3f9365a067d05a7c600f6294efe2853915a
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 1 reduce tasks
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::worker] Sending materializer schema task with input <TaskInput -/00209f6b09271e1a71ae118d1db23247e3f9365a067d05a7c600f6294efe2853915a> to the task queue.
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::tasks::schema] Working on <TaskInput -/00209f6b09271e1a71ae118d1db23247e3f9365a067d05a7c600f6294efe2853915a>
[2023-06-21T04:25:53Z DEBUG aquadoggo::materializer::worker] Silently failing worker schema with task <QueueItem 1 w. <TaskInput -/00209f6b09271e1a71ae118d1db23247e3f9365a067d05a7c600f6294efe2853915a>>: Related schema definition not given (yet)
[2023-06-21T04:25:53Z WARN  aquadoggo::network::peers::handler] Error decoding inbound message: CBOR error: invalid type: string "\u{18}5\u{18}0\u{18}2", expected p2panda replication message
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: Incoming data could not be ingested: Entry's claimed seq num of 1 does not match expected seq num of 2 for given public key and log
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1) failed: No session found with id 1 for peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:53Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 1 with peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:25:58Z DEBUG libp2p_swarm] Connection attempt to unknown peer failed with Transport([("/ip4/172.17.0.1/udp/2022/quic-v1", Other(Custom { kind: Other, error: HandshakeTimedOut }))])
[2023-06-21T04:25:58Z WARN  aquadoggo::network::service] Outgoing connection error occurred: Failed to negotiate transport protocol(s): [(/ip4/172.17.0.1/udp/2022/quic-v1: : Handshake with the remote timed out.)]
[2023-06-21T04:25:58Z DEBUG libp2p_swarm] Incoming connection failed: Transport(Other(Custom { kind: Other, error: HandshakeTimedOut }))
[2023-06-21T04:25:58Z WARN  aquadoggo::network::service] Incoming connection error occurred with /ip4/0.0.0.0/udp/2024/quic-v1 and /ip4/192.168.40.110/udp/2022/quic-v1: Listen error: Failed to negotiate transport protocol(s)
[2023-06-21T04:26:08Z DEBUG libp2p_autonat::behaviour::as_client] Outbound dial-back request aborted: No qualified server.
[2023-06-21T04:26:33Z DEBUG libp2p_swarm] Connection closed with error KeepAliveTimeout: Connected { endpoint: Dialer { address: "/ip4/192.168.40.110/udp/2022/quic-v1", role_override: Dialer }, peer_id: PeerId("12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk") }; Total (peer): 0.
[2023-06-21T04:26:33Z DEBUG aquadoggo::network::service] Connection timed out with peer 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk
[2023-06-21T04:26:33Z INFO  aquadoggo::replication::service] Disconnected from peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
[2023-06-21T04:26:33Z DEBUG aquadoggo::replication::service] Remove peer: 12D3KooWHy186KwkZ4CAWmmNDQdPCoVQuBAmyn7QJBChwCzWPWxk (1)
@sandreae
Copy link
Member Author

It seems like it could be related to: #397 but I'm not sure.

@sandreae
Copy link
Member Author

Interesting line: [2023-06-21T04:25:53Z WARN aquadoggo::network::peers::handler] Error decoding inbound message: CBOR error: invalid type: string "\u{18}5\u{18}0\u{18}2", expected p2panda replication message

@adzialocha
Copy link
Member

adzialocha commented Jun 21, 2023 •
edited
Loading

At 2023-06-21T04:25:53Z Peer B fails with:

CBOR error: invalid type: integer `100`, expected p2panda replication message

Which should lead to the sub stream being closed ("critical error"). I guess that's why sending data on the other's Peer A stream fails with at the same time as Peer B failed:

Error flushing outbound message: I/O error: stopped by peer: code 0

So far that's roughly expected behaviour! Both peers consider the stream as faulty and close it.

The question is, why did the stream fail?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@adzialocha @sandreae