Anvil stops producing blocks and then mines a bunch of blocks at once

[golden-expiriensu]

Component

Anvil

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (f8a07c3 2023-08-22T21:43:06.347698566Z)

What command(s) is the bug in?

anvil

Operating System

Linux

Describe the bug

I am running anvil in the kubernetes with following dockerfile and entrypoint:

Dockerfile:

FROM ghcr.io/foundry-rs/foundry

WORKDIR /app

COPY . ./

RUN ["chmod", "+x", "./entrypoint.sh"]

EXPOSE 8545

ENTRYPOINT ["./entrypoint.sh"]

entrypoint.sh

#!/bin/sh
anvil \
    -b 12 \
    --host '0.0.0.0' \
    --prune-history 300 \
    --state "/app/blockchain_data" \
    --transaction-block-keeper 3600 \
    -f ${ETHEREUM_URL}

Sometimes the anvil hangs up and stops producing blocks for about 5 minutes. During this time it does not respond to most requests and just hangs. The amount of RAM consumption does not increase, the process does not crash. After this delay, the anvil produces a bunch of blocks at a time and starts responding to some requests. After 2-3 such delays, the node starts responding steadily, and nothing crashes

In the attached logs the bug starts at 2023-10-13T09:31:54.519257387Z
anvil.log

[mattsse]

I see this is in forking mode,

are you making any requests during that time?

[golden-expiriensu]

This happened again and I think it is happening when metamask makes requests to node in order to get info about a newly imported account. The RAM consumption raised from ~500 MiB to 1.2 GiB after this particular delay

Additional context: RAM limit on pod is 4 GiB, anvil is running for 46 hours

[golden-expiriensu]

I see this is in forking mode,

are you making any requests during that time?

Yeah, it looks like anvil is not able to handle all the incoming requests from metamask + backend. But the stange thing tho is that it working good then account is not newly imported, so I think when you importing a new account metamask makes a ton of requests

[mattsse]

I see there are a few eth_call requests

I wonder if this has something to do with rpc rate limits,
could you try this with --no-rate-limit

[golden-expiriensu]

Sure, I'll add a flag, redeploy the pod and report back with results

[golden-expiriensu]

Unfortunately disabling rate-limits didnt fix the problem, here is the new logs
first batch: 2023-10-13T11:29:52.754852809Z
second batch: 2023-10-13T11:34:41.848433939Z
normal block: 2023-10-13T11:34:44.932910143Z
anvil_no_rate_limits.log

[golden-expiriensu]

The changes in command:

[golden-expiriensu]

Hello, I am still experiencing this issue on ghcr.io/foundry-rs/foundry:nightly image :(

[mattsse]

looking at the logs, I'm pretty sure this is an issue with tx execution in forked mode, which delays block production.
I think we we should tune the block interval so that missed intervals aren't resulting in bursts of blocks?

[golden-expiriensu]

Hmmm, I use 12 seconds to mock Ethereum, what are your suggestions?

[golden-expiriensu]

By the way, the reason is actually metamask, we don't experience any problems while not creating new accounts/adding anvil as a new network

[mattsse]

Hmmm, I use 12 seconds to mock Ethereum, what are your suggestions?

I meant in anvil we should delay missed blocks (for what ever reason) and if a block is late, still target the next block for 12s because rn I think we just mined all missed blocks in bulk

[mattsse]

By the way, the reason is actually metamask, we don't experience any problems while not creating new accounts/adding anvil as a new network

hmm, interesting, I guess the call chainId blocknumber a lot, need to check those if they somehow interfere with blockproduction in forking mode

[KELs7]

I had a similar issue but in a non-fork mode with Metamask. I was getting a lot of frustrating -32603: Internal JSON-RPC error errors from Metamask. The recommendation here by @mattsse seems to work.

I also tried using the prefix RUST_LOG=node,backend,api,rpc=warn anvil and realised that transactions seem to fail when they come in too early before the next block is mined and that is why shortening the block time interval seem to make the problem go away (my interpretation might be wrong).

If I was proficient in rust and understood anvil enough, I would have attempted solving this issue.

[KELs7]

Thank you @mattsse . Hoping to test this soon

[KELs7]

I have built anvil from your branch. With a block time of 5s I get the internal RPC error once in a while. I suspect I might be sending too many requests to anvil.

I am wondering about anvil keeping unprocessed txns in a pool until they're mined.

[zerosnacks]

Tentatively marking as resolved by #7328

@golden-expiriensu / @KELs7 feel free to re-open with additional details / minimal reproduction if this is still an issue

[jtaaa]

Just adding to this thread incase useful. MM does the following eth_call when importing a new account. It's a massive request that attempts to load token balances for many different tokens. For me running anvil locally it takes between 3–6 minutes to respond to that request, during which time anvil cannot respond to other requests — other requests timeout.
mm_eth_call.json

It does balances method on this contract https://etherscan.io/address/0xb1f8e55c7f64d203c1400b9d8555d050f94adf39#readContract