[Bug] Subgraph is requesting a block range that exceeds the current most recent block

[jurgenwerk]

Bug report

We are running a graph node that is consistently around 20-100 blocks behind the latest block indexed in the archive node.

This is on gnosis chain mainnet. We are hosting our own graph node v0.27.0.

The error that keeps appearing is:

May 23 10:57:46.726 WARN Trying again after eth_getLogs RPC call for block range: [28087279..28087335] failed (attempt #10) with result Err(Rpc(Error { code: ServerError(-32001), message: "28087335 could not be found", data: None })), sgd: 3, subgraph_id: QmfBJ5mdAkVbaugY1mcrcqFU2KsTysgwG6smguRFiDDVAC, component: BlockStream

This means the subgraph is requesting a block range that includes some blocks that are not available yet.

I did some testing and the offending block number from the warning message above, at the time of the output, is:

1. Higher than what is the latest block returned from the archive node
2. Higher than what is the latest block in the block explorer website (gnosis scan)

It looks like the subgraph has a wrong idea about which block range to request for indexing. It wants blocks that are not available yet.

Perhaps related - this started happening after our subgraph got stuck due to a combination of a reorg and archive node issues and we performed grafting from a lower block number.

Any ideas why the subgraph is behaving like this now and how to mitigate the problem?

Relevant log output

No response

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• Tick this box if this bug is caused by a regression found in the latest release.
• Tick this box if this bug is specific to the hosted service.
• I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

Linux

[azf20]

What does the graph node report as the latest block on the chain?
You can check the index node on port 8030/graphql (schema)

[jurgenwerk]

@azf20 latestBlock from 8030/graphql is returning a lower value than what is the latest block on chain. For example, it returns 28088547 while the latest block from the archive node and in https://gnosisscan.io/ is 28088607.

[jurgenwerk]

@azf20 I discovered something very concerning. If I query for the chain head block, I get a value that is out of touch with reality. The result is 28465419 while the latest block on (https://gnosisscan.io/block/28091860) is 28091860.

Is there a way to reset the chain head block to the real value? If I update it with in the database, it gets immediately overwritten with the old incorrect value. I suppose that is happening because it's overwriting it with the max number value from the blocks table, because there are records in that table that have block numbers far higher that any of the block in existence. I am really not sure how we got into this situation. Perhaps because we used grafting to get the graph unstuck due to reorg or archive node issues. Is there a way to reset values in the graph node to correctly reflect the blockchain state? We can't afford to do a rebuild because it takes many days.

# curl http://localhost:8030/graphql -X POST -d '{"query": "query { indexingStatusForCurrentVersion(subgraphName: \"subgraph-xdai\") { chains { network chainHeadBlock { hash number } } } }"}' -H "Content-Type: application/json"

{"data":{"indexingStatusForCurrentVersion":{"chains":[{"network":"xdai","chainHeadBlock":{"hash":"46c7038711697ae75773470fff9f946d317a7e4a66cadd3d88215b205af5c73e","number":"28465419"}}]}}}#

[azf20]

hey @jurgenwerk that suggests that the RPC you are using is returning the problematic data, have you checked what block it returns?

[jurgenwerk]

@azf20 it's returning valid data now. Does this mean it was returning invalid data at some point, possibly from a fork or during reorg?

[azf20]

If the block numbers are much higher than expected that suggests an error rather than a fork (forked blocks would also be automatically removed). If you delete the erroneous blocks from the blocks table that should resolve the issue

[jurgenwerk]

We decided to rebuild the subgraph from scratch and ended up in the same state as described above.

To repeat, graph node is trying to request blocks that don't exist yet - blocks whose number is larger than what the rpc node is returning as latest. The latest block value from the rpc node is correct, but the graph node is still requesting ranges of blocks that don't exist yet.

In addition to that, the chainHeadBlock does not reflect the reality; its current value is 28490129 while the real chain head block on gnosis is 28118998.

What could be the reason for this? Is it possible the rpc node is returning incorrect blocks data?

[azf20]

did you remove the blocks from the blocks table? that is populated from the RPC

[jurgenwerk]

I did but the graph got stuck then and could not continue syncing. The error was something like "can't find block 0x..."

[github-actions]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.