[Feature] Do not ingest blocks if the new head block number is significantly higher than the previously known block head block number

[paymog]

Description

Logic was recently introduced which guarded against the head block going backwards. It would be great if logic could also be introduced to prevent the head block from skipping forwards by a lot, 10% for example.

Some reasons for why this could be useful:

1. there's an issue with upstream rpc providers where they misconfigure their infra and switch nodes for a given endpoint, ie: the mainnet endpoint starts serving arbitrum blocks
2. an indexer does a config update and accidentally configures, eg, the mainnet ingester to use an aribtrum endpoint

If this edge case happens, the block cache gets corrupted and requires significant manual intervention. It would be great if graph node could attempt to detect this issue and prevent the block cache from getting corrupted during the block ingestion process.

Are you aware of any blockers that must be resolved before implementing this feature? If so, which? Link to any relevant GitHub issues.

I'm not aware of any blockers.

Some information to help us out

• Tick this box if you plan on implementing this feature yourself.
• I have searched the issue tracker to make sure this issue is not a duplicate.

[github-actions]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.

[azf20]

@paymog there is some potential complexity here, for example if the Graph Node is stopped for some time that could be hard to distinguish. I think there is currently a check of the genesis hash which should mitigate this kind of issue on configuration for RPC providers, but I think that wouldn't cover the case where the upstream RPC is inadvertently switched. Have you seen this kind of issue regularly?

[paymog]

Ah, great point about the potential complexity with the Graph Node being stopped for a time. Did not think of that!

No, this isn't an issue that happens regularly but it did happen once or twice for me.

I believe the genesis hash check only happens on Graph Node startup, is that correct? If so, that absolutely doesn't help against issues like this during startup, but during regular runtime where other infra could change underneath graph node.