[Bug] some subgraphs encounter "Subgraph stopped, WASM runtime thread terminated" and stop indexing

[chenzongxiong]

Bug report

I use the official subgraphs

• rocket-pool-mainet: https://github.com/Data-Nexus/rocket-pool-mainnet
• messari/uniswap-v2/v3: https://github.com/messari/subgraphs/tree/master/subgraphs/uniswap-forks
• messari/curve-finance: https://github.com/messari/subgraphs/tree/master/subgraphs/curve-finance

And I find it will stop indexing at specific blocks, e.g.

• rocket-pool-mainet at block 17,794,258
• curve-finance-ethereum at block 17,797,894
• ...

And there is no bugs or errors throw from graph-node process, but stop processing further blocks.

Relevant log output

contracts, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: BlockStream
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.146 DEBG Requesting logs for blocks [17794862, 17795270], event 0xef51b4c870b8b0100eae2072e91db01222a303072af3728e58c9d4d2da33127f, 0
 contracts, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: BlockStream
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.157 DEBG Found 49 relevant block(s), sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: BlockStream
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.167 DEBG eth_call, block_number: 17794265, block_hash: f34ffcd8a5b6880ffc787f3675022979b4aa196df6c5d23298cd7cd5bb8b963e, data: e71501
34, address: 0a882c9059cc2e97c860b80018c27145884d694b, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.177 DEBG eth_call, block_number: 17794265, block_hash: f34ffcd8a5b6880ffc787f3675022979b4aa196df6c5d23298cd7cd5bb8b963e, data: f0d19b
89000000000000000000000000c3af24f9d2850d83dbcffc4c35b15c34f84966a2, address: 3019227b2b8493e45bf5d25302139c9a2713bf15, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: Subg
raphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.186 DEBG eth_call, block_number: 17794265, block_hash: f34ffcd8a5b6880ffc787f3675022979b4aa196df6c5d23298cd7cd5bb8b963e, data: 03fa87
b4000000000000000000000000c3af24f9d2850d83dbcffc4c35b15c34f84966a2, address: 3019227b2b8493e45bf5d25302139c9a2713bf15, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: Subg
raphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.191 DEBG eth_call, block_number: 17794265, block_hash: f34ffcd8a5b6880ffc787f3675022979b4aa196df6c5d23298cd7cd5bb8b963e, data: 4e58ff6e000000000000000000000000c3af24f9d2850d83dbcffc4c35b15c34f84966a2, address: 3019227b2b8493e45bf5d25302139c9a2713bf15, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: Subg
raphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.220 INFO Create data source, params: 0x92db5083ab4219e5305100d8318026ea19a73620, name: rocketMinipoolDelegate, sgd: 39, subgraph_id:
QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.220 INFO Done processing trigger, gas_used: 20475216043, data_source: rocketMinipoolManagerAtlas, handler: handleMinipoolCreated, tot
al_ms: 90, transaction: 0xd9b4…6021, address: 0x6d01…49bf, signature: MinipoolCreated(indexed address,indexed address,uint256), sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, compon
ent: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.222 INFO Done processing trigger, gas_used: 93631351, data_source: rocketMinipoolDelegate, handler: handleEtherDeposited, total_ms: 1
, transaction: 0xd9b4…6021, address: 0x92db…3620, signature: EtherDeposited(indexed address,uint256,uint256), sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: SubgraphInsta
nceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.224 INFO Done processing trigger, gas_used: 201438674, data_source: rocketMinipoolDelegate, handler: handleStatusUpdated, total_ms: 2
, transaction: 0xd9b4…6021, address: 0x92db…3620, signature: StatusUpdated(indexed uint8,uint256), sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: , gas_used: 93720492, data_source: rocketMinipoolDelegate, handler: handleEtherDeposited, total_ms: 1, transaction: 0xd9b4…6021, address: 0x92db…3620
, signature: EtherDeposited(indexed address,uint256,uint256), sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, component: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.300 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, comp
onent: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.301 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, comp
onent: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.301 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, comp
onent: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.301 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, comp
onent: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.301 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, comp
onent: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.301 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, comp
onent: SubgraphInstanceManager
Sep 29 14:27:50 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:27:50.301 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 39, subgraph_id: QmfLKcvxhEJNcde67yUWTnjbxrRnuPQfe8adfVPCCg1anR, comp

Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.186 DEBG eth_call, block_number: 17820767, block_hash: 475c97c84b4fad89f251ce17d3805ec48200122f9a1d946d71967f2df7c9101f, data: bdf475c30000000000000000000000004ebdf703948ddcea3b11f675b4d1fba9d2414a14, address: 4aacf35761d06aa7142b9326612a42a2b9170e33, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.195 WARN Could not find pool for lp token 0x4ebdf703948ddcea3b11f675b4d1fba9d2414a14, data_source: Booster, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager > UserMapping
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.198 DEBG eth_call, block_number: 17820767, block_hash: 475c97c84b4fad89f251ce17d3805ec48200122f9a1d946d71967f2df7c9101f, data: 50940618, address: f403c135812408bfbe8713b5a23a04b3d48aae31, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.200 DEBG eth_call, block_number: 17820767, block_hash: 475c97c84b4fad89f251ce17d3805ec48200122f9a1d946d71967f2df7c9101f, data: 3a088cd2, address: f403c135812408bfbe8713b5a23a04b3d48aae31, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.205 DEBG eth_call, block_number: 17820767, block_hash: 475c97c84b4fad89f251ce17d3805ec48200122f9a1d946d71967f2df7c9101f, data: 62d28ac7, address: f403c135812408bfbe8713b5a23a04b3d48aae31, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.210 DEBG eth_call, block_number: 17820767, block_hash: 475c97c84b4fad89f251ce17d3805ec48200122f9a1d946d71967f2df7c9101f, data: 26232a2e, address: f403c135812408bfbe8713b5a23a04b3d48aae31, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.230 INFO Create data source, params: 0xf956a46dba1a0a567168db8655bc18e9050c7738, name: PoolCrvRewards, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.238 WARN [NewVault] poolId: 211, name: TriCRV, inputToken: 0x4ebdf703948ddcea3b11f675b4d1fba9d2414a14, pool: 0x0000000000000000000000000000000000000000, data_source: Booster, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager > UserMapping
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.238 INFO Done processing trigger, gas_used: 87293035912, data_source: Booster, handler: handleAddPool, total_ms: 460, transaction: 0xf2ea…3a0f, to: 0xf403…ae31, function: addPool(address,address,uint256), sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.240 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.240 DEBG Subgraph stopped, WASM runtime thread terminated, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: SubgraphInstanceManager
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.967 DEBG Received 3 traces for blocks [17821358, 17821407], sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: BlockStream
Sep 29 14:15:55 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:55.967 DEBG Found 45 relevant block(s), sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: BlockStream
Sep 29 14:15:56 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:56.902 DEBG Requesting 0 block(s), sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: BlockStream
Sep 29 14:15:56 ubuntu-8gb-nbg1-1 graph-node[3825978]: Sep 29 14:15:56.903 DEBG Processing 63 triggers, sgd: 20, subgraph_id: QmYgXVANbehheH6uq6Arzd5Ci6YQreRw9WyoyhqEBtLvQz, component: BlockStream

### IPFS hash

_No response_

### Subgraph name or link to explorer

_No response_

### Some information to help us out

- [ ] Tick this box if this bug is caused by a regression found in the latest release.
- [X] Tick this box if this bug is specific to the hosted service.
- [X] I have searched the issue tracker to make sure this issue is not a duplicate.

### OS information

Linux

[azf20]

hello @chenzongxiong do you see consistent behaviour (i.e. the same stop block even if you restart the Graph Nodes)? what Ethereum provider are you using for your Graph Node?

[chenzongxiong]

@azf20 I'm using own erigon node, and the graph node runs locally.

1. Restarting the service doesn't help solve this issue.
2. Rewinding the subgraphs several blocks doesn't help either. The wasm thread will stop at the same block.

I'm deploying a new subgraph and see what will happen.

[incrypto32]

@chenzongxiong any updates on the new subgraph you deployed? I'd be happy to help investigate this issue.

[chenzongxiong]

@incrypto32
By re-deploy the rocket-pool-mainet, the problem solves.
messari/curve-finance is unclear since it's still under indexing.

The deployed wasm and previous wasm of rocket-pool-mainet are definitely the same.
So I think the issue is due to graph-node itself, which can fail suspiciously.

[chenzongxiong]

By redeploying the rest subgraphs, this issue is also resolved. So I think it's due to internal graph-node problems, not subgraph self.

[incrypto32]

Thats Wierd, I'll see if its reproducable.

[github-actions]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.

[fordN]

Closing this old issue. We can reopen or start a new one if anyone is having similar issues.