feat: sqlite indexer

[hanabi1224]

Summary of changes

Changes introduced in this pull request:

benchmark:

# command
oha -n 1000 -c 1 -m POST http://127.0.0.1:2345/rpc/v1 -H 'Content-Type: application/json' -D eth_getLogs.json

# payload (using max allowed range 2880 with topic filter)
{
	"method": "eth_getLogs",
	"params": [
	{
		"fromBlock": "0x3c4d80",
		"toBlock": "0x3c58c0",
		"topics": [
			"0x58b7742b13c8873fc0ba58f695b33ca0044b2db7ff9c5208181dbaec2a5b291e",
			"0x0000000000000000000000000000000000000000000000000000000000002169"
		]
	}
	],
	"id": 1,
	"jsonrpc": "2.0"
}

# NOSQL with cache
  Total:────────82910.3099 ms────────────────────────┘└───────────────────────────────────────────────────┘
  Slowest:      379.3777 ms
  Fastest:      66.1746 ms
  Average:      82.8963 ms
  Requests/sec: 12.0612

  Total data:   5.63 MiB
  Size/request: 5.77 KiB
  Size/sec:     69.56 KiB

Response time histogram:
   66.175 ms [1]   |
   97.495 ms [868] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  128.815 ms [129] |■■■■
  160.136 ms [1]   |
  191.456 ms [0]   |
  222.776 ms [0]   |
  254.096 ms [0]   |
  285.417 ms [0]   |
  316.737 ms [0]   |
  348.057 ms [0]   |
  379.378 ms [1]   |

Response time distribution:
  10.00% in 73.8108 ms
  25.00% in 75.8106 ms
  50.00% in 78.8940 ms
  75.00% in 82.6644 ms
  90.00% in 107.4672 ms
  95.00% in 114.2749 ms
  99.00% in 124.6876 ms
  99.90% in 379.3777 ms
  99.99% in 379.3777 ms
  
# NOSQL without cache
  Total:        175673.6178 ms
  Slowest:      445.6916 ms
  Fastest:      161.4016 ms
  Average:      175.6593 ms
  Requests/sec: 5.6924

  Total data:   5.63 MiB
  Size/request: 5.77 KiB
  Size/sec:     32.83 KiB

Response time histogram:
  161.402 ms [1]   |
  189.831 ms [977] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  218.260 ms [19]  |
  246.689 ms [0]   |
  275.118 ms [2]   |
  303.547 ms [0]   |
  331.976 ms [0]   |
  360.405 ms [0]   |
  388.834 ms [0]   |
  417.263 ms [0]   |
  445.692 ms [1]   |

Response time distribution:
  10.00% in 167.7529 ms
  25.00% in 171.4235 ms
  50.00% in 175.0497 ms
  75.00% in 178.6948 ms
  90.00% in 182.3066 ms
  95.00% in 184.6605 ms
  99.00% in 194.7276 ms
  99.90% in 445.6916 ms
  99.99% in 445.6916 ms

# SQL  
  Total:────────146722.3188 ms───────────────────────┘└───────────────────────────────────────────────────┘
  Slowest:      199.1718 ms
  Fastest:      60.5419 ms
  Average:      146.7080 ms
  Requests/sec: 6.8156

  Total data:   5.63 MiB
  Size/request: 5.77 KiB
  Size/sec:     39.31 KiB

Response time histogram:
   60.542 ms [1]   |
   74.405 ms [25]  |■■
   88.268 ms [72]  |■■■■■■■■
  102.131 ms [52]  |■■■■■
  115.994 ms [16]  |■
  129.857 ms [30]  |■■■
  143.720 ms [135] |■■■■■■■■■■■■■■■
  157.583 ms [280] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  171.446 ms [83]  |■■■■■■■■■
  185.309 ms [287] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  199.172 ms [19]  |■■

Response time distribution:
  10.00% in 89.2464 ms
  25.00% in 132.2283 ms
  50.00% in 146.4321 ms
  75.00% in 179.1558 ms
  90.00% in 182.5456 ms
  95.00% in 183.8301 ms
  99.00% in 186.5293 ms
  99.90% in 199.1718 ms
  99.99% in 199.1718 ms

Reference issue to close (if applicable)

Closes #7017

Other information and links

Change checklist

• I have performed a self-review of my own code,
• I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
• I have added tests that prove my fix is effective or that my feature works (if possible),
• I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Outside contributions

• I have read and agree to the CONTRIBUTING document.
• I have read and agree to the AI Policy document. I understand that failure to comply with the guidelines will lead to rejection of the pull request.

Summary by CodeRabbit

• New Features

  • Added SQL-backed chain indexing with automatic population, maintenance, garbage collection, and fork handling.
  • Added Filecoin.ChainValidateIndex and the forest-cli index validate-backfill command for validating and optionally backfilling index data.
  • Large-range EthGetLogs requests can now use the SQL index for improved retrieval.

• Documentation

  • Documented FOREST_RPC_SQL_RANGE_THRESHOLD, configurable with a default of 500.
  • Added unreleased changelog information for SQL index backfilling and validation.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds a SQLite-backed chain indexer with schema initialization, validation, backfilling, indexed event queries, daemon integration, RPC support, and the forest-cli index validate-backfill command.

Changes

SQLite chain indexer

Layer / File(s)	Summary
SQLite storage and indexer construction Cargo.toml, src/utils/sqlite/*, src/db/mod.rs, src/chain/store/indexer/*, src/shim/*	Adds SQLite setup, index tables, prepared statements, indexer construction, and supporting address and actor types.
Index population, validation, and event persistence src/chain/store/indexer.rs, src/daemon/context.rs, src/daemon/mod.rs, src/chain/store/indexer/tests.rs	Adds tipset indexing, fork handling, garbage collection, snapshot population, backfilling, validation, event persistence, and daemon lifecycle wiring.
Indexed EthGetLogs queries src/chain/store/indexer/events.rs, src/rpc/methods/eth.rs, src/rpc/methods/eth/filter/mod.rs, docs/docs/users/reference/env_variables.md	Adds indexed event filters and routes large tipset ranges through SQLite using the configurable threshold.
RPC and CLI index validation src/rpc/methods/chain.rs, src/rpc/methods/chain/types.rs, src/rpc/mod.rs, src/cli/subcommands/index_cmd.rs, scripts/tests/api_compare/*, CHANGELOG.md	Adds Filecoin.ChainValidateIndex, the validation response type, CLI range processing, integration-test setup, and changelog documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• ChainSafe/forest#7408: Adds related SQLite index validation and backfilling APIs.
• ChainSafe/forest#7437: Updates the same API comparison service for index validation.

Suggested reviewers: lesnyrumcajs, sudo-shashank, akaladarshi

Sequence Diagram(s)

sequenceDiagram
  participant ForestCLI
  participant ChainValidateIndex
  participant SqliteIndexer
  participant ChainStore
  ForestCLI->>ChainValidateIndex: validate epoch with backfill flag
  ChainValidateIndex->>SqliteIndexer: validate_index(epoch, backfill)
  SqliteIndexer->>ChainStore: load canonical tipset data
  ChainStore-->>SqliteIndexer: return chain data
  SqliteIndexer-->>ChainValidateIndex: return ChainIndexValidation
  ChainValidateIndex-->>ForestCLI: return validation result

Loading

sequenceDiagram
  participant EthClient
  participant EthGetLogs
  participant ParsedFilter
  participant SqliteIndexer
  participant SQLite
  EthClient->>EthGetLogs: submit event filter
  EthGetLogs->>ParsedFilter: parse tipset range
  ParsedFilter-->>EthGetLogs: select SQL index for large range
  EthGetLogs->>SqliteIndexer: query indexed events
  SqliteIndexer->>SQLite: execute filtered event query
  SQLite-->>EthGetLogs: return collected events

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The PR includes unrelated cleanup and wording changes, such as the F3 description edit and variable renames.	Remove unrelated cleanup and wording changes, or move them into a separate pull request.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR adds a SQLite indexer and integrates it with large-range EthGetLogs queries, addressing issue #7017.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: adding a SQLite indexer.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch hm/sqlite-indexer

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch hm/sqlite-indexer

---

_{Comment @coderabbitai help to get the list of available commands.}

[LesnyRumcajs]

As agreed, let's have it disabled for now.

[LesnyRumcajs]

If you took it from Lotus, let's have a permalink.