Service boundaries, events, and consistency without drowning in complexity.

You usually start with “I just need to read some data from the chain”. You build one indexer that talks to one node and one database. It works.

Then it grows:

  • multiple chains,
  • multiple consumers (explorer, DEX, analytics, monitoring),
  • higher throughput and deeper history.

At that point the “one big indexer” becomes painful. Every change risks re‑indexing; deployments are scary; a single bad query can starve everything else.

This is where a microservices architecture and event‑driven pipelines make sense. I’ll focus on how I slice blockchain indexers into services, how those services talk to each other, and how to keep data consistent when the chain can reorg under your feet.

1. Prerequisites

I’ll assume you’re comfortable with:

  • basic microservices concepts (services, bounded contexts, database‑per‑service);
  • Kafka or a similar event broker;
  • what a “classic” indexer does: read blocks/transactions from a node, decode them, write to a relational schema, expose an API.

No code here, just architecture, diagrams, and patterns.

2. Why indexers want microservices and events

A blockchain indexer has three properties that push you towards microservices + events:

  1. High‑volume, append‑only streams.
    Blocks, logs, UTXOs, delegations, swaps… all arrive as ordered streams. Distributed log systems (Kafka, Pulsar, Kinesis, managed “stream” products) are built exactly for this kind of workload.

  2. Many independent consumers.
    Explorers, DEX backends, portfolio apps, analytics jobs, monitors, and risk engines all want slightly different views of the same events. An event‑driven architecture lets each consumer build its own view without fighting over one giant database.

  3. Different scaling and latency profiles.
    Block ingestion is sequential and moderately latency‑tolerant. User‑facing APIs want low latency and horizontal scale with read‑optimised stores and caches.

Event‑driven microservices are a natural fit: ingestion produces “chain events”; other services subscribe and transform them. This is the same pattern you see in generic event‑driven microservice architectures, just with blocks and transactions instead of “order created” events.

Polygon’s Chain Indexer Framework is a public example: Kafka is the backbone for pipelines that turn EVM blocks and logs into downstream views and APIs.

3. Service boundaries for blockchain indexers

The first architectural decision is where to cut.

I slice by responsibility, not by technology. At minimum:

  • ingest raw chain data,
  • normalise into domain events,
  • build read models (SQL, time‑series, aggregates),
  • expose APIs.

ASCII view:

          +----------------------------+
          |         Ingestion          |
          |  - connect to node         |
          |  - read blocks/tx/logs     |
          +-------------+--------------+
                        |
                        |  raw-chain events
                        v
          +----------------------------+
          |   Normalisation / Decode   |
          |  - ABI/script decoding     |
          |  - classify events:        |
          |    transfers, swaps, etc.  |
          +-------------+--------------+
                        |
                        |  domain events
                        v
          +----------------------------+
          |     Projection Services    |
          |  - balances, positions     |
          |  - DEX state, orderbooks   |
          |  - staking / voting views  |
          |  - time-series stats       |
          +-------------+--------------+
                        |
                        |  materialised views
                        v
          +----------------------------+
          |       API / Query          |
          |  - REST / GraphQL          |
          |  - pagination, filters     |
          +----------------------------+

Each box can be one or more microservices. They communicate via events, not via a mesh of synchronous HTTP calls. That keeps coupling low and makes replay straightforward.

For multi‑chain systems I repeat “ingestion + normalisation” per chain, and share or specialise projection/API services depending on whether the use case is chain‑specific or cross‑chain.

From the trenches. The first big reliability win on a BSC indexer that started life as a monolith was splitting “raw chain ingestion” from “projections”. When the DEX projection had a bug, we could stop just that consumer group, fix it, redeploy, and replay from Kafka without touching ingestion or any other projection.

4. Event‑driven communication

Once you have multiple services, you need to get data from one to another without building a fragile web of HTTP dependencies.

For indexers I almost always use an event broker:

[ Ingestion svc ] ---> [ Kafka / broker ] ---> [ Projection svc A ]
                                      \---->   [ Projection svc B ]
                                      \---->   [ Analytics svc    ]

Kafka is the usual suspect (especially in Spring Boot / Kotlin stacks), but the same ideas apply to Pulsar, NATS JetStream, or managed streaming products.

4.1 Event types and schemas

Wire‑level events should be simple, explicit, and versioned.

Conceptually:

block.raw:
  chain_id, height, hash, parent_hash,
  timestamp, raw_block (bytes/CBOR/JSON),
  node_source, received_at

tx.raw:
  chain_id, height, tx_index, tx_hash,
  raw_tx, status, fee, gas_used, logs...

event.domain:
  type = "transfer" | "swap" | "stake_delegation" | ...
  chain_id, height, tx_hash, event_index,
  normalized fields (from, to, amount, asset, pool_id, ...)

reorg:
  chain_id, old_tip_hash, new_tip_hash,
  old_tip_height, new_tip_height, common_ancestor_height

Choose a schema format (Avro, Protobuf, JSON Schema) and treat it as a contract between services. That’s standard event‑driven practice and helps keep independent teams aligned.

4.2 Topic and partition strategy

You also need to decide how to structure topics and partitions.

A typical layout:

Topics:
  chain.{chain_id}.blocks.raw
  chain.{chain_id}.txs.raw
  chain.{chain_id}.events.domain

Partition keys:
  - by block height
  - by tx_hash
  - by entity (address / contract / pool / policy_id)

Trade‑offs:

  • Partition by height Preserves global block order per chain, but may constrain parallelism. On very fast chains, a single “totally ordered” stream can become a bottleneck.

  • Partition by entity Great for scaling projections that are per entity (address, contract, pool). All events for a given entity land on the same partition and are consumed by one instance in order. Global ordering across entities is no longer trivial, but most projections don’t need it.

Kafka patterns for microservices emphasize the same point: choose keys based on the ordering guarantees each consumer actually needs.

For high‑throughput chains I often do:

- one topic per chain + event type
- partitioned by entity key (e.g. address or contract)
- consumer groups sized to match the number of partitions

The rule: all events that affect one logical state machine should land on the same partition.

4.3 Delivery guarantees and idempotence

End‑to‑end “exactly once” across many services is possible but operationally heavy. With Kafka, the pragmatic model is:

  • at‑least‑once delivery from broker to consumer;
  • idempotent consumers.

For indexers that means:

  • deterministic primary keys (for example (chain_id, height, tx_hash, event_index) for events);
  • upserts / ON CONFLICT DO NOTHING/UPDATE instead of blind inserts;
  • “already processed” treated as a normal path, not an exception.

When a service both writes its DB and publishes follow‑up events, patterns like Transactional Outbox give you local atomicity: write the state and the “outbox” row in one transaction, then a relay publishes outbox entries to the broker.

5. Data consistency patterns

Indexers are read‑heavy systems sitting on append‑only, reorg‑prone ledgers. You don’t need global strong consistency; you need eventual consistency with controlled divergence and replay.

Three patterns dominate: CQRS, Event Sourcing, and Sagas/Outbox.

5.1 CQRS for read models

CQRS (Command Query Responsibility Segregation) splits writes (commands) from reads (queries), often with separate models and databases.

In an indexer context:

Write side:
  - consume chain events (blocks/txs/domain events)
  - update internal state / projections
  - emit derived events if needed

Read side:
  - serve denormalised, query-optimised views:
      balances, portfolios, pool states, staking dashboards
  - REST/GraphQL APIs, internal analytics endpoints

Each projection service owns its database per service, tuned for its queries:

balances-db   -> keyed by (address, asset)
dex-db        -> keyed by (pool_id, time)
staking-db    -> keyed by (pool_id, delegator, epoch)

This matches mainstream guidance on microservices and CQRS: separate read models when queries become complex or heavily loaded.

5.2 Event sourcing on top of chain + broker

Event Sourcing says: treat state as a function of the event history; store events as the primary source of truth; rebuild state by replaying them.

For indexers you almost get this for free:

  • the chain itself is an event log (blocks and transactions);
  • Kafka (or equivalent) is an internal event log (raw + decoded + domain events).

Pattern:

Truth:
  - chain.{chain_id}.blocks.raw
  - chain.{chain_id}.txs.raw
  - chain.{chain_id}.events.domain

Caches:
  - balances-db, dex-db, staking-db, analytics-db, ...

If a projection changes:
  - drop / migrate its DB
  - replay from the relevant topics

Polygon’s Chain Indexer and similar frameworks lean heavily on this: Kafka topics are treated as an event store for rebuilding downstream state.

5.3 Reorg handling as explicit events

Reorgs are where indexer consistency gets interesting.

You need:

  • a clear representation of chain state (tip hash, tip height, cumulative work/stake);
  • a way to mark blocks as canonical vs orphaned;
  • a stream of reorg events so projections can react.

I usually model reorg‑related changes as explicit events:

block.canonicalised:
  chain_id, height, hash, previous_status, new_status

block.orphaned:
  chain_id, height, hash,
  new_tip_hash, common_ancestor_height

Projections then have two kinds of logic:

  • forward‑apply effects for canonical blocks (credit balances, add trades, update pools);
  • reverse‑apply effects for orphaned blocks (debit balances, remove trades, revert pools).

With an event‑sourced design you can always rebuild: walk back to the common ancestor, follow the winning branch, replay projections. You avoid one‑off “fix scripts” that nobody fully trusts.

5.4 Cross‑service workflows: Sagas and Outbox

Sometimes one chain event must trigger coordinated changes in multiple services (e.g. indexer + risk engine + notifications).

Distributed 2PC across microservices doesn’t scale. Instead you use:

  • Sagas for long‑running workflows coordinated through events and local transactions;
  • Transactional Outbox to ensure “write state and publish event” happens atomically inside one service.

For indexers, Outbox is usually enough. Saga‑style coordination shows up more where indexer outputs drive off‑chain actions (e.g. sending emails or triggering risk limits) that may fail and need compensation.

6. Scaling strategies

The main reason to move to microservices is to let different parts scale in different ways.

6.1 Scale ingestion by chain and bandwidth

Ingestion is closest to the nodes and gets the raw firehose. It’s naturally partitioned by chain:

[ BTC ingestor ]   [ ADA ingestor ]   [ ATOM ingestor ]   [ EVM-L2 ingestor ]
      |                   |                 |                    |
      v                   v                 v                    v
  topics.btc.*       topics.ada.*     topics.atom.*        topics.evm-l2.*

Within a chain you can scale by:

  • responsibility split: one ingestor for blocks, others for logs of specific contracts or modules;
  • horizontal scaling over partitions: multiple ingestor instances each consuming a subset of partitions or node endpoints.

Stream‑based indexers built on Kafka/Streams, Pulsar Functions, or managed streaming products follow the same pattern: parallel consumers over partitioned streams handle big chains in real time.

From the trenches. I strongly prefer a single ingestion service per chain talking to the nodes, then fanning out via events. Letting every downstream service talk to full nodes directly leads to rate‑limit pain, duplicate logic, and non‑replayable side effects.

6.2 Scale projections and APIs by read profile

Projections and APIs are read‑heavy. That’s exactly where CQRS and “database per service” shine.

Example split:

balances-service:
  - consumes transfer / mint / burn events
  - writes balances-db (PostgreSQL/TimeSeries)
  - exposes /address/{addr}/balances

dex-service:
  - consumes swap / liquidity events
  - writes dex-db (PostgreSQL/ClickHouse)
  - exposes /pools, /trades, /volume

staking-service:
  - consumes delegation / reward events
  - writes staking-db
  - exposes /pools, /delegators, /rewards

Each service:

  • owns its schema and migration path;
  • can tune its DB for its access patterns;
  • scales independently behind a load balancer.

If one projection falls behind, others can keep up; if you need more capacity for trades but not for balances, you scale only dex-service.

7. Testing and operations

More services means more moving parts, but also more isolation for tests and observability.

7.1 Testing strategy

I like three layers.

Component tests per service. Each service gets tests that feed it synthetic events and inspect its database and emitted events. For ingestion, that means fake node data or canned block streams; for projections, short sequences including reorgs.

Contract tests for events. Treat event schemas as contracts. Producers and consumers both validate against the same Avro/Protobuf/JSON Schema. This catches “producer changed a field, consumers silently broke” early.

End‑to‑end tests for reorgs and lag. Spin up a reduced pipeline (broker + ingestion + a couple of projections), feed it a small chain with a controlled reorg, and assert that projections match the winning branch. Simulate slow consumers and verify lag metrics and alerts behave as expected.

7.2 Observability and operations

Operationally you want:

  • per‑topic consumer lag per consumer group (how far each projection is behind the head);
  • per‑service metrics: throughput, error ratios, DB latency, queue sizes;
  • reorg counters (count, last depth) per chain;
  • clear runbooks for “replay projection X from topic Y”.

From the trenches. On one platform we decided early that “replay from topics” was a supported operation. Every projection treated its DB as cache and the event log as truth. That turned schema changes and bugfixes into operational tasks: reset the projection, replay, watch lag go to zero. It was far less stressful than “surgery on a monolith database in production”.

8. Conclusion

A microservices architecture for blockchain indexers isn’t about chasing buzzwords. It’s a response to concrete constraints:

  • high‑volume, append‑only chain data;
  • multiple independent consumers with different needs;
  • noisy dependencies (nodes, RPC providers);
  • reorgs and protocol evolution.

If you:

  • slice services by responsibility (ingestion, normalisation, projections, APIs);
  • use an event broker as your backbone;
  • apply proven patterns (event‑driven architecture, CQRS, event sourcing, Outbox/Saga);
  • design for replay and reorgs from day zero;

…you end up with an indexer platform that can:

  • keep up with fast chains;
  • support many downstream consumers;
  • survive node failures and reorgs;
  • evolve without “reindex from genesis” every time you add a column.

That’s the architecture I’m aiming for whenever I build or refactor a serious blockchain indexing system.

References and further reading

  • Polygon’s Chain Indexer Framework: Kafka‑based event‑driven pipelines for EVM indexing. (Polygon)
  • Microservices.io patterns for microservice data and collaboration: CQRS, Saga, Outbox. (microservices.io)
  • Event‑driven architecture and microservices with Kafka and Spring Boot. (GeeksforGeeks)
  • CQRS and Event Sourcing guides (Azure Architecture Center, AWS patterns, CQRS articles). (Microsoft Learn)
  • QuickNode and other stream‑based indexer examples for understanding real‑time indexing flows. (Quicknode)