Suppose I have a DApp that lets users create spl tokens. The frontend communicates with the spl token program to create the tokens. The main page shows all tokens created through the DApp (not all spl tokens) and updates live as new tokens are created. From what I understand fetching and searching through all the token accounts using the rpc method getProgramAccounts doesn't scale.

I am thinking of having a separate database that stores tokens created through the DApp to process these search requests. However since I now have a distributed system, how should I go about ensuring that the data in the DB is consistent with the on-chain data? A naive implementation -- call a Rest API that persists the token in the db, when a token is created -- is not fault-tolerant, because if the DB crashes, the DB and the on-chain data will be permanently inconsistent.

A solution that is maybe more fault-tolerant involves using Kafka for its durability guarantees. When a token is created we would publish a TokenCreated event to a Kafka topic, and a separate service reads from the topic and persists the tokens in the DB. Technically writing to Kafka could also fail, but I think it's not as likely...

In general, how does the architecture of a typical DApp that works with large amounts of data look like? Is there a better solution than replicating the data off-chain in its own database?

Anyways, I was wondering if anyone faced a similar problem and has any recommendations

1 Answer 1


A solution here could be using webhooks from a provider like Helius, and streaming events from a given program to an off-chain DB. This once again isn't perfect, but gives good guarantees of consistent data, and doesn't require lots of infra to keep up with.

A larger app with tons of users might opt to run a geyser plugin and stream events to something like Postgres, however for most people this is overkill

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.