Suppose I have a DApp that lets users create spl tokens. The frontend communicates with the spl token program to create the tokens. The main page shows all tokens created through the DApp (not all spl tokens) and updates live as new tokens are created. From what I understand fetching and searching through all the token accounts using the rpc method getProgramAccounts
doesn't scale.
I am thinking of having a separate database that stores tokens created through the DApp to process these search requests. However since I now have a distributed system, how should I go about ensuring that the data in the DB is consistent with the on-chain data? A naive implementation -- call a Rest API that persists the token in the db, when a token is created -- is not fault-tolerant, because if the DB crashes, the DB and the on-chain data will be permanently inconsistent.
A solution that is maybe more fault-tolerant involves using Kafka for its durability guarantees. When a token is created we would publish a TokenCreated event to a Kafka topic, and a separate service reads from the topic and persists the tokens in the DB. Technically writing to Kafka could also fail, but I think it's not as likely...
In general, how does the architecture of a typical DApp that works with large amounts of data look like? Is there a better solution than replicating the data off-chain in its own database?
Anyways, I was wondering if anyone faced a similar problem and has any recommendations