5

The Ask

We are building a game and we want to also present trade stats to our players. We need to be able to efficiently index and track not just mints, but transfers of nfts in real time.

The Problem

Collections are largely described and configured by Metaplex which extends SPL. SPL is not aware of metaplex and does not mention it, or the first creator or the metaplex collection nft when transferred.

What this means is that there is not an efficient way, to subscribe to all NFTs in a collection, short of first, finding all nfts within the collection, and then independently subscribing to transfers for all of them independently.

For large collections this could result in 10,000 or even 100,000 subscribe calls to the RPC node, and those would need to be re-established every time the link was broken.

This is of course the dreaded N+1 scaling problem. It works fine for a single nft, or even a handful, but it does not work at large scales.

More Backstory Stuff

When I asked initially on the Solana discord, I was referred to Star Atlas as a reference for a company with similar needs and similar scale. However they gate all mints and transfers through their platform, which then makes it of course very easy to track them. We would like our NFTs to be transferrable on any exchange.

One option of course may be to run our own validator and brute force it. But that's not, ideal. There is significant cost and R&D and planning needed to host a validator, and then we need to sort out how to get all that data that we don't care about, filtered so that we aren't pumping excessive , unwanted data into our nft transfer history service.

One option that came to mind was to register an "observer" on an account. All token transfers would be required to mention this observer. But much like the validator , we don't have a lot of cycles to invest in developing and submitting a feature proposal to SPL Token Program just now.

But then I stumbled on this.

I'm seeing some new features on this SPL Token 2022. https://www.youtube.com/watch?v=Yg-k1Tx5PZk that may be at least a start on a pathway to a native solution.

https://docs.rs/spl-token-2022/0.1.0/spl_token_2022/extension/index.html

I'm not seeing many details on this yet but one thing that smells very similar to the 'observable' concept I mentioned is this 'memo_transfer Memo Transfer extension'

I stumbled upon these new program ids while I was rummaging around through the spl web3 source.

https://github.com/solana-labs/solana-program-library/blob/master/token/js/src/constants.ts

export const TOKEN_2022_PROGRAM_ID = new PublicKey('TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb');

You can set a "memo_required' .

https://github.com/solana-labs/solana-program-library/blob/master/token/program-2022/src/extension/memo_transfer/processor.rs

It seems to be for transfers INTO an account. It's not clear if we can enforce that the receiving account can be configured to require memos.

It appears it will assert that the immediately preceding instruction be a memo, targeting the "memo" program https://github.com/solana-labs/solana-program-library/tree/master/memo

It's not clear, if we could enforce that the memo include a specific account. But this could potentially be a basis for the observer solution.

We could possibly add an additional extension that builds on top of memo requirement that requires mention of any specific address set on the token account's memo offset. And then if we could push it further and require the memo if it's on the token account, rather than the token receive account. Unless that's already covered here some how.

The Question:

My question then is, could this new "require memo" be used to enforce that all transfers of a given token, mention a pre-established observer?

Then we could upgrade the metaplex candy machine, to configure itself, or require itself, as the observer ( required memo account ) on any tokens it adds to a collection.

And BOOM, just as soon as all the exchanges get on board with memos....

UPDATE:

I'm seeing some interesting codes in and around the default_account_state extensions.

It LOOKS Like, we can attach default_account_state to the "mint" account. This will get applied to the token accounts whenever we create them via

process_create_associated_token_account

I believe then, that we could, set the memo requirement as a default. Then every time a transfer is executed, the account created to receive the token will have, by default, the memo requirement set. It would have to be explicitly unset for that account to not be there ( Again this is my current understanding of how this will work, I'm still digging )

This would ensure that the vast majority of transfers ( without some intentional manipulation of the transfer process ), would have memos.

The problem that remains , is that I don't see a way to place any structure requirements or content requirements on the memo. While the memo would be required, it may not contain anything we can actually key off of.

UPDATE #2

Actually... we don't even need the new memo extension. We need the enforced transfer fee extension. Again along with account state defaults. (Actually, I think the transfer fee goes on the mint account rather than the ownership account )

Then all we have to do is listen for every time we get paid. Lookup the transaction, and see if it includes a token transfer.

Yeah that... that might just work.

Again, I think if someone overrides the transfer fee, we'd be in trouble, but there is a new AuthorityType.TransferFeeConfig that makes me think we could assert control over that. :)

When I get time to test this out, if it works, I'll post it as an answer.

Update #3

I've dug into the blockSubscribe + getBlock solution a bit. There are some issues with this approach.

First off, while the rate may not be an issue if your filter is efficient, the network bandwidth for getting every transaction from every block is non trivial. It could be 10s of TBs per month unless I'm mistaken.

Block Subscribe works well enough, because you can filter. You can filter out blocks and even transactions within the block, that you don't care about. You only get to put in one "mentions" filter, but it works to a degree.

However, if, if you are filtering, then you don't know which blocks, if absent from your index, need to be retrieved. "getBlock" doesn't offer the ability to configure it with the same filters, or any filters for that matter.

This means that if you are trying to identify missed blocks and go get them, you don't know if a block was skipped because it was filtered out, or if it was missed. If you get every block that isn't already marked as processed, you'll eliminate the benefit of your blockSubscribe filter and end up pulling all data for nearly every block.

I can possibly use getSignaturesForAddress() but you can't range limit by slot. One option may be, to record transaction signatures along with any blocks you get from your block subscribe. Then you'd use getSignaturesForAddress() to scan between the signatures of two recorded blocks to see if anything was missed.

It's a pretty complicated setup, but it may work.

However, are signatures viable "stopping points" for a range scan? I thought some transactions could be processed concurrently. Wouldn't a transaction signature based range limiter have an undefined or ambiguous or inconsistent stopping point?

Consider the following sequence of transactions, listed from top to bottom in the order that they were validated.

Transaction Xray
Transaction Niner
Transaction Bravo (in parallel with ) Transaction Mike
Transaction India

If I'm scanning "before: 'Mike'" and "until: 'Xray'", will the scan include Bravo? Sometimes? Reliably?

3 Answers 3

4

I think the 'inefficiency' that you're trying to avoid is not as bad as you think it is, as long as you don't use individual account subscriptions.

A simple and reasonably efficient solution:

  • Build a map of your 100,000 accounts
  • Subscribe to all blocks as they are created out of Solana (can use getBlock or blockSubscribe if its supported)
  • Scan each transaction in the block for ones referencing your accounts (would be a vanishingly small percentage of all transactions)
  • Then and only then, parse the corresponding transaction for system transfers

If a block is produced every 400ms, you have a ton of time to do the above work, which even if done inefficiently should be complete in a few ms.

The main thing to maintain will be the high watermark for blocks processed so you don't miss any, but this would occur regardless of the solution.

Your solution sounds 'elegant' but very complex. Suggest trying to quite simple solution above and seeing if it doesn't work for your needs. I'd be very surprised. This is close to how exchanges manage deposits today, see this: https://docs.solana.com/integrations/exchange#listening-for-deposits

0
1

RPC isn't suited for something like this at scale. You'd want to build a Geyser Plugin that streams account updates, filters and builds a database optimized for your application

2
  • 1
    Can we use a geyser plugin without running our own validator?
    – itsfarseen
    Aug 23, 2022 at 6:32
  • Late to the party but... you can user gayser-as-a-service providers like Helius and Shyft instead of maintaining and paying expensive infra necessary to run a validator. Jul 14, 2023 at 3:26
0

Now that Solana ecosystem has matured and gave us the necessary tooling... the correct answer is to rely on services such as Helius and their webhooks for pushing on-chain events whenever you need them pushed at.

https://docs.helius.xyz/webhooks/what-are-webhooks

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.