The Ask
We are building a game and we want to also present trade stats to our players. We need to be able to efficiently index and track not just mints, but transfers of nfts in real time.
The Problem
Collections are largely described and configured by Metaplex which extends SPL. SPL is not aware of metaplex and does not mention it, or the first creator or the metaplex collection nft when transferred.
What this means is that there is not an efficient way, to subscribe to all NFTs in a collection, short of first, finding all nfts within the collection, and then independently subscribing to transfers for all of them independently.
For large collections this could result in 10,000 or even 100,000 subscribe calls to the RPC node, and those would need to be re-established every time the link was broken.
This is of course the dreaded N+1 scaling problem. It works fine for a single nft, or even a handful, but it does not work at large scales.
More Backstory Stuff
When I asked initially on the Solana discord, I was referred to Star Atlas as a reference for a company with similar needs and similar scale. However they gate all mints and transfers through their platform, which then makes it of course very easy to track them. We would like our NFTs to be transferrable on any exchange.
One option of course may be to run our own validator and brute force it. But that's not, ideal. There is significant cost and R&D and planning needed to host a validator, and then we need to sort out how to get all that data that we don't care about, filtered so that we aren't pumping excessive , unwanted data into our nft transfer history service.
One option that came to mind was to register an "observer" on an account. All token transfers would be required to mention this observer. But much like the validator , we don't have a lot of cycles to invest in developing and submitting a feature proposal to SPL Token Program just now.
But then I stumbled on this.
I'm seeing some new features on this SPL Token 2022. https://www.youtube.com/watch?v=Yg-k1Tx5PZk that may be at least a start on a pathway to a native solution.
https://docs.rs/spl-token-2022/0.1.0/spl_token_2022/extension/index.html
I'm not seeing many details on this yet but one thing that smells very similar to the 'observable' concept I mentioned is this 'memo_transfer Memo Transfer extension'
I stumbled upon these new program ids while I was rummaging around through the spl web3 source.
https://github.com/solana-labs/solana-program-library/blob/master/token/js/src/constants.ts
export const TOKEN_2022_PROGRAM_ID = new PublicKey('TokenzQdBNbLqP5VEhdkAS6EPFLC1PHnBqCXEpPxuEb');
You can set a "memo_required' .
It seems to be for transfers INTO an account. It's not clear if we can enforce that the receiving account can be configured to require memos.
It appears it will assert that the immediately preceding instruction be a memo, targeting the "memo" program https://github.com/solana-labs/solana-program-library/tree/master/memo
It's not clear, if we could enforce that the memo include a specific account. But this could potentially be a basis for the observer solution.
We could possibly add an additional extension that builds on top of memo requirement that requires mention of any specific address set on the token account's memo offset. And then if we could push it further and require the memo if it's on the token account, rather than the token receive account. Unless that's already covered here some how.
The Question:
My question then is, could this new "require memo" be used to enforce that all transfers of a given token, mention a pre-established observer?
Then we could upgrade the metaplex candy machine, to configure itself, or require itself, as the observer ( required memo account ) on any tokens it adds to a collection.
And BOOM, just as soon as all the exchanges get on board with memos....
UPDATE:
I'm seeing some interesting codes in and around the default_account_state extensions.
It LOOKS Like, we can attach default_account_state to the "mint" account. This will get applied to the token accounts whenever we create them via
process_create_associated_token_account
I believe then, that we could, set the memo requirement as a default. Then every time a transfer is executed, the account created to receive the token will have, by default, the memo requirement set. It would have to be explicitly unset for that account to not be there ( Again this is my current understanding of how this will work, I'm still digging )
This would ensure that the vast majority of transfers ( without some intentional manipulation of the transfer process ), would have memos.
The problem that remains , is that I don't see a way to place any structure requirements or content requirements on the memo. While the memo would be required, it may not contain anything we can actually key off of.
UPDATE #2
Actually... we don't even need the new memo extension. We need the enforced transfer fee extension. Again along with account state defaults. (Actually, I think the transfer fee goes on the mint account rather than the ownership account )
Then all we have to do is listen for every time we get paid. Lookup the transaction, and see if it includes a token transfer.
Yeah that... that might just work.
Again, I think if someone overrides the transfer fee, we'd be in trouble, but there is a new AuthorityType.TransferFeeConfig that makes me think we could assert control over that. :)
When I get time to test this out, if it works, I'll post it as an answer.
Update #3
I've dug into the blockSubscribe + getBlock solution a bit. There are some issues with this approach.
First off, while the rate may not be an issue if your filter is efficient, the network bandwidth for getting every transaction from every block is non trivial. It could be 10s of TBs per month unless I'm mistaken.
Block Subscribe works well enough, because you can filter. You can filter out blocks and even transactions within the block, that you don't care about. You only get to put in one "mentions" filter, but it works to a degree.
However, if, if you are filtering, then you don't know which blocks, if absent from your index, need to be retrieved. "getBlock" doesn't offer the ability to configure it with the same filters, or any filters for that matter.
This means that if you are trying to identify missed blocks and go get them, you don't know if a block was skipped because it was filtered out, or if it was missed. If you get every block that isn't already marked as processed, you'll eliminate the benefit of your blockSubscribe filter and end up pulling all data for nearly every block.
I can possibly use getSignaturesForAddress() but you can't range limit by slot. One option may be, to record transaction signatures along with any blocks you get from your block subscribe. Then you'd use getSignaturesForAddress() to scan between the signatures of two recorded blocks to see if anything was missed.
It's a pretty complicated setup, but it may work.
However, are signatures viable "stopping points" for a range scan? I thought some transactions could be processed concurrently. Wouldn't a transaction signature based range limiter have an undefined or ambiguous or inconsistent stopping point?
Consider the following sequence of transactions, listed from top to bottom in the order that they were validated.
Transaction Xray
Transaction Niner
Transaction Bravo (in parallel with ) Transaction Mike
Transaction India
If I'm scanning "before: 'Mike'" and "until: 'Xray'", will the scan include Bravo? Sometimes? Reliably?
