How to actually store 10k Pubkeys in a Solana Account

Question

Im trying to store, lets say 10,000 Public Keys inside a vector in a Solana Account. I have created the account with 500KB on devnet (8q8rTUeW9Wn3seaYiCf7HU6bpimzQTt7zEDe3RgVteBc). In theory for each Pubkey I will need 32 bytes of space, so for 10k pubkeys I will need 320,000 bytes, which is less than the 500KB of my account. When I test this, Solana stores just more or less 500 pubkeys in the vector and after that number is reached i start getting this error: "Error: memory allocation failed, out of memory". I made some research and it looks like although the account space is 500KB, the program stack space and heap is 4kb and 32kb, and this may be limiting the space.

Here it is my code: Its just a for loop to push the pubkeys to the account data (as I am testing, the pubkey is always the same, and is the payer pubkey)

#[account]
#[derive(Default)]
pub struct ListPDA {
    pub pubkeys: Vec<Pubkey>,
}

#[derive(Accounts)]
pub struct TestAcc<'info> {
    pub rent: Sysvar<'info, Rent>,
    pub system_program: Program<'info, System>,
    /// CHECK: Just testing
    #[account(mut)]
    pub payer: Signer<'info>,
    /// CHECK: Just testing
    #[account(mut)]
    store_data: Account<'info, ListPDA>,
}

declare_id!("AS8VZPnX3SfwQCCAj26WSt88DLWJriUMAjthV1xcbyKD");

#[program]
pub mod solana_devnet_test_3 {
    use super::*;

    pub fn test_acc(ctx: Context<TestAcc>) -> Result<()> {

        // check length before pushing new pubkeys
        msg!(
            "length1...{}",
            ctx.accounts.store_data.pubkeys.len()
        );

        for i in 0..100 {
            ctx.accounts
                .store_data
                .pubkeys
                .push(ctx.accounts.payer.key());
        }

        // check length after pushing new pubkeys
        msg!(
            "length2...{}",
            ctx.accounts.store_data.pubkeys.len()
        );

        Ok(())
    }
}

I have tried using zero_copy on the ListPDA struct but couldn't make it work, maybe it is not possible with a vector..

Any help will be really appreciated, thanks.

Answer 1

Given the transaction size limit, there's no way you're going to be storing 10k pubkeys at once anyway. I'd suggest using multiple transactions to load a subset into different regions of the account space. This is how the program loader program's deploy functionality works.

More likely, you're holding it wrong and really want something like a merkle distributor

Answer 2

First of all I agree with trent's answer that you probably don't actually want to store 10k public keys in one account. But I'll continue assuming you do.

As far as I can tell you have two options:

Anchor ZeroCopy: To use zero_copy your struct has to implement the Pod trait. This is so you can map from struct access straight to byte access in the serialized account without deserializing the whole thing. I would suggest using a [Pubkey; N] (though you will have to manage the expandable array aspect yourself). Vec doesn't and can't implement Pod .

Manage the Memory Yourself: You could write your own serialization/deserialization solution that only reads and writes from the region of memory you are interested in. If you use Borsh Serialization, a Vec is stored as a u32 specifying the length followed by the packed elements. You could directly access the 9th-12th bytes (since the first 8 will probably be a discriminator) to get the current length of the vector, then copy in your Pubkeys to the region of memory from [13 + start_length * 32 : 13 + (start_length + number_to_insert) * 32], then edit the length field.

If you think about it these solutions are basically equivalent. It's just that using ZeroCopy will manage some of the bit access math for you.