1

Im trying to store, lets say 10,000 Public Keys inside a vector in a Solana Account. I have created the account with 500KB on devnet (8q8rTUeW9Wn3seaYiCf7HU6bpimzQTt7zEDe3RgVteBc). In theory for each Pubkey I will need 32 bytes of space, so for 10k pubkeys I will need 320,000 bytes, which is less than the 500KB of my account. When I test this, Solana stores just more or less 500 pubkeys in the vector and after that number is reached i start getting this error: "Error: memory allocation failed, out of memory". I made some research and it looks like although the account space is 500KB, the program stack space and heap is 4kb and 32kb, and this may be limiting the space.

Here it is my code: Its just a for loop to push the pubkeys to the account data (as I am testing, the pubkey is always the same, and is the payer pubkey)

#[account]
#[derive(Default)]
pub struct ListPDA {
    pub pubkeys: Vec<Pubkey>,
}

#[derive(Accounts)]
pub struct TestAcc<'info> {
    pub rent: Sysvar<'info, Rent>,
    pub system_program: Program<'info, System>,
    /// CHECK: Just testing
    #[account(mut)]
    pub payer: Signer<'info>,
    /// CHECK: Just testing
    #[account(mut)]
    store_data: Account<'info, ListPDA>,
}

declare_id!("AS8VZPnX3SfwQCCAj26WSt88DLWJriUMAjthV1xcbyKD");

#[program]
pub mod solana_devnet_test_3 {
    use super::*;

    pub fn test_acc(ctx: Context<TestAcc>) -> Result<()> {

        // check length before pushing new pubkeys
        msg!(
            "length1...{}",
            ctx.accounts.store_data.pubkeys.len()
        );

        for i in 0..100 {
            ctx.accounts
                .store_data
                .pubkeys
                .push(ctx.accounts.payer.key());
        }

        // check length after pushing new pubkeys
        msg!(
            "length2...{}",
            ctx.accounts.store_data.pubkeys.len()
        );

        Ok(())
    }
}

I have tried using zero_copy on the ListPDA struct but couldn't make it work, maybe it is not possible with a vector..

Any help will be really appreciated, thanks.

1

2 Answers 2

2

Given the transaction size limit, there's no way you're going to be storing 10k pubkeys at once anyway. I'd suggest using multiple transactions to load a subset into different regions of the account space. This is how the program loader program's deploy functionality works.

More likely, you're holding it wrong and really want something like a merkle distributor

3
  • 1
    Hi, thanks for the answer. I am not trying to store all 10k pubkeys at once, I'm storing 100 each time. For the first 500 everything goes fine, but then I start getting this error: "Error: memory allocation failed, out of memory" Jul 14, 2022 at 18:40
  • 1
    Using Vec::push() like that requires allocating the entire account's state. Instead you can use Vec::copy_from_slice() directly into the mutably borrowed account data field. The loader program does it like this
    – trent.sol
    Jul 14, 2022 at 19:08
  • @trent.sol: Can you explain the difference between the situation using Vec::push() and Vec:: copy_from_slice()? Why the former requires allocating the entire account's state, but not the latter? Jun 9, 2023 at 20:54
1

First of all I agree with trent's answer that you probably don't actually want to store 10k public keys in one account. But I'll continue assuming you do.

As far as I can tell you have two options:

Anchor ZeroCopy: To use zero_copy your struct has to implement the Pod trait. This is so you can map from struct access straight to byte access in the serialized account without deserializing the whole thing. I would suggest using a [Pubkey; N] (though you will have to manage the expandable array aspect yourself). Vec doesn't and can't implement Pod .

Manage the Memory Yourself: You could write your own serialization/deserialization solution that only reads and writes from the region of memory you are interested in. If you use Borsh Serialization, a Vec is stored as a u32 specifying the length followed by the packed elements. You could directly access the 9th-12th bytes (since the first 8 will probably be a discriminator) to get the current length of the vector, then copy in your Pubkeys to the region of memory from [13 + start_length * 32 : 13 + (start_length + number_to_insert) * 32], then edit the length field.

If you think about it these solutions are basically equivalent. It's just that using ZeroCopy will manage some of the bit access math for you.

2
  • 1
    Why he shouldn't store 10k pubkey in an account? What is wrong with this approach?
    – curiosity
    Jul 15, 2022 at 4:19
  • 2
    Just that there is probably a better approach. If you're maintaining an append-only list, you could use a series of smaller accounts that can fit into memory. If you're maintaining a set to do inclusion testing on, I'd create a tiny account for each member. But maybe there's a use-case for this I'm not thinking of, I don't know OP's plan.
    – Ellie High
    Jul 15, 2022 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.