Getting Error: memory allocation failed whenever serializing the data to insert into the account

Question

I am developing a program on Solana and I have to store around 1 MB of state into the account (Given accounts can store up to 10 MB of data) however, whenever I try to serialize that data to add it to the account, right at the serialization operation it fails and spits out the Error: memory allocation failed, out of memory which I believe is happening because of the 32kb limit for bpf programs. I tried several serializers Borsh, Bincode, and serde_json nothing seems to work. Any help would be appreciated.

The exact error messages are included below:

EDIT: I am not using native and have written my program using the solana program library.

  Program log: Error: memory allocation failed, out of memory
  Program 9Sn8j5ZRtGqUM14WQG9KLf5QBgrSCnJL3pWEKrVrLy54 consumed 365701 of 1400000 compute units
  Program failed to complete: BPF program panicked
  Program 9Sn8j5ZRtGqUM14WQG9KLf5QBgrSCnJL3pWEKrVrLy54 failed: Program failed to complete
    

Answer 1

looks like you would need to use a zero copy account and then not load the whole account into memory but instead only work with the memory and then you can add around one kb per transaction. (The transaction size limit is 1232bytes) Basically in anchor you need to add zero copy to your account macro like so for example:

    #[account(zero_copy)]
    #[repr(packed)]
    pub struct DataHolder {
        // 40952 = 40960 - 8 (account desciminator)
        pub long_string: [u8; 40952],
    }

and then use AccountLoader instead of Account.

    #[account(
        mut,
        seeds = [b"level"],
        bump,
    )]
    pub game_data_account: AccountLoader<'info, GameDataAccount>,

It doesn't need to be a byte array any other struct works as well.

Here is another example:

#[account(zero_copy(unsafe))]
#[repr(packed)]
#[derive(Default)]
pub struct GameDataAccount {
    board: [[Tile; BOARD_SIZE_X]; BOARD_SIZE_Y],
    action_id: u64,
}

#[zero_copy(unsafe)]
#[repr(packed)]
#[derive(Default)]
pub struct Tile {
    player: Pubkey,      // 32
    state: u8,           // 1
    health: u64,         // 8
    damage: u64,         // 8
    range: u16,          // 2
    collect_reward: u64, // 8
    avatar: Pubkey,      // 32 used in the client to display the avatar
    look_direction: u8,  // 1 (Up, right, down, left)
    ship_level: u16,     // 2
    start_health: u64,   // 8 (used to calculate the length of the health bar in the client)
}

This is from an example game: https://github.com/solana-developers/solana-game-examples/blob/main/seven-seas/program/programs/seven-seas/src/state/game.rs

You can realloc your account to be bigger than 10kb up to 10mb like so for example:

    #[derive(Accounts)]
    #[instruction(len: u16)]
    pub struct IncreaseZeroCopy<'info> {
        #[account(mut, 
            realloc = len as usize, 
            realloc::zero = true, 
            realloc::payer=signer)]
        pub data_holder: AccountLoader<'info, DataHolder>,
        #[account(mut)]
        pub signer: Signer<'info>,
        #[account(address = system_program::ID)]
        pub system_program: Program<'info, System>,
    }

Here is some code that you can look at on how to use zero copy accounts: https://github.com/solana-developers/anchor-zero-copy-example