How can one paginate large number of accounts fetch on solana?

Question

Like fetching all candy machines using get .all() api of anchor ? How can one paginate the data or we can't until all accounts data is fetched ?

Answer 1

You can achieve it by using configuration object key named 'dataSlice'.

const conn = new Connection('https://xxx'); // mainnet-beta connection
const accountInfos = await conn.getAccountInfo(publicKey, { dataSlice: { offset:5, length: 10 } });

You can then get 50~60th account infos from data accounts.

Answer 2

here is a great explanation of paginating accounts by loris leiva that also happens to use candymachine as an example:

https://lorisleiva.com/paginating-and-ordering-accounts-in-solana

Content from article below:

The key is to pre-fetch the accounts once without any data by providing a dataSlice parameter with a length of zero.

Pre-fetching accounts without data will allow us to scan all the accounts we need and paginate them by fetching their data page by page.

const accounts = await connection.getProgramAccounts(candyMachineV2Program, {
    dataSlice: { offset: 0, length: 0 }, // Fetch without any data.
    filters: [
        { memcmp: { offset: 0, bytes: bs58.encode(candyMachineDiscriminator) } }, // Ensure it's a CandyMachine account.
    ],
})

Then use the getMultipleAccounts RPC method to fetch all of the accounts within the page.

const accountPublicKeys = accounts.map(account => account.pubkey)

const getPage = async (page, perPage) => {
    const paginatedPublicKeys = accountPublicKeys.slice(
        (page - 1) * perPage,
        page * perPage,
    );

    if (paginatedPublicKeys.length === 0) {
        return [];
    }

    const accountsWithData = await connection.getMultipleAccountsInfo(paginatedPublicKeys);

    return accountsWithData;
}