Do you know why this modification of the banking-bench results in poor performance?

Question

Long story short. I modified the banking-bench code to allow benchmarking the banking stage with different levels of contention via the number of accounts used and performance has significantly degraded

The idea for account-based contention is simple:
create a vector of public keys.
Randomly generate a transaction between any two of the keys.

By varying the number of public keys in the vector, you can modify contention. 2 public keys is completely sequential, 10 is less contentious and so on.

I've written the implementation of this idea and added it alongside some handling logic to the original program. The main code block is:

   if is_accounts {
        let accounts: Vec<pubkey::Pubkey> = (0..num_accounts).map(|_| pubkey::new_rand()).collect();
        (0..total_num_transactions)
            .into_par_iter()
            .map(|_| {
                let mut rng = thread_rng();
                let send_pubkey = accounts.choose(&mut rng).unwrap();
                let recv_pubkey = loop {
                    let candidate = accounts.choose(&mut rng).unwrap();
                    if candidate != send_pubkey {
                        break candidate;
                    }
                };

                let compute_unit_price = 1;
                let mut new = make_transfer_transaction_with_compute_unit_price(
                    &payer_key,
                    &to_pubkey,
                    1,
                    hash,
                    compute_unit_price,
                );
                //You can have duplicate transactions in a batch, not just duplicate signatures.
                //And considering how large signature is, the odds of a collison are sufficiently small for this experiment.
                let sig: [u8; 64] = std::array::from_fn(|_| thread_rng().gen::<u8>());
                new.message.account_keys[0] = *send_pubkey;
                new.message.account_keys[1] = *recv_pubkey;
                new.signatures = vec![Signature::from(sig)];
                new
            })
            .collect()
    }

The problem now is that performance compiling the new code dropped TPS (as low as 5 TPS when 2 accounts were used). In the existing implementation, the same level of contention has over 70k TPS.

The difference is too staggering and is consistent until the number of accounts is very large. I can't spot any errors in the code that would cause that behavior and would love to hear your opinions on if/what the problem is and how to fix it.

Link to modified program

Answer 1

I recently discovered the solution to this problem. Long story short the transactions were getting filtered out during banking because their message hashes were being detected as duplicates.

I fixed this by modifying the amount of lamports each transactioned performed. The implementation uses the index of the iterator as seen below:

    //add accounts based contention logic
if is_accounts {
    let accounts: Vec<pubkey::Pubkey> = (0..num_accounts).map(|_| pubkey::new_rand()).collect();
    (0..total_num_transactions)
        .into_par_iter()
        .map(|i| { //replaced underscore with i to allow reuse of the value
            let mut rng = thread_rng();
            let send_pubkey = accounts.choose(&mut rng).unwrap();
            let recv_pubkey = loop {
                let candidate = accounts.choose(&mut rng).unwrap();
                if candidate != send_pubkey {
                    break candidate;
                }
            };

            let compute_unit_price = 1;
            let mut new = make_transfer_transaction_with_compute_unit_price(
                &payer_key,
                &to_pubkey,
                (i+1).try_into().unwrap(), //changed from 1 lamport to a value that's different per packet because of duplicated txs
                hash, //check signature status in bank.
                compute_unit_price,
            );
            
            let sig: [u8; 64] = std::array::from_fn(|_| thread_rng().gen::<u8>());
            new.message.account_keys[0] = *send_pubkey;
            new.message.account_keys[1] = *recv_pubkey;
            new.signatures = vec![Signature::from(sig)];
            new
        })
        .collect()
}

The code works great now!