Private Node drops transactions while solana-test-validator does not

Question

I am trying to run a custom benchmark for Solana and have noticed the following odd (or maybe not) behaviour.

I create a single node cluster by following the instructions in: https://docs.solana.com/running-validator/validator-start. Then, I deploy a program and start sending random transactions to it at a rate that I provide as an argument to my benchmark. I want to send a total of 10000txs at a rate of 160 tps which seems not to be very large.

Problem: After some time, I check the logs and the node starts retrying transactions and eventually drops them after they expire.

The odd thing I notice is that although I miss approximately a 40% of transactions sent, this won't happen if instead of the bootstrap-node.sh, I use the solana-test-validator without any extra arguments, where the transactions are all successfully executed even if use a way bigger tps rate.

The command I execute is the following:

./multinode-demo/bootstrap-validator.sh \
                        --enable-rpc-transaction-history \
                        --enable-cpi-and-log-storage \
                        --gossip-host $1 \
                        --allow-private-addr \
                        --log $LOGS/validator.log & 

Both $LOGS and $1 are guaranteed to be there and correct.

Since I use the same machine for both nodes, I cannot figure out how the node behaves differently in each run. I guess I miss something in my node configuration but cannot tell what it is.

I am using solana version 1.10.24

Also, I notice that when running the benchmark with solana-test-validator, the resources spent are low, despite the fact that all the transactions are executed:

On the contrary, the benchmark with bootstrap-validator.sh consumes way more resources while also dropping a handful of transactions, which is weird:

This is why I believe it has something to do with the node configuration.

Edit: Following I present the code with which I send each request. This code is executed 10000 times.

// The transaction RPC configuration
let RPC_SEND_CONFIG: RpcSendTransactionConfig = RpcSendTransactionConfig {
        skip_preflight: true,
        preflight_commitment: Some(CommitmentLevel::Confirmed),
        encoding: None,
        max_retries: None,
        min_context_slot: None
    };

// Create random triplet to use as instruction data to have different signatures
let random_data = Self::get_random_u8_triple();

// Serialize instruction
let instruction = Instruction::new_with_borsh(driver.program, &random_data, vec![]);

// Get recent blockhash
let recent_blockhash = driver.connection.get_latest_blockhash().unwrap();

// Create and sign  the transaction
let read_tx = Transaction::new_signed_with_payer(
    &[instruction],
    Some(&driver.fee_payer.pubkey()),
    &[&driver.fee_payer],
    recent_blockhash,
);

let sig = driver.connection.send_transaction_with_config(&read_tx, RPC_SEND_CONFIG).unwrap()

The driver struct contains the RPC connection object to communicate with the node. This is created as follows:

// url is the node's endpoint
let connection = RpcClient::new_with_commitment(url, CommitmentConfig::confirmed());

Disclaimer: I know my server specs are way lower than the recommended, however this seems not to be the case here.

Please let me know if I need to provide any additional information on this topic. Any help will be appreciated, thank you very much.

Answer 1

After some searching, I think I noticed what went wrong in my case.

The problem was that the bootstrap-validator.sh script did not run the cli command solana-validator. Instead, it ran a cargo run --bin solana-validator -- <ARGS> command which seems to retry incoming transactions when throughput is increased. I managed to solve my problem by replacing the $program variable in bootstrap-validator.sh with solana-validator.

Also, I noticed that when running the cargo command, all of my cores reach a 100% even when idle, while the solana-validator uses far less resources.

Could anyone explain why the cargo run command creates this issue?