2

I am trying to run a custom benchmark for Solana and have noticed the following odd (or maybe not) behaviour.

I create a single node cluster by following the instructions in: https://docs.solana.com/running-validator/validator-start. Then, I deploy a program and start sending random transactions to it at a rate that I provide as an argument to my benchmark. I want to send a total of 10000txs at a rate of 160 tps which seems not to be very large.

Problem: After some time, I check the logs and the node starts retrying transactions and eventually drops them after they expire.

enter image description here

The odd thing I notice is that although I miss approximately a 40% of transactions sent, this won't happen if instead of the bootstrap-node.sh, I use the solana-test-validator without any extra arguments, where the transactions are all successfully executed even if use a way bigger tps rate.

The command I execute is the following:

./multinode-demo/bootstrap-validator.sh \
                        --enable-rpc-transaction-history \
                        --enable-cpi-and-log-storage \
                        --gossip-host $1 \
                        --allow-private-addr \
                        --log $LOGS/validator.log & 

Both $LOGS and $1 are guaranteed to be there and correct.

Since I use the same machine for both nodes, I cannot figure out how the node behaves differently in each run. I guess I miss something in my node configuration but cannot tell what it is.

I am using solana version 1.10.24

Also, I notice that when running the benchmark with solana-test-validator, the resources spent are low, despite the fact that all the transactions are executed:

enter image description here

On the contrary, the benchmark with bootstrap-validator.sh consumes way more resources while also dropping a handful of transactions, which is weird:

enter image description here

This is why I believe it has something to do with the node configuration.

Edit: Following I present the code with which I send each request. This code is executed 10000 times.

// The transaction RPC configuration
let RPC_SEND_CONFIG: RpcSendTransactionConfig = RpcSendTransactionConfig {
        skip_preflight: true,
        preflight_commitment: Some(CommitmentLevel::Confirmed),
        encoding: None,
        max_retries: None,
        min_context_slot: None
    };

// Create random triplet to use as instruction data to have different signatures
let random_data = Self::get_random_u8_triple();

// Serialize instruction
let instruction = Instruction::new_with_borsh(driver.program, &random_data, vec![]);

// Get recent blockhash
let recent_blockhash = driver.connection.get_latest_blockhash().unwrap();

// Create and sign  the transaction
let read_tx = Transaction::new_signed_with_payer(
    &[instruction],
    Some(&driver.fee_payer.pubkey()),
    &[&driver.fee_payer],
    recent_blockhash,
);

let sig = driver.connection.send_transaction_with_config(&read_tx, RPC_SEND_CONFIG).unwrap()

The driver struct contains the RPC connection object to communicate with the node. This is created as follows:

// url is the node's endpoint
let connection = RpcClient::new_with_commitment(url, CommitmentConfig::confirmed());

Disclaimer: I know my server specs are way lower than the recommended, however this seems not to be the case here.

Please let me know if I need to provide any additional information on this topic. Any help will be appreciated, thank you very much.

5
  • please supply the client code used to send the transactions
    – trent.sol
    Jul 28, 2022 at 17:55
  • @trent.sol I added the Rust code above. However, I am positive that the code is correct since it works with the solana-test-validator.
    – al96
    Jul 28, 2022 at 19:07
  • First, solana-test-validator isn't a real full fledged validator but a stubbed out validator to allow developers to test on a lightweight setup so resource usage comparison isn't very relevant. I think modifying your example to use the memo program would allow people to attempt to reproduce with less code/complexity
    – Arowana
    Jul 29, 2022 at 1:17
  • @Arowana I can understand what you tell me for the solana-test-validator and it makes sense why it uses less resources. However, this makes me wonder how it is possible for the solana-test-validator to successfully execute all the transactions since as you say is not a full validator. I guess I should somehow modify the configurations of my actual validator, however I cannot find any sources on how to do so. Do you think it is acceptable to check the pending transaction blockhashes and manually retry them when it expires? I will check the memo program since I have never used it before. Thanks!
    – al96
    Jul 29, 2022 at 10:38
  • @trent.sol I think I figured out the problem, could you have a look on my answer and provide any comments? Thanks
    – al96
    Aug 2, 2022 at 12:51

1 Answer 1

2

After some searching, I think I noticed what went wrong in my case.

The problem was that the bootstrap-validator.sh script did not run the cli command solana-validator. Instead, it ran a cargo run --bin solana-validator -- <ARGS> command which seems to retry incoming transactions when throughput is increased. I managed to solve my problem by replacing the $program variable in bootstrap-validator.sh with solana-validator.

Also, I noticed that when running the cargo command, all of my cores reach a 100% even when idle, while the solana-validator uses far less resources.

Could anyone explain why the cargo run command creates this issue?

3
  • cargo run ... just builds the bin from source and executes that instead of searching for it in PATH. if you're observing a difference in behavior then it is between whatever version of the code base that you have checked out and whatever version of the solana cli tools you have installed
    – trent.sol
    Aug 2, 2022 at 16:13
  • actually I am running same versions for both source and cli tools
    – al96
    Aug 2, 2022 at 16:25
  • assuming you got your cli tools from the labs binary release repository (which are release mode bins), is cargo run ... passing --release? it will run a debug build by default
    – trent.sol
    Aug 2, 2022 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.