bench-tps Looping verifications

Question

I setup a private chain cluster(one validator node and one rpc node). I want to run benchmark on other host, but I always get Looping verifications. Anyone know why? Did I miss something?

bench-tps

solana-bench-tps \
    --url http://<<RPC_NODE_IP>>:8111 \
    --faucet <<RPC_NODE_IP>>:9900 \
    --entrypoint <<VALIDATOR_NODE_IP>>:8200 \
    --duration 90 \
    --tx-count 50000 \
    --thread-batch-sleep-ms 0 \
    --bind-address 0.0.0.0 \
    --client-node-id /opt/benchmark/validator-keypair.json \
    --identity "/opt/benchmark/benchmark-keypair.json" \
    --threads 10 \
    --keypair-multiplier 2 \
    --use-rpc-client

log

[2024-05-20T07:53:35.542034433Z INFO  solana_bench_tps] Connecting to the cluster
[2024-05-20T07:53:35.548441362Z INFO  solana_bench_tps] Stake for specified client_node_id: 999977696, total stake: 999977696
[2024-05-20T07:53:35.553183295Z INFO  solana_bench_tps::bench] Creating 100000 keypairs...
[2024-05-20T07:53:38.346094971Z INFO  solana_bench_tps::bench] Get lamports...
[2024-05-20T07:53:38.347576883Z INFO  solana_bench_tps::bench] Funding keypair balance: 349528046643354 max_fee: 50 lamports_per_account: 1000008704 extra: 87381 total: 349528046634650
[2024-05-20T07:53:38.348741455Z INFO  solana_bench_tps::send_batch] transferring 87381761656474 each to 4 accounts in 1 txs
[2024-05-20T07:53:39.350144114Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:39.350168815Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:39.450763883Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:39.450782933Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:39.551390782Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:39.551410882Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:39.651495996Z INFO  solana_bench_tps::send_batch]  retrying 87381761656474 each to 4 accounts in 1 txs
[2024-05-20T07:53:40.653445400Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:40.653468371Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:40.754048919Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:40.754068549Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:40.854650893Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:40.854672094Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:40.954758668Z INFO  solana_bench_tps::send_batch]  retrying 87381761656474 each to 4 accounts in 1 txs
[2024-05-20T07:53:41.956277562Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:41.956300493Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:42.056884584Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:42.056901875Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:42.157516659Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:42.157540090Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:42.257615056Z INFO  solana_bench_tps::send_batch]  retrying 87381761656474 each to 4 accounts in 1 txs
[2024-05-20T07:53:43.259302268Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:43.259324489Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:43.359939859Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:43.359961750Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:43.460566031Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:43.460585851Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:43.560675141Z INFO  solana_bench_tps::send_batch]  retrying 87381761656474 each to 4 accounts in 1 txs
[2024-05-20T07:53:44.562291766Z INFO  solana_bench_tps::send_batch] Looping verifications
[2024-05-20T07:53:44.562313857Z INFO  solana_bench_tps::send_batch] Verifying transfers... 0 remaining, 0 verified, 1 failures
[2024-05-20T07:53:44.662904616Z INFO  solana_bench_tps::send_batch] Looping verifications

validator node

solana-validator \
    --ledger /media/ledger \
    --accounts /media/ledger/accounts \
    --snapshots /media/ledger/snapshots \
    --limit-ledger-size 50000000 \
    --rpc-port 8111 \
    --public-rpc-address <<VALIDATOR_NODE_IP>>:8111 \
    --gossip-port 8200 \
    --gossip-host <<VALIDATOR_NODE_IP>> \
    --dynamic-port-range 8201-8300 \
    --full-snapshot-interval-slots 5000 \
    --incremental-snapshot-interval-slots 100 \
    --log /media/ledger/validator.log \
    --allow-private-addr \
    --enable-cpi-and-log-storage \
    --enable-rpc-transaction-history \
    --full-rpc-api \
    --rpc-faucet-address 127.0.0.1:9900 \
    --identity /opt/svm/secrets/identity-keypair.json \
    --vote-account /opt/svm/secrets/vote-keypair.json \
    --disable-accounts-disk-index \
    --no-wait-for-vote-to-start-leader \
    --no-poh-speed-test \
    --no-snapshot-fetch \
    --rpc-niceness-adjustment=-100 \
    --tpu-enable-udp

rpc node

solana-validator \
    --ledger /media/ledger \
    --accounts /media/ledger/accounts \
    --snapshots /media/ledger/snapshots \
    --limit-ledger-size 50000000 \
    --rpc-port 8111 \
    --public-rpc-address <<RPC_NODE_IP>>:8111 \
    --gossip-port 8200 \
    --gossip-host <<RPC_NODE_IP>>\
    --dynamic-port-range 8201-8300 \
    --full-snapshot-interval-slots 5000 \
    --incremental-snapshot-interval-slots 100 \
    --log /media/ledger/validator.log \
    --allow-private-addr \
    --enable-cpi-and-log-storage \
    --enable-rpc-transaction-history \
    --full-rpc-api \
    --entrypoint <<VALIDATOR_IP>>:8200 \
    --known-validator <<VALIDATOR_PUBLIC_KEY>> \
    --rpc-faucet-address 127.0.0.1:9900 \
    --expected-genesis-hash 2DS9zUTsRb4QHCHGky6dHz2CJMWGAvMoBDBiYY9RgKbi \
    --identity /opt/svm/secrets/rpc-keypair.json \
    --snapshot-packager-niceness-adjustment=-1 \
    --no-voting

Answer 1

You might be providing too many parameters to bench-tps. I tried locally with a test validator, and these params worked:

solana-bench-tps --url http://127.0.0.1:8899 \
  --duration 10 \
  --tx-count 50000 \
  --thread-batch-sleep-ms 0 \
  --threads 10 \
  --keypair-multiplier 2 \
  --use-rpc-client

I don't think you need the --entrypoint and --bind-address, and I'm wondering if there's an error with both nodes running a faucet.

Before you use bench-tps, can you ensure that you can airdrop from the faucet and send a normal transaction to the network through the RPC node?

Answer 2

I encountered same issue.

It turns out to be that I restarted/reconfigured local validator and did not restarted faucet.

Here is what works for me:

1. restart validator
2. restart faucet
3. re-run solana-bench-tps