I'm running an validator, but after two days, still can't catchup the latest slot and keep downloading snapshots, I found the in_errors
and rcvbuf_errors
are always high and keep increasing, not sure if it's related.
Here's my running script and iptables rules:
#!/bin/bash
exec solana-validator \
--identity /root/validator-keypair.json \
--no-voting \
--ledger /root/sol/ledger \
--accounts /root/sol/accounts \
--entrypoint entrypoint.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint2.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint3.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint4.mainnet-beta.solana.com:8001 \
--entrypoint entrypoint5.mainnet-beta.solana.com:8001 \
--geyser-plugin-config /root/yellowstone-config.json \
--known-validator 7Np41oeYqPefeNQEHSv1UDhYrehxin3NStELsSKCT4K2 \
--known-validator GdnSyH3YtwcxFvQrVVJMm1JhTS4QVX7MFsX56uJLUfiZ \
--known-validator DE1bawNcRJB9rVm3buyMVfr8mBEoyyu73NBovf2oXJsJ \
--known-validator CakcnaRDHka2gXyfbEd2d3xsvkJkqsLw2akB3zsN1D2S \
--known-validator 7w3DYgChJqkfdt25y4ihj8r3mxsEEkoxfWY13u1uHeYx \
--known-validator BPF7fD7z1cWyK6Rh5GqHhbsCkUG6SN4V3wDnq12CWj8Y \
--known-validator DE1jnsfDvq6wosTtaC9PSzF7vhJdGVWBrKrwBfMDxBk7 \
--expected-genesis-hash 5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d \
--wal-recovery-mode skip_any_corrupted_record \
--account-index-exclude-key kinXdEcpDQeHPEuQnqmUgtYykqKGVFq6CeVX5iAHJq6 \
--account-index-exclude-key TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA \
--rpc-bind-address 0.0.0.0 \
--full-rpc-api \
--private-rpc \
--rpc-port 8899 \
--gossip-port 8001 \
--dynamic-port-range 8000-8020 \
--limit-ledger-size 50000000 \
--enable-rpc-transaction-history \
--init-complete-file /root/init-completed \
--enable-cpi-and-log-storage \
--log /root/solana-rpc.log \
--minimal-snapshot-download-speed 1073741824 \
--maximum-snapshot-download-abort 3
iptables rules:
root@123# iptables -nvL
Chain INPUT (policy DROP 7004 packets, 443K bytes)
pkts bytes target prot opt in out source destination
458 50653 ACCEPT 0 -- lo * 0.0.0.0/0 0.0.0.0/0
84M 415G ACCEPT 0 -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
2995 390K ACCEPT 1 -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 8
0 0 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:20415
124 6612 ACCEPT 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpts:8000:8020
54385 22M ACCEPT 17 -- * * 0.0.0.0/0 0.0.0.0/0 udp dpts:8000:8020
0 0 ACCEPT 6 -- * * myip 0.0.0.0/0 tcp dpt:8899
0 0 ACCEPT 6 -- * * myip 0.0.0.0/0 tcp dpt:8900
0 0 ACCEPT 6 -- * * myip 0.0.0.0/0 tcp dpt:10001
27 1456 DROP 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8899
0 0 DROP 6 -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8900
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 22M packets, 5701M bytes)
pkts bytes target prot opt in out source destination
Here is logs:
[2024-12-12T15:22:25.837376405Z INFO solana_metrics::metrics] datapoint: memory-stats total=405241454592i swap_total=8589930496i free_percent=19.044428381519175 used_bytes=19562012672i avail_percent=95.17275134359213 buffers_percent=0.04144097256019357 cached_percent=75.78612261502403 swap_free_percent=97.86634343449757
[2024-12-12T15:22:25.838277851Z INFO solana_metrics::metrics] datapoint: cpu-stats cpu_num=48i cpu0_freq_mhz=4029i average_load_one_minute=2.01 average_load_five_minutes=1.46 average_load_fifteen_minutes=2.79 total_num_threads=780i
[2024-12-12T15:22:25.838482760Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=23772i reads_merged=0i sectors_read=5984944i time_reading_ms=1476i writes_completed=57963i writes_merged=47788i sectors_written=12269568i time_writing_ms=602378i io_in_progress=320i time_io_ms=3965i time_io_weighted_ms=603853i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=4i
[2024-12-12T15:22:26.161385994Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 62338/424840 slots with 0 collisions
[2024-12-12T15:22:27.838897352Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=2i out_datagrams_delta=0i rcvbuf_errors_delta=2i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=30984911i rcvbuf_errors=30984911i sndbuf_errors=0i rx_bytes_delta=702i rx_packets_delta=6i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=0i tx_packets_delta=0i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2024-12-12T15:22:28.162702290Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 68882/424840 slots with 0 collisions
[2024-12-12T15:22:29.839350508Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=2i out_datagrams_delta=0i rcvbuf_errors_delta=2i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=30984913i rcvbuf_errors=30984913i sndbuf_errors=0i rx_bytes_delta=9394i rx_packets_delta=108i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=6030i tx_packets_delta=57i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2024-12-12T15:22:30.164123879Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 75291/424840 slots with 0 collisions
[2024-12-12T15:22:30.839557894Z INFO solana_metrics::metrics] datapoint: memory-stats total=405241454592i swap_total=8589930496i free_percent=23.08760069726761 used_bytes=14453731328i avail_percent=96.43330385768354 buffers_percent=0.04307940710946366 cached_percent=72.99778193764281 swap_free_percent=97.86634343449757
[2024-12-12T15:22:30.839782993Z INFO solana_metrics::metrics] datapoint: disk-stats reads_completed=23116i reads_merged=0i sectors_read=5820480i time_reading_ms=1503i writes_completed=90626i writes_merged=42251i sectors_written=19760240i time_writing_ms=999338i io_in_progress=347i time_io_ms=5869i time_io_weighted_ms=1000842i discards_completed=0i discards_merged=0i sectors_discarded=0i time_discarding=0i flushes_completed=0i time_flushing=0i num_disks=4i
[2024-12-12T15:22:32.165466511Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 81706/424840 slots with 0 collisions
[2024-12-12T15:22:32.340141498Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=2i out_datagrams_delta=0i rcvbuf_errors_delta=2i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=30984915i rcvbuf_errors=30984915i sndbuf_errors=0i rx_bytes_delta=20830i rx_packets_delta=243i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=16382i tx_packets_delta=125i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i
[2024-12-12T15:22:34.166787854Z INFO solana_runtime::snapshot_utils::snapshot_storage_rebuilder] rebuilt storages for 88185/424840 slots with 0 collisions
[2024-12-12T15:22:34.340561204Z INFO solana_metrics::metrics] datapoint: net-stats-validator in_datagrams_delta=0i no_ports_delta=0i in_errors_delta=2i out_datagrams_delta=0i rcvbuf_errors_delta=2i sndbuf_errors_delta=0i in_csum_errors_delta=0i ignored_multi_delta=0i in_errors=30984917i rcvbuf_errors=30984917i sndbuf_errors=0i rx_bytes_delta=9540i rx_packets_delta=113i rx_errs_delta=0i rx_drops_delta=0i rx_fifo_delta=0i rx_frame_delta=0i tx_bytes_delta=5922i tx_packets_delta=51i tx_errs_delta=0i tx_drops_delta=0i tx_fifo_delta=0i tx_colls_delta=0i