Duplicate slot : At this point manual intervention is needed to make progress

Question

Pruning expired primary index 0 up to slot 0 (max requested: 248834748)
thread 'solReplayStage' panicked at 'We have tried to repair duplicate slot: 248834747 
more than 10 times and are unable to freeze a block with bankhash 
6FEbLdjU1UhNYzpnj1VQaW4QoJ6usacQhvE3igEDgesZ, instead we have a block with bankhash 
Some(9EzNj88y1ia7BS3Pyd2Uvk8PHFkMVQDtbuvMwkJ5GMTa). This is most likely a bug in the 
runtime. At this point manual intervention is needed to make progress. Exiting', 
core/src/replay_stage.rs:1240:25

Configuration : As suggested here https://docs.solana.com/clusters#devnet

#!/bin/sh

set -e
# Run the command with sudo and the -E option to preserve the environment
 set -o errexit
 set -o nounset
# set -o pipefail
# Remove empty snapshots
sudo find "/solana/ledger" -name 'snapshot-*' -size 0 -print -exec rm {} \; || true
 export RUST_LOG=error
 export RUST_BACKTRACE=full

sudo /home/debian/.local/share/solana/install/active_release/bin/solana-validator \
  --identity /home/debian/validator-keypair.json \
    --vote-account vote-account-keypair.json \
    --known-validator dv1ZAGvdsz5hHLwWXsVnM94hWf1pjbKVau1QVkaMJ92 \
    --known-validator dv2eQHeP4RFrJZ6UeiZWoc3XTtmtZCUKxxCApCDcRNV \
    --known-validator dv4ACNkpYPcE3aKmYDqZm9G5EB3J4MRoeE7WNDRBVJB \
    --known-validator dv3qDFk1DTF36Z62bNvrCXe9sKATA6xvVy6A798xxAS \
    --only-known-rpc \
    --rpc-port 8899 \
    --gossip-port 8801 \
  --dynamic-port-range 8802-8816 \
    --entrypoint entrypoint.devnet.solana.com:8001 \
    --entrypoint entrypoint2.devnet.solana.com:8001 \
    --entrypoint entrypoint3.devnet.solana.com:8001 \
    --entrypoint entrypoint4.devnet.solana.com:8001 \
    --entrypoint entrypoint5.devnet.solana.com:8001 \
    --expected-genesis-hash EtWTRABZaYq6iMfeYKouRu166VU2xqa1wcaWoxPkrZBG \
    --wal-recovery-mode skip_any_corrupted_record \
    --limit-ledger-size \
        --log ~/log/solana-validator.log \
--ledger /solana/ledger 

Answer 1

As you have seemingly already determined, your issue was running too old of a version. More specifically, a feature gate was activated on devnet that required a v1.16 release. Feature gates are the mechanism in which features that alter consensus are rolled out to the clusters. The new feature was added in v1.16, so the v1.14 client was not aware of the change and fell out of sync with the cluster majority of v1.16.

If you're running a validator or rpc node on any cluster, you need to keep up with minimum software versions. At least one place where these updates are broadcasted is through the Solana discord in the cluster-announcement channels (there is one for each of devnet, testnet and mnb).

Answer 2

Update solana validator cli to latest version solana-cli 1.16.16 . Getting error on solana-cli 1.14.29