1

We are running a solana node since around April 2023. It was running fine since that time except some minor issues. But it looks like since the migration to the v1.16.17 it’s very difficult to make it sync.

curl http://127.0.0.1:8899 -X POST -H "Content-Type: application/json" -d '
  {
    "jsonrpc":"2.0","id":1,
    "method":"getHealth"
  }
'
{"jsonrpc":"2.0","error":{"code":-32005,"message":"Node is behind by 7390 slots","data":{"numSlotsBehind":7390}},"id":1}

Usually when this happen we wipe the node volume where all the data are located and restart a new node, and in approximately 1h it’s up and running again. But this time it doesn’t work. The node lags behind all the time.

We have a very powerful machine with 40CPU and 320GB memory and super fast NVMe SSD.

Is there anything I can do to investigate what is slowing down the validator ?

Thx.

EDIT: Do you know if there is some parameters that we can tune in order for solana-validatord to perform less write operations, but bigger ones. Because apparently our SSD can’t handle all the write. But in terms of bandwith it’s ok.

1 Answer 1

1

There could be several reasons for this. v1.16.17 has been problematic for servers with lower memory, but this doesn't seem to apply to you.

The 1st thing to do is to make sure you are following the latest documentation as the process of starting a validator differs slightly with time. Make sure you are doing the steps from "System Tunning", I highly recommend you use separate disks for accounts and ledger as the documentation outlines, and check you are passing appropriate command line arguments at start-up.

If you're still having trouble, my best recommendation is to go with this problem to the Solana discord and ask for help in the dedicated channel (#validator-support). These is a wealth of information there and people with deep experience ready to assist.

1
  • 1
    OK thank for the answer. I have actually more data on this issue. I monitored the disk activity and found out that it was overloaded. Even with our NVMe SSD. I also found out about the account storage on disk. And actually we have already disabled the index storage and rely purely on RAM. We have asked on the discord channel already. They mentioned that we run our node in a VM inside an Hypervisor. Which is not recommended. But that’s annoying because we don’t have any other hosting solution in our cloud provider for now. Oct 26, 2023 at 9:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.