Cannot allocate memory error

Question

I'm running a container based on the official solana image. Dockerfile:

ARG SOLANA_VERSION=v1.11.3
FROM solanalabs/solana:${SOLANA_VERSION}
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh:

#!/bin/bash
echo $VALIDATOR_IDENTITY
set -e
if [[ -e $VALIDATOR_IDENTITY ]]; then
  echo "Use existing validator keypair"
else
  solana-keygen new --no-passphrase -so "$VALIDATOR_IDENTITY"
fi
exec solana-validator "$@"

I'm starting a docker-compose file with the following args:

version: '3'
services:
  solana:
    image: ${REGISTRY_URL}/${IMAGE_NAME}:${SOLANA_IMAGE_TAG}
    container_name: solana
    command: 
      --ledger /solana/ledger
      --identity /solana/data/validator-identity.json 
      --expected-genesis-hash 4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY
      --rpc-port 8899 
      --gossip-host $BLOCKCHAIN_HOST_EXTERNAL
      --gossip-port 8801
      --dynamic-port-range 8802-8815
      --full-rpc-api 
      --no-voting 
      --enable-rpc-transaction-history 
      --limit-ledger-size 
      --known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on 
      --known-validator dDzy5SR3AXdYWVqbDEkVFdvSPCtS9ihF5kJkHCtXoFs 
      --known-validator Ft5fbkqNa76vnsjYNwjDZUXoTWpP7VYm3mtsaQckQADN 
      --known-validator eoKpUABi59aT4rR9HGS3LcMecfut9x7zJyodWWP43YQ 
      --known-validator 9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv 
      --entrypoint entrypoint.testnet.solana.com:8001 
      --entrypoint entrypoint2.testnet.solana.com:8001 
      --entrypoint entrypoint3.testnet.solana.com:8001  
      --only-known-rpc 
      --log - 
      --no-os-network-limits-test 
      --init-complete-file /solana/data/init-completed 
      --no-port-check
    ports:
      - 8899:8899
    ulimits:
      nofile:
        soft: 1000000
        hard: 1000000

After starting a container and after getting some init messages, I'm getting an error:

solana          | [2022-07-21T12:50:28.414766618Z INFO  solana_metrics::metrics] datapoint: cpu-stats cpu_num=28i cpu0_freq_mhz=3230i average_load_one_minute=3.3 average_load_five_minutes=2.53 average_load_fifteen_minutes=2.14 total_num_threads=2000i
solana          | [2022-07-21T12:50:28.914924215Z INFO  solana_metrics::metrics] datapoint: memory-stats total=134721150976i swap_total=9663668224i free_percent=3.5714674445270678 used_bytes=58626637824i avail_percent=56.48297435163385 buffers_percent=0.7306305274777168 cached_percent=50.49884609441892 swap_free_percent=1.8690760880161608
solana          | thread 'main' panicked at 'Load from snapshot failed: Serialize(Io(Os { code: 12, kind: OutOfMemory, message: "Cannot allocate memory" }))', ledger/src/bank_forks_utils.rs:212:10
solana          | stack backtrace:
solana          |    0: rust_begin_unwind
solana          |              at ./rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/std/src/panicking.rs:584:5
solana          |    1: core::panicking::panic_fmt
solana          |              at ./rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/panicking.rs:143:14
solana          |    2: core::result::unwrap_failed
solana          |              at ./rustc/7737e0b5c4103216d6fd8cf941b7ab9bdbaace7c/library/core/src/result.rs:1749:5
solana          |    3: solana_ledger::bank_forks_utils::load_bank_forks
solana          |    4: solana_core::validator::load_blockstore
solana          |    5: solana_core::validator::Validator::new
solana          |    6: solana_validator::main
solana          | note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
solana          | [2022-07-21T12:50:29.232095785Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="main" one=1i message="panicked at 'Load from snapshot failed: Serialize(Io(Os { code: 12, kind: OutOfMemory, message: \"Cannot allocate memory\" }))', ledger/src/bank_forks_utils.rs:212:10" location="ledger/src/bank_forks_utils.rs:212:10" version="\"1.11.3 (src:1db136a8; feat:3270869161)\""

A question is how to set limit to the solana daemon? I'm trying to run a test node on a shared host with other blockchains.

Answer 1

For starters running a validator with Docker is unsupported. The image you're using is purely for use in Solana Labs CI jobs and is not intended for production.

You need to perform the system tuning steps on both the host and guest