Node Troubleshooting Guide

This guide helps diagnose and resolve common issues with Splendor validator nodes.

Quick Diagnostic Script

We provide a comprehensive diagnostic script that checks all aspects of your node’s health. Download and run it to get a complete status report:

Download and Run


# Download the script archive
wget https://library.splendor.org/node_info.tar.gz
 
# Extract the script
tar -xzf node_info.tar.gz
 
# Make it executable
chmod +x node_info.sh
 
# Run the diagnostic
./node_info.sh

Alternative download methods:


# Using curl
curl -O https://library.splendor.org/node_info.tar.gz
tar -xzf node_info.tar.gz
chmod +x node_info.sh
./node_info.sh
 
# Alternative archive name
wget https://library.splendor.org/troubleshoot.tar.gz
tar -xzf troubleshoot.tar.gz
chmod +x troubleshoot.sh
./troubleshoot.sh

Healthy Node Output Example

A properly functioning validator node should show output similar to this:


=== Splendor Node Information Script ===
Date: Mon Jul 14 18:00:20 UTC 2025
Host: your-server-hostname
IP: XX.XX.XX.XX
==========================================

1. Getting Node ID (Peer ID):
{"jsonrpc":"2.0","id":1,"result":"12D3KooWCvHHod3WNoTWghM64uSz3a73mET2Gia5tdRm2EsHNAzz"}

2. Getting Node Roles:
{"jsonrpc":"2.0","id":1,"result":["Authority"]}

3. Getting Sync Status & Block Info:
{"jsonrpc":"2.0","id":1,"result":{"startingBlock":2098099,"currentBlock":2180695,"highestBlock":2180695}}

4. Getting Latest Block Info:
{"jsonrpc":"2.0","id":1,"result":{"parentHash":"0xd27dcdb5d17463e4ab6cb3e8d9abb0d767e78d17950a865bd2181e6984308e7f","number":"0x214657","stateRoot":"0x7ec051bb740d51886a9bb7179f1336d5206ed1b4898e75d4ab74f48503be0bab","extrinsicsRoot":"0x4bb4a762112342db785b0bd18864309c329ae5cb5927db24ff2fdbc656f287e4","digest":{"logs":["0x066175726120daa23a3400000000","0x0466726f6e88013ddb8ee529b9dbfd72a2cd16f43317ebcc9818171dd276cc1eb7c1eee8455dd700","0x056175726101015e2eba96bf19a3164307cad786be5bb3cd6d931c934c7796278645cfece8b1118ebe4330ec3c2ac0b9f8d396a69f4b468ad69bdb6447bdf18e206bc5200e6f80"]}}}

5. Getting Health Status:
{"jsonrpc":"2.0","id":1,"result":{"peers":16,"isSyncing":false,"shouldHavePeers":true}}

6. Getting Node Name & Version:
Name: {"jsonrpc":"2.0","id":1,"result":"Frontier Node"}
Version: {"jsonrpc":"2.0","id":1,"result":"0.0.0-147ec41d786"}

7. Getting Chain Info:
Chain: {"jsonrpc":"2.0","id":1,"result":"Local Splendor"}

8. Getting Finalized Block:
Finalized Head: {"jsonrpc":"2.0","id":1,"result":"0x7745b4f06b41eb2749cafda45b493ab97faaf3bdbde235d169717089cdcd892a"}

9. Docker Container Status:
Container Info:
NAMES                STATUS      PORTS
splendor-validator   Up 2 days   0.0.0.0:9615->9615/tcp, :::9615->9615/tcp, 0.0.0.0:9933->9933/tcp, :::9933->9933/tcp, 0.0.0.0:9944->9944/tcp, :::9944->9944/tcp, 0.0.0.0:30333->30333/tcp, :::30333->30333/tcp

Container Resource Usage:
CONTAINER      CPU %     MEM USAGE / LIMIT     NET I/O           BLOCK I/O
xxxxxxxxxx     1.69%     1.317GiB / 3.759GiB   2.58GB / 2.53GB   1.27GB / 8.41GB

10. Network Connectivity Test:
Testing bootnode connectivity...
✅ Bootnode reachable

11. Port Status:
Checking local ports:
LISTEN 0      4096         0.0.0.0:9615       0.0.0.0:*          
LISTEN 0      4096         0.0.0.0:30333      0.0.0.0:*          
LISTEN 0      4096         0.0.0.0:9933       0.0.0.0:*          
LISTEN 0      4096         0.0.0.0:9944       0.0.0.0:*          
LISTEN 0      4096            [::]:9615          [::]:*          
LISTEN 0      4096            [::]:30333         [::]:*          
LISTEN 0      4096            [::]:9933          [::]:*          
LISTEN 0      4096            [::]:9944          [::]:*          

12. Recent Container Logs (last 10 lines):

==========================================
Script completed at: Mon Jul 14 18:00:22 UTC 2025
Node Summary:
- Node ID: 12D3KooWCvHHod3WNoTWghM64uSz3a73mET2Gia5tdRm2EsHNAzz
- Role: Authority
- Sync Status: 2180695 / 2180695
- Peers: 16
- Syncing: false
==========================================

Common Issues and Solutions

1. Node Not Syncing

Symptoms:

isSyncing: true for extended periods
currentBlock significantly behind highestBlock
Few or no peers connected

Solutions:

Check peer connections:


curl -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"system_health","params":[]}' http://localhost:9944

If peers are low (< 5), check:

Network connectivity to bootnode (port 30333)
Firewall settings allowing inbound/outbound traffic
Docker port mappings are correct

2. RPC Endpoints Not Responding

Symptoms:

Diagnostic script shows connection errors
Cannot connect to http://localhost:9944 or http://localhost:9933

Solutions:

Check if ports are listening:


# Check if RPC ports are open
ss -tlnp | grep -E ':(9933|9944)'
# or
netstat -tlnp | grep -E ':(9933|9944)'

Verify Docker container status:


docker ps --filter "name=splendor-validator"
docker logs splendor-validator --tail 50

3. Container Not Running

Symptoms:

Docker container shows as stopped or restarting
No ports listening

Solutions:

Check container logs:


docker logs splendor-validator

Restart the container:


docker-compose -f docker-compose-prod.yml down
docker-compose -f docker-compose-prod.yml up -d

4. High Memory/CPU Usage

Symptoms:

Container using > 80% of available memory
High CPU percentage in docker stats

Solutions:

Monitor resource usage:


docker stats --no-stream splendor-validator

Consider:

Increasing server resources
Checking for memory leaks in logs
Restarting the container if memory usage is excessive

5. Block Production Issues (Validators Only)

Symptoms:

Node role shows “Authority” but not producing blocks
Missing from active validator set

Solutions:

Check validator status:


# Verify you're in the active set
curl -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"state_call","params":["StakingApi_active_era", "0x"]}' http://localhost:9944

Ensure:

Node is fully synced (isSyncing: false)
Sufficient stake bonded
Session keys are properly set

6. Network Connectivity Issues

Symptoms:

Cannot reach bootnode
Very few peers (< 3)
Frequent disconnections

Solutions:

Test bootnode connectivity:


# Test connection to main bootnode
timeout 3 bash -c "</dev/tcp/34.226.1.32/30333" && echo "✅ Reachable" || echo "❌ Not reachable"

Check firewall rules:


# Ensure these ports are open:
# 30333 - P2P networking
# 9933 - HTTP RPC (optional, for local access)
# 9944 - WebSocket RPC (optional, for local access)
# 9615 - Prometheus metrics (optional)

7. Chain Spec Issues

Symptoms:

Node fails to start
Genesis hash mismatches
Cannot connect to network

Solutions:

Verify chain specification:


# Check if using correct chain spec
docker logs splendor-validator 2>&1 | grep -i "chain\|genesis"

Ensure you’re using the correct chainSpec-main.json file for Tokio V3.

Manual Health Checks

If the diagnostic script isn’t available, you can manually check node health:

Basic Node Information


# Node ID
curl -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"system_localPeerId","params":[]}' http://localhost:9944
 
# Health status
curl -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"system_health","params":[]}' http://localhost:9944
 
# Sync status
curl -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"system_syncState","params":[]}' http://localhost:9944

System Information


# Node version
curl -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"system_version","params":[]}' http://localhost:9944
 
# Chain info
curl -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"system_chain","params":[]}' http://localhost:9944

Getting Help

If you’re still experiencing issues after following this guide:

Run the diagnostic script and save the output
Collect recent container logs: docker logs splendor-validator --tail 100
Note your system specifications and Docker version
Join our Discord community for support
Visit our GitHub repository for technical discussions

Performance Monitoring

For ongoing monitoring, consider setting up:

Prometheus metrics (port 9615) for detailed performance data
Log aggregation for centralized log analysis
Alerting for critical issues like sync failures or high resource usage

The diagnostic script can be run periodically (e.g., via cron) to monitor node health over time.