vendredi 10 avril 2020

ElasticSearch - Unassigned shards and replication failed


Hi,

Today, I want to share a workaround to correct ElasticSearch shards replication problem:

In my case, a shard was unassigned and my cluster state was yellow..

First, I had to determine which index was impacted :


root@server [ /root ] curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED 
2158047191530222659_-823974979_201908_001-2 3 r UNASSIGNED REPLICATION FAILED


The I had to see, why the replication failed, in the log file on master node I found these kind of errors:


...
try to recover [2158047191530222659_-823974979_201908_001-2][3] from primary shard with sync id but number of docs differ: 17986 (ElasticSearch Data2, primary) vs 20064(ElasticSearch Data1)];
...


The problem is due to a corrupted recover file on a slave node.

In this case, simply decrease number of replicates to "0" and then reconfigure the replicates to its initial config to force the replication (Elasticsearch will recreate metadata on the slave nodes). If you don't know the number of replicates, you can use this command:


root@server [ /root ] curl -X GET "localhost:9200/2158047191530222659_-823974979_201908_001-2/_settings?pretty"


  • Decrease the number of replicas


root@server [ /root ] curl -X PUT "localhost:9200/2158047191530222659_-823974979_201908_001-2/_settings?pretty" -H 'Content-Type: application/json' -d'
{
    "index" : {
        "number_of_replicas" : 0
    }
}'



  • Wait 5 seconds, and then reconfigure to the original value.



root@server [ /root ] curl -X PUT "localhost:9200/2158047191530222659_-823974979_201908_001-2/_settings?pretty" -H 'Content-Type: application/json' -d'
{
    "index" : {
        "number_of_replicas" : 1
    }
}'



  • Check if UNASSIGNED shards are still exist.



root@server [ /root ] curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

root@server 



No more unassigned shards :)

Aucun commentaire:

Publier un commentaire