vendredi 10 avril 2020

ElasticSearch - Unassigned shards and replication failed


Hi,

Today, I want to share a workaround to correct ElasticSearch shards replication problem:

In my case, a shard was unassigned and my cluster state was yellow..

First, I had to determine which index was impacted :


root@server [ /root ] curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED 
2158047191530222659_-823974979_201908_001-2 3 r UNASSIGNED REPLICATION FAILED


The I had to see, why the replication failed, in the log file on master node I found these kind of errors:


...
try to recover [2158047191530222659_-823974979_201908_001-2][3] from primary shard with sync id but number of docs differ: 17986 (ElasticSearch Data2, primary) vs 20064(ElasticSearch Data1)];
...


The problem is due to a corrupted recover file on a slave node.

In this case, simply decrease number of replicates to "0" and then reconfigure the replicates to its initial config to force the replication (Elasticsearch will recreate metadata on the slave nodes). If you don't know the number of replicates, you can use this command:


root@server [ /root ] curl -X GET "localhost:9200/2158047191530222659_-823974979_201908_001-2/_settings?pretty"


  • Decrease the number of replicas


root@server [ /root ] curl -X PUT "localhost:9200/2158047191530222659_-823974979_201908_001-2/_settings?pretty" -H 'Content-Type: application/json' -d'
{
    "index" : {
        "number_of_replicas" : 0
    }
}'



  • Wait 5 seconds, and then reconfigure to the original value.



root@server [ /root ] curl -X PUT "localhost:9200/2158047191530222659_-823974979_201908_001-2/_settings?pretty" -H 'Content-Type: application/json' -d'
{
    "index" : {
        "number_of_replicas" : 1
    }
}'



  • Check if UNASSIGNED shards are still exist.



root@server [ /root ] curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

root@server 



No more unassigned shards :)

SSL - retrieve website certicate from cli with openssl


Hi,

Here is a simple tip to check ssl certificate with openssl. Sometimes you can't do otherwise if the website is requesting client cert and you don't have, but you have to renew the certificate on server side...
With this command you will be able to see if the new server certificate is correct and up to date.

  • openssl s_client -connect will connect to the website.
  • 2>&1 will display the all the output. 
  • < /dev/null will send empty block to the command (to exit)
  • then sed command will remove the BEGIN and END lines
  • then openssl x509 -noout -text will print the certificat as text

root@servername # openssl s_client -connect www.google.com:443  2>&1 < /dev/null | sed -n '/-----BEGIN/,/-----END/p' | openssl x509 -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            81:14:a8:04:f4:b6:a5:a4:02:00:00:00:00:5f:97:72
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Google Trust Services, CN=GTS CA 1O1
        Validity
            Not Before: Mar 24 06:35:44 2020 GMT
            Not After : Jun 16 06:35:44 2020 GMT
        Subject: C=US, ST=California, L=Mountain View, O=Google LLC, CN=www.google.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a3:b1:f2:78:b3:39:98:28:49:46:f2:c3:54:1b:
                    89:c1:70:2d:43:0c:1f:d2:64:96:97:1a:d3:2b:fb:
                    4a:a1:56:87:40:aa:c7:c7:bc:e9:1b:28:08:ae:b6:
                    25:02:e0:c5:61:14:86:32:6b:17:9d:67:81:79:a8:
                    4e:a5:0e:32:0f:50:be:5d:c8:f6:0c:4c:37:b1:e7:
                    6c:7d:4e:f4:97:86:99:2e:88:3c:ad:d4:b7:ec:1b:
                    92:33:d7:58:8d:8c:00:b7:66:99:f0:f3:ee:bb:8a:
                    7c:f2:da:22:87:bb:5f:e1:c0:57:b8:3e:7d:73:45:
                    27:cc:05:28:2f:95:ef:98:ac:cc:5d:a7:80:85:04:
                    71:d8:8b:d9:0a:4e:c2:3b:75:b6:e8:65:e7:de:ba:
                    d2:35:03:61:54:4f:55:6b:e4:7e:ca:89:cd:e0:7d:
                    b4:18:e2:85:8e:11:d8:8e:d9:08:7c:e5:62:a2:8f:
                    38:76:b5:5b:ad:24:63:4e:3d:ab:3f:04:b4:cc:ea:
                    c1:fa:cb:55:5a:f8:bb:9a:0d:2f:6a:19:b0:a1:a5:
                    8b:a9:e3:a7:f8:87:0e:90:8f:e5:68:23:32:ca:d1:
                    71:cc:eb:98:6c:1a:3c:06:fd:1f:c6:60:49:a7:52:
                    90:90:c8:59:5b:5a:d7:d7:1e:07:cf:20:ee:c7:f5:
                    58:77
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                E1:8A:BF:37:55:F5:37:C1:A0:4A:B6:BE:3F:F1:9C:8A:38:54:D3:6B
            X509v3 Authority Key Identifier:
                keyid:98:D1:F8:6E:10:EB:CF:9B:EC:60:9F:18:90:1B:A0:EB:7D:09:FD:2B

            Authority Information Access:
                OCSP - URI:http://ocsp.pki.goog/gts1o1
                CA Issuers - URI:http://pki.goog/gsr2/GTS1O1.crt

            X509v3 Subject Alternative Name:
                DNS:www.google.com
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.2
                Policy: 1.3.6.1.4.1.11129.2.5.3

            X509v3 CRL Distribution Points:
                Full Name:
                  URI:http://crl.pki.goog/GTS1O1.crl

            1.3.6.1.4.1.11129.2.4.2:
                ......w......... N.f.+..% gk..p..IS-...^...q.w.......H0F.!...h-..@Y8Z.zY\...qyl..+... .M.wB.!..&.l.....r...Z.1..L.....)Qw.G..5.u.^.s..V...6H}.I.2z.........u..qEX...q.w.......F0D. a..........I..C..n..A...#..
}.yQ. ...E.; ....p..]&@.....>....Y..u.
    Signature Algorithm: sha256WithRSAEncryption
         52:6e:de:a5:0e:32:d8:3a:5f:be:3a:27:b0:69:43:ef:22:05:
         57:bc:0e:0f:99:6e:71:89:33:8c:b0:7e:e3:73:dd:19:b2:65:
         85:a7:4a:4d:f2:2b:27:5d:05:61:b5:b4:86:70:37:15:e3:cd:
         48:5a:11:77:d9:a5:9b:44:96:50:5e:02:69:d3:8c:aa:cd:38:
         8e:40:84:a7:91:29:46:81:d6:75:a0:1b:69:31:8f:61:ec:6b:
         b6:f2:23:15:5f:61:96:12:b3:31:34:06:9d:a8:67:7b:60:e6:
         5f:2c:2f:6a:3d:18:9d:6c:80:c7:94:af:68:9a:6b:1e:b6:2c:
         3b:f2:a9:2b:72:e8:57:4e:c8:68:ed:c6:88:2b:83:24:de:60:
         07:c4:61:c7:e0:0e:a2:f6:92:8c:09:9d:1d:2c:bd:35:8d:51:
         92:43:1f:5e:8f:26:9e:6f:3e:65:ff:a5:79:6f:1a:1c:b6:3c:
         c8:48:85:38:a8:1b:ea:64:35:bf:df:42:0c:4e:c8:65:da:ce:
         8e:c3:7f:e5:8f:fc:53:ac:32:7d:4a:90:c3:5d:c0:85:5c:84:
         b3:f3:0c:0b:dc:1c:45:f5:a6:a7:ea:9e:09:e1:28:d8:d7:9e:
         e3:6b:9b:85:92:dc:5e:c5:04:7b:8a:d8:c0:60:13:c1:a0:df:
         1b:96:b0:13




mercredi 11 mars 2020

Generate ansible host file from redhat satellite

Hi,

Here is a little script to generate ansible host list from redhat satellite.
The idea is to use HostCollection to create ansible groups.

You need to have hammer installed on the server.
My satellite-api.conf contains this keys:

   [main]
   SATELLITE_URL = https://mysatellite.domain.dns
   SATELLITE_LOGIN = ****************
   SATELLITE_PASSWORD = *****************
   SATELLITE_ORG = myOrg


Here is the script

#!/bin/bash 

fn_isHammer() {
    if [ ! -f /usr/bin/hammer ]
    then
       echo "Error: hammer is not installed on this machine"
       exit -1
    fi
}

fn_isConfFile() {
    if [ ! -s /usr/local/etc/satellite-api.conf ]
    then
       echo "Error: satellite-api.conf not found"
       exit -1
    fi
}


fn_getConfig() {
    fn_isConfFile
    login=$(grep SATELLITE_LOGIN /usr/local/etc/satellite-api.conf|awk  '{print $NF}')
    password=$(grep SATELLITE_PASSWORD /usr/local/etc/satellite-api.conf|awk '{print $NF}')
    url=$(grep SATELLITE_URL /usr/local/etc/satellite-api.conf|awk '{print $NF}')
    ORG=$(grep SATELLITE_ORG /usr/local/etc/satellite-api.conf|awk -F= '{print $NF}')
}

fn_getHostsByCollections() {
    for HC in $(hammer --no-headers -u $login -p$password \
                     -s $url host-collection list | awk '{print $3}')
    do
       echo "[${HC}]"
       hammer --no-headers -u $login -p$password -s $url \
                     host-collection hosts --name "${HC}" \
                     --organization $ORG | awk '{print $NF}'
       echo ""
    done
}

main() {
    fn_isHammer
    fn_getConfig
    fn_getHostsByCollections
}

main