Infrastructure

Network

Jumbo frames

$ ping -M do -s 8972 -c 3 10.30.50.11
PING 10.30.50.11 (10.30.50.11) 8972(9000) bytes of data.
8980 bytes from 10.30.50.11: icmp_seq=1 ttl=64 time=0.255 ms
8980 bytes from 10.30.50.11: icmp_seq=2 ttl=64 time=0.206 ms
8980 bytes from 10.30.50.11: icmp_seq=3 ttl=64 time=0.191 ms

--- 10.30.50.11 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.191/0.217/0.255/0.029 ms

Libvirtd

$ docker exec -it nova_libvirt virsh nodeinfo
CPU model:           x86_64
CPU(s):              32
CPU frequency:       800 MHz
CPU socket(s):       1
Core(s) per socket:  8
Thread(s) per core:  2
NUMA cell(s):        2
Memory size:         263784260 KiB
$ docker exec -it nova_libvirt virsh sysinfo
<sysinfo type='smbios'>
  <bios>
    <entry name='vendor'>American Megatrends Inc.</entry>
    <entry name='version'>2.0b</entry>
    <entry name='date'>02/28/2018</entry>
    <entry name='release'>5.12</entry>
  </bios>
  <system>
    <entry name='manufacturer'>Supermicro</entry>
[...]
$ docker exec -it nova_libvirt virsh capabilities
<capabilities>

  <host>
    <uuid>00000000-0000-0000-0000-ac1f6b09a1de</uuid>
    <cpu>
      <arch>x86_64</arch>
      <model>Skylake-Client-IBRS</model>
      <vendor>Intel</vendor>
[...]

Memcached

$ echo stats | nc 10.49.20.10 11211
STAT pid 7
STAT uptime 2524
STAT time 1528967802
STAT version 1.4.25 Ubuntu
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 0.044000
STAT rusage_system 0.088000
STAT curr_connections 1
STAT total_connections 2
STAT connection_structures 2
STAT reserved_fds 20
[...]

Open vSwitch

$ docker exec -it openvswitch_vswitchd ovs-vsctl -V
ovs-vsctl (Open vSwitch) 2.8.1
DB Schema 7.15.0

On network nodes and compute nodes with provider networks, after the initial start of the service, a br-ex exists with the external interfaces.

$ docker exec -it openvswitch_vswitchd ovs-vsctl show
a2f9dbad-519e-4873-aea4-0719abcd9e2a
    Bridge br-ex
        Port br-ex
            Interface br-ex
                type: internal
        Port "enp24s0f1"
            Interface "enp24s0f1"

RabbitMQ

dragon@20-10:~$ docker exec -it rabbitmq rabbitmqctl cluster_status
Cluster status of node 'rabbit@20-10'
[{nodes,[{disc,['rabbit@20-10','rabbit@20-11','rabbit@20-12']}]},
 {running_nodes,['rabbit@20-12','rabbit@20-11','rabbit@20-10']},
 {cluster_name,<<"rabbit@20-10.betacloud.xyz">>},
 {partitions,[]},
 {alarms,[{'rabbit@20-12',[]},{'rabbit@20-11',[]},{'rabbit@20-10',[]}]}]

Alternatively, log in to the web interface and check the status of the nodes there.

../_images/rabbitmq-nodes.png

MariaDB

Login to the mariadb databaserver (run docker exec -it mariadb mysql -u root -p on one of the database nodes or use phpMyAdmin running on the manager node on port 8110) and run the following query.

The password for MariaDB can be found in the file environments/kolla/secrets.yml in the variable database_password.

MariaDB [(none)]> SHOW GLOBAL STATUS LIKE 'wsrep_%';
+------------------------------+----------------------------------------------------+
| Variable_name                | Value                                              |
+------------------------------+----------------------------------------------------+
[...]
| wsrep_local_state_comment    | Synced                                             |
| wsrep_incoming_addresses     | 10.49.20.10:3306,10.49.20.11:3306,10.49.20.12:3306 |
| wsrep_evs_state              | OPERATIONAL                                        |
| wsrep_cluster_size           | 3                                                  |
| wsrep_cluster_status         | Primary                                            |
| wsrep_connected              | ON                                                 |
| wsrep_ready                  | ON                                                 |
[...]
+------------------------------+----------------------------------------------------+

Elasticsearch

Note

Run this command on the manager node.

$ curl -s http://10.49.0.100:9200/_cluster/health | python -m json.tool
{
    "active_primary_shards": 321,
    "active_shards": 642,
    "active_shards_percent_as_number": 100.0,
    "cluster_name": "kolla_logging",
    "delayed_unassigned_shards": 0,
    "initializing_shards": 0,
    "number_of_data_nodes": 3,
    "number_of_in_flight_fetch": 0,
    "number_of_nodes": 3,
    "number_of_pending_tasks": 0,
    "relocating_shards": 0,
    "status": "green",
    "task_max_waiting_in_queue_millis": 0,
    "timed_out": false,
    "unassigned_shards": 0
}
  • number_of_data_nodes should be the number of available Elasticsearch nodes

  • status should be green

Fluentd

$ docker logs fluentd
[...]
2018-06-14 08:15:52 +0000 [info]: #0 listening syslog socket on 10.49.10.11:5140 with udp
[...]
2018-06-14 08:27:05 +0000 [info]: #0 Connection opened to Elasticsearch cluster => {:host=>"10.49.0.100", :port=>9200, :scheme=>"http"}