SuccessConsole Output

Skipping 582 KB.. Full Log
h project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 257 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 256 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 255 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 254 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 253 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 252 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 251 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 250 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 249 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 248 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 247 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 246 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 245 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 244 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 243 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 242 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message db59991c-66b3-4eac-9f5f-b1638fb0a8a6
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:db59991c-66b3-4eac-9f5f-b1638fb0a8a6'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 242 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for db59991c-66b3-4eac-9f5f-b1638fb0a8a6
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for db59991c-66b3-4eac-9f5f-b1638fb0a8a6
++ espod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ myproject=.operations
++ mymessage=6c20b8b8-591a-417d-a537-96833ab88103
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER 6c20b8b8-591a-417d-a537-96833ab88103
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:6c20b8b8-591a-417d-a537-96833ab88103' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:6c20b8b8-591a-417d-a537-96833ab88103'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:6c20b8b8-591a-417d-a537-96833ab88103'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 6c20b8b8-591a-417d-a537-96833ab88103
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 6c20b8b8-591a-417d-a537-96833ab88103
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496956477
+++ expr 1496956477 - 1496956395
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 82 seconds at 2017-06-08 21:14:37.579728035+00:00
++ echo END wait_for_fluentd_to_catch_up took 82 seconds at 2017-06-08 21:14:37.579728035+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.4.251" labeled
++ wait_for_pod_ACTION stop logging-fluentd-t2b89
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-t2b89
++ '[' -z logging-fluentd-t2b89 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-t2b89
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
daemonset "logging-forward-fluentd" deleted
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '      <match **>' ']'
++ oc get configmap/logging-fluentd -o yaml
++ sed -e '/<match \*\*>/ d' -e '/@include configs\.d\/user\/secure-forward\.conf/ d' -e '/<\/match>/ d'
++ oc replace -f -
configmap "logging-fluentd" replaced
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.4.251" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-5l8ss
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-5l8ss ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-5l8ss
++ oc get events -o yaml
++ write_and_verify_logs 1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up '' ''
+++ date +%s
++ local starttime=1496956501
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 21:15:01.124478950+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 21:15:01.124478950+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ '[' -z logging-es-ops-data-master-10aztz22-1-j3dz5 ']'
+++ uuidgen
++ local uuid_es=9fc34db7-1675-4341-bf60-90cdf75c335b
+++ uuidgen
++ local uuid_es_ops=43688c23-92bb-4bef-82ba-264c4ddd6510
++ local expected=1
++ local timeout=300
++ add_test_message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-6cs5r
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/9fc34db7-1675-4341-bf60-90cdf75c335b
++ echo added es message 9fc34db7-1675-4341-bf60-90cdf75c335b
added es message 9fc34db7-1675-4341-bf60-90cdf75c335b
++ logger -i -p local6.info -t 43688c23-92bb-4bef-82ba-264c4ddd6510 43688c23-92bb-4bef-82ba-264c4ddd6510
added es-ops message 43688c23-92bb-4bef-82ba-264c4ddd6510
++ echo added es-ops message 43688c23-92bb-4bef-82ba-264c4ddd6510
++ local rc=0
++ espod=logging-es-data-master-5ufvzrs6-1-27kfh
++ myproject=project.logging
++ mymessage=9fc34db7-1675-4341-bf60-90cdf75c335b
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message 9fc34db7-1675-4341-bf60-90cdf75c335b
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9fc34db7-1675-4341-bf60-90cdf75c335b'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 290 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 9fc34db7-1675-4341-bf60-90cdf75c335b
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 9fc34db7-1675-4341-bf60-90cdf75c335b
++ espod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ myproject=.operations
++ mymessage=43688c23-92bb-4bef-82ba-264c4ddd6510
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER 43688c23-92bb-4bef-82ba-264c4ddd6510
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:43688c23-92bb-4bef-82ba-264c4ddd6510' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:43688c23-92bb-4bef-82ba-264c4ddd6510'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:43688c23-92bb-4bef-82ba-264c4ddd6510'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 43688c23-92bb-4bef-82ba-264c4ddd6510
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 43688c23-92bb-4bef-82ba-264c4ddd6510
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496956516
+++ expr 1496956516 - 1496956501
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-08 21:15:16.686255778+00:00
++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-08 21:15:16.686255778+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.4.251" labeled
++ wait_for_pod_ACTION stop logging-fluentd-5l8ss
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-5l8ss
++ '[' -z logging-fluentd-5l8ss -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-5l8ss
++ '[' -n 1 ']'
pod logging-fluentd-5l8ss still running
++ echo pod logging-fluentd-5l8ss still running
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' stop = start ']'
++ '[' 110 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-5l8ss
++ '[' stop = start ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
Error from server (NotFound): daemonsets.extensions "logging-forward-fluentd" not found
++ :
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '' ']'
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.4.251" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-94h3t
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-94h3t ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-94h3t
++ oc get events -o yaml
running test test-json-parsing.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ es_pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ uuidgen
Adding test message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d to Kibana . . .
++ uuid_es=2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
++ echo Adding test message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d to Kibana . . .
++ add_test_message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-6cs5r
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
++ rc=0
++ timeout=600
++ echo Waiting 600 seconds for 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d to show up in Elasticsearch . . .
Waiting 600 seconds for 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d to show up in Elasticsearch . . .
++ espod=logging-es-data-master-5ufvzrs6-1-27kfh
++ myproject=project.logging.
++ mymessage=2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 600
++ let ii=600
++ local interval=1
++ '[' 600 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 599 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 598 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 597 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 596 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ get_count_from_json
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 595 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 594 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 593 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 592 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 591 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 590 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 589 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 588 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _count message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 588 -le 0 ']'
++ return 0
good - ./logging.sh: found 1 record project logging for 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
++ echo good - ./logging.sh: found 1 record project logging for 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
Testing if record is in correct format . . .
++ echo Testing if record is in correct format . . .
++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging. _search message 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
++ python test-json-parsing.py 2a050c6f-f40f-423b-82d1-4a6a0aae2e9d
++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging.*/_search?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d' --connect-timeout 1
++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
++ local 'endpoint=/project.logging.*/_search?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
++ shift
++ shift
++ args=("${@:-}")
++ local args
++ local secret_dir=/etc/elasticsearch/secret/
++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_search?q=message:2a050c6f-f40f-423b-82d1-4a6a0aae2e9d'
Success: record contains all of the expected fields/values
Success: ./logging.sh passed
++ echo Success: ./logging.sh passed
++ exit 0
running test test-mux.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ '[' false == false -o false == false ']'
Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.
++ echo 'Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.'
++ exit 0
SKIPPING upgrade test for now
running test test-viaq-data-model.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ [[ 1 -ne 1 ]]
++ [[ true = \f\a\l\s\e ]]
++ CLUSTER=true
++ ops=-ops
++ INDEX_PREFIX=
++ PROJ_PREFIX=project.
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
++ get_test_user_token
++ local current_project
+++ oc project -q
++ current_project=logging
++ oc login --username=admin --password=admin
+++ oc whoami -t
++ test_token=Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw
+++ oc whoami
++ test_name=admin
++ test_ip=127.0.0.1
++ oc login --username=system:admin
++ oc project logging
++ TEST_DIVIDER=------------------------------------------
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-94h3t
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
+++ mktemp
++ cfg=/tmp/tmp.jIqwwFkgay
++ cat
++ add_test_volume /tmp/tmp.jIqwwFkgay
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"].append({"name": "cdmtest", "mountPath": "/etc/fluent/configs.d/openshift/filter-pre-cdm-test.conf", "readOnly": True}); obj["objects"][0]["spec"]["template"]["spec"]["volumes"].append({"name": "cdmtest", "hostPath": {"path": "/tmp/tmp.jIqwwFkgay"}}); print json.dumps(obj, indent=2)'
++ oc get template logging-fluentd-template -o json
template "logging-fluentd-template" replaced
++ trap cleanup INT TERM EXIT
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-94h3t
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-94h3t
++ '[' -z logging-fluentd-94h3t -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-94h3t
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-0673q
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-0673q ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-0673q
++ keep_fields=method,statusCode,type,@timestamp,req,res
++ write_and_verify_logs test1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496956580
+++ date -u --rfc-3339=ns
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 21:16:20.146614175+00:00
START wait_for_fluentd_to_catch_up at 2017-06-08 21:16:20.146614175+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ '[' -z logging-es-ops-data-master-10aztz22-1-j3dz5 ']'
+++ uuidgen
++ local uuid_es=eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ uuidgen
++ local uuid_es_ops=c80413f6-6736-4afc-afdc-8bbbf53c2f62
++ local expected=1
++ local timeout=300
++ add_test_message eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-6cs5r
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/eb4d1dd5-06d1-46f1-9953-3888541d8470
added es message eb4d1dd5-06d1-46f1-9953-3888541d8470
++ echo added es message eb4d1dd5-06d1-46f1-9953-3888541d8470
++ logger -i -p local6.info -t c80413f6-6736-4afc-afdc-8bbbf53c2f62 c80413f6-6736-4afc-afdc-8bbbf53c2f62
added es-ops message c80413f6-6736-4afc-afdc-8bbbf53c2f62
++ echo added es-ops message c80413f6-6736-4afc-afdc-8bbbf53c2f62
++ local rc=0
++ espod=logging-es-data-master-5ufvzrs6-1-27kfh
++ myproject=project.logging
++ mymessage=eb4d1dd5-06d1-46f1-9953-3888541d8470
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message eb4d1dd5-06d1-46f1-9953-3888541d8470
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 293 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for eb4d1dd5-06d1-46f1-9953-3888541d8470
good - wait_for_fluentd_to_catch_up: found 1 record project logging for eb4d1dd5-06d1-46f1-9953-3888541d8470
++ espod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ myproject=.operations
++ mymessage=c80413f6-6736-4afc-afdc-8bbbf53c2f62
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER c80413f6-6736-4afc-afdc-8bbbf53c2f62
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:c80413f6-6736-4afc-afdc-8bbbf53c2f62' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:c80413f6-6736-4afc-afdc-8bbbf53c2f62'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:c80413f6-6736-4afc-afdc-8bbbf53c2f62'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for c80413f6-6736-4afc-afdc-8bbbf53c2f62
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for c80413f6-6736-4afc-afdc-8bbbf53c2f62
++ '[' -n get_logmessage ']'
++ get_logmessage eb4d1dd5-06d1-46f1-9953-3888541d8470
++ logmessage=eb4d1dd5-06d1-46f1-9953-3888541d8470
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 c80413f6-6736-4afc-afdc-8bbbf53c2f62
++ logmessage2=c80413f6-6736-4afc-afdc-8bbbf53c2f62
+++ date +%s
++ local endtime=1496956591
+++ expr 1496956591 - 1496956580
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 11 seconds at 2017-06-08 21:16:31.578492461+00:00
++ echo END wait_for_fluentd_to_catch_up took 11 seconds at 2017-06-08 21:16:31.578492461+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-6cs5r
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es project.logging _search message eb4d1dd5-06d1-46f1-9953-3888541d8470
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:eb4d1dd5-06d1-46f1-9953-3888541d8470'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es-ops .operations _search message c80413f6-6736-4afc-afdc-8bbbf53c2f62
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:c80413f6-6736-4afc-afdc-8bbbf53c2f62'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_USE_UNDEFINED '"true"'
+++ mktemp
++ junk=/tmp/tmp.CnTGHTqGW8
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.CnTGHTqGW8'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.CnTGHTqGW8
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.LXH4VEVG5q
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.LXH4VEVG5q'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.LXH4VEVG5q
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-0673q
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-0673q
++ '[' -z logging-fluentd-0673q -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-0673q
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-1v909
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-1v909 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-1v909
++ write_and_verify_logs test2
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496956610
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 21:16:50.798298957+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 21:16:50.798298957+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ '[' -z logging-es-ops-data-master-10aztz22-1-j3dz5 ']'
+++ uuidgen
++ local uuid_es=cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ uuidgen
++ local uuid_es_ops=d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
++ local expected=1
++ local timeout=300
++ add_test_message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-6cs5r
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/cacdeb81-7e1f-46f7-ad14-ce8d266e1579
added es message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
++ echo added es message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
++ logger -i -p local6.info -t d3a6e0cd-53bc-4c75-89ec-d9007b280fbb d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
added es-ops message d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
++ echo added es-ops message d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
++ local rc=0
++ espod=logging-es-data-master-5ufvzrs6-1-27kfh
++ myproject=project.logging
++ mymessage=cacdeb81-7e1f-46f7-ad14-ce8d266e1579
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 293 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for cacdeb81-7e1f-46f7-ad14-ce8d266e1579
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for cacdeb81-7e1f-46f7-ad14-ce8d266e1579
++ espod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ myproject=.operations
++ mymessage=d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d3a6e0cd-53bc-4c75-89ec-d9007b280fbb' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d3a6e0cd-53bc-4c75-89ec-d9007b280fbb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d3a6e0cd-53bc-4c75-89ec-d9007b280fbb'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
++ '[' -n get_logmessage ']'
++ get_logmessage cacdeb81-7e1f-46f7-ad14-ce8d266e1579
++ logmessage=cacdeb81-7e1f-46f7-ad14-ce8d266e1579
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
++ logmessage2=d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
+++ date +%s
++ local endtime=1496956622
+++ expr 1496956622 - 1496956610
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 12 seconds at 2017-06-08 21:17:02.351405824+00:00
++ echo END wait_for_fluentd_to_catch_up took 12 seconds at 2017-06-08 21:17:02.351405824+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-6cs5r
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es project.logging _search message cacdeb81-7e1f-46f7-ad14-ce8d266e1579
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:cacdeb81-7e1f-46f7-ad14-ce8d266e1579'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es-ops .operations _search message d3a6e0cd-53bc-4c75-89ec-d9007b280fbb
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:d3a6e0cd-53bc-4c75-89ec-d9007b280fbb'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.E9UpCG1w5Y
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.E9UpCG1w5Y'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.E9UpCG1w5Y
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-1v909
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-1v909
++ '[' -z logging-fluentd-1v909 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-1v909
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-sczwq
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-sczwq ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-sczwq
++ write_and_verify_logs test3
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496956638
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 21:17:18.313007022+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 21:17:18.313007022+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ get_running_pod es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
+++ oc get pods -l component=es-ops
++ local es_ops_pod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ '[' -z logging-es-ops-data-master-10aztz22-1-j3dz5 ']'
+++ uuidgen
++ local uuid_es=b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ uuidgen
++ local uuid_es_ops=e0e86215-8941-4820-9e32-1ba48d8c40f8
++ local expected=1
++ local timeout=300
++ add_test_message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-6cs5r
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
added es message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
++ echo added es message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
++ logger -i -p local6.info -t e0e86215-8941-4820-9e32-1ba48d8c40f8 e0e86215-8941-4820-9e32-1ba48d8c40f8
added es-ops message e0e86215-8941-4820-9e32-1ba48d8c40f8
++ echo added es-ops message e0e86215-8941-4820-9e32-1ba48d8c40f8
++ local rc=0
++ espod=logging-es-data-master-5ufvzrs6-1-27kfh
++ myproject=project.logging
++ mymessage=b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 291 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
++ espod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ myproject=.operations
++ mymessage=e0e86215-8941-4820-9e32-1ba48d8c40f8
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER e0e86215-8941-4820-9e32-1ba48d8c40f8
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:e0e86215-8941-4820-9e32-1ba48d8c40f8' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:e0e86215-8941-4820-9e32-1ba48d8c40f8'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:e0e86215-8941-4820-9e32-1ba48d8c40f8'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for e0e86215-8941-4820-9e32-1ba48d8c40f8
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for e0e86215-8941-4820-9e32-1ba48d8c40f8
++ '[' -n get_logmessage ']'
++ get_logmessage b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
++ logmessage=b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 e0e86215-8941-4820-9e32-1ba48d8c40f8
++ logmessage2=e0e86215-8941-4820-9e32-1ba48d8c40f8
+++ date +%s
++ local endtime=1496956652
+++ expr 1496956652 - 1496956638
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-08 21:17:32.633147759+00:00
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-08 21:17:32.633147759+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-6cs5r
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es project.logging _search message b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:b7b3ee81-fdf2-4b1d-8f2c-ea3e7462e564'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es-ops .operations _search message e0e86215-8941-4820-9e32-1ba48d8c40f8
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:e0e86215-8941-4820-9e32-1ba48d8c40f8'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_UNDEFINED_NAME myname
+++ mktemp
++ junk=/tmp/tmp.HMtNjdxoVS
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.HMtNjdxoVS'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.HMtNjdxoVS
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-sczwq
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-sczwq
++ '[' -z logging-fluentd-sczwq -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-sczwq
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-wrwj2
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-wrwj2 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-wrwj2
++ write_and_verify_logs test4
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496956680
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 21:18:00.428579592+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 21:18:00.428579592+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ '[' -z logging-es-ops-data-master-10aztz22-1-j3dz5 ']'
+++ uuidgen
++ local uuid_es=b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ uuidgen
++ local uuid_es_ops=008d5948-e554-40b4-aa09-d9ec51eeb1de
++ local expected=1
++ local timeout=300
++ add_test_message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-6cs5r
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
added es message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
++ echo added es message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
++ logger -i -p local6.info -t 008d5948-e554-40b4-aa09-d9ec51eeb1de 008d5948-e554-40b4-aa09-d9ec51eeb1de
added es-ops message 008d5948-e554-40b4-aa09-d9ec51eeb1de
++ echo added es-ops message 008d5948-e554-40b4-aa09-d9ec51eeb1de
++ local rc=0
++ espod=logging-es-data-master-5ufvzrs6-1-27kfh
++ myproject=project.logging
++ mymessage=b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 293 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
++ espod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ myproject=.operations
++ mymessage=008d5948-e554-40b4-aa09-d9ec51eeb1de
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER 008d5948-e554-40b4-aa09-d9ec51eeb1de
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER 008d5948-e554-40b4-aa09-d9ec51eeb1de
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER 008d5948-e554-40b4-aa09-d9ec51eeb1de
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER 008d5948-e554-40b4-aa09-d9ec51eeb1de
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:008d5948-e554-40b4-aa09-d9ec51eeb1de'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 297 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 008d5948-e554-40b4-aa09-d9ec51eeb1de
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 008d5948-e554-40b4-aa09-d9ec51eeb1de
++ '[' -n get_logmessage ']'
++ get_logmessage b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
++ logmessage=b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 008d5948-e554-40b4-aa09-d9ec51eeb1de
++ logmessage2=008d5948-e554-40b4-aa09-d9ec51eeb1de
+++ date +%s
++ local endtime=1496956696
+++ expr 1496956696 - 1496956680
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-08 21:18:16.487783293+00:00
++ echo END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-08 21:18:16.487783293+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-6cs5r
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es project.logging _search message b059bbfb-e5b1-4861-93c6-c32d1e12c7b2
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:b059bbfb-e5b1-4861-93c6-c32d1e12c7b2'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es-ops .operations _search message 008d5948-e554-40b4-aa09-d9ec51eeb1de
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:008d5948-e554-40b4-aa09-d9ec51eeb1de'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,empty1,undefined3,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.Q2nRXNHKGw
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.Q2nRXNHKGw'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.Q2nRXNHKGw
++ add_cdm_env_var_val CDM_KEEP_EMPTY_FIELDS undefined4,undefined5,empty1,undefined3
+++ mktemp
++ junk=/tmp/tmp.ldfZbMg1uM
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.ldfZbMg1uM'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.ldfZbMg1uM
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-wrwj2
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-wrwj2
++ '[' -z logging-fluentd-wrwj2 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-wrwj2
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-kwl45
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-kwl45 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-kwl45
++ write_and_verify_logs test5 allow_empty
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496956714
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 21:18:34.939112073+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 21:18:34.939112073+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ '[' -z logging-es-ops-data-master-10aztz22-1-j3dz5 ']'
+++ uuidgen
++ local uuid_es=d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ uuidgen
++ local uuid_es_ops=7986ccf1-7a6a-4f62-b8a7-8b6063bee347
++ local expected=1
++ local timeout=300
++ add_test_message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-6cs5r
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/d56a7a34-b504-46cb-baa7-7bcce291d8ec
added es message d56a7a34-b504-46cb-baa7-7bcce291d8ec
++ echo added es message d56a7a34-b504-46cb-baa7-7bcce291d8ec
++ logger -i -p local6.info -t 7986ccf1-7a6a-4f62-b8a7-8b6063bee347 7986ccf1-7a6a-4f62-b8a7-8b6063bee347
added es-ops message 7986ccf1-7a6a-4f62-b8a7-8b6063bee347
++ echo added es-ops message 7986ccf1-7a6a-4f62-b8a7-8b6063bee347
++ local rc=0
++ espod=logging-es-data-master-5ufvzrs6-1-27kfh
++ myproject=project.logging
++ mymessage=d56a7a34-b504-46cb-baa7-7bcce291d8ec
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
+++ shift
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec' --connect-timeout 1
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-5ufvzrs6-1-27kfh project.logging _count message d56a7a34-b504-46cb-baa7-7bcce291d8ec
+++ get_count_from_json
+++ curl_es logging-es-data-master-5ufvzrs6-1-27kfh '/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-5ufvzrs6-1-27kfh
+++ local 'endpoint=/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-5ufvzrs6-1-27kfh -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
good - wait_for_fluentd_to_catch_up: found 1 record project logging for d56a7a34-b504-46cb-baa7-7bcce291d8ec
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 292 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for d56a7a34-b504-46cb-baa7-7bcce291d8ec
++ espod=logging-es-ops-data-master-10aztz22-1-j3dz5
++ myproject=.operations
++ mymessage=7986ccf1-7a6a-4f62-b8a7-8b6063bee347
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-10aztz22-1-j3dz5 .operations _count systemd.u.SYSLOG_IDENTIFIER 7986ccf1-7a6a-4f62-b8a7-8b6063bee347
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-ops-data-master-10aztz22-1-j3dz5 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:7986ccf1-7a6a-4f62-b8a7-8b6063bee347' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-10aztz22-1-j3dz5
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:7986ccf1-7a6a-4f62-b8a7-8b6063bee347'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-10aztz22-1-j3dz5 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:7986ccf1-7a6a-4f62-b8a7-8b6063bee347'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 7986ccf1-7a6a-4f62-b8a7-8b6063bee347
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 7986ccf1-7a6a-4f62-b8a7-8b6063bee347
++ '[' -n get_logmessage ']'
++ get_logmessage d56a7a34-b504-46cb-baa7-7bcce291d8ec
++ logmessage=d56a7a34-b504-46cb-baa7-7bcce291d8ec
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 7986ccf1-7a6a-4f62-b8a7-8b6063bee347
++ logmessage2=7986ccf1-7a6a-4f62-b8a7-8b6063bee347
+++ date +%s
++ local endtime=1496956727
+++ expr 1496956727 - 1496956714
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-08 21:18:47.749868902+00:00
++ echo END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-08 21:18:47.749868902+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-6cs5r
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es project.logging _search message d56a7a34-b504-46cb-baa7-7bcce291d8ec
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:d56a7a34-b504-46cb-baa7-7bcce291d8ec'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-6cs5r logging-es-ops .operations _search message 7986ccf1-7a6a-4f62-b8a7-8b6063bee347
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-6cs5r -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer Xg2qk8R7Brf41l2vkBpWtF5cllH8mEmkD4xyLClV3dw' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:7986ccf1-7a6a-4f62-b8a7-8b6063bee347'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ cleanup
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ remove_cdm_env
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.jIqwwFkgay
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-kwl45
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-kwl45
++ '[' -z logging-fluentd-kwl45 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-kwl45
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
pod for component=fluentd not running yet
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-vwsj9
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-vwsj9 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
SKIPPING reinstall test for now
/data/src/github.com/openshift/origin-aggregated-logging/hack/lib/log/system.sh: line 31:  4608 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"  (wd: /data/src/github.com/openshift/origin-aggregated-logging)
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to /tmp/origin-aggregated-logging/artifacts/events.txt
[INFO] [CLEANUP] Dumping etcd contents to /tmp/origin-aggregated-logging/artifacts/etcd
[WARNING] No compiled `etcdhelper` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh tools/etcdhelper
++ Building go targets for linux/amd64: tools/etcdhelper
/data/src/github.com/openshift/origin-aggregated-logging/../origin/hack/build-go.sh took 168 seconds
2017-06-08 17:22:02.846728 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
[INFO] [CLEANUP] Dumping container logs to /tmp/origin-aggregated-logging/logs/containers
[INFO] [CLEANUP] Truncating log files over 200M
[INFO] [CLEANUP] Stopping docker containers
[INFO] [CLEANUP] Removing docker containers
Error response from daemon: You cannot remove a running container 14879e244f53e6115e475efacf7d06322d3b754071a8aa7f12e58b06cd76cc3c. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 509695f2711ecf4e76089fe6d62f2846bdf0934be477e0638cbc7ac4938bc02a. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 45974e779b733a158e761382417e8f3563a3222636a16e6c8ec94f494f437c31. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container fb09cfee904bf16872cea210d42c0f7b435fe36bc5745958aae5d0de0a3d2b58. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 5211d4e78e87aa72c8e6326fd694dc172b6051071242f034331e8d2c73b1a065. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 0c9448a89af276718eea58053ed7c045103f0b4b73c3fe116bcea3c35a0ed434. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 2d8ec2ed48546367834efddbc675106ba90ef120a3634a73c54c680a1e082150. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container c7311d7ba0f9241716fbf29da26697616afa010498e8630f1596b5c7e670cc23. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 05a7632f12ef983fb42e40c821dffb3298cf71f4ab26377e72c7c480ea024152. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container e3ef265548a68f6128e82b962c825a6627d0cc09365f4a01c8c368573073e214. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 4036753f0c528a6c1580b130a78885695153a9bd4bfb9e438a1818d981a5533b. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 5c9156b7b27e6aac44c0d3d382fe0a91faf69908a1d34819ba678c8e8a66383d. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container dd215534d1f22e6838770fcf238c7009fb999e4540ba924d8f9e628f97d5692d. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 99f0620d19beef4f306754be6c669f17f791d73c09569475ff2d92d9a56e3836. Stop the container before attempting removal or use -f
[INFO] [CLEANUP] Killing child processes
[INFO] [CLEANUP] Pruning etcd data directory
[INFO] /data/src/github.com/openshift/origin-aggregated-logging/logging.sh exited with code 0 after 01h 00m 46s

real	60m46.013s
user	5m21.853s
sys	1m1.724s
Finished GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh
***************************************************
==> openshiftdev: Downloading logs
==> openshiftdev: Downloading artifacts from '/var/log/yum.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/yum.log'
==> openshiftdev: Downloading artifacts from '/var/log/secure' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/secure'
==> openshiftdev: Downloading artifacts from '/var/log/audit/audit.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/audit.log'
==> openshiftdev: Downloading artifacts from '/tmp/origin-aggregated-logging/' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts'
+ false
+ false
+ test_pull_requests --mark_test_success --repo origin-aggregated-logging --config /var/lib/jenkins/.test_pull_requests_logging.json
Rate limit remaining: 2736
  Marking SUCCESS for pull request #--repo in repo ''
/usr/share/ruby/net/http/response.rb:119:in `error!': 404 "Not Found" (Net::HTTPServerException)
	from /bin/test_pull_requests:543:in `block in get_comments'
	from /bin/test_pull_requests:535:in `each'
	from /bin/test_pull_requests:535:in `get_comments'
	from /bin/test_pull_requests:1074:in `get_comment_matching_regex'
	from /bin/test_pull_requests:1090:in `get_comment_with_prefix'
	from /bin/test_pull_requests:779:in `mark_test_success'
	from /bin/test_pull_requests:2361:in `<main>'
+ true
[description-setter] Could not determine description.
[PostBuildScript] - Execution post build scripts.
[workspace] $ /bin/sh -xe /tmp/hudson2141670511664926205.sh
+ INSTANCE_NAME=origin_logging-rhel7-1638
+ pushd origin
~/jobs/test-origin-aggregated-logging/workspace/origin ~/jobs/test-origin-aggregated-logging/workspace
+ rc=0
+ '[' -f .vagrant-openshift.json ']'
++ /usr/bin/vagrant ssh -c 'sudo ausearch -m avc'
+ ausearchresult='<no matches>'
+ rc=1
+ '[' '<no matches>' = '<no matches>' ']'
+ rc=0
+ /usr/bin/vagrant destroy -f
==> openshiftdev: Terminating the instance...
==> openshiftdev: Running cleanup tasks for 'shell' provisioner...
+ popd
~/jobs/test-origin-aggregated-logging/workspace
+ exit 0
Finished: SUCCESS