SuccessConsole Output

Skipping 566 KB.. Full Log
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 269 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 268 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 267 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 266 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 265 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 264 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 263 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 262 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 261 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 260 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 259 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 258 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 257 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 256 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 255 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 254 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 253 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 252 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 251 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 250 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 249 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 248 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 247 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 246 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b0b106a0-927f-46b5-abbc-17e086a3fc07
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b0b106a0-927f-46b5-abbc-17e086a3fc07'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 246 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for b0b106a0-927f-46b5-abbc-17e086a3fc07
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for b0b106a0-927f-46b5-abbc-17e086a3fc07
++ espod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ myproject=.operations
++ mymessage=398e37aa-dd8b-4cc3-926c-2a65f3166b72
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-tycs4wrj-1-70xts .operations _count systemd.u.SYSLOG_IDENTIFIER 398e37aa-dd8b-4cc3-926c-2a65f3166b72
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-tycs4wrj-1-70xts '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:398e37aa-dd8b-4cc3-926c-2a65f3166b72' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-tycs4wrj-1-70xts
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:398e37aa-dd8b-4cc3-926c-2a65f3166b72'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-tycs4wrj-1-70xts -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:398e37aa-dd8b-4cc3-926c-2a65f3166b72'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 398e37aa-dd8b-4cc3-926c-2a65f3166b72
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 398e37aa-dd8b-4cc3-926c-2a65f3166b72
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496438719
+++ expr 1496438719 - 1496438643
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 76 seconds at 2017-06-02 21:25:19.644214624+00:00
++ echo END wait_for_fluentd_to_catch_up took 76 seconds at 2017-06-02 21:25:19.644214624+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.8.225" labeled
++ wait_for_pod_ACTION stop logging-fluentd-w9gvv
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-w9gvv
++ '[' -z logging-fluentd-w9gvv -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-w9gvv
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
daemonset "logging-forward-fluentd" deleted
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '      <match **>' ']'
++ oc get configmap/logging-fluentd -o yaml
++ sed -e '/<match \*\*>/ d' -e '/@include configs\.d\/user\/secure-forward\.conf/ d' -e '/<\/match>/ d'
++ oc replace -f -
configmap "logging-fluentd" replaced
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.8.225" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 110 - 10
++ ii=100
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-6n0kv
++ '[' 100 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-6n0kv ']'
++ break
++ '[' 100 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-6n0kv
++ oc get events -o yaml
++ write_and_verify_logs 1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up '' ''
+++ date +%s
++ local starttime=1496438747
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-02 21:25:47.443453894+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-02 21:25:47.443453894+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-i5jtydma-1-21qpf
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ '[' -z logging-es-ops-data-master-tycs4wrj-1-70xts ']'
+++ uuidgen
++ local uuid_es=aa2e688a-d063-457f-bd86-6416333cd3c4
+++ uuidgen
++ local uuid_es_ops=f9247e63-f1a7-4515-a0aa-54cdc684bf24
++ local expected=1
++ local timeout=300
++ add_test_message aa2e688a-d063-457f-bd86-6416333cd3c4
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-54cs7
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/aa2e688a-d063-457f-bd86-6416333cd3c4
++ echo added es message aa2e688a-d063-457f-bd86-6416333cd3c4
added es message aa2e688a-d063-457f-bd86-6416333cd3c4
++ logger -i -p local6.info -t f9247e63-f1a7-4515-a0aa-54cdc684bf24 f9247e63-f1a7-4515-a0aa-54cdc684bf24
added es-ops message f9247e63-f1a7-4515-a0aa-54cdc684bf24
++ echo added es-ops message f9247e63-f1a7-4515-a0aa-54cdc684bf24
++ local rc=0
++ espod=logging-es-data-master-i5jtydma-1-21qpf
++ myproject=project.logging
++ mymessage=aa2e688a-d063-457f-bd86-6416333cd3c4
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message aa2e688a-d063-457f-bd86-6416333cd3c4
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message aa2e688a-d063-457f-bd86-6416333cd3c4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message aa2e688a-d063-457f-bd86-6416333cd3c4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message aa2e688a-d063-457f-bd86-6416333cd3c4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message aa2e688a-d063-457f-bd86-6416333cd3c4
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message aa2e688a-d063-457f-bd86-6416333cd3c4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message aa2e688a-d063-457f-bd86-6416333cd3c4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message aa2e688a-d063-457f-bd86-6416333cd3c4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:aa2e688a-d063-457f-bd86-6416333cd3c4'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 293 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for aa2e688a-d063-457f-bd86-6416333cd3c4
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for aa2e688a-d063-457f-bd86-6416333cd3c4
++ espod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ myproject=.operations
++ mymessage=f9247e63-f1a7-4515-a0aa-54cdc684bf24
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-tycs4wrj-1-70xts .operations _count systemd.u.SYSLOG_IDENTIFIER f9247e63-f1a7-4515-a0aa-54cdc684bf24
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-tycs4wrj-1-70xts '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:f9247e63-f1a7-4515-a0aa-54cdc684bf24' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-tycs4wrj-1-70xts
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:f9247e63-f1a7-4515-a0aa-54cdc684bf24'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-tycs4wrj-1-70xts -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:f9247e63-f1a7-4515-a0aa-54cdc684bf24'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for f9247e63-f1a7-4515-a0aa-54cdc684bf24
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for f9247e63-f1a7-4515-a0aa-54cdc684bf24
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496438759
+++ expr 1496438759 - 1496438747
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 12 seconds at 2017-06-02 21:25:59.110162025+00:00
++ echo END wait_for_fluentd_to_catch_up took 12 seconds at 2017-06-02 21:25:59.110162025+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.8.225" labeled
++ wait_for_pod_ACTION stop logging-fluentd-6n0kv
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-6n0kv
++ '[' -z logging-fluentd-6n0kv -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-6n0kv
++ '[' -n 1 ']'
pod logging-fluentd-6n0kv still running
++ echo pod logging-fluentd-6n0kv still running
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' stop = start ']'
++ '[' 110 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-6n0kv
++ '[' -n 1 ']'
pod logging-fluentd-6n0kv still running
++ echo pod logging-fluentd-6n0kv still running
++ sleep 10
+++ expr 110 - 10
++ ii=100
++ '[' stop = start ']'
++ '[' 100 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-6n0kv
++ '[' stop = start ']'
++ break
++ '[' 100 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
Error from server (NotFound): daemonsets.extensions "logging-forward-fluentd" not found
++ :
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '' ']'
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.8.225" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
pod for component=fluentd not running yet
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-30qz3
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-30qz3 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-30qz3
++ oc get events -o yaml
running test test-json-parsing.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
Adding test message 6bd1519d-7b5b-465b-bf80-bbeacd28faab to Kibana . . .
++ es_pod=logging-es-data-master-i5jtydma-1-21qpf
+++ uuidgen
++ uuid_es=6bd1519d-7b5b-465b-bf80-bbeacd28faab
++ echo Adding test message 6bd1519d-7b5b-465b-bf80-bbeacd28faab to Kibana . . .
++ add_test_message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-54cs7
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/6bd1519d-7b5b-465b-bf80-bbeacd28faab
++ rc=0
++ timeout=600
Waiting 600 seconds for 6bd1519d-7b5b-465b-bf80-bbeacd28faab to show up in Elasticsearch . . .
++ echo Waiting 600 seconds for 6bd1519d-7b5b-465b-bf80-bbeacd28faab to show up in Elasticsearch . . .
++ espod=logging-es-data-master-i5jtydma-1-21qpf
++ myproject=project.logging.
++ mymessage=6bd1519d-7b5b-465b-bf80-bbeacd28faab
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 600
++ let ii=600
++ local interval=1
++ '[' 600 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _count message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 599 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _count message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 598 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _count message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 597 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _count message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 596 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _count message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 595 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _count message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 594 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _count message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 593 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _count message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 592 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _count message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 592 -le 0 ']'
++ return 0
good - ./logging.sh: found 1 record project logging for 6bd1519d-7b5b-465b-bf80-bbeacd28faab
++ echo good - ./logging.sh: found 1 record project logging for 6bd1519d-7b5b-465b-bf80-bbeacd28faab
Testing if record is in correct format . . .
++ echo Testing if record is in correct format . . .
++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging. _search message 6bd1519d-7b5b-465b-bf80-bbeacd28faab
++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging.*/_search?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab' --connect-timeout 1
++ python test-json-parsing.py 6bd1519d-7b5b-465b-bf80-bbeacd28faab
++ local pod=logging-es-data-master-i5jtydma-1-21qpf
++ local 'endpoint=/project.logging.*/_search?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
++ shift
++ shift
++ args=("${@:-}")
++ local args
++ local secret_dir=/etc/elasticsearch/secret/
++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_search?q=message:6bd1519d-7b5b-465b-bf80-bbeacd28faab'
Success: record contains all of the expected fields/values
Success: ./logging.sh passed
++ echo Success: ./logging.sh passed
++ exit 0
running test test-mux.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ '[' false == false -o false == false ']'
Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.
++ echo 'Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.'
++ exit 0
SKIPPING upgrade test for now
running test test-viaq-data-model.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ [[ 1 -ne 1 ]]
++ [[ true = \f\a\l\s\e ]]
++ CLUSTER=true
++ ops=-ops
++ INDEX_PREFIX=
++ PROJ_PREFIX=project.
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
++ get_test_user_token
++ oc login --username=admin --password=admin
+++ oc whoami -t
++ test_token=M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg
+++ oc whoami
++ test_name=admin
++ test_ip=127.0.0.1
++ oc login --username=system:admin
++ TEST_DIVIDER=------------------------------------------
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-30qz3
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
template "logging-fluentd-template" replaced
+++ mktemp
++ cfg=/tmp/tmp.Y2LYH1Lbjk
++ cat
++ add_test_volume /tmp/tmp.Y2LYH1Lbjk
++ oc get template logging-fluentd-template -o json
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"].append({"name": "cdmtest", "mountPath": "/etc/fluent/configs.d/openshift/filter-pre-cdm-test.conf", "readOnly": True}); obj["objects"][0]["spec"]["template"]["spec"]["volumes"].append({"name": "cdmtest", "hostPath": {"path": "/tmp/tmp.Y2LYH1Lbjk"}}); print json.dumps(obj, indent=2)'
template "logging-fluentd-template" replaced
++ trap cleanup INT TERM EXIT
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-30qz3
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-30qz3
++ '[' -z logging-fluentd-30qz3 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-30qz3
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
pod for component=fluentd not running yet
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-9nvhp
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-9nvhp ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-9nvhp
++ keep_fields=method,statusCode,type,@timestamp,req,res
++ write_and_verify_logs test1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496438825
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-02 21:27:05.797865404+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-02 21:27:05.797865404+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-i5jtydma-1-21qpf
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ '[' -z logging-es-ops-data-master-tycs4wrj-1-70xts ']'
+++ uuidgen
++ local uuid_es=0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ uuidgen
++ local uuid_es_ops=609b5ff8-8735-4292-a3d6-45d3f4ed8112
++ local expected=1
++ local timeout=300
++ add_test_message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-54cs7
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
++ echo added es message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
added es message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
++ logger -i -p local6.info -t 609b5ff8-8735-4292-a3d6-45d3f4ed8112 609b5ff8-8735-4292-a3d6-45d3f4ed8112
added es-ops message 609b5ff8-8735-4292-a3d6-45d3f4ed8112
++ echo added es-ops message 609b5ff8-8735-4292-a3d6-45d3f4ed8112
++ local rc=0
++ espod=logging-es-data-master-i5jtydma-1-21qpf
++ myproject=project.logging
++ mymessage=0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 292 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
++ espod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ myproject=.operations
++ mymessage=609b5ff8-8735-4292-a3d6-45d3f4ed8112
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-tycs4wrj-1-70xts .operations _count systemd.u.SYSLOG_IDENTIFIER 609b5ff8-8735-4292-a3d6-45d3f4ed8112
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-tycs4wrj-1-70xts '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:609b5ff8-8735-4292-a3d6-45d3f4ed8112' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-tycs4wrj-1-70xts
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:609b5ff8-8735-4292-a3d6-45d3f4ed8112'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-tycs4wrj-1-70xts -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:609b5ff8-8735-4292-a3d6-45d3f4ed8112'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 609b5ff8-8735-4292-a3d6-45d3f4ed8112
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 609b5ff8-8735-4292-a3d6-45d3f4ed8112
++ '[' -n get_logmessage ']'
++ get_logmessage 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
++ logmessage=0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 609b5ff8-8735-4292-a3d6-45d3f4ed8112
++ logmessage2=609b5ff8-8735-4292-a3d6-45d3f4ed8112
+++ date +%s
++ local endtime=1496438838
+++ expr 1496438838 - 1496438825
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-02 21:27:18.651049356+00:00
++ echo END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-02 21:27:18.651049356+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-54cs7
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es project.logging _search message 0bc7d44a-2d06-4eb8-8564-6671cd0f48ad
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:0bc7d44a-2d06-4eb8-8564-6671cd0f48ad'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es-ops .operations _search message 609b5ff8-8735-4292-a3d6-45d3f4ed8112
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:609b5ff8-8735-4292-a3d6-45d3f4ed8112'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_USE_UNDEFINED '"true"'
+++ mktemp
++ junk=/tmp/tmp.THsMbzPNum
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.THsMbzPNum'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.THsMbzPNum
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.EDDAYNBuT0
++ cat
++ oc get template logging-fluentd-template -o yaml
++ oc replace -f -
++ sed '/env:/r /tmp/tmp.EDDAYNBuT0'
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.EDDAYNBuT0
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-9nvhp
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-9nvhp
++ '[' -z logging-fluentd-9nvhp -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-9nvhp
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
pod for component=fluentd not running yet
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-tr8c5
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-tr8c5 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-tr8c5
++ write_and_verify_logs test2
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496438866
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-02 21:27:46.139183619+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-02 21:27:46.139183619+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-i5jtydma-1-21qpf
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ '[' -z logging-es-ops-data-master-tycs4wrj-1-70xts ']'
+++ uuidgen
++ local uuid_es=d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ uuidgen
++ local uuid_es_ops=9203c199-5347-4f79-b503-79289825a37f
++ local expected=1
++ local timeout=300
++ add_test_message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-54cs7
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/d8d12809-1721-4b10-bf1b-4d93921b63f4
++ echo added es message d8d12809-1721-4b10-bf1b-4d93921b63f4
added es message d8d12809-1721-4b10-bf1b-4d93921b63f4
++ logger -i -p local6.info -t 9203c199-5347-4f79-b503-79289825a37f 9203c199-5347-4f79-b503-79289825a37f
added es-ops message 9203c199-5347-4f79-b503-79289825a37f
++ echo added es-ops message 9203c199-5347-4f79-b503-79289825a37f
++ local rc=0
++ espod=logging-es-data-master-i5jtydma-1-21qpf
++ myproject=project.logging
++ mymessage=d8d12809-1721-4b10-bf1b-4d93921b63f4
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4' --connect-timeout 1
+++ get_count_from_json
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message d8d12809-1721-4b10-bf1b-4d93921b63f4
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 292 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for d8d12809-1721-4b10-bf1b-4d93921b63f4
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for d8d12809-1721-4b10-bf1b-4d93921b63f4
++ espod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ myproject=.operations
++ mymessage=9203c199-5347-4f79-b503-79289825a37f
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-tycs4wrj-1-70xts .operations _count systemd.u.SYSLOG_IDENTIFIER 9203c199-5347-4f79-b503-79289825a37f
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-tycs4wrj-1-70xts '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:9203c199-5347-4f79-b503-79289825a37f' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-tycs4wrj-1-70xts
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:9203c199-5347-4f79-b503-79289825a37f'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-tycs4wrj-1-70xts -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:9203c199-5347-4f79-b503-79289825a37f'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 9203c199-5347-4f79-b503-79289825a37f
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 9203c199-5347-4f79-b503-79289825a37f
++ '[' -n get_logmessage ']'
++ get_logmessage d8d12809-1721-4b10-bf1b-4d93921b63f4
++ logmessage=d8d12809-1721-4b10-bf1b-4d93921b63f4
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 9203c199-5347-4f79-b503-79289825a37f
++ logmessage2=9203c199-5347-4f79-b503-79289825a37f
+++ date +%s
++ local endtime=1496438879
+++ expr 1496438879 - 1496438866
+++ date -u --rfc-3339=ns
++ echo END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-02 21:27:59.133590718+00:00
END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-02 21:27:59.133590718+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-54cs7
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es project.logging _search message d8d12809-1721-4b10-bf1b-4d93921b63f4
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:d8d12809-1721-4b10-bf1b-4d93921b63f4'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es-ops .operations _search message 9203c199-5347-4f79-b503-79289825a37f
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:9203c199-5347-4f79-b503-79289825a37f'
++ python test-viaq-data-model.py test2
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.HveLGO6sEV
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.HveLGO6sEV'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.HveLGO6sEV
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-tr8c5
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-tr8c5
++ '[' -z logging-fluentd-tr8c5 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-tr8c5
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
pod for component=fluentd not running yet
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-4gn2k
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-4gn2k ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-4gn2k
++ write_and_verify_logs test3
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496438906
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-02 21:28:26.167129417+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-02 21:28:26.167129417+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-i5jtydma-1-21qpf
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ '[' -z logging-es-ops-data-master-tycs4wrj-1-70xts ']'
+++ uuidgen
++ local uuid_es=20dc649f-fd8c-4337-8e51-96293e418695
+++ uuidgen
++ local uuid_es_ops=eb0713dc-21ed-4841-9983-17ed6b91e840
++ local expected=1
++ local timeout=300
++ add_test_message 20dc649f-fd8c-4337-8e51-96293e418695
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-54cs7
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/20dc649f-fd8c-4337-8e51-96293e418695
added es message 20dc649f-fd8c-4337-8e51-96293e418695
++ echo added es message 20dc649f-fd8c-4337-8e51-96293e418695
++ logger -i -p local6.info -t eb0713dc-21ed-4841-9983-17ed6b91e840 eb0713dc-21ed-4841-9983-17ed6b91e840
added es-ops message eb0713dc-21ed-4841-9983-17ed6b91e840
++ echo added es-ops message eb0713dc-21ed-4841-9983-17ed6b91e840
++ local rc=0
++ espod=logging-es-data-master-i5jtydma-1-21qpf
++ myproject=project.logging
++ mymessage=20dc649f-fd8c-4337-8e51-96293e418695
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 20dc649f-fd8c-4337-8e51-96293e418695
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 20dc649f-fd8c-4337-8e51-96293e418695
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 20dc649f-fd8c-4337-8e51-96293e418695
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 20dc649f-fd8c-4337-8e51-96293e418695
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 20dc649f-fd8c-4337-8e51-96293e418695
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 20dc649f-fd8c-4337-8e51-96293e418695
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 20dc649f-fd8c-4337-8e51-96293e418695
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 20dc649f-fd8c-4337-8e51-96293e418695
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message 20dc649f-fd8c-4337-8e51-96293e418695
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 292 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 20dc649f-fd8c-4337-8e51-96293e418695
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 20dc649f-fd8c-4337-8e51-96293e418695
++ espod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ myproject=.operations
++ mymessage=eb0713dc-21ed-4841-9983-17ed6b91e840
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-tycs4wrj-1-70xts .operations _count systemd.u.SYSLOG_IDENTIFIER eb0713dc-21ed-4841-9983-17ed6b91e840
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-tycs4wrj-1-70xts '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eb0713dc-21ed-4841-9983-17ed6b91e840' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-tycs4wrj-1-70xts
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eb0713dc-21ed-4841-9983-17ed6b91e840'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-tycs4wrj-1-70xts -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eb0713dc-21ed-4841-9983-17ed6b91e840'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for eb0713dc-21ed-4841-9983-17ed6b91e840
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for eb0713dc-21ed-4841-9983-17ed6b91e840
++ '[' -n get_logmessage ']'
++ get_logmessage 20dc649f-fd8c-4337-8e51-96293e418695
++ logmessage=20dc649f-fd8c-4337-8e51-96293e418695
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 eb0713dc-21ed-4841-9983-17ed6b91e840
++ logmessage2=eb0713dc-21ed-4841-9983-17ed6b91e840
+++ date +%s
++ local endtime=1496438919
+++ expr 1496438919 - 1496438906
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-02 21:28:39.188484943+00:00
++ echo END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-02 21:28:39.188484943+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-54cs7
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es project.logging _search message 20dc649f-fd8c-4337-8e51-96293e418695
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:20dc649f-fd8c-4337-8e51-96293e418695'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es-ops .operations _search message eb0713dc-21ed-4841-9983-17ed6b91e840
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:eb0713dc-21ed-4841-9983-17ed6b91e840'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_UNDEFINED_NAME myname
+++ mktemp
++ junk=/tmp/tmp.Xq2tcYBcQq
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.Xq2tcYBcQq'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.Xq2tcYBcQq
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-4gn2k
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-4gn2k
++ '[' -z logging-fluentd-4gn2k -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-4gn2k
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-z3676
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-z3676 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-z3676
++ write_and_verify_logs test4
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496438936
+++ date -u --rfc-3339=ns
++ echo START wait_for_fluentd_to_catch_up at 2017-06-02 21:28:56.958786052+00:00
START wait_for_fluentd_to_catch_up at 2017-06-02 21:28:56.958786052+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-i5jtydma-1-21qpf
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ '[' -z logging-es-ops-data-master-tycs4wrj-1-70xts ']'
+++ uuidgen
++ local uuid_es=b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ uuidgen
++ local uuid_es_ops=5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
++ local expected=1
++ local timeout=300
++ add_test_message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-54cs7
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
added es message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
++ echo added es message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
++ logger -i -p local6.info -t 5fb5acdc-a9d3-4caa-b8a7-93152fdf7682 5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
added es-ops message 5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
++ echo added es-ops message 5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
++ local rc=0
++ espod=logging-es-data-master-i5jtydma-1-21qpf
++ myproject=project.logging
++ mymessage=b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6' --connect-timeout 1
+++ get_count_from_json
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 292 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
++ espod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ myproject=.operations
++ mymessage=5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-tycs4wrj-1-70xts .operations _count systemd.u.SYSLOG_IDENTIFIER 5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-tycs4wrj-1-70xts '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:5fb5acdc-a9d3-4caa-b8a7-93152fdf7682' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-tycs4wrj-1-70xts
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:5fb5acdc-a9d3-4caa-b8a7-93152fdf7682'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-tycs4wrj-1-70xts -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:5fb5acdc-a9d3-4caa-b8a7-93152fdf7682'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
++ '[' -n get_logmessage ']'
++ get_logmessage b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
++ logmessage=b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
++ logmessage2=5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
+++ date +%s
++ local endtime=1496438950
+++ expr 1496438950 - 1496438936
+++ date -u --rfc-3339=ns
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-02 21:29:10.011706458+00:00
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-02 21:29:10.011706458+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-54cs7
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es project.logging _search message b7fa6b26-dc4c-4519-ac2d-5cff58205cf6
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:b7fa6b26-dc4c-4519-ac2d-5cff58205cf6'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es-ops .operations _search message 5fb5acdc-a9d3-4caa-b8a7-93152fdf7682
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:5fb5acdc-a9d3-4caa-b8a7-93152fdf7682'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,empty1,undefined3,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.cH8HSDnxbd
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.cH8HSDnxbd'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.cH8HSDnxbd
++ add_cdm_env_var_val CDM_KEEP_EMPTY_FIELDS undefined4,undefined5,empty1,undefined3
+++ mktemp
++ junk=/tmp/tmp.wMMvTzBeiF
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.wMMvTzBeiF'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.wMMvTzBeiF
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-z3676
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-z3676
++ '[' -z logging-fluentd-z3676 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-z3676
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-0s6st
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-0s6st ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-0s6st
++ write_and_verify_logs test5 allow_empty
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496438975
+++ date -u --rfc-3339=ns
++ echo START wait_for_fluentd_to_catch_up at 2017-06-02 21:29:35.721178873+00:00
START wait_for_fluentd_to_catch_up at 2017-06-02 21:29:35.721178873+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-i5jtydma-1-21qpf
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ '[' -z logging-es-ops-data-master-tycs4wrj-1-70xts ']'
+++ uuidgen
++ local uuid_es=e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ uuidgen
++ local uuid_es_ops=9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
++ local expected=1
++ local timeout=300
++ add_test_message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-54cs7
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/e315915e-8c11-4f05-8e1a-785caa3e4bbf
++ echo added es message e315915e-8c11-4f05-8e1a-785caa3e4bbf
added es message e315915e-8c11-4f05-8e1a-785caa3e4bbf
++ logger -i -p local6.info -t 9de196e6-e1cd-4ed2-af66-a3bcf6618fe0 9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
added es-ops message 9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
++ echo added es-ops message 9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
++ local rc=0
++ espod=logging-es-data-master-i5jtydma-1-21qpf
++ myproject=project.logging
++ mymessage=e315915e-8c11-4f05-8e1a-785caa3e4bbf
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-i5jtydma-1-21qpf project.logging _count message e315915e-8c11-4f05-8e1a-785caa3e4bbf
+++ get_count_from_json
+++ curl_es logging-es-data-master-i5jtydma-1-21qpf '/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-i5jtydma-1-21qpf
+++ local 'endpoint=/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-i5jtydma-1-21qpf -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 291 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for e315915e-8c11-4f05-8e1a-785caa3e4bbf
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for e315915e-8c11-4f05-8e1a-785caa3e4bbf
++ espod=logging-es-ops-data-master-tycs4wrj-1-70xts
++ myproject=.operations
++ mymessage=9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-tycs4wrj-1-70xts .operations _count systemd.u.SYSLOG_IDENTIFIER 9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-tycs4wrj-1-70xts '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:9de196e6-e1cd-4ed2-af66-a3bcf6618fe0' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-tycs4wrj-1-70xts
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:9de196e6-e1cd-4ed2-af66-a3bcf6618fe0'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-tycs4wrj-1-70xts -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:9de196e6-e1cd-4ed2-af66-a3bcf6618fe0'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
++ '[' -n get_logmessage ']'
++ get_logmessage e315915e-8c11-4f05-8e1a-785caa3e4bbf
++ logmessage=e315915e-8c11-4f05-8e1a-785caa3e4bbf
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
++ logmessage2=9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
+++ date +%s
++ local endtime=1496438989
+++ expr 1496438989 - 1496438975
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-02 21:29:49.915367539+00:00
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-02 21:29:49.915367539+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-54cs7
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es project.logging _search message e315915e-8c11-4f05-8e1a-785caa3e4bbf
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:e315915e-8c11-4f05-8e1a-785caa3e4bbf'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-54cs7 logging-es-ops .operations _search message 9de196e6-e1cd-4ed2-af66-a3bcf6618fe0
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-54cs7 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer M06ucLbXQmf1r0rTr11YC9NsgRypaEh3NTq39If9gPg' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:9de196e6-e1cd-4ed2-af66-a3bcf6618fe0'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ cleanup
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ remove_cdm_env
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.Y2LYH1Lbjk
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-0s6st
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-0s6st
++ '[' -z logging-fluentd-0s6st -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-0s6st
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-7hhbv
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-7hhbv ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
SKIPPING reinstall test for now
/data/src/github.com/openshift/origin/hack/lib/log/system.sh: line 31:  4604 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"  (wd: /data/src/github.com/openshift/origin)
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to /tmp/origin-aggregated-logging/artifacts/events.txt
[INFO] [CLEANUP] Dumping etcd contents to /tmp/origin-aggregated-logging/artifacts/etcd
[WARNING] No compiled `etcdhelper` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh tools/etcdhelper
++ Building go targets for linux/amd64: tools/etcdhelper
/data/src/github.com/openshift/origin/hack/build-go.sh took 166 seconds
2017-06-02 17:33:05.232436 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
[INFO] [CLEANUP] Dumping container logs to /tmp/origin-aggregated-logging/logs/containers
[INFO] [CLEANUP] Truncating log files over 100M
[INFO] [CLEANUP] Stopping docker containers
[INFO] [CLEANUP] Removing docker containers
Error response from daemon: No such container: 794853d3226873782b75eb5508530f38ea1fe01206c69f4e664593d6fe9d6268
Error response from daemon: No such container: 46f6ac9b33b77903c633fa07aa9e864da56245983b8f497b152a6c79abc2941b
Error response from daemon: No such container: 7b58cde36dfe27350e62f8167dc69370086494f862a6fc21fb8d9bfedfcf0b7b
Error response from daemon: No such container: 3a123be74c95a10599c3ccf289540b06f755c1bd8dc21d6f5cef11552b86b965
Error response from daemon: No such container: 4c8c6d24a2933419cf1a2eab77e7e734bbd1558a0f3db24aa602c5fb95a70c18
Error response from daemon: No such container: 9fff3da16c904e41054a5212d95fc36c994ed4f3ab6d3b3ea3f0ed75cbbf8e11
Error response from daemon: No such container: 7ea161620b46db33dbc5e41a41efc976effea48b3d94a82bfc2815f1819eae69
Error response from daemon: No such container: 8ab707e605107648abde7abd8603867f93b01e1f291b28348345e446f660bf10
Error response from daemon: No such container: 2d41466f02d3dd14f4d0c036334f20368199bb39b428582cafae1f1695388053
Error response from daemon: No such container: 4fdb7a5991a4e321805ac8ef4fbdb03017787e4d1c589d1297833f21128e490c
Error response from daemon: No such container: c48a6dcda310f3334d3e5a0d092f621090eba7e0e34ebf23726c001cef262b27
Error response from daemon: No such container: c0f9839de01bd0239ef269f0b426974f3d52cc28923c10541783194eeb5ff719
Error response from daemon: No such container: e8ecf00948da746a39294af6dcc9196f5bde6f735b788c0b24ba55cad16ec51f
[INFO] [CLEANUP] Killing child processes
[INFO] [CLEANUP] Pruning etcd data directory
[INFO] /data/src/github.com/openshift/origin/logging.sh exited with code 0 after 00h 55m 24s
Finished GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh
***************************************************

real	55m24.556s
user	9m6.802s
sys	1m16.460s
==> openshiftdev: Downloading logs
==> openshiftdev: Downloading artifacts from '/var/log/yum.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/yum.log'
==> openshiftdev: Downloading artifacts from '/var/log/secure' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/secure'
==> openshiftdev: Downloading artifacts from '/var/log/audit/audit.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/audit.log'
==> openshiftdev: Downloading artifacts from '/tmp/origin-aggregated-logging/' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts'
+ false
+ false
+ test_pull_requests --mark_test_success 462 --repo origin-aggregated-logging --config /var/lib/jenkins/.test_pull_requests_logging.json
Rate limit remaining: 1421
  Marking SUCCESS for pull request #462 in repo 'origin-aggregated-logging'
  Recreating comment #305901746 with Aggregated Logging Test Results: SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test-origin-aggregated-logging/1550/) (Base Commit: 0986045cc4f5bb216c462883f4f9ae77025f3e62)
  Deleting comment #305901746
  Updating status of 'f1648e6796ba5adbad0fc4b3a3b56bb144328091' with state: success
Rate limit remaining: 1416; delta: 5
[description-setter] Description set: <a href="https://github.com/openshift/origin-aggregated-logging/pull/462">https://github.com/openshift/origin-aggregated-logging/pull/462</a>
[PostBuildScript] - Execution post build scripts.
[workspace@2] $ /bin/sh -xe /tmp/hudson1656869019568887179.sh
+ INSTANCE_NAME=origin_logging-rhel7-1550
+ pushd origin
~/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2
+ rc=0
+ '[' -f .vagrant-openshift.json ']'
++ /usr/bin/vagrant ssh -c 'sudo ausearch -m avc'
+ ausearchresult='<no matches>'
+ rc=1
+ '[' '<no matches>' = '<no matches>' ']'
+ rc=0
+ /usr/bin/vagrant destroy -f
==> openshiftdev: Terminating the instance...
==> openshiftdev: Running cleanup tasks for 'shell' provisioner...
+ popd
~/jobs/test-origin-aggregated-logging/workspace@2
+ exit 0
Finished: SUCCESS