SuccessConsole Output

Skipping 576 KB.. Full Log
 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 261 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 260 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 259 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 258 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 257 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 256 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 255 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 254 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 253 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 252 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 251 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 250 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 249 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 248 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 247 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message e8f18efe-b099-46aa-87e3-b035c2fb26a0
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8f18efe-b099-46aa-87e3-b035c2fb26a0'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 247 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for e8f18efe-b099-46aa-87e3-b035c2fb26a0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for e8f18efe-b099-46aa-87e3-b035c2fb26a0
++ espod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ myproject=.operations
++ mymessage=4f377eae-4929-4aa5-8feb-216259955e20
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER 4f377eae-4929-4aa5-8feb-216259955e20
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4f377eae-4929-4aa5-8feb-216259955e20' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4f377eae-4929-4aa5-8feb-216259955e20'
+++ shift
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4f377eae-4929-4aa5-8feb-216259955e20'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 4f377eae-4929-4aa5-8feb-216259955e20
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 4f377eae-4929-4aa5-8feb-216259955e20
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496959486
+++ expr 1496959486 - 1496959410
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 76 seconds at 2017-06-08 22:04:46.130698190+00:00
++ echo END wait_for_fluentd_to_catch_up took 76 seconds at 2017-06-08 22:04:46.130698190+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.5.71" labeled
++ wait_for_pod_ACTION stop logging-fluentd-hxjz4
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-hxjz4
++ '[' -z logging-fluentd-hxjz4 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-hxjz4
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
daemonset "logging-forward-fluentd" deleted
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '      <match **>' ']'
++ oc get configmap/logging-fluentd -o yaml
++ sed -e '/<match \*\*>/ d' -e '/@include configs\.d\/user\/secure-forward\.conf/ d' -e '/<\/match>/ d'
++ oc replace -f -
configmap "logging-fluentd" replaced
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.5.71" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-kqrk0
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-kqrk0 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-kqrk0
++ oc get events -o yaml
++ write_and_verify_logs 1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up '' ''
+++ date +%s
++ local starttime=1496959500
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 22:05:00.801582778+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 22:05:00.801582778+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-nh0txix5-1-psfc8
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ '[' -z logging-es-ops-data-master-kw4v1jbu-1-h8r1q ']'
+++ uuidgen
++ local uuid_es=ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ uuidgen
++ local uuid_es_ops=03ee4287-9b0d-4025-8c53-6ad319f391b3
++ local expected=1
++ local timeout=300
++ add_test_message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-jg9t8
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/ff5992e0-d890-4d4b-b8ad-f821abab28bf
added es message ff5992e0-d890-4d4b-b8ad-f821abab28bf
++ echo added es message ff5992e0-d890-4d4b-b8ad-f821abab28bf
++ logger -i -p local6.info -t 03ee4287-9b0d-4025-8c53-6ad319f391b3 03ee4287-9b0d-4025-8c53-6ad319f391b3
added es-ops message 03ee4287-9b0d-4025-8c53-6ad319f391b3
++ echo added es-ops message 03ee4287-9b0d-4025-8c53-6ad319f391b3
++ local rc=0
++ espod=logging-es-data-master-nh0txix5-1-psfc8
++ myproject=project.logging
++ mymessage=ff5992e0-d890-4d4b-b8ad-f821abab28bf
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message ff5992e0-d890-4d4b-b8ad-f821abab28bf
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ff5992e0-d890-4d4b-b8ad-f821abab28bf'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 290 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for ff5992e0-d890-4d4b-b8ad-f821abab28bf
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for ff5992e0-d890-4d4b-b8ad-f821abab28bf
++ espod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ myproject=.operations
++ mymessage=03ee4287-9b0d-4025-8c53-6ad319f391b3
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER 03ee4287-9b0d-4025-8c53-6ad319f391b3
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:03ee4287-9b0d-4025-8c53-6ad319f391b3' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:03ee4287-9b0d-4025-8c53-6ad319f391b3'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:03ee4287-9b0d-4025-8c53-6ad319f391b3'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 03ee4287-9b0d-4025-8c53-6ad319f391b3
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 03ee4287-9b0d-4025-8c53-6ad319f391b3
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496959516
+++ expr 1496959516 - 1496959500
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-08 22:05:16.567436428+00:00
++ echo END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-08 22:05:16.567436428+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.5.71" labeled
++ wait_for_pod_ACTION stop logging-fluentd-kqrk0
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-kqrk0
++ '[' -z logging-fluentd-kqrk0 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-kqrk0
++ '[' -n 1 ']'
pod logging-fluentd-kqrk0 still running
++ echo pod logging-fluentd-kqrk0 still running
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' stop = start ']'
++ '[' 110 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-kqrk0
++ '[' -n 1 ']'
pod logging-fluentd-kqrk0 still running
++ echo pod logging-fluentd-kqrk0 still running
++ sleep 10
+++ expr 110 - 10
++ ii=100
++ '[' stop = start ']'
++ '[' 100 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-kqrk0
++ '[' stop = start ']'
++ break
++ '[' 100 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
Error from server (NotFound): daemonsets.extensions "logging-forward-fluentd" not found
++ :
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '' ']'
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" not patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.5.71" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-httd4
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-httd4 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-httd4
++ oc get events -o yaml
running test test-json-parsing.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ es_pod=logging-es-data-master-nh0txix5-1-psfc8
+++ uuidgen
++ uuid_es=afc61d63-13b2-40d5-97a6-28f059ab165c
++ echo Adding test message afc61d63-13b2-40d5-97a6-28f059ab165c to Kibana . . .
Adding test message afc61d63-13b2-40d5-97a6-28f059ab165c to Kibana . . .
++ add_test_message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-jg9t8
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/afc61d63-13b2-40d5-97a6-28f059ab165c
++ rc=0
++ timeout=600
++ echo Waiting 600 seconds for afc61d63-13b2-40d5-97a6-28f059ab165c to show up in Elasticsearch . . .
Waiting 600 seconds for afc61d63-13b2-40d5-97a6-28f059ab165c to show up in Elasticsearch . . .
++ espod=logging-es-data-master-nh0txix5-1-psfc8
++ myproject=project.logging.
++ mymessage=afc61d63-13b2-40d5-97a6-28f059ab165c
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 600
++ let ii=600
++ local interval=1
++ '[' 600 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _count message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 599 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _count message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 598 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _count message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 597 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _count message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 596 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _count message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 595 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _count message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 594 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _count message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 593 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _count message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 592 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _count message afc61d63-13b2-40d5-97a6-28f059ab165c
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 592 -le 0 ']'
++ return 0
good - ./logging.sh: found 1 record project logging for afc61d63-13b2-40d5-97a6-28f059ab165c
++ echo good - ./logging.sh: found 1 record project logging for afc61d63-13b2-40d5-97a6-28f059ab165c
Testing if record is in correct format . . .
++ echo Testing if record is in correct format . . .
++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging. _search message afc61d63-13b2-40d5-97a6-28f059ab165c
++ python test-json-parsing.py afc61d63-13b2-40d5-97a6-28f059ab165c
++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging.*/_search?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c' --connect-timeout 1
++ local pod=logging-es-data-master-nh0txix5-1-psfc8
++ local 'endpoint=/project.logging.*/_search?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
++ shift
++ shift
++ args=("${@:-}")
++ local args
++ local secret_dir=/etc/elasticsearch/secret/
++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_search?q=message:afc61d63-13b2-40d5-97a6-28f059ab165c'
Success: record contains all of the expected fields/values
Success: ./logging.sh passed
++ echo Success: ./logging.sh passed
++ exit 0
running test test-mux.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ '[' false == false -o false == false ']'
Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.
++ echo 'Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.'
++ exit 0
SKIPPING upgrade test for now
running test test-viaq-data-model.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ [[ 1 -ne 1 ]]
++ [[ true = \f\a\l\s\e ]]
++ CLUSTER=true
++ ops=-ops
++ INDEX_PREFIX=
++ PROJ_PREFIX=project.
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
++ get_test_user_token
++ local current_project
+++ oc project -q
++ current_project=logging
++ oc login --username=admin --password=admin
+++ oc whoami -t
++ test_token=bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E
+++ oc whoami
++ test_name=admin
++ test_ip=127.0.0.1
++ oc login --username=system:admin
++ oc project logging
++ TEST_DIVIDER=------------------------------------------
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-httd4
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
+++ mktemp
++ cfg=/tmp/tmp.poloaJ2AK9
++ cat
++ add_test_volume /tmp/tmp.poloaJ2AK9
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"].append({"name": "cdmtest", "mountPath": "/etc/fluent/configs.d/openshift/filter-pre-cdm-test.conf", "readOnly": True}); obj["objects"][0]["spec"]["template"]["spec"]["volumes"].append({"name": "cdmtest", "hostPath": {"path": "/tmp/tmp.poloaJ2AK9"}}); print json.dumps(obj, indent=2)'
++ oc replace -f -
++ oc get template logging-fluentd-template -o json
template "logging-fluentd-template" replaced
++ trap cleanup INT TERM EXIT
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-httd4
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-httd4
++ '[' -z logging-fluentd-httd4 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-httd4
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-k5jwz
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-k5jwz ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-k5jwz
++ keep_fields=method,statusCode,type,@timestamp,req,res
++ write_and_verify_logs test1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496959580
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 22:06:20.813820986+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 22:06:20.813820986+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-nh0txix5-1-psfc8
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ '[' -z logging-es-ops-data-master-kw4v1jbu-1-h8r1q ']'
+++ uuidgen
++ local uuid_es=1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ uuidgen
++ local uuid_es_ops=d6429b8e-589c-4567-8720-f84f7fd3b477
++ local expected=1
++ local timeout=300
++ add_test_message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-jg9t8
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
added es message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
++ echo added es message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
++ logger -i -p local6.info -t d6429b8e-589c-4567-8720-f84f7fd3b477 d6429b8e-589c-4567-8720-f84f7fd3b477
added es-ops message d6429b8e-589c-4567-8720-f84f7fd3b477
++ echo added es-ops message d6429b8e-589c-4567-8720-f84f7fd3b477
++ local rc=0
++ espod=logging-es-data-master-nh0txix5-1-psfc8
++ myproject=project.logging
++ mymessage=1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ get_count_from_json
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 290 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
++ espod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ myproject=.operations
++ mymessage=d6429b8e-589c-4567-8720-f84f7fd3b477
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER d6429b8e-589c-4567-8720-f84f7fd3b477
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d6429b8e-589c-4567-8720-f84f7fd3b477' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d6429b8e-589c-4567-8720-f84f7fd3b477'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d6429b8e-589c-4567-8720-f84f7fd3b477'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for d6429b8e-589c-4567-8720-f84f7fd3b477
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for d6429b8e-589c-4567-8720-f84f7fd3b477
++ '[' -n get_logmessage ']'
++ get_logmessage 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
++ logmessage=1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 d6429b8e-589c-4567-8720-f84f7fd3b477
++ logmessage2=d6429b8e-589c-4567-8720-f84f7fd3b477
+++ date +%s
++ local endtime=1496959596
+++ expr 1496959596 - 1496959580
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-08 22:06:36.581066652+00:00
++ echo END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-08 22:06:36.581066652+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-jg9t8
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es project.logging _search message 1dc7936a-ce14-4d6b-a694-4ffa0132f3ab
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:1dc7936a-ce14-4d6b-a694-4ffa0132f3ab'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es-ops .operations _search message d6429b8e-589c-4567-8720-f84f7fd3b477
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:d6429b8e-589c-4567-8720-f84f7fd3b477'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_USE_UNDEFINED '"true"'
+++ mktemp
++ junk=/tmp/tmp.h5YNlA2Pys
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.h5YNlA2Pys'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.h5YNlA2Pys
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.TpulKdng48
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.TpulKdng48'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.TpulKdng48
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-k5jwz
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-k5jwz
++ '[' -z logging-fluentd-k5jwz -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-k5jwz
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
pod for component=fluentd not running yet
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-xnr18
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-xnr18 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-xnr18
++ write_and_verify_logs test2
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496959619
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 22:06:59.957318709+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 22:06:59.957318709+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-nh0txix5-1-psfc8
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ '[' -z logging-es-ops-data-master-kw4v1jbu-1-h8r1q ']'
+++ uuidgen
++ local uuid_es=5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ uuidgen
++ local uuid_es_ops=68367465-8f39-48ca-8e4b-2ab73921ef1e
++ local expected=1
++ local timeout=300
++ add_test_message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-jg9t8
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/5db938ea-e092-4bc3-b7c8-280bb8e69a90
++ echo added es message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
added es message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
++ logger -i -p local6.info -t 68367465-8f39-48ca-8e4b-2ab73921ef1e 68367465-8f39-48ca-8e4b-2ab73921ef1e
added es-ops message 68367465-8f39-48ca-8e4b-2ab73921ef1e
++ echo added es-ops message 68367465-8f39-48ca-8e4b-2ab73921ef1e
++ local rc=0
++ espod=logging-es-data-master-nh0txix5-1-psfc8
++ myproject=project.logging
++ mymessage=5db938ea-e092-4bc3-b7c8-280bb8e69a90
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 289 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 289 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 5db938ea-e092-4bc3-b7c8-280bb8e69a90
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 5db938ea-e092-4bc3-b7c8-280bb8e69a90
++ espod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ myproject=.operations
++ mymessage=68367465-8f39-48ca-8e4b-2ab73921ef1e
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER 68367465-8f39-48ca-8e4b-2ab73921ef1e
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:68367465-8f39-48ca-8e4b-2ab73921ef1e' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:68367465-8f39-48ca-8e4b-2ab73921ef1e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:68367465-8f39-48ca-8e4b-2ab73921ef1e'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 68367465-8f39-48ca-8e4b-2ab73921ef1e
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 68367465-8f39-48ca-8e4b-2ab73921ef1e
++ '[' -n get_logmessage ']'
++ get_logmessage 5db938ea-e092-4bc3-b7c8-280bb8e69a90
++ logmessage=5db938ea-e092-4bc3-b7c8-280bb8e69a90
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 68367465-8f39-48ca-8e4b-2ab73921ef1e
++ logmessage2=68367465-8f39-48ca-8e4b-2ab73921ef1e
+++ date +%s
++ local endtime=1496959637
+++ expr 1496959637 - 1496959619
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 18 seconds at 2017-06-08 22:07:17.119904932+00:00
++ echo END wait_for_fluentd_to_catch_up took 18 seconds at 2017-06-08 22:07:17.119904932+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-jg9t8
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es project.logging _search message 5db938ea-e092-4bc3-b7c8-280bb8e69a90
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:5db938ea-e092-4bc3-b7c8-280bb8e69a90'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es-ops .operations _search message 68367465-8f39-48ca-8e4b-2ab73921ef1e
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:68367465-8f39-48ca-8e4b-2ab73921ef1e'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.kp8V5S1DGO
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.kp8V5S1DGO'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.kp8V5S1DGO
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-xnr18
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-xnr18
++ '[' -z logging-fluentd-xnr18 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-xnr18
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-v86b9
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-v86b9 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-v86b9
++ write_and_verify_logs test3
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496959654
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 22:07:34.186541631+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 22:07:34.186541631+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-nh0txix5-1-psfc8
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ '[' -z logging-es-ops-data-master-kw4v1jbu-1-h8r1q ']'
+++ uuidgen
++ local uuid_es=60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ uuidgen
++ local uuid_es_ops=bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
++ local expected=1
++ local timeout=300
++ add_test_message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-jg9t8
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/60cf9a0c-f2d5-455b-823d-74a8878d64a1
added es message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
++ echo added es message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
++ logger -i -p local6.info -t bc4aa684-9f7d-4054-bcff-ea6fa8b291f2 bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
added es-ops message bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
++ echo added es-ops message bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
++ local rc=0
++ espod=logging-es-data-master-nh0txix5-1-psfc8
++ myproject=project.logging
++ mymessage=60cf9a0c-f2d5-455b-823d-74a8878d64a1
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 292 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 60cf9a0c-f2d5-455b-823d-74a8878d64a1
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 60cf9a0c-f2d5-455b-823d-74a8878d64a1
++ espod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ myproject=.operations
++ mymessage=bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ get_count_from_json
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bc4aa684-9f7d-4054-bcff-ea6fa8b291f2' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bc4aa684-9f7d-4054-bcff-ea6fa8b291f2'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bc4aa684-9f7d-4054-bcff-ea6fa8b291f2'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
++ '[' -n get_logmessage ']'
++ get_logmessage 60cf9a0c-f2d5-455b-823d-74a8878d64a1
++ logmessage=60cf9a0c-f2d5-455b-823d-74a8878d64a1
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
++ logmessage2=bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
+++ date +%s
++ local endtime=1496959667
+++ expr 1496959667 - 1496959654
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-08 22:07:47.244561351+00:00
++ echo END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-08 22:07:47.244561351+00:00
++ return 0
+++ get_running_pod kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
+++ oc get pods -l component=kibana
++ kpod=logging-kibana-1-jg9t8
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es project.logging _search message 60cf9a0c-f2d5-455b-823d-74a8878d64a1
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:60cf9a0c-f2d5-455b-823d-74a8878d64a1'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es-ops .operations _search message bc4aa684-9f7d-4054-bcff-ea6fa8b291f2
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:bc4aa684-9f7d-4054-bcff-ea6fa8b291f2'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_UNDEFINED_NAME myname
+++ mktemp
++ junk=/tmp/tmp.dQZ2PdIwRR
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.dQZ2PdIwRR'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.dQZ2PdIwRR
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-v86b9
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-v86b9
++ '[' -z logging-fluentd-v86b9 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-v86b9
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
+++ oc get pods -l component=fluentd
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-m9zlg
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-m9zlg ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-m9zlg
++ write_and_verify_logs test4
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496959690
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 22:08:10.098572502+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 22:08:10.098572502+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-nh0txix5-1-psfc8
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ '[' -z logging-es-ops-data-master-kw4v1jbu-1-h8r1q ']'
+++ uuidgen
++ local uuid_es=3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ uuidgen
++ local uuid_es_ops=02851942-1d45-4547-907b-08a7b152ad92
++ local expected=1
++ local timeout=300
++ add_test_message 3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-jg9t8
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/3bee3368-9972-49ab-8e9d-a4b428e44e70
++ echo added es message 3bee3368-9972-49ab-8e9d-a4b428e44e70
added es message 3bee3368-9972-49ab-8e9d-a4b428e44e70
++ logger -i -p local6.info -t 02851942-1d45-4547-907b-08a7b152ad92 02851942-1d45-4547-907b-08a7b152ad92
added es-ops message 02851942-1d45-4547-907b-08a7b152ad92
++ echo added es-ops message 02851942-1d45-4547-907b-08a7b152ad92
++ local rc=0
++ espod=logging-es-data-master-nh0txix5-1-psfc8
++ myproject=project.logging
++ mymessage=3bee3368-9972-49ab-8e9d-a4b428e44e70
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 3bee3368-9972-49ab-8e9d-a4b428e44e70
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 293 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 3bee3368-9972-49ab-8e9d-a4b428e44e70
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 3bee3368-9972-49ab-8e9d-a4b428e44e70
++ espod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ myproject=.operations
++ mymessage=02851942-1d45-4547-907b-08a7b152ad92
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER 02851942-1d45-4547-907b-08a7b152ad92
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER 02851942-1d45-4547-907b-08a7b152ad92
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER 02851942-1d45-4547-907b-08a7b152ad92
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER 02851942-1d45-4547-907b-08a7b152ad92
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:02851942-1d45-4547-907b-08a7b152ad92'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 297 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 02851942-1d45-4547-907b-08a7b152ad92
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 02851942-1d45-4547-907b-08a7b152ad92
++ '[' -n get_logmessage ']'
++ get_logmessage 3bee3368-9972-49ab-8e9d-a4b428e44e70
++ logmessage=3bee3368-9972-49ab-8e9d-a4b428e44e70
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 02851942-1d45-4547-907b-08a7b152ad92
++ logmessage2=02851942-1d45-4547-907b-08a7b152ad92
+++ date +%s
++ local endtime=1496959705
+++ expr 1496959705 - 1496959690
+++ date -u --rfc-3339=ns
++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-08 22:08:25.905279127+00:00
END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-08 22:08:25.905279127+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-jg9t8
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es project.logging _search message 3bee3368-9972-49ab-8e9d-a4b428e44e70
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:3bee3368-9972-49ab-8e9d-a4b428e44e70'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es-ops .operations _search message 02851942-1d45-4547-907b-08a7b152ad92
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:02851942-1d45-4547-907b-08a7b152ad92'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,empty1,undefined3,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.SPm2QLuW4s
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.SPm2QLuW4s'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.SPm2QLuW4s
++ add_cdm_env_var_val CDM_KEEP_EMPTY_FIELDS undefined4,undefined5,empty1,undefined3
+++ mktemp
++ junk=/tmp/tmp.43LALPXVY3
++ cat
++ oc get template logging-fluentd-template -o yaml
++ oc replace -f -
++ sed '/env:/r /tmp/tmp.43LALPXVY3'
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.43LALPXVY3
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-m9zlg
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-m9zlg
++ '[' -z logging-fluentd-m9zlg -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-m9zlg
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-70n8c
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-70n8c ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-70n8c
++ write_and_verify_logs test5 allow_empty
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496959729
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 22:08:49.723871851+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 22:08:49.723871851+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-nh0txix5-1-psfc8
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ '[' -z logging-es-ops-data-master-kw4v1jbu-1-h8r1q ']'
+++ uuidgen
++ local uuid_es=9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ uuidgen
++ local uuid_es_ops=593d06ec-4aab-4b35-b2c8-d1711895e526
++ local expected=1
++ local timeout=300
++ add_test_message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-jg9t8
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/9cb1e56d-6610-44f1-b3c2-d16651b4b12f
++ echo added es message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
added es message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
++ logger -i -p local6.info -t 593d06ec-4aab-4b35-b2c8-d1711895e526 593d06ec-4aab-4b35-b2c8-d1711895e526
added es-ops message 593d06ec-4aab-4b35-b2c8-d1711895e526
++ echo added es-ops message 593d06ec-4aab-4b35-b2c8-d1711895e526
++ local rc=0
++ espod=logging-es-data-master-nh0txix5-1-psfc8
++ myproject=project.logging
++ mymessage=9cb1e56d-6610-44f1-b3c2-d16651b4b12f
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ get_count_from_json
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-nh0txix5-1-psfc8 project.logging _count message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
+++ curl_es logging-es-data-master-nh0txix5-1-psfc8 '/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f' --connect-timeout 1
+++ local pod=logging-es-data-master-nh0txix5-1-psfc8
+++ local 'endpoint=/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ get_count_from_json
+++ oc exec logging-es-data-master-nh0txix5-1-psfc8 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 293 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
++ espod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
++ myproject=.operations
++ mymessage=593d06ec-4aab-4b35-b2c8-d1711895e526
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q .operations _count systemd.u.SYSLOG_IDENTIFIER 593d06ec-4aab-4b35-b2c8-d1711895e526
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-kw4v1jbu-1-h8r1q '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:593d06ec-4aab-4b35-b2c8-d1711895e526' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-kw4v1jbu-1-h8r1q
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:593d06ec-4aab-4b35-b2c8-d1711895e526'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-kw4v1jbu-1-h8r1q -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:593d06ec-4aab-4b35-b2c8-d1711895e526'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 593d06ec-4aab-4b35-b2c8-d1711895e526
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 593d06ec-4aab-4b35-b2c8-d1711895e526
++ '[' -n get_logmessage ']'
++ get_logmessage 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
++ logmessage=9cb1e56d-6610-44f1-b3c2-d16651b4b12f
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 593d06ec-4aab-4b35-b2c8-d1711895e526
++ logmessage2=593d06ec-4aab-4b35-b2c8-d1711895e526
+++ date +%s
++ local endtime=1496959741
+++ expr 1496959741 - 1496959729
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 12 seconds at 2017-06-08 22:09:01.456910743+00:00
++ echo END wait_for_fluentd_to_catch_up took 12 seconds at 2017-06-08 22:09:01.456910743+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-jg9t8
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es project.logging _search message 9cb1e56d-6610-44f1-b3c2-d16651b4b12f
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:9cb1e56d-6610-44f1-b3c2-d16651b4b12f'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-jg9t8 logging-es-ops .operations _search message 593d06ec-4aab-4b35-b2c8-d1711895e526
++ oc exec logging-kibana-1-jg9t8 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bUkbJLQX-5PA5Zxr0POXhpyVK2i4z1CtyPR-eteXA3E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:593d06ec-4aab-4b35-b2c8-d1711895e526'
++ python test-viaq-data-model.py test5 allow_empty
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ cleanup
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
template "logging-fluentd-template" replaced
++ remove_cdm_env
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.poloaJ2AK9
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-70n8c
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-70n8c
++ '[' -z logging-fluentd-70n8c -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-70n8c
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-0wwf4
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-0wwf4 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
SKIPPING reinstall test for now
/data/src/github.com/openshift/origin-aggregated-logging/hack/lib/log/system.sh: line 31:  4607 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"  (wd: /data/src/github.com/openshift/origin-aggregated-logging)
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to /tmp/origin-aggregated-logging/artifacts/events.txt
[INFO] [CLEANUP] Dumping etcd contents to /tmp/origin-aggregated-logging/artifacts/etcd
[WARNING] No compiled `etcdhelper` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh tools/etcdhelper
++ Building go targets for linux/amd64: tools/etcdhelper
/data/src/github.com/openshift/origin-aggregated-logging/../origin/hack/build-go.sh took 110 seconds
2017-06-08 18:11:25.490241 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
[INFO] [CLEANUP] Dumping container logs to /tmp/origin-aggregated-logging/logs/containers
[INFO] [CLEANUP] Truncating log files over 200M
[INFO] [CLEANUP] Stopping docker containers
[INFO] [CLEANUP] Removing docker containers
Error response from daemon: You cannot remove a running container c1a1185338d2a75d79d4c1224236f278fb48a4d5fa498d44f78e6cbd414d946d. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container f437255b8c8c602272b0890c0fb02f74e81c3fa26236460ec85fdb02e028e125. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 9602e26a4981150f161a62f203d3c161e83167b39a49de0e913c4509be03a002. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 26908dd9d66dc62381a593cbf8599b01040d7641080bac7032d1fb07335f1a6f. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 7bc65b63f052281d95ff7dc89c9bca42a504804e74d60608434bc1de7fe13611. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 349f74594f14626bdd565c4cd7b4f3026eff6945fec3134a5931c2820f362874. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 5afae608915e332198bbfcc323e8fc56e80fa4d65e5d8844e7d92c78449b4306. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container f07af443777ed409dd855d1345e5ec1934ea78b678e96b3b979884eed8d0cb5c. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container cd45499f0bc4ef6a7e891e03fed3691fbce55130fc9baf2e7f76dae67c9c9e19. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container a631ed27fa0f6454ea87498c7a77a06a3d044b07be7dec6c267f57d43b4bcb24. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 5dbb3e957aa0854bc6b5b0255104c8079a70438dfba62311dc2b07b6778f8717. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 3a3bf6c2d5093b72108f11fd7cfec70e6a5c590b55c4776681e036950dcd0c31. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 8be8b5b3fc2efa6c179c1b74755dfd331f823820f6c09c2d8a77584523ed375a. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 19451b08b42b963e4fd9bc1dff929e742e27df9a94c7c86284712bac77d4ab70. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container d594a193c5ade671f9ff74ab06c4855a4f92fafe341fc5152a6568f7ee2f8b47. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 056ac3bb38917b786bc43ed5abd4397aa033596b4ae7777272c7f65be17b7a2d. Stop the container before attempting removal or use -f
[INFO] [CLEANUP] Killing child processes
[INFO] [CLEANUP] Pruning etcd data directory
[INFO] /data/src/github.com/openshift/origin-aggregated-logging/logging.sh exited with code 0 after 00h 57m 03s
Finished GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh
***************************************************

real	57m2.570s
user	5m30.852s
sys	1m5.691s
==> openshiftdev: Downloading logs
==> openshiftdev: Downloading artifacts from '/var/log/yum.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/yum.log'
==> openshiftdev: Downloading artifacts from '/var/log/secure' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/secure'
==> openshiftdev: Downloading artifacts from '/var/log/audit/audit.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/audit.log'
==> openshiftdev: Downloading artifacts from '/tmp/origin-aggregated-logging/' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts'
+ false
+ false
+ test_pull_requests --mark_test_success --repo origin-aggregated-logging --config /var/lib/jenkins/.test_pull_requests_logging.json
Rate limit remaining: 3377
  Marking SUCCESS for pull request #--repo in repo ''
/usr/share/ruby/net/http/response.rb:119:in `error!': 404 "Not Found" (Net::HTTPServerException)
	from /bin/test_pull_requests:543:in `block in get_comments'
	from /bin/test_pull_requests:535:in `each'
	from /bin/test_pull_requests:535:in `get_comments'
	from /bin/test_pull_requests:1074:in `get_comment_matching_regex'
	from /bin/test_pull_requests:1090:in `get_comment_with_prefix'
	from /bin/test_pull_requests:779:in `mark_test_success'
	from /bin/test_pull_requests:2361:in `<main>'
+ true
[description-setter] Could not determine description.
[PostBuildScript] - Execution post build scripts.
[workspace@2] $ /bin/sh -xe /tmp/hudson3870563122715187706.sh
+ INSTANCE_NAME=origin_logging-rhel7-1639
+ pushd origin
~/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2
+ rc=0
+ '[' -f .vagrant-openshift.json ']'
++ /usr/bin/vagrant ssh -c 'sudo ausearch -m avc'
+ ausearchresult='<no matches>'
+ rc=1
+ '[' '<no matches>' = '<no matches>' ']'
+ rc=0
+ /usr/bin/vagrant destroy -f
==> openshiftdev: Terminating the instance...
==> openshiftdev: Running cleanup tasks for 'shell' provisioner...
+ popd
~/jobs/test-origin-aggregated-logging/workspace@2
+ exit 0
Finished: SUCCESS