SuccessConsole Output

Skipping 597 KB.. Full Log
ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 255 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 254 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 253 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 252 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 251 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 250 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 249 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 248 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 247 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 246 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6106ea2f-c9ec-4193-ac06-66762bcd8b0e'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 246 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 6106ea2f-c9ec-4193-ac06-66762bcd8b0e
++ espod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ myproject=.operations
++ mymessage=eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 297 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for eae7bc77-bfe8-454d-ba0f-6a44e1c1ba39
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496966031
+++ expr 1496966031 - 1496965950
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 81 seconds at 2017-06-08 23:53:51.788690307+00:00
++ echo END wait_for_fluentd_to_catch_up took 81 seconds at 2017-06-08 23:53:51.788690307+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.13.16" labeled
++ wait_for_pod_ACTION stop logging-fluentd-hq1lx
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-hq1lx
++ '[' -z logging-fluentd-hq1lx -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-hq1lx
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
daemonset "logging-forward-fluentd" deleted
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '      <match **>' ']'
++ oc get configmap/logging-fluentd -o yaml
++ sed -e '/<match \*\*>/ d' -e '/@include configs\.d\/user\/secure-forward\.conf/ d' -e '/<\/match>/ d'
++ oc replace -f -
configmap "logging-fluentd" replaced
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.13.16" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-1lhxf
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-1lhxf ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-1lhxf
++ oc get events -o yaml
++ write_and_verify_logs 1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up '' ''
+++ date +%s
++ local starttime=1496966046
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 23:54:06.326008504+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 23:54:06.326008504+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ '[' -z logging-es-ops-data-master-hkbpoe6l-1-20qsg ']'
+++ uuidgen
++ local uuid_es=3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ uuidgen
++ local uuid_es_ops=edba78a5-27ad-404b-9269-f169f64fbfb2
++ local expected=1
++ local timeout=300
++ add_test_message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-vqfs4
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/3910db15-7ddc-4dcc-83f8-5ecc4742f355
added es message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
++ echo added es message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
++ logger -i -p local6.info -t edba78a5-27ad-404b-9269-f169f64fbfb2 edba78a5-27ad-404b-9269-f169f64fbfb2
++ echo added es-ops message edba78a5-27ad-404b-9269-f169f64fbfb2
added es-ops message edba78a5-27ad-404b-9269-f169f64fbfb2
++ local rc=0
++ espod=logging-es-data-master-52f7bjgb-1-26h6x
++ myproject=project.logging
++ mymessage=3910db15-7ddc-4dcc-83f8-5ecc4742f355
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 289 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 288 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 287 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 286 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 3910db15-7ddc-4dcc-83f8-5ecc4742f355
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3910db15-7ddc-4dcc-83f8-5ecc4742f355'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 286 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 3910db15-7ddc-4dcc-83f8-5ecc4742f355
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 3910db15-7ddc-4dcc-83f8-5ecc4742f355
++ espod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ myproject=.operations
++ mymessage=edba78a5-27ad-404b-9269-f169f64fbfb2
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER edba78a5-27ad-404b-9269-f169f64fbfb2
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:edba78a5-27ad-404b-9269-f169f64fbfb2' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:edba78a5-27ad-404b-9269-f169f64fbfb2'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:edba78a5-27ad-404b-9269-f169f64fbfb2'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for edba78a5-27ad-404b-9269-f169f64fbfb2
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for edba78a5-27ad-404b-9269-f169f64fbfb2
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496966067
+++ expr 1496966067 - 1496966046
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 21 seconds at 2017-06-08 23:54:27.662772318+00:00
++ echo END wait_for_fluentd_to_catch_up took 21 seconds at 2017-06-08 23:54:27.662772318+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.13.16" labeled
++ wait_for_pod_ACTION stop logging-fluentd-1lhxf
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-1lhxf
++ '[' -z logging-fluentd-1lhxf -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-1lhxf
++ '[' -n 1 ']'
pod logging-fluentd-1lhxf still running
++ echo pod logging-fluentd-1lhxf still running
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' stop = start ']'
++ '[' 110 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-1lhxf
++ '[' stop = start ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
Error from server (NotFound): daemonsets.extensions "logging-forward-fluentd" not found
++ :
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '' ']'
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.13.16" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-c5jsc
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-c5jsc ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-c5jsc
++ oc get events -o yaml
running test test-json-parsing.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ es_pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ uuidgen
++ uuid_es=793c505c-472b-4a9a-8c54-f0675a5d6ad6
++ echo Adding test message 793c505c-472b-4a9a-8c54-f0675a5d6ad6 to Kibana . . .
Adding test message 793c505c-472b-4a9a-8c54-f0675a5d6ad6 to Kibana . . .
++ add_test_message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-vqfs4
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/793c505c-472b-4a9a-8c54-f0675a5d6ad6
++ rc=0
++ timeout=600
++ echo Waiting 600 seconds for 793c505c-472b-4a9a-8c54-f0675a5d6ad6 to show up in Elasticsearch . . .
Waiting 600 seconds for 793c505c-472b-4a9a-8c54-f0675a5d6ad6 to show up in Elasticsearch . . .
++ espod=logging-es-data-master-52f7bjgb-1-26h6x
++ myproject=project.logging.
++ mymessage=793c505c-472b-4a9a-8c54-f0675a5d6ad6
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 600
++ let ii=600
++ local interval=1
++ '[' 600 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _count message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 599 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _count message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 598 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _count message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 597 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _count message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 596 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _count message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 595 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _count message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 594 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _count message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 593 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _count message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 592 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _count message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 592 -le 0 ']'
++ return 0
good - ./logging.sh: found 1 record project logging for 793c505c-472b-4a9a-8c54-f0675a5d6ad6
++ echo good - ./logging.sh: found 1 record project logging for 793c505c-472b-4a9a-8c54-f0675a5d6ad6
Testing if record is in correct format . . .
++ echo Testing if record is in correct format . . .
++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging. _search message 793c505c-472b-4a9a-8c54-f0675a5d6ad6
++ python test-json-parsing.py 793c505c-472b-4a9a-8c54-f0675a5d6ad6
++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging.*/_search?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6' --connect-timeout 1
++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
++ local 'endpoint=/project.logging.*/_search?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
++ shift
++ shift
++ args=("${@:-}")
++ local args
++ local secret_dir=/etc/elasticsearch/secret/
++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_search?q=message:793c505c-472b-4a9a-8c54-f0675a5d6ad6'
Success: record contains all of the expected fields/values
Success: ./logging.sh passed
++ echo Success: ./logging.sh passed
++ exit 0
running test test-mux.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ '[' false == false -o false == false ']'
Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.
++ echo 'Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.'
++ exit 0
SKIPPING upgrade test for now
running test test-viaq-data-model.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ [[ 1 -ne 1 ]]
++ [[ true = \f\a\l\s\e ]]
++ CLUSTER=true
++ ops=-ops
++ INDEX_PREFIX=
++ PROJ_PREFIX=project.
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
++ get_test_user_token
++ local current_project
+++ oc project -q
++ current_project=logging
++ oc login --username=admin --password=admin
+++ oc whoami -t
++ test_token=bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4
+++ oc whoami
++ test_name=admin
++ test_ip=127.0.0.1
++ oc login --username=system:admin
++ oc project logging
++ TEST_DIVIDER=------------------------------------------
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-c5jsc
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
+++ mktemp
++ cfg=/tmp/tmp.7bNHeOSuNr
++ cat
++ add_test_volume /tmp/tmp.7bNHeOSuNr
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"].append({"name": "cdmtest", "mountPath": "/etc/fluent/configs.d/openshift/filter-pre-cdm-test.conf", "readOnly": True}); obj["objects"][0]["spec"]["template"]["spec"]["volumes"].append({"name": "cdmtest", "hostPath": {"path": "/tmp/tmp.7bNHeOSuNr"}}); print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ trap cleanup INT TERM EXIT
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-c5jsc
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-c5jsc
++ '[' -z logging-fluentd-c5jsc -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-c5jsc
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-11f0t
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-11f0t ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-11f0t
++ keep_fields=method,statusCode,type,@timestamp,req,res
++ write_and_verify_logs test1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496966123
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 23:55:23.275614105+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 23:55:23.275614105+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ '[' -z logging-es-ops-data-master-hkbpoe6l-1-20qsg ']'
+++ uuidgen
++ local uuid_es=fff516c9-022d-4ea4-9324-4c35877d9315
+++ uuidgen
++ local uuid_es_ops=1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
++ local expected=1
++ local timeout=300
++ add_test_message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-vqfs4
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/fff516c9-022d-4ea4-9324-4c35877d9315
added es message fff516c9-022d-4ea4-9324-4c35877d9315
++ echo added es message fff516c9-022d-4ea4-9324-4c35877d9315
++ logger -i -p local6.info -t 1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d 1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
added es-ops message 1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
++ echo added es-ops message 1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
++ local rc=0
++ espod=logging-es-data-master-52f7bjgb-1-26h6x
++ myproject=project.logging
++ mymessage=fff516c9-022d-4ea4-9324-4c35877d9315
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message fff516c9-022d-4ea4-9324-4c35877d9315
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 291 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for fff516c9-022d-4ea4-9324-4c35877d9315
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for fff516c9-022d-4ea4-9324-4c35877d9315
++ espod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ myproject=.operations
++ mymessage=1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER 1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
++ '[' -n get_logmessage ']'
++ get_logmessage fff516c9-022d-4ea4-9324-4c35877d9315
++ logmessage=fff516c9-022d-4ea4-9324-4c35877d9315
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
++ logmessage2=1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
+++ date +%s
++ local endtime=1496966137
+++ expr 1496966137 - 1496966123
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-08 23:55:37.569668117+00:00
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-08 23:55:37.569668117+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-vqfs4
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es project.logging _search message fff516c9-022d-4ea4-9324-4c35877d9315
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:fff516c9-022d-4ea4-9324-4c35877d9315'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es-ops .operations _search message 1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:1467e76b-76a7-4241-b5c2-c5ee0d2e8e8d'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_USE_UNDEFINED '"true"'
+++ mktemp
++ junk=/tmp/tmp.IGR5U0CkJ9
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.IGR5U0CkJ9'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.IGR5U0CkJ9
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.zP70QctVcF
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.zP70QctVcF'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.zP70QctVcF
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-11f0t
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-11f0t
++ '[' -z logging-fluentd-11f0t -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-11f0t
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-rc9jb
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-rc9jb ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-rc9jb
++ write_and_verify_logs test2
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496966162
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 23:56:02.910220783+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 23:56:02.910220783+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ '[' -z logging-es-ops-data-master-hkbpoe6l-1-20qsg ']'
+++ uuidgen
++ local uuid_es=85fba70c-3780-4f2c-b01e-d7223c295439
+++ uuidgen
++ local uuid_es_ops=185932a3-2d04-4ee3-91cc-f37efe3517f7
++ local expected=1
++ local timeout=300
++ add_test_message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-vqfs4
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/85fba70c-3780-4f2c-b01e-d7223c295439
added es message 85fba70c-3780-4f2c-b01e-d7223c295439
++ echo added es message 85fba70c-3780-4f2c-b01e-d7223c295439
++ logger -i -p local6.info -t 185932a3-2d04-4ee3-91cc-f37efe3517f7 185932a3-2d04-4ee3-91cc-f37efe3517f7
added es-ops message 185932a3-2d04-4ee3-91cc-f37efe3517f7
++ echo added es-ops message 185932a3-2d04-4ee3-91cc-f37efe3517f7
++ local rc=0
++ espod=logging-es-data-master-52f7bjgb-1-26h6x
++ myproject=project.logging
++ mymessage=85fba70c-3780-4f2c-b01e-d7223c295439
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 85fba70c-3780-4f2c-b01e-d7223c295439
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 291 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 85fba70c-3780-4f2c-b01e-d7223c295439
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 85fba70c-3780-4f2c-b01e-d7223c295439
++ espod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ myproject=.operations
++ mymessage=185932a3-2d04-4ee3-91cc-f37efe3517f7
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER 185932a3-2d04-4ee3-91cc-f37efe3517f7
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:185932a3-2d04-4ee3-91cc-f37efe3517f7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:185932a3-2d04-4ee3-91cc-f37efe3517f7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:185932a3-2d04-4ee3-91cc-f37efe3517f7'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 185932a3-2d04-4ee3-91cc-f37efe3517f7
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 185932a3-2d04-4ee3-91cc-f37efe3517f7
++ '[' -n get_logmessage ']'
++ get_logmessage 85fba70c-3780-4f2c-b01e-d7223c295439
++ logmessage=85fba70c-3780-4f2c-b01e-d7223c295439
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 185932a3-2d04-4ee3-91cc-f37efe3517f7
++ logmessage2=185932a3-2d04-4ee3-91cc-f37efe3517f7
+++ date +%s
++ local endtime=1496966177
+++ expr 1496966177 - 1496966162
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-08 23:56:17.174893792+00:00
++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-08 23:56:17.174893792+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-vqfs4
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es project.logging _search message 85fba70c-3780-4f2c-b01e-d7223c295439
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:85fba70c-3780-4f2c-b01e-d7223c295439'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es-ops .operations _search message 185932a3-2d04-4ee3-91cc-f37efe3517f7
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:185932a3-2d04-4ee3-91cc-f37efe3517f7'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.IM3KpeNQzt
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.IM3KpeNQzt'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.IM3KpeNQzt
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-rc9jb
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-rc9jb
++ '[' -z logging-fluentd-rc9jb -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-rc9jb
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
+++ oc get pods -l component=fluentd
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-zhbv5
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-zhbv5 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-zhbv5
++ write_and_verify_logs test3
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496966195
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 23:56:35.406171836+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 23:56:35.406171836+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ '[' -z logging-es-ops-data-master-hkbpoe6l-1-20qsg ']'
+++ uuidgen
++ local uuid_es=d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ uuidgen
++ local uuid_es_ops=d7278a8f-78c4-460a-9931-58fb1a14ba2d
++ local expected=1
++ local timeout=300
++ add_test_message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-vqfs4
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/d266c3ed-25b1-4751-a165-9ae4c3c88436
++ echo added es message d266c3ed-25b1-4751-a165-9ae4c3c88436
added es message d266c3ed-25b1-4751-a165-9ae4c3c88436
++ logger -i -p local6.info -t d7278a8f-78c4-460a-9931-58fb1a14ba2d d7278a8f-78c4-460a-9931-58fb1a14ba2d
added es-ops message d7278a8f-78c4-460a-9931-58fb1a14ba2d
++ echo added es-ops message d7278a8f-78c4-460a-9931-58fb1a14ba2d
++ local rc=0
++ espod=logging-es-data-master-52f7bjgb-1-26h6x
++ myproject=project.logging
++ mymessage=d266c3ed-25b1-4751-a165-9ae4c3c88436
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
+++ shift
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message d266c3ed-25b1-4751-a165-9ae4c3c88436
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 292 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for d266c3ed-25b1-4751-a165-9ae4c3c88436
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for d266c3ed-25b1-4751-a165-9ae4c3c88436
++ espod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ myproject=.operations
++ mymessage=d7278a8f-78c4-460a-9931-58fb1a14ba2d
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER d7278a8f-78c4-460a-9931-58fb1a14ba2d
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d7278a8f-78c4-460a-9931-58fb1a14ba2d' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d7278a8f-78c4-460a-9931-58fb1a14ba2d'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d7278a8f-78c4-460a-9931-58fb1a14ba2d'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for d7278a8f-78c4-460a-9931-58fb1a14ba2d
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for d7278a8f-78c4-460a-9931-58fb1a14ba2d
++ '[' -n get_logmessage ']'
++ get_logmessage d266c3ed-25b1-4751-a165-9ae4c3c88436
++ logmessage=d266c3ed-25b1-4751-a165-9ae4c3c88436
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 d7278a8f-78c4-460a-9931-58fb1a14ba2d
++ logmessage2=d7278a8f-78c4-460a-9931-58fb1a14ba2d
+++ date +%s
++ local endtime=1496966208
+++ expr 1496966208 - 1496966195
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-08 23:56:48.141270349+00:00
++ echo END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-08 23:56:48.141270349+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-vqfs4
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es project.logging _search message d266c3ed-25b1-4751-a165-9ae4c3c88436
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:d266c3ed-25b1-4751-a165-9ae4c3c88436'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es-ops .operations _search message d7278a8f-78c4-460a-9931-58fb1a14ba2d
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:d7278a8f-78c4-460a-9931-58fb1a14ba2d'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_UNDEFINED_NAME myname
+++ mktemp
++ junk=/tmp/tmp.mTjfdljCsT
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.mTjfdljCsT'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.mTjfdljCsT
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-zhbv5
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-zhbv5
++ '[' -z logging-fluentd-zhbv5 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-zhbv5
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-2br3n
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-2br3n ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-2br3n
++ write_and_verify_logs test4
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496966232
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 23:57:12.898477137+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 23:57:12.898477137+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ '[' -z logging-es-ops-data-master-hkbpoe6l-1-20qsg ']'
+++ uuidgen
++ local uuid_es=e223d1f1-dc4e-43ad-9676-0588d549e962
+++ uuidgen
++ local uuid_es_ops=2d708dae-cff8-4500-8451-68e8ed646dc7
++ local expected=1
++ local timeout=300
++ add_test_message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-vqfs4
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/e223d1f1-dc4e-43ad-9676-0588d549e962
added es message e223d1f1-dc4e-43ad-9676-0588d549e962
++ echo added es message e223d1f1-dc4e-43ad-9676-0588d549e962
++ logger -i -p local6.info -t 2d708dae-cff8-4500-8451-68e8ed646dc7 2d708dae-cff8-4500-8451-68e8ed646dc7
added es-ops message 2d708dae-cff8-4500-8451-68e8ed646dc7
++ echo added es-ops message 2d708dae-cff8-4500-8451-68e8ed646dc7
++ local rc=0
++ espod=logging-es-data-master-52f7bjgb-1-26h6x
++ myproject=project.logging
++ mymessage=e223d1f1-dc4e-43ad-9676-0588d549e962
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message e223d1f1-dc4e-43ad-9676-0588d549e962
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 291 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for e223d1f1-dc4e-43ad-9676-0588d549e962
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for e223d1f1-dc4e-43ad-9676-0588d549e962
++ espod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ myproject=.operations
++ mymessage=2d708dae-cff8-4500-8451-68e8ed646dc7
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER 2d708dae-cff8-4500-8451-68e8ed646dc7
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:2d708dae-cff8-4500-8451-68e8ed646dc7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:2d708dae-cff8-4500-8451-68e8ed646dc7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:2d708dae-cff8-4500-8451-68e8ed646dc7'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 2d708dae-cff8-4500-8451-68e8ed646dc7
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 2d708dae-cff8-4500-8451-68e8ed646dc7
++ '[' -n get_logmessage ']'
++ get_logmessage e223d1f1-dc4e-43ad-9676-0588d549e962
++ logmessage=e223d1f1-dc4e-43ad-9676-0588d549e962
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 2d708dae-cff8-4500-8451-68e8ed646dc7
++ logmessage2=2d708dae-cff8-4500-8451-68e8ed646dc7
+++ date +%s
++ local endtime=1496966247
+++ expr 1496966247 - 1496966232
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-08 23:57:27.059703018+00:00
++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-08 23:57:27.059703018+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-vqfs4
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es project.logging _search message e223d1f1-dc4e-43ad-9676-0588d549e962
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:e223d1f1-dc4e-43ad-9676-0588d549e962'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es-ops .operations _search message 2d708dae-cff8-4500-8451-68e8ed646dc7
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:2d708dae-cff8-4500-8451-68e8ed646dc7'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,empty1,undefined3,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.GuekcHdE3A
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.GuekcHdE3A'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.GuekcHdE3A
++ add_cdm_env_var_val CDM_KEEP_EMPTY_FIELDS undefined4,undefined5,empty1,undefined3
+++ mktemp
++ junk=/tmp/tmp.SKLDkpBb8D
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.SKLDkpBb8D'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.SKLDkpBb8D
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-2br3n
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-2br3n
++ '[' -z logging-fluentd-2br3n -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-2br3n
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-b4zm6
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-b4zm6 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-b4zm6
++ write_and_verify_logs test5 allow_empty
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496966272
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-08 23:57:52.698345983+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-08 23:57:52.698345983+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ '[' -z logging-es-ops-data-master-hkbpoe6l-1-20qsg ']'
+++ uuidgen
++ local uuid_es=0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ uuidgen
++ local uuid_es_ops=90708abd-1adc-45ba-9495-f6372d016c1d
++ local expected=1
++ local timeout=300
++ add_test_message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-vqfs4
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
added es message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
++ echo added es message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
++ logger -i -p local6.info -t 90708abd-1adc-45ba-9495-f6372d016c1d 90708abd-1adc-45ba-9495-f6372d016c1d
added es-ops message 90708abd-1adc-45ba-9495-f6372d016c1d
++ echo added es-ops message 90708abd-1adc-45ba-9495-f6372d016c1d
++ local rc=0
++ espod=logging-es-data-master-52f7bjgb-1-26h6x
++ myproject=project.logging
++ mymessage=0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-52f7bjgb-1-26h6x project.logging _count message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
+++ get_count_from_json
+++ curl_es logging-es-data-master-52f7bjgb-1-26h6x '/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-52f7bjgb-1-26h6x
+++ local 'endpoint=/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-52f7bjgb-1-26h6x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 291 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
++ espod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
++ myproject=.operations
++ mymessage=90708abd-1adc-45ba-9495-f6372d016c1d
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-hkbpoe6l-1-20qsg .operations _count systemd.u.SYSLOG_IDENTIFIER 90708abd-1adc-45ba-9495-f6372d016c1d
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-hkbpoe6l-1-20qsg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:90708abd-1adc-45ba-9495-f6372d016c1d' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-hkbpoe6l-1-20qsg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:90708abd-1adc-45ba-9495-f6372d016c1d'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-hkbpoe6l-1-20qsg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:90708abd-1adc-45ba-9495-f6372d016c1d'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 90708abd-1adc-45ba-9495-f6372d016c1d
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 90708abd-1adc-45ba-9495-f6372d016c1d
++ '[' -n get_logmessage ']'
++ get_logmessage 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
++ logmessage=0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 90708abd-1adc-45ba-9495-f6372d016c1d
++ logmessage2=90708abd-1adc-45ba-9495-f6372d016c1d
+++ date +%s
++ local endtime=1496966286
+++ expr 1496966286 - 1496966272
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-08 23:58:06.946066273+00:00
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-08 23:58:06.946066273+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-vqfs4
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es project.logging _search message 0e6fe5f9-a496-4822-ba3f-e38bf1c31c93
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:0e6fe5f9-a496-4822-ba3f-e38bf1c31c93'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-vqfs4 logging-es-ops .operations _search message 90708abd-1adc-45ba-9495-f6372d016c1d
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-vqfs4 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer bsyiPZ6YQcvqHsIEbGxgUk5X_nF5iPDFjQkB18BK1i4' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:90708abd-1adc-45ba-9495-f6372d016c1d'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ cleanup
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
template "logging-fluentd-template" replaced
++ remove_cdm_env
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.7bNHeOSuNr
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-b4zm6
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-b4zm6
++ '[' -z logging-fluentd-b4zm6 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-b4zm6
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-3f79x
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-3f79x ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
SKIPPING reinstall test for now
/data/src/github.com/openshift/origin-aggregated-logging/hack/lib/log/system.sh: line 31:  4609 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"  (wd: /data/src/github.com/openshift/origin-aggregated-logging)
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to /tmp/origin-aggregated-logging/artifacts/events.txt
[INFO] [CLEANUP] Dumping etcd contents to /tmp/origin-aggregated-logging/artifacts/etcd
[WARNING] No compiled `etcdhelper` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh tools/etcdhelper
++ Building go targets for linux/amd64: tools/etcdhelper
/data/src/github.com/openshift/origin-aggregated-logging/../origin/hack/build-go.sh took 158 seconds
2017-06-08 20:01:16.859295 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
[INFO] [CLEANUP] Dumping container logs to /tmp/origin-aggregated-logging/logs/containers
[INFO] [CLEANUP] Truncating log files over 200M
[INFO] [CLEANUP] Stopping docker containers
[INFO] [CLEANUP] Removing docker containers
Error response from daemon: You cannot remove a running container 9f19282522d6e927182aa45295810539638639b988b6cf0745c65e3b9a659d3f. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 789a10e4f01c0903a3aa895ce0219e18460fcecbce73391fbf2c29b6d504e923. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 0bc3047107250920a39eb1b51f1f3f2e19de4f9cd889c36fe2ed61a1aeed6d62. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 38c41de81987737de3f649d061c55e5b02520af73757f872ceb7f61c3585934e. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container eca8819b5f2a2104b950654ae275060e14e9c9253fe9b00f3a378afa7b89d617. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 4d8e0d98fee21b517c30423ac7a35c2c73bbe8484a91f479cea5f5b435ad566f. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 9b82247fd72577772d8360a7e024a92e483c6201c2aef529a2b6417cbd81515d. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container b90fff11d1a7b01b1234ccca2f6d73244932a0c5d5290068a6db264f6e127912. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 3dbdd3a939b629175d64e7b35d98250ca8d60b697ccfa174b4bfc6ceb7bcc046. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 8713d3b5c4223e7303629540012e5064e000fd974645b4227943a9c2ef82c376. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 894afedc4a36fa21a11f1ec5bf477a58cb5a287cefe066419ccbbc266700c705. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container dc2f908634ef081347274d2e4ace2322b4d39af61714bb0a0a06f2f2f89d5e2e. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 245fe9f690ac881643e2a77ac3549578f430ccd9969d3670579953b147759402. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 8872ba13db5cb64cf14c8d0dc3e85a780304271a0f5713403398b92f06818851. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container bb6fb1841f9e666e16b995599a7607381ab69257f3674a66c1c22664e8dfb149. Stop the container before attempting removal or use -f
[INFO] [CLEANUP] Killing child processes
[INFO] [CLEANUP] Pruning etcd data directory
[INFO] /data/src/github.com/openshift/origin-aggregated-logging/logging.sh exited with code 0 after 01h 22m 55s

real	82m55.851s
user	6m3.086s
sys	1m6.212s
Finished GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh
***************************************************
==> openshiftdev: Downloading logs
==> openshiftdev: Downloading artifacts from '/var/log/yum.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/yum.log'
==> openshiftdev: Downloading artifacts from '/var/log/secure' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/secure'
==> openshiftdev: Downloading artifacts from '/var/log/audit/audit.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/audit.log'
==> openshiftdev: Downloading artifacts from '/tmp/origin-aggregated-logging/' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts'
+ false
+ false
+ test_pull_requests --mark_test_success --repo origin-aggregated-logging --config /var/lib/jenkins/.test_pull_requests_logging.json
Rate limit remaining: 4728
  Marking SUCCESS for pull request #--repo in repo ''
/usr/share/ruby/net/http/response.rb:119:in `error!': 404 "Not Found" (Net::HTTPServerException)
	from /bin/test_pull_requests:543:in `block in get_comments'
	from /bin/test_pull_requests:535:in `each'
	from /bin/test_pull_requests:535:in `get_comments'
	from /bin/test_pull_requests:1074:in `get_comment_matching_regex'
	from /bin/test_pull_requests:1090:in `get_comment_with_prefix'
	from /bin/test_pull_requests:779:in `mark_test_success'
	from /bin/test_pull_requests:2361:in `<main>'
+ true
[description-setter] Could not determine description.
[PostBuildScript] - Execution post build scripts.
[workspace@2] $ /bin/sh -xe /tmp/hudson8463658973853361436.sh
+ INSTANCE_NAME=origin_logging-rhel7-1645
+ pushd origin
~/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2
+ rc=0
+ '[' -f .vagrant-openshift.json ']'
++ /usr/bin/vagrant ssh -c 'sudo ausearch -m avc'
+ ausearchresult='<no matches>'
+ rc=1
+ '[' '<no matches>' = '<no matches>' ']'
+ rc=0
+ /usr/bin/vagrant destroy -f
==> openshiftdev: Terminating the instance...
==> openshiftdev: Running cleanup tasks for 'shell' provisioner...
+ popd
~/jobs/test-origin-aggregated-logging/workspace@2
+ exit 0
Finished: SUCCESS