SuccessConsole Output

Skipping 624 KB.. Full Log
101378090'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 289 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 288 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 287 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 286 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 285 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ get_count_from_json
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 284 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6bad3ab3-2cc8-4300-96bc-e41101378090
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6bad3ab3-2cc8-4300-96bc-e41101378090'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 284 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 6bad3ab3-2cc8-4300-96bc-e41101378090
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 6bad3ab3-2cc8-4300-96bc-e41101378090
++ espod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ myproject=.operations
++ mymessage=5ea1ffb4-6463-4adf-a458-500fd38b8ff7
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-xc2h70yx-1-08w7b .operations _count systemd.u.SYSLOG_IDENTIFIER 5ea1ffb4-6463-4adf-a458-500fd38b8ff7
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-xc2h70yx-1-08w7b '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:5ea1ffb4-6463-4adf-a458-500fd38b8ff7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:5ea1ffb4-6463-4adf-a458-500fd38b8ff7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-xc2h70yx-1-08w7b -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:5ea1ffb4-6463-4adf-a458-500fd38b8ff7'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 5ea1ffb4-6463-4adf-a458-500fd38b8ff7
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 5ea1ffb4-6463-4adf-a458-500fd38b8ff7
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496974923
+++ expr 1496974923 - 1496974899
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 24 seconds at 2017-06-09 02:22:03.676196682+00:00
++ echo END wait_for_fluentd_to_catch_up took 24 seconds at 2017-06-09 02:22:03.676196682+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.11.188" labeled
++ wait_for_pod_ACTION stop logging-fluentd-g06k8
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-g06k8
++ '[' -z logging-fluentd-g06k8 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-g06k8
++ '[' -n 1 ']'
pod logging-fluentd-g06k8 still running
++ echo pod logging-fluentd-g06k8 still running
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' stop = start ']'
++ '[' 110 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-g06k8
++ '[' stop = start ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
Error from server (NotFound): daemonsets.extensions "logging-forward-fluentd" not found
++ :
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '' ']'
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" not patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.11.188" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-kgk6c
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-kgk6c ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-kgk6c
++ oc get events -o yaml
running test test-json-parsing.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ es_pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ uuidgen
++ uuid_es=24405998-374a-4a32-8d2c-2d151d0bd85f
Adding test message 24405998-374a-4a32-8d2c-2d151d0bd85f to Kibana . . .
++ echo Adding test message 24405998-374a-4a32-8d2c-2d151d0bd85f to Kibana . . .
++ add_test_message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-fz6pt
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/24405998-374a-4a32-8d2c-2d151d0bd85f
++ rc=0
++ timeout=600
++ echo Waiting 600 seconds for 24405998-374a-4a32-8d2c-2d151d0bd85f to show up in Elasticsearch . . .
Waiting 600 seconds for 24405998-374a-4a32-8d2c-2d151d0bd85f to show up in Elasticsearch . . .
++ espod=logging-es-data-master-8nzz83ik-1-cqnxl
++ myproject=project.logging.
++ mymessage=24405998-374a-4a32-8d2c-2d151d0bd85f
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 600
++ let ii=600
++ local interval=1
++ '[' 600 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 599 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 598 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 597 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 596 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 595 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 594 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 593 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 592 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 591 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 590 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 589 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 588 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 587 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 586 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 585 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _count message 24405998-374a-4a32-8d2c-2d151d0bd85f
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 585 -le 0 ']'
++ return 0
++ echo good - ./logging.sh: found 1 record project logging for 24405998-374a-4a32-8d2c-2d151d0bd85f
good - ./logging.sh: found 1 record project logging for 24405998-374a-4a32-8d2c-2d151d0bd85f
Testing if record is in correct format . . .
++ echo Testing if record is in correct format . . .
++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging. _search message 24405998-374a-4a32-8d2c-2d151d0bd85f
++ python test-json-parsing.py 24405998-374a-4a32-8d2c-2d151d0bd85f
++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging.*/_search?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f' --connect-timeout 1
++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
++ local 'endpoint=/project.logging.*/_search?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
++ shift
++ shift
++ args=("${@:-}")
++ local args
++ local secret_dir=/etc/elasticsearch/secret/
++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_search?q=message:24405998-374a-4a32-8d2c-2d151d0bd85f'
Success: record contains all of the expected fields/values
Success: ./logging.sh passed
++ echo Success: ./logging.sh passed
++ exit 0
running test test-mux.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ '[' false == false -o false == false ']'
Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.
++ echo 'Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.'
++ exit 0
SKIPPING upgrade test for now
running test test-viaq-data-model.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ [[ 1 -ne 1 ]]
++ [[ true = \f\a\l\s\e ]]
++ CLUSTER=true
++ ops=-ops
++ INDEX_PREFIX=
++ PROJ_PREFIX=project.
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
++ get_test_user_token
++ local current_project
+++ oc project -q
++ current_project=logging
++ oc login --username=admin --password=admin
+++ oc whoami -t
++ test_token=b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo
+++ oc whoami
++ test_name=admin
++ test_ip=127.0.0.1
++ oc login --username=system:admin
++ oc project logging
++ TEST_DIVIDER=------------------------------------------
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-kgk6c
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
+++ mktemp
++ cfg=/tmp/tmp.HEU3elyHps
++ cat
++ add_test_volume /tmp/tmp.HEU3elyHps
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"].append({"name": "cdmtest", "mountPath": "/etc/fluent/configs.d/openshift/filter-pre-cdm-test.conf", "readOnly": True}); obj["objects"][0]["spec"]["template"]["spec"]["volumes"].append({"name": "cdmtest", "hostPath": {"path": "/tmp/tmp.HEU3elyHps"}}); print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ trap cleanup INT TERM EXIT
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-kgk6c
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-kgk6c
++ '[' -z logging-fluentd-kgk6c -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-kgk6c
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
pod for component=fluentd not running yet
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-0v2kd
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-0v2kd ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-0v2kd
++ keep_fields=method,statusCode,type,@timestamp,req,res
++ write_and_verify_logs test1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496974986
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-09 02:23:06.029844143+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 02:23:06.029844143+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ '[' -z logging-es-ops-data-master-xc2h70yx-1-08w7b ']'
+++ uuidgen
++ local uuid_es=a7abe340-b73c-4ec2-aad9-49d606659644
+++ uuidgen
++ local uuid_es_ops=613467a4-5c7e-4052-8448-798b82d56d70
++ local expected=1
++ local timeout=300
++ add_test_message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-fz6pt
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/a7abe340-b73c-4ec2-aad9-49d606659644
added es message a7abe340-b73c-4ec2-aad9-49d606659644
++ echo added es message a7abe340-b73c-4ec2-aad9-49d606659644
++ logger -i -p local6.info -t 613467a4-5c7e-4052-8448-798b82d56d70 613467a4-5c7e-4052-8448-798b82d56d70
added es-ops message 613467a4-5c7e-4052-8448-798b82d56d70
++ echo added es-ops message 613467a4-5c7e-4052-8448-798b82d56d70
++ local rc=0
++ espod=logging-es-data-master-8nzz83ik-1-cqnxl
++ myproject=project.logging
++ mymessage=a7abe340-b73c-4ec2-aad9-49d606659644
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ get_count_from_json
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 289 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a7abe340-b73c-4ec2-aad9-49d606659644
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ local nrecs=1
++ test 1 = 1
good - wait_for_fluentd_to_catch_up: found 1 record project logging for a7abe340-b73c-4ec2-aad9-49d606659644
++ break
++ '[' 289 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for a7abe340-b73c-4ec2-aad9-49d606659644
++ espod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ myproject=.operations
++ mymessage=613467a4-5c7e-4052-8448-798b82d56d70
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-xc2h70yx-1-08w7b .operations _count systemd.u.SYSLOG_IDENTIFIER 613467a4-5c7e-4052-8448-798b82d56d70
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-xc2h70yx-1-08w7b '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:613467a4-5c7e-4052-8448-798b82d56d70' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:613467a4-5c7e-4052-8448-798b82d56d70'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-xc2h70yx-1-08w7b -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:613467a4-5c7e-4052-8448-798b82d56d70'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 613467a4-5c7e-4052-8448-798b82d56d70
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 613467a4-5c7e-4052-8448-798b82d56d70
++ '[' -n get_logmessage ']'
++ get_logmessage a7abe340-b73c-4ec2-aad9-49d606659644
++ logmessage=a7abe340-b73c-4ec2-aad9-49d606659644
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 613467a4-5c7e-4052-8448-798b82d56d70
++ logmessage2=613467a4-5c7e-4052-8448-798b82d56d70
+++ date +%s
++ local endtime=1496975003
+++ expr 1496975003 - 1496974986
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 17 seconds at 2017-06-09 02:23:23.366573646+00:00
++ echo END wait_for_fluentd_to_catch_up took 17 seconds at 2017-06-09 02:23:23.366573646+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-fz6pt
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es project.logging _search message a7abe340-b73c-4ec2-aad9-49d606659644
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:a7abe340-b73c-4ec2-aad9-49d606659644'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es-ops .operations _search message 613467a4-5c7e-4052-8448-798b82d56d70
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:613467a4-5c7e-4052-8448-798b82d56d70'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_USE_UNDEFINED '"true"'
+++ mktemp
++ junk=/tmp/tmp.kG2LLs13n7
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.kG2LLs13n7'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.kG2LLs13n7
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.XiVJAnAKTo
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.XiVJAnAKTo'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.XiVJAnAKTo
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-0v2kd
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-0v2kd
++ '[' -z logging-fluentd-0v2kd -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-0v2kd
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-lbfsw
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-lbfsw ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-lbfsw
++ write_and_verify_logs test2
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496975020
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-09 02:23:40.627196783+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 02:23:40.627196783+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ '[' -z logging-es-ops-data-master-xc2h70yx-1-08w7b ']'
+++ uuidgen
++ local uuid_es=a057b0f5-4090-45a0-9e40-18a2056a304a
+++ uuidgen
++ local uuid_es_ops=3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
++ local expected=1
++ local timeout=300
++ add_test_message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-fz6pt
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/a057b0f5-4090-45a0-9e40-18a2056a304a
added es message a057b0f5-4090-45a0-9e40-18a2056a304a
++ echo added es message a057b0f5-4090-45a0-9e40-18a2056a304a
++ logger -i -p local6.info -t 3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09 3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
added es-ops message 3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
++ echo added es-ops message 3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
++ local rc=0
++ espod=logging-es-data-master-8nzz83ik-1-cqnxl
++ myproject=project.logging
++ mymessage=a057b0f5-4090-45a0-9e40-18a2056a304a
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message a057b0f5-4090-45a0-9e40-18a2056a304a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 292 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for a057b0f5-4090-45a0-9e40-18a2056a304a
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for a057b0f5-4090-45a0-9e40-18a2056a304a
++ espod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ myproject=.operations
++ mymessage=3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-xc2h70yx-1-08w7b .operations _count systemd.u.SYSLOG_IDENTIFIER 3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-xc2h70yx-1-08w7b '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-xc2h70yx-1-08w7b -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
++ '[' -n get_logmessage ']'
++ get_logmessage a057b0f5-4090-45a0-9e40-18a2056a304a
++ logmessage=a057b0f5-4090-45a0-9e40-18a2056a304a
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
++ logmessage2=3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
+++ date +%s
++ local endtime=1496975033
+++ expr 1496975033 - 1496975020
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-09 02:23:53.459437173+00:00
++ echo END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-09 02:23:53.459437173+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-fz6pt
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es project.logging _search message a057b0f5-4090-45a0-9e40-18a2056a304a
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:a057b0f5-4090-45a0-9e40-18a2056a304a'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es-ops .operations _search message 3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:3fe1750d-3ed3-4e5b-89a9-32ca7ca29e09'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.rj9kTzb5q9
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.rj9kTzb5q9'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.rj9kTzb5q9
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-lbfsw
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-lbfsw
++ '[' -z logging-fluentd-lbfsw -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-lbfsw
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-glmjj
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-glmjj ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-glmjj
++ write_and_verify_logs test3
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496975050
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-09 02:24:10.414177003+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 02:24:10.414177003+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ '[' -z logging-es-ops-data-master-xc2h70yx-1-08w7b ']'
+++ uuidgen
++ local uuid_es=6e69cb38-df43-4143-a9ba-93a709257e49
+++ uuidgen
++ local uuid_es_ops=bdb35d8f-1ada-4b3b-887b-444d32a06c4b
++ local expected=1
++ local timeout=300
++ add_test_message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-fz6pt
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/6e69cb38-df43-4143-a9ba-93a709257e49
added es message 6e69cb38-df43-4143-a9ba-93a709257e49
++ echo added es message 6e69cb38-df43-4143-a9ba-93a709257e49
++ logger -i -p local6.info -t bdb35d8f-1ada-4b3b-887b-444d32a06c4b bdb35d8f-1ada-4b3b-887b-444d32a06c4b
added es-ops message bdb35d8f-1ada-4b3b-887b-444d32a06c4b
++ echo added es-ops message bdb35d8f-1ada-4b3b-887b-444d32a06c4b
++ local rc=0
++ espod=logging-es-data-master-8nzz83ik-1-cqnxl
++ myproject=project.logging
++ mymessage=6e69cb38-df43-4143-a9ba-93a709257e49
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 289 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 288 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message 6e69cb38-df43-4143-a9ba-93a709257e49
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 288 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 6e69cb38-df43-4143-a9ba-93a709257e49
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 6e69cb38-df43-4143-a9ba-93a709257e49
++ espod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ myproject=.operations
++ mymessage=bdb35d8f-1ada-4b3b-887b-444d32a06c4b
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-xc2h70yx-1-08w7b .operations _count systemd.u.SYSLOG_IDENTIFIER bdb35d8f-1ada-4b3b-887b-444d32a06c4b
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-xc2h70yx-1-08w7b '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bdb35d8f-1ada-4b3b-887b-444d32a06c4b' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bdb35d8f-1ada-4b3b-887b-444d32a06c4b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-xc2h70yx-1-08w7b -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bdb35d8f-1ada-4b3b-887b-444d32a06c4b'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for bdb35d8f-1ada-4b3b-887b-444d32a06c4b
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for bdb35d8f-1ada-4b3b-887b-444d32a06c4b
++ '[' -n get_logmessage ']'
++ get_logmessage 6e69cb38-df43-4143-a9ba-93a709257e49
++ logmessage=6e69cb38-df43-4143-a9ba-93a709257e49
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 bdb35d8f-1ada-4b3b-887b-444d32a06c4b
++ logmessage2=bdb35d8f-1ada-4b3b-887b-444d32a06c4b
+++ date +%s
++ local endtime=1496975068
+++ expr 1496975068 - 1496975050
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 18 seconds at 2017-06-09 02:24:28.849563900+00:00
++ echo END wait_for_fluentd_to_catch_up took 18 seconds at 2017-06-09 02:24:28.849563900+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-fz6pt
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es project.logging _search message 6e69cb38-df43-4143-a9ba-93a709257e49
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:6e69cb38-df43-4143-a9ba-93a709257e49'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es-ops .operations _search message bdb35d8f-1ada-4b3b-887b-444d32a06c4b
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:bdb35d8f-1ada-4b3b-887b-444d32a06c4b'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_UNDEFINED_NAME myname
+++ mktemp
++ junk=/tmp/tmp.aqzHKf7Tel
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.aqzHKf7Tel'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.aqzHKf7Tel
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-glmjj
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-glmjj
++ '[' -z logging-fluentd-glmjj -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-glmjj
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-hlc4r
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-hlc4r ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-hlc4r
++ write_and_verify_logs test4
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496975092
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-09 02:24:52.568953465+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 02:24:52.568953465+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ '[' -z logging-es-ops-data-master-xc2h70yx-1-08w7b ']'
+++ uuidgen
++ local uuid_es=bb55c0ed-170c-4147-9334-5cfe0960003b
+++ uuidgen
++ local uuid_es_ops=5270be67-d874-4aab-b9a9-9bd5d3644d2e
++ local expected=1
++ local timeout=300
++ add_test_message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-fz6pt
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/bb55c0ed-170c-4147-9334-5cfe0960003b
added es message bb55c0ed-170c-4147-9334-5cfe0960003b
++ echo added es message bb55c0ed-170c-4147-9334-5cfe0960003b
++ logger -i -p local6.info -t 5270be67-d874-4aab-b9a9-9bd5d3644d2e 5270be67-d874-4aab-b9a9-9bd5d3644d2e
added es-ops message 5270be67-d874-4aab-b9a9-9bd5d3644d2e
++ echo added es-ops message 5270be67-d874-4aab-b9a9-9bd5d3644d2e
++ local rc=0
++ espod=logging-es-data-master-8nzz83ik-1-cqnxl
++ myproject=project.logging
++ mymessage=bb55c0ed-170c-4147-9334-5cfe0960003b
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 289 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 288 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 287 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 286 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message bb55c0ed-170c-4147-9334-5cfe0960003b
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
good - wait_for_fluentd_to_catch_up: found 1 record project logging for bb55c0ed-170c-4147-9334-5cfe0960003b
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 286 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for bb55c0ed-170c-4147-9334-5cfe0960003b
++ espod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ myproject=.operations
++ mymessage=5270be67-d874-4aab-b9a9-9bd5d3644d2e
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-xc2h70yx-1-08w7b .operations _count systemd.u.SYSLOG_IDENTIFIER 5270be67-d874-4aab-b9a9-9bd5d3644d2e
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-xc2h70yx-1-08w7b '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:5270be67-d874-4aab-b9a9-9bd5d3644d2e' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:5270be67-d874-4aab-b9a9-9bd5d3644d2e'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-xc2h70yx-1-08w7b -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:5270be67-d874-4aab-b9a9-9bd5d3644d2e'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 5270be67-d874-4aab-b9a9-9bd5d3644d2e
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 5270be67-d874-4aab-b9a9-9bd5d3644d2e
++ '[' -n get_logmessage ']'
++ get_logmessage bb55c0ed-170c-4147-9334-5cfe0960003b
++ logmessage=bb55c0ed-170c-4147-9334-5cfe0960003b
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 5270be67-d874-4aab-b9a9-9bd5d3644d2e
++ logmessage2=5270be67-d874-4aab-b9a9-9bd5d3644d2e
+++ date +%s
++ local endtime=1496975113
+++ expr 1496975113 - 1496975092
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 21 seconds at 2017-06-09 02:25:13.915658620+00:00
++ echo END wait_for_fluentd_to_catch_up took 21 seconds at 2017-06-09 02:25:13.915658620+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-fz6pt
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es project.logging _search message bb55c0ed-170c-4147-9334-5cfe0960003b
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:bb55c0ed-170c-4147-9334-5cfe0960003b'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es-ops .operations _search message 5270be67-d874-4aab-b9a9-9bd5d3644d2e
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:5270be67-d874-4aab-b9a9-9bd5d3644d2e'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,empty1,undefined3,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.pRo2brmMc0
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.pRo2brmMc0'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.pRo2brmMc0
++ add_cdm_env_var_val CDM_KEEP_EMPTY_FIELDS undefined4,undefined5,empty1,undefined3
+++ mktemp
++ junk=/tmp/tmp.5Ub9HiDpfi
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.5Ub9HiDpfi'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.5Ub9HiDpfi
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-hlc4r
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-hlc4r
++ '[' -z logging-fluentd-hlc4r -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-hlc4r
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-pv704
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-pv704 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-pv704
++ write_and_verify_logs test5 allow_empty
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496975142
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-09 02:25:42.440149479+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 02:25:42.440149479+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ '[' -z logging-es-ops-data-master-xc2h70yx-1-08w7b ']'
+++ uuidgen
++ local uuid_es=d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ uuidgen
++ local uuid_es_ops=50d50fad-2e5e-465b-ab08-dde7225f9ed6
++ local expected=1
++ local timeout=300
++ add_test_message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-fz6pt
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
added es message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
++ echo added es message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
++ logger -i -p local6.info -t 50d50fad-2e5e-465b-ab08-dde7225f9ed6 50d50fad-2e5e-465b-ab08-dde7225f9ed6
added es-ops message 50d50fad-2e5e-465b-ab08-dde7225f9ed6
++ echo added es-ops message 50d50fad-2e5e-465b-ab08-dde7225f9ed6
++ local rc=0
++ espod=logging-es-data-master-8nzz83ik-1-cqnxl
++ myproject=project.logging
++ mymessage=d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 289 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 288 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 287 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 286 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-8nzz83ik-1-cqnxl project.logging _count message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
+++ get_count_from_json
+++ curl_es logging-es-data-master-8nzz83ik-1-cqnxl '/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-8nzz83ik-1-cqnxl
+++ local 'endpoint=/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-8nzz83ik-1-cqnxl -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
good - wait_for_fluentd_to_catch_up: found 1 record project logging for d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 286 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
++ espod=logging-es-ops-data-master-xc2h70yx-1-08w7b
++ myproject=.operations
++ mymessage=50d50fad-2e5e-465b-ab08-dde7225f9ed6
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-xc2h70yx-1-08w7b .operations _count systemd.u.SYSLOG_IDENTIFIER 50d50fad-2e5e-465b-ab08-dde7225f9ed6
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-xc2h70yx-1-08w7b '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50d50fad-2e5e-465b-ab08-dde7225f9ed6' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-xc2h70yx-1-08w7b
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50d50fad-2e5e-465b-ab08-dde7225f9ed6'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-xc2h70yx-1-08w7b -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50d50fad-2e5e-465b-ab08-dde7225f9ed6'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 50d50fad-2e5e-465b-ab08-dde7225f9ed6
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 50d50fad-2e5e-465b-ab08-dde7225f9ed6
++ '[' -n get_logmessage ']'
++ get_logmessage d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
++ logmessage=d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 50d50fad-2e5e-465b-ab08-dde7225f9ed6
++ logmessage2=50d50fad-2e5e-465b-ab08-dde7225f9ed6
+++ date +%s
++ local endtime=1496975163
+++ expr 1496975163 - 1496975142
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 21 seconds at 2017-06-09 02:26:03.934954787+00:00
++ echo END wait_for_fluentd_to_catch_up took 21 seconds at 2017-06-09 02:26:03.934954787+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-fz6pt
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es project.logging _search message d37c6bc9-d55c-438a-ac57-52a2b4c40f3a
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:d37c6bc9-d55c-438a-ac57-52a2b4c40f3a'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-fz6pt logging-es-ops .operations _search message 50d50fad-2e5e-465b-ab08-dde7225f9ed6
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-fz6pt -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer b9YXT3gcexKSesTOvxPU0vSYJF_8FU6TsX8JT5BEmyo' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:50d50fad-2e5e-465b-ab08-dde7225f9ed6'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ cleanup
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
template "logging-fluentd-template" replaced
++ remove_cdm_env
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.HEU3elyHps
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-pv704
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-pv704
++ '[' -z logging-fluentd-pv704 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-pv704
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-pwxwh
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-pwxwh ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
SKIPPING reinstall test for now
/data/src/github.com/openshift/origin-aggregated-logging/hack/lib/log/system.sh: line 31:  4612 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"  (wd: /data/src/github.com/openshift/origin-aggregated-logging)
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to /tmp/origin-aggregated-logging/artifacts/events.txt
[INFO] [CLEANUP] Dumping etcd contents to /tmp/origin-aggregated-logging/artifacts/etcd
[WARNING] No compiled `etcdhelper` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh tools/etcdhelper
++ Building go targets for linux/amd64: tools/etcdhelper
/data/src/github.com/openshift/origin-aggregated-logging/../origin/hack/build-go.sh took 129 seconds
2017-06-08 22:28:45.031539 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
[INFO] [CLEANUP] Dumping container logs to /tmp/origin-aggregated-logging/logs/containers
[INFO] [CLEANUP] Truncating log files over 200M
[INFO] [CLEANUP] Stopping docker containers
[INFO] [CLEANUP] Removing docker containers
Error response from daemon: You cannot remove a running container f0631b367c5332163cdf3e73daa1cf76eeffa31d95859c57ed4062bb60b6e40e. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container b2aa234ddccf8547a90479ed6e520c746c0c96386925d67f375da6f3fff6ee3e. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container edd62c4bf2a472750ec4e209bec8ff2b7ce4b65d3307f63be0d76b5d8d163c41. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 1ee3978fdbda0756c77708273ea59f732da42e46d48413581f5a9e347eb96977. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container b8224c8a6c08dc0a7a5cf8bbaadd117af5090cc0ba2c0e3cbb377519b4b1741c. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 2cf1a2c9eeda28221bd2174b8297fb26fb599fd380d6d37f6dda16a31d6db729. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container b5dfd700823a300d1bac1c6446990661c1df1f981b417154f678d6b39b931847. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 06a30ac01cb8fc54bd4a6b15a9908e5ae30ccfefa266a97c7ca867ade1256417. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container cdfd396b9afb270155a20f29f6e27b4015ec2c7242aa713da768df235473d906. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 17c584219c01665563559ddba2a4e3013079a7427961544850e9c01fb69fa335. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 47f29e470a80373ce04cfb55f41073033dafa2b002d4cc257a3697a532d1b971. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container b0bb5976debdb2790a39fb70a1b4957488776c7d5bcbf3721aa7d258690e9c03. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 9f6154d1ebac4e13f9ebd2d9e135fd6669a2393f40203e61903a79e6750f2648. Stop the container before attempting removal or use -f
[INFO] [CLEANUP] Killing child processes
[INFO] [CLEANUP] Pruning etcd data directory
[INFO] /data/src/github.com/openshift/origin-aggregated-logging/logging.sh exited with code 0 after 00h 58m 59s

real	58m59.075s
user	5m39.639s
sys	1m5.121s
Finished GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh
***************************************************
==> openshiftdev: Downloading logs
==> openshiftdev: Downloading artifacts from '/var/log/yum.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/yum.log'
==> openshiftdev: Downloading artifacts from '/var/log/secure' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/secure'
==> openshiftdev: Downloading artifacts from '/var/log/audit/audit.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/audit.log'
==> openshiftdev: Downloading artifacts from '/tmp/origin-aggregated-logging/' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts'
+ false
+ false
+ test_pull_requests --mark_test_success --repo origin-aggregated-logging --config /var/lib/jenkins/.test_pull_requests_logging.json
Rate limit remaining: 2349
  Marking SUCCESS for pull request #--repo in repo ''
/usr/share/ruby/net/http/response.rb:119:in `error!': 404 "Not Found" (Net::HTTPServerException)
	from /bin/test_pull_requests:543:in `block in get_comments'
	from /bin/test_pull_requests:535:in `each'
	from /bin/test_pull_requests:535:in `get_comments'
	from /bin/test_pull_requests:1074:in `get_comment_matching_regex'
	from /bin/test_pull_requests:1090:in `get_comment_with_prefix'
	from /bin/test_pull_requests:779:in `mark_test_success'
	from /bin/test_pull_requests:2361:in `<main>'
+ true
[description-setter] Could not determine description.
[PostBuildScript] - Execution post build scripts.
[workspace] $ /bin/sh -xe /tmp/hudson1064897459589279789.sh
+ INSTANCE_NAME=origin_logging-rhel7-1648
+ pushd origin
~/jobs/test-origin-aggregated-logging/workspace/origin ~/jobs/test-origin-aggregated-logging/workspace
+ rc=0
+ '[' -f .vagrant-openshift.json ']'
++ /usr/bin/vagrant ssh -c 'sudo ausearch -m avc'
+ ausearchresult='<no matches>'
+ rc=1
+ '[' '<no matches>' = '<no matches>' ']'
+ rc=0
+ /usr/bin/vagrant destroy -f
==> openshiftdev: Terminating the instance...
==> openshiftdev: Running cleanup tasks for 'shell' provisioner...
+ popd
~/jobs/test-origin-aggregated-logging/workspace
+ exit 0
Finished: SUCCESS