SuccessConsole Output

Skipping 588 KB.. Full Log
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 250 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 1fb0e21e-def6-49b0-b117-50aecfb6b422
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 249 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 1fb0e21e-def6-49b0-b117-50aecfb6b422
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 248 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 1fb0e21e-def6-49b0-b117-50aecfb6b422
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 247 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 1fb0e21e-def6-49b0-b117-50aecfb6b422
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 246 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 1fb0e21e-def6-49b0-b117-50aecfb6b422
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:1fb0e21e-def6-49b0-b117-50aecfb6b422'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 246 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 1fb0e21e-def6-49b0-b117-50aecfb6b422
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 1fb0e21e-def6-49b0-b117-50aecfb6b422
++ espod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ myproject=.operations
++ mymessage=ec094958-2d72-441d-9b80-e140988cc0aa
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER ec094958-2d72-441d-9b80-e140988cc0aa
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER ec094958-2d72-441d-9b80-e140988cc0aa
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER ec094958-2d72-441d-9b80-e140988cc0aa
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER ec094958-2d72-441d-9b80-e140988cc0aa
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ec094958-2d72-441d-9b80-e140988cc0aa'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 297 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for ec094958-2d72-441d-9b80-e140988cc0aa
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for ec094958-2d72-441d-9b80-e140988cc0aa
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496862738
+++ expr 1496862738 - 1496862657
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 81 seconds at 2017-06-07 19:12:18.018900467+00:00
++ echo END wait_for_fluentd_to_catch_up took 81 seconds at 2017-06-07 19:12:18.018900467+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.3.61" labeled
++ wait_for_pod_ACTION stop logging-fluentd-2cql8
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-2cql8
++ '[' -z logging-fluentd-2cql8 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-2cql8
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
daemonset "logging-forward-fluentd" deleted
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '      <match **>' ']'
++ oc get configmap/logging-fluentd -o yaml
++ sed -e '/<match \*\*>/ d' -e '/@include configs\.d\/user\/secure-forward\.conf/ d' -e '/<\/match>/ d'
++ oc replace -f -
configmap "logging-fluentd" replaced
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.3.61" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-qz86p
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-qz86p ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-qz86p
++ oc get events -o yaml
++ write_and_verify_logs 1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up '' ''
+++ date +%s
++ local starttime=1496862753
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 19:12:33.655900216+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 19:12:33.655900216+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ '[' -z logging-es-ops-data-master-9g9jl08p-1-j4b1v ']'
+++ uuidgen
++ local uuid_es=d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ uuidgen
++ local uuid_es_ops=a338a5d9-c306-422e-ac30-9b706433d327
++ local expected=1
++ local timeout=300
++ add_test_message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-bk2hh
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/d8224cd7-8215-49d0-a699-fff8dc4ce5ee
++ echo added es message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
added es message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
++ logger -i -p local6.info -t a338a5d9-c306-422e-ac30-9b706433d327 a338a5d9-c306-422e-ac30-9b706433d327
added es-ops message a338a5d9-c306-422e-ac30-9b706433d327
++ echo added es-ops message a338a5d9-c306-422e-ac30-9b706433d327
++ local rc=0
++ espod=logging-es-data-master-crpvxjde-1-7kj2m
++ myproject=project.logging
++ mymessage=d8224cd7-8215-49d0-a699-fff8dc4ce5ee
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 289 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 288 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 287 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message d8224cd7-8215-49d0-a699-fff8dc4ce5ee
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:d8224cd7-8215-49d0-a699-fff8dc4ce5ee'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 287 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for d8224cd7-8215-49d0-a699-fff8dc4ce5ee
good - wait_for_fluentd_to_catch_up: found 1 record project logging for d8224cd7-8215-49d0-a699-fff8dc4ce5ee
++ espod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ myproject=.operations
++ mymessage=a338a5d9-c306-422e-ac30-9b706433d327
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER a338a5d9-c306-422e-ac30-9b706433d327
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:a338a5d9-c306-422e-ac30-9b706433d327' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:a338a5d9-c306-422e-ac30-9b706433d327'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:a338a5d9-c306-422e-ac30-9b706433d327'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for a338a5d9-c306-422e-ac30-9b706433d327
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for a338a5d9-c306-422e-ac30-9b706433d327
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496862773
+++ expr 1496862773 - 1496862753
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-07 19:12:53.612505515+00:00
++ echo END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-07 19:12:53.612505515+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.3.61" labeled
++ wait_for_pod_ACTION stop logging-fluentd-qz86p
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-qz86p
++ '[' -z logging-fluentd-qz86p -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-qz86p
pod logging-fluentd-qz86p still running
++ '[' -n 1 ']'
++ echo pod logging-fluentd-qz86p still running
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' stop = start ']'
++ '[' 110 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-qz86p
++ '[' -n 1 ']'
pod logging-fluentd-qz86p still running
++ echo pod logging-fluentd-qz86p still running
++ sleep 10
+++ expr 110 - 10
++ ii=100
++ '[' stop = start ']'
++ '[' 100 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-qz86p
++ '[' stop = start ']'
++ break
++ '[' 100 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
Error from server (NotFound): daemonsets.extensions "logging-forward-fluentd" not found
++ :
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '' ']'
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.3.61" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-mm567
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-mm567 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-mm567
++ oc get events -o yaml
running test test-json-parsing.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ es_pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ uuidgen
++ uuid_es=121f00fd-bda6-4420-885b-e1a427800bdc
++ echo Adding test message 121f00fd-bda6-4420-885b-e1a427800bdc to Kibana . . .
Adding test message 121f00fd-bda6-4420-885b-e1a427800bdc to Kibana . . .
++ add_test_message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-bk2hh
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/121f00fd-bda6-4420-885b-e1a427800bdc
++ rc=0
++ timeout=600
Waiting 600 seconds for 121f00fd-bda6-4420-885b-e1a427800bdc to show up in Elasticsearch . . .
++ echo Waiting 600 seconds for 121f00fd-bda6-4420-885b-e1a427800bdc to show up in Elasticsearch . . .
++ espod=logging-es-data-master-crpvxjde-1-7kj2m
++ myproject=project.logging.
++ mymessage=121f00fd-bda6-4420-885b-e1a427800bdc
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 600
++ let ii=600
++ local interval=1
++ '[' 600 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _count message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 599 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _count message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 598 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _count message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 597 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _count message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 596 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _count message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 595 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _count message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 594 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _count message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 593 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _count message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 592 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _count message 121f00fd-bda6-4420-885b-e1a427800bdc
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 592 -le 0 ']'
++ return 0
good - ./logging.sh: found 1 record project logging for 121f00fd-bda6-4420-885b-e1a427800bdc
++ echo good - ./logging.sh: found 1 record project logging for 121f00fd-bda6-4420-885b-e1a427800bdc
Testing if record is in correct format . . .
++ echo Testing if record is in correct format . . .
++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging. _search message 121f00fd-bda6-4420-885b-e1a427800bdc
++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging.*/_search?q=message:121f00fd-bda6-4420-885b-e1a427800bdc' --connect-timeout 1
++ python test-json-parsing.py 121f00fd-bda6-4420-885b-e1a427800bdc
++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
++ local 'endpoint=/project.logging.*/_search?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
++ shift
++ shift
++ args=("${@:-}")
++ local args
++ local secret_dir=/etc/elasticsearch/secret/
++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_search?q=message:121f00fd-bda6-4420-885b-e1a427800bdc'
Success: record contains all of the expected fields/values
++ echo Success: ./logging.sh passed
Success: ./logging.sh passed
++ exit 0
running test test-mux.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ '[' false == false -o false == false ']'
Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.
++ echo 'Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.'
++ exit 0
SKIPPING upgrade test for now
running test test-viaq-data-model.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ [[ 1 -ne 1 ]]
++ [[ true = \f\a\l\s\e ]]
++ CLUSTER=true
++ ops=-ops
++ INDEX_PREFIX=
++ PROJ_PREFIX=project.
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
++ get_test_user_token
++ local current_project
+++ oc project -q
++ current_project=logging
++ oc login --username=admin --password=admin
+++ oc whoami -t
++ test_token=UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0
+++ oc whoami
++ test_name=admin
++ test_ip=127.0.0.1
++ oc login --username=system:admin
++ oc project logging
++ TEST_DIVIDER=------------------------------------------
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-mm567
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
+++ mktemp
++ cfg=/tmp/tmp.rzycFu1E6i
++ cat
++ add_test_volume /tmp/tmp.rzycFu1E6i
++ oc get template logging-fluentd-template -o json
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"].append({"name": "cdmtest", "mountPath": "/etc/fluent/configs.d/openshift/filter-pre-cdm-test.conf", "readOnly": True}); obj["objects"][0]["spec"]["template"]["spec"]["volumes"].append({"name": "cdmtest", "hostPath": {"path": "/tmp/tmp.rzycFu1E6i"}}); print json.dumps(obj, indent=2)'
template "logging-fluentd-template" replaced
++ trap cleanup INT TERM EXIT
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-mm567
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-mm567
++ '[' -z logging-fluentd-mm567 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-mm567
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-kzjsv
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-kzjsv ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-kzjsv
++ keep_fields=method,statusCode,type,@timestamp,req,res
++ write_and_verify_logs test1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496862837
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 19:13:57.649590025+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 19:13:57.649590025+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ '[' -z logging-es-ops-data-master-9g9jl08p-1-j4b1v ']'
+++ uuidgen
++ local uuid_es=21565127-4a37-4d18-bfd2-f0f0b501000f
+++ uuidgen
++ local uuid_es_ops=e68d8e1f-8813-4df6-8834-4436c0cefe22
++ local expected=1
++ local timeout=300
++ add_test_message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-bk2hh
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/21565127-4a37-4d18-bfd2-f0f0b501000f
++ echo added es message 21565127-4a37-4d18-bfd2-f0f0b501000f
added es message 21565127-4a37-4d18-bfd2-f0f0b501000f
++ logger -i -p local6.info -t e68d8e1f-8813-4df6-8834-4436c0cefe22 e68d8e1f-8813-4df6-8834-4436c0cefe22
added es-ops message e68d8e1f-8813-4df6-8834-4436c0cefe22
++ echo added es-ops message e68d8e1f-8813-4df6-8834-4436c0cefe22
++ local rc=0
++ espod=logging-es-data-master-crpvxjde-1-7kj2m
++ myproject=project.logging
++ mymessage=21565127-4a37-4d18-bfd2-f0f0b501000f
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 21565127-4a37-4d18-bfd2-f0f0b501000f
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 290 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 21565127-4a37-4d18-bfd2-f0f0b501000f
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 21565127-4a37-4d18-bfd2-f0f0b501000f
++ espod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ myproject=.operations
++ mymessage=e68d8e1f-8813-4df6-8834-4436c0cefe22
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER e68d8e1f-8813-4df6-8834-4436c0cefe22
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:e68d8e1f-8813-4df6-8834-4436c0cefe22' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:e68d8e1f-8813-4df6-8834-4436c0cefe22'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:e68d8e1f-8813-4df6-8834-4436c0cefe22'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for e68d8e1f-8813-4df6-8834-4436c0cefe22
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for e68d8e1f-8813-4df6-8834-4436c0cefe22
++ '[' -n get_logmessage ']'
++ get_logmessage 21565127-4a37-4d18-bfd2-f0f0b501000f
++ logmessage=21565127-4a37-4d18-bfd2-f0f0b501000f
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 e68d8e1f-8813-4df6-8834-4436c0cefe22
++ logmessage2=e68d8e1f-8813-4df6-8834-4436c0cefe22
+++ date +%s
++ local endtime=1496862853
+++ expr 1496862853 - 1496862837
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-07 19:14:13.424945484+00:00
++ echo END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-07 19:14:13.424945484+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-bk2hh
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es project.logging _search message 21565127-4a37-4d18-bfd2-f0f0b501000f
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:21565127-4a37-4d18-bfd2-f0f0b501000f'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es-ops .operations _search message e68d8e1f-8813-4df6-8834-4436c0cefe22
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:e68d8e1f-8813-4df6-8834-4436c0cefe22'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_USE_UNDEFINED '"true"'
+++ mktemp
++ junk=/tmp/tmp.MyRJdF87mp
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.MyRJdF87mp'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.MyRJdF87mp
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.9wysJ5pxTK
++ cat
++ oc get template logging-fluentd-template -o yaml
++ oc replace -f -
++ sed '/env:/r /tmp/tmp.9wysJ5pxTK'
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.9wysJ5pxTK
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-kzjsv
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-kzjsv
++ '[' -z logging-fluentd-kzjsv -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-kzjsv
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-489sh
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-489sh ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-489sh
++ write_and_verify_logs test2
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496862870
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 19:14:30.722014741+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 19:14:30.722014741+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ '[' -z logging-es-ops-data-master-9g9jl08p-1-j4b1v ']'
+++ uuidgen
++ local uuid_es=9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ uuidgen
++ local uuid_es_ops=686bb930-72b3-4a44-b524-421f7482f6d1
++ local expected=1
++ local timeout=300
++ add_test_message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-bk2hh
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/9f73ac28-0a01-4f84-aa32-de0d1584a6c1
++ echo added es message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
added es message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
++ logger -i -p local6.info -t 686bb930-72b3-4a44-b524-421f7482f6d1 686bb930-72b3-4a44-b524-421f7482f6d1
added es-ops message 686bb930-72b3-4a44-b524-421f7482f6d1
++ echo added es-ops message 686bb930-72b3-4a44-b524-421f7482f6d1
++ local rc=0
++ espod=logging-es-data-master-crpvxjde-1-7kj2m
++ myproject=project.logging
++ mymessage=9f73ac28-0a01-4f84-aa32-de0d1584a6c1
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1' --connect-timeout 1
+++ get_count_from_json
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 292 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
++ espod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ myproject=.operations
++ mymessage=686bb930-72b3-4a44-b524-421f7482f6d1
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER 686bb930-72b3-4a44-b524-421f7482f6d1
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:686bb930-72b3-4a44-b524-421f7482f6d1' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:686bb930-72b3-4a44-b524-421f7482f6d1'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:686bb930-72b3-4a44-b524-421f7482f6d1'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 686bb930-72b3-4a44-b524-421f7482f6d1
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 686bb930-72b3-4a44-b524-421f7482f6d1
++ '[' -n get_logmessage ']'
++ get_logmessage 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
++ logmessage=9f73ac28-0a01-4f84-aa32-de0d1584a6c1
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 686bb930-72b3-4a44-b524-421f7482f6d1
++ logmessage2=686bb930-72b3-4a44-b524-421f7482f6d1
+++ date +%s
++ local endtime=1496862883
+++ expr 1496862883 - 1496862870
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-07 19:14:43.785362989+00:00
++ echo END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-07 19:14:43.785362989+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-bk2hh
++ '[' 0 = 0 ']'
++ python test-viaq-data-model.py test2
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es project.logging _search message 9f73ac28-0a01-4f84-aa32-de0d1584a6c1
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:9f73ac28-0a01-4f84-aa32-de0d1584a6c1'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es-ops .operations _search message 686bb930-72b3-4a44-b524-421f7482f6d1
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:686bb930-72b3-4a44-b524-421f7482f6d1'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.weZM0arui3
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.weZM0arui3'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.weZM0arui3
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-489sh
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-489sh
++ '[' -z logging-fluentd-489sh -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-489sh
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-p7xn7
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-p7xn7 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-p7xn7
++ write_and_verify_logs test3
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496862908
+++ date -u --rfc-3339=ns
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 19:15:08.770727927+00:00
START wait_for_fluentd_to_catch_up at 2017-06-07 19:15:08.770727927+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ '[' -z logging-es-ops-data-master-9g9jl08p-1-j4b1v ']'
+++ uuidgen
++ local uuid_es=468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ uuidgen
++ local uuid_es_ops=f938ae3f-866d-4d16-a16c-0f6dcec09f0c
++ local expected=1
++ local timeout=300
++ add_test_message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-bk2hh
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/468ae57c-0fe2-425c-bbc7-9cb866948e67
added es message 468ae57c-0fe2-425c-bbc7-9cb866948e67
++ echo added es message 468ae57c-0fe2-425c-bbc7-9cb866948e67
++ logger -i -p local6.info -t f938ae3f-866d-4d16-a16c-0f6dcec09f0c f938ae3f-866d-4d16-a16c-0f6dcec09f0c
added es-ops message f938ae3f-866d-4d16-a16c-0f6dcec09f0c
++ echo added es-ops message f938ae3f-866d-4d16-a16c-0f6dcec09f0c
++ local rc=0
++ espod=logging-es-data-master-crpvxjde-1-7kj2m
++ myproject=project.logging
++ mymessage=468ae57c-0fe2-425c-bbc7-9cb866948e67
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 289 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 288 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 287 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 468ae57c-0fe2-425c-bbc7-9cb866948e67
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 468ae57c-0fe2-425c-bbc7-9cb866948e67
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 287 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 468ae57c-0fe2-425c-bbc7-9cb866948e67
++ espod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ myproject=.operations
++ mymessage=f938ae3f-866d-4d16-a16c-0f6dcec09f0c
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER f938ae3f-866d-4d16-a16c-0f6dcec09f0c
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:f938ae3f-866d-4d16-a16c-0f6dcec09f0c' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:f938ae3f-866d-4d16-a16c-0f6dcec09f0c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:f938ae3f-866d-4d16-a16c-0f6dcec09f0c'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for f938ae3f-866d-4d16-a16c-0f6dcec09f0c
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for f938ae3f-866d-4d16-a16c-0f6dcec09f0c
++ '[' -n get_logmessage ']'
++ get_logmessage 468ae57c-0fe2-425c-bbc7-9cb866948e67
++ logmessage=468ae57c-0fe2-425c-bbc7-9cb866948e67
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 f938ae3f-866d-4d16-a16c-0f6dcec09f0c
++ logmessage2=f938ae3f-866d-4d16-a16c-0f6dcec09f0c
+++ date +%s
++ local endtime=1496862928
+++ expr 1496862928 - 1496862908
+++ date -u --rfc-3339=ns
++ echo END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-07 19:15:28.581575450+00:00
END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-07 19:15:28.581575450+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-bk2hh
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es project.logging _search message 468ae57c-0fe2-425c-bbc7-9cb866948e67
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:468ae57c-0fe2-425c-bbc7-9cb866948e67'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es-ops .operations _search message f938ae3f-866d-4d16-a16c-0f6dcec09f0c
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:f938ae3f-866d-4d16-a16c-0f6dcec09f0c'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_UNDEFINED_NAME myname
+++ mktemp
++ junk=/tmp/tmp.t7VR8pVmJX
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.t7VR8pVmJX'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.t7VR8pVmJX
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-p7xn7
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-p7xn7
++ '[' -z logging-fluentd-p7xn7 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-p7xn7
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-4m7zd
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-4m7zd ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-4m7zd
++ write_and_verify_logs test4
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496862948
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 19:15:48.408814211+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 19:15:48.408814211+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ '[' -z logging-es-ops-data-master-9g9jl08p-1-j4b1v ']'
+++ uuidgen
++ local uuid_es=e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ uuidgen
++ local uuid_es_ops=db8e1104-bc59-4982-abff-ef0523ca3ebb
++ local expected=1
++ local timeout=300
++ add_test_message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-bk2hh
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/e8c81c60-957c-4c70-9036-57bb3b28e8f9
++ echo added es message e8c81c60-957c-4c70-9036-57bb3b28e8f9
added es message e8c81c60-957c-4c70-9036-57bb3b28e8f9
++ logger -i -p local6.info -t db8e1104-bc59-4982-abff-ef0523ca3ebb db8e1104-bc59-4982-abff-ef0523ca3ebb
++ echo added es-ops message db8e1104-bc59-4982-abff-ef0523ca3ebb
added es-ops message db8e1104-bc59-4982-abff-ef0523ca3ebb
++ local rc=0
++ espod=logging-es-data-master-crpvxjde-1-7kj2m
++ myproject=project.logging
++ mymessage=e8c81c60-957c-4c70-9036-57bb3b28e8f9
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message e8c81c60-957c-4c70-9036-57bb3b28e8f9
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 290 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for e8c81c60-957c-4c70-9036-57bb3b28e8f9
good - wait_for_fluentd_to_catch_up: found 1 record project logging for e8c81c60-957c-4c70-9036-57bb3b28e8f9
++ espod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ myproject=.operations
++ mymessage=db8e1104-bc59-4982-abff-ef0523ca3ebb
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER db8e1104-bc59-4982-abff-ef0523ca3ebb
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:db8e1104-bc59-4982-abff-ef0523ca3ebb' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:db8e1104-bc59-4982-abff-ef0523ca3ebb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:db8e1104-bc59-4982-abff-ef0523ca3ebb'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for db8e1104-bc59-4982-abff-ef0523ca3ebb
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for db8e1104-bc59-4982-abff-ef0523ca3ebb
++ '[' -n get_logmessage ']'
++ get_logmessage e8c81c60-957c-4c70-9036-57bb3b28e8f9
++ logmessage=e8c81c60-957c-4c70-9036-57bb3b28e8f9
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 db8e1104-bc59-4982-abff-ef0523ca3ebb
++ logmessage2=db8e1104-bc59-4982-abff-ef0523ca3ebb
+++ date +%s
++ local endtime=1496862964
+++ expr 1496862964 - 1496862948
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-07 19:16:04.086250694+00:00
++ echo END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-07 19:16:04.086250694+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-bk2hh
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es project.logging _search message e8c81c60-957c-4c70-9036-57bb3b28e8f9
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:e8c81c60-957c-4c70-9036-57bb3b28e8f9'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es-ops .operations _search message db8e1104-bc59-4982-abff-ef0523ca3ebb
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:db8e1104-bc59-4982-abff-ef0523ca3ebb'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,empty1,undefined3,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.haf4eWS1Qk
++ cat
++ oc get template logging-fluentd-template -o yaml
++ oc replace -f -
++ sed '/env:/r /tmp/tmp.haf4eWS1Qk'
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.haf4eWS1Qk
++ add_cdm_env_var_val CDM_KEEP_EMPTY_FIELDS undefined4,undefined5,empty1,undefined3
+++ mktemp
++ junk=/tmp/tmp.8VUP7Q0D74
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.8VUP7Q0D74'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.8VUP7Q0D74
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-4m7zd
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-4m7zd
++ '[' -z logging-fluentd-4m7zd -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-4m7zd
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-fh2bz
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-fh2bz ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-fh2bz
++ write_and_verify_logs test5 allow_empty
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496862989
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 19:16:29.154534906+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 19:16:29.154534906+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ '[' -z logging-es-ops-data-master-9g9jl08p-1-j4b1v ']'
+++ uuidgen
++ local uuid_es=82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ uuidgen
++ local uuid_es_ops=363ef7b8-6a30-4507-a178-414e3bbc1621
++ local expected=1
++ local timeout=300
++ add_test_message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-bk2hh
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/82a6f07b-80d4-4b24-9c84-31bf6899fedb
added es message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
++ echo added es message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
++ logger -i -p local6.info -t 363ef7b8-6a30-4507-a178-414e3bbc1621 363ef7b8-6a30-4507-a178-414e3bbc1621
added es-ops message 363ef7b8-6a30-4507-a178-414e3bbc1621
++ echo added es-ops message 363ef7b8-6a30-4507-a178-414e3bbc1621
++ local rc=0
++ espod=logging-es-data-master-crpvxjde-1-7kj2m
++ myproject=project.logging
++ mymessage=82a6f07b-80d4-4b24-9c84-31bf6899fedb
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ get_count_from_json
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-crpvxjde-1-7kj2m project.logging _count message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
+++ get_count_from_json
+++ curl_es logging-es-data-master-crpvxjde-1-7kj2m '/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb' --connect-timeout 1
+++ local pod=logging-es-data-master-crpvxjde-1-7kj2m
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-crpvxjde-1-7kj2m -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 291 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 82a6f07b-80d4-4b24-9c84-31bf6899fedb
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 82a6f07b-80d4-4b24-9c84-31bf6899fedb
++ espod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
++ myproject=.operations
++ mymessage=363ef7b8-6a30-4507-a178-414e3bbc1621
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-9g9jl08p-1-j4b1v .operations _count systemd.u.SYSLOG_IDENTIFIER 363ef7b8-6a30-4507-a178-414e3bbc1621
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-9g9jl08p-1-j4b1v '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:363ef7b8-6a30-4507-a178-414e3bbc1621' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-9g9jl08p-1-j4b1v
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:363ef7b8-6a30-4507-a178-414e3bbc1621'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-9g9jl08p-1-j4b1v -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:363ef7b8-6a30-4507-a178-414e3bbc1621'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 363ef7b8-6a30-4507-a178-414e3bbc1621
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 363ef7b8-6a30-4507-a178-414e3bbc1621
++ '[' -n get_logmessage ']'
++ get_logmessage 82a6f07b-80d4-4b24-9c84-31bf6899fedb
++ logmessage=82a6f07b-80d4-4b24-9c84-31bf6899fedb
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 363ef7b8-6a30-4507-a178-414e3bbc1621
++ logmessage2=363ef7b8-6a30-4507-a178-414e3bbc1621
+++ date +%s
++ local endtime=1496863003
+++ expr 1496863003 - 1496862989
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 19:16:43.439751946+00:00
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 19:16:43.439751946+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-bk2hh
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es project.logging _search message 82a6f07b-80d4-4b24-9c84-31bf6899fedb
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:82a6f07b-80d4-4b24-9c84-31bf6899fedb'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-bk2hh logging-es-ops .operations _search message 363ef7b8-6a30-4507-a178-414e3bbc1621
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-bk2hh -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer UUXKWepsK4eWNn0k-6n3VUwlssWTrpoGycLmsqmGOM0' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:363ef7b8-6a30-4507-a178-414e3bbc1621'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ cleanup
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
template "logging-fluentd-template" replaced
++ remove_cdm_env
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.rzycFu1E6i
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-fh2bz
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-fh2bz
++ '[' -z logging-fluentd-fh2bz -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-fh2bz
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-kqsz8
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-kqsz8 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
SKIPPING reinstall test for now
/data/src/github.com/openshift/origin-aggregated-logging/hack/lib/log/system.sh: line 31:  4606 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"  (wd: /data/src/github.com/openshift/origin-aggregated-logging)
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to /tmp/origin-aggregated-logging/artifacts/events.txt
[INFO] [CLEANUP] Dumping etcd contents to /tmp/origin-aggregated-logging/artifacts/etcd
[WARNING] No compiled `etcdhelper` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh tools/etcdhelper
++ Building go targets for linux/amd64: tools/etcdhelper
/data/src/github.com/openshift/origin-aggregated-logging/../origin/hack/build-go.sh took 223 seconds
2017-06-07 15:20:56.998477 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
[INFO] [CLEANUP] Dumping container logs to /tmp/origin-aggregated-logging/logs/containers
[INFO] [CLEANUP] Truncating log files over 200M
[INFO] [CLEANUP] Stopping docker containers
[INFO] [CLEANUP] Removing docker containers
Error response from daemon: You cannot remove a running container be54ab89f33e772d9f1c1b323f65f22118a85d27009bc212d03d1eeabd6e13e0. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 108c742e7e19fb3af64a56b2a328924d6fa49456fdcb89178cdde25453611e05. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 8072b2a04041693aa75aa28f68c3572ac4fd35170433b40b384d4997d9384f21. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 887300d1f2fdec639588c68d80470f3cdfd1c40362b6a405471954e05992593f. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container e0f5225344af6f67edfb29eaa348a24b90910b2b1bc8d15a23c9930915152032. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 43d2c600bf778b25d68361bd7dd6c403b7c646e0cb12f7121583f7603b519b24. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 543882db5d17ae3cb462ff8f0aa9f04a18f7b41befb1817bed47960eab904447. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 992c4c53a2449d0418a2f743c74287749b5a5f7ace571b45aae5803efc03d81f. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 965349cad43ae723a63a29891c2e3452279ed4c6d7cc171a9f8e7b22453fd803. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 9225dca0c90b7663ebd30614d5fb1f3690865a739f45ec7a810bb6ab2adc8971. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 25a548bd7899ee53cae94820bde26060db703257693d74177932cb4ff78e4b71. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 470df71ab48e83334167b89e387d77037042fb640d58aa2ba05f9017397a0527. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container a376ae1200db2a3d6033f9d6f1c9d81c4c1a3e6681d37a51e717541487462701. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 89449582989e926c95d9133aa1f3f20729a9ba5fd9e73770b33d7195e45adc7c. Stop the container before attempting removal or use -f
[INFO] [CLEANUP] Killing child processes
[INFO] [CLEANUP] Pruning etcd data directory
[INFO] /data/src/github.com/openshift/origin-aggregated-logging/logging.sh exited with code 0 after 01h 03m 00s

real	62m59.354s
user	5m43.270s
sys	1m2.949s
Finished GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh
***************************************************
==> openshiftdev: Downloading logs
==> openshiftdev: Downloading artifacts from '/var/log/yum.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/yum.log'
==> openshiftdev: Downloading artifacts from '/var/log/secure' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/secure'
==> openshiftdev: Downloading artifacts from '/var/log/audit/audit.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/audit.log'
==> openshiftdev: Downloading artifacts from '/tmp/origin-aggregated-logging/' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts'
+ false
+ false
+ test_pull_requests --mark_test_success --repo origin-aggregated-logging --config /var/lib/jenkins/.test_pull_requests_logging.json
Rate limit remaining: 2285
  Marking SUCCESS for pull request #--repo in repo ''
/usr/share/ruby/net/http/response.rb:119:in `error!': 404 "Not Found" (Net::HTTPServerException)
	from /bin/test_pull_requests:543:in `block in get_comments'
	from /bin/test_pull_requests:535:in `each'
	from /bin/test_pull_requests:535:in `get_comments'
	from /bin/test_pull_requests:1074:in `get_comment_matching_regex'
	from /bin/test_pull_requests:1090:in `get_comment_with_prefix'
	from /bin/test_pull_requests:779:in `mark_test_success'
	from /bin/test_pull_requests:2361:in `<main>'
+ true
[description-setter] Could not determine description.
[PostBuildScript] - Execution post build scripts.
[workspace@2] $ /bin/sh -xe /tmp/hudson5239609848582340799.sh
+ INSTANCE_NAME=origin_logging-rhel7-1586
+ pushd origin
~/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2
+ rc=0
+ '[' -f .vagrant-openshift.json ']'
++ /usr/bin/vagrant ssh -c 'sudo ausearch -m avc'
+ ausearchresult='<no matches>'
+ rc=1
+ '[' '<no matches>' = '<no matches>' ']'
+ rc=0
+ /usr/bin/vagrant destroy -f
==> openshiftdev: Terminating the instance...
==> openshiftdev: Running cleanup tasks for 'shell' provisioner...
+ popd
~/jobs/test-origin-aggregated-logging/workspace@2
+ exit 0
Finished: SUCCESS