Console Output

Skipping 578 KB.. Full Log
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 262 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 261 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 260 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 259 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 258 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 257 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 256 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 255 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 254 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 253 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 252 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 251 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 250 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 249 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:9c17c198-0b7e-4c4a-b3a8-9f864aa58708'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 249 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 9c17c198-0b7e-4c4a-b3a8-9f864aa58708
++ espod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ myproject=.operations
++ mymessage=50a2e765-a42f-41bf-894a-ab3010d8be79
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 50a2e765-a42f-41bf-894a-ab3010d8be79
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50a2e765-a42f-41bf-894a-ab3010d8be79' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50a2e765-a42f-41bf-894a-ab3010d8be79'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50a2e765-a42f-41bf-894a-ab3010d8be79'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 50a2e765-a42f-41bf-894a-ab3010d8be79
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50a2e765-a42f-41bf-894a-ab3010d8be79' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50a2e765-a42f-41bf-894a-ab3010d8be79'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50a2e765-a42f-41bf-894a-ab3010d8be79'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 50a2e765-a42f-41bf-894a-ab3010d8be79
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50a2e765-a42f-41bf-894a-ab3010d8be79' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50a2e765-a42f-41bf-894a-ab3010d8be79'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:50a2e765-a42f-41bf-894a-ab3010d8be79'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 298 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 50a2e765-a42f-41bf-894a-ab3010d8be79
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 50a2e765-a42f-41bf-894a-ab3010d8be79
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496809204
+++ expr 1496809204 - 1496809129
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 75 seconds at 2017-06-07 04:20:04.169109212+00:00
++ echo END wait_for_fluentd_to_catch_up took 75 seconds at 2017-06-07 04:20:04.169109212+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.3.241" labeled
++ wait_for_pod_ACTION stop logging-fluentd-qs50t
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-qs50t
++ '[' -z logging-fluentd-qs50t -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-qs50t
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
daemonset "logging-forward-fluentd" deleted
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '      <match **>' ']'
++ oc get configmap/logging-fluentd -o yaml
++ oc replace -f -
++ sed -e '/<match \*\*>/ d' -e '/@include configs\.d\/user\/secure-forward\.conf/ d' -e '/<\/match>/ d'
configmap "logging-fluentd" replaced
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.3.241" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
+++ oc get pods -l component=fluentd
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 110 - 10
++ ii=100
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-c9jkp
++ '[' 100 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-c9jkp ']'
++ break
++ '[' 100 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-c9jkp
++ oc get events -o yaml
++ write_and_verify_logs 1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up '' ''
+++ date +%s
++ local starttime=1496809230
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 04:20:30.003216138+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 04:20:30.003216138+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ '[' -z logging-es-ops-data-master-it0wbr7x-1-71pzg ']'
+++ uuidgen
++ local uuid_es=e016242a-84fc-4c80-8b84-4334415dcd3f
+++ uuidgen
++ local uuid_es_ops=12cec94f-9c12-4577-8e0d-2b8621154a77
++ local expected=1
++ local timeout=300
++ add_test_message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-n0b8v
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/e016242a-84fc-4c80-8b84-4334415dcd3f
added es message e016242a-84fc-4c80-8b84-4334415dcd3f
++ echo added es message e016242a-84fc-4c80-8b84-4334415dcd3f
++ logger -i -p local6.info -t 12cec94f-9c12-4577-8e0d-2b8621154a77 12cec94f-9c12-4577-8e0d-2b8621154a77
added es-ops message 12cec94f-9c12-4577-8e0d-2b8621154a77
++ echo added es-ops message 12cec94f-9c12-4577-8e0d-2b8621154a77
++ local rc=0
++ espod=logging-es-data-master-jpwqs1vo-1-6r16x
++ myproject=project.logging
++ mymessage=e016242a-84fc-4c80-8b84-4334415dcd3f
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e016242a-84fc-4c80-8b84-4334415dcd3f
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e016242a-84fc-4c80-8b84-4334415dcd3f'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 290 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for e016242a-84fc-4c80-8b84-4334415dcd3f
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for e016242a-84fc-4c80-8b84-4334415dcd3f
++ espod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ myproject=.operations
++ mymessage=12cec94f-9c12-4577-8e0d-2b8621154a77
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 12cec94f-9c12-4577-8e0d-2b8621154a77
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:12cec94f-9c12-4577-8e0d-2b8621154a77' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:12cec94f-9c12-4577-8e0d-2b8621154a77'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:12cec94f-9c12-4577-8e0d-2b8621154a77'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 12cec94f-9c12-4577-8e0d-2b8621154a77
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 12cec94f-9c12-4577-8e0d-2b8621154a77
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496809245
+++ expr 1496809245 - 1496809230
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 04:20:45.956647462+00:00
++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 04:20:45.956647462+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.3.241" labeled
++ wait_for_pod_ACTION stop logging-fluentd-c9jkp
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-c9jkp
++ '[' -z logging-fluentd-c9jkp -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-c9jkp
++ '[' -n 1 ']'
pod logging-fluentd-c9jkp still running
++ echo pod logging-fluentd-c9jkp still running
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' stop = start ']'
++ '[' 110 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-c9jkp
++ '[' stop = start ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
Error from server (NotFound): daemonsets.extensions "logging-forward-fluentd" not found
++ :
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '' ']'
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.3.241" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-vdj2f
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-vdj2f ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-vdj2f
++ oc get events -o yaml
running test test-json-parsing.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ es_pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ uuidgen
Adding test message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9 to Kibana . . .
++ uuid_es=32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
++ echo Adding test message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9 to Kibana . . .
++ add_test_message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-n0b8v
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
++ rc=0
++ timeout=600
Waiting 600 seconds for 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9 to show up in Elasticsearch . . .
++ echo Waiting 600 seconds for 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9 to show up in Elasticsearch . . .
++ espod=logging-es-data-master-jpwqs1vo-1-6r16x
++ myproject=project.logging.
++ mymessage=32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 600
++ let ii=600
++ local interval=1
++ '[' 600 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _count message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 599 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _count message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 598 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _count message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
+++ get_count_from_json
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 597 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _count message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 596 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _count message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 595 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _count message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 594 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _count message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 593 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _count message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 592 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _count message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 592 -le 0 ']'
++ return 0
good - ./logging.sh: found 1 record project logging for 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
++ echo good - ./logging.sh: found 1 record project logging for 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
Testing if record is in correct format . . .
++ echo Testing if record is in correct format . . .
++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging. _search message 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
++ python test-json-parsing.py 32ef6027-8eaa-443d-bd27-b8c56d3bc0b9
++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging.*/_search?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9' --connect-timeout 1
++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
++ local 'endpoint=/project.logging.*/_search?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
++ shift
++ shift
++ args=("${@:-}")
++ local args
++ local secret_dir=/etc/elasticsearch/secret/
++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_search?q=message:32ef6027-8eaa-443d-bd27-b8c56d3bc0b9'
Success: record contains all of the expected fields/values
Success: ./logging.sh passed
++ echo Success: ./logging.sh passed
++ exit 0
running test test-mux.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ '[' false == false -o false == false ']'
Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.
++ echo 'Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.'
++ exit 0
SKIPPING upgrade test for now
running test test-viaq-data-model.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ [[ 1 -ne 1 ]]
++ [[ true = \f\a\l\s\e ]]
++ CLUSTER=true
++ ops=-ops
++ INDEX_PREFIX=
++ PROJ_PREFIX=project.
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
++ get_test_user_token
++ local current_project
+++ oc project -q
++ current_project=logging
++ oc login --username=admin --password=admin
+++ oc whoami -t
++ test_token=eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E
+++ oc whoami
++ test_name=admin
++ test_ip=127.0.0.1
++ oc login --username=system:admin
++ oc project logging
++ TEST_DIVIDER=------------------------------------------
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-vdj2f
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
+++ mktemp
++ cfg=/tmp/tmp.tUXJwDsGVa
++ cat
++ add_test_volume /tmp/tmp.tUXJwDsGVa
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"].append({"name": "cdmtest", "mountPath": "/etc/fluent/configs.d/openshift/filter-pre-cdm-test.conf", "readOnly": True}); obj["objects"][0]["spec"]["template"]["spec"]["volumes"].append({"name": "cdmtest", "hostPath": {"path": "/tmp/tmp.tUXJwDsGVa"}}); print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ trap cleanup INT TERM EXIT
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-vdj2f
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-vdj2f
++ '[' -z logging-fluentd-vdj2f -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-vdj2f
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-73ss5
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-73ss5 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-73ss5
++ keep_fields=method,statusCode,type,@timestamp,req,res
++ write_and_verify_logs test1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496809299
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 04:21:39.182275876+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 04:21:39.182275876+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ '[' -z logging-es-ops-data-master-it0wbr7x-1-71pzg ']'
+++ uuidgen
++ local uuid_es=e34004d4-90ce-468b-a197-5df24b9292ed
+++ uuidgen
++ local uuid_es_ops=3dedbb89-66ae-4740-97c8-706d3fb2de06
++ local expected=1
++ local timeout=300
++ add_test_message e34004d4-90ce-468b-a197-5df24b9292ed
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-n0b8v
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/e34004d4-90ce-468b-a197-5df24b9292ed
added es message e34004d4-90ce-468b-a197-5df24b9292ed
++ echo added es message e34004d4-90ce-468b-a197-5df24b9292ed
++ logger -i -p local6.info -t 3dedbb89-66ae-4740-97c8-706d3fb2de06 3dedbb89-66ae-4740-97c8-706d3fb2de06
added es-ops message 3dedbb89-66ae-4740-97c8-706d3fb2de06
++ echo added es-ops message 3dedbb89-66ae-4740-97c8-706d3fb2de06
++ local rc=0
++ espod=logging-es-data-master-jpwqs1vo-1-6r16x
++ myproject=project.logging
++ mymessage=e34004d4-90ce-468b-a197-5df24b9292ed
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e34004d4-90ce-468b-a197-5df24b9292ed
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e34004d4-90ce-468b-a197-5df24b9292ed
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e34004d4-90ce-468b-a197-5df24b9292ed
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e34004d4-90ce-468b-a197-5df24b9292ed
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e34004d4-90ce-468b-a197-5df24b9292ed
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e34004d4-90ce-468b-a197-5df24b9292ed
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e34004d4-90ce-468b-a197-5df24b9292ed
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message e34004d4-90ce-468b-a197-5df24b9292ed
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 293 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for e34004d4-90ce-468b-a197-5df24b9292ed
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for e34004d4-90ce-468b-a197-5df24b9292ed
++ espod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ myproject=.operations
++ mymessage=3dedbb89-66ae-4740-97c8-706d3fb2de06
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 3dedbb89-66ae-4740-97c8-706d3fb2de06
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3dedbb89-66ae-4740-97c8-706d3fb2de06' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3dedbb89-66ae-4740-97c8-706d3fb2de06'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3dedbb89-66ae-4740-97c8-706d3fb2de06'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 3dedbb89-66ae-4740-97c8-706d3fb2de06
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 3dedbb89-66ae-4740-97c8-706d3fb2de06
++ '[' -n get_logmessage ']'
++ get_logmessage e34004d4-90ce-468b-a197-5df24b9292ed
++ logmessage=e34004d4-90ce-468b-a197-5df24b9292ed
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 3dedbb89-66ae-4740-97c8-706d3fb2de06
++ logmessage2=3dedbb89-66ae-4740-97c8-706d3fb2de06
+++ date +%s
++ local endtime=1496809310
+++ expr 1496809310 - 1496809299
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 11 seconds at 2017-06-07 04:21:50.607823527+00:00
++ echo END wait_for_fluentd_to_catch_up took 11 seconds at 2017-06-07 04:21:50.607823527+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-n0b8v
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es project.logging _search message e34004d4-90ce-468b-a197-5df24b9292ed
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:e34004d4-90ce-468b-a197-5df24b9292ed'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es-ops .operations _search message 3dedbb89-66ae-4740-97c8-706d3fb2de06
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:3dedbb89-66ae-4740-97c8-706d3fb2de06'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_USE_UNDEFINED '"true"'
+++ mktemp
++ junk=/tmp/tmp.XFrnggbS1b
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.XFrnggbS1b'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.XFrnggbS1b
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.EJL7fTyPvQ
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.EJL7fTyPvQ'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.EJL7fTyPvQ
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-73ss5
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-73ss5
++ '[' -z logging-fluentd-73ss5 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-73ss5
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
pod for component=fluentd not running yet
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-gq587
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-gq587 ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-gq587
++ write_and_verify_logs test2
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496809330
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 04:22:10.875542566+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 04:22:10.875542566+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ '[' -z logging-es-ops-data-master-it0wbr7x-1-71pzg ']'
+++ uuidgen
++ local uuid_es=0745c532-5ba7-4b4d-aeb7-355576654e20
+++ uuidgen
++ local uuid_es_ops=3c20cd12-0300-4f03-8ae7-585d1996948c
++ local expected=1
++ local timeout=300
++ add_test_message 0745c532-5ba7-4b4d-aeb7-355576654e20
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-n0b8v
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/0745c532-5ba7-4b4d-aeb7-355576654e20
++ echo added es message 0745c532-5ba7-4b4d-aeb7-355576654e20
added es message 0745c532-5ba7-4b4d-aeb7-355576654e20
++ logger -i -p local6.info -t 3c20cd12-0300-4f03-8ae7-585d1996948c 3c20cd12-0300-4f03-8ae7-585d1996948c
added es-ops message 3c20cd12-0300-4f03-8ae7-585d1996948c
++ echo added es-ops message 3c20cd12-0300-4f03-8ae7-585d1996948c
++ local rc=0
++ espod=logging-es-data-master-jpwqs1vo-1-6r16x
++ myproject=project.logging
++ mymessage=0745c532-5ba7-4b4d-aeb7-355576654e20
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 0745c532-5ba7-4b4d-aeb7-355576654e20
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 0745c532-5ba7-4b4d-aeb7-355576654e20
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 0745c532-5ba7-4b4d-aeb7-355576654e20
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 0745c532-5ba7-4b4d-aeb7-355576654e20
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 0745c532-5ba7-4b4d-aeb7-355576654e20
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 0745c532-5ba7-4b4d-aeb7-355576654e20
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 0745c532-5ba7-4b4d-aeb7-355576654e20
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 294 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 0745c532-5ba7-4b4d-aeb7-355576654e20
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 0745c532-5ba7-4b4d-aeb7-355576654e20
++ espod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ myproject=.operations
++ mymessage=3c20cd12-0300-4f03-8ae7-585d1996948c
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 3c20cd12-0300-4f03-8ae7-585d1996948c
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 3c20cd12-0300-4f03-8ae7-585d1996948c
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 3c20cd12-0300-4f03-8ae7-585d1996948c
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c'
+++ shift
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 3c20cd12-0300-4f03-8ae7-585d1996948c
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:3c20cd12-0300-4f03-8ae7-585d1996948c'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 297 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 3c20cd12-0300-4f03-8ae7-585d1996948c
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 3c20cd12-0300-4f03-8ae7-585d1996948c
++ '[' -n get_logmessage ']'
++ get_logmessage 0745c532-5ba7-4b4d-aeb7-355576654e20
++ logmessage=0745c532-5ba7-4b4d-aeb7-355576654e20
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 3c20cd12-0300-4f03-8ae7-585d1996948c
++ logmessage2=3c20cd12-0300-4f03-8ae7-585d1996948c
+++ date +%s
++ local endtime=1496809345
+++ expr 1496809345 - 1496809330
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 04:22:25.172849822+00:00
++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 04:22:25.172849822+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-n0b8v
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es project.logging _search message 0745c532-5ba7-4b4d-aeb7-355576654e20
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:0745c532-5ba7-4b4d-aeb7-355576654e20'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es-ops .operations _search message 3c20cd12-0300-4f03-8ae7-585d1996948c
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:3c20cd12-0300-4f03-8ae7-585d1996948c'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.rm0a3OMucC
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.rm0a3OMucC'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.rm0a3OMucC
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-gq587
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-gq587
++ '[' -z logging-fluentd-gq587 -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-gq587
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-4rk7g
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-4rk7g ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-4rk7g
++ write_and_verify_logs test3
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496809370
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 04:22:50.205699617+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 04:22:50.205699617+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ '[' -z logging-es-ops-data-master-it0wbr7x-1-71pzg ']'
+++ uuidgen
++ local uuid_es=2706073a-a609-41da-ab04-ca4285cd6a43
+++ uuidgen
++ local uuid_es_ops=1d689a1b-3df6-4849-8338-ca46a49f6ff9
++ local expected=1
++ local timeout=300
++ add_test_message 2706073a-a609-41da-ab04-ca4285cd6a43
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-n0b8v
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/2706073a-a609-41da-ab04-ca4285cd6a43
added es message 2706073a-a609-41da-ab04-ca4285cd6a43
++ echo added es message 2706073a-a609-41da-ab04-ca4285cd6a43
++ logger -i -p local6.info -t 1d689a1b-3df6-4849-8338-ca46a49f6ff9 1d689a1b-3df6-4849-8338-ca46a49f6ff9
added es-ops message 1d689a1b-3df6-4849-8338-ca46a49f6ff9
++ echo added es-ops message 1d689a1b-3df6-4849-8338-ca46a49f6ff9
++ local rc=0
++ espod=logging-es-data-master-jpwqs1vo-1-6r16x
++ myproject=project.logging
++ mymessage=2706073a-a609-41da-ab04-ca4285cd6a43
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 2706073a-a609-41da-ab04-ca4285cd6a43
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 2706073a-a609-41da-ab04-ca4285cd6a43
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 2706073a-a609-41da-ab04-ca4285cd6a43
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 2706073a-a609-41da-ab04-ca4285cd6a43
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 2706073a-a609-41da-ab04-ca4285cd6a43
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 2706073a-a609-41da-ab04-ca4285cd6a43
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 2706073a-a609-41da-ab04-ca4285cd6a43
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 2706073a-a609-41da-ab04-ca4285cd6a43
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 293 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 2706073a-a609-41da-ab04-ca4285cd6a43
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 2706073a-a609-41da-ab04-ca4285cd6a43
++ espod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ myproject=.operations
++ mymessage=1d689a1b-3df6-4849-8338-ca46a49f6ff9
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 1d689a1b-3df6-4849-8338-ca46a49f6ff9
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1d689a1b-3df6-4849-8338-ca46a49f6ff9' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1d689a1b-3df6-4849-8338-ca46a49f6ff9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1d689a1b-3df6-4849-8338-ca46a49f6ff9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 1d689a1b-3df6-4849-8338-ca46a49f6ff9
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1d689a1b-3df6-4849-8338-ca46a49f6ff9' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1d689a1b-3df6-4849-8338-ca46a49f6ff9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1d689a1b-3df6-4849-8338-ca46a49f6ff9'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 1d689a1b-3df6-4849-8338-ca46a49f6ff9
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1d689a1b-3df6-4849-8338-ca46a49f6ff9' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1d689a1b-3df6-4849-8338-ca46a49f6ff9'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:1d689a1b-3df6-4849-8338-ca46a49f6ff9'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 298 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 1d689a1b-3df6-4849-8338-ca46a49f6ff9
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 1d689a1b-3df6-4849-8338-ca46a49f6ff9
++ '[' -n get_logmessage ']'
++ get_logmessage 2706073a-a609-41da-ab04-ca4285cd6a43
++ logmessage=2706073a-a609-41da-ab04-ca4285cd6a43
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 1d689a1b-3df6-4849-8338-ca46a49f6ff9
++ logmessage2=1d689a1b-3df6-4849-8338-ca46a49f6ff9
+++ date +%s
++ local endtime=1496809384
+++ expr 1496809384 - 1496809370
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 04:23:04.337941342+00:00
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 04:23:04.337941342+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-n0b8v
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es project.logging _search message 2706073a-a609-41da-ab04-ca4285cd6a43
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:2706073a-a609-41da-ab04-ca4285cd6a43'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es-ops .operations _search message 1d689a1b-3df6-4849-8338-ca46a49f6ff9
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:1d689a1b-3df6-4849-8338-ca46a49f6ff9'
++ python test-viaq-data-model.py test3
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_UNDEFINED_NAME myname
+++ mktemp
++ junk=/tmp/tmp.o4EmYpSBC0
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.o4EmYpSBC0'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.o4EmYpSBC0
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-4rk7g
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-4rk7g
++ '[' -z logging-fluentd-4rk7g -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-4rk7g
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-vfr0z
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-vfr0z ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-vfr0z
++ write_and_verify_logs test4
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496809401
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 04:23:21.106316224+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 04:23:21.106316224+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ '[' -z logging-es-ops-data-master-it0wbr7x-1-71pzg ']'
+++ uuidgen
++ local uuid_es=3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ uuidgen
++ local uuid_es_ops=e1ad81bb-06be-40a5-9f0a-248d07223b59
++ local expected=1
++ local timeout=300
++ add_test_message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-n0b8v
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/3a4de0b0-1bf5-4fe9-b8cf-266492bce825
++ echo added es message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
added es message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
++ logger -i -p local6.info -t e1ad81bb-06be-40a5-9f0a-248d07223b59 e1ad81bb-06be-40a5-9f0a-248d07223b59
added es-ops message e1ad81bb-06be-40a5-9f0a-248d07223b59
++ echo added es-ops message e1ad81bb-06be-40a5-9f0a-248d07223b59
++ local rc=0
++ espod=logging-es-data-master-jpwqs1vo-1-6r16x
++ myproject=project.logging
++ mymessage=3a4de0b0-1bf5-4fe9-b8cf-266492bce825
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 290 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
++ espod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ myproject=.operations
++ mymessage=e1ad81bb-06be-40a5-9f0a-248d07223b59
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER e1ad81bb-06be-40a5-9f0a-248d07223b59
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:e1ad81bb-06be-40a5-9f0a-248d07223b59' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:e1ad81bb-06be-40a5-9f0a-248d07223b59'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:e1ad81bb-06be-40a5-9f0a-248d07223b59'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for e1ad81bb-06be-40a5-9f0a-248d07223b59
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for e1ad81bb-06be-40a5-9f0a-248d07223b59
++ '[' -n get_logmessage ']'
++ get_logmessage 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
++ logmessage=3a4de0b0-1bf5-4fe9-b8cf-266492bce825
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 e1ad81bb-06be-40a5-9f0a-248d07223b59
++ logmessage2=e1ad81bb-06be-40a5-9f0a-248d07223b59
+++ date +%s
++ local endtime=1496809416
+++ expr 1496809416 - 1496809401
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 04:23:36.888281491+00:00
++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 04:23:36.888281491+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-n0b8v
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es project.logging _search message 3a4de0b0-1bf5-4fe9-b8cf-266492bce825
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:3a4de0b0-1bf5-4fe9-b8cf-266492bce825'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es-ops .operations _search message e1ad81bb-06be-40a5-9f0a-248d07223b59
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:e1ad81bb-06be-40a5-9f0a-248d07223b59'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,empty1,undefined3,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.9y8UEBv8HF
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.9y8UEBv8HF'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.9y8UEBv8HF
++ add_cdm_env_var_val CDM_KEEP_EMPTY_FIELDS undefined4,undefined5,empty1,undefined3
+++ mktemp
++ junk=/tmp/tmp.n3aYUlSS5X
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.n3aYUlSS5X'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.n3aYUlSS5X
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-vfr0z
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-vfr0z
++ '[' -z logging-fluentd-vfr0z -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-vfr0z
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-wtd7k
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-wtd7k ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-wtd7k
++ write_and_verify_logs test5 allow_empty
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496809435
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 04:23:55.279163725+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 04:23:55.279163725+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ '[' -z logging-es-ops-data-master-it0wbr7x-1-71pzg ']'
+++ uuidgen
++ local uuid_es=09630389-a489-48d5-ae4f-4a85250d2409
+++ uuidgen
++ local uuid_es_ops=31eb2b98-a173-47d3-bffc-7102c8c39d49
++ local expected=1
++ local timeout=300
++ add_test_message 09630389-a489-48d5-ae4f-4a85250d2409
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-n0b8v
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/09630389-a489-48d5-ae4f-4a85250d2409
added es message 09630389-a489-48d5-ae4f-4a85250d2409
++ echo added es message 09630389-a489-48d5-ae4f-4a85250d2409
++ logger -i -p local6.info -t 31eb2b98-a173-47d3-bffc-7102c8c39d49 31eb2b98-a173-47d3-bffc-7102c8c39d49
added es-ops message 31eb2b98-a173-47d3-bffc-7102c8c39d49
++ echo added es-ops message 31eb2b98-a173-47d3-bffc-7102c8c39d49
++ local rc=0
++ espod=logging-es-data-master-jpwqs1vo-1-6r16x
++ myproject=project.logging
++ mymessage=09630389-a489-48d5-ae4f-4a85250d2409
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 09630389-a489-48d5-ae4f-4a85250d2409
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 09630389-a489-48d5-ae4f-4a85250d2409
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 09630389-a489-48d5-ae4f-4a85250d2409
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 09630389-a489-48d5-ae4f-4a85250d2409
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 09630389-a489-48d5-ae4f-4a85250d2409
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 09630389-a489-48d5-ae4f-4a85250d2409
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 09630389-a489-48d5-ae4f-4a85250d2409
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-jpwqs1vo-1-6r16x project.logging _count message 09630389-a489-48d5-ae4f-4a85250d2409
+++ get_count_from_json
+++ curl_es logging-es-data-master-jpwqs1vo-1-6r16x '/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409' --connect-timeout 1
+++ local pod=logging-es-data-master-jpwqs1vo-1-6r16x
+++ local 'endpoint=/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-jpwqs1vo-1-6r16x -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 293 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 09630389-a489-48d5-ae4f-4a85250d2409
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 09630389-a489-48d5-ae4f-4a85250d2409
++ espod=logging-es-ops-data-master-it0wbr7x-1-71pzg
++ myproject=.operations
++ mymessage=31eb2b98-a173-47d3-bffc-7102c8c39d49
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 31eb2b98-a173-47d3-bffc-7102c8c39d49
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:31eb2b98-a173-47d3-bffc-7102c8c39d49' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:31eb2b98-a173-47d3-bffc-7102c8c39d49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:31eb2b98-a173-47d3-bffc-7102c8c39d49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 31eb2b98-a173-47d3-bffc-7102c8c39d49
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:31eb2b98-a173-47d3-bffc-7102c8c39d49' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:31eb2b98-a173-47d3-bffc-7102c8c39d49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:31eb2b98-a173-47d3-bffc-7102c8c39d49'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-it0wbr7x-1-71pzg .operations _count systemd.u.SYSLOG_IDENTIFIER 31eb2b98-a173-47d3-bffc-7102c8c39d49
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-it0wbr7x-1-71pzg '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:31eb2b98-a173-47d3-bffc-7102c8c39d49' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-it0wbr7x-1-71pzg
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:31eb2b98-a173-47d3-bffc-7102c8c39d49'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-it0wbr7x-1-71pzg -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:31eb2b98-a173-47d3-bffc-7102c8c39d49'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 298 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 31eb2b98-a173-47d3-bffc-7102c8c39d49
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 31eb2b98-a173-47d3-bffc-7102c8c39d49
++ '[' -n get_logmessage ']'
++ get_logmessage 09630389-a489-48d5-ae4f-4a85250d2409
++ logmessage=09630389-a489-48d5-ae4f-4a85250d2409
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 31eb2b98-a173-47d3-bffc-7102c8c39d49
++ logmessage2=31eb2b98-a173-47d3-bffc-7102c8c39d49
+++ date +%s
++ local endtime=1496809449
+++ expr 1496809449 - 1496809435
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 04:24:09.466243659+00:00
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 04:24:09.466243659+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-n0b8v
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es project.logging _search message 09630389-a489-48d5-ae4f-4a85250d2409
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:09630389-a489-48d5-ae4f-4a85250d2409'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-n0b8v logging-es-ops .operations _search message 31eb2b98-a173-47d3-bffc-7102c8c39d49
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-n0b8v -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer eBdz5NJmnyWWXc0XNqrN6VNoDnoJNxaf5aKyC_Fc_2E' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:31eb2b98-a173-47d3-bffc-7102c8c39d49'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ cleanup
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
template "logging-fluentd-template" replaced
++ remove_cdm_env
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.tUXJwDsGVa
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-wtd7k
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-wtd7k
++ '[' -z logging-fluentd-wtd7k -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-wtd7k
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
+++ oc get pods -l component=fluentd
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-8l4ql
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-8l4ql ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
SKIPPING reinstall test for now
/data/src/github.com/openshift/origin-aggregated-logging/hack/lib/log/system.sh: line 31:  4612 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"  (wd: /data/src/github.com/openshift/origin-aggregated-logging)
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to /tmp/origin-aggregated-logging/artifacts/events.txt
[INFO] [CLEANUP] Dumping etcd contents to /tmp/origin-aggregated-logging/artifacts/etcd
[WARNING] No compiled `etcdhelper` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh tools/etcdhelper
++ Building go targets for linux/amd64: tools/etcdhelper
/data/src/github.com/openshift/origin-aggregated-logging/../origin/hack/build-go.sh took 112 seconds
2017-06-07 00:26:27.084428 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
[INFO] [CLEANUP] Dumping container logs to /tmp/origin-aggregated-logging/logs/containers
[INFO] [CLEANUP] Truncating log files over 200M
[INFO] [CLEANUP] Stopping docker containers
[INFO] [CLEANUP] Removing docker containers
Error: No such image, container or task: a4b5d466ee2e
json: cannot unmarshal array into Go value of type types.ContainerJSON
Error response from daemon: devmapper: Unknown device 8e34678e8ecd5c733c65c1582eac1f0c866247019ec6d7fc918b75add0cd15a5
json: cannot unmarshal array into Go value of type types.ContainerJSON
Error: No such image, container or task: 06f43cdd166e
json: cannot unmarshal array into Go value of type types.ContainerJSON
Error response from daemon: You cannot remove a running container cb235acc891bbd082a6b37422c984c187eecfd7235dd783f0a632710873cf86b. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container ce7a8b36d2c7f333d1c70bf05ce6580e779789a2f77aeb69d21ee246d92a1721. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 40fb8740e15ea55f503b79afc1962d388447e2f3b06d8d6f45f89190db10ba2c. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container acf0c89a34441e2722ebb21e9a8d0134c7276515bd13342a766077ba0164012a. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container bc13df358fbd417895e36a226ae25e71279bdb52b9eb51177963d66122ed3829. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 59786e7126d0f530171e59140efe3a5c67b31150870cc64d9a6b02e3f8158368. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 54c5fec689825e16b232e9d5643b2f6c9c4881ee709507c7db05bc700cd65598. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 89d6cac7fa6200188c5f9a2e168ba33854341bf79a7371335a966847e2ceba04. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 53b48fb16d590f2b7ca66f470fe5c7629f61e7b9b99cfb4b73cfe0ee6e09402e. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 35cbdba0a916bb7e96ffcc5dedff941fb3879e8bea535a7c8175259d79321623. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container dc4e838d29af2e3003c2055352be030e6521b853d263702ad5635e85da87796a. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 8f7e5eab1bdf046e174b883bace3961d395d507a36787e677d1a1810a4869e0d. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 77ef90ccd2eca661b9c5e9bc5b364b092c9852b6a04a06429ac18e0e22377912. Stop the container before attempting removal or use -f
[INFO] [CLEANUP] Killing child processes
[INFO] [CLEANUP] Pruning etcd data directory
[INFO] /data/src/github.com/openshift/origin-aggregated-logging/logging.sh exited with code 0 after 01h 05m 01s

real	65m1.121s
user	6m3.989s
sys	1m5.833s
Finished GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh
***************************************************
==> openshiftdev: Downloading logs
==> openshiftdev: Downloading artifacts from '/var/log/yum.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/yum.log'
==> openshiftdev: Downloading artifacts from '/var/log/secure' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/secure'
==> openshiftdev: Downloading artifacts from '/var/log/audit/audit.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/audit.log'
==> openshiftdev: Downloading artifacts from '/tmp/origin-aggregated-logging/' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts'
+ false
+ false
+ test_pull_requests --mark_test_success --repo origin-aggregated-logging --config /var/lib/jenkins/.test_pull_requests_logging.json
Rate limit remaining: 2660
  Marking SUCCESS for pull request #--repo in repo ''
/usr/share/ruby/net/http/response.rb:119:in `error!': 404 "Not Found" (Net::HTTPServerException)
	from /bin/test_pull_requests:543:in `block in get_comments'
	from /bin/test_pull_requests:535:in `each'
	from /bin/test_pull_requests:535:in `get_comments'
	from /bin/test_pull_requests:1074:in `get_comment_matching_regex'
	from /bin/test_pull_requests:1090:in `get_comment_with_prefix'
	from /bin/test_pull_requests:779:in `mark_test_success'
	from /bin/test_pull_requests:2361:in `<main>'
+ true
[description-setter] Could not determine description.
[PostBuildScript] - Execution post build scripts.
[workspace] $ /bin/sh -xe /tmp/hudson5016764017260192906.sh
+ INSTANCE_NAME=origin_logging-rhel7-1578
+ pushd origin
~/jobs/test-origin-aggregated-logging/workspace/origin ~/jobs/test-origin-aggregated-logging/workspace
+ rc=0
+ '[' -f .vagrant-openshift.json ']'
++ /usr/bin/vagrant ssh -c 'sudo ausearch -m avc'
+ ausearchresult='<no matches>'
+ rc=1
+ '[' '<no matches>' = '<no matches>' ']'
+ rc=0
+ /usr/bin/vagrant destroy -f
==> openshiftdev: Terminating the instance...
==> openshiftdev: Running cleanup tasks for 'shell' provisioner...
+ popd
~/jobs/test-origin-aggregated-logging/workspace
+ exit 0
Finished: SUCCESS