Console Output

Skipping 569 KB.. Full Log
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 264 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 263 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ get_count_from_json
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 262 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 261 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 260 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 259 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 258 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 257 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 256 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 255 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ get_count_from_json
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 254 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 253 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 252 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 251 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 250 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 249 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 248 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 247 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 246 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message f5114d56-0bfd-4a1a-9e31-a92da288aab7
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:f5114d56-0bfd-4a1a-9e31-a92da288aab7'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 246 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for f5114d56-0bfd-4a1a-9e31-a92da288aab7
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for f5114d56-0bfd-4a1a-9e31-a92da288aab7
++ espod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ myproject=.operations
++ mymessage=fd83df75-f2cd-4291-a039-366435b8cee7
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER fd83df75-f2cd-4291-a039-366435b8cee7
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fd83df75-f2cd-4291-a039-366435b8cee7' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fd83df75-f2cd-4291-a039-366435b8cee7'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fd83df75-f2cd-4291-a039-366435b8cee7'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for fd83df75-f2cd-4291-a039-366435b8cee7
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for fd83df75-f2cd-4291-a039-366435b8cee7
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496797699
+++ expr 1496797699 - 1496797623
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 76 seconds at 2017-06-07 01:08:19.622873900+00:00
++ echo END wait_for_fluentd_to_catch_up took 76 seconds at 2017-06-07 01:08:19.622873900+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.5.204" labeled
++ wait_for_pod_ACTION stop logging-fluentd-286xs
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-286xs
++ '[' -z logging-fluentd-286xs -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-286xs
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
daemonset "logging-forward-fluentd" deleted
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '      <match **>' ']'
++ oc get configmap/logging-fluentd -o yaml
++ sed -e '/<match \*\*>/ d' -e '/@include configs\.d\/user\/secure-forward\.conf/ d' -e '/<\/match>/ d'
++ oc replace -f -
configmap "logging-fluentd" replaced
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.5.204" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
pod for component=fluentd not running yet
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-gwj6q
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-gwj6q ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-gwj6q
++ oc get events -o yaml
++ write_and_verify_logs 1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up '' ''
+++ date +%s
++ local starttime=1496797715
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 01:08:35.128813457+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 01:08:35.128813457+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ '[' -z logging-es-ops-data-master-nhj4cm2p-1-077m0 ']'
+++ uuidgen
++ local uuid_es=ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ uuidgen
++ local uuid_es_ops=58089843-e122-407a-b3b5-c0be1719e586
++ local expected=1
++ local timeout=300
++ add_test_message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-2d3s6
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/ad683c69-dfc0-4d01-babf-eb3406a41cce
added es message ad683c69-dfc0-4d01-babf-eb3406a41cce
++ echo added es message ad683c69-dfc0-4d01-babf-eb3406a41cce
++ logger -i -p local6.info -t 58089843-e122-407a-b3b5-c0be1719e586 58089843-e122-407a-b3b5-c0be1719e586
added es-ops message 58089843-e122-407a-b3b5-c0be1719e586
++ echo added es-ops message 58089843-e122-407a-b3b5-c0be1719e586
++ local rc=0
++ espod=logging-es-data-master-pr08bexf-1-hc6s6
++ myproject=project.logging
++ mymessage=ad683c69-dfc0-4d01-babf-eb3406a41cce
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 293 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 292 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 291 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 290 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message ad683c69-dfc0-4d01-babf-eb3406a41cce
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:ad683c69-dfc0-4d01-babf-eb3406a41cce'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 290 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for ad683c69-dfc0-4d01-babf-eb3406a41cce
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for ad683c69-dfc0-4d01-babf-eb3406a41cce
++ espod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ myproject=.operations
++ mymessage=58089843-e122-407a-b3b5-c0be1719e586
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER 58089843-e122-407a-b3b5-c0be1719e586
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:58089843-e122-407a-b3b5-c0be1719e586' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:58089843-e122-407a-b3b5-c0be1719e586'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:58089843-e122-407a-b3b5-c0be1719e586'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 58089843-e122-407a-b3b5-c0be1719e586
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 58089843-e122-407a-b3b5-c0be1719e586
++ '[' -n '' ']'
++ '[' -n '' ']'
+++ date +%s
++ local endtime=1496797730
+++ expr 1496797730 - 1496797715
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 01:08:50.567359144+00:00
++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 01:08:50.567359144+00:00
++ return 0
++ return 0
++ cleanup
++ cleanup_forward
++ oc label node --all logging-infra-fluentd-
node "172.18.5.204" labeled
++ wait_for_pod_ACTION stop logging-fluentd-gwj6q
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-gwj6q
++ '[' -z logging-fluentd-gwj6q -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-gwj6q
++ '[' -n 1 ']'
pod logging-fluentd-gwj6q still running
++ echo pod logging-fluentd-gwj6q still running
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' stop = start ']'
++ '[' 110 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-gwj6q
++ '[' stop = start ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
++ oc delete daemonset/logging-forward-fluentd
Error from server (NotFound): daemonsets.extensions "logging-forward-fluentd" not found
++ :
+++ oc get configmap/logging-fluentd -o yaml
+++ grep '<match \*\*>'
++ '[' -n '' ']'
++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\
# @type secure_forward\n\
# self_hostname forwarding-${HOSTNAME}\n\
# shared_key aggregated_logging_ci_testing\n\
#  secure no\n\
#  <server>\n\
#   host ${FLUENTD_FORWARD}\n\
#   port 24284\n\
#  </server>"}]'
configmap "logging-fluentd" patched
++ oc label node --all logging-infra-fluentd=true
node "172.18.5.204" labeled
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-01rrj
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-01rrj ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-01rrj
++ oc get events -o yaml
running test test-json-parsing.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ es_pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ uuidgen
++ uuid_es=488a29ab-be55-4338-b1e8-389451114f64
++ echo Adding test message 488a29ab-be55-4338-b1e8-389451114f64 to Kibana . . .
Adding test message 488a29ab-be55-4338-b1e8-389451114f64 to Kibana . . .
++ add_test_message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-2d3s6
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/488a29ab-be55-4338-b1e8-389451114f64
++ rc=0
++ timeout=600
Waiting 600 seconds for 488a29ab-be55-4338-b1e8-389451114f64 to show up in Elasticsearch . . .
++ echo Waiting 600 seconds for 488a29ab-be55-4338-b1e8-389451114f64 to show up in Elasticsearch . . .
++ espod=logging-es-data-master-pr08bexf-1-hc6s6
++ myproject=project.logging.
++ mymessage=488a29ab-be55-4338-b1e8-389451114f64
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 600
++ let ii=600
++ local interval=1
++ '[' 600 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _count message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 599 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _count message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 598 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _count message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 597 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _count message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 596 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _count message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 595 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _count message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 594 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _count message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 593 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _count message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 592 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _count message 488a29ab-be55-4338-b1e8-389451114f64
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 592 -le 0 ']'
++ return 0
good - ./logging.sh: found 1 record project logging for 488a29ab-be55-4338-b1e8-389451114f64
++ echo good - ./logging.sh: found 1 record project logging for 488a29ab-be55-4338-b1e8-389451114f64
Testing if record is in correct format . . .
++ echo Testing if record is in correct format . . .
++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging. _search message 488a29ab-be55-4338-b1e8-389451114f64
++ python test-json-parsing.py 488a29ab-be55-4338-b1e8-389451114f64
++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging.*/_search?q=message:488a29ab-be55-4338-b1e8-389451114f64' --connect-timeout 1
++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
++ local 'endpoint=/project.logging.*/_search?q=message:488a29ab-be55-4338-b1e8-389451114f64'
++ shift
++ shift
++ args=("${@:-}")
++ local args
++ local secret_dir=/etc/elasticsearch/secret/
++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_search?q=message:488a29ab-be55-4338-b1e8-389451114f64'
Success: record contains all of the expected fields/values
Success: ./logging.sh passed
++ echo Success: ./logging.sh passed
++ exit 0
running test test-mux.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ '[' false == false -o false == false ']'
Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.
++ echo 'Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.'
++ exit 0
SKIPPING upgrade test for now
running test test-viaq-data-model.sh
++ set -o nounset
++ set -o pipefail
++ type get_running_pod
++ [[ 1 -ne 1 ]]
++ [[ true = \f\a\l\s\e ]]
++ CLUSTER=true
++ ops=-ops
++ INDEX_PREFIX=
++ PROJ_PREFIX=project.
++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts
++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']'
++ get_test_user_token
++ local current_project
+++ oc project -q
++ current_project=logging
++ oc login --username=admin --password=admin
+++ oc whoami -t
++ test_token=s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY
+++ oc whoami
++ test_name=admin
++ test_ip=127.0.0.1
++ oc login --username=system:admin
++ oc project logging
++ TEST_DIVIDER=------------------------------------------
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-01rrj
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
+++ mktemp
++ cfg=/tmp/tmp.MwVdiKdT1n
++ cat
++ add_test_volume /tmp/tmp.MwVdiKdT1n
++ oc get template logging-fluentd-template -o json
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"].append({"name": "cdmtest", "mountPath": "/etc/fluent/configs.d/openshift/filter-pre-cdm-test.conf", "readOnly": True}); obj["objects"][0]["spec"]["template"]["spec"]["volumes"].append({"name": "cdmtest", "hostPath": {"path": "/tmp/tmp.MwVdiKdT1n"}}); print json.dumps(obj, indent=2)'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ trap cleanup INT TERM EXIT
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-01rrj
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-01rrj
++ '[' -z logging-fluentd-01rrj -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-01rrj
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-nx9lg
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-nx9lg ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-nx9lg
++ keep_fields=method,statusCode,type,@timestamp,req,res
++ write_and_verify_logs test1
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496797789
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 01:09:49.891202554+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 01:09:49.891202554+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ '[' -z logging-es-ops-data-master-nhj4cm2p-1-077m0 ']'
+++ uuidgen
++ local uuid_es=de11f973-7956-4487-bc1f-5ea2b882dafe
+++ uuidgen
++ local uuid_es_ops=fae3ac60-26de-4840-bf61-85a9e433cf0c
++ local expected=1
++ local timeout=300
++ add_test_message de11f973-7956-4487-bc1f-5ea2b882dafe
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-2d3s6
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/de11f973-7956-4487-bc1f-5ea2b882dafe
added es message de11f973-7956-4487-bc1f-5ea2b882dafe
++ echo added es message de11f973-7956-4487-bc1f-5ea2b882dafe
++ logger -i -p local6.info -t fae3ac60-26de-4840-bf61-85a9e433cf0c fae3ac60-26de-4840-bf61-85a9e433cf0c
added es-ops message fae3ac60-26de-4840-bf61-85a9e433cf0c
++ echo added es-ops message fae3ac60-26de-4840-bf61-85a9e433cf0c
++ local rc=0
++ espod=logging-es-data-master-pr08bexf-1-hc6s6
++ myproject=project.logging
++ mymessage=de11f973-7956-4487-bc1f-5ea2b882dafe
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message de11f973-7956-4487-bc1f-5ea2b882dafe
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message de11f973-7956-4487-bc1f-5ea2b882dafe
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message de11f973-7956-4487-bc1f-5ea2b882dafe
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message de11f973-7956-4487-bc1f-5ea2b882dafe
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message de11f973-7956-4487-bc1f-5ea2b882dafe
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message de11f973-7956-4487-bc1f-5ea2b882dafe
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message de11f973-7956-4487-bc1f-5ea2b882dafe
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 294 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for de11f973-7956-4487-bc1f-5ea2b882dafe
good - wait_for_fluentd_to_catch_up: found 1 record project logging for de11f973-7956-4487-bc1f-5ea2b882dafe
++ espod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ myproject=.operations
++ mymessage=fae3ac60-26de-4840-bf61-85a9e433cf0c
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER fae3ac60-26de-4840-bf61-85a9e433cf0c
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c' --connect-timeout 1
+++ get_count_from_json
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER fae3ac60-26de-4840-bf61-85a9e433cf0c
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c'
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER fae3ac60-26de-4840-bf61-85a9e433cf0c
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER fae3ac60-26de-4840-bf61-85a9e433cf0c
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:fae3ac60-26de-4840-bf61-85a9e433cf0c'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 297 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for fae3ac60-26de-4840-bf61-85a9e433cf0c
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for fae3ac60-26de-4840-bf61-85a9e433cf0c
++ '[' -n get_logmessage ']'
++ get_logmessage de11f973-7956-4487-bc1f-5ea2b882dafe
++ logmessage=de11f973-7956-4487-bc1f-5ea2b882dafe
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 fae3ac60-26de-4840-bf61-85a9e433cf0c
++ logmessage2=fae3ac60-26de-4840-bf61-85a9e433cf0c
+++ date +%s
++ local endtime=1496797804
+++ expr 1496797804 - 1496797789
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 01:10:04.445384313+00:00
++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-07 01:10:04.445384313+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-2d3s6
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es project.logging _search message de11f973-7956-4487-bc1f-5ea2b882dafe
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:de11f973-7956-4487-bc1f-5ea2b882dafe'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es-ops .operations _search message fae3ac60-26de-4840-bf61-85a9e433cf0c
++ python test-viaq-data-model.py test1
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:fae3ac60-26de-4840-bf61-85a9e433cf0c'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_USE_UNDEFINED '"true"'
+++ mktemp
++ junk=/tmp/tmp.rmJU9HAhTC
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.rmJU9HAhTC'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.rmJU9HAhTC
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.KNO3JhfRRw
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.KNO3JhfRRw'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.KNO3JhfRRw
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-nx9lg
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-nx9lg
++ '[' -z logging-fluentd-nx9lg -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-nx9lg
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-mt3kz
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-mt3kz ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-mt3kz
++ write_and_verify_logs test2
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496797829
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 01:10:29.653103750+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 01:10:29.653103750+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ '[' -z logging-es-ops-data-master-nhj4cm2p-1-077m0 ']'
+++ uuidgen
++ local uuid_es=2143e38d-b4ff-4a80-a945-e0d173e59a59
+++ uuidgen
++ local uuid_es_ops=ceedc0ad-951b-4148-b8e3-fd98127bfbe8
++ local expected=1
++ local timeout=300
++ add_test_message 2143e38d-b4ff-4a80-a945-e0d173e59a59
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-2d3s6
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/2143e38d-b4ff-4a80-a945-e0d173e59a59
added es message 2143e38d-b4ff-4a80-a945-e0d173e59a59
++ echo added es message 2143e38d-b4ff-4a80-a945-e0d173e59a59
++ logger -i -p local6.info -t ceedc0ad-951b-4148-b8e3-fd98127bfbe8 ceedc0ad-951b-4148-b8e3-fd98127bfbe8
added es-ops message ceedc0ad-951b-4148-b8e3-fd98127bfbe8
++ echo added es-ops message ceedc0ad-951b-4148-b8e3-fd98127bfbe8
++ local rc=0
++ espod=logging-es-data-master-pr08bexf-1-hc6s6
++ myproject=project.logging
++ mymessage=2143e38d-b4ff-4a80-a945-e0d173e59a59
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 2143e38d-b4ff-4a80-a945-e0d173e59a59
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 2143e38d-b4ff-4a80-a945-e0d173e59a59
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 2143e38d-b4ff-4a80-a945-e0d173e59a59
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 2143e38d-b4ff-4a80-a945-e0d173e59a59
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 2143e38d-b4ff-4a80-a945-e0d173e59a59
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 2143e38d-b4ff-4a80-a945-e0d173e59a59
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 2143e38d-b4ff-4a80-a945-e0d173e59a59
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59' --connect-timeout 1
+++ get_count_from_json
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 294 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 2143e38d-b4ff-4a80-a945-e0d173e59a59
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 2143e38d-b4ff-4a80-a945-e0d173e59a59
++ espod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ myproject=.operations
++ mymessage=ceedc0ad-951b-4148-b8e3-fd98127bfbe8
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER ceedc0ad-951b-4148-b8e3-fd98127bfbe8
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER ceedc0ad-951b-4148-b8e3-fd98127bfbe8
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER ceedc0ad-951b-4148-b8e3-fd98127bfbe8
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER ceedc0ad-951b-4148-b8e3-fd98127bfbe8
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:ceedc0ad-951b-4148-b8e3-fd98127bfbe8'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 297 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for ceedc0ad-951b-4148-b8e3-fd98127bfbe8
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for ceedc0ad-951b-4148-b8e3-fd98127bfbe8
++ '[' -n get_logmessage ']'
++ get_logmessage 2143e38d-b4ff-4a80-a945-e0d173e59a59
++ logmessage=2143e38d-b4ff-4a80-a945-e0d173e59a59
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 ceedc0ad-951b-4148-b8e3-fd98127bfbe8
++ logmessage2=ceedc0ad-951b-4148-b8e3-fd98127bfbe8
+++ date +%s
++ local endtime=1496797843
+++ expr 1496797843 - 1496797829
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 01:10:43.771937971+00:00
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 01:10:43.771937971+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-2d3s6
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es project.logging _search message 2143e38d-b4ff-4a80-a945-e0d173e59a59
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:2143e38d-b4ff-4a80-a945-e0d173e59a59'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es-ops .operations _search message ceedc0ad-951b-4148-b8e3-fd98127bfbe8
++ python test-viaq-data-model.py test2
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:ceedc0ad-951b-4148-b8e3-fd98127bfbe8'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.w6aDSjS0h3
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.w6aDSjS0h3'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.w6aDSjS0h3
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-mt3kz
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-mt3kz
++ '[' -z logging-fluentd-mt3kz -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-mt3kz
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-k2trs
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-k2trs ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-k2trs
++ write_and_verify_logs test3
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496797869
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 01:11:09.676444004+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 01:11:09.676444004+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ '[' -z logging-es-ops-data-master-nhj4cm2p-1-077m0 ']'
+++ uuidgen
++ local uuid_es=51b35061-a2b9-409f-ba15-a1edb5543a35
+++ uuidgen
++ local uuid_es_ops=265bf708-2eb2-4a3b-8ba3-78645bfc03bd
++ local expected=1
++ local timeout=300
++ add_test_message 51b35061-a2b9-409f-ba15-a1edb5543a35
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-2d3s6
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/51b35061-a2b9-409f-ba15-a1edb5543a35
++ echo added es message 51b35061-a2b9-409f-ba15-a1edb5543a35
added es message 51b35061-a2b9-409f-ba15-a1edb5543a35
++ logger -i -p local6.info -t 265bf708-2eb2-4a3b-8ba3-78645bfc03bd 265bf708-2eb2-4a3b-8ba3-78645bfc03bd
++ echo added es-ops message 265bf708-2eb2-4a3b-8ba3-78645bfc03bd
added es-ops message 265bf708-2eb2-4a3b-8ba3-78645bfc03bd
++ local rc=0
++ espod=logging-es-data-master-pr08bexf-1-hc6s6
++ myproject=project.logging
++ mymessage=51b35061-a2b9-409f-ba15-a1edb5543a35
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 51b35061-a2b9-409f-ba15-a1edb5543a35
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 51b35061-a2b9-409f-ba15-a1edb5543a35
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 51b35061-a2b9-409f-ba15-a1edb5543a35
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 51b35061-a2b9-409f-ba15-a1edb5543a35
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 51b35061-a2b9-409f-ba15-a1edb5543a35
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 51b35061-a2b9-409f-ba15-a1edb5543a35
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message 51b35061-a2b9-409f-ba15-a1edb5543a35
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 294 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 51b35061-a2b9-409f-ba15-a1edb5543a35
good - wait_for_fluentd_to_catch_up: found 1 record project logging for 51b35061-a2b9-409f-ba15-a1edb5543a35
++ espod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ myproject=.operations
++ mymessage=265bf708-2eb2-4a3b-8ba3-78645bfc03bd
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER 265bf708-2eb2-4a3b-8ba3-78645bfc03bd
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:265bf708-2eb2-4a3b-8ba3-78645bfc03bd' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:265bf708-2eb2-4a3b-8ba3-78645bfc03bd'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:265bf708-2eb2-4a3b-8ba3-78645bfc03bd'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 300 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 265bf708-2eb2-4a3b-8ba3-78645bfc03bd
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 265bf708-2eb2-4a3b-8ba3-78645bfc03bd
++ '[' -n get_logmessage ']'
++ get_logmessage 51b35061-a2b9-409f-ba15-a1edb5543a35
++ logmessage=51b35061-a2b9-409f-ba15-a1edb5543a35
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 265bf708-2eb2-4a3b-8ba3-78645bfc03bd
++ logmessage2=265bf708-2eb2-4a3b-8ba3-78645bfc03bd
+++ date +%s
++ local endtime=1496797879
+++ expr 1496797879 - 1496797869
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 10 seconds at 2017-06-07 01:11:19.604139026+00:00
++ echo END wait_for_fluentd_to_catch_up took 10 seconds at 2017-06-07 01:11:19.604139026+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-2d3s6
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es project.logging _search message 51b35061-a2b9-409f-ba15-a1edb5543a35
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:51b35061-a2b9-409f-ba15-a1edb5543a35'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es-ops .operations _search message 265bf708-2eb2-4a3b-8ba3-78645bfc03bd
++ python test-viaq-data-model.py test3
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:265bf708-2eb2-4a3b-8ba3-78645bfc03bd'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ add_cdm_env_var_val CDM_UNDEFINED_NAME myname
+++ mktemp
++ junk=/tmp/tmp.O9RbwpFDa4
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.O9RbwpFDa4'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.O9RbwpFDa4
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-k2trs
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-k2trs
++ '[' -z logging-fluentd-k2trs -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-k2trs
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-2xvlm
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-2xvlm ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-2xvlm
++ write_and_verify_logs test4
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496797896
+++ date -u --rfc-3339=ns
START wait_for_fluentd_to_catch_up at 2017-06-07 01:11:36.398189590+00:00
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 01:11:36.398189590+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ '[' -z logging-es-ops-data-master-nhj4cm2p-1-077m0 ']'
+++ uuidgen
++ local uuid_es=fc57d773-680e-49db-bed8-6dd982c49cfd
+++ uuidgen
++ local uuid_es_ops=bcdfd86d-d943-420a-9993-287064da09c3
++ local expected=1
++ local timeout=300
++ add_test_message fc57d773-680e-49db-bed8-6dd982c49cfd
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-2d3s6
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/fc57d773-680e-49db-bed8-6dd982c49cfd
added es message fc57d773-680e-49db-bed8-6dd982c49cfd
++ echo added es message fc57d773-680e-49db-bed8-6dd982c49cfd
++ logger -i -p local6.info -t bcdfd86d-d943-420a-9993-287064da09c3 bcdfd86d-d943-420a-9993-287064da09c3
added es-ops message bcdfd86d-d943-420a-9993-287064da09c3
++ echo added es-ops message bcdfd86d-d943-420a-9993-287064da09c3
++ local rc=0
++ espod=logging-es-data-master-pr08bexf-1-hc6s6
++ myproject=project.logging
++ mymessage=fc57d773-680e-49db-bed8-6dd982c49cfd
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message fc57d773-680e-49db-bed8-6dd982c49cfd
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
+++ shift
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message fc57d773-680e-49db-bed8-6dd982c49cfd
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message fc57d773-680e-49db-bed8-6dd982c49cfd
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message fc57d773-680e-49db-bed8-6dd982c49cfd
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message fc57d773-680e-49db-bed8-6dd982c49cfd
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message fc57d773-680e-49db-bed8-6dd982c49cfd
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 295 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project logging for fc57d773-680e-49db-bed8-6dd982c49cfd
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for fc57d773-680e-49db-bed8-6dd982c49cfd
++ espod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ myproject=.operations
++ mymessage=bcdfd86d-d943-420a-9993-287064da09c3
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER bcdfd86d-d943-420a-9993-287064da09c3
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER bcdfd86d-d943-420a-9993-287064da09c3
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER bcdfd86d-d943-420a-9993-287064da09c3
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER bcdfd86d-d943-420a-9993-287064da09c3
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:bcdfd86d-d943-420a-9993-287064da09c3'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 297 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for bcdfd86d-d943-420a-9993-287064da09c3
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for bcdfd86d-d943-420a-9993-287064da09c3
++ '[' -n get_logmessage ']'
++ get_logmessage fc57d773-680e-49db-bed8-6dd982c49cfd
++ logmessage=fc57d773-680e-49db-bed8-6dd982c49cfd
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 bcdfd86d-d943-420a-9993-287064da09c3
++ logmessage2=bcdfd86d-d943-420a-9993-287064da09c3
+++ date +%s
++ local endtime=1496797909
+++ expr 1496797909 - 1496797896
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-07 01:11:49.145175416+00:00
++ echo END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-07 01:11:49.145175416+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-2d3s6
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es project.logging _search message fc57d773-680e-49db-bed8-6dd982c49cfd
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:fc57d773-680e-49db-bed8-6dd982c49cfd'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es-ops .operations _search message bcdfd86d-d943-420a-9993-287064da09c3
++ python test-viaq-data-model.py test4
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:bcdfd86d-d943-420a-9993-287064da09c3'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,empty1,undefined3,method,statusCode,type,@timestamp,req,res
+++ mktemp
++ junk=/tmp/tmp.jzeYLBfr6J
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.jzeYLBfr6J'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.jzeYLBfr6J
++ add_cdm_env_var_val CDM_KEEP_EMPTY_FIELDS undefined4,undefined5,empty1,undefined3
+++ mktemp
++ junk=/tmp/tmp.2pqeF8Y9dX
++ cat
++ oc get template logging-fluentd-template -o yaml
++ sed '/env:/r /tmp/tmp.2pqeF8Y9dX'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.2pqeF8Y9dX
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-2xvlm
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-2xvlm
++ '[' -z logging-fluentd-2xvlm -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-2xvlm
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
pod for component=fluentd not running yet
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-8zwrl
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-8zwrl ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ fpod=logging-fluentd-8zwrl
++ write_and_verify_logs test5 allow_empty
++ expected=1
++ rc=0
++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2
+++ date +%s
++ local starttime=1496797929
+++ date -u --rfc-3339=ns
++ echo START wait_for_fluentd_to_catch_up at 2017-06-07 01:12:09.602328802+00:00
START wait_for_fluentd_to_catch_up at 2017-06-07 01:12:09.602328802+00:00
+++ get_running_pod es
+++ oc get pods -l component=es
+++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ get_running_pod es-ops
+++ oc get pods -l component=es-ops
+++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}'
++ local es_ops_pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ '[' -z logging-es-ops-data-master-nhj4cm2p-1-077m0 ']'
+++ uuidgen
++ local uuid_es=a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
+++ uuidgen
++ local uuid_es_ops=27cd05af-eb95-4066-8c32-530fc7793910
++ local expected=1
++ local timeout=300
++ add_test_message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ local kib_pod=logging-kibana-1-2d3s6
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
added es message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
++ echo added es message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
++ logger -i -p local6.info -t 27cd05af-eb95-4066-8c32-530fc7793910 27cd05af-eb95-4066-8c32-530fc7793910
added es-ops message 27cd05af-eb95-4066-8c32-530fc7793910
++ echo added es-ops message 27cd05af-eb95-4066-8c32-530fc7793910
++ local rc=0
++ espod=logging-es-data-master-pr08bexf-1-hc6s6
++ myproject=project.logging
++ mymessage=a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
++ expected=1
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 296 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36' --connect-timeout 1
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 295 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 294 -gt 0 ']'
++ test_count_expected
++ myfield=message
+++ query_es_from_es logging-es-data-master-pr08bexf-1-hc6s6 project.logging _count message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
+++ get_count_from_json
+++ curl_es logging-es-data-master-pr08bexf-1-hc6s6 '/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-data-master-pr08bexf-1-hc6s6
+++ local 'endpoint=/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-data-master-pr08bexf-1-hc6s6 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 294 -le 0 ']'
++ return 0
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
good - wait_for_fluentd_to_catch_up: found 1 record project logging for a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
++ espod=logging-es-ops-data-master-nhj4cm2p-1-077m0
++ myproject=.operations
++ mymessage=27cd05af-eb95-4066-8c32-530fc7793910
++ expected=1
++ myfield=systemd.u.SYSLOG_IDENTIFIER
++ wait_until_cmd_or_err test_count_expected test_count_err 300
++ let ii=300
++ local interval=1
++ '[' 300 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER 27cd05af-eb95-4066-8c32-530fc7793910
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 299 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER 27cd05af-eb95-4066-8c32-530fc7793910
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 298 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER 27cd05af-eb95-4066-8c32-530fc7793910
+++ get_count_from_json
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910' --connect-timeout 1
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910'
++ local nrecs=0
++ test 0 = 1
++ sleep 1
++ let ii=ii-1
++ '[' 297 -gt 0 ']'
++ test_count_expected
++ myfield=systemd.u.SYSLOG_IDENTIFIER
+++ query_es_from_es logging-es-ops-data-master-nhj4cm2p-1-077m0 .operations _count systemd.u.SYSLOG_IDENTIFIER 27cd05af-eb95-4066-8c32-530fc7793910
+++ get_count_from_json
+++ curl_es logging-es-ops-data-master-nhj4cm2p-1-077m0 '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910' --connect-timeout 1
+++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)'
+++ local pod=logging-es-ops-data-master-nhj4cm2p-1-077m0
+++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910'
+++ shift
+++ shift
+++ args=("${@:-}")
+++ local args
+++ local secret_dir=/etc/elasticsearch/secret/
+++ oc exec logging-es-ops-data-master-nhj4cm2p-1-077m0 -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:27cd05af-eb95-4066-8c32-530fc7793910'
++ local nrecs=1
++ test 1 = 1
++ break
++ '[' 297 -le 0 ']'
++ return 0
good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 27cd05af-eb95-4066-8c32-530fc7793910
++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 27cd05af-eb95-4066-8c32-530fc7793910
++ '[' -n get_logmessage ']'
++ get_logmessage a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
++ logmessage=a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
++ '[' -n get_logmessage2 ']'
++ get_logmessage2 27cd05af-eb95-4066-8c32-530fc7793910
++ logmessage2=27cd05af-eb95-4066-8c32-530fc7793910
+++ date +%s
++ local endtime=1496797943
+++ expr 1496797943 - 1496797929
+++ date -u --rfc-3339=ns
END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 01:12:23.547789331+00:00
++ echo END wait_for_fluentd_to_catch_up took 14 seconds at 2017-06-07 01:12:23.547789331+00:00
++ return 0
+++ get_running_pod kibana
+++ oc get pods -l component=kibana
+++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}'
++ kpod=logging-kibana-1-2d3s6
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es project.logging _search message a284a4f8-75f2-4fb6-97d2-742ef0f6ae36
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:a284a4f8-75f2-4fb6-97d2-742ef0f6ae36'
++ :
++ '[' 0 = 0 ']'
++ curl_es_from_kibana logging-kibana-1-2d3s6 logging-es-ops .operations _search message 27cd05af-eb95-4066-8c32-530fc7793910
++ python test-viaq-data-model.py test5 allow_empty
++ oc exec logging-kibana-1-2d3s6 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer s4p2usNsKnFo_6PGIfuqiEDeRtmYGSyb-PoqkHAwceY' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:27cd05af-eb95-4066-8c32-530fc7793910'
++ :
++ '[' 0 '!=' 0 ']'
++ return 0
++ cleanup
++ remove_test_volume
++ oc get template logging-fluentd-template -o json
++ oc replace -f -
++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)'
template "logging-fluentd-template" replaced
++ remove_cdm_env
++ oc get template logging-fluentd-template -o yaml
++ sed '/- name: CDM_/,/value:/d'
++ oc replace -f -
template "logging-fluentd-template" replaced
++ rm -f /tmp/tmp.MwVdiKdT1n
++ restart_fluentd
++ oc delete daemonset logging-fluentd
daemonset "logging-fluentd" deleted
++ wait_for_pod_ACTION stop logging-fluentd-8zwrl
++ local ii=120
++ local incr=10
++ '[' stop = start ']'
++ curpod=logging-fluentd-8zwrl
++ '[' -z logging-fluentd-8zwrl -a -n '' ']'
++ '[' 120 -gt 0 ']'
++ '[' stop = stop ']'
++ oc describe pod/logging-fluentd-8zwrl
++ '[' stop = start ']'
++ break
++ '[' 120 -le 0 ']'
++ return 0
++ oc process logging-fluentd-template
++ oc create -f -
daemonset "logging-fluentd" created
++ wait_for_pod_ACTION start fluentd
++ local ii=120
++ local incr=10
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
pod for component=fluentd not running yet
++ curpod=
++ '[' 120 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z '' ']'
++ '[' -n '' ']'
++ '[' -n 1 ']'
++ echo pod for component=fluentd not running yet
++ sleep 10
+++ expr 120 - 10
++ ii=110
++ '[' start = start ']'
+++ get_running_pod fluentd
+++ oc get pods -l component=fluentd
+++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}'
++ curpod=logging-fluentd-wmxjj
++ '[' 110 -gt 0 ']'
++ '[' start = stop ']'
++ '[' start = start ']'
++ '[' -z logging-fluentd-wmxjj ']'
++ break
++ '[' 110 -le 0 ']'
++ return 0
SKIPPING reinstall test for now
/data/src/github.com/openshift/origin-aggregated-logging/hack/lib/log/system.sh: line 31:  4609 Terminated              sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}"  (wd: /data/src/github.com/openshift/origin-aggregated-logging)
[INFO] [CLEANUP] Beginning cleanup routines...
[INFO] [CLEANUP] Dumping cluster events to /tmp/origin-aggregated-logging/artifacts/events.txt
[INFO] [CLEANUP] Dumping etcd contents to /tmp/origin-aggregated-logging/artifacts/etcd
[WARNING] No compiled `etcdhelper` binary was found. Attempting to build one using:
[WARNING]   $ hack/build-go.sh tools/etcdhelper
++ Building go targets for linux/amd64: tools/etcdhelper
/data/src/github.com/openshift/origin-aggregated-logging/../origin/hack/build-go.sh took 324 seconds
2017-06-06 21:18:16.901671 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated
[INFO] [CLEANUP] Dumping container logs to /tmp/origin-aggregated-logging/logs/containers
[INFO] [CLEANUP] Truncating log files over 200M
[INFO] [CLEANUP] Stopping docker containers
[INFO] [CLEANUP] Removing docker containers
Error response from daemon: You cannot remove a running container c8ba71e187b7aa0172942c9e9d04490f90a2d0d1d72a0f67176264aea3583926. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container d9506c15b967ab2ae59ee8c3c115bd7e2fdd43665ac93f03b04e5636430c4a7e. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 0899d14681cff82a85d5cfa030079352f9edcdfd3cde560b4cafd97d8a028c5b. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container d1840dbe61376114fcc2cab9bf06ea5205afd324c1c0157ff00a83faac669b23. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 763e879cd96fe515151aa6fea6de8e6f9b4c97c1a09bdb22ff7e63326c7e2b77. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container d80fcbb0227fa203078f7359e2b87d6b0ccb0a2558ca2533251930c7296a955d. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 0d5cbaea2e343a24e1300bca1d623c7ded2eb88615ea21741973b106c572676c. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 45535e49c38689480fd958c920bd4344e35b2155dbdb5a7c2179fd8148cc116c. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 0ed79efa61e4c50d0be14dff1650736dbfc95e68489e6e1d5c846d809206da7e. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container ff7d25d49aad8c65a41520fcdc680856798376b2e7ffc32de7bbc3b95582044b. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 319c8bd720357a2599da34ad5bbb4c63322433455851aa3eade2830534cace7c. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 066f644d0b47ee3c0a02002e0752e06f5c61c37f78f36c16e4b7633fb2f17d78. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container 38bc1a2278d22a0450cd83161b249538cd4436874d92d30ce9b1623d0e1d7e0f. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container a16541823fa87a1a64b4af0bf182720224926f776779f3656d1ae04450f6320e. Stop the container before attempting removal or use -f
Error response from daemon: You cannot remove a running container d620b085d9711331e16bcc54e21aad7387f2788d59ca7b7d08c3e7a139df6d01. Stop the container before attempting removal or use -f
[INFO] [CLEANUP] Killing child processes
[INFO] [CLEANUP] Pruning etcd data directory
[INFO] /data/src/github.com/openshift/origin-aggregated-logging/logging.sh exited with code 0 after 01h 01m 05s

real	61m5.035s
user	5m16.446s
sys	0m57.879s
Finished GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh
***************************************************
==> openshiftdev: Downloading logs
==> openshiftdev: Downloading artifacts from '/var/log/yum.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/yum.log'
==> openshiftdev: Downloading artifacts from '/var/log/secure' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/secure'
==> openshiftdev: Downloading artifacts from '/var/log/audit/audit.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts/audit.log'
==> openshiftdev: Downloading artifacts from '/tmp/origin-aggregated-logging/' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace/origin/artifacts'
+ false
+ false
+ test_pull_requests --mark_test_success --repo origin-aggregated-logging --config /var/lib/jenkins/.test_pull_requests_logging.json
Rate limit remaining: 3266
  Marking SUCCESS for pull request #--repo in repo ''
/usr/share/ruby/net/http/response.rb:119:in `error!': 404 "Not Found" (Net::HTTPServerException)
	from /bin/test_pull_requests:543:in `block in get_comments'
	from /bin/test_pull_requests:535:in `each'
	from /bin/test_pull_requests:535:in `get_comments'
	from /bin/test_pull_requests:1074:in `get_comment_matching_regex'
	from /bin/test_pull_requests:1090:in `get_comment_with_prefix'
	from /bin/test_pull_requests:779:in `mark_test_success'
	from /bin/test_pull_requests:2361:in `<main>'
+ true
[description-setter] Could not determine description.
[PostBuildScript] - Execution post build scripts.
[workspace] $ /bin/sh -xe /tmp/hudson5763500608720750246.sh
+ INSTANCE_NAME=origin_logging-rhel7-1577
+ pushd origin
~/jobs/test-origin-aggregated-logging/workspace/origin ~/jobs/test-origin-aggregated-logging/workspace
+ rc=0
+ '[' -f .vagrant-openshift.json ']'
++ /usr/bin/vagrant ssh -c 'sudo ausearch -m avc'
+ ausearchresult='<no matches>'
+ rc=1
+ '[' '<no matches>' = '<no matches>' ']'
+ rc=0
+ /usr/bin/vagrant destroy -f
==> openshiftdev: Terminating the instance...
==> openshiftdev: Running cleanup tasks for 'shell' provisioner...
+ popd
~/jobs/test-origin-aggregated-logging/workspace
+ exit 0
Finished: SUCCESS