Started by upstream project "test_pull_request_origin_aggregated_logging" build number 92 originally caused by: Started by remote host 50.17.198.52 [EnvInject] - Loading node environment variables. Building in workspace /var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties content OS_ROOT=/data/src/github.com/openshift/origin INSTANCE_TYPE=c4.xlarge GITHUB_REPO=openshift OS=rhel7 TESTNAME=logging [EnvInject] - Variables injected successfully. [workspace@2] $ /bin/sh -xe /tmp/hudson218699302287761060.sh + false + unset GOPATH + REPO_NAME=origin-aggregated-logging + rm -rf origin-aggregated-logging + vagrant origin-local-checkout --replace --repo origin-aggregated-logging -b master You don't seem to have the GOPATH environment variable set on your system. See: 'go help gopath' for more details about GOPATH. Waiting for the cloning process to finish Cloning origin-aggregated-logging ... Submodule 'deployer/common' (https://github.com/openshift/origin-integration-common) registered for path 'deployer/common' Submodule 'kibana-proxy' (https://github.com/fabric8io/openshift-auth-proxy.git) registered for path 'kibana-proxy' Cloning into 'deployer/common'... Submodule path 'deployer/common': checked out '45bf993212cdcbab5cbce3b3fab74a72b851402e' Cloning into 'kibana-proxy'... Submodule path 'kibana-proxy': checked out '118dfb40f7a8082d370ba7f4805255c9ec7c8178' Origin repositories cloned into /var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2 + pushd origin-aggregated-logging ~/jobs/test-origin-aggregated-logging/workspace@2/origin-aggregated-logging ~/jobs/test-origin-aggregated-logging/workspace@2 + git checkout master Already on 'master' + popd ~/jobs/test-origin-aggregated-logging/workspace@2 + '[' -n '' ']' + vagrant origin-local-checkout --replace You don't seem to have the GOPATH environment variable set on your system. See: 'go help gopath' for more details about GOPATH. Waiting for the cloning process to finish Checking repo integrity for /var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2 # On branch master # Untracked files: # (use "git add <file>..." to include in what will be committed) # # artifacts/ nothing added to commit but untracked files present (use "git add" to track) ~/jobs/test-origin-aggregated-logging/workspace@2 Replacing: /var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2 From https://github.com/openshift/origin cc2ed8f..1565480 master -> origin/master Already on 'master' Your branch is behind 'origin/master' by 4 commits, and can be fast-forwarded. (use "git pull" to update your local branch) HEAD is now at 1565480 Merge pull request #13663 from imcsk8/haproxy-ipv6 Removing .vagrant-openshift.json Removing .vagrant/ Removing artifacts/ fatal: branch name required ~/jobs/test-origin-aggregated-logging/workspace@2 Origin repositories cloned into /var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2 + pushd origin ~/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2 + INSTANCE_NAME=origin_logging-rhel7-1655 + GIT_URL=https://github.com/openshift/origin-aggregated-logging ++ echo https://github.com/openshift/origin-aggregated-logging ++ sed s,https://,, + OAL_LOCAL_PATH=github.com/openshift/origin-aggregated-logging + OS_O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging + env + sort _=/bin/env BRANCH=master BUILD_CAUSE=UPSTREAMTRIGGER BUILD_CAUSE_UPSTREAMTRIGGER=true BUILD_DISPLAY_NAME=#1655 BUILD_ID=1655 BUILD_NUMBER=1655 BUILD_TAG=jenkins-test-origin-aggregated-logging-1655 BUILD_URL=https://ci.openshift.redhat.com/jenkins/job/test-origin-aggregated-logging/1655/ EXECUTOR_NUMBER=51 GITHUB_REPO=openshift HOME=/var/lib/jenkins HUDSON_COOKIE=8b29e6d6-e658-4d77-b3fb-24636c280482 HUDSON_HOME=/var/lib/jenkins HUDSON_SERVER_COOKIE=ec11f8b2841c966f HUDSON_URL=https://ci.openshift.redhat.com/jenkins/ INSTANCE_TYPE=c4.xlarge JENKINS_HOME=/var/lib/jenkins JENKINS_SERVER_COOKIE=ec11f8b2841c966f JENKINS_URL=https://ci.openshift.redhat.com/jenkins/ JOB_BASE_NAME=test-origin-aggregated-logging JOB_DISPLAY_URL=https://ci.openshift.redhat.com/jenkins/job/test-origin-aggregated-logging/display/redirect JOB_NAME=test-origin-aggregated-logging JOB_URL=https://ci.openshift.redhat.com/jenkins/job/test-origin-aggregated-logging/ LANG=en_US.UTF-8 LOGNAME=jenkins MERGE=false MERGE_SEVERITY=none NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat NODE_LABELS=master NODE_NAME=master OLDPWD=/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2 OPENSHIFT_ANSIBLE_TARGET_BRANCH=master ORIGIN_AGGREGATED_LOGGING_PULL_ID=464 ORIGIN_AGGREGATED_LOGGING_TARGET_BRANCH=master OS_ANSIBLE_BRANCH=master OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS=rhel7 OS_ROOT=/data/src/github.com/openshift/origin PATH=/sbin:/usr/sbin:/bin:/usr/bin PWD=/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin ROOT_BUILD_CAUSE=REMOTECAUSE ROOT_BUILD_CAUSE_REMOTECAUSE=true RUN_CHANGES_DISPLAY_URL=https://ci.openshift.redhat.com/jenkins/job/test-origin-aggregated-logging/1655/display/redirect?page=changes RUN_DISPLAY_URL=https://ci.openshift.redhat.com/jenkins/job/test-origin-aggregated-logging/1655/display/redirect SHELL=/bin/bash SHLVL=3 TESTNAME=logging TEST_PERF=false USER=jenkins WORKSPACE=/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2 XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt + vagrant origin-init --stage inst --os rhel7 --instance-type c4.xlarge origin_logging-rhel7-1655 Reading AWS credentials from /var/lib/jenkins/.awscred Searching devenv-rhel7_* for latest base AMI (required_name_tag=) Found: ami-83a1fc95 (devenv-rhel7_6323) ++ seq 0 2 + for i in '$(seq 0 2)' + vagrant up --provider aws Bringing machine 'openshiftdev' up with 'aws' provider... ==> openshiftdev: Warning! The AWS provider doesn't support any of the Vagrant ==> openshiftdev: high-level network configurations (`config.vm.network`). They ==> openshiftdev: will be silently ignored. ==> openshiftdev: Warning! You're launching this instance into a VPC without an ==> openshiftdev: elastic IP. Please verify you're properly connected to a VPN so ==> openshiftdev: you can access this machine, otherwise Vagrant will not be able ==> openshiftdev: to SSH into it. ==> openshiftdev: Launching an instance with the following settings... ==> openshiftdev: -- Type: c4.xlarge ==> openshiftdev: -- AMI: ami-83a1fc95 ==> openshiftdev: -- Region: us-east-1 ==> openshiftdev: -- Keypair: libra ==> openshiftdev: -- Subnet ID: subnet-cf57c596 ==> openshiftdev: -- User Data: yes ==> openshiftdev: -- User Data: ==> openshiftdev: # cloud-config ==> openshiftdev: ==> openshiftdev: growpart: ==> openshiftdev: mode: auto ==> openshiftdev: devices: ['/'] ==> openshiftdev: runcmd: ==> openshiftdev: - [ sh, -xc, "sed -i s/^Defaults.*requiretty/#Defaults requiretty/g /etc/sudoers"] ==> openshiftdev: ==> openshiftdev: -- Block Device Mapping: [{"DeviceName"=>"/dev/sda1", "Ebs.VolumeSize"=>25, "Ebs.VolumeType"=>"gp2"}, {"DeviceName"=>"/dev/sdb", "Ebs.VolumeSize"=>35, "Ebs.VolumeType"=>"gp2"}] ==> openshiftdev: -- Terminate On Shutdown: false ==> openshiftdev: -- Monitoring: false ==> openshiftdev: -- EBS optimized: false ==> openshiftdev: -- Assigning a public IP address in a VPC: false ==> openshiftdev: Waiting for instance to become "ready"... ==> openshiftdev: Waiting for SSH to become available... ==> openshiftdev: Machine is booted and ready for use! ==> openshiftdev: Running provisioner: setup (shell)... openshiftdev: Running: /tmp/vagrant-shell20170609-19009-1rojyhp.sh ==> openshiftdev: Host: ec2-54-152-244-105.compute-1.amazonaws.com + break + vagrant sync-origin-aggregated-logging -c -s Running ssh/sudo command 'rm -rf /data/src/github.com/openshift/origin-aggregated-logging-bare; ' with timeout 14400. Attempt #0 Running ssh/sudo command 'mkdir -p /ec2-user/.ssh; mv /tmp/file20170609-19492-442miy /ec2-user/.ssh/config && chown ec2-user:ec2-user /ec2-user/.ssh/config && chmod 0600 /ec2-user/.ssh/config' with timeout 14400. Attempt #0 Running ssh/sudo command 'mkdir -p /data/src/github.com/openshift/' with timeout 14400. Attempt #0 Running ssh/sudo command 'mkdir -p /data/src/github.com/openshift/builder && chown -R ec2-user:ec2-user /data/src/github.com/openshift/' with timeout 14400. Attempt #0 Running ssh/sudo command 'set -e rm -fr /data/src/github.com/openshift/origin-aggregated-logging-bare; if [ ! -d /data/src/github.com/openshift/origin-aggregated-logging-bare ]; then git clone --quiet --bare https://github.com/openshift/origin-aggregated-logging.git /data/src/github.com/openshift/origin-aggregated-logging-bare >/dev/null fi ' with timeout 14400. Attempt #0 Synchronizing local sources Synchronizing [origin-aggregated-logging@master] from origin-aggregated-logging... Warning: Permanently added '54.152.244.105' (ECDSA) to the list of known hosts. Running ssh/sudo command 'set -e if [ -d /data/src/github.com/openshift/origin-aggregated-logging-bare ]; then rm -rf /data/src/github.com/openshift/origin-aggregated-logging echo 'Cloning origin-aggregated-logging ...' git clone --quiet --recurse-submodules /data/src/github.com/openshift/origin-aggregated-logging-bare /data/src/github.com/openshift/origin-aggregated-logging else MISSING_REPO+='origin-aggregated-logging-bare' fi if [ -n "$MISSING_REPO" ]; then echo 'Missing required upstream repositories:' echo $MISSING_REPO echo 'To fix, execute command: vagrant clone-upstream-repos' fi ' with timeout 14400. Attempt #0 Cloning origin-aggregated-logging ... Submodule 'deployer/common' (https://github.com/openshift/origin-integration-common) registered for path 'deployer/common' Submodule 'kibana-proxy' (https://github.com/fabric8io/openshift-auth-proxy.git) registered for path 'kibana-proxy' Cloning into 'deployer/common'... Submodule path 'deployer/common': checked out '45bf993212cdcbab5cbce3b3fab74a72b851402e' Cloning into 'kibana-proxy'... Submodule path 'kibana-proxy': checked out '118dfb40f7a8082d370ba7f4805255c9ec7c8178' + vagrant ssh -c 'if [ ! -d /tmp/openshift ] ; then mkdir /tmp/openshift ; fi ; sudo chmod 777 /tmp/openshift' + for image in openshift/base-centos7 centos:centos7 openshift/origin-logging-elasticsearch openshift/origin-logging-fluentd openshift/origin-logging-curator openshift/origin-logging-kibana + echo pulling image openshift/base-centos7 ... pulling image openshift/base-centos7 ... + vagrant ssh -c 'docker pull openshift/base-centos7' -- -n Using default tag: latest Trying to pull repository docker.io/openshift/base-centos7 ... latest: Pulling from docker.io/openshift/base-centos7 45a2e645736c: Pulling fs layer 734fb161cf89: Pulling fs layer 78efc9e155c4: Pulling fs layer 8a3400b7e31a: Pulling fs layer 8a3400b7e31a: Waiting 734fb161cf89: Verifying Checksum 734fb161cf89: Download complete 8a3400b7e31a: Verifying Checksum 8a3400b7e31a: Download complete 78efc9e155c4: Verifying Checksum 78efc9e155c4: Download complete 45a2e645736c: Verifying Checksum 45a2e645736c: Download complete 45a2e645736c: Pull complete 734fb161cf89: Pull complete 78efc9e155c4: Pull complete 8a3400b7e31a: Pull complete Digest: sha256:aea292a3bddba020cde0ee83e6a45807931eb607c164ec6a3674f67039d8cd7c + echo done with openshift/base-centos7 done with openshift/base-centos7 + for image in openshift/base-centos7 centos:centos7 openshift/origin-logging-elasticsearch openshift/origin-logging-fluentd openshift/origin-logging-curator openshift/origin-logging-kibana + echo pulling image centos:centos7 ... pulling image centos:centos7 ... + vagrant ssh -c 'docker pull centos:centos7' -- -n Trying to pull repository docker.io/library/centos ... centos7: Pulling from docker.io/library/centos Digest: sha256:aebf12af704307dfa0079b3babdca8d7e8ff6564696882bcb5d11f1d461f9ee9 + echo done with centos:centos7 done with centos:centos7 + for image in openshift/base-centos7 centos:centos7 openshift/origin-logging-elasticsearch openshift/origin-logging-fluentd openshift/origin-logging-curator openshift/origin-logging-kibana + echo pulling image openshift/origin-logging-elasticsearch ... pulling image openshift/origin-logging-elasticsearch ... + vagrant ssh -c 'docker pull openshift/origin-logging-elasticsearch' -- -n Using default tag: latest Trying to pull repository docker.io/openshift/origin-logging-elasticsearch ... latest: Pulling from docker.io/openshift/origin-logging-elasticsearch d5e46245fe40: Already exists c6633336ebec: Pulling fs layer c8922b1f556c: Pulling fs layer 8efa1082b536: Pulling fs layer 1b2efd2f86a8: Pulling fs layer 0c23282fef8c: Pulling fs layer 1181763a1fee: Pulling fs layer 4943cf20f63b: Pulling fs layer 3ca1d3f72a67: Pulling fs layer ff2b5d1f1edd: Pulling fs layer f633748cd255: Pulling fs layer 1181763a1fee: Waiting 4943cf20f63b: Waiting 3ca1d3f72a67: Waiting ff2b5d1f1edd: Waiting f633748cd255: Waiting 1b2efd2f86a8: Waiting 0c23282fef8c: Waiting c6633336ebec: Verifying Checksum c6633336ebec: Download complete 8efa1082b536: Verifying Checksum 8efa1082b536: Download complete 1b2efd2f86a8: Verifying Checksum 1b2efd2f86a8: Download complete 1181763a1fee: Verifying Checksum 1181763a1fee: Download complete 4943cf20f63b: Verifying Checksum 0c23282fef8c: Verifying Checksum 0c23282fef8c: Download complete ff2b5d1f1edd: Verifying Checksum f633748cd255: Verifying Checksum f633748cd255: Download complete 3ca1d3f72a67: Verifying Checksum 3ca1d3f72a67: Download complete c8922b1f556c: Verifying Checksum c8922b1f556c: Download complete c6633336ebec: Pull complete c8922b1f556c: Pull complete 8efa1082b536: Pull complete 1b2efd2f86a8: Pull complete 0c23282fef8c: Pull complete 1181763a1fee: Pull complete 4943cf20f63b: Pull complete 3ca1d3f72a67: Pull complete ff2b5d1f1edd: Pull complete f633748cd255: Pull complete Digest: sha256:6296f1719676e970438cac4d912542b35ac786c14a15df892507007c4ecbe490 + echo done with openshift/origin-logging-elasticsearch done with openshift/origin-logging-elasticsearch + for image in openshift/base-centos7 centos:centos7 openshift/origin-logging-elasticsearch openshift/origin-logging-fluentd openshift/origin-logging-curator openshift/origin-logging-kibana + echo pulling image openshift/origin-logging-fluentd ... pulling image openshift/origin-logging-fluentd ... + vagrant ssh -c 'docker pull openshift/origin-logging-fluentd' -- -n Using default tag: latest Trying to pull repository docker.io/openshift/origin-logging-fluentd ... latest: Pulling from docker.io/openshift/origin-logging-fluentd d5e46245fe40: Already exists d7c28cc24dc2: Pulling fs layer 9175f9d06c1f: Pulling fs layer 91e5bb34ef30: Pulling fs layer 0c100caa1a42: Pulling fs layer 19d549a53f32: Pulling fs layer 0c100caa1a42: Waiting 19d549a53f32: Waiting 91e5bb34ef30: Verifying Checksum 91e5bb34ef30: Download complete 0c100caa1a42: Verifying Checksum 0c100caa1a42: Download complete 19d549a53f32: Verifying Checksum 19d549a53f32: Download complete 9175f9d06c1f: Verifying Checksum 9175f9d06c1f: Download complete d7c28cc24dc2: Download complete d7c28cc24dc2: Pull complete 9175f9d06c1f: Pull complete 91e5bb34ef30: Pull complete 0c100caa1a42: Pull complete 19d549a53f32: Pull complete Digest: sha256:cae7c21c9f111d4f5b481c14a65c597c67e715a8ffe3aee4c483100ee77296d7 + echo done with openshift/origin-logging-fluentd done with openshift/origin-logging-fluentd + for image in openshift/base-centos7 centos:centos7 openshift/origin-logging-elasticsearch openshift/origin-logging-fluentd openshift/origin-logging-curator openshift/origin-logging-kibana + echo pulling image openshift/origin-logging-curator ... pulling image openshift/origin-logging-curator ... + vagrant ssh -c 'docker pull openshift/origin-logging-curator' -- -n Using default tag: latest Trying to pull repository docker.io/openshift/origin-logging-curator ... latest: Pulling from docker.io/openshift/origin-logging-curator d5e46245fe40: Already exists 73c202020d66: Pulling fs layer b330097dd4ed: Pulling fs layer 73c202020d66: Download complete b330097dd4ed: Verifying Checksum b330097dd4ed: Download complete 73c202020d66: Pull complete b330097dd4ed: Pull complete Digest: sha256:daded10ff4e08dfb6659c964e305f16679596312da558af095835202cf66f703 + echo done with openshift/origin-logging-curator done with openshift/origin-logging-curator + for image in openshift/base-centos7 centos:centos7 openshift/origin-logging-elasticsearch openshift/origin-logging-fluentd openshift/origin-logging-curator openshift/origin-logging-kibana + echo pulling image openshift/origin-logging-kibana ... pulling image openshift/origin-logging-kibana ... + vagrant ssh -c 'docker pull openshift/origin-logging-kibana' -- -n Using default tag: latest Trying to pull repository docker.io/openshift/origin-logging-kibana ... latest: Pulling from docker.io/openshift/origin-logging-kibana 45a2e645736c: Already exists 734fb161cf89: Already exists 78efc9e155c4: Already exists 8a3400b7e31a: Already exists 4a36f2160feb: Pulling fs layer c746963bb35a: Pulling fs layer 1a70ab5cdf21: Pulling fs layer 41752a3d77bd: Pulling fs layer a8fda808a3c1: Pulling fs layer 3b950177e658: Pulling fs layer 41752a3d77bd: Waiting 3b950177e658: Waiting a8fda808a3c1: Waiting 1a70ab5cdf21: Verifying Checksum 1a70ab5cdf21: Download complete 4a36f2160feb: Download complete a8fda808a3c1: Verifying Checksum a8fda808a3c1: Download complete 41752a3d77bd: Verifying Checksum 41752a3d77bd: Download complete 4a36f2160feb: Pull complete c746963bb35a: Download complete 3b950177e658: Verifying Checksum 3b950177e658: Download complete c746963bb35a: Pull complete 1a70ab5cdf21: Pull complete 41752a3d77bd: Pull complete a8fda808a3c1: Pull complete 3b950177e658: Pull complete Digest: sha256:950568237cc7d0ff14ea9fe22c3967d888996db70c66181421ad68caeb5ba75f + echo done with openshift/origin-logging-kibana done with openshift/origin-logging-kibana + vagrant test-origin-aggregated-logging -d --env GIT_URL=https://github.com/openshift/origin-aggregated-logging --env GIT_BRANCH=master --env O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging --env OS_ROOT=/data/src/github.com/openshift/origin --env ENABLE_OPS_CLUSTER=true --env USE_LOCAL_SOURCE=true --env TEST_PERF=false --env VERBOSE=1 --env OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible --env OS_ANSIBLE_BRANCH=master *************************************************** Running GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh... /data/src/github.com/openshift/origin /data/src/github.com/openshift/origin-aggregated-logging/hack/testing /data/src/github.com/openshift/origin-aggregated-logging/hack/testing /data/src/github.com/openshift/origin-aggregated-logging /data/src/github.com/openshift/origin-aggregated-logging/hack/testing /data/src/github.com/openshift/origin-aggregated-logging/hack/testing Loaded plugins: amazon-id, rhui-lb, search-disabled-repos Metadata Cache Created Loaded plugins: amazon-id, rhui-lb, search-disabled-repos Resolving Dependencies --> Running transaction check ---> Package ansible.noarch 0:2.3.0.0-3.el7 will be installed --> Processing Dependency: sshpass for package: ansible-2.3.0.0-3.el7.noarch --> Processing Dependency: python-paramiko for package: ansible-2.3.0.0-3.el7.noarch --> Processing Dependency: python-keyczar for package: ansible-2.3.0.0-3.el7.noarch --> Processing Dependency: python-httplib2 for package: ansible-2.3.0.0-3.el7.noarch --> Processing Dependency: python-crypto for package: ansible-2.3.0.0-3.el7.noarch ---> Package python2-pip.noarch 0:8.1.2-5.el7 will be installed ---> Package python2-ruamel-yaml.x86_64 0:0.12.14-9.el7 will be installed --> Processing Dependency: python2-typing for package: python2-ruamel-yaml-0.12.14-9.el7.x86_64 --> Processing Dependency: python2-ruamel-ordereddict for package: python2-ruamel-yaml-0.12.14-9.el7.x86_64 --> Running transaction check ---> Package python-httplib2.noarch 0:0.9.1-2.el7aos will be installed ---> Package python-keyczar.noarch 0:0.71c-2.el7aos will be installed --> Processing Dependency: python-pyasn1 for package: python-keyczar-0.71c-2.el7aos.noarch ---> Package python-paramiko.noarch 0:2.1.1-1.el7 will be installed --> Processing Dependency: python-cryptography for package: python-paramiko-2.1.1-1.el7.noarch ---> Package python2-crypto.x86_64 0:2.6.1-13.el7 will be installed --> Processing Dependency: libtomcrypt.so.0()(64bit) for package: python2-crypto-2.6.1-13.el7.x86_64 ---> Package python2-ruamel-ordereddict.x86_64 0:0.4.9-3.el7 will be installed ---> Package python2-typing.noarch 0:3.5.2.2-3.el7 will be installed ---> Package sshpass.x86_64 0:1.06-1.el7 will be installed --> Running transaction check ---> Package libtomcrypt.x86_64 0:1.17-23.el7 will be installed --> Processing Dependency: libtommath >= 0.42.0 for package: libtomcrypt-1.17-23.el7.x86_64 --> Processing Dependency: libtommath.so.0()(64bit) for package: libtomcrypt-1.17-23.el7.x86_64 ---> Package python2-cryptography.x86_64 0:1.3.1-3.el7 will be installed --> Processing Dependency: python-idna >= 2.0 for package: python2-cryptography-1.3.1-3.el7.x86_64 --> Processing Dependency: python-cffi >= 1.4.1 for package: python2-cryptography-1.3.1-3.el7.x86_64 --> Processing Dependency: python-ipaddress for package: python2-cryptography-1.3.1-3.el7.x86_64 --> Processing Dependency: python-enum34 for package: python2-cryptography-1.3.1-3.el7.x86_64 ---> Package python2-pyasn1.noarch 0:0.1.9-7.el7 will be installed --> Running transaction check ---> Package libtommath.x86_64 0:0.42.0-4.el7 will be installed ---> Package python-cffi.x86_64 0:1.6.0-5.el7 will be installed --> Processing Dependency: python-pycparser for package: python-cffi-1.6.0-5.el7.x86_64 ---> Package python-enum34.noarch 0:1.0.4-1.el7 will be installed ---> Package python-idna.noarch 0:2.0-1.el7 will be installed ---> Package python-ipaddress.noarch 0:1.0.16-2.el7 will be installed --> Running transaction check ---> Package python-pycparser.noarch 0:2.14-1.el7 will be installed --> Processing Dependency: python-ply for package: python-pycparser-2.14-1.el7.noarch --> Running transaction check ---> Package python-ply.noarch 0:3.4-10.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: ansible noarch 2.3.0.0-3.el7 epel 5.7 M python2-pip noarch 8.1.2-5.el7 epel 1.7 M python2-ruamel-yaml x86_64 0.12.14-9.el7 li 245 k Installing for dependencies: libtomcrypt x86_64 1.17-23.el7 epel 224 k libtommath x86_64 0.42.0-4.el7 epel 35 k python-cffi x86_64 1.6.0-5.el7 oso-rhui-rhel-server-releases 218 k python-enum34 noarch 1.0.4-1.el7 oso-rhui-rhel-server-releases 52 k python-httplib2 noarch 0.9.1-2.el7aos li 115 k python-idna noarch 2.0-1.el7 oso-rhui-rhel-server-releases 92 k python-ipaddress noarch 1.0.16-2.el7 oso-rhui-rhel-server-releases 34 k python-keyczar noarch 0.71c-2.el7aos rhel-7-server-ose-3.1-rpms 217 k python-paramiko noarch 2.1.1-1.el7 rhel-7-server-ose-3.4-rpms 266 k python-ply noarch 3.4-10.el7 oso-rhui-rhel-server-releases 123 k python-pycparser noarch 2.14-1.el7 oso-rhui-rhel-server-releases 105 k python2-crypto x86_64 2.6.1-13.el7 epel 476 k python2-cryptography x86_64 1.3.1-3.el7 oso-rhui-rhel-server-releases 471 k python2-pyasn1 noarch 0.1.9-7.el7 oso-rhui-rhel-server-releases 100 k python2-ruamel-ordereddict x86_64 0.4.9-3.el7 li 38 k python2-typing noarch 3.5.2.2-3.el7 epel 39 k sshpass x86_64 1.06-1.el7 epel 21 k Transaction Summary ================================================================================ Install 3 Packages (+17 Dependent packages) Total download size: 10 M Installed size: 47 M Downloading packages: -------------------------------------------------------------------------------- Total 5.4 MB/s | 10 MB 00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : python2-pyasn1-0.1.9-7.el7.noarch 1/20 Installing : sshpass-1.06-1.el7.x86_64 2/20 Installing : libtommath-0.42.0-4.el7.x86_64 3/20 Installing : libtomcrypt-1.17-23.el7.x86_64 4/20 Installing : python2-crypto-2.6.1-13.el7.x86_64 5/20 Installing : python-keyczar-0.71c-2.el7aos.noarch 6/20 Installing : python-enum34-1.0.4-1.el7.noarch 7/20 Installing : python-ply-3.4-10.el7.noarch 8/20 Installing : python-pycparser-2.14-1.el7.noarch 9/20 Installing : python-cffi-1.6.0-5.el7.x86_64 10/20 Installing : python-httplib2-0.9.1-2.el7aos.noarch 11/20 Installing : python-idna-2.0-1.el7.noarch 12/20 Installing : python2-ruamel-ordereddict-0.4.9-3.el7.x86_64 13/20 Installing : python2-typing-3.5.2.2-3.el7.noarch 14/20 Installing : python-ipaddress-1.0.16-2.el7.noarch 15/20 Installing : python2-cryptography-1.3.1-3.el7.x86_64 16/20 Installing : python-paramiko-2.1.1-1.el7.noarch 17/20 Installing : ansible-2.3.0.0-3.el7.noarch 18/20 Installing : python2-ruamel-yaml-0.12.14-9.el7.x86_64 19/20 Installing : python2-pip-8.1.2-5.el7.noarch 20/20 Verifying : python-pycparser-2.14-1.el7.noarch 1/20 Verifying : python-ipaddress-1.0.16-2.el7.noarch 2/20 Verifying : ansible-2.3.0.0-3.el7.noarch 3/20 Verifying : python2-typing-3.5.2.2-3.el7.noarch 4/20 Verifying : python2-pip-8.1.2-5.el7.noarch 5/20 Verifying : python2-pyasn1-0.1.9-7.el7.noarch 6/20 Verifying : libtomcrypt-1.17-23.el7.x86_64 7/20 Verifying : python-cffi-1.6.0-5.el7.x86_64 8/20 Verifying : python2-ruamel-yaml-0.12.14-9.el7.x86_64 9/20 Verifying : python2-ruamel-ordereddict-0.4.9-3.el7.x86_64 10/20 Verifying : python-idna-2.0-1.el7.noarch 11/20 Verifying : python-httplib2-0.9.1-2.el7aos.noarch 12/20 Verifying : python-ply-3.4-10.el7.noarch 13/20 Verifying : python-enum34-1.0.4-1.el7.noarch 14/20 Verifying : python-keyczar-0.71c-2.el7aos.noarch 15/20 Verifying : libtommath-0.42.0-4.el7.x86_64 16/20 Verifying : sshpass-1.06-1.el7.x86_64 17/20 Verifying : python2-cryptography-1.3.1-3.el7.x86_64 18/20 Verifying : python-paramiko-2.1.1-1.el7.noarch 19/20 Verifying : python2-crypto-2.6.1-13.el7.x86_64 20/20 Installed: ansible.noarch 0:2.3.0.0-3.el7 python2-pip.noarch 0:8.1.2-5.el7 python2-ruamel-yaml.x86_64 0:0.12.14-9.el7 Dependency Installed: libtomcrypt.x86_64 0:1.17-23.el7 libtommath.x86_64 0:0.42.0-4.el7 python-cffi.x86_64 0:1.6.0-5.el7 python-enum34.noarch 0:1.0.4-1.el7 python-httplib2.noarch 0:0.9.1-2.el7aos python-idna.noarch 0:2.0-1.el7 python-ipaddress.noarch 0:1.0.16-2.el7 python-keyczar.noarch 0:0.71c-2.el7aos python-paramiko.noarch 0:2.1.1-1.el7 python-ply.noarch 0:3.4-10.el7 python-pycparser.noarch 0:2.14-1.el7 python2-crypto.x86_64 0:2.6.1-13.el7 python2-cryptography.x86_64 0:1.3.1-3.el7 python2-pyasn1.noarch 0:0.1.9-7.el7 python2-ruamel-ordereddict.x86_64 0:0.4.9-3.el7 python2-typing.noarch 0:3.5.2.2-3.el7 sshpass.x86_64 0:1.06-1.el7 Complete! Cloning into '/tmp/tmp.yPQutins2A/openhift-ansible'... Copying oc from path to /usr/local/bin for use by openshift-ansible Copying oc from path to /usr/bin for use by openshift-ansible Copying oadm from path to /usr/local/bin for use by openshift-ansible Copying oadm from path to /usr/bin for use by openshift-ansible [INFO] Starting logging tests at Fri Jun 9 12:21:16 EDT 2017 Generated new key pair as /tmp/openshift/origin-aggregated-logging/openshift.local.config/master/serviceaccounts.public.key and /tmp/openshift/origin-aggregated-logging/openshift.local.config/master/serviceaccounts.private.key Generating node credentials ... Created node config for 172.18.12.108 in /tmp/openshift/origin-aggregated-logging/openshift.local.config/node-172.18.12.108 Wrote master config to: /tmp/openshift/origin-aggregated-logging/openshift.local.config/master/master-config.yaml Running hack/lib/start.sh:352: executing 'oc get --raw /healthz --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting any result and text 'ok'; re-trying every 0.25s until completion or 80.000s... SUCCESS after 31.182s: hack/lib/start.sh:352: executing 'oc get --raw /healthz --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting any result and text 'ok'; re-trying every 0.25s until completion or 80.000s Standard output from the command: ok Standard error from the command: The connection to the server 172.18.12.108:8443 was refused - did you specify the right host or port? ... repeated 65 times Error from server (Forbidden): User "system:admin" cannot "get" on "/healthz" ... repeated 6 times Running hack/lib/start.sh:353: executing 'oc get --raw https://172.18.12.108:10250/healthz --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting any result and text 'ok'; re-trying every 0.5s until completion or 120.000s... SUCCESS after 0.213s: hack/lib/start.sh:353: executing 'oc get --raw https://172.18.12.108:10250/healthz --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting any result and text 'ok'; re-trying every 0.5s until completion or 120.000s Standard output from the command: ok There was no error output from the command. Running hack/lib/start.sh:354: executing 'oc get --raw /healthz/ready --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting any result and text 'ok'; re-trying every 0.25s until completion or 80.000s... SUCCESS after 0.897s: hack/lib/start.sh:354: executing 'oc get --raw /healthz/ready --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting any result and text 'ok'; re-trying every 0.25s until completion or 80.000s Standard output from the command: ok Standard error from the command: Error from server (InternalError): an error on the server ("") has prevented the request from succeeding Running hack/lib/start.sh:355: executing 'oc get service kubernetes --namespace default --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting success; re-trying every 0.25s until completion or 160.000s... SUCCESS after 0.493s: hack/lib/start.sh:355: executing 'oc get service kubernetes --namespace default --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting success; re-trying every 0.25s until completion or 160.000s Standard output from the command: NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 172.30.0.1 <none> 443/TCP,53/UDP,53/TCP 5s There was no error output from the command. Running hack/lib/start.sh:356: executing 'oc get --raw /api/v1/nodes/172.18.12.108 --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting success; re-trying every 0.25s until completion or 80.000s... SUCCESS after 0.261s: hack/lib/start.sh:356: executing 'oc get --raw /api/v1/nodes/172.18.12.108 --config='/tmp/openshift/origin-aggregated-logging/openshift.local.config/master/admin.kubeconfig'' expecting success; re-trying every 0.25s until completion or 80.000s Standard output from the command: {"kind":"Node","apiVersion":"v1","metadata":{"name":"172.18.12.108","selfLink":"/api/v1/nodes/172.18.12.108","uid":"ca4dd811-4d2f-11e7-8061-0ed679898f5a","resourceVersion":"286","creationTimestamp":"2017-06-09T16:22:11Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"172.18.12.108"},"annotations":{"volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"externalID":"172.18.12.108","providerID":"aws:////i-08ba2905c8dea9d7e"},"status":{"capacity":{"cpu":"4","memory":"7231688Ki","pods":"40"},"allocatable":{"cpu":"4","memory":"7129288Ki","pods":"40"},"conditions":[{"type":"OutOfDisk","status":"False","lastHeartbeatTime":"2017-06-09T16:22:11Z","lastTransitionTime":"2017-06-09T16:22:11Z","reason":"KubeletHasSufficientDisk","message":"kubelet has sufficient disk space available"},{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2017-06-09T16:22:11Z","lastTransitionTime":"2017-06-09T16:22:11Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2017-06-09T16:22:11Z","lastTransitionTime":"2017-06-09T16:22:11Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"Ready","status":"True","lastHeartbeatTime":"2017-06-09T16:22:11Z","lastTransitionTime":"2017-06-09T16:22:11Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"LegacyHostIP","address":"172.18.12.108"},{"type":"InternalIP","address":"172.18.12.108"},{"type":"Hostname","address":"172.18.12.108"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"f9370ed252a14f73b014c1301a9b6d1b","systemUUID":"EC2E3595-6DF8-F87D-D6D8-1B4169ECFD13","bootID":"64a1913d-5a6f-485c-8e07-024fe2b25e0b","kernelVersion":"3.10.0-327.22.2.el7.x86_64","osImage":"Red Hat Enterprise Linux Server 7.3 (Maipo)","containerRuntimeVersion":"docker://1.12.6","kubeletVersion":"v1.6.1+5115d708d7","kubeProxyVersion":"v1.6.1+5115d708d7","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["openshift/origin-federation:6acabdc","openshift/origin-federation:latest"],"sizeBytes":1205885664},{"names":["openshift/origin-docker-registry:6acabdc","openshift/origin-docker-registry:latest"],"sizeBytes":1100164272},{"names":["openshift/origin-gitserver:6acabdc","openshift/origin-gitserver:latest"],"sizeBytes":1086520226},{"names":["openshift/openvswitch:6acabdc","openshift/openvswitch:latest"],"sizeBytes":1053403667},{"names":["openshift/node:6acabdc","openshift/node:latest"],"sizeBytes":1051721928},{"names":["openshift/origin-keepalived-ipfailover:6acabdc","openshift/origin-keepalived-ipfailover:latest"],"sizeBytes":1028529711},{"names":["openshift/origin-haproxy-router:6acabdc","openshift/origin-haproxy-router:latest"],"sizeBytes":1022758742},{"names":["openshift/origin:6acabdc","openshift/origin:latest"],"sizeBytes":1001728427},{"names":["openshift/origin-f5-router:6acabdc","openshift/origin-f5-router:latest"],"sizeBytes":1001728427},{"names":["openshift/origin-sti-builder:6acabdc","openshift/origin-sti-builder:latest"],"sizeBytes":1001728427},{"names":["openshift/origin-recycler:6acabdc","openshift/origin-recycler:latest"],"sizeBytes":1001728427},{"names":["openshift/origin-deployer:6acabdc","openshift/origin-deployer:latest"],"sizeBytes":1001728427},{"names":["openshift/origin-docker-builder:6acabdc","openshift/origin-docker-builder:latest"],"sizeBytes":1001728427},{"names":["openshift/origin-cluster-capacity:6acabdc","openshift/origin-cluster-capacity:latest"],"sizeBytes":962455026},{"names":["rhel7.1:latest"],"sizeBytes":765301508},{"names":["openshift/dind-master:latest"],"sizeBytes":731456758},{"names":["openshift/dind-node:latest"],"sizeBytes":731453034},{"names":["\u003cnone\u003e@\u003cnone\u003e","\u003cnone\u003e:\u003cnone\u003e"],"sizeBytes":709532011},{"names":["docker.io/openshift/origin-logging-kibana@sha256:950568237cc7d0ff14ea9fe22c3967d888996db70c66181421ad68caeb5ba75f","docker.io/openshift/origin-logging-kibana:latest"],"sizeBytes":682851513},{"names":["openshift/dind:latest"],"sizeBytes":640650210},{"names":["docker.io/openshift/origin-logging-elasticsearch@sha256:6296f1719676e970438cac4d912542b35ac786c14a15df892507007c4ecbe490","docker.io/openshift/origin-logging-elasticsearch:latest"],"sizeBytes":425567196},{"names":["docker.io/openshift/base-centos7@sha256:aea292a3bddba020cde0ee83e6a45807931eb607c164ec6a3674f67039d8cd7c","docker.io/openshift/base-centos7:latest"],"sizeBytes":383049978},{"names":["rhel7.2:latest"],"sizeBytes":377493597},{"names":["openshift/origin-egress-router:6acabdc","openshift/origin-egress-router:latest"],"sizeBytes":364745713},{"names":["openshift/origin-base:latest"],"sizeBytes":363070172},{"names":["\u003cnone\u003e@\u003cnone\u003e","\u003cnone\u003e:\u003cnone\u003e"],"sizeBytes":363024702},{"names":["docker.io/openshift/origin-logging-fluentd@sha256:cae7c21c9f111d4f5b481c14a65c597c67e715a8ffe3aee4c483100ee77296d7","docker.io/openshift/origin-logging-fluentd:latest"],"sizeBytes":359223728},{"names":["docker.io/fedora@sha256:69281ddd7b2600e5f2b17f1e12d7fba25207f459204fb2d15884f8432c479136","docker.io/fedora:25"],"sizeBytes":230864375},{"names":["docker.io/openshift/origin-logging-curator@sha256:daded10ff4e08dfb6659c964e305f16679596312da558af095835202cf66f703","docker.io/openshift/origin-logging-curator:latest"],"sizeBytes":224977669},{"names":["rhel7.3:latest","rhel7:latest"],"sizeBytes":219121266},{"names":["openshift/origin-pod:6acabdc","openshift/origin-pod:latest"],"sizeBytes":213199843},{"names":["registry.access.redhat.com/rhel7.2@sha256:98e6ca5d226c26e31a95cd67716afe22833c943e1926a21daf1a030906a02249","registry.access.redhat.com/rhel7.2:latest"],"sizeBytes":201376319},{"names":["registry.access.redhat.com/rhel7.3@sha256:1e232401d8e0ba53b36b757b4712fbcbd1dab9c21db039c45a84871a74e89e68","registry.access.redhat.com/rhel7.3:latest"],"sizeBytes":192693772},{"names":["docker.io/centos@sha256:bba1de7c9d900a898e3cadbae040dfe8a633c06bc104a0df76ae24483e03c077"],"sizeBytes":192548999},{"names":["openshift/origin-source:latest"],"sizeBytes":192548894},{"names":["docker.io/centos@sha256:aebf12af704307dfa0079b3babdca8d7e8ff6564696882bcb5d11f1d461f9ee9","docker.io/centos:7","docker.io/centos:centos7"],"sizeBytes":192548537},{"names":["registry.access.redhat.com/rhel7.1@sha256:1bc5a4c43bbb29a5a96a61896ff696933be3502e2f5fdc4cde02d9e101731fdd","registry.access.redhat.com/rhel7.1:latest"],"sizeBytes":158229901},{"names":["openshift/hello-openshift:6acabdc","openshift/hello-openshift:latest"],"sizeBytes":5643318}]}} There was no error output from the command. serviceaccount "registry" created clusterrolebinding "registry-registry-role" created deploymentconfig "docker-registry" created service "docker-registry" created info: password for stats user admin has been set to rnwCp6QgGh --> Creating router router ... serviceaccount "router" created clusterrolebinding "router-router-role" created deploymentconfig "router" created service "router" created --> Success Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:162: executing 'oadm new-project logging --node-selector=''' expecting success... SUCCESS after 0.941s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:162: executing 'oadm new-project logging --node-selector=''' expecting success Standard output from the command: Created project logging There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:163: executing 'oc project logging > /dev/null' expecting success... SUCCESS after 0.335s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:163: executing 'oc project logging > /dev/null' expecting success There was no output from the command. There was no error output from the command. apiVersion: v1 items: - apiVersion: v1 kind: ImageStream metadata: labels: build: logging-elasticsearch component: development logging-infra: development provider: openshift name: logging-elasticsearch spec: {} - apiVersion: v1 kind: ImageStream metadata: labels: build: logging-fluentd component: development logging-infra: development provider: openshift name: logging-fluentd spec: {} - apiVersion: v1 kind: ImageStream metadata: labels: build: logging-kibana component: development logging-infra: development provider: openshift name: logging-kibana spec: {} - apiVersion: v1 kind: ImageStream metadata: labels: build: logging-curator component: development logging-infra: development provider: openshift name: logging-curator spec: {} - apiVersion: v1 kind: ImageStream metadata: labels: build: logging-auth-proxy component: development logging-infra: development provider: openshift name: logging-auth-proxy spec: {} - apiVersion: v1 kind: ImageStream metadata: labels: build: logging-deployment component: development logging-infra: development provider: openshift name: origin spec: dockerImageRepository: openshift/origin tags: - from: kind: DockerImage name: openshift/origin:v1.5.0-alpha.2 name: v1.5.0-alpha.2 - apiVersion: v1 kind: BuildConfig metadata: labels: app: logging-elasticsearch component: development logging-infra: development provider: openshift name: logging-elasticsearch spec: output: to: kind: ImageStreamTag name: logging-elasticsearch:latest resources: {} source: contextDir: elasticsearch git: ref: master uri: https://github.com/openshift/origin-aggregated-logging type: Git strategy: dockerStrategy: from: kind: DockerImage name: openshift/base-centos7 type: Docker - apiVersion: v1 kind: BuildConfig metadata: labels: build: logging-fluentd component: development logging-infra: development provider: openshift name: logging-fluentd spec: output: to: kind: ImageStreamTag name: logging-fluentd:latest resources: {} source: contextDir: fluentd git: ref: master uri: https://github.com/openshift/origin-aggregated-logging type: Git strategy: dockerStrategy: from: kind: DockerImage name: openshift/base-centos7 type: Docker - apiVersion: v1 kind: BuildConfig metadata: labels: build: logging-kibana component: development logging-infra: development provider: openshift name: logging-kibana spec: output: to: kind: ImageStreamTag name: logging-kibana:latest resources: {} source: contextDir: kibana git: ref: master uri: https://github.com/openshift/origin-aggregated-logging type: Git strategy: dockerStrategy: from: kind: DockerImage name: openshift/base-centos7 type: Docker - apiVersion: v1 kind: BuildConfig metadata: labels: build: logging-curator component: development logging-infra: development provider: openshift name: logging-curator spec: output: to: kind: ImageStreamTag name: logging-curator:latest resources: {} source: contextDir: curator git: ref: master uri: https://github.com/openshift/origin-aggregated-logging type: Git strategy: dockerStrategy: from: kind: DockerImage name: openshift/base-centos7 type: Docker - apiVersion: v1 kind: BuildConfig metadata: labels: build: logging-auth-proxy component: development logging-infra: development provider: openshift name: logging-auth-proxy spec: output: to: kind: ImageStreamTag name: logging-auth-proxy:latest resources: {} source: contextDir: kibana-proxy git: ref: master uri: https://github.com/openshift/origin-aggregated-logging type: Git strategy: dockerStrategy: from: kind: DockerImage name: library/node:0.10.36 type: Docker kind: List metadata: {} Running hack/testing/build-images:31: executing 'oc process -o yaml -f /data/src/github.com/openshift/origin-aggregated-logging/hack/templates/dev-builds-wo-deployer.yaml -p LOGGING_FORK_URL=https://github.com/openshift/origin-aggregated-logging -p LOGGING_FORK_BRANCH=master | build_filter | oc create -f -' expecting success... SUCCESS after 0.405s: hack/testing/build-images:31: executing 'oc process -o yaml -f /data/src/github.com/openshift/origin-aggregated-logging/hack/templates/dev-builds-wo-deployer.yaml -p LOGGING_FORK_URL=https://github.com/openshift/origin-aggregated-logging -p LOGGING_FORK_BRANCH=master | build_filter | oc create -f -' expecting success Standard output from the command: imagestream "logging-elasticsearch" created imagestream "logging-fluentd" created imagestream "logging-kibana" created imagestream "logging-curator" created imagestream "logging-auth-proxy" created imagestream "origin" created buildconfig "logging-elasticsearch" created buildconfig "logging-fluentd" created buildconfig "logging-kibana" created buildconfig "logging-curator" created buildconfig "logging-auth-proxy" created There was no error output from the command. Running hack/testing/build-images:9: executing 'oc get imagestreamtag origin:latest' expecting success; re-trying every 0.2s until completion or 60.000s... SUCCESS after 0.628s: hack/testing/build-images:9: executing 'oc get imagestreamtag origin:latest' expecting success; re-trying every 0.2s until completion or 60.000s Standard output from the command: NAME DOCKER REF UPDATED IMAGENAME origin:latest openshift/origin@sha256:4019c4653c0ce067bc0d92b83a9e125c9526bae3c0f4645225b298b460c07a19 Less than a second ago sha256:4019c4653c0ce067bc0d92b83a9e125c9526bae3c0f4645225b298b460c07a19 Standard error from the command: Error from server (NotFound): imagestreamtags.image.openshift.io "origin:latest" not found Uploading directory "/data/src/github.com/openshift/origin-aggregated-logging" as binary input for the build ... build "logging-auth-proxy-1" started Uploading directory "/data/src/github.com/openshift/origin-aggregated-logging" as binary input for the build ... build "logging-curator-1" started Uploading directory "/data/src/github.com/openshift/origin-aggregated-logging" as binary input for the build ... build "logging-elasticsearch-1" started Uploading directory "/data/src/github.com/openshift/origin-aggregated-logging" as binary input for the build ... build "logging-fluentd-1" started Uploading directory "/data/src/github.com/openshift/origin-aggregated-logging" as binary input for the build ... build "logging-kibana-1" started Running hack/testing/build-images:33: executing 'wait_for_builds_complete' expecting success... SUCCESS after 60.644s: hack/testing/build-images:33: executing 'wait_for_builds_complete' expecting success Standard output from the command: Builds are complete There was no error output from the command. /tmp/tmp.yPQutins2A/openhift-ansible /data/src/github.com/openshift/origin-aggregated-logging ### Created host inventory file ### [oo_first_master] openshift [oo_first_master:vars] ansible_become=true ansible_connection=local containerized=true docker_protect_installed_version=true openshift_deployment_type=origin deployment_type=origin required_packages=[] openshift_hosted_logging_hostname=kibana.127.0.0.1.xip.io openshift_master_logging_public_url=https://kibana.127.0.0.1.xip.io openshift_logging_master_public_url=https://172.18.12.108:8443 openshift_logging_image_prefix=172.30.180.71:5000/logging/ openshift_logging_use_ops=true openshift_logging_fluentd_journal_read_from_head=False openshift_logging_es_log_appenders=['console'] openshift_logging_use_mux=false openshift_logging_mux_allow_external=false openshift_logging_use_mux_client=false ################################### Running hack/testing/init-log-stack:58: executing 'oc login -u system:admin' expecting success... SUCCESS after 0.220s: hack/testing/init-log-stack:58: executing 'oc login -u system:admin' expecting success Standard output from the command: Logged into "https://172.18.12.108:8443" as "system:admin" using existing credentials. You have access to the following projects and can switch between them with 'oc project <projectname>': default kube-public kube-system * logging openshift openshift-infra Using project "logging". There was no error output from the command. Using /tmp/tmp.yPQutins2A/openhift-ansible/ansible.cfg as config file PLAYBOOK: openshift-logging.yml ************************************************ 4 plays in /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/byo/openshift-cluster/openshift-logging.yml PLAY [Create initial host groups for localhost] ******************************** META: ran handlers TASK [include_vars] ************************************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/byo/openshift-cluster/initialize_groups.yml:10 ok: [localhost] => { "ansible_facts": { "g_all_hosts": "{{ g_master_hosts | union(g_node_hosts) | union(g_etcd_hosts) | union(g_lb_hosts) | union(g_nfs_hosts) | union(g_new_node_hosts)| union(g_new_master_hosts) | default([]) }}", "g_etcd_hosts": "{{ groups.etcd | default([]) }}", "g_glusterfs_hosts": "{{ groups.glusterfs | default([]) }}", "g_glusterfs_registry_hosts": "{{ groups.glusterfs_registry | default(g_glusterfs_hosts) }}", "g_lb_hosts": "{{ groups.lb | default([]) }}", "g_master_hosts": "{{ groups.masters | default([]) }}", "g_new_master_hosts": "{{ groups.new_masters | default([]) }}", "g_new_node_hosts": "{{ groups.new_nodes | default([]) }}", "g_nfs_hosts": "{{ groups.nfs | default([]) }}", "g_node_hosts": "{{ groups.nodes | default([]) }}" }, "changed": false } META: ran handlers META: ran handlers PLAY [Populate config host groups] ********************************************* META: ran handlers TASK [Evaluate groups - g_etcd_hosts required] ********************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:8 skipping: [localhost] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [Evaluate groups - g_master_hosts or g_new_master_hosts required] ********* task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:13 skipping: [localhost] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [Evaluate groups - g_node_hosts or g_new_node_hosts required] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:18 skipping: [localhost] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [Evaluate groups - g_lb_hosts required] *********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:23 skipping: [localhost] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [Evaluate groups - g_nfs_hosts required] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:28 skipping: [localhost] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [Evaluate groups - g_nfs_hosts is single host] **************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:33 skipping: [localhost] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [Evaluate groups - g_glusterfs_hosts required] **************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:38 skipping: [localhost] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [Evaluate oo_all_hosts] *************************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:43 TASK [Evaluate oo_masters] ***************************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:52 TASK [Evaluate oo_first_master] ************************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:61 skipping: [localhost] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [Evaluate oo_masters_to_config] ******************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:70 TASK [Evaluate oo_etcd_to_config] ********************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:79 TASK [Evaluate oo_first_etcd] ************************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:88 skipping: [localhost] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [Evaluate oo_etcd_hosts_to_upgrade] *************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:100 TASK [Evaluate oo_etcd_hosts_to_backup] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:107 creating host via 'add_host': hostname=openshift ok: [localhost] => (item=openshift) => { "add_host": { "groups": [ "oo_etcd_hosts_to_backup" ], "host_name": "openshift", "host_vars": {} }, "changed": false, "item": "openshift" } TASK [Evaluate oo_nodes_to_config] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:114 TASK [Add master to oo_nodes_to_config] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:124 TASK [Evaluate oo_lb_to_config] ************************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:134 TASK [Evaluate oo_nfs_to_config] *********************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:143 TASK [Evaluate oo_glusterfs_to_config] ***************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/playbooks/common/openshift-cluster/evaluate_groups.yml:152 META: ran handlers META: ran handlers PLAY [OpenShift Aggregated Logging] ******************************************** TASK [Gathering Facts] ********************************************************* ok: [openshift] META: ran handlers TASK [openshift_sanitize_inventory : Abort when conflicting deployment type variables are set] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_sanitize_inventory/tasks/main.yml:2 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_sanitize_inventory : Standardize on latest variable names] ***** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_sanitize_inventory/tasks/main.yml:15 ok: [openshift] => { "ansible_facts": { "deployment_type": "origin", "openshift_deployment_type": "origin" }, "changed": false } TASK [openshift_sanitize_inventory : Abort when deployment type is invalid] **** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_sanitize_inventory/tasks/main.yml:23 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_sanitize_inventory : Normalize openshift_release] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_sanitize_inventory/tasks/main.yml:31 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_sanitize_inventory : Abort when openshift_release is invalid] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_sanitize_inventory/tasks/main.yml:41 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_facts : Detecting Operating System] **************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:2 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_facts : set_fact] ********************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:8 ok: [openshift] => { "ansible_facts": { "l_is_atomic": false }, "changed": false } TASK [openshift_facts : set_fact] ********************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:10 ok: [openshift] => { "ansible_facts": { "l_is_containerized": true, "l_is_etcd_system_container": false, "l_is_master_system_container": false, "l_is_node_system_container": false, "l_is_openvswitch_system_container": false }, "changed": false } TASK [openshift_facts : set_fact] ********************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:16 ok: [openshift] => { "ansible_facts": { "l_any_system_container": false }, "changed": false } TASK [openshift_facts : set_fact] ********************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:18 ok: [openshift] => { "ansible_facts": { "l_etcd_runtime": "docker" }, "changed": false } TASK [openshift_facts : Validate python version] ******************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:22 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_facts : Validate python version] ******************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:29 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_facts : Determine Atomic Host Docker Version] ****************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:42 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_facts : assert] ************************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:46 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_facts : Load variables] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:53 ok: [openshift] => (item=/tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/vars/default.yml) => { "ansible_facts": { "required_packages": [ "iproute", "python-dbus", "PyYAML", "yum-utils" ] }, "item": "/tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/vars/default.yml" } TASK [openshift_facts : Ensure various deps are installed] ********************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:59 ok: [openshift] => (item=iproute) => { "changed": false, "item": "iproute", "rc": 0, "results": [ "iproute-3.10.0-74.el7.x86_64 providing iproute is already installed" ] } ok: [openshift] => (item=python-dbus) => { "changed": false, "item": "python-dbus", "rc": 0, "results": [ "dbus-python-1.1.1-9.el7.x86_64 providing python-dbus is already installed" ] } ok: [openshift] => (item=PyYAML) => { "changed": false, "item": "PyYAML", "rc": 0, "results": [ "PyYAML-3.10-11.el7.x86_64 providing PyYAML is already installed" ] } ok: [openshift] => (item=yum-utils) => { "changed": false, "item": "yum-utils", "rc": 0, "results": [ "yum-utils-1.1.31-40.el7.noarch providing yum-utils is already installed" ] } TASK [openshift_facts : Ensure various deps for running system containers are installed] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:64 skipping: [openshift] => (item=atomic) => { "changed": false, "item": "atomic", "skip_reason": "Conditional result was False", "skipped": true } skipping: [openshift] => (item=ostree) => { "changed": false, "item": "ostree", "skip_reason": "Conditional result was False", "skipped": true } skipping: [openshift] => (item=runc) => { "changed": false, "item": "runc", "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_facts : Gather Cluster facts and set is_containerized if needed] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:71 changed: [openshift] => { "ansible_facts": { "openshift": { "common": { "admin_binary": "/usr/local/bin/oadm", "all_hostnames": [ "172.18.12.108", "ec2-54-152-244-105.compute-1.amazonaws.com", "ip-172-18-12-108.ec2.internal", "54.152.244.105" ], "cli_image": "openshift/origin", "client_binary": "/usr/local/bin/oc", "cluster_id": "default", "config_base": "/etc/origin", "data_dir": "/var/lib/origin", "debug_level": "2", "deployer_image": "openshift/origin-deployer", "deployment_subtype": "basic", "deployment_type": "origin", "dns_domain": "cluster.local", "etcd_runtime": "docker", "examples_content_version": "v3.6", "generate_no_proxy_hosts": true, "hostname": "ip-172-18-12-108.ec2.internal", "install_examples": true, "internal_hostnames": [ "172.18.12.108", "ip-172-18-12-108.ec2.internal" ], "ip": "172.18.12.108", "is_atomic": false, "is_containerized": true, "is_etcd_system_container": false, "is_master_system_container": false, "is_node_system_container": false, "is_openvswitch_system_container": false, "kube_svc_ip": "172.30.0.1", "pod_image": "openshift/origin-pod", "portal_net": "172.30.0.0/16", "public_hostname": "ec2-54-152-244-105.compute-1.amazonaws.com", "public_ip": "54.152.244.105", "registry_image": "openshift/origin-docker-registry", "router_image": "openshift/origin-haproxy-router", "sdn_network_plugin_name": "redhat/openshift-ovs-subnet", "service_type": "origin", "use_calico": false, "use_contiv": false, "use_dnsmasq": true, "use_flannel": false, "use_manageiq": true, "use_nuage": false, "use_openshift_sdn": true, "version_gte_3_1_1_or_1_1_1": true, "version_gte_3_1_or_1_1": true, "version_gte_3_2_or_1_2": true, "version_gte_3_3_or_1_3": true, "version_gte_3_4_or_1_4": true, "version_gte_3_5_or_1_5": true, "version_gte_3_6": true }, "current_config": { "roles": [ "node", "docker" ] }, "docker": { "api_version": 1.24, "disable_push_dockerhub": false, "gte_1_10": true, "options": "--log-driver=journald", "service_name": "docker", "version": "1.12.6" }, "hosted": { "logging": { "selector": null }, "metrics": { "selector": null }, "registry": { "selector": "region=infra" }, "router": { "selector": "region=infra" } }, "node": { "annotations": {}, "iptables_sync_period": "30s", "kubelet_args": { "node-labels": [] }, "labels": {}, "local_quota_per_fsgroup": "", "node_image": "openshift/node", "node_system_image": "openshift/node", "nodename": "ip-172-18-12-108.ec2.internal", "ovs_image": "openshift/openvswitch", "ovs_system_image": "openshift/openvswitch", "registry_url": "openshift/origin-${component}:${version}", "schedulable": true, "sdn_mtu": "8951", "set_node_ip": false, "storage_plugin_deps": [ "ceph", "glusterfs", "iscsi" ] }, "provider": { "metadata": { "ami-id": "ami-83a1fc95", "ami-launch-index": "0", "ami-manifest-path": "(unknown)", "block-device-mapping": { "ami": "/dev/sda1", "ebs17": "sdb", "root": "/dev/sda1" }, "hostname": "ip-172-18-12-108.ec2.internal", "instance-action": "none", "instance-id": "i-08ba2905c8dea9d7e", "instance-type": "c4.xlarge", "local-hostname": "ip-172-18-12-108.ec2.internal", "local-ipv4": "172.18.12.108", "mac": "0e:d6:79:89:8f:5a", "metrics": { "vhostmd": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" }, "network": { "interfaces": { "macs": { "0e:d6:79:89:8f:5a": { "device-number": "0", "interface-id": "eni-6417f2b9", "ipv4-associations": { "54.152.244.105": "172.18.12.108" }, "local-hostname": "ip-172-18-12-108.ec2.internal", "local-ipv4s": "172.18.12.108", "mac": "0e:d6:79:89:8f:5a", "owner-id": "531415883065", "public-hostname": "ec2-54-152-244-105.compute-1.amazonaws.com", "public-ipv4s": "54.152.244.105", "security-group-ids": "sg-7e73221a", "security-groups": "default", "subnet-id": "subnet-cf57c596", "subnet-ipv4-cidr-block": "172.18.0.0/20", "vpc-id": "vpc-69705d0c", "vpc-ipv4-cidr-block": "172.18.0.0/16", "vpc-ipv4-cidr-blocks": "172.18.0.0/16" } } } }, "placement": { "availability-zone": "us-east-1d" }, "profile": "default-hvm", "public-hostname": "ec2-54-152-244-105.compute-1.amazonaws.com", "public-ipv4": "54.152.244.105", "public-keys/": "0=libra", "reservation-id": "r-0b42effe1b1ff7b19", "security-groups": "default", "services": { "domain": "amazonaws.com", "partition": "aws" } }, "name": "aws", "network": { "hostname": "ip-172-18-12-108.ec2.internal", "interfaces": [ { "ips": [ "172.18.12.108" ], "network_id": "subnet-cf57c596", "network_type": "vpc", "public_ips": [ "54.152.244.105" ] } ], "ip": "172.18.12.108", "ipv6_enabled": false, "public_hostname": "ec2-54-152-244-105.compute-1.amazonaws.com", "public_ip": "54.152.244.105" }, "zone": "us-east-1d" } } }, "changed": true } TASK [openshift_facts : Set repoquery command] ********************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_facts/tasks/main.yml:99 ok: [openshift] => { "ansible_facts": { "repoquery_cmd": "repoquery --plugins" }, "changed": false } TASK [openshift_logging : fail] ************************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/main.yaml:2 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Set default image variables based on deployment_type] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/main.yaml:6 ok: [openshift] => (item=/tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/vars/default_images.yml) => { "ansible_facts": { "__openshift_logging_image_prefix": "{{ openshift_hosted_logging_deployer_prefix | default('docker.io/openshift/origin-') }}", "__openshift_logging_image_version": "{{ openshift_hosted_logging_deployer_version | default('latest') }}" }, "item": "/tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/vars/default_images.yml" } TASK [openshift_logging : Set logging image facts] ***************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/main.yaml:12 ok: [openshift] => { "ansible_facts": { "openshift_logging_image_prefix": "172.30.180.71:5000/logging/", "openshift_logging_image_version": "latest" }, "changed": false } TASK [openshift_logging : Create temp directory for doing work in] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/main.yaml:17 ok: [openshift] => { "changed": false, "cmd": [ "mktemp", "-d", "/tmp/openshift-logging-ansible-XXXXXX" ], "delta": "0:00:00.002129", "end": "2017-06-09 12:35:52.687883", "rc": 0, "start": "2017-06-09 12:35:52.685754" } STDOUT: /tmp/openshift-logging-ansible-uqrVXU TASK [openshift_logging : debug] *********************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/main.yaml:24 ok: [openshift] => { "changed": false } MSG: Created temp dir /tmp/openshift-logging-ansible-uqrVXU TASK [openshift_logging : Create local temp directory for doing work in] ******* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/main.yaml:26 ok: [openshift -> 127.0.0.1] => { "changed": false, "cmd": [ "mktemp", "-d", "/tmp/openshift-logging-ansible-XXXXXX" ], "delta": "0:00:00.002055", "end": "2017-06-09 12:35:52.844773", "rc": 0, "start": "2017-06-09 12:35:52.842718" } STDOUT: /tmp/openshift-logging-ansible-YJJ6SM TASK [openshift_logging : include] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/main.yaml:33 included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml for openshift TASK [openshift_logging : Gather OpenShift Logging Facts] ********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:2 ok: [openshift] => { "ansible_facts": { "openshift_logging_facts": { "curator": { "clusterrolebindings": {}, "configmaps": {}, "daemonsets": {}, "deploymentconfigs": {}, "oauthclients": {}, "pvcs": {}, "rolebindings": {}, "routes": {}, "sccs": {}, "secrets": {}, "services": {} }, "curator_ops": { "clusterrolebindings": {}, "configmaps": {}, "daemonsets": {}, "deploymentconfigs": {}, "oauthclients": {}, "pvcs": {}, "rolebindings": {}, "routes": {}, "sccs": {}, "secrets": {}, "services": {} }, "elasticsearch": { "clusterrolebindings": {}, "configmaps": {}, "daemonsets": {}, "deploymentconfigs": {}, "oauthclients": {}, "pvcs": {}, "rolebindings": {}, "routes": {}, "sccs": {}, "secrets": {}, "services": {} }, "elasticsearch_ops": { "clusterrolebindings": {}, "configmaps": {}, "daemonsets": {}, "deploymentconfigs": {}, "oauthclients": {}, "pvcs": {}, "rolebindings": {}, "routes": {}, "sccs": {}, "secrets": {}, "services": {} }, "fluentd": { "clusterrolebindings": {}, "configmaps": {}, "daemonsets": {}, "deploymentconfigs": {}, "oauthclients": {}, "pvcs": {}, "rolebindings": {}, "routes": {}, "sccs": {}, "secrets": {}, "services": {} }, "kibana": { "clusterrolebindings": {}, "configmaps": {}, "daemonsets": {}, "deploymentconfigs": {}, "oauthclients": {}, "pvcs": {}, "rolebindings": {}, "routes": {}, "sccs": {}, "secrets": {}, "services": {} }, "kibana_ops": { "clusterrolebindings": {}, "configmaps": {}, "daemonsets": {}, "deploymentconfigs": {}, "oauthclients": {}, "pvcs": {}, "rolebindings": {}, "routes": {}, "sccs": {}, "secrets": {}, "services": {} } } }, "changed": false } TASK [openshift_logging : Set logging project] ********************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:7 ok: [openshift] => { "changed": false, "results": { "cmd": "/bin/oc get namespace logging -o json", "results": { "apiVersion": "v1", "kind": "Namespace", "metadata": { "annotations": { "openshift.io/description": "", "openshift.io/display-name": "", "openshift.io/node-selector": "", "openshift.io/sa.scc.mcs": "s0:c7,c4", "openshift.io/sa.scc.supplemental-groups": "1000050000/10000", "openshift.io/sa.scc.uid-range": "1000050000/10000" }, "creationTimestamp": "2017-06-09T16:22:13Z", "name": "logging", "resourceVersion": "609", "selfLink": "/api/v1/namespaces/logging", "uid": "cb9bf79f-4d2f-11e7-8061-0ed679898f5a" }, "spec": { "finalizers": [ "openshift.io/origin", "kubernetes" ] }, "status": { "phase": "Active" } }, "returncode": 0 }, "state": "present" } TASK [openshift_logging : Labeling logging project] **************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:13 TASK [openshift_logging : Labeling logging project] **************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:26 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Create logging cert directory] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:39 ok: [openshift] => { "changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/origin/logging", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 6, "state": "directory", "uid": 0 } TASK [openshift_logging : include] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:47 included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml for openshift TASK [openshift_logging : Checking for ca.key] ********************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:3 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for ca.crt] ********************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:8 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for ca.serial.txt] ************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:13 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Generate certificates] ******************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:18 changed: [openshift] => { "changed": true, "cmd": [ "/usr/local/bin/oc", "adm", "--config=/tmp/openshift-logging-ansible-uqrVXU/admin.kubeconfig", "ca", "create-signer-cert", "--key=/etc/origin/logging/ca.key", "--cert=/etc/origin/logging/ca.crt", "--serial=/etc/origin/logging/ca.serial.txt", "--name=logging-signer-test" ], "delta": "0:00:00.554159", "end": "2017-06-09 12:35:57.454349", "rc": 0, "start": "2017-06-09 12:35:56.900190" } TASK [openshift_logging : Checking for signing.conf] *************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:29 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : template] ******************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:34 changed: [openshift] => { "changed": true, "checksum": "a5a1bda430be44f982fa9097778b7d35d2e42780", "dest": "/etc/origin/logging/signing.conf", "gid": 0, "group": "root", "md5sum": "449087446670073f2899aac33113350c", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 4263, "src": "/root/.ansible/tmp/ansible-tmp-1497026157.62-189747179592030/source", "state": "file", "uid": 0 } TASK [openshift_logging : include] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:39 included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml for openshift included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml for openshift included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml for openshift TASK [openshift_logging : Checking for kibana.crt] ***************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:2 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for kibana.key] ***************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:7 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Trying to discover server cert variable name for kibana] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Trying to discover the server key variable name for kibana] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:20 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Creating signed server cert and key for kibana] ****** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:28 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Copying server key for kibana to generated certs directory] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:40 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Copying Server cert for kibana to generated certs directory] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:50 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Checking for kibana-ops.crt] ************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:2 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for kibana-ops.key] ************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:7 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Trying to discover server cert variable name for kibana-ops] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Trying to discover the server key variable name for kibana-ops] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:20 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Creating signed server cert and key for kibana-ops] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:28 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Copying server key for kibana-ops to generated certs directory] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:40 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Copying Server cert for kibana-ops to generated certs directory] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:50 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Checking for kibana-internal.crt] ******************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:2 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for kibana-internal.key] ******************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:7 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Trying to discover server cert variable name for kibana-internal] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Trying to discover the server key variable name for kibana-internal] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:20 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Creating signed server cert and key for kibana-internal] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:28 changed: [openshift] => { "changed": true, "cmd": [ "/usr/local/bin/oc", "adm", "--config=/tmp/openshift-logging-ansible-uqrVXU/admin.kubeconfig", "ca", "create-server-cert", "--key=/etc/origin/logging/kibana-internal.key", "--cert=/etc/origin/logging/kibana-internal.crt", "--hostnames=kibana, kibana-ops, kibana.127.0.0.1.xip.io, kibana-ops.router.default.svc.cluster.local", "--signer-cert=/etc/origin/logging/ca.crt", "--signer-key=/etc/origin/logging/ca.key", "--signer-serial=/etc/origin/logging/ca.serial.txt" ], "delta": "0:00:00.479385", "end": "2017-06-09 12:35:59.819353", "rc": 0, "start": "2017-06-09 12:35:59.339968" } TASK [openshift_logging : Copying server key for kibana-internal to generated certs directory] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:40 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Copying Server cert for kibana-internal to generated certs directory] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/procure_server_certs.yaml:50 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : include] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:48 skipping: [openshift] => (item={u'procure_component': u'mux', u'hostnames': u'logging-mux, mux.router.default.svc.cluster.local'}) => { "cert_info": { "hostnames": "logging-mux, mux.router.default.svc.cluster.local", "procure_component": "mux" }, "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : include] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:56 skipping: [openshift] => (item={u'procure_component': u'mux'}) => { "changed": false, "shared_key_info": { "procure_component": "mux" }, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : include] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:63 skipping: [openshift] => (item={u'procure_component': u'es', u'hostnames': u'es, es.router.default.svc.cluster.local'}) => { "cert_info": { "hostnames": "es, es.router.default.svc.cluster.local", "procure_component": "es" }, "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : include] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:71 skipping: [openshift] => (item={u'procure_component': u'es-ops', u'hostnames': u'es-ops, es-ops.router.default.svc.cluster.local'}) => { "cert_info": { "hostnames": "es-ops, es-ops.router.default.svc.cluster.local", "procure_component": "es-ops" }, "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Copy proxy TLS configuration file] ******************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:81 changed: [openshift] => { "changed": true, "checksum": "36991681e03970736a99be9f084773521c44db06", "dest": "/etc/origin/logging/server-tls.json", "gid": 0, "group": "root", "md5sum": "2a954195add2b2fdde4ed09ff5c8e1c5", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 321, "src": "/root/.ansible/tmp/ansible-tmp-1497026160.28-122634310982295/source", "state": "file", "uid": 0 } TASK [openshift_logging : Copy proxy TLS configuration file] ******************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:86 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Checking for ca.db] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:91 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : copy] ************************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:96 changed: [openshift] => { "changed": true, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "dest": "/etc/origin/logging/ca.db", "gid": 0, "group": "root", "md5sum": "d41d8cd98f00b204e9800998ecf8427e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 0, "src": "/root/.ansible/tmp/ansible-tmp-1497026160.65-278940876479359/source", "state": "file", "uid": 0 } TASK [openshift_logging : Checking for ca.crt.srl] ***************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:101 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : copy] ************************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:106 changed: [openshift] => { "changed": true, "checksum": "da39a3ee5e6b4b0d3255bfef95601890afd80709", "dest": "/etc/origin/logging/ca.crt.srl", "gid": 0, "group": "root", "md5sum": "d41d8cd98f00b204e9800998ecf8427e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 0, "src": "/root/.ansible/tmp/ansible-tmp-1497026161.0-99884603306756/source", "state": "file", "uid": 0 } TASK [openshift_logging : Generate PEM certs] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:111 included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml for openshift included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml for openshift included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml for openshift included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml for openshift TASK [openshift_logging : Checking for system.logging.fluentd.key] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:2 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for system.logging.fluentd.crt] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:7 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Creating cert req for system.logging.fluentd] ******** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Creating cert req for system.logging.fluentd] ******** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:22 changed: [openshift] => { "changed": true, "cmd": [ "openssl", "req", "-out", "/etc/origin/logging/system.logging.fluentd.csr", "-new", "-newkey", "rsa:2048", "-keyout", "/etc/origin/logging/system.logging.fluentd.key", "-subj", "/CN=system.logging.fluentd/OU=OpenShift/O=Logging", "-days", "712", "-nodes" ], "delta": "0:00:00.073215", "end": "2017-06-09 12:36:01.871752", "rc": 0, "start": "2017-06-09 12:36:01.798537" } STDERR: Generating a 2048 bit RSA private key ..................................+++ ...........+++ writing new private key to '/etc/origin/logging/system.logging.fluentd.key' ----- TASK [openshift_logging : Sign cert request with CA for system.logging.fluentd] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:31 changed: [openshift] => { "changed": true, "cmd": [ "openssl", "ca", "-in", "/etc/origin/logging/system.logging.fluentd.csr", "-notext", "-out", "/etc/origin/logging/system.logging.fluentd.crt", "-config", "/etc/origin/logging/signing.conf", "-extensions", "v3_req", "-batch", "-extensions", "server_ext" ], "delta": "0:00:00.007932", "end": "2017-06-09 12:36:02.003331", "rc": 0, "start": "2017-06-09 12:36:01.995399" } STDERR: Using configuration from /etc/origin/logging/signing.conf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 2 (0x2) Validity Not Before: Jun 9 16:36:01 2017 GMT Not After : Jun 9 16:36:01 2019 GMT Subject: organizationName = Logging organizationalUnitName = OpenShift commonName = system.logging.fluentd X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: CA:FALSE X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Key Identifier: 99:1E:83:FA:34:40:1A:AB:57:E4:71:C8:66:9B:29:B8:3D:00:C8:BB X509v3 Authority Key Identifier: 0. Certificate is to be certified until Jun 9 16:36:01 2019 GMT (730 days) Write out database with 1 new entries Data Base Updated TASK [openshift_logging : Checking for system.logging.kibana.key] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:2 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for system.logging.kibana.crt] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:7 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Creating cert req for system.logging.kibana] ********* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Creating cert req for system.logging.kibana] ********* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:22 changed: [openshift] => { "changed": true, "cmd": [ "openssl", "req", "-out", "/etc/origin/logging/system.logging.kibana.csr", "-new", "-newkey", "rsa:2048", "-keyout", "/etc/origin/logging/system.logging.kibana.key", "-subj", "/CN=system.logging.kibana/OU=OpenShift/O=Logging", "-days", "712", "-nodes" ], "delta": "0:00:00.128854", "end": "2017-06-09 12:36:02.527864", "rc": 0, "start": "2017-06-09 12:36:02.399010" } STDERR: Generating a 2048 bit RSA private key ..............................+++ ...........................................................+++ writing new private key to '/etc/origin/logging/system.logging.kibana.key' ----- TASK [openshift_logging : Sign cert request with CA for system.logging.kibana] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:31 changed: [openshift] => { "changed": true, "cmd": [ "openssl", "ca", "-in", "/etc/origin/logging/system.logging.kibana.csr", "-notext", "-out", "/etc/origin/logging/system.logging.kibana.crt", "-config", "/etc/origin/logging/signing.conf", "-extensions", "v3_req", "-batch", "-extensions", "server_ext" ], "delta": "0:00:00.008185", "end": "2017-06-09 12:36:02.659354", "rc": 0, "start": "2017-06-09 12:36:02.651169" } STDERR: Using configuration from /etc/origin/logging/signing.conf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 3 (0x3) Validity Not Before: Jun 9 16:36:02 2017 GMT Not After : Jun 9 16:36:02 2019 GMT Subject: organizationName = Logging organizationalUnitName = OpenShift commonName = system.logging.kibana X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: CA:FALSE X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Key Identifier: 3B:6C:AC:89:4E:D2:9F:77:24:66:DA:1A:2C:95:35:63:12:2B:34:1D X509v3 Authority Key Identifier: 0. Certificate is to be certified until Jun 9 16:36:02 2019 GMT (730 days) Write out database with 1 new entries Data Base Updated TASK [openshift_logging : Checking for system.logging.curator.key] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:2 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for system.logging.curator.crt] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:7 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Creating cert req for system.logging.curator] ******** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Creating cert req for system.logging.curator] ******** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:22 changed: [openshift] => { "changed": true, "cmd": [ "openssl", "req", "-out", "/etc/origin/logging/system.logging.curator.csr", "-new", "-newkey", "rsa:2048", "-keyout", "/etc/origin/logging/system.logging.curator.key", "-subj", "/CN=system.logging.curator/OU=OpenShift/O=Logging", "-days", "712", "-nodes" ], "delta": "0:00:00.136444", "end": "2017-06-09 12:36:03.199402", "rc": 0, "start": "2017-06-09 12:36:03.062958" } STDERR: Generating a 2048 bit RSA private key ..............................+++ .................................................................+++ writing new private key to '/etc/origin/logging/system.logging.curator.key' ----- TASK [openshift_logging : Sign cert request with CA for system.logging.curator] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:31 changed: [openshift] => { "changed": true, "cmd": [ "openssl", "ca", "-in", "/etc/origin/logging/system.logging.curator.csr", "-notext", "-out", "/etc/origin/logging/system.logging.curator.crt", "-config", "/etc/origin/logging/signing.conf", "-extensions", "v3_req", "-batch", "-extensions", "server_ext" ], "delta": "0:00:00.007946", "end": "2017-06-09 12:36:03.331861", "rc": 0, "start": "2017-06-09 12:36:03.323915" } STDERR: Using configuration from /etc/origin/logging/signing.conf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 4 (0x4) Validity Not Before: Jun 9 16:36:03 2017 GMT Not After : Jun 9 16:36:03 2019 GMT Subject: organizationName = Logging organizationalUnitName = OpenShift commonName = system.logging.curator X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: CA:FALSE X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Key Identifier: 25:AA:E3:54:07:89:AD:24:B1:36:67:79:76:43:2E:84:32:76:E9:69 X509v3 Authority Key Identifier: 0. Certificate is to be certified until Jun 9 16:36:03 2019 GMT (730 days) Write out database with 1 new entries Data Base Updated TASK [openshift_logging : Checking for system.admin.key] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:2 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for system.admin.crt] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:7 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Creating cert req for system.admin] ****************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Creating cert req for system.admin] ****************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:22 changed: [openshift] => { "changed": true, "cmd": [ "openssl", "req", "-out", "/etc/origin/logging/system.admin.csr", "-new", "-newkey", "rsa:2048", "-keyout", "/etc/origin/logging/system.admin.key", "-subj", "/CN=system.admin/OU=OpenShift/O=Logging", "-days", "712", "-nodes" ], "delta": "0:00:00.106658", "end": "2017-06-09 12:36:03.828190", "rc": 0, "start": "2017-06-09 12:36:03.721532" } STDERR: Generating a 2048 bit RSA private key ..................+++ .......................................................+++ writing new private key to '/etc/origin/logging/system.admin.key' ----- TASK [openshift_logging : Sign cert request with CA for system.admin] ********** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_pems.yaml:31 changed: [openshift] => { "changed": true, "cmd": [ "openssl", "ca", "-in", "/etc/origin/logging/system.admin.csr", "-notext", "-out", "/etc/origin/logging/system.admin.crt", "-config", "/etc/origin/logging/signing.conf", "-extensions", "v3_req", "-batch", "-extensions", "server_ext" ], "delta": "0:00:00.008087", "end": "2017-06-09 12:36:03.961627", "rc": 0, "start": "2017-06-09 12:36:03.953540" } STDERR: Using configuration from /etc/origin/logging/signing.conf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 5 (0x5) Validity Not Before: Jun 9 16:36:03 2017 GMT Not After : Jun 9 16:36:03 2019 GMT Subject: organizationName = Logging organizationalUnitName = OpenShift commonName = system.admin X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: CA:FALSE X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Key Identifier: BE:94:89:E9:00:B6:46:D2:E5:F2:BA:8E:BB:1E:28:7B:3D:F1:E9:CE X509v3 Authority Key Identifier: 0. Certificate is to be certified until Jun 9 16:36:03 2019 GMT (730 days) Write out database with 1 new entries Data Base Updated TASK [openshift_logging : Generate PEM cert for mux] *************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:121 skipping: [openshift] => (item=system.logging.mux) => { "changed": false, "node_name": "system.logging.mux", "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Generate PEM cert for Elasticsearch external route] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:129 skipping: [openshift] => (item=system.logging.es) => { "changed": false, "node_name": "system.logging.es", "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Creating necessary JKS certs] ************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:137 included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml for openshift TASK [openshift_logging : Checking for elasticsearch.jks] ********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:3 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for logging-es.jks] ************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:8 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for system.admin.jks] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:13 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Checking for truststore.jks] ************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:18 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging : Create placeholder for previously created JKS certs to prevent recreating...] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:23 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Create placeholder for previously created JKS certs to prevent recreating...] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:28 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Create placeholder for previously created JKS certs to prevent recreating...] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:33 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Create placeholder for previously created JKS certs to prevent recreating...] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:38 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : pulling down signing items from host] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:43 changed: [openshift] => (item=ca.crt) => { "changed": true, "checksum": "888ace2a8904586885ff7bed41bbdbbc87d1c679", "dest": "/tmp/openshift-logging-ansible-YJJ6SM/ca.crt", "item": "ca.crt", "md5sum": "b1f646f3ce19fb0a86eb597e9ed5aa6f", "remote_checksum": "888ace2a8904586885ff7bed41bbdbbc87d1c679", "remote_md5sum": null } changed: [openshift] => (item=ca.key) => { "changed": true, "checksum": "6c5d8b0ee1876fb57453e8c0febc0176ec2914cd", "dest": "/tmp/openshift-logging-ansible-YJJ6SM/ca.key", "item": "ca.key", "md5sum": "9c3e5537101257f52e1506e153d68337", "remote_checksum": "6c5d8b0ee1876fb57453e8c0febc0176ec2914cd", "remote_md5sum": null } changed: [openshift] => (item=ca.serial.txt) => { "changed": true, "checksum": "b649682b92a811746098e5c91e891e5142a41950", "dest": "/tmp/openshift-logging-ansible-YJJ6SM/ca.serial.txt", "item": "ca.serial.txt", "md5sum": "76b01ce73ac53fdac1c67d27ac040473", "remote_checksum": "b649682b92a811746098e5c91e891e5142a41950", "remote_md5sum": null } ok: [openshift] => (item=ca.crl.srl) => { "changed": false, "file": "/etc/origin/logging/ca.crl.srl", "item": "ca.crl.srl" } MSG: the remote file does not exist, not transferring, ignored changed: [openshift] => (item=ca.db) => { "changed": true, "checksum": "57fc15c09a71115076d8b534eec4b15dbcf53524", "dest": "/tmp/openshift-logging-ansible-YJJ6SM/ca.db", "item": "ca.db", "md5sum": "851c9a95b6fddd2127099580e7fc59dc", "remote_checksum": "57fc15c09a71115076d8b534eec4b15dbcf53524", "remote_md5sum": null } TASK [openshift_logging : template] ******************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:56 changed: [openshift -> 127.0.0.1] => { "changed": true, "checksum": "efbc44e10e419a94c50750a513b5f70d3fb06daa", "dest": "/tmp/openshift-logging-ansible-YJJ6SM/signing.conf", "gid": 0, "group": "root", "md5sum": "2212608ffe863b74ef500a3a1818a5da", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 4281, "src": "/root/.ansible/tmp/ansible-tmp-1497026165.47-133736239980327/source", "state": "file", "uid": 0 } TASK [openshift_logging : Run JKS generation script] *************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:61 changed: [openshift -> 127.0.0.1] => { "changed": true, "rc": 0 } STDOUT: Generating keystore and certificate for node system.admin Generating certificate signing request for node system.admin Sign certificate request with CA Import back to keystore (including CA chain) All done for system.admin Generating keystore and certificate for node elasticsearch Generating certificate signing request for node elasticsearch Sign certificate request with CA Import back to keystore (including CA chain) All done for elasticsearch Generating keystore and certificate for node logging-es Generating certificate signing request for node logging-es Sign certificate request with CA Import back to keystore (including CA chain) All done for logging-es Import CA to truststore for validating client certs STDERR: + '[' 2 -lt 1 ']' + dir=/tmp/openshift-logging-ansible-YJJ6SM + SCRATCH_DIR=/tmp/openshift-logging-ansible-YJJ6SM + PROJECT=logging + [[ ! -f /tmp/openshift-logging-ansible-YJJ6SM/system.admin.jks ]] + generate_JKS_client_cert system.admin + NODE_NAME=system.admin + ks_pass=kspass + ts_pass=tspass + dir=/tmp/openshift-logging-ansible-YJJ6SM + echo Generating keystore and certificate for node system.admin + keytool -genkey -alias system.admin -keystore /tmp/openshift-logging-ansible-YJJ6SM/system.admin.jks -keyalg RSA -keysize 2048 -validity 712 -keypass kspass -storepass kspass -dname 'CN=system.admin, OU=OpenShift, O=Logging' + echo Generating certificate signing request for node system.admin + keytool -certreq -alias system.admin -keystore /tmp/openshift-logging-ansible-YJJ6SM/system.admin.jks -file /tmp/openshift-logging-ansible-YJJ6SM/system.admin.jks.csr -keyalg rsa -keypass kspass -storepass kspass -dname 'CN=system.admin, OU=OpenShift, O=Logging' + echo Sign certificate request with CA + openssl ca -in /tmp/openshift-logging-ansible-YJJ6SM/system.admin.jks.csr -notext -out /tmp/openshift-logging-ansible-YJJ6SM/system.admin.jks.crt -config /tmp/openshift-logging-ansible-YJJ6SM/signing.conf -extensions v3_req -batch -extensions server_ext Using configuration from /tmp/openshift-logging-ansible-YJJ6SM/signing.conf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 6 (0x6) Validity Not Before: Jun 9 16:36:16 2017 GMT Not After : Jun 9 16:36:16 2019 GMT Subject: organizationName = Logging organizationalUnitName = OpenShift commonName = system.admin X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: CA:FALSE X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Key Identifier: 89:85:AC:32:BE:B2:0C:07:CC:3C:47:60:D1:59:F6:AF:42:60:71:A5 X509v3 Authority Key Identifier: 0. Certificate is to be certified until Jun 9 16:36:16 2019 GMT (730 days) Write out database with 1 new entries Data Base Updated + echo 'Import back to keystore (including CA chain)' + keytool -import -file /tmp/openshift-logging-ansible-YJJ6SM/ca.crt -keystore /tmp/openshift-logging-ansible-YJJ6SM/system.admin.jks -storepass kspass -noprompt -alias sig-ca Certificate was added to keystore + keytool -import -file /tmp/openshift-logging-ansible-YJJ6SM/system.admin.jks.crt -keystore /tmp/openshift-logging-ansible-YJJ6SM/system.admin.jks -storepass kspass -noprompt -alias system.admin Certificate reply was installed in keystore + echo All done for system.admin + [[ ! -f /tmp/openshift-logging-ansible-YJJ6SM/elasticsearch.jks ]] ++ join , logging-es logging-es-ops ++ local IFS=, ++ shift ++ echo logging-es,logging-es-ops + generate_JKS_chain true elasticsearch logging-es,logging-es-ops + dir=/tmp/openshift-logging-ansible-YJJ6SM + ADD_OID=true + NODE_NAME=elasticsearch + CERT_NAMES=logging-es,logging-es-ops + ks_pass=kspass + ts_pass=tspass + rm -rf elasticsearch + extension_names= + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es,dns:logging-es-ops + '[' true = true ']' + extension_names=,dns:logging-es,dns:logging-es-ops,oid:1.2.3.4.5.5 + echo Generating keystore and certificate for node elasticsearch + keytool -genkey -alias elasticsearch -keystore /tmp/openshift-logging-ansible-YJJ6SM/elasticsearch.jks -keypass kspass -storepass kspass -keyalg RSA -keysize 2048 -validity 712 -dname 'CN=elasticsearch, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es-ops,oid:1.2.3.4.5.5 + echo Generating certificate signing request for node elasticsearch + keytool -certreq -alias elasticsearch -keystore /tmp/openshift-logging-ansible-YJJ6SM/elasticsearch.jks -storepass kspass -file /tmp/openshift-logging-ansible-YJJ6SM/elasticsearch.csr -keyalg rsa -dname 'CN=elasticsearch, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es-ops,oid:1.2.3.4.5.5 + echo Sign certificate request with CA + openssl ca -in /tmp/openshift-logging-ansible-YJJ6SM/elasticsearch.csr -notext -out /tmp/openshift-logging-ansible-YJJ6SM/elasticsearch.crt -config /tmp/openshift-logging-ansible-YJJ6SM/signing.conf -extensions v3_req -batch -extensions server_ext Using configuration from /tmp/openshift-logging-ansible-YJJ6SM/signing.conf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 7 (0x7) Validity Not Before: Jun 9 16:36:17 2017 GMT Not After : Jun 9 16:36:17 2019 GMT Subject: organizationName = Logging organizationalUnitName = OpenShift commonName = elasticsearch X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: CA:FALSE X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Key Identifier: AE:7E:04:F8:D4:8D:DC:53:08:7F:B0:A9:31:4F:EA:35:B6:F3:76:0E X509v3 Authority Key Identifier: 0. X509v3 Subject Alternative Name: DNS:localhost, IP Address:127.0.0.1, DNS:logging-es, DNS:logging-es-ops, Registered ID:1.2.3.4.5.5 Certificate is to be certified until Jun 9 16:36:17 2019 GMT (730 days) Write out database with 1 new entries Data Base Updated + echo 'Import back to keystore (including CA chain)' + keytool -import -file /tmp/openshift-logging-ansible-YJJ6SM/ca.crt -keystore /tmp/openshift-logging-ansible-YJJ6SM/elasticsearch.jks -storepass kspass -noprompt -alias sig-ca Certificate was added to keystore + keytool -import -file /tmp/openshift-logging-ansible-YJJ6SM/elasticsearch.crt -keystore /tmp/openshift-logging-ansible-YJJ6SM/elasticsearch.jks -storepass kspass -noprompt -alias elasticsearch Certificate reply was installed in keystore + echo All done for elasticsearch + [[ ! -f /tmp/openshift-logging-ansible-YJJ6SM/logging-es.jks ]] ++ join , logging-es logging-es.logging.svc.cluster.local logging-es-cluster logging-es-cluster.logging.svc.cluster.local logging-es-ops logging-es-ops.logging.svc.cluster.local logging-es-ops-cluster logging-es-ops-cluster.logging.svc.cluster.local ++ local IFS=, ++ shift ++ echo logging-es,logging-es.logging.svc.cluster.local,logging-es-cluster,logging-es-cluster.logging.svc.cluster.local,logging-es-ops,logging-es-ops.logging.svc.cluster.local,logging-es-ops-cluster,logging-es-ops-cluster.logging.svc.cluster.local + generate_JKS_chain false logging-es logging-es,logging-es.logging.svc.cluster.local,logging-es-cluster,logging-es-cluster.logging.svc.cluster.local,logging-es-ops,logging-es-ops.logging.svc.cluster.local,logging-es-ops-cluster,logging-es-ops-cluster.logging.svc.cluster.local + dir=/tmp/openshift-logging-ansible-YJJ6SM + ADD_OID=false + NODE_NAME=logging-es + CERT_NAMES=logging-es,logging-es.logging.svc.cluster.local,logging-es-cluster,logging-es-cluster.logging.svc.cluster.local,logging-es-ops,logging-es-ops.logging.svc.cluster.local,logging-es-ops-cluster,logging-es-ops-cluster.logging.svc.cluster.local + ks_pass=kspass + ts_pass=tspass + rm -rf logging-es + extension_names= + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster + for name in '${CERT_NAMES//,/ }' + extension_names=,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local + '[' false = true ']' + echo Generating keystore and certificate for node logging-es + keytool -genkey -alias logging-es -keystore /tmp/openshift-logging-ansible-YJJ6SM/logging-es.jks -keypass kspass -storepass kspass -keyalg RSA -keysize 2048 -validity 712 -dname 'CN=logging-es, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local + echo Generating certificate signing request for node logging-es + keytool -certreq -alias logging-es -keystore /tmp/openshift-logging-ansible-YJJ6SM/logging-es.jks -storepass kspass -file /tmp/openshift-logging-ansible-YJJ6SM/logging-es.csr -keyalg rsa -dname 'CN=logging-es, OU=OpenShift, O=Logging' -ext san=dns:localhost,ip:127.0.0.1,dns:logging-es,dns:logging-es.logging.svc.cluster.local,dns:logging-es-cluster,dns:logging-es-cluster.logging.svc.cluster.local,dns:logging-es-ops,dns:logging-es-ops.logging.svc.cluster.local,dns:logging-es-ops-cluster,dns:logging-es-ops-cluster.logging.svc.cluster.local + echo Sign certificate request with CA + openssl ca -in /tmp/openshift-logging-ansible-YJJ6SM/logging-es.csr -notext -out /tmp/openshift-logging-ansible-YJJ6SM/logging-es.crt -config /tmp/openshift-logging-ansible-YJJ6SM/signing.conf -extensions v3_req -batch -extensions server_ext Using configuration from /tmp/openshift-logging-ansible-YJJ6SM/signing.conf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 8 (0x8) Validity Not Before: Jun 9 16:36:18 2017 GMT Not After : Jun 9 16:36:18 2019 GMT Subject: organizationName = Logging organizationalUnitName = OpenShift commonName = logging-es X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Basic Constraints: CA:FALSE X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication X509v3 Subject Key Identifier: 75:B0:7F:3E:2B:46:20:19:3D:8E:06:C5:E6:93:11:91:BD:70:4F:4A X509v3 Authority Key Identifier: 0. X509v3 Subject Alternative Name: DNS:localhost, IP Address:127.0.0.1, DNS:logging-es, DNS:logging-es.logging.svc.cluster.local, DNS:logging-es-cluster, DNS:logging-es-cluster.logging.svc.cluster.local, DNS:logging-es-ops, DNS:logging-es-ops.logging.svc.cluster.local, DNS:logging-es-ops-cluster, DNS:logging-es-ops-cluster.logging.svc.cluster.local Certificate is to be certified until Jun 9 16:36:18 2019 GMT (730 days) Write out database with 1 new entries Data Base Updated + echo 'Import back to keystore (including CA chain)' + keytool -import -file /tmp/openshift-logging-ansible-YJJ6SM/ca.crt -keystore /tmp/openshift-logging-ansible-YJJ6SM/logging-es.jks -storepass kspass -noprompt -alias sig-ca Certificate was added to keystore + keytool -import -file /tmp/openshift-logging-ansible-YJJ6SM/logging-es.crt -keystore /tmp/openshift-logging-ansible-YJJ6SM/logging-es.jks -storepass kspass -noprompt -alias logging-es Certificate reply was installed in keystore + echo All done for logging-es + '[' '!' -f /tmp/openshift-logging-ansible-YJJ6SM/truststore.jks ']' + createTruststore + echo 'Import CA to truststore for validating client certs' + keytool -import -file /tmp/openshift-logging-ansible-YJJ6SM/ca.crt -keystore /tmp/openshift-logging-ansible-YJJ6SM/truststore.jks -storepass tspass -noprompt -alias sig-ca Certificate was added to keystore + exit 0 TASK [openshift_logging : Pushing locally generated JKS certs to remote host...] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:66 changed: [openshift] => { "changed": true, "checksum": "4009bb7a4f459b9aa5750c53cd68ae46c385422c", "dest": "/etc/origin/logging/elasticsearch.jks", "gid": 0, "group": "root", "md5sum": "bb85e7f03e6a1511a9372af3bbefde4f", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 3767, "src": "/root/.ansible/tmp/ansible-tmp-1497026179.37-128716580312780/source", "state": "file", "uid": 0 } TASK [openshift_logging : Pushing locally generated JKS certs to remote host...] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:72 changed: [openshift] => { "changed": true, "checksum": "b852cd1cacc49dc0556bc50d8f966e1d6b9ed8de", "dest": "/etc/origin/logging/logging-es.jks", "gid": 0, "group": "root", "md5sum": "a08ce2784ff90fbe515d8024e414bf56", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 3982, "src": "/root/.ansible/tmp/ansible-tmp-1497026179.6-196216547862854/source", "state": "file", "uid": 0 } TASK [openshift_logging : Pushing locally generated JKS certs to remote host...] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:78 changed: [openshift] => { "changed": true, "checksum": "355fff97bde7f1b3878b025724211339536a7bc1", "dest": "/etc/origin/logging/system.admin.jks", "gid": 0, "group": "root", "md5sum": "d39dd274f0a6071029ab2000fc436103", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 3700, "src": "/root/.ansible/tmp/ansible-tmp-1497026179.82-277326758949488/source", "state": "file", "uid": 0 } TASK [openshift_logging : Pushing locally generated JKS certs to remote host...] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_jks.yaml:84 changed: [openshift] => { "changed": true, "checksum": "6ac4079616dcf7111d6c74160d1f55c1cd0d4982", "dest": "/etc/origin/logging/truststore.jks", "gid": 0, "group": "root", "md5sum": "8645a70586bb76ce14e932856db9344a", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 797, "src": "/root/.ansible/tmp/ansible-tmp-1497026180.05-232010964180718/source", "state": "file", "uid": 0 } TASK [openshift_logging : Generate proxy session] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:141 ok: [openshift] => { "ansible_facts": { "session_secret": "Ho9ijP7CJc8sLfDVVWAK2QYpD26cqc5t16TEKP9OxRYtUZxqpaI5kcI85vOKdbVKrq8bMWX95hgqqaZhpEidFbuO8IALgTFYRYkOlF8njDtFspCAvaMylyaQe95BcwLMkP51CEkJRFwLF6aWANULVTHnid6NBMuPYnLM8MHJrdkmwwv9VbMtdk1Rw7yoDKnWq4FTfg8X" }, "changed": false } TASK [openshift_logging : Generate oauth client secret] ************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/generate_certs.yaml:146 ok: [openshift] => { "ansible_facts": { "oauth_secret": "fHXvW6fbE0oKuRSxEcVgjbqiLDGGACuBoOIBDXxyM9GRXlXaTWUEMLPKPvDTU3zI" }, "changed": false } TASK [openshift_logging : set_fact] ******************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:53 TASK [openshift_logging : set_fact] ******************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:57 ok: [openshift] => { "ansible_facts": { "es_indices": "[]" }, "changed": false } TASK [openshift_logging : set_fact] ******************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:60 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:64 TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:85 statically included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml TASK [openshift_logging_elasticsearch : Validate Elasticsearch cluster size] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:2 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Validate Elasticsearch Ops cluster size] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:6 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : fail] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:10 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:14 ok: [openshift] => { "ansible_facts": { "elasticsearch_name": "logging-elasticsearch", "es_component": "es" }, "changed": false } TASK [openshift_logging_elasticsearch : fail] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:3 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:7 ok: [openshift] => { "ansible_facts": { "es_version": "3_5" }, "changed": false } TASK [openshift_logging_elasticsearch : debug] ********************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:11 ok: [openshift] => { "changed": false, "openshift_logging_image_version": "latest" } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:14 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : fail] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:17 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Create temp directory for doing work in] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:21 ok: [openshift] => { "changed": false, "cmd": [ "mktemp", "-d", "/tmp/openshift-logging-ansible-XXXXXX" ], "delta": "0:00:00.002030", "end": "2017-06-09 12:36:20.979396", "rc": 0, "start": "2017-06-09 12:36:20.977366" } STDOUT: /tmp/openshift-logging-ansible-9FHFr2 TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:26 ok: [openshift] => { "ansible_facts": { "tempdir": "/tmp/openshift-logging-ansible-9FHFr2" }, "changed": false } TASK [openshift_logging_elasticsearch : Create templates subdirectory] ********* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:30 ok: [openshift] => { "changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/tmp/openshift-logging-ansible-9FHFr2/templates", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 6, "state": "directory", "uid": 0 } TASK [openshift_logging_elasticsearch : Create ES service account] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:40 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Create ES service account] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:48 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get sa aggregated-logging-elasticsearch -o json -n logging", "results": [ { "apiVersion": "v1", "imagePullSecrets": [ { "name": "aggregated-logging-elasticsearch-dockercfg-81j90" } ], "kind": "ServiceAccount", "metadata": { "creationTimestamp": "2017-06-09T16:36:21Z", "name": "aggregated-logging-elasticsearch", "namespace": "logging", "resourceVersion": "1283", "selfLink": "/api/v1/namespaces/logging/serviceaccounts/aggregated-logging-elasticsearch", "uid": "c55ea4a0-4d31-11e7-8061-0ed679898f5a" }, "secrets": [ { "name": "aggregated-logging-elasticsearch-token-ms48q" }, { "name": "aggregated-logging-elasticsearch-dockercfg-81j90" } ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : copy] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:57 changed: [openshift] => { "changed": true, "checksum": "e5015364391ac609da8655a9a1224131599a5cea", "dest": "/tmp/openshift-logging-ansible-9FHFr2/rolebinding-reader.yml", "gid": 0, "group": "root", "md5sum": "446fb96447527f48f97e69bb41bad7be", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 135, "src": "/root/.ansible/tmp/ansible-tmp-1497026182.16-90174521482233/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : Create rolebinding-reader role] ******** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:61 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get clusterrole rolebinding-reader -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "ClusterRole", "metadata": { "creationTimestamp": "2017-06-09T16:36:22Z", "name": "rolebinding-reader", "resourceVersion": "122", "selfLink": "/oapi/v1/clusterroles/rolebinding-reader", "uid": "c607da6a-4d31-11e7-8061-0ed679898f5a" }, "rules": [ { "apiGroups": [ "" ], "attributeRestrictions": null, "resources": [ "clusterrolebindings" ], "verbs": [ "get" ] } ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Set rolebinding-reader permissions for ES] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:72 changed: [openshift] => { "changed": true, "present": "present", "results": { "cmd": "/bin/oc adm policy add-cluster-role-to-user rolebinding-reader system:serviceaccount:logging:aggregated-logging-elasticsearch -n logging", "results": "", "returncode": 0 } } TASK [openshift_logging_elasticsearch : Generate logging-elasticsearch-view-role] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:81 ok: [openshift] => { "changed": false, "checksum": "d752c09323565f80ed14fa806d42284f0c5aef2a", "dest": "/tmp/openshift-logging-ansible-9FHFr2/logging-elasticsearch-view-role.yaml", "gid": 0, "group": "root", "md5sum": "8299dca2fb036c06ba7c4f620680e0f6", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 183, "src": "/root/.ansible/tmp/ansible-tmp-1497026183.9-132312945734937/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : Set logging-elasticsearch-view-role role] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:94 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get rolebinding logging-elasticsearch-view-role -o json -n logging", "results": [ { "apiVersion": "v1", "groupNames": null, "kind": "RoleBinding", "metadata": { "creationTimestamp": "2017-06-09T16:36:24Z", "name": "logging-elasticsearch-view-role", "namespace": "logging", "resourceVersion": "711", "selfLink": "/oapi/v1/namespaces/logging/rolebindings/logging-elasticsearch-view-role", "uid": "c70ec6cc-4d31-11e7-8061-0ed679898f5a" }, "roleRef": { "name": "view" }, "subjects": [ { "kind": "ServiceAccount", "name": "aggregated-logging-elasticsearch", "namespace": "logging" } ], "userNames": [ "system:serviceaccount:logging:aggregated-logging-elasticsearch" ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : template] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:105 ok: [openshift] => { "changed": false, "checksum": "f91458d5dad42c496e2081ef872777a6f6eb9ff9", "dest": "/tmp/openshift-logging-ansible-9FHFr2/elasticsearch-logging.yml", "gid": 0, "group": "root", "md5sum": "e4be7c33c1927bbdd8c909bfbe3d9f0b", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 2171, "src": "/root/.ansible/tmp/ansible-tmp-1497026184.91-254937888745160/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : template] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:111 ok: [openshift] => { "changed": false, "checksum": "6d4f976f6e77a6e0c8dca7e01fb5bedb68678b1d", "dest": "/tmp/openshift-logging-ansible-9FHFr2/elasticsearch.yml", "gid": 0, "group": "root", "md5sum": "75abfd3a190832e593a8e5e7c5695e8e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 2454, "src": "/root/.ansible/tmp/ansible-tmp-1497026185.15-9765826295016/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : copy] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:121 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : copy] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:127 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Set ES configmap] ********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:133 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get configmap logging-elasticsearch -o json -n logging", "results": [ { "apiVersion": "v1", "data": { "elasticsearch.yml": "cluster:\n name: ${CLUSTER_NAME}\n\nscript:\n inline: on\n indexed: on\n\nindex:\n number_of_shards: 1\n number_of_replicas: 0\n unassigned.node_left.delayed_timeout: 2m\n translog:\n flush_threshold_size: 256mb\n flush_threshold_period: 5m\n\nnode:\n master: ${IS_MASTER}\n data: ${HAS_DATA}\n\nnetwork:\n host: 0.0.0.0\n\ncloud:\n kubernetes:\n service: ${SERVICE_DNS}\n namespace: ${NAMESPACE}\n\ndiscovery:\n type: kubernetes\n zen.ping.multicast.enabled: false\n zen.minimum_master_nodes: ${NODE_QUORUM}\n\ngateway:\n recover_after_nodes: ${NODE_QUORUM}\n expected_nodes: ${RECOVER_EXPECTED_NODES}\n recover_after_time: ${RECOVER_AFTER_TIME}\n\nio.fabric8.elasticsearch.authentication.users: [\"system.logging.kibana\", \"system.logging.fluentd\", \"system.logging.curator\", \"system.admin\"]\nio.fabric8.elasticsearch.kibana.mapping.app: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json\nio.fabric8.elasticsearch.kibana.mapping.ops: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json\nio.fabric8.elasticsearch.kibana.mapping.empty: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json\n\nopenshift.config:\n use_common_data_model: true\n project_index_prefix: \"project\"\n time_field_name: \"@timestamp\"\n\nopenshift.searchguard:\n keystore.path: /etc/elasticsearch/secret/admin.jks\n truststore.path: /etc/elasticsearch/secret/searchguard.truststore\n\nopenshift.operations.allow_cluster_reader: false\n\npath:\n data: /elasticsearch/persistent/${CLUSTER_NAME}/data\n logs: /elasticsearch/${CLUSTER_NAME}/logs\n work: /elasticsearch/${CLUSTER_NAME}/work\n scripts: /elasticsearch/${CLUSTER_NAME}/scripts\n\nsearchguard:\n authcz.admin_dn:\n - CN=system.admin,OU=OpenShift,O=Logging\n config_index_name: \".searchguard.${HOSTNAME}\"\n ssl:\n transport:\n enabled: true\n enforce_hostname_verification: false\n keystore_type: JKS\n keystore_filepath: /etc/elasticsearch/secret/searchguard.key\n keystore_password: kspass\n truststore_type: JKS\n truststore_filepath: /etc/elasticsearch/secret/searchguard.truststore\n truststore_password: tspass\n http:\n enabled: true\n keystore_type: JKS\n keystore_filepath: /etc/elasticsearch/secret/key\n keystore_password: kspass\n clientauth_mode: OPTIONAL\n truststore_type: JKS\n truststore_filepath: /etc/elasticsearch/secret/truststore\n truststore_password: tspass\n", "logging.yml": "# you can override this using by setting a system property, for example -Des.logger.level=DEBUG\nes.logger.level: INFO\nrootLogger: ${es.logger.level}, console, file\nlogger:\n # log action execution errors for easier debugging\n action: WARN\n # reduce the logging for aws, too much is logged under the default INFO\n com.amazonaws: WARN\n io.fabric8.elasticsearch: ${PLUGIN_LOGLEVEL}\n io.fabric8.kubernetes: ${PLUGIN_LOGLEVEL}\n\n # gateway\n #gateway: DEBUG\n #index.gateway: DEBUG\n\n # peer shard recovery\n #indices.recovery: DEBUG\n\n # discovery\n #discovery: TRACE\n\n index.search.slowlog: TRACE, index_search_slow_log_file\n index.indexing.slowlog: TRACE, index_indexing_slow_log_file\n\n # search-guard\n com.floragunn.searchguard: WARN\n\nadditivity:\n index.search.slowlog: false\n index.indexing.slowlog: false\n\nappender:\n console:\n type: console\n layout:\n type: consolePattern\n conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n\n file:\n type: dailyRollingFile\n file: ${path.logs}/${cluster.name}.log\n datePattern: \"'.'yyyy-MM-dd\"\n layout:\n type: pattern\n conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n\n # Use the following log4j-extras RollingFileAppender to enable gzip compression of log files.\n # For more information see https://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html\n #file:\n #type: extrasRollingFile\n #file: ${path.logs}/${cluster.name}.log\n #rollingPolicy: timeBased\n #rollingPolicy.FileNamePattern: ${path.logs}/${cluster.name}.log.%d{yyyy-MM-dd}.gz\n #layout:\n #type: pattern\n #conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n\n index_search_slow_log_file:\n type: dailyRollingFile\n file: ${path.logs}/${cluster.name}_index_search_slowlog.log\n datePattern: \"'.'yyyy-MM-dd\"\n layout:\n type: pattern\n conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n\n index_indexing_slow_log_file:\n type: dailyRollingFile\n file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log\n datePattern: \"'.'yyyy-MM-dd\"\n layout:\n type: pattern\n conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n" }, "kind": "ConfigMap", "metadata": { "creationTimestamp": "2017-06-09T16:36:26Z", "name": "logging-elasticsearch", "namespace": "logging", "resourceVersion": "1290", "selfLink": "/api/v1/namespaces/logging/configmaps/logging-elasticsearch", "uid": "c7e3ae9c-4d31-11e7-8061-0ed679898f5a" } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Set ES secret] ************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:144 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc secrets new logging-elasticsearch key=/etc/origin/logging/logging-es.jks truststore=/etc/origin/logging/truststore.jks searchguard.key=/etc/origin/logging/elasticsearch.jks searchguard.truststore=/etc/origin/logging/truststore.jks admin-key=/etc/origin/logging/system.admin.key admin-cert=/etc/origin/logging/system.admin.crt admin-ca=/etc/origin/logging/ca.crt admin.jks=/etc/origin/logging/system.admin.jks -n logging", "results": "", "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Set logging-es-cluster service] ******** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:168 changed: [openshift] => { "changed": true, "results": { "clusterip": "172.30.206.91", "cmd": "/bin/oc get service logging-es-cluster -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "Service", "metadata": { "creationTimestamp": "2017-06-09T16:36:27Z", "name": "logging-es-cluster", "namespace": "logging", "resourceVersion": "1293", "selfLink": "/api/v1/namespaces/logging/services/logging-es-cluster", "uid": "c8ed8991-4d31-11e7-8061-0ed679898f5a" }, "spec": { "clusterIP": "172.30.206.91", "ports": [ { "port": 9300, "protocol": "TCP", "targetPort": 9300 } ], "selector": { "component": "es", "provider": "openshift" }, "sessionAffinity": "None", "type": "ClusterIP" }, "status": { "loadBalancer": {} } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Set logging-es service] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:182 changed: [openshift] => { "changed": true, "results": { "clusterip": "172.30.128.77", "cmd": "/bin/oc get service logging-es -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "Service", "metadata": { "creationTimestamp": "2017-06-09T16:36:28Z", "name": "logging-es", "namespace": "logging", "resourceVersion": "1298", "selfLink": "/api/v1/namespaces/logging/services/logging-es", "uid": "c98a5ef0-4d31-11e7-8061-0ed679898f5a" }, "spec": { "clusterIP": "172.30.128.77", "ports": [ { "port": 9200, "protocol": "TCP", "targetPort": "restapi" } ], "selector": { "component": "es", "provider": "openshift" }, "sessionAffinity": "None", "type": "ClusterIP" }, "status": { "loadBalancer": {} } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Creating ES storage template] ********** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:197 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Creating ES storage template] ********** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:210 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Set ES storage] ************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:225 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:237 ok: [openshift] => { "ansible_facts": { "es_deploy_name": "logging-es-data-master-03l0u89m" }, "changed": false } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:241 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Set ES dc templates] ******************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:246 changed: [openshift] => { "changed": true, "checksum": "9a19134a0c4df8a1a79c3faa428b07c0af49022e", "dest": "/tmp/openshift-logging-ansible-9FHFr2/templates/logging-es-dc.yml", "gid": 0, "group": "root", "md5sum": "0e5f38dad26c6217bcedb74c68982aa7", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 3138, "src": "/root/.ansible/tmp/ansible-tmp-1497026189.31-49456034179108/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : Set ES dc] ***************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:262 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get dc logging-es-data-master-03l0u89m -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "DeploymentConfig", "metadata": { "creationTimestamp": "2017-06-09T16:36:30Z", "generation": 2, "labels": { "component": "es", "deployment": "logging-es-data-master-03l0u89m", "logging-infra": "elasticsearch", "provider": "openshift" }, "name": "logging-es-data-master-03l0u89m", "namespace": "logging", "resourceVersion": "1312", "selfLink": "/oapi/v1/namespaces/logging/deploymentconfigs/logging-es-data-master-03l0u89m", "uid": "ca490840-4d31-11e7-8061-0ed679898f5a" }, "spec": { "replicas": 1, "selector": { "component": "es", "deployment": "logging-es-data-master-03l0u89m", "logging-infra": "elasticsearch", "provider": "openshift" }, "strategy": { "activeDeadlineSeconds": 21600, "recreateParams": { "timeoutSeconds": 600 }, "resources": {}, "type": "Recreate" }, "template": { "metadata": { "creationTimestamp": null, "labels": { "component": "es", "deployment": "logging-es-data-master-03l0u89m", "logging-infra": "elasticsearch", "provider": "openshift" }, "name": "logging-es-data-master-03l0u89m" }, "spec": { "containers": [ { "env": [ { "name": "NAMESPACE", "valueFrom": { "fieldRef": { "apiVersion": "v1", "fieldPath": "metadata.namespace" } } }, { "name": "KUBERNETES_TRUST_CERT", "value": "true" }, { "name": "SERVICE_DNS", "value": "logging-es-cluster" }, { "name": "CLUSTER_NAME", "value": "logging-es" }, { "name": "INSTANCE_RAM", "value": "8Gi" }, { "name": "NODE_QUORUM", "value": "1" }, { "name": "RECOVER_EXPECTED_NODES", "value": "1" }, { "name": "RECOVER_AFTER_TIME", "value": "5m" }, { "name": "READINESS_PROBE_TIMEOUT", "value": "30" }, { "name": "IS_MASTER", "value": "true" }, { "name": "HAS_DATA", "value": "true" } ], "image": "172.30.180.71:5000/logging/logging-elasticsearch:latest", "imagePullPolicy": "Always", "name": "elasticsearch", "ports": [ { "containerPort": 9200, "name": "restapi", "protocol": "TCP" }, { "containerPort": 9300, "name": "cluster", "protocol": "TCP" } ], "readinessProbe": { "exec": { "command": [ "/usr/share/elasticsearch/probe/readiness.sh" ] }, "failureThreshold": 3, "initialDelaySeconds": 10, "periodSeconds": 5, "successThreshold": 1, "timeoutSeconds": 30 }, "resources": { "limits": { "cpu": "1", "memory": "8Gi" }, "requests": { "memory": "512Mi" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/etc/elasticsearch/secret", "name": "elasticsearch", "readOnly": true }, { "mountPath": "/usr/share/java/elasticsearch/config", "name": "elasticsearch-config", "readOnly": true }, { "mountPath": "/elasticsearch/persistent", "name": "elasticsearch-storage" } ] } ], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": { "supplementalGroups": [ 65534 ] }, "serviceAccount": "aggregated-logging-elasticsearch", "serviceAccountName": "aggregated-logging-elasticsearch", "terminationGracePeriodSeconds": 30, "volumes": [ { "name": "elasticsearch", "secret": { "defaultMode": 420, "secretName": "logging-elasticsearch" } }, { "configMap": { "defaultMode": 420, "name": "logging-elasticsearch" }, "name": "elasticsearch-config" }, { "emptyDir": {}, "name": "elasticsearch-storage" } ] } }, "test": false, "triggers": [ { "type": "ConfigChange" } ] }, "status": { "availableReplicas": 0, "conditions": [ { "lastTransitionTime": "2017-06-09T16:36:30Z", "lastUpdateTime": "2017-06-09T16:36:30Z", "message": "Deployment config does not have minimum availability.", "status": "False", "type": "Available" }, { "lastTransitionTime": "2017-06-09T16:36:30Z", "lastUpdateTime": "2017-06-09T16:36:30Z", "message": "replication controller \"logging-es-data-master-03l0u89m-1\" is waiting for pod \"logging-es-data-master-03l0u89m-1-deploy\" to run", "status": "Unknown", "type": "Progressing" } ], "details": { "causes": [ { "type": "ConfigChange" } ], "message": "config change" }, "latestVersion": 1, "observedGeneration": 2, "replicas": 0, "unavailableReplicas": 0, "updatedReplicas": 0 } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Delete temp directory] ***************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:274 ok: [openshift] => { "changed": false, "path": "/tmp/openshift-logging-ansible-9FHFr2", "state": "absent" } TASK [openshift_logging : set_fact] ******************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:99 TASK [openshift_logging : set_fact] ******************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:105 ok: [openshift] => { "ansible_facts": { "es_ops_indices": "[]" }, "changed": false } TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:109 TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:132 statically included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml TASK [openshift_logging_elasticsearch : Validate Elasticsearch cluster size] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:2 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Validate Elasticsearch Ops cluster size] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:6 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : fail] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:10 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:14 ok: [openshift] => { "ansible_facts": { "elasticsearch_name": "logging-elasticsearch-ops", "es_component": "es-ops" }, "changed": false } TASK [openshift_logging_elasticsearch : fail] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:3 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:7 ok: [openshift] => { "ansible_facts": { "es_version": "3_5" }, "changed": false } TASK [openshift_logging_elasticsearch : debug] ********************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:11 ok: [openshift] => { "changed": false, "openshift_logging_image_version": "latest" } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:14 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : fail] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/determine_version.yaml:17 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Create temp directory for doing work in] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:21 ok: [openshift] => { "changed": false, "cmd": [ "mktemp", "-d", "/tmp/openshift-logging-ansible-XXXXXX" ], "delta": "0:00:00.002177", "end": "2017-06-09 12:36:31.314983", "rc": 0, "start": "2017-06-09 12:36:31.312806" } STDOUT: /tmp/openshift-logging-ansible-VAlbdx TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:26 ok: [openshift] => { "ansible_facts": { "tempdir": "/tmp/openshift-logging-ansible-VAlbdx" }, "changed": false } TASK [openshift_logging_elasticsearch : Create templates subdirectory] ********* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:30 ok: [openshift] => { "changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/tmp/openshift-logging-ansible-VAlbdx/templates", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 6, "state": "directory", "uid": 0 } TASK [openshift_logging_elasticsearch : Create ES service account] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:40 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Create ES service account] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:48 ok: [openshift] => { "changed": false, "results": { "cmd": "/bin/oc get sa aggregated-logging-elasticsearch -o json -n logging", "results": [ { "apiVersion": "v1", "imagePullSecrets": [ { "name": "aggregated-logging-elasticsearch-dockercfg-81j90" } ], "kind": "ServiceAccount", "metadata": { "creationTimestamp": "2017-06-09T16:36:21Z", "name": "aggregated-logging-elasticsearch", "namespace": "logging", "resourceVersion": "1283", "selfLink": "/api/v1/namespaces/logging/serviceaccounts/aggregated-logging-elasticsearch", "uid": "c55ea4a0-4d31-11e7-8061-0ed679898f5a" }, "secrets": [ { "name": "aggregated-logging-elasticsearch-token-ms48q" }, { "name": "aggregated-logging-elasticsearch-dockercfg-81j90" } ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : copy] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:57 changed: [openshift] => { "changed": true, "checksum": "e5015364391ac609da8655a9a1224131599a5cea", "dest": "/tmp/openshift-logging-ansible-VAlbdx/rolebinding-reader.yml", "gid": 0, "group": "root", "md5sum": "446fb96447527f48f97e69bb41bad7be", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 135, "src": "/root/.ansible/tmp/ansible-tmp-1497026192.07-31968981539646/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : Create rolebinding-reader role] ******** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:61 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get clusterrole rolebinding-reader -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "ClusterRole", "metadata": { "creationTimestamp": "2017-06-09T16:36:22Z", "name": "rolebinding-reader", "resourceVersion": "122", "selfLink": "/oapi/v1/clusterroles/rolebinding-reader", "uid": "c607da6a-4d31-11e7-8061-0ed679898f5a" }, "rules": [ { "apiGroups": [ "" ], "attributeRestrictions": null, "resources": [ "clusterrolebindings" ], "verbs": [ "get" ] } ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Set rolebinding-reader permissions for ES] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:72 ok: [openshift] => { "changed": false, "present": "present" } TASK [openshift_logging_elasticsearch : Generate logging-elasticsearch-view-role] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:81 ok: [openshift] => { "changed": false, "checksum": "d752c09323565f80ed14fa806d42284f0c5aef2a", "dest": "/tmp/openshift-logging-ansible-VAlbdx/logging-elasticsearch-view-role.yaml", "gid": 0, "group": "root", "md5sum": "8299dca2fb036c06ba7c4f620680e0f6", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 183, "src": "/root/.ansible/tmp/ansible-tmp-1497026193.78-172718547938487/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : Set logging-elasticsearch-view-role role] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:94 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get rolebinding logging-elasticsearch-view-role -o json -n logging", "results": [ { "apiVersion": "v1", "groupNames": null, "kind": "RoleBinding", "metadata": { "creationTimestamp": "2017-06-09T16:36:24Z", "name": "logging-elasticsearch-view-role", "namespace": "logging", "resourceVersion": "1288", "selfLink": "/oapi/v1/namespaces/logging/rolebindings/logging-elasticsearch-view-role", "uid": "c70ec6cc-4d31-11e7-8061-0ed679898f5a" }, "roleRef": { "name": "view" }, "subjects": [ { "kind": "ServiceAccount", "name": "aggregated-logging-elasticsearch", "namespace": "logging" } ], "userNames": [ "system:serviceaccount:logging:aggregated-logging-elasticsearch" ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : template] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:105 ok: [openshift] => { "changed": false, "checksum": "f91458d5dad42c496e2081ef872777a6f6eb9ff9", "dest": "/tmp/openshift-logging-ansible-VAlbdx/elasticsearch-logging.yml", "gid": 0, "group": "root", "md5sum": "e4be7c33c1927bbdd8c909bfbe3d9f0b", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 2171, "src": "/root/.ansible/tmp/ansible-tmp-1497026195.15-35890814142635/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : template] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:111 ok: [openshift] => { "changed": false, "checksum": "6d4f976f6e77a6e0c8dca7e01fb5bedb68678b1d", "dest": "/tmp/openshift-logging-ansible-VAlbdx/elasticsearch.yml", "gid": 0, "group": "root", "md5sum": "75abfd3a190832e593a8e5e7c5695e8e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 2454, "src": "/root/.ansible/tmp/ansible-tmp-1497026195.42-111937644196865/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : copy] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:121 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : copy] ********************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:127 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Set ES configmap] ********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:133 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get configmap logging-elasticsearch-ops -o json -n logging", "results": [ { "apiVersion": "v1", "data": { "elasticsearch.yml": "cluster:\n name: ${CLUSTER_NAME}\n\nscript:\n inline: on\n indexed: on\n\nindex:\n number_of_shards: 1\n number_of_replicas: 0\n unassigned.node_left.delayed_timeout: 2m\n translog:\n flush_threshold_size: 256mb\n flush_threshold_period: 5m\n\nnode:\n master: ${IS_MASTER}\n data: ${HAS_DATA}\n\nnetwork:\n host: 0.0.0.0\n\ncloud:\n kubernetes:\n service: ${SERVICE_DNS}\n namespace: ${NAMESPACE}\n\ndiscovery:\n type: kubernetes\n zen.ping.multicast.enabled: false\n zen.minimum_master_nodes: ${NODE_QUORUM}\n\ngateway:\n recover_after_nodes: ${NODE_QUORUM}\n expected_nodes: ${RECOVER_EXPECTED_NODES}\n recover_after_time: ${RECOVER_AFTER_TIME}\n\nio.fabric8.elasticsearch.authentication.users: [\"system.logging.kibana\", \"system.logging.fluentd\", \"system.logging.curator\", \"system.admin\"]\nio.fabric8.elasticsearch.kibana.mapping.app: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json\nio.fabric8.elasticsearch.kibana.mapping.ops: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json\nio.fabric8.elasticsearch.kibana.mapping.empty: /usr/share/elasticsearch/index_patterns/com.redhat.viaq-openshift.index-pattern.json\n\nopenshift.config:\n use_common_data_model: true\n project_index_prefix: \"project\"\n time_field_name: \"@timestamp\"\n\nopenshift.searchguard:\n keystore.path: /etc/elasticsearch/secret/admin.jks\n truststore.path: /etc/elasticsearch/secret/searchguard.truststore\n\nopenshift.operations.allow_cluster_reader: false\n\npath:\n data: /elasticsearch/persistent/${CLUSTER_NAME}/data\n logs: /elasticsearch/${CLUSTER_NAME}/logs\n work: /elasticsearch/${CLUSTER_NAME}/work\n scripts: /elasticsearch/${CLUSTER_NAME}/scripts\n\nsearchguard:\n authcz.admin_dn:\n - CN=system.admin,OU=OpenShift,O=Logging\n config_index_name: \".searchguard.${HOSTNAME}\"\n ssl:\n transport:\n enabled: true\n enforce_hostname_verification: false\n keystore_type: JKS\n keystore_filepath: /etc/elasticsearch/secret/searchguard.key\n keystore_password: kspass\n truststore_type: JKS\n truststore_filepath: /etc/elasticsearch/secret/searchguard.truststore\n truststore_password: tspass\n http:\n enabled: true\n keystore_type: JKS\n keystore_filepath: /etc/elasticsearch/secret/key\n keystore_password: kspass\n clientauth_mode: OPTIONAL\n truststore_type: JKS\n truststore_filepath: /etc/elasticsearch/secret/truststore\n truststore_password: tspass\n", "logging.yml": "# you can override this using by setting a system property, for example -Des.logger.level=DEBUG\nes.logger.level: INFO\nrootLogger: ${es.logger.level}, console, file\nlogger:\n # log action execution errors for easier debugging\n action: WARN\n # reduce the logging for aws, too much is logged under the default INFO\n com.amazonaws: WARN\n io.fabric8.elasticsearch: ${PLUGIN_LOGLEVEL}\n io.fabric8.kubernetes: ${PLUGIN_LOGLEVEL}\n\n # gateway\n #gateway: DEBUG\n #index.gateway: DEBUG\n\n # peer shard recovery\n #indices.recovery: DEBUG\n\n # discovery\n #discovery: TRACE\n\n index.search.slowlog: TRACE, index_search_slow_log_file\n index.indexing.slowlog: TRACE, index_indexing_slow_log_file\n\n # search-guard\n com.floragunn.searchguard: WARN\n\nadditivity:\n index.search.slowlog: false\n index.indexing.slowlog: false\n\nappender:\n console:\n type: console\n layout:\n type: consolePattern\n conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n\n file:\n type: dailyRollingFile\n file: ${path.logs}/${cluster.name}.log\n datePattern: \"'.'yyyy-MM-dd\"\n layout:\n type: pattern\n conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n\n # Use the following log4j-extras RollingFileAppender to enable gzip compression of log files.\n # For more information see https://logging.apache.org/log4j/extras/apidocs/org/apache/log4j/rolling/RollingFileAppender.html\n #file:\n #type: extrasRollingFile\n #file: ${path.logs}/${cluster.name}.log\n #rollingPolicy: timeBased\n #rollingPolicy.FileNamePattern: ${path.logs}/${cluster.name}.log.%d{yyyy-MM-dd}.gz\n #layout:\n #type: pattern\n #conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n\n index_search_slow_log_file:\n type: dailyRollingFile\n file: ${path.logs}/${cluster.name}_index_search_slowlog.log\n datePattern: \"'.'yyyy-MM-dd\"\n layout:\n type: pattern\n conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n\n index_indexing_slow_log_file:\n type: dailyRollingFile\n file: ${path.logs}/${cluster.name}_index_indexing_slowlog.log\n datePattern: \"'.'yyyy-MM-dd\"\n layout:\n type: pattern\n conversionPattern: \"[%d{ISO8601}][%-5p][%-25c] %m%n\"\n" }, "kind": "ConfigMap", "metadata": { "creationTimestamp": "2017-06-09T16:36:36Z", "name": "logging-elasticsearch-ops", "namespace": "logging", "resourceVersion": "1340", "selfLink": "/api/v1/namespaces/logging/configmaps/logging-elasticsearch-ops", "uid": "ce10fdb8-4d31-11e7-8061-0ed679898f5a" } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Set ES secret] ************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:144 ok: [openshift] => { "changed": false, "results": { "apiVersion": "v1", "data": { "admin-ca": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMyakNDQWNLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpVMU5sb1hEVEl5TURZd09ERTJNelUxTjFvdwpIakVjTUJvR0ExVUVBeE1UYkc5bloybHVaeTF6YVdkdVpYSXRkR1Z6ZERDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1IRDJVYWdmRzVYQitRYnRkcTc4TXRmUWMxRVkydHRaZVdzNHdHTGhIRUMKR1VzMFFpZEVzZ0k3cmo1a2ZySDdrQ01sZnd2N20yRmpYL1grNGJJN2g5QlhhY05Mc2RKUkRNbUhVLytzNjdBKwo1TWM1VjFJU0VqYWNMeG5PZmZrNTNzVGg0U282WkNrQmkrcDlqVmxTN1hCdi9nV2doaU1odEVpLzJoaUIybE9iCmplcCsrTWVKVDRCQVJGNjI3OTRnV0w3eklNc2t3a1ZhYkxKYUlubWtWSVZLOTJqOEdTaVJDa0VNcUJJNFZqRDgKQnVRa09pN3pHZWxlQ3RDMTRMTW01T21NMkx3Vi9ERktpY2hBRjBuVkxLRG9TWUNtZmVIalNLeUVVVk54d01lagpsM3ZDYjRjOTRRcCtXWlZFQSt4UE91T2p1eFIrUkFmN2p0bVl4dXd2c0M4Q0F3RUFBYU1qTUNFd0RnWURWUjBQCkFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGcWYKZU1HZEdVellycHFIT3BnTzhvdzFEai9GZzA5dG5yd2dVQUZwd0NmQ1lheXdxUFJYUmdtMHBVUngrTFl2UnI1SQpSbWEvYzI4LzhjbVdXUG5ReEFHOUYyWVFnZU5TbExZNWY0U2NJemFzaDVQMWZIQlVRWUlWUzNkWnB1QXJrR1hOCkpySkszbFBHY0hWNm5ZRys5ZUs3b1FWS0Nrd1IvWnMvQ21FSHNOTGdPcDZRSFhPS0hTcWtlMmpwNHZmSG1TdC8Kb0R4VFRVSFIzN3g2cTEyTTRoYVhYM2NTWCthYkNRYzFZUU5kcUxiZHhYbjZ6OE14dVptRkI5YnJnSkVoU2N2Rgp2VHh6TnRpUnkzVHNkZUxROForWjF0cHNiR1NvSmFPUFJUcit3ZWNzM3RETEpuR1hwaGc0NHc2UW0zNnZkNENtCjhoaTN6ODczbEdSNmVGL05ZN1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K", "admin-cert": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURQRENDQWlTZ0F3SUJBZ0lCQlRBTkJna3Foa2lHOXcwQkFRVUZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpZd00xb1hEVEU1TURZd09URTJNell3TTFvdwpQVEVRTUE0R0ExVUVDZ3dIVEc5bloybHVaekVTTUJBR0ExVUVDd3dKVDNCbGJsTm9hV1owTVJVd0V3WURWUVFECkRBeHplWE4wWlcwdVlXUnRhVzR3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRRGgKdVpkRm10cGdYa0hzSnR0NVViT0pzRGZXcE5OR2hYMDkzK2pLVGRQV040ZCtnVXlpVUU4YjR2dWFpNHJnUThYbwp5b25QeXhPRDJ5RmM0L3RFcDN2N245ZnhKY1dnNHN1VGNmN1M0YzNsNjkwY0hXVVRDUWxjczhRT1lTQkxoSzcvCkN4T0NaU3ZKMm1Lem5YQWRnQU9NS1YxZmttNVVGMTQ1b0hqWmMxcHpzYU9iUUpQMWw2dW54Nk5KWlA5UXoycHoKNGVNQmhEM1ZScFBZNjFqYllXdVFiNzNSOTFOdHM2RzJGMDFqdUJ3dzhpQ1RjWmdPUzlJVmR3S2h3SUpiMnFyKwp4aitoemhsS09vR2lhVXRHUWVGLzZMTW1NeHUySUEvcEQ4dGVBbldsYXZnWVFDRVRiM3p5Znk0Q3JpWlVpNUViCmdSelluSlBwTFc1aFoyVU5OT0N2QWdNQkFBR2paakJrTUE0R0ExVWREd0VCL3dRRUF3SUZvREFKQmdOVkhSTUUKQWpBQU1CMEdBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFkQmdOVkhRNEVGZ1FVdnBTSgo2UUMyUnRMbDhycU91eDRvZXozeDZjNHdDUVlEVlIwakJBSXdBREFOQmdrcWhraUc5dzBCQVFVRkFBT0NBUUVBClJaNEsxS3d4bFVqeVZvZVVBUklZUlcyeVA2dW9QbGhuTmY0bndZL1hjTXhIWFhSSU1ya0NKY1NkSmp5ZjdzQXUKWTcyT0ptQ01LVFpYaUo0a2UzWEZoTG5ZTktUT2hqcEFUTlpUYWdqUUZiUWZPMFJHYTVGOXNhVmk2aUdyZUVpOQpHcGUybm1PTDFDNUkrTmVETS9QMDdVcWh4STA1WVRGdlR3aUJqYVJSaTVGTlkxS1prV3h2Z01SVG5tN2pRN0txCkZIb3pDcVhFcnRRdHJFUmRJZ0p5cFFxUzA1L2tWbTBBU1VqdmJod0NXTi9ZNW1RL2lMdC8rMDFVUTNML1dLVVkKOERXRW1JMUYyNGFpeW80ZklDbldYbzdybXRMd2IyTVhCSDFpdUZBUWRUZTNVbGpQRWpxZS8zSU5NL3U1WWdvSwpkclpsZ3k0NTdNL1phSkZnUHVqNHpnPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", "admin-key": "LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV3QUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktvd2dnU21BZ0VBQW9JQkFRRGh1WmRGbXRwZ1hrSHMKSnR0NVViT0pzRGZXcE5OR2hYMDkzK2pLVGRQV040ZCtnVXlpVUU4YjR2dWFpNHJnUThYb3lvblB5eE9EMnlGYwo0L3RFcDN2N245ZnhKY1dnNHN1VGNmN1M0YzNsNjkwY0hXVVRDUWxjczhRT1lTQkxoSzcvQ3hPQ1pTdkoybUt6Cm5YQWRnQU9NS1YxZmttNVVGMTQ1b0hqWmMxcHpzYU9iUUpQMWw2dW54Nk5KWlA5UXoycHo0ZU1CaEQzVlJwUFkKNjFqYllXdVFiNzNSOTFOdHM2RzJGMDFqdUJ3dzhpQ1RjWmdPUzlJVmR3S2h3SUpiMnFyK3hqK2h6aGxLT29HaQphVXRHUWVGLzZMTW1NeHUySUEvcEQ4dGVBbldsYXZnWVFDRVRiM3p5Znk0Q3JpWlVpNUViZ1J6WW5KUHBMVzVoCloyVU5OT0N2QWdNQkFBRUNnZ0VCQUxTUlRHdUg1cTAwZWJPNW1kUkJBVWhoeVNsUmpSbGE2OGMxZVBsc3JNc3UKZkpzOFA5dVFOdGh1Q1N1bCtTQlIxdDNVTjg0Yis2NTgrd0ZkOXNFQzRmRjErcjZUb3JhQzlxc3FnOWlhZmFVeAp1eDBrZnBodG9DbWlQSEhzVEhOYzNzaHpTZFlEckxXNmx6NXI4U1lJSTg1WndMSVZaMkNwbU55c2ZFQW5RUEpPClpQWHZDR1VQUXhIeldsVUFMdjVKNTljbXhtRzBMQ0NQaXN3K3FaOUVUVjlmbXBBZEltZVFsb0JXT1RjSE1kbUgKVzVTVC9qNFBua1dLVzltS0hHMTdKZkJVK3BiQUh5RlN4ZXdFK1VrNG91V1BBbGw2NStpSEFPL1BFMzErUzFHcQprR2dMR2JyeGpPeHpwMHpTZkVhZXBCTnpTeThJNi9WSmI5eThuaE1RdHBFQ2dZRUEveFpMZVZ6OEFUTFVLL3NBCkVycmh4RUtSc1M3MVhNbU5kQlNUNUZ2WkltN3NGWWY2TnE1NWRTQVJlWjlKeitFSEhJV0l4UEw4b3UraW4za0wKckh5NmJYVmoxWWlKbHRubzQwSm5TR1Rwa1ZNS1VobWFLUHBIcFRtamVkaFpZTnZ0S01vMUtFNmkzOGkrazZJYwphamRoM1B5Z2EwU2xaZmZCeVNDaEVNeEJwRk1DZ1lFQTRvaGxLTGV2aFJEc2tkdlF0cEVaZGFGVDlQVWdxTUFKCmdLcnV5RHZEOUNYQi9yaUdBSGgzZDFkamtKdnl1NGFJd05NU0RhVUt2Y3ZVM20yRTAwR1puZXZhSFJXTmtoWTYKQWY3aEVFeEkzMUVVL2VraFpOMGRPMHZDY0pOZmljUWo0ZndKOTRtNk5KUFIxT3hHYWJZbnBqa1MxdmlRN0UzRApLZ05mWTlVUFJyVUNnWUVBdjlkOG02R2F4U3Y4OWJhbnIyNXRMTFVEN1VWcnNvVENwWEowWjJITVEyRTRuRmpQCmpTQmFxQU9lSU9IekdTd2J0UStZZE1acHlnQi9SWUE3ajFXZXY0Tnh1YjRkL0IyV1VKUTRLUEg3MXhla3cyb04KenBPc0hrN2k2MWdTb0EzR3BzNnNlOHZvSW5WUEhxamt4ZDB3OUlucCs4Sjh3ZHM3R3BBM2tmS0x5NUVDZ1lFQQpqSndLNWw0MXRVdk8rNjVNZTdYTzdzUEhvQ216THV1Y09tQmZQS3hVTWl4cG9Oa2FYQnBxVlFwS1E4MTR1a3VFCmdPUEN0Zk4ybWw1MTcrWmM1dUEySU43dmtOUDYvdTJLN2dndDNQYnMyS2kwZGZLS0J0aDB0NFNMMldLd2o0dk8KbGd2aWgrTFdVS0pjUVdTelJCU0x3ZEoxVGpnVmhJZTBGMVRtbGNuKzVHMENnWUVBaVFjdnhTSXpnaHBib0RrcApqd2pGOCt5bS9qVlRpK0tuNEM4OHQrZDg0MnZUWUhmeHJQZXhHZzhaT0xCOWlobWpINjlhN0dxd2ExQWNGaG5SClo4bXN6VkI0MU93QjdVMjljZk5kaFQ2ajY3VTBJcElSY0JnRWY2V244WDIzTVh3cnZadXNOM3lBcWVyZzJXYnMKdUNaalZ6S2J6bWtxTTR0eFl5dk1ITGQrOEU0PQotLS0tLUVORCBQUklWQVRFIEtFWS0tLS0tCg==", "admin.jks": "/u3+7QAAAAIAAAACAAAAAQAMc3lzdGVtLmFkbWluAAABXI22pnAAAAUBMIIE/TAOBgorBgEEASoCEQEBBQAEggTpFWoGw/9gmjPp/XQVId2dpWHnvIqPNPId5rwdd7BGeuD4SgSRVzey+PveM2pLkIZziqhIJ1qefK8UaSPqzT+jfVfS8TDyhTISJWYaPDWaCLmq3YZrgTJvFA+x45hFJCCDUUxyTx3sZhJMJggI8yhGxQ6TTN0w4GuiaNnisEogZVjSZF99QRCtaXMcYX4+4S7ceSh/0Vt7t1hLL1Go3NqHto/H327BFHt4Xb+8HpdXc4CRjH3vT4ejxBIxednztEkewWpEHVdjKbwvI+ZUCr283jmNG6qfnptmL4TzwptQDPurVq9CF2DG5Xt5AXZ4M5+9VO/1BCR+lIavwT/UlrAx7dMxVTjZ2W/mHpjqrR3eBiHTTf+bwP5+LlEiLYlszRDI3RHkn8Ft/bmhoO+IBjiGZrmm/U+KsoTK+o87qihvETSfMWzY55l7QPbQmbOa9yi4c8aKSjlxH3T5GELFAL2eGj81EKs+tDPbx7Hipx3CM5g3OuS2XOmeFd6fcNZ5Xk7/h/UV0tZmTyLH3djD2BtdWfnqTfhjRbfirCTrUcHo87wSAhBVwF0sEJZQj7xsXssfVH1uHUz8wkz00t8Wpit9eRHmbWulhqUZ/fnI4KCAGaMM20EoWPA+Q0Vehl+LSrXiblaXq76ivXni9iDsDXdEgac5UkVowLId9yrt7rwVqh+vTOr5UNTe9q9ICFxfvyv/i7+xShLpKkaCZ9zxpsMe6qD35siah9D7aWQmITk/9sVR32IrF1RRFvrDKf/XpVS+9qXPRuSibil9CSKrxGcJcgXD16fnb7LuSChq9IjV+6lAi/x7tUEf5EFSSZ9PMk2KakUEpP4qxzdZynn5ys0FKMvzlbF3vRJl/7Y30LrlVn34v6YW5eSQ/YYQSobMFZ0BQAyXnboVpjpzTYuzbyJ10LeQYPbHP1WcasvL8Ccn3n99OI2krLeblSQc10VaCQYEsnhzupbqd1gt0j1vHW2ZD2X7S2wrS9vTb51neU0I4RT7vasEZFT1DFZVkQJQ+VMaw4AjdnHY5YgijPWYwid8tbwDI3t1OAf3aQQe6zQHfvEk1wQp33xrXW5AWCAnoGeG3fYTZaYj03dnFsVeC1E+E4cPM1phsQAx9ch5liRitzgGwHi9ROSu0eZcWYG40UeYlwhtoUaejRsMFeslFjsTxHBl15OpMYm/PbAG52sbizgtqX0B1dLTTEAahdcSfQkJwFWUOkBcSV4cfv0ij4DVAAlU/uHkIMFgPc2p9fntiBjVJiNiFw5YTGPT3nwGSckAY5V5oHIUd9YaU/A3uxN6CuwizY1G/UeQS2EoTBbvufAeSvQcu/j4EM3GYDA2l+Sl2svnVhLrgjqmauroXFk6FOGtAuwsqZrD/1gL1JN3KvclNmeQ78xH5ViJ15DVDBXaJBoZoJGY9/X5Psu45jur9AZZ9LCaLA1hZkTrcX7qKXgtEhzfAF/0SZfpED9O0f/M02uJPVPVZ+o57rfba0st98k4zzvxpfCZlyTFULQK6fE6B44Z0dcmQ0S1mqe9JQNRVwG5c4NCpDcjUnNeuDOc9iAJ8zgs5MV3AJFwr68OuofbGBdzOPfwQjGHN3Z1KTvaOkxli93MCV7vT2A0SHBtnXTposOu3OXvNG5ECWoC4JB8j7mO5gV4nHrZ+adaSv9L2t+wAXiRG0Z2AAAAAgAFWC41MDkAAANAMIIDPDCCAiSgAwIBAgIBBjANBgkqhkiG9w0BAQUFADAeMRwwGgYDVQQDExNsb2dnaW5nLXNpZ25lci10ZXN0MB4XDTE3MDYwOTE2MzYxNloXDTE5MDYwOTE2MzYxNlowPTEQMA4GA1UEChMHTG9nZ2luZzESMBAGA1UECxMJT3BlblNoaWZ0MRUwEwYDVQQDEwxzeXN0ZW0uYWRtaW4wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCw4XsC/H+0rdq9HUDICMJTrjJqXrsWwtlZN3mvH+EmXsOxKvuwH2RyMA3LyK/7xu7lN0CAhRoZLPqgNc82qN5aCVkgFMZjphul4HgI097IbgM4ireMyCuaKIz+nDxisvEk+tChsG0RXiajBdCX8tArd7tCUzL7XARsC4bHVdOzxHYUzAE4U1ZwMv6u9RWsSycjXcxUao6AIE/ciSNw7olniR+7cAw6w1i2M18LV//0w9pmI4F/FkU/b4anbI8n7PCsIP6KA463FWcWLcwf0rs1pPPp2zEx7O4o5wdqycjbuSjiDJDupfbadkiCtoLv6Cy7fI1lQawYmV7Wpx8AMc8hAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIFoDAJBgNVHRMEAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUiYWsMr6yDAfMPEdg0Vn2r0JgcaUwCQYDVR0jBAIwADANBgkqhkiG9w0BAQUFAAOCAQEAfr1S01tw0Nj42Q11fM+2zmNLQxIE7JhqMZKeyJl/qUuyLvrXHAFxKTHL3i03V5MA1cfiapBPZ0/6iKYOk5RJzNfufs8voJTEA7JLIevzmZj3xOk8dAMGu8iAcDGyjd2kcPO5hSOdNujPLWwf7x3+jczqlAkKe3bIcf6RJzMnZ4tf3WnH28n7CoZB95lI4OByQXepHdI6QGvIcVs+2I/xpzNMXcnaDGH4KN5wv0hRoyTLiKNlIbggKcnWeAltV9Tkgh2g1I/fYDYfsyV29uY0Suje/YqRwuy8U6HM5gqWruJQe5X9t9GPC+oVb7I6tBSKL7iUhua1MlDxHf8FLUb92wAFWC41MDkAAALeMIIC2jCCAcKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAeMRwwGgYDVQQDExNsb2dnaW5nLXNpZ25lci10ZXN0MB4XDTE3MDYwOTE2MzU1NloXDTIyMDYwODE2MzU1N1owHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMHD2UagfG5XB+Qbtdq78MtfQc1EY2ttZeWs4wGLhHECGUs0QidEsgI7rj5kfrH7kCMlfwv7m2FjX/X+4bI7h9BXacNLsdJRDMmHU/+s67A+5Mc5V1ISEjacLxnOffk53sTh4So6ZCkBi+p9jVlS7XBv/gWghiMhtEi/2hiB2lObjep++MeJT4BARF62794gWL7zIMskwkVabLJaInmkVIVK92j8GSiRCkEMqBI4VjD8BuQkOi7zGeleCtC14LMm5OmM2LwV/DFKichAF0nVLKDoSYCmfeHjSKyEUVNxwMejl3vCb4c94Qp+WZVEA+xPOuOjuxR+RAf7jtmYxuwvsC8CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFqfeMGdGUzYrpqHOpgO8ow1Dj/Fg09tnrwgUAFpwCfCYaywqPRXRgm0pURx+LYvRr5IRma/c28/8cmWWPnQxAG9F2YQgeNSlLY5f4ScIzash5P1fHBUQYIVS3dZpuArkGXNJrJK3lPGcHV6nYG+9eK7oQVKCkwR/Zs/CmEHsNLgOp6QHXOKHSqke2jp4vfHmSt/oDxTTUHR37x6q12M4haXX3cSX+abCQc1YQNdqLbdxXn6z8MxuZmFB9brgJEhScvFvTxzNtiRy3TsdeLQ8Z+Z1tpsbGSoJaOPRTr+wecs3tDLJnGXphg44w6Qm36vd4Cm8hi3z873lGR6eF/NY7UAAAACAAZzaWctY2EAAAFcjbal5wAFWC41MDkAAALeMIIC2jCCAcKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAeMRwwGgYDVQQDExNsb2dnaW5nLXNpZ25lci10ZXN0MB4XDTE3MDYwOTE2MzU1NloXDTIyMDYwODE2MzU1N1owHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMHD2UagfG5XB+Qbtdq78MtfQc1EY2ttZeWs4wGLhHECGUs0QidEsgI7rj5kfrH7kCMlfwv7m2FjX/X+4bI7h9BXacNLsdJRDMmHU/+s67A+5Mc5V1ISEjacLxnOffk53sTh4So6ZCkBi+p9jVlS7XBv/gWghiMhtEi/2hiB2lObjep++MeJT4BARF62794gWL7zIMskwkVabLJaInmkVIVK92j8GSiRCkEMqBI4VjD8BuQkOi7zGeleCtC14LMm5OmM2LwV/DFKichAF0nVLKDoSYCmfeHjSKyEUVNxwMejl3vCb4c94Qp+WZVEA+xPOuOjuxR+RAf7jtmYxuwvsC8CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFqfeMGdGUzYrpqHOpgO8ow1Dj/Fg09tnrwgUAFpwCfCYaywqPRXRgm0pURx+LYvRr5IRma/c28/8cmWWPnQxAG9F2YQgeNSlLY5f4ScIzash5P1fHBUQYIVS3dZpuArkGXNJrJK3lPGcHV6nYG+9eK7oQVKCkwR/Zs/CmEHsNLgOp6QHXOKHSqke2jp4vfHmSt/oDxTTUHR37x6q12M4haXX3cSX+abCQc1YQNdqLbdxXn6z8MxuZmFB9brgJEhScvFvTxzNtiRy3TsdeLQ8Z+Z1tpsbGSoJaOPRTr+wecs3tDLJnGXphg44w6Qm36vd4Cm8hi3z873lGR6eF/NY7WmWk7ewh4YG+hhBMHtqXNGd3cZJA==", "key": "/u3+7QAAAAIAAAACAAAAAQAKbG9nZ2luZy1lcwAAAVyNtrAkAAAFATCCBP0wDgYKKwYBBAEqAhEBAQUABIIE6T4M6y7EGaMU9r/rQAONU2aSSYJ4b0FzL3ccTsm/vQbJ11vf9tuocWrUzqJwLnE0iGnjE0/5K3ASPZbW5TrQA2upmJFArgigQ82zI/gQBrbKHQlAJRcrAv8YF6+GeVPhy+GKqmaYm8LrXuCcwRGlFmwOgPTD53mJqge6wyCyTlA/MzyvASYGxG74mnCpqrBT2zXYKaw3ZsSKwqlff1/tVGEZPXZqPozkQ9CEs+nCAdaBLxgpiUGigy2MOZ5EN78zpMXmcrrZfr/1szv9Qx3jqi3SENCsZjosaJQMEE/UlWrd7j2oJn5OTlutpw73aC2Iek2LpBCF0kCdOivdV6TE9bNMMCPeOR8XBO+LabcahGXgwdzudLqqT0Dahm2czHoSwaxARTtnFN1+GZMu3WPuTMuG5x+Qp3k1jQj9guRVEaDeCPgnlX13tkY/i1+eYYA4Jjr+K4O4xNy/0Ba+ITDi+429FcZqklsFUtnzL+iuI56O8I5CBgH34dmFtEw1IWxiZJHZuZXTkG+B7KC1w0EGhrVyCxIKvU45ir+0dYfpUUW6XzI0M8KShviMM6RYn6pxnNnQtcCUFgK5XrvaZLrfymur3jeF1C1xgtqCwNHjy1jgkqVZxXcwjpOO2x78CDOLBdOHE+ehw1I8KY+fivZyBPS8TNk1HufH1TgIjywSyQwTRPl/2htrSkV6Wude5EQucCZB7QLby09I8SbcgeXV7qns+q3t5bXwUrUB3z64wAtSwIxUW71jO4b4GP98NfVLI0TNARpH6AJeyHJ/bjMaNcX/spsnx4eyHllRyQmZvLazYwsZiL/4nVFPRoscqKonSX1onbR0jtls40nNFU32qQg3XedixG95uaIgwq8jvsmCYD8PZIUBJ9d+YeVB9AB5LQ1xlvJDUagH0UBPuRge2ED00j85Yu60yZA31F7eVbjUuxzVbm+WIUVGrko40t2aq8VM/Z3ceFJFug5jhZ2dzpEBdjsVtJs7fgFCiQqDu2TeLQhNzCuZU3aWZPiI9ROZ/3JfCbhCTeYH2PooqXasjoF5xYTRCDBVob8qX4AVKzEVZpsnz8tvBlqbbutZOE8Hc2w8n54pLTpObooNOCbqYMpfTG5+eq160mnkmlP5sNpuA2RgeYBoVPOOI8ROknMEZaHGCzqyCTXxKFxJ8J0higd0Z5z8c8eCHoyUCx5MQqj2S6QSao6a3Qh1YGqYAcgQYZMrj0AynDRH1GFDgu9PmSt2VWQE2yvSMNK77Zk1Ogb+B6v7xsdzb2b/suExrV+Zrwm4A57Jf/Z3l6A3PJt3iqB75erY/iunrUnXce/5rfy6IWId1g3xui/fJXjpMAz9CYJF7WgHt9MMPw+WPnUYNGjn40JaluoU6iDl/0HHTUJwJIPiFZN3I3/xfDOzxXDm9uz/CxFUyVUGKPdVuUyvfHEVCB/y8RCKXw/nS6ublSJ1l5DQT6uUQgOyu49Oyoncmbb4coGeMGbVK2O9pQCabe9i0SuYA5JOf/Y2K0KOduWNsG7qDEB+E51+3rnIc15tTcWcJbjnlK9I3ABrMmWpbcjGODVfl4whSjIpliwJ37RD3PZk8m17QjxgtQI6OIPRtFY9TDoud+w9mHE6ygy6KuOE12eFBxCvBuu7m8wYjmPyksU42MLp7udT2XxT2ZE4j35503BFP4IHcQAAAAIABVguNTA5AAAEXDCCBFgwggNAoAMCAQICAQgwDQYJKoZIhvcNAQEFBQAwHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDAeFw0xNzA2MDkxNjM2MThaFw0xOTA2MDkxNjM2MThaMDsxEDAOBgNVBAoTB0xvZ2dpbmcxEjAQBgNVBAsTCU9wZW5TaGlmdDETMBEGA1UEAxMKbG9nZ2luZy1lczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMDvxlSBHccqMizVQxqf1WN1cvIwAt2bUsup2E+sZHOS6Omzl1CiMuF5gPPDV4v1qVu41P0HPlx7Zzp1GyK/8CRFZXctM1xFy1BE77C/AgssAoLu0rL0//P0A1Yznex8nKdnWVMNh8y+iruPLfYrwFDDPY3WTtx/nc9WAstQkMtFQTTFHhR24NuDXxJvsJsvng2l5lJIQO2Rl2F48iCg30cKT22y4SgAWXospdfP1WKg3WptzW8jdbRKWROb2ChZoymKRQlwasuMAdrTX6X6TpwneBfNau4c5ZxDt0CacEgPf5+Tv1W4sGQ+IaP7pmum9OowvHfxBqUzVfFU3lXveAsCAwEAAaOCAYIwggF+MA4GA1UdDwEB/wQEAwIFoDAJBgNVHRMEAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUdbB/PitGIBk9jgbF5pMRkb1wT0owCQYDVR0jBAIwADCCARYGA1UdEQSCAQ0wggEJgglsb2NhbGhvc3SHBH8AAAGCCmxvZ2dpbmctZXOCJGxvZ2dpbmctZXMubG9nZ2luZy5zdmMuY2x1c3Rlci5sb2NhbIISbG9nZ2luZy1lcy1jbHVzdGVygixsb2dnaW5nLWVzLWNsdXN0ZXIubG9nZ2luZy5zdmMuY2x1c3Rlci5sb2NhbIIObG9nZ2luZy1lcy1vcHOCKGxvZ2dpbmctZXMtb3BzLmxvZ2dpbmcuc3ZjLmNsdXN0ZXIubG9jYWyCFmxvZ2dpbmctZXMtb3BzLWNsdXN0ZXKCMGxvZ2dpbmctZXMtb3BzLWNsdXN0ZXIubG9nZ2luZy5zdmMuY2x1c3Rlci5sb2NhbDANBgkqhkiG9w0BAQUFAAOCAQEAn8yafDwYeMqRQmWugKI2KDmtc5fXJnphjw1luhIzOY9Mr0nv+oiN4TT/uKvF7Vf7GC/pEZLSoEvfYJdNkBnhHWx3FrodqC6Wl3p//kNb2y4ZHpZCoPYuu+g7QoiahcN6cZEZu7kCDR7H6nQxn2x/OzLH3MkWswJ6YCYGr8ApXNLf8/3pye84/3WCQJsSXmb8HbwtUupU/Ya7zWDwhPgdzsDUni4t76D1j18R5JJl26QT7KRSSasZ4m4pF1dzN7lB/BkCCgVksTv3UI/hGRyCbH5TWE7x1qVP4snQqEl4ZKhIHgxob+LFMgZAmjRbKDgceQpK+Z+o8xHjDI0aOOC/9wAFWC41MDkAAALeMIIC2jCCAcKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAeMRwwGgYDVQQDExNsb2dnaW5nLXNpZ25lci10ZXN0MB4XDTE3MDYwOTE2MzU1NloXDTIyMDYwODE2MzU1N1owHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMHD2UagfG5XB+Qbtdq78MtfQc1EY2ttZeWs4wGLhHECGUs0QidEsgI7rj5kfrH7kCMlfwv7m2FjX/X+4bI7h9BXacNLsdJRDMmHU/+s67A+5Mc5V1ISEjacLxnOffk53sTh4So6ZCkBi+p9jVlS7XBv/gWghiMhtEi/2hiB2lObjep++MeJT4BARF62794gWL7zIMskwkVabLJaInmkVIVK92j8GSiRCkEMqBI4VjD8BuQkOi7zGeleCtC14LMm5OmM2LwV/DFKichAF0nVLKDoSYCmfeHjSKyEUVNxwMejl3vCb4c94Qp+WZVEA+xPOuOjuxR+RAf7jtmYxuwvsC8CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFqfeMGdGUzYrpqHOpgO8ow1Dj/Fg09tnrwgUAFpwCfCYaywqPRXRgm0pURx+LYvRr5IRma/c28/8cmWWPnQxAG9F2YQgeNSlLY5f4ScIzash5P1fHBUQYIVS3dZpuArkGXNJrJK3lPGcHV6nYG+9eK7oQVKCkwR/Zs/CmEHsNLgOp6QHXOKHSqke2jp4vfHmSt/oDxTTUHR37x6q12M4haXX3cSX+abCQc1YQNdqLbdxXn6z8MxuZmFB9brgJEhScvFvTxzNtiRy3TsdeLQ8Z+Z1tpsbGSoJaOPRTr+wecs3tDLJnGXphg44w6Qm36vd4Cm8hi3z873lGR6eF/NY7UAAAACAAZzaWctY2EAAAFcjbavkwAFWC41MDkAAALeMIIC2jCCAcKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAeMRwwGgYDVQQDExNsb2dnaW5nLXNpZ25lci10ZXN0MB4XDTE3MDYwOTE2MzU1NloXDTIyMDYwODE2MzU1N1owHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMHD2UagfG5XB+Qbtdq78MtfQc1EY2ttZeWs4wGLhHECGUs0QidEsgI7rj5kfrH7kCMlfwv7m2FjX/X+4bI7h9BXacNLsdJRDMmHU/+s67A+5Mc5V1ISEjacLxnOffk53sTh4So6ZCkBi+p9jVlS7XBv/gWghiMhtEi/2hiB2lObjep++MeJT4BARF62794gWL7zIMskwkVabLJaInmkVIVK92j8GSiRCkEMqBI4VjD8BuQkOi7zGeleCtC14LMm5OmM2LwV/DFKichAF0nVLKDoSYCmfeHjSKyEUVNxwMejl3vCb4c94Qp+WZVEA+xPOuOjuxR+RAf7jtmYxuwvsC8CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFqfeMGdGUzYrpqHOpgO8ow1Dj/Fg09tnrwgUAFpwCfCYaywqPRXRgm0pURx+LYvRr5IRma/c28/8cmWWPnQxAG9F2YQgeNSlLY5f4ScIzash5P1fHBUQYIVS3dZpuArkGXNJrJK3lPGcHV6nYG+9eK7oQVKCkwR/Zs/CmEHsNLgOp6QHXOKHSqke2jp4vfHmSt/oDxTTUHR37x6q12M4haXX3cSX+abCQc1YQNdqLbdxXn6z8MxuZmFB9brgJEhScvFvTxzNtiRy3TsdeLQ8Z+Z1tpsbGSoJaOPRTr+wecs3tDLJnGXphg44w6Qm36vd4Cm8hi3z873lGR6eF/NY7VxM82Jdwsq8Z8Rvx02AxrDkzJnFg==", "searchguard.key": "/u3+7QAAAAIAAAACAAAAAQANZWxhc3RpY3NlYXJjaAAAAVyNtqraAAAFATCCBP0wDgYKKwYBBAEqAhEBAQUABIIE6e8onPYwoACGzptF1jQbtfGP1f0jBb76pJfaAqEAgf4BjJX04GpJFLNNnL+leNdWhGGn1CWBIXOTbwjG+vP/1qd3jTxA/jgm1/M6eysmUMOkuilEiytJSGBHUYVRqX7SOoryXKtqhpdEskuKsTHse1UCpfPDTPy0iAVaqt7SV2ZKMCjJGLSDOPq8f2V7Fg+vpWEe+8wDIT68Clhw07mXJ0PrRbmv9ADPYKoiicR5EYui2mh+K7NrfhesUsUmuD8IPwpsOHY177GvAnm1kmbCBhGYJSLVBLB7D5QQTJKqAHTa0TzIx5wMHqCLqqfc4ZspIlN54MLnVxp3W5jmstZ13Y30lnBBlnoQOPjYWX1iTXWreUc4tRaRRG7tTezoiVaT4EX46H2Qv2cO8GVBLM9J7+2umZ4+tRNtIYSoiakpFIUkFSNfPBFWWPX4LLNfzetReByXXduRYx5fzJtWy3SlweUdfUzIDxJ/5JCUEpVwE+AGMegfrt947kSsp55h2jVPYZ/Czv7lM+NmJOvv3dz+mv5cmpy/vi3E05hTHluijp3/8qLSic7MLLftJYKTkJKU9gbOOj8M7PISpaWr8mdZ9wuGPXkjey8yI5k5zMNuL5YQlJJWJkjIPkzLXJ6L7tGlH8c2GoMWRaqmu3h+yHrLsGhRUbLWAxzNrrYm4W+OM2UDhXhagun9I7qC9RQFoC8/gsOFuraI59L4KV7thef8z+CDr4up2yHukwzy8jeHXBw+y74Bm7FhNJTCRmTkcrvUHWUUCucQGeNWKCq8jglTnBKfbfZMpT7RFrKv87Pwsqj9LHacUMD29pWuxeKI13/vGNO0ZfTbb2duHe1ppzQrqrvKdJfo0mABCUDcDn1fs6FbHSxQQTjQrRdv/jGGTftw5wLCLr2klqaK1/v3phy8NAQODerafAkstJs3l9w6z4epGaxw0bgQNtmXMsoCYa2an85556rBS0H1oNwjw1HW90SaQ8enoQLY4F35VMgYBhe/kYLPmBPi4j/5H3FRPdh20Sz3/W+eSIv53o1j9/53YpeE3ySS5Zdkg+9DW2o0jcOgcADGgpjLKO61ssoHcA2GyckOWxp1DSH+B7Jk47fyCYr+lmp3VVjlNHVbdlD/k39+cebevMu9cy+BTcmrN8BvbiQG6ahiC7g6SMqfTNwMhmrYWpTve8Vzara7CMpaSvSMzX/Vy+FX48Zrt1MGD55iXuOUmVxVe3AYgljIyMTTRwMauew7FDVNo9+Gev2pDJbCswlFkaD8wj4qAu2l+WlT7bPcgHyOa+6bNatR5aYOj0HidKcBDb20FFOhqazMKGvENkfYdCdzveYvwMuqfekv+NezVfOY96nh1TmF44sngykqaQUwcCOb054ze8eGDq+Q7OzRB3KczkOtZSGY6LaeMhMxGGyJdfK0bvMX552hsNgmuYLLyg45ybFx9HB5GJw+niyMQ0ZNJP7vtmRDFzPHJg58NAvf7MjS6dveMG4Ir4TwUa2+LbPjgApIXdSYtC1ozac+pUmYWrgvPwhiouydS38eJCbu5YrZfMHHqLtZ7PU1rWAtTo1cF31UWhzsjdmGKDGgm/O3D1YkrB1OHhC0LORlC9GMz+cNB6CGBuYXeIqSTvvDqKaprTK4kwIDYUdfjGkteamWwFvDh1iStOonx6/dkJDke4JcGQAAAAIABVguNTA5AAADgjCCA34wggJmoAMCAQICAQcwDQYJKoZIhvcNAQEFBQAwHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDAeFw0xNzA2MDkxNjM2MTdaFw0xOTA2MDkxNjM2MTdaMD4xEDAOBgNVBAoTB0xvZ2dpbmcxEjAQBgNVBAsTCU9wZW5TaGlmdDEWMBQGA1UEAxMNZWxhc3RpY3NlYXJjaDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJYJkHT5baO+0eDTEWmG6ByFwdyMWsjyLewX9/aRNhX5XI8KbReKHrO2BZSkZfTXGDJNxoFeO6blpoOuxddu2Ox1C4ZCG/xafKwA1sQJF7NTL6AMZlr4EtJrZRv4oT60z+uXs0LJU+zNDnKZE/ATKtrgNlrnVmSI42PUy5SVfg82iCYzoFz80g3H+MMFrwJBATuJqHxmQmx7M6rvdI/Fat+ii7HHU2lhDAMINX2ODdwr2qf2NGSw88L/lDSNpgsWowZKeZyEqsy71y5ArG0fMACujq7rgGsClFNOuSPQIKq/xc1dtr0sSY3lfZ1mjCBG3yS8XDvr9EFOF/QtV7uaookCAwEAAaOBpjCBozAOBgNVHQ8BAf8EBAMCBaAwCQYDVR0TBAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHQYDVR0OBBYEFK5+BPjUjdxTCH+wqTFP6jW283YOMAkGA1UdIwQCMAAwPQYDVR0RBDYwNIIJbG9jYWxob3N0hwR/AAABggpsb2dnaW5nLWVzgg5sb2dnaW5nLWVzLW9wc4gFKgMEBQUwDQYJKoZIhvcNAQEFBQADggEBABOa9YOcP4oN3v1PTMgWugOVlFj+cFFJLTQdkakxGc736sTDwzJ0z5mlBLElyyu+uIlFcaXNEp4UbNdJGxOsjyeCpX51ss0hxzd45E7BjmEeYvEAgmW3W4sNOQIvx8uEZe1oxgPGMkBJ+2EoyKIynZbSkP88MKSi3dLldZ3mD/f0N3QyM5gf4mrka7620xl79Jd6cWBRtU19yR/tJ+i/kzTKwx7L6cApP6jlFjnmdr/oLa+r7obRUaJI0u6pFrt90MlblQBrv3ZaTltIGrUOrsTtPtsMFyWhWF50/341SlCh9BMor8Ty8lX0eUSdKbQU9ZoSL6lwA0rspJEge1wUXzoABVguNTA5AAAC3jCCAtowggHCoAMCAQICAQEwDQYJKoZIhvcNAQELBQAwHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDAeFw0xNzA2MDkxNjM1NTZaFw0yMjA2MDgxNjM1NTdaMB4xHDAaBgNVBAMTE2xvZ2dpbmctc2lnbmVyLXRlc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDBw9lGoHxuVwfkG7Xau/DLX0HNRGNrbWXlrOMBi4RxAhlLNEInRLICO64+ZH6x+5AjJX8L+5thY1/1/uGyO4fQV2nDS7HSUQzJh1P/rOuwPuTHOVdSEhI2nC8Zzn35Od7E4eEqOmQpAYvqfY1ZUu1wb/4FoIYjIbRIv9oYgdpTm43qfvjHiU+AQERetu/eIFi+8yDLJMJFWmyyWiJ5pFSFSvdo/BkokQpBDKgSOFYw/AbkJDou8xnpXgrQteCzJuTpjNi8FfwxSonIQBdJ1Syg6EmApn3h40ishFFTccDHo5d7wm+HPeEKflmVRAPsTzrjo7sUfkQH+47ZmMbsL7AvAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBan3jBnRlM2K6ahzqYDvKMNQ4/xYNPbZ68IFABacAnwmGssKj0V0YJtKVEcfi2L0a+SEZmv3NvP/HJllj50MQBvRdmEIHjUpS2OX+EnCM2rIeT9XxwVEGCFUt3WabgK5BlzSaySt5TxnB1ep2BvvXiu6EFSgpMEf2bPwphB7DS4DqekB1zih0qpHto6eL3x5krf6A8U01B0d+8eqtdjOIWl193El/mmwkHNWEDXai23cV5+s/DMbmZhQfW64CRIUnLxb08czbYkct07HXi0PGfmdbabGxkqCWjj0U6/sHnLN7QyyZxl6YYOOMOkJt+r3eApvIYt8/O95RkenhfzWO1AAAAAgAGc2lnLWNhAAABXI22qk4ABVguNTA5AAAC3jCCAtowggHCoAMCAQICAQEwDQYJKoZIhvcNAQELBQAwHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDAeFw0xNzA2MDkxNjM1NTZaFw0yMjA2MDgxNjM1NTdaMB4xHDAaBgNVBAMTE2xvZ2dpbmctc2lnbmVyLXRlc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDBw9lGoHxuVwfkG7Xau/DLX0HNRGNrbWXlrOMBi4RxAhlLNEInRLICO64+ZH6x+5AjJX8L+5thY1/1/uGyO4fQV2nDS7HSUQzJh1P/rOuwPuTHOVdSEhI2nC8Zzn35Od7E4eEqOmQpAYvqfY1ZUu1wb/4FoIYjIbRIv9oYgdpTm43qfvjHiU+AQERetu/eIFi+8yDLJMJFWmyyWiJ5pFSFSvdo/BkokQpBDKgSOFYw/AbkJDou8xnpXgrQteCzJuTpjNi8FfwxSonIQBdJ1Syg6EmApn3h40ishFFTccDHo5d7wm+HPeEKflmVRAPsTzrjo7sUfkQH+47ZmMbsL7AvAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBan3jBnRlM2K6ahzqYDvKMNQ4/xYNPbZ68IFABacAnwmGssKj0V0YJtKVEcfi2L0a+SEZmv3NvP/HJllj50MQBvRdmEIHjUpS2OX+EnCM2rIeT9XxwVEGCFUt3WabgK5BlzSaySt5TxnB1ep2BvvXiu6EFSgpMEf2bPwphB7DS4DqekB1zih0qpHto6eL3x5krf6A8U01B0d+8eqtdjOIWl193El/mmwkHNWEDXai23cV5+s/DMbmZhQfW64CRIUnLxb08czbYkct07HXi0PGfmdbabGxkqCWjj0U6/sHnLN7QyyZxl6YYOOMOkJt+r3eApvIYt8/O95RkenhfzWO1NzgO3d2Y2584ZKgOgjbteBhGeoY=", "searchguard.truststore": "/u3+7QAAAAIAAAABAAAAAgAGc2lnLWNhAAABXI22sKsABVguNTA5AAAC3jCCAtowggHCoAMCAQICAQEwDQYJKoZIhvcNAQELBQAwHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDAeFw0xNzA2MDkxNjM1NTZaFw0yMjA2MDgxNjM1NTdaMB4xHDAaBgNVBAMTE2xvZ2dpbmctc2lnbmVyLXRlc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDBw9lGoHxuVwfkG7Xau/DLX0HNRGNrbWXlrOMBi4RxAhlLNEInRLICO64+ZH6x+5AjJX8L+5thY1/1/uGyO4fQV2nDS7HSUQzJh1P/rOuwPuTHOVdSEhI2nC8Zzn35Od7E4eEqOmQpAYvqfY1ZUu1wb/4FoIYjIbRIv9oYgdpTm43qfvjHiU+AQERetu/eIFi+8yDLJMJFWmyyWiJ5pFSFSvdo/BkokQpBDKgSOFYw/AbkJDou8xnpXgrQteCzJuTpjNi8FfwxSonIQBdJ1Syg6EmApn3h40ishFFTccDHo5d7wm+HPeEKflmVRAPsTzrjo7sUfkQH+47ZmMbsL7AvAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBan3jBnRlM2K6ahzqYDvKMNQ4/xYNPbZ68IFABacAnwmGssKj0V0YJtKVEcfi2L0a+SEZmv3NvP/HJllj50MQBvRdmEIHjUpS2OX+EnCM2rIeT9XxwVEGCFUt3WabgK5BlzSaySt5TxnB1ep2BvvXiu6EFSgpMEf2bPwphB7DS4DqekB1zih0qpHto6eL3x5krf6A8U01B0d+8eqtdjOIWl193El/mmwkHNWEDXai23cV5+s/DMbmZhQfW64CRIUnLxb08czbYkct07HXi0PGfmdbabGxkqCWjj0U6/sHnLN7QyyZxl6YYOOMOkJt+r3eApvIYt8/O95RkenhfzWO1xvbvwAyNbPnCdKUcTgcDPPmVMC4=", "truststore": "/u3+7QAAAAIAAAABAAAAAgAGc2lnLWNhAAABXI22sKsABVguNTA5AAAC3jCCAtowggHCoAMCAQICAQEwDQYJKoZIhvcNAQELBQAwHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDAeFw0xNzA2MDkxNjM1NTZaFw0yMjA2MDgxNjM1NTdaMB4xHDAaBgNVBAMTE2xvZ2dpbmctc2lnbmVyLXRlc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDBw9lGoHxuVwfkG7Xau/DLX0HNRGNrbWXlrOMBi4RxAhlLNEInRLICO64+ZH6x+5AjJX8L+5thY1/1/uGyO4fQV2nDS7HSUQzJh1P/rOuwPuTHOVdSEhI2nC8Zzn35Od7E4eEqOmQpAYvqfY1ZUu1wb/4FoIYjIbRIv9oYgdpTm43qfvjHiU+AQERetu/eIFi+8yDLJMJFWmyyWiJ5pFSFSvdo/BkokQpBDKgSOFYw/AbkJDou8xnpXgrQteCzJuTpjNi8FfwxSonIQBdJ1Syg6EmApn3h40ishFFTccDHo5d7wm+HPeEKflmVRAPsTzrjo7sUfkQH+47ZmMbsL7AvAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBan3jBnRlM2K6ahzqYDvKMNQ4/xYNPbZ68IFABacAnwmGssKj0V0YJtKVEcfi2L0a+SEZmv3NvP/HJllj50MQBvRdmEIHjUpS2OX+EnCM2rIeT9XxwVEGCFUt3WabgK5BlzSaySt5TxnB1ep2BvvXiu6EFSgpMEf2bPwphB7DS4DqekB1zih0qpHto6eL3x5krf6A8U01B0d+8eqtdjOIWl193El/mmwkHNWEDXai23cV5+s/DMbmZhQfW64CRIUnLxb08czbYkct07HXi0PGfmdbabGxkqCWjj0U6/sHnLN7QyyZxl6YYOOMOkJt+r3eApvIYt8/O95RkenhfzWO1xvbvwAyNbPnCdKUcTgcDPPmVMC4=" }, "kind": "Secret", "metadata": { "creationTimestamp": null, "name": "logging-elasticsearch" }, "type": "Opaque" }, "state": "present" } TASK [openshift_logging_elasticsearch : Set logging-es-ops-cluster service] **** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:168 changed: [openshift] => { "changed": true, "results": { "clusterip": "172.30.230.239", "cmd": "/bin/oc get service logging-es-ops-cluster -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "Service", "metadata": { "creationTimestamp": "2017-06-09T16:36:38Z", "name": "logging-es-ops-cluster", "namespace": "logging", "resourceVersion": "1344", "selfLink": "/api/v1/namespaces/logging/services/logging-es-ops-cluster", "uid": "cf26058a-4d31-11e7-8061-0ed679898f5a" }, "spec": { "clusterIP": "172.30.230.239", "ports": [ { "port": 9300, "protocol": "TCP", "targetPort": 9300 } ], "selector": { "component": "es-ops", "provider": "openshift" }, "sessionAffinity": "None", "type": "ClusterIP" }, "status": { "loadBalancer": {} } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Set logging-es-ops service] ************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:182 changed: [openshift] => { "changed": true, "results": { "clusterip": "172.30.189.53", "cmd": "/bin/oc get service logging-es-ops -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "Service", "metadata": { "creationTimestamp": "2017-06-09T16:36:39Z", "name": "logging-es-ops", "namespace": "logging", "resourceVersion": "1349", "selfLink": "/api/v1/namespaces/logging/services/logging-es-ops", "uid": "cffc21c5-4d31-11e7-8061-0ed679898f5a" }, "spec": { "clusterIP": "172.30.189.53", "ports": [ { "port": 9200, "protocol": "TCP", "targetPort": "restapi" } ], "selector": { "component": "es-ops", "provider": "openshift" }, "sessionAffinity": "None", "type": "ClusterIP" }, "status": { "loadBalancer": {} } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Creating ES storage template] ********** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:197 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Creating ES storage template] ********** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:210 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Set ES storage] ************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:225 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:237 ok: [openshift] => { "ansible_facts": { "es_deploy_name": "logging-es-ops-data-master-kq3j3au5" }, "changed": false } TASK [openshift_logging_elasticsearch : set_fact] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:241 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_elasticsearch : Set ES dc templates] ******************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:246 changed: [openshift] => { "changed": true, "checksum": "131ba08134e422b1fbcc9486e491f713f4552a76", "dest": "/tmp/openshift-logging-ansible-VAlbdx/templates/logging-es-dc.yml", "gid": 0, "group": "root", "md5sum": "e19bfc3f1ca82187aa209b6c31f18261", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 3178, "src": "/root/.ansible/tmp/ansible-tmp-1497026200.2-251318336304069/source", "state": "file", "uid": 0 } TASK [openshift_logging_elasticsearch : Set ES dc] ***************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:262 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get dc logging-es-ops-data-master-kq3j3au5 -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "DeploymentConfig", "metadata": { "creationTimestamp": "2017-06-09T16:36:41Z", "generation": 2, "labels": { "component": "es-ops", "deployment": "logging-es-ops-data-master-kq3j3au5", "logging-infra": "elasticsearch", "provider": "openshift" }, "name": "logging-es-ops-data-master-kq3j3au5", "namespace": "logging", "resourceVersion": "1363", "selfLink": "/oapi/v1/namespaces/logging/deploymentconfigs/logging-es-ops-data-master-kq3j3au5", "uid": "d0f1cd7c-4d31-11e7-8061-0ed679898f5a" }, "spec": { "replicas": 1, "selector": { "component": "es-ops", "deployment": "logging-es-ops-data-master-kq3j3au5", "logging-infra": "elasticsearch", "provider": "openshift" }, "strategy": { "activeDeadlineSeconds": 21600, "recreateParams": { "timeoutSeconds": 600 }, "resources": {}, "type": "Recreate" }, "template": { "metadata": { "creationTimestamp": null, "labels": { "component": "es-ops", "deployment": "logging-es-ops-data-master-kq3j3au5", "logging-infra": "elasticsearch", "provider": "openshift" }, "name": "logging-es-ops-data-master-kq3j3au5" }, "spec": { "containers": [ { "env": [ { "name": "NAMESPACE", "valueFrom": { "fieldRef": { "apiVersion": "v1", "fieldPath": "metadata.namespace" } } }, { "name": "KUBERNETES_TRUST_CERT", "value": "true" }, { "name": "SERVICE_DNS", "value": "logging-es-ops-cluster" }, { "name": "CLUSTER_NAME", "value": "logging-es-ops" }, { "name": "INSTANCE_RAM", "value": "8Gi" }, { "name": "NODE_QUORUM", "value": "1" }, { "name": "RECOVER_EXPECTED_NODES", "value": "1" }, { "name": "RECOVER_AFTER_TIME", "value": "5m" }, { "name": "READINESS_PROBE_TIMEOUT", "value": "30" }, { "name": "IS_MASTER", "value": "true" }, { "name": "HAS_DATA", "value": "true" } ], "image": "172.30.180.71:5000/logging/logging-elasticsearch:latest", "imagePullPolicy": "Always", "name": "elasticsearch", "ports": [ { "containerPort": 9200, "name": "restapi", "protocol": "TCP" }, { "containerPort": 9300, "name": "cluster", "protocol": "TCP" } ], "readinessProbe": { "exec": { "command": [ "/usr/share/elasticsearch/probe/readiness.sh" ] }, "failureThreshold": 3, "initialDelaySeconds": 10, "periodSeconds": 5, "successThreshold": 1, "timeoutSeconds": 30 }, "resources": { "limits": { "cpu": "1", "memory": "8Gi" }, "requests": { "memory": "512Mi" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/etc/elasticsearch/secret", "name": "elasticsearch", "readOnly": true }, { "mountPath": "/usr/share/java/elasticsearch/config", "name": "elasticsearch-config", "readOnly": true }, { "mountPath": "/elasticsearch/persistent", "name": "elasticsearch-storage" } ] } ], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": { "supplementalGroups": [ 65534 ] }, "serviceAccount": "aggregated-logging-elasticsearch", "serviceAccountName": "aggregated-logging-elasticsearch", "terminationGracePeriodSeconds": 30, "volumes": [ { "name": "elasticsearch", "secret": { "defaultMode": 420, "secretName": "logging-elasticsearch" } }, { "configMap": { "defaultMode": 420, "name": "logging-elasticsearch" }, "name": "elasticsearch-config" }, { "emptyDir": {}, "name": "elasticsearch-storage" } ] } }, "test": false, "triggers": [ { "type": "ConfigChange" } ] }, "status": { "availableReplicas": 0, "conditions": [ { "lastTransitionTime": "2017-06-09T16:36:41Z", "lastUpdateTime": "2017-06-09T16:36:41Z", "message": "Deployment config does not have minimum availability.", "status": "False", "type": "Available" }, { "lastTransitionTime": "2017-06-09T16:36:41Z", "lastUpdateTime": "2017-06-09T16:36:41Z", "message": "replication controller \"logging-es-ops-data-master-kq3j3au5-1\" is waiting for pod \"logging-es-ops-data-master-kq3j3au5-1-deploy\" to run", "status": "Unknown", "type": "Progressing" } ], "details": { "causes": [ { "type": "ConfigChange" } ], "message": "config change" }, "latestVersion": 1, "observedGeneration": 2, "replicas": 0, "unavailableReplicas": 0, "updatedReplicas": 0 } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_elasticsearch : Delete temp directory] ***************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_elasticsearch/tasks/main.yaml:274 ok: [openshift] => { "changed": false, "path": "/tmp/openshift-logging-ansible-VAlbdx", "state": "absent" } TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:151 statically included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml TASK [openshift_logging_kibana : fail] ***************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml:3 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml:7 ok: [openshift] => { "ansible_facts": { "kibana_version": "3_5" }, "changed": false } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : fail] ***************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml:15 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : Create temp directory for doing work in] ****** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:7 ok: [openshift] => { "changed": false, "cmd": [ "mktemp", "-d", "/tmp/openshift-logging-ansible-XXXXXX" ], "delta": "0:00:00.002301", "end": "2017-06-09 12:36:42.610534", "rc": 0, "start": "2017-06-09 12:36:42.608233" } STDOUT: /tmp/openshift-logging-ansible-LosCgQ TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:12 ok: [openshift] => { "ansible_facts": { "tempdir": "/tmp/openshift-logging-ansible-LosCgQ" }, "changed": false } TASK [openshift_logging_kibana : Create templates subdirectory] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:16 ok: [openshift] => { "changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/tmp/openshift-logging-ansible-LosCgQ/templates", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 6, "state": "directory", "uid": 0 } TASK [openshift_logging_kibana : Create Kibana service account] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:26 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : Create Kibana service account] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:34 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get sa aggregated-logging-kibana -o json -n logging", "results": [ { "apiVersion": "v1", "imagePullSecrets": [ { "name": "aggregated-logging-kibana-dockercfg-z98xg" } ], "kind": "ServiceAccount", "metadata": { "creationTimestamp": "2017-06-09T16:36:43Z", "name": "aggregated-logging-kibana", "namespace": "logging", "resourceVersion": "1374", "selfLink": "/api/v1/namespaces/logging/serviceaccounts/aggregated-logging-kibana", "uid": "d25aca93-4d31-11e7-8061-0ed679898f5a" }, "secrets": [ { "name": "aggregated-logging-kibana-dockercfg-z98xg" }, { "name": "aggregated-logging-kibana-token-k6wbc" } ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:42 ok: [openshift] => { "ansible_facts": { "kibana_component": "kibana", "kibana_name": "logging-kibana" }, "changed": false } TASK [openshift_logging_kibana : Checking for session_secret] ****************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:47 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging_kibana : Checking for oauth_secret] ******************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:51 ok: [openshift] => { "changed": false, "stat": { "exists": false } } TASK [openshift_logging_kibana : Generate session secret] ********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:56 changed: [openshift] => { "changed": true, "checksum": "dedc90ba17550e4b3c37f18c2023ffdbdf02ce71", "dest": "/etc/origin/logging/session_secret", "gid": 0, "group": "root", "md5sum": "158064f037a7e80f847c37337f139707", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 200, "src": "/root/.ansible/tmp/ansible-tmp-1497026204.45-204817268509874/source", "state": "file", "uid": 0 } TASK [openshift_logging_kibana : Generate oauth secret] ************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:64 changed: [openshift] => { "changed": true, "checksum": "2cbf634d9888a78465dfdf839adf753d69aa3ac5", "dest": "/etc/origin/logging/oauth_secret", "gid": 0, "group": "root", "md5sum": "d105f0fb627e409074114f10bc5a50c7", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 64, "src": "/root/.ansible/tmp/ansible-tmp-1497026204.8-48198521171991/source", "state": "file", "uid": 0 } TASK [openshift_logging_kibana : Retrieving the cert to use when generating secrets for the logging components] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:71 ok: [openshift] => (item={u'name': u'ca_file', u'file': u'ca.crt'}) => { "changed": false, "content": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMyakNDQWNLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpVMU5sb1hEVEl5TURZd09ERTJNelUxTjFvdwpIakVjTUJvR0ExVUVBeE1UYkc5bloybHVaeTF6YVdkdVpYSXRkR1Z6ZERDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1IRDJVYWdmRzVYQitRYnRkcTc4TXRmUWMxRVkydHRaZVdzNHdHTGhIRUMKR1VzMFFpZEVzZ0k3cmo1a2ZySDdrQ01sZnd2N20yRmpYL1grNGJJN2g5QlhhY05Mc2RKUkRNbUhVLytzNjdBKwo1TWM1VjFJU0VqYWNMeG5PZmZrNTNzVGg0U282WkNrQmkrcDlqVmxTN1hCdi9nV2doaU1odEVpLzJoaUIybE9iCmplcCsrTWVKVDRCQVJGNjI3OTRnV0w3eklNc2t3a1ZhYkxKYUlubWtWSVZLOTJqOEdTaVJDa0VNcUJJNFZqRDgKQnVRa09pN3pHZWxlQ3RDMTRMTW01T21NMkx3Vi9ERktpY2hBRjBuVkxLRG9TWUNtZmVIalNLeUVVVk54d01lagpsM3ZDYjRjOTRRcCtXWlZFQSt4UE91T2p1eFIrUkFmN2p0bVl4dXd2c0M4Q0F3RUFBYU1qTUNFd0RnWURWUjBQCkFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGcWYKZU1HZEdVellycHFIT3BnTzhvdzFEai9GZzA5dG5yd2dVQUZwd0NmQ1lheXdxUFJYUmdtMHBVUngrTFl2UnI1SQpSbWEvYzI4LzhjbVdXUG5ReEFHOUYyWVFnZU5TbExZNWY0U2NJemFzaDVQMWZIQlVRWUlWUzNkWnB1QXJrR1hOCkpySkszbFBHY0hWNm5ZRys5ZUs3b1FWS0Nrd1IvWnMvQ21FSHNOTGdPcDZRSFhPS0hTcWtlMmpwNHZmSG1TdC8Kb0R4VFRVSFIzN3g2cTEyTTRoYVhYM2NTWCthYkNRYzFZUU5kcUxiZHhYbjZ6OE14dVptRkI5YnJnSkVoU2N2Rgp2VHh6TnRpUnkzVHNkZUxROForWjF0cHNiR1NvSmFPUFJUcit3ZWNzM3RETEpuR1hwaGc0NHc2UW0zNnZkNENtCjhoaTN6ODczbEdSNmVGL05ZN1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K", "encoding": "base64", "item": { "file": "ca.crt", "name": "ca_file" }, "source": "/etc/origin/logging/ca.crt" } ok: [openshift] => (item={u'name': u'kibana_internal_key', u'file': u'kibana-internal.key'}) => { "changed": false, "content": "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBcDRrNEdrazE4UnRMdWROa1BDemZsSjA1YXFta0VuQndiNEd0RzREeEcxS3I1cG8vClg1dENkaG1va2VGcGlFajE0Q0p0MFQ0a2wvKytVczdvNTFWampYUU5UbVBCVWZXckJJaW5YOHVpQ2RvRCtqVk0KbU1JbVFKUkIzdUxNK1R0UFpMeFladDVNbytnV3F6OVo5RG00TXJ2M1VTaVFpSng5RklYQWs5WW9selFpV0ZhdgpYMW1XSE5uTldQOXBVbW1aQW8rdHhPWGVCSGp3RjNRUFZDMjFobzRmVGkxek1Bb0xadXNUUXNiZFhsWS82K2lhCkZGc09scWN5d09jblQ5MUxmL1c3c1BrVHZiYXBVVFRmTE12dWpGWGJyT0FzTmY0RHg0LzRwTzhIcGRxMXZlU3EKWk9YZDF6MGZMdFFYRmpYM3gvQVR5dklPRlJxSndyM3NidzllZFFJREFRQUJBb0lCQUZ6eHJXUTMxV3NyWDNsNgpJeE9YT1hwek54VkxnN0NQSEhGZ0dhODJlNHhTOHZiOS84YUVsZng3QTUvYTAvYzlzdHBqZjFVa0NwMGd6bStXClJuT0FWYmgxOTZQVGg3NTJjaWdUdi9kUE8xZ0gwVWtRSkt4aW1XSWVYZVRQSWpQcWlZTXpVai9QZCtrK3FzVGEKWW9jeE5sVS9IbGV5RDIwYWsvRzh1amZLQ2I3SDBabi9kYkhIUFdFdGJVQ0ZiRGp6QTRCdXBZWUdva1NvdHBUZQovdk8wRTJrbXVGZi9oNXhHdHZMd3JwMGFxQ29PRjVPNlNRd0Y2a0NvdTBaU24raFNiQlU4Umltc2xnZFpXWE9iCjVNaWYvZ21WUGIwY1h3VkhjRzVyL1lCVXBEaWwvSGdYRG1GdmxKTU9VdVNFWmlMSktjcklhU0Zzek83Q0k2T2QKU3luU2NVa0NnWUVBMVc1a0ErMzhpNDBoanBmcXpCRFhoS1V4SjY2cDByazczT0NNNkl4c2N0WXJZc215YUdVcQpTRDFxNGRCNC9sK2JKOU5GbDhxdFlrL1FCZjJBUVVxOHd3RzJJMTBMYnZZL21PNDczVjdza2o4dXhXZnJKMnFjClhYNk5FV05HdFlxQmRQbVNzaW9yRU5ZcXJyQ2RVdTd2TUQ1bE9oK2RsV1BydDNiOUU4bExVYk1DZ1lFQXlQTjEKdzAyc01HVzA5Z0dMUS9qMFJYeStTRzRwbjE5KzlQUnNobXZxOHpOVndaVC8reEJsUUxpbUU4NFZPVDRrSHlONwpJd2JXYk1NVXFIdGZUbjAwR3NSblFUT2xpNFFrZnBuaU83K3NGU2xESkxxeWpNWkVNUkRyUmlQVWwxcWF0cFB2CjZYNzQ0NW9ESllONHVQcFlYVEJpKy9YbFJxbEdkbkNIaHJ6WmF6Y0NnWUVBbTNkUHp6VDVzc1ZZM2o5MnZySzkKNmxkU0dPKzNzRkZSYzdEMXN5Y21qL2ZGbnE0QkVONHdFSEw4ZWdTYlBkOXlhMjRQMFdQSExFZXE1ZElrTEdzKwpXR1hVd0ttei9GM1BNWnpIYXBjTWgvTVNKS1Z2dTBTREplOWEzb0dNTGd6NEE3Y1B6a3U5RndTS0RybUR2N2RmCjc2cGpuaFVOZXBaUVMxR1hRMlJyUXRVQ2dZRUF4dmhoWXpjMlJsbCtXNU5kekxqbFRTUmQyS1BUYksyR3JJNlUKSW1rYW95SHlTSkg3QUNIVXo4ZkZFWmtkWktlRjVwc2dJcWR6aXcrQ0ZlektpcEhvRkpUaDFDcUtRT0ZmclVjNwpQWVBrcExOTmUwR3M1enl0YXBQaEVYc0sxYTV0d0ZnczFkcldINjV3YlNXSFZFajFVNWhFZWt5WWIyWE9raG1MCjU3cDM0UE1DZ1lFQWg2a3NLUWtaTmQ1czh1RG12YW9PZ1NrblVoRmVacDI2YTM3OVdUdGM2TWNOZ21hWWN5UnQKcU5NdUY1bWp6b2tYSXJOTUxYQ2kwcUl1cmhsa0N6YmV4TXJTOFFzQ2xqdUYxRzJjZTlMNGVTNnRpTjZoaTE2aQpNSk1zN2NaWTVtME43VzIyUWNOdDFNeDZpb2hnRkpwRkJPMW9GVU92V09uZmNKbHBPbjZQV1VrPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=", "encoding": "base64", "item": { "file": "kibana-internal.key", "name": "kibana_internal_key" }, "source": "/etc/origin/logging/kibana-internal.key" } ok: [openshift] => (item={u'name': u'kibana_internal_cert', u'file': u'kibana-internal.crt'}) => { "changed": false, "content": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUakNDQWphZ0F3SUJBZ0lCQWpBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpVMU9Gb1hEVEU1TURZd09URTJNelUxT1ZvdwpGakVVTUJJR0ExVUVBeE1MSUd0cFltRnVZUzF2Y0hNd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3CmdnRUtBb0lCQVFDbmlUZ2FTVFh4RzB1NTAyUThMTitVblRscXFhUVNjSEJ2Z2EwYmdQRWJVcXZtbWo5Zm0wSjIKR2FpUjRXbUlTUFhnSW0zUlBpU1gvNzVTenVqblZXT05kQTFPWThGUjlhc0VpS2RmeTZJSjJnUDZOVXlZd2laQQpsRUhlNHN6NU8wOWt2RmhtM2t5ajZCYXJQMW4wT2JneXUvZFJLSkNJbkgwVWhjQ1QxaWlYTkNKWVZxOWZXWlljCjJjMVkvMmxTYVprQ2o2M0U1ZDRFZVBBWGRBOVVMYldHamg5T0xYTXdDZ3RtNnhOQ3h0MWVWai9yNkpvVVd3NlcKcHpMQTV5ZFAzVXQvOWJ1dytSTzl0cWxSTk44c3krNk1WZHVzNEN3MS9nUEhqL2lrN3dlbDJyVzk1S3BrNWQzWApQUjh1MUJjV05mZkg4QlBLOGc0VkdvbkN2ZXh2RDE1MUFnTUJBQUdqZ1o0d2dac3dEZ1lEVlIwUEFRSC9CQVFECkFnV2dNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUZCd01CTUF3R0ExVWRFd0VCL3dRQ01BQXdaZ1lEVlIwUkJGOHcKWFlJTElHdHBZbUZ1WVMxdmNIT0NMQ0JyYVdKaGJtRXRiM0J6TG5KdmRYUmxjaTVrWldaaGRXeDBMbk4yWXk1agpiSFZ6ZEdWeUxteHZZMkZzZ2hnZ2EybGlZVzVoTGpFeU55NHdMakF1TVM1NGFYQXVhVytDQm10cFltRnVZVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQXV1eFZ3S3QrcDNCdVpWQVd4VktmZGUwM1BVWksxRm0xZnZ5c0xIOWMKL0dJcnB6UC8rVlR6YUQydlpWbGtQd2FzRDkzbVRoNGFiQ1htZWtyZld6YmZjdmNhTFBwVHYvc2hISUVsczNNcwpyT05USGJlSmxyK3dMTkJMREFVdSt6NFZTaGlJcEhlQTIzWW5wUnByMlcyUnhoQ0hzNGZlUGlyanl4OEFiRklCCjZUVUxWZC9jWDhwZHVVME9BR3dYVGIzN1Zma1dQbHBKNFdYeGMvQWFxWmxnVEZWWVFZY2VDT3B6SHp6bzJsWlQKMkgzYzRCYnhrekpybVVMU09qV1BTdmhiV2FkTkV6L3BOaDdINnB3cm1jT0ZQRVl2RWQ0NmNoNjV6MkNnMjJwcAppRWZKWmxvTWl4d3FUYjVzQkxrSVdaa05hbkVUdjYwa2JGYzd1Uis2Z2V1ck5nPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJQzJqQ0NBY0tnQXdJQkFnSUJBVEFOQmdrcWhraUc5dzBCQVFzRkFEQWVNUnd3R2dZRFZRUURFeE5zYjJkbgphVzVuTFhOcFoyNWxjaTEwWlhOME1CNFhEVEUzTURZd09URTJNelUxTmxvWERUSXlNRFl3T0RFMk16VTFOMW93CkhqRWNNQm9HQTFVRUF4TVRiRzluWjJsdVp5MXphV2R1WlhJdGRHVnpkRENDQVNJd0RRWUpLb1pJaHZjTkFRRUIKQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUhEMlVhZ2ZHNVhCK1FidGRxNzhNdGZRYzFFWTJ0dFplV3M0d0dMaEhFQwpHVXMwUWlkRXNnSTdyajVrZnJIN2tDTWxmd3Y3bTJGalgvWCs0Ykk3aDlCWGFjTkxzZEpSRE1tSFUvK3M2N0ErCjVNYzVWMUlTRWphY0x4bk9mZms1M3NUaDRTbzZaQ2tCaStwOWpWbFM3WEJ2L2dXZ2hpTWh0RWkvMmhpQjJsT2IKamVwKytNZUpUNEJBUkY2Mjc5NGdXTDd6SU1za3drVmFiTEphSW5ta1ZJVks5Mmo4R1NpUkNrRU1xQkk0VmpEOApCdVFrT2k3ekdlbGVDdEMxNExNbTVPbU0yTHdWL0RGS2ljaEFGMG5WTEtEb1NZQ21mZUhqU0t5RVVWTnh3TWVqCmwzdkNiNGM5NFFwK1daVkVBK3hQT3VPanV4UitSQWY3anRtWXh1d3ZzQzhDQXdFQUFhTWpNQ0V3RGdZRFZSMFAKQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUZxZgplTUdkR1V6WXJwcUhPcGdPOG93MURqL0ZnMDl0bnJ3Z1VBRnB3Q2ZDWWF5d3FQUlhSZ20wcFVSeCtMWXZScjVJClJtYS9jMjgvOGNtV1dQblF4QUc5RjJZUWdlTlNsTFk1ZjRTY0l6YXNoNVAxZkhCVVFZSVZTM2RacHVBcmtHWE4KSnJKSzNsUEdjSFY2bllHKzllSzdvUVZLQ2t3Ui9acy9DbUVIc05MZ09wNlFIWE9LSFNxa2UyanA0dmZIbVN0LwpvRHhUVFVIUjM3eDZxMTJNNGhhWFgzY1NYK2FiQ1FjMVlRTmRxTGJkeFhuNno4TXh1Wm1GQjlicmdKRWhTY3ZGCnZUeHpOdGlSeTNUc2RlTFE4WitaMXRwc2JHU29KYU9QUlRyK3dlY3MzdERMSm5HWHBoZzQ0dzZRbTM2dmQ0Q20KOGhpM3o4NzNsR1I2ZUYvTlk3VT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", "encoding": "base64", "item": { "file": "kibana-internal.crt", "name": "kibana_internal_cert" }, "source": "/etc/origin/logging/kibana-internal.crt" } ok: [openshift] => (item={u'name': u'server_tls', u'file': u'server-tls.json'}) => { "changed": false, "content": "Ly8gU2VlIGZvciBhdmFpbGFibGUgb3B0aW9uczogaHR0cHM6Ly9ub2RlanMub3JnL2FwaS90bHMuaHRtbCN0bHNfdGxzX2NyZWF0ZXNlcnZlcl9vcHRpb25zX3NlY3VyZWNvbm5lY3Rpb25saXN0ZW5lcgp0bHNfb3B0aW9ucyA9IHsKCWNpcGhlcnM6ICdrRUVDREg6K2tFRUNESCtTSEE6a0VESDora0VESCtTSEE6K2tFREgrQ0FNRUxMSUE6a0VDREg6K2tFQ0RIK1NIQTprUlNBOitrUlNBK1NIQTora1JTQStDQU1FTExJQTohYU5VTEw6IWVOVUxMOiFTU0x2MjohUkM0OiFERVM6IUVYUDohU0VFRDohSURFQTorM0RFUycsCglob25vckNpcGhlck9yZGVyOiB0cnVlCn0K", "encoding": "base64", "item": { "file": "server-tls.json", "name": "server_tls" }, "source": "/etc/origin/logging/server-tls.json" } ok: [openshift] => (item={u'name': u'session_secret', u'file': u'session_secret'}) => { "changed": false, "content": "dWJ3bzJDNUxGVXdjMU03SkVmcExhWDJCWG15bHBtaWthMzhEaERyRnJPaXNJY2FOZ2JLaE9YZlQ5eTEzZE1HZXVKa0RMR1d5WjBlVVhseEZzdG4xZlRSdzhvS3pxV2dFYkhhN0NTeDVQMk0yZWNwdWduNWdDSVk4THJCb2xqM1hkdzhSdDk5Q1RUc3l3Vmo5TVNUdE1FRlA5QVk4YVV4Sk1SV1dOelZoYWpEcEtvcFJ3RmZXT2VTOTg0VzJVSUlBZWNsUEVsUnE=", "encoding": "base64", "item": { "file": "session_secret", "name": "session_secret" }, "source": "/etc/origin/logging/session_secret" } ok: [openshift] => (item={u'name': u'oauth_secret', u'file': u'oauth_secret'}) => { "changed": false, "content": "RDBhTVcwUTVoQXBwbnB0N0gyQ1NkWFJzVlBqelFJNnRmVjVSMVpmMWZycUhkeVRZckFLZ3djZk92a0lRNlA2TA==", "encoding": "base64", "item": { "file": "oauth_secret", "name": "oauth_secret" }, "source": "/etc/origin/logging/oauth_secret" } TASK [openshift_logging_kibana : Set logging-kibana service] ******************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:84 changed: [openshift] => { "changed": true, "results": { "clusterip": "172.30.59.132", "cmd": "/bin/oc get service logging-kibana -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "Service", "metadata": { "creationTimestamp": "2017-06-09T16:36:46Z", "name": "logging-kibana", "namespace": "logging", "resourceVersion": "1393", "selfLink": "/api/v1/namespaces/logging/services/logging-kibana", "uid": "d44436bd-4d31-11e7-8061-0ed679898f5a" }, "spec": { "clusterIP": "172.30.59.132", "ports": [ { "port": 443, "protocol": "TCP", "targetPort": "oaproxy" } ], "selector": { "component": "kibana", "provider": "openshift" }, "sessionAffinity": "None", "type": "ClusterIP" }, "status": { "loadBalancer": {} } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:101 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ openshift_logging_kibana_key | trim | length > 0 }} skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:106 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ openshift_logging_kibana_cert | trim | length > 0 }} skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:111 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ openshift_logging_kibana_ca | trim | length > 0 }} skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:116 ok: [openshift] => { "ansible_facts": { "kibana_ca": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMyakNDQWNLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpVMU5sb1hEVEl5TURZd09ERTJNelUxTjFvdwpIakVjTUJvR0ExVUVBeE1UYkc5bloybHVaeTF6YVdkdVpYSXRkR1Z6ZERDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1IRDJVYWdmRzVYQitRYnRkcTc4TXRmUWMxRVkydHRaZVdzNHdHTGhIRUMKR1VzMFFpZEVzZ0k3cmo1a2ZySDdrQ01sZnd2N20yRmpYL1grNGJJN2g5QlhhY05Mc2RKUkRNbUhVLytzNjdBKwo1TWM1VjFJU0VqYWNMeG5PZmZrNTNzVGg0U282WkNrQmkrcDlqVmxTN1hCdi9nV2doaU1odEVpLzJoaUIybE9iCmplcCsrTWVKVDRCQVJGNjI3OTRnV0w3eklNc2t3a1ZhYkxKYUlubWtWSVZLOTJqOEdTaVJDa0VNcUJJNFZqRDgKQnVRa09pN3pHZWxlQ3RDMTRMTW01T21NMkx3Vi9ERktpY2hBRjBuVkxLRG9TWUNtZmVIalNLeUVVVk54d01lagpsM3ZDYjRjOTRRcCtXWlZFQSt4UE91T2p1eFIrUkFmN2p0bVl4dXd2c0M4Q0F3RUFBYU1qTUNFd0RnWURWUjBQCkFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGcWYKZU1HZEdVellycHFIT3BnTzhvdzFEai9GZzA5dG5yd2dVQUZwd0NmQ1lheXdxUFJYUmdtMHBVUngrTFl2UnI1SQpSbWEvYzI4LzhjbVdXUG5ReEFHOUYyWVFnZU5TbExZNWY0U2NJemFzaDVQMWZIQlVRWUlWUzNkWnB1QXJrR1hOCkpySkszbFBHY0hWNm5ZRys5ZUs3b1FWS0Nrd1IvWnMvQ21FSHNOTGdPcDZRSFhPS0hTcWtlMmpwNHZmSG1TdC8Kb0R4VFRVSFIzN3g2cTEyTTRoYVhYM2NTWCthYkNRYzFZUU5kcUxiZHhYbjZ6OE14dVptRkI5YnJnSkVoU2N2Rgp2VHh6TnRpUnkzVHNkZUxROForWjF0cHNiR1NvSmFPUFJUcit3ZWNzM3RETEpuR1hwaGc0NHc2UW0zNnZkNENtCjhoaTN6ODczbEdSNmVGL05ZN1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K" }, "changed": false } TASK [openshift_logging_kibana : Generating Kibana route template] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:121 ok: [openshift] => { "changed": false, "checksum": "dcf08b6882eca4f3d8b51b4eedd30966374a3e8c", "dest": "/tmp/openshift-logging-ansible-LosCgQ/templates/kibana-route.yaml", "gid": 0, "group": "root", "md5sum": "a47cff0434381011c01e654fa1970cb5", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 2714, "src": "/root/.ansible/tmp/ansible-tmp-1497026207.39-212770588717126/source", "state": "file", "uid": 0 } TASK [openshift_logging_kibana : Setting Kibana route] ************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:141 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get route logging-kibana -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "Route", "metadata": { "creationTimestamp": "2017-06-09T16:36:48Z", "labels": { "component": "support", "logging-infra": "support", "provider": "openshift" }, "name": "logging-kibana", "namespace": "logging", "resourceVersion": "1399", "selfLink": "/oapi/v1/namespaces/logging/routes/logging-kibana", "uid": "d5291bf6-4d31-11e7-8061-0ed679898f5a" }, "spec": { "host": "kibana.router.default.svc.cluster.local", "tls": { "caCertificate": "-----BEGIN CERTIFICATE-----\nMIIC2jCCAcKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAeMRwwGgYDVQQDExNsb2dn\naW5nLXNpZ25lci10ZXN0MB4XDTE3MDYwOTE2MzU1NloXDTIyMDYwODE2MzU1N1ow\nHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAMHD2UagfG5XB+Qbtdq78MtfQc1EY2ttZeWs4wGLhHEC\nGUs0QidEsgI7rj5kfrH7kCMlfwv7m2FjX/X+4bI7h9BXacNLsdJRDMmHU/+s67A+\n5Mc5V1ISEjacLxnOffk53sTh4So6ZCkBi+p9jVlS7XBv/gWghiMhtEi/2hiB2lOb\njep++MeJT4BARF62794gWL7zIMskwkVabLJaInmkVIVK92j8GSiRCkEMqBI4VjD8\nBuQkOi7zGeleCtC14LMm5OmM2LwV/DFKichAF0nVLKDoSYCmfeHjSKyEUVNxwMej\nl3vCb4c94Qp+WZVEA+xPOuOjuxR+RAf7jtmYxuwvsC8CAwEAAaMjMCEwDgYDVR0P\nAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFqf\neMGdGUzYrpqHOpgO8ow1Dj/Fg09tnrwgUAFpwCfCYaywqPRXRgm0pURx+LYvRr5I\nRma/c28/8cmWWPnQxAG9F2YQgeNSlLY5f4ScIzash5P1fHBUQYIVS3dZpuArkGXN\nJrJK3lPGcHV6nYG+9eK7oQVKCkwR/Zs/CmEHsNLgOp6QHXOKHSqke2jp4vfHmSt/\noDxTTUHR37x6q12M4haXX3cSX+abCQc1YQNdqLbdxXn6z8MxuZmFB9brgJEhScvF\nvTxzNtiRy3TsdeLQ8Z+Z1tpsbGSoJaOPRTr+wecs3tDLJnGXphg44w6Qm36vd4Cm\n8hi3z873lGR6eF/NY7U=\n-----END CERTIFICATE-----\n", "destinationCACertificate": "-----BEGIN CERTIFICATE-----\nMIIC2jCCAcKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAeMRwwGgYDVQQDExNsb2dn\naW5nLXNpZ25lci10ZXN0MB4XDTE3MDYwOTE2MzU1NloXDTIyMDYwODE2MzU1N1ow\nHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAMHD2UagfG5XB+Qbtdq78MtfQc1EY2ttZeWs4wGLhHEC\nGUs0QidEsgI7rj5kfrH7kCMlfwv7m2FjX/X+4bI7h9BXacNLsdJRDMmHU/+s67A+\n5Mc5V1ISEjacLxnOffk53sTh4So6ZCkBi+p9jVlS7XBv/gWghiMhtEi/2hiB2lOb\njep++MeJT4BARF62794gWL7zIMskwkVabLJaInmkVIVK92j8GSiRCkEMqBI4VjD8\nBuQkOi7zGeleCtC14LMm5OmM2LwV/DFKichAF0nVLKDoSYCmfeHjSKyEUVNxwMej\nl3vCb4c94Qp+WZVEA+xPOuOjuxR+RAf7jtmYxuwvsC8CAwEAAaMjMCEwDgYDVR0P\nAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFqf\neMGdGUzYrpqHOpgO8ow1Dj/Fg09tnrwgUAFpwCfCYaywqPRXRgm0pURx+LYvRr5I\nRma/c28/8cmWWPnQxAG9F2YQgeNSlLY5f4ScIzash5P1fHBUQYIVS3dZpuArkGXN\nJrJK3lPGcHV6nYG+9eK7oQVKCkwR/Zs/CmEHsNLgOp6QHXOKHSqke2jp4vfHmSt/\noDxTTUHR37x6q12M4haXX3cSX+abCQc1YQNdqLbdxXn6z8MxuZmFB9brgJEhScvF\nvTxzNtiRy3TsdeLQ8Z+Z1tpsbGSoJaOPRTr+wecs3tDLJnGXphg44w6Qm36vd4Cm\n8hi3z873lGR6eF/NY7U=\n-----END CERTIFICATE-----\n", "insecureEdgeTerminationPolicy": "Redirect", "termination": "reencrypt" }, "to": { "kind": "Service", "name": "logging-kibana", "weight": 100 }, "wildcardPolicy": "None" }, "status": { "ingress": [ { "conditions": [ { "lastTransitionTime": "2017-06-09T16:36:48Z", "status": "True", "type": "Admitted" } ], "host": "kibana.router.default.svc.cluster.local", "routerName": "router", "wildcardPolicy": "None" } ] } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : Get current oauthclient hostnames] ************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:151 ok: [openshift] => { "changed": false, "results": { "cmd": "/bin/oc get oauthclient kibana-proxy -o json -n logging", "results": [ {} ], "returncode": 0, "stderr": "Error from server (NotFound): oauthclients.oauth.openshift.io \"kibana-proxy\" not found\n", "stdout": "" }, "state": "list" } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:159 ok: [openshift] => { "ansible_facts": { "proxy_hostnames": [ "https://kibana.router.default.svc.cluster.local" ] }, "changed": false } TASK [openshift_logging_kibana : Create oauth-client template] ***************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:162 changed: [openshift] => { "changed": true, "checksum": "45f21b897c8bf1555ea0474ba1d7d4cdc5bcf061", "dest": "/tmp/openshift-logging-ansible-LosCgQ/templates/oauth-client.yml", "gid": 0, "group": "root", "md5sum": "e83ab0637a6663a493301cd00b6559f0", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 328, "src": "/root/.ansible/tmp/ansible-tmp-1497026209.14-226082743168640/source", "state": "file", "uid": 0 } TASK [openshift_logging_kibana : Set kibana-proxy oauth-client] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:170 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get oauthclient kibana-proxy -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "OAuthClient", "metadata": { "creationTimestamp": "2017-06-09T16:36:50Z", "labels": { "logging-infra": "support" }, "name": "kibana-proxy", "resourceVersion": "1405", "selfLink": "/oapi/v1/oauthclients/kibana-proxy", "uid": "d637f4f0-4d31-11e7-8061-0ed679898f5a" }, "redirectURIs": [ "https://kibana.router.default.svc.cluster.local" ], "scopeRestrictions": [ { "literals": [ "user:info", "user:check-access", "user:list-projects" ] } ], "secret": "D0aMW0Q5hAppnpt7H2CSdXRsVPjzQI6tfV5R1Zf1frqHdyTYrAKgwcfOvkIQ6P6L" } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : Set Kibana secret] **************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:181 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc secrets new logging-kibana ca=/etc/origin/logging/ca.crt key=/etc/origin/logging/system.logging.kibana.key cert=/etc/origin/logging/system.logging.kibana.crt -n logging", "results": "", "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : Set Kibana Proxy secret] ********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:195 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc secrets new logging-kibana-proxy oauth-secret=/tmp/oauth-secret-gNto6g session-secret=/tmp/session-secret-MyNyWB server-key=/tmp/server-key-KOiGsN server-cert=/tmp/server-cert-O99Ciy server-tls.json=/tmp/server-tls.json-au4RmW -n logging", "results": "", "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : Generate Kibana DC template] ****************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:221 changed: [openshift] => { "changed": true, "checksum": "e53b4339febe77a8072436827cf7c7054629050a", "dest": "/tmp/openshift-logging-ansible-LosCgQ/templates/kibana-dc.yaml", "gid": 0, "group": "root", "md5sum": "79b8f886df51a07c3b43053ce64373e7", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 3739, "src": "/root/.ansible/tmp/ansible-tmp-1497026212.31-135901078042696/source", "state": "file", "uid": 0 } TASK [openshift_logging_kibana : Set Kibana DC] ******************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:240 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get dc logging-kibana -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "DeploymentConfig", "metadata": { "creationTimestamp": "2017-06-09T16:36:53Z", "generation": 2, "labels": { "component": "kibana", "logging-infra": "kibana", "provider": "openshift" }, "name": "logging-kibana", "namespace": "logging", "resourceVersion": "1420", "selfLink": "/oapi/v1/namespaces/logging/deploymentconfigs/logging-kibana", "uid": "d83757a9-4d31-11e7-8061-0ed679898f5a" }, "spec": { "replicas": 1, "selector": { "component": "kibana", "logging-infra": "kibana", "provider": "openshift" }, "strategy": { "activeDeadlineSeconds": 21600, "resources": {}, "rollingParams": { "intervalSeconds": 1, "maxSurge": "25%", "maxUnavailable": "25%", "timeoutSeconds": 600, "updatePeriodSeconds": 1 }, "type": "Rolling" }, "template": { "metadata": { "creationTimestamp": null, "labels": { "component": "kibana", "logging-infra": "kibana", "provider": "openshift" }, "name": "logging-kibana" }, "spec": { "containers": [ { "env": [ { "name": "ES_HOST", "value": "logging-es" }, { "name": "ES_PORT", "value": "9200" }, { "name": "KIBANA_MEMORY_LIMIT", "valueFrom": { "resourceFieldRef": { "containerName": "kibana", "divisor": "0", "resource": "limits.memory" } } } ], "image": "172.30.180.71:5000/logging/logging-kibana:latest", "imagePullPolicy": "Always", "name": "kibana", "readinessProbe": { "exec": { "command": [ "/usr/share/kibana/probe/readiness.sh" ] }, "failureThreshold": 3, "initialDelaySeconds": 5, "periodSeconds": 5, "successThreshold": 1, "timeoutSeconds": 4 }, "resources": { "limits": { "memory": "736Mi" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/etc/kibana/keys", "name": "kibana", "readOnly": true } ] }, { "env": [ { "name": "OAP_BACKEND_URL", "value": "http://localhost:5601" }, { "name": "OAP_AUTH_MODE", "value": "oauth2" }, { "name": "OAP_TRANSFORM", "value": "user_header,token_header" }, { "name": "OAP_OAUTH_ID", "value": "kibana-proxy" }, { "name": "OAP_MASTER_URL", "value": "https://kubernetes.default.svc.cluster.local" }, { "name": "OAP_PUBLIC_MASTER_URL", "value": "https://172.18.12.108:8443" }, { "name": "OAP_LOGOUT_REDIRECT", "value": "https://172.18.12.108:8443/console/logout" }, { "name": "OAP_MASTER_CA_FILE", "value": "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" }, { "name": "OAP_DEBUG", "value": "False" }, { "name": "OAP_OAUTH_SECRET_FILE", "value": "/secret/oauth-secret" }, { "name": "OAP_SERVER_CERT_FILE", "value": "/secret/server-cert" }, { "name": "OAP_SERVER_KEY_FILE", "value": "/secret/server-key" }, { "name": "OAP_SERVER_TLS_FILE", "value": "/secret/server-tls.json" }, { "name": "OAP_SESSION_SECRET_FILE", "value": "/secret/session-secret" }, { "name": "OCP_AUTH_PROXY_MEMORY_LIMIT", "valueFrom": { "resourceFieldRef": { "containerName": "kibana-proxy", "divisor": "0", "resource": "limits.memory" } } } ], "image": "172.30.180.71:5000/logging/logging-auth-proxy:latest", "imagePullPolicy": "Always", "name": "kibana-proxy", "ports": [ { "containerPort": 3000, "name": "oaproxy", "protocol": "TCP" } ], "resources": { "limits": { "memory": "96Mi" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/secret", "name": "kibana-proxy", "readOnly": true } ] } ], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "aggregated-logging-kibana", "serviceAccountName": "aggregated-logging-kibana", "terminationGracePeriodSeconds": 30, "volumes": [ { "name": "kibana", "secret": { "defaultMode": 420, "secretName": "logging-kibana" } }, { "name": "kibana-proxy", "secret": { "defaultMode": 420, "secretName": "logging-kibana-proxy" } } ] } }, "test": false, "triggers": [ { "type": "ConfigChange" } ] }, "status": { "availableReplicas": 0, "conditions": [ { "lastTransitionTime": "2017-06-09T16:36:53Z", "lastUpdateTime": "2017-06-09T16:36:53Z", "message": "Deployment config does not have minimum availability.", "status": "False", "type": "Available" }, { "lastTransitionTime": "2017-06-09T16:36:53Z", "lastUpdateTime": "2017-06-09T16:36:53Z", "message": "replication controller \"logging-kibana-1\" is waiting for pod \"logging-kibana-1-deploy\" to run", "status": "Unknown", "type": "Progressing" } ], "details": { "causes": [ { "type": "ConfigChange" } ], "message": "config change" }, "latestVersion": 1, "observedGeneration": 2, "replicas": 0, "unavailableReplicas": 0, "updatedReplicas": 0 } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : Delete temp directory] ************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:252 ok: [openshift] => { "changed": false, "path": "/tmp/openshift-logging-ansible-LosCgQ", "state": "absent" } TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:166 statically included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml TASK [openshift_logging_kibana : fail] ***************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml:3 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml:7 ok: [openshift] => { "ansible_facts": { "kibana_version": "3_5" }, "changed": false } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : fail] ***************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/determine_version.yaml:15 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : Create temp directory for doing work in] ****** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:7 ok: [openshift] => { "changed": false, "cmd": [ "mktemp", "-d", "/tmp/openshift-logging-ansible-XXXXXX" ], "delta": "0:00:00.008197", "end": "2017-06-09 12:36:56.004383", "rc": 0, "start": "2017-06-09 12:36:55.996186" } STDOUT: /tmp/openshift-logging-ansible-gvBLkA TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:12 ok: [openshift] => { "ansible_facts": { "tempdir": "/tmp/openshift-logging-ansible-gvBLkA" }, "changed": false } TASK [openshift_logging_kibana : Create templates subdirectory] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:16 ok: [openshift] => { "changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/tmp/openshift-logging-ansible-gvBLkA/templates", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 6, "state": "directory", "uid": 0 } TASK [openshift_logging_kibana : Create Kibana service account] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:26 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : Create Kibana service account] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:34 ok: [openshift] => { "changed": false, "results": { "cmd": "/bin/oc get sa aggregated-logging-kibana -o json -n logging", "results": [ { "apiVersion": "v1", "imagePullSecrets": [ { "name": "aggregated-logging-kibana-dockercfg-z98xg" } ], "kind": "ServiceAccount", "metadata": { "creationTimestamp": "2017-06-09T16:36:43Z", "name": "aggregated-logging-kibana", "namespace": "logging", "resourceVersion": "1374", "selfLink": "/api/v1/namespaces/logging/serviceaccounts/aggregated-logging-kibana", "uid": "d25aca93-4d31-11e7-8061-0ed679898f5a" }, "secrets": [ { "name": "aggregated-logging-kibana-dockercfg-z98xg" }, { "name": "aggregated-logging-kibana-token-k6wbc" } ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:42 ok: [openshift] => { "ansible_facts": { "kibana_component": "kibana-ops", "kibana_name": "logging-kibana-ops" }, "changed": false } TASK [openshift_logging_kibana : Checking for session_secret] ****************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:47 ok: [openshift] => { "changed": false, "stat": { "atime": 1497026205.6136134, "attr_flags": "", "attributes": [], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "checksum": "dedc90ba17550e4b3c37f18c2023ffdbdf02ce71", "ctime": 1497026204.6096144, "dev": 51714, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 118889939, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "md5": "158064f037a7e80f847c37337f139707", "mimetype": "text/plain", "mode": "0644", "mtime": 1497026204.4856143, "nlink": 1, "path": "/etc/origin/logging/session_secret", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 200, "uid": 0, "version": "1444090880", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false } } TASK [openshift_logging_kibana : Checking for oauth_secret] ******************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:51 ok: [openshift] => { "changed": false, "stat": { "atime": 1497026205.7216132, "attr_flags": "", "attributes": [], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "checksum": "2cbf634d9888a78465dfdf839adf753d69aa3ac5", "ctime": 1497026204.9326138, "dev": 51714, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 1905450, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "md5": "d105f0fb627e409074114f10bc5a50c7", "mimetype": "text/plain", "mode": "0644", "mtime": 1497026204.818614, "nlink": 1, "path": "/etc/origin/logging/oauth_secret", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 64, "uid": 0, "version": "18446744072498685517", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false } } TASK [openshift_logging_kibana : Generate session secret] ********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:56 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : Generate oauth secret] ************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:64 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : Retrieving the cert to use when generating secrets for the logging components] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:71 ok: [openshift] => (item={u'name': u'ca_file', u'file': u'ca.crt'}) => { "changed": false, "content": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMyakNDQWNLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpVMU5sb1hEVEl5TURZd09ERTJNelUxTjFvdwpIakVjTUJvR0ExVUVBeE1UYkc5bloybHVaeTF6YVdkdVpYSXRkR1Z6ZERDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1IRDJVYWdmRzVYQitRYnRkcTc4TXRmUWMxRVkydHRaZVdzNHdHTGhIRUMKR1VzMFFpZEVzZ0k3cmo1a2ZySDdrQ01sZnd2N20yRmpYL1grNGJJN2g5QlhhY05Mc2RKUkRNbUhVLytzNjdBKwo1TWM1VjFJU0VqYWNMeG5PZmZrNTNzVGg0U282WkNrQmkrcDlqVmxTN1hCdi9nV2doaU1odEVpLzJoaUIybE9iCmplcCsrTWVKVDRCQVJGNjI3OTRnV0w3eklNc2t3a1ZhYkxKYUlubWtWSVZLOTJqOEdTaVJDa0VNcUJJNFZqRDgKQnVRa09pN3pHZWxlQ3RDMTRMTW01T21NMkx3Vi9ERktpY2hBRjBuVkxLRG9TWUNtZmVIalNLeUVVVk54d01lagpsM3ZDYjRjOTRRcCtXWlZFQSt4UE91T2p1eFIrUkFmN2p0bVl4dXd2c0M4Q0F3RUFBYU1qTUNFd0RnWURWUjBQCkFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGcWYKZU1HZEdVellycHFIT3BnTzhvdzFEai9GZzA5dG5yd2dVQUZwd0NmQ1lheXdxUFJYUmdtMHBVUngrTFl2UnI1SQpSbWEvYzI4LzhjbVdXUG5ReEFHOUYyWVFnZU5TbExZNWY0U2NJemFzaDVQMWZIQlVRWUlWUzNkWnB1QXJrR1hOCkpySkszbFBHY0hWNm5ZRys5ZUs3b1FWS0Nrd1IvWnMvQ21FSHNOTGdPcDZRSFhPS0hTcWtlMmpwNHZmSG1TdC8Kb0R4VFRVSFIzN3g2cTEyTTRoYVhYM2NTWCthYkNRYzFZUU5kcUxiZHhYbjZ6OE14dVptRkI5YnJnSkVoU2N2Rgp2VHh6TnRpUnkzVHNkZUxROForWjF0cHNiR1NvSmFPUFJUcit3ZWNzM3RETEpuR1hwaGc0NHc2UW0zNnZkNENtCjhoaTN6ODczbEdSNmVGL05ZN1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K", "encoding": "base64", "item": { "file": "ca.crt", "name": "ca_file" }, "source": "/etc/origin/logging/ca.crt" } ok: [openshift] => (item={u'name': u'kibana_internal_key', u'file': u'kibana-internal.key'}) => { "changed": false, "content": "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBcDRrNEdrazE4UnRMdWROa1BDemZsSjA1YXFta0VuQndiNEd0RzREeEcxS3I1cG8vClg1dENkaG1va2VGcGlFajE0Q0p0MFQ0a2wvKytVczdvNTFWampYUU5UbVBCVWZXckJJaW5YOHVpQ2RvRCtqVk0KbU1JbVFKUkIzdUxNK1R0UFpMeFladDVNbytnV3F6OVo5RG00TXJ2M1VTaVFpSng5RklYQWs5WW9selFpV0ZhdgpYMW1XSE5uTldQOXBVbW1aQW8rdHhPWGVCSGp3RjNRUFZDMjFobzRmVGkxek1Bb0xadXNUUXNiZFhsWS82K2lhCkZGc09scWN5d09jblQ5MUxmL1c3c1BrVHZiYXBVVFRmTE12dWpGWGJyT0FzTmY0RHg0LzRwTzhIcGRxMXZlU3EKWk9YZDF6MGZMdFFYRmpYM3gvQVR5dklPRlJxSndyM3NidzllZFFJREFRQUJBb0lCQUZ6eHJXUTMxV3NyWDNsNgpJeE9YT1hwek54VkxnN0NQSEhGZ0dhODJlNHhTOHZiOS84YUVsZng3QTUvYTAvYzlzdHBqZjFVa0NwMGd6bStXClJuT0FWYmgxOTZQVGg3NTJjaWdUdi9kUE8xZ0gwVWtRSkt4aW1XSWVYZVRQSWpQcWlZTXpVai9QZCtrK3FzVGEKWW9jeE5sVS9IbGV5RDIwYWsvRzh1amZLQ2I3SDBabi9kYkhIUFdFdGJVQ0ZiRGp6QTRCdXBZWUdva1NvdHBUZQovdk8wRTJrbXVGZi9oNXhHdHZMd3JwMGFxQ29PRjVPNlNRd0Y2a0NvdTBaU24raFNiQlU4Umltc2xnZFpXWE9iCjVNaWYvZ21WUGIwY1h3VkhjRzVyL1lCVXBEaWwvSGdYRG1GdmxKTU9VdVNFWmlMSktjcklhU0Zzek83Q0k2T2QKU3luU2NVa0NnWUVBMVc1a0ErMzhpNDBoanBmcXpCRFhoS1V4SjY2cDByazczT0NNNkl4c2N0WXJZc215YUdVcQpTRDFxNGRCNC9sK2JKOU5GbDhxdFlrL1FCZjJBUVVxOHd3RzJJMTBMYnZZL21PNDczVjdza2o4dXhXZnJKMnFjClhYNk5FV05HdFlxQmRQbVNzaW9yRU5ZcXJyQ2RVdTd2TUQ1bE9oK2RsV1BydDNiOUU4bExVYk1DZ1lFQXlQTjEKdzAyc01HVzA5Z0dMUS9qMFJYeStTRzRwbjE5KzlQUnNobXZxOHpOVndaVC8reEJsUUxpbUU4NFZPVDRrSHlONwpJd2JXYk1NVXFIdGZUbjAwR3NSblFUT2xpNFFrZnBuaU83K3NGU2xESkxxeWpNWkVNUkRyUmlQVWwxcWF0cFB2CjZYNzQ0NW9ESllONHVQcFlYVEJpKy9YbFJxbEdkbkNIaHJ6WmF6Y0NnWUVBbTNkUHp6VDVzc1ZZM2o5MnZySzkKNmxkU0dPKzNzRkZSYzdEMXN5Y21qL2ZGbnE0QkVONHdFSEw4ZWdTYlBkOXlhMjRQMFdQSExFZXE1ZElrTEdzKwpXR1hVd0ttei9GM1BNWnpIYXBjTWgvTVNKS1Z2dTBTREplOWEzb0dNTGd6NEE3Y1B6a3U5RndTS0RybUR2N2RmCjc2cGpuaFVOZXBaUVMxR1hRMlJyUXRVQ2dZRUF4dmhoWXpjMlJsbCtXNU5kekxqbFRTUmQyS1BUYksyR3JJNlUKSW1rYW95SHlTSkg3QUNIVXo4ZkZFWmtkWktlRjVwc2dJcWR6aXcrQ0ZlektpcEhvRkpUaDFDcUtRT0ZmclVjNwpQWVBrcExOTmUwR3M1enl0YXBQaEVYc0sxYTV0d0ZnczFkcldINjV3YlNXSFZFajFVNWhFZWt5WWIyWE9raG1MCjU3cDM0UE1DZ1lFQWg2a3NLUWtaTmQ1czh1RG12YW9PZ1NrblVoRmVacDI2YTM3OVdUdGM2TWNOZ21hWWN5UnQKcU5NdUY1bWp6b2tYSXJOTUxYQ2kwcUl1cmhsa0N6YmV4TXJTOFFzQ2xqdUYxRzJjZTlMNGVTNnRpTjZoaTE2aQpNSk1zN2NaWTVtME43VzIyUWNOdDFNeDZpb2hnRkpwRkJPMW9GVU92V09uZmNKbHBPbjZQV1VrPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=", "encoding": "base64", "item": { "file": "kibana-internal.key", "name": "kibana_internal_key" }, "source": "/etc/origin/logging/kibana-internal.key" } ok: [openshift] => (item={u'name': u'kibana_internal_cert', u'file': u'kibana-internal.crt'}) => { "changed": false, "content": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUakNDQWphZ0F3SUJBZ0lCQWpBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpVMU9Gb1hEVEU1TURZd09URTJNelUxT1ZvdwpGakVVTUJJR0ExVUVBeE1MSUd0cFltRnVZUzF2Y0hNd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3CmdnRUtBb0lCQVFDbmlUZ2FTVFh4RzB1NTAyUThMTitVblRscXFhUVNjSEJ2Z2EwYmdQRWJVcXZtbWo5Zm0wSjIKR2FpUjRXbUlTUFhnSW0zUlBpU1gvNzVTenVqblZXT05kQTFPWThGUjlhc0VpS2RmeTZJSjJnUDZOVXlZd2laQQpsRUhlNHN6NU8wOWt2RmhtM2t5ajZCYXJQMW4wT2JneXUvZFJLSkNJbkgwVWhjQ1QxaWlYTkNKWVZxOWZXWlljCjJjMVkvMmxTYVprQ2o2M0U1ZDRFZVBBWGRBOVVMYldHamg5T0xYTXdDZ3RtNnhOQ3h0MWVWai9yNkpvVVd3NlcKcHpMQTV5ZFAzVXQvOWJ1dytSTzl0cWxSTk44c3krNk1WZHVzNEN3MS9nUEhqL2lrN3dlbDJyVzk1S3BrNWQzWApQUjh1MUJjV05mZkg4QlBLOGc0VkdvbkN2ZXh2RDE1MUFnTUJBQUdqZ1o0d2dac3dEZ1lEVlIwUEFRSC9CQVFECkFnV2dNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUZCd01CTUF3R0ExVWRFd0VCL3dRQ01BQXdaZ1lEVlIwUkJGOHcKWFlJTElHdHBZbUZ1WVMxdmNIT0NMQ0JyYVdKaGJtRXRiM0J6TG5KdmRYUmxjaTVrWldaaGRXeDBMbk4yWXk1agpiSFZ6ZEdWeUxteHZZMkZzZ2hnZ2EybGlZVzVoTGpFeU55NHdMakF1TVM1NGFYQXVhVytDQm10cFltRnVZVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQXV1eFZ3S3QrcDNCdVpWQVd4VktmZGUwM1BVWksxRm0xZnZ5c0xIOWMKL0dJcnB6UC8rVlR6YUQydlpWbGtQd2FzRDkzbVRoNGFiQ1htZWtyZld6YmZjdmNhTFBwVHYvc2hISUVsczNNcwpyT05USGJlSmxyK3dMTkJMREFVdSt6NFZTaGlJcEhlQTIzWW5wUnByMlcyUnhoQ0hzNGZlUGlyanl4OEFiRklCCjZUVUxWZC9jWDhwZHVVME9BR3dYVGIzN1Zma1dQbHBKNFdYeGMvQWFxWmxnVEZWWVFZY2VDT3B6SHp6bzJsWlQKMkgzYzRCYnhrekpybVVMU09qV1BTdmhiV2FkTkV6L3BOaDdINnB3cm1jT0ZQRVl2RWQ0NmNoNjV6MkNnMjJwcAppRWZKWmxvTWl4d3FUYjVzQkxrSVdaa05hbkVUdjYwa2JGYzd1Uis2Z2V1ck5nPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJQzJqQ0NBY0tnQXdJQkFnSUJBVEFOQmdrcWhraUc5dzBCQVFzRkFEQWVNUnd3R2dZRFZRUURFeE5zYjJkbgphVzVuTFhOcFoyNWxjaTEwWlhOME1CNFhEVEUzTURZd09URTJNelUxTmxvWERUSXlNRFl3T0RFMk16VTFOMW93CkhqRWNNQm9HQTFVRUF4TVRiRzluWjJsdVp5MXphV2R1WlhJdGRHVnpkRENDQVNJd0RRWUpLb1pJaHZjTkFRRUIKQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUhEMlVhZ2ZHNVhCK1FidGRxNzhNdGZRYzFFWTJ0dFplV3M0d0dMaEhFQwpHVXMwUWlkRXNnSTdyajVrZnJIN2tDTWxmd3Y3bTJGalgvWCs0Ykk3aDlCWGFjTkxzZEpSRE1tSFUvK3M2N0ErCjVNYzVWMUlTRWphY0x4bk9mZms1M3NUaDRTbzZaQ2tCaStwOWpWbFM3WEJ2L2dXZ2hpTWh0RWkvMmhpQjJsT2IKamVwKytNZUpUNEJBUkY2Mjc5NGdXTDd6SU1za3drVmFiTEphSW5ta1ZJVks5Mmo4R1NpUkNrRU1xQkk0VmpEOApCdVFrT2k3ekdlbGVDdEMxNExNbTVPbU0yTHdWL0RGS2ljaEFGMG5WTEtEb1NZQ21mZUhqU0t5RVVWTnh3TWVqCmwzdkNiNGM5NFFwK1daVkVBK3hQT3VPanV4UitSQWY3anRtWXh1d3ZzQzhDQXdFQUFhTWpNQ0V3RGdZRFZSMFAKQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUZxZgplTUdkR1V6WXJwcUhPcGdPOG93MURqL0ZnMDl0bnJ3Z1VBRnB3Q2ZDWWF5d3FQUlhSZ20wcFVSeCtMWXZScjVJClJtYS9jMjgvOGNtV1dQblF4QUc5RjJZUWdlTlNsTFk1ZjRTY0l6YXNoNVAxZkhCVVFZSVZTM2RacHVBcmtHWE4KSnJKSzNsUEdjSFY2bllHKzllSzdvUVZLQ2t3Ui9acy9DbUVIc05MZ09wNlFIWE9LSFNxa2UyanA0dmZIbVN0LwpvRHhUVFVIUjM3eDZxMTJNNGhhWFgzY1NYK2FiQ1FjMVlRTmRxTGJkeFhuNno4TXh1Wm1GQjlicmdKRWhTY3ZGCnZUeHpOdGlSeTNUc2RlTFE4WitaMXRwc2JHU29KYU9QUlRyK3dlY3MzdERMSm5HWHBoZzQ0dzZRbTM2dmQ0Q20KOGhpM3o4NzNsR1I2ZUYvTlk3VT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", "encoding": "base64", "item": { "file": "kibana-internal.crt", "name": "kibana_internal_cert" }, "source": "/etc/origin/logging/kibana-internal.crt" } ok: [openshift] => (item={u'name': u'server_tls', u'file': u'server-tls.json'}) => { "changed": false, "content": "Ly8gU2VlIGZvciBhdmFpbGFibGUgb3B0aW9uczogaHR0cHM6Ly9ub2RlanMub3JnL2FwaS90bHMuaHRtbCN0bHNfdGxzX2NyZWF0ZXNlcnZlcl9vcHRpb25zX3NlY3VyZWNvbm5lY3Rpb25saXN0ZW5lcgp0bHNfb3B0aW9ucyA9IHsKCWNpcGhlcnM6ICdrRUVDREg6K2tFRUNESCtTSEE6a0VESDora0VESCtTSEE6K2tFREgrQ0FNRUxMSUE6a0VDREg6K2tFQ0RIK1NIQTprUlNBOitrUlNBK1NIQTora1JTQStDQU1FTExJQTohYU5VTEw6IWVOVUxMOiFTU0x2MjohUkM0OiFERVM6IUVYUDohU0VFRDohSURFQTorM0RFUycsCglob25vckNpcGhlck9yZGVyOiB0cnVlCn0K", "encoding": "base64", "item": { "file": "server-tls.json", "name": "server_tls" }, "source": "/etc/origin/logging/server-tls.json" } ok: [openshift] => (item={u'name': u'session_secret', u'file': u'session_secret'}) => { "changed": false, "content": "dWJ3bzJDNUxGVXdjMU03SkVmcExhWDJCWG15bHBtaWthMzhEaERyRnJPaXNJY2FOZ2JLaE9YZlQ5eTEzZE1HZXVKa0RMR1d5WjBlVVhseEZzdG4xZlRSdzhvS3pxV2dFYkhhN0NTeDVQMk0yZWNwdWduNWdDSVk4THJCb2xqM1hkdzhSdDk5Q1RUc3l3Vmo5TVNUdE1FRlA5QVk4YVV4Sk1SV1dOelZoYWpEcEtvcFJ3RmZXT2VTOTg0VzJVSUlBZWNsUEVsUnE=", "encoding": "base64", "item": { "file": "session_secret", "name": "session_secret" }, "source": "/etc/origin/logging/session_secret" } ok: [openshift] => (item={u'name': u'oauth_secret', u'file': u'oauth_secret'}) => { "changed": false, "content": "RDBhTVcwUTVoQXBwbnB0N0gyQ1NkWFJzVlBqelFJNnRmVjVSMVpmMWZycUhkeVRZckFLZ3djZk92a0lRNlA2TA==", "encoding": "base64", "item": { "file": "oauth_secret", "name": "oauth_secret" }, "source": "/etc/origin/logging/oauth_secret" } TASK [openshift_logging_kibana : Set logging-kibana-ops service] *************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:84 changed: [openshift] => { "changed": true, "results": { "clusterip": "172.30.46.235", "cmd": "/bin/oc get service logging-kibana-ops -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "Service", "metadata": { "creationTimestamp": "2017-06-09T16:37:00Z", "name": "logging-kibana-ops", "namespace": "logging", "resourceVersion": "1450", "selfLink": "/api/v1/namespaces/logging/services/logging-kibana-ops", "uid": "dc37333f-4d31-11e7-8061-0ed679898f5a" }, "spec": { "clusterIP": "172.30.46.235", "ports": [ { "port": 443, "protocol": "TCP", "targetPort": "oaproxy" } ], "selector": { "component": "kibana-ops", "provider": "openshift" }, "sessionAffinity": "None", "type": "ClusterIP" }, "status": { "loadBalancer": {} } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:101 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ openshift_logging_kibana_key | trim | length > 0 }} skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:106 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ openshift_logging_kibana_cert | trim | length > 0 }} skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:111 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ openshift_logging_kibana_ca | trim | length > 0 }} skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:116 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_kibana : Generating Kibana route template] ************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:121 ok: [openshift] => { "changed": false, "checksum": "9d336a16b72b7f74dcff15e4bdc537dd41aa1a2f", "dest": "/tmp/openshift-logging-ansible-gvBLkA/templates/kibana-route.yaml", "gid": 0, "group": "root", "md5sum": "f7e94baa3d918e713deea3b3c6ac860f", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 2726, "src": "/root/.ansible/tmp/ansible-tmp-1497026220.78-89743656864150/source", "state": "file", "uid": 0 } TASK [openshift_logging_kibana : Setting Kibana route] ************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:141 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get route logging-kibana-ops -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "Route", "metadata": { "creationTimestamp": "2017-06-09T16:37:01Z", "labels": { "component": "support", "logging-infra": "support", "provider": "openshift" }, "name": "logging-kibana-ops", "namespace": "logging", "resourceVersion": "1456", "selfLink": "/oapi/v1/namespaces/logging/routes/logging-kibana-ops", "uid": "dd1f2e5a-4d31-11e7-8061-0ed679898f5a" }, "spec": { "host": "kibana-ops.router.default.svc.cluster.local", "tls": { "caCertificate": "-----BEGIN CERTIFICATE-----\nMIIC2jCCAcKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAeMRwwGgYDVQQDExNsb2dn\naW5nLXNpZ25lci10ZXN0MB4XDTE3MDYwOTE2MzU1NloXDTIyMDYwODE2MzU1N1ow\nHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAMHD2UagfG5XB+Qbtdq78MtfQc1EY2ttZeWs4wGLhHEC\nGUs0QidEsgI7rj5kfrH7kCMlfwv7m2FjX/X+4bI7h9BXacNLsdJRDMmHU/+s67A+\n5Mc5V1ISEjacLxnOffk53sTh4So6ZCkBi+p9jVlS7XBv/gWghiMhtEi/2hiB2lOb\njep++MeJT4BARF62794gWL7zIMskwkVabLJaInmkVIVK92j8GSiRCkEMqBI4VjD8\nBuQkOi7zGeleCtC14LMm5OmM2LwV/DFKichAF0nVLKDoSYCmfeHjSKyEUVNxwMej\nl3vCb4c94Qp+WZVEA+xPOuOjuxR+RAf7jtmYxuwvsC8CAwEAAaMjMCEwDgYDVR0P\nAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFqf\neMGdGUzYrpqHOpgO8ow1Dj/Fg09tnrwgUAFpwCfCYaywqPRXRgm0pURx+LYvRr5I\nRma/c28/8cmWWPnQxAG9F2YQgeNSlLY5f4ScIzash5P1fHBUQYIVS3dZpuArkGXN\nJrJK3lPGcHV6nYG+9eK7oQVKCkwR/Zs/CmEHsNLgOp6QHXOKHSqke2jp4vfHmSt/\noDxTTUHR37x6q12M4haXX3cSX+abCQc1YQNdqLbdxXn6z8MxuZmFB9brgJEhScvF\nvTxzNtiRy3TsdeLQ8Z+Z1tpsbGSoJaOPRTr+wecs3tDLJnGXphg44w6Qm36vd4Cm\n8hi3z873lGR6eF/NY7U=\n-----END CERTIFICATE-----\n", "destinationCACertificate": "-----BEGIN CERTIFICATE-----\nMIIC2jCCAcKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAeMRwwGgYDVQQDExNsb2dn\naW5nLXNpZ25lci10ZXN0MB4XDTE3MDYwOTE2MzU1NloXDTIyMDYwODE2MzU1N1ow\nHjEcMBoGA1UEAxMTbG9nZ2luZy1zaWduZXItdGVzdDCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAMHD2UagfG5XB+Qbtdq78MtfQc1EY2ttZeWs4wGLhHEC\nGUs0QidEsgI7rj5kfrH7kCMlfwv7m2FjX/X+4bI7h9BXacNLsdJRDMmHU/+s67A+\n5Mc5V1ISEjacLxnOffk53sTh4So6ZCkBi+p9jVlS7XBv/gWghiMhtEi/2hiB2lOb\njep++MeJT4BARF62794gWL7zIMskwkVabLJaInmkVIVK92j8GSiRCkEMqBI4VjD8\nBuQkOi7zGeleCtC14LMm5OmM2LwV/DFKichAF0nVLKDoSYCmfeHjSKyEUVNxwMej\nl3vCb4c94Qp+WZVEA+xPOuOjuxR+RAf7jtmYxuwvsC8CAwEAAaMjMCEwDgYDVR0P\nAQH/BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAFqf\neMGdGUzYrpqHOpgO8ow1Dj/Fg09tnrwgUAFpwCfCYaywqPRXRgm0pURx+LYvRr5I\nRma/c28/8cmWWPnQxAG9F2YQgeNSlLY5f4ScIzash5P1fHBUQYIVS3dZpuArkGXN\nJrJK3lPGcHV6nYG+9eK7oQVKCkwR/Zs/CmEHsNLgOp6QHXOKHSqke2jp4vfHmSt/\noDxTTUHR37x6q12M4haXX3cSX+abCQc1YQNdqLbdxXn6z8MxuZmFB9brgJEhScvF\nvTxzNtiRy3TsdeLQ8Z+Z1tpsbGSoJaOPRTr+wecs3tDLJnGXphg44w6Qm36vd4Cm\n8hi3z873lGR6eF/NY7U=\n-----END CERTIFICATE-----\n", "insecureEdgeTerminationPolicy": "Redirect", "termination": "reencrypt" }, "to": { "kind": "Service", "name": "logging-kibana-ops", "weight": 100 }, "wildcardPolicy": "None" }, "status": { "ingress": [ { "conditions": [ { "lastTransitionTime": "2017-06-09T16:37:01Z", "status": "True", "type": "Admitted" } ], "host": "kibana-ops.router.default.svc.cluster.local", "routerName": "router", "wildcardPolicy": "None" } ] } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : Get current oauthclient hostnames] ************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:151 ok: [openshift] => { "changed": false, "results": { "cmd": "/bin/oc get oauthclient kibana-proxy -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "OAuthClient", "metadata": { "creationTimestamp": "2017-06-09T16:36:50Z", "labels": { "logging-infra": "support" }, "name": "kibana-proxy", "resourceVersion": "1405", "selfLink": "/oapi/v1/oauthclients/kibana-proxy", "uid": "d637f4f0-4d31-11e7-8061-0ed679898f5a" }, "redirectURIs": [ "https://kibana.router.default.svc.cluster.local" ], "scopeRestrictions": [ { "literals": [ "user:info", "user:check-access", "user:list-projects" ] } ], "secret": "D0aMW0Q5hAppnpt7H2CSdXRsVPjzQI6tfV5R1Zf1frqHdyTYrAKgwcfOvkIQ6P6L" } ], "returncode": 0 }, "state": "list" } TASK [openshift_logging_kibana : set_fact] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:159 ok: [openshift] => { "ansible_facts": { "proxy_hostnames": [ "https://kibana.router.default.svc.cluster.local", "https://kibana-ops.router.default.svc.cluster.local" ] }, "changed": false } TASK [openshift_logging_kibana : Create oauth-client template] ***************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:162 changed: [openshift] => { "changed": true, "checksum": "d5cb634d6af6d8930d0735d80038cf3fead4fa60", "dest": "/tmp/openshift-logging-ansible-gvBLkA/templates/oauth-client.yml", "gid": 0, "group": "root", "md5sum": "3c41a2a0bad3f101c0613e53bba75efe", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 382, "src": "/root/.ansible/tmp/ansible-tmp-1497026222.53-194217586283417/source", "state": "file", "uid": 0 } TASK [openshift_logging_kibana : Set kibana-proxy oauth-client] **************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:170 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get oauthclient kibana-proxy -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "OAuthClient", "metadata": { "creationTimestamp": "2017-06-09T16:36:50Z", "labels": { "logging-infra": "support" }, "name": "kibana-proxy", "resourceVersion": "1462", "selfLink": "/oapi/v1/oauthclients/kibana-proxy", "uid": "d637f4f0-4d31-11e7-8061-0ed679898f5a" }, "redirectURIs": [ "https://kibana.router.default.svc.cluster.local", "https://kibana-ops.router.default.svc.cluster.local" ], "scopeRestrictions": [ { "literals": [ "user:info", "user:check-access", "user:list-projects" ] } ], "secret": "D0aMW0Q5hAppnpt7H2CSdXRsVPjzQI6tfV5R1Zf1frqHdyTYrAKgwcfOvkIQ6P6L" } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : Set Kibana secret] **************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:181 ok: [openshift] => { "changed": false, "results": { "apiVersion": "v1", "data": { "ca": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMyakNDQWNLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpVMU5sb1hEVEl5TURZd09ERTJNelUxTjFvdwpIakVjTUJvR0ExVUVBeE1UYkc5bloybHVaeTF6YVdkdVpYSXRkR1Z6ZERDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1IRDJVYWdmRzVYQitRYnRkcTc4TXRmUWMxRVkydHRaZVdzNHdHTGhIRUMKR1VzMFFpZEVzZ0k3cmo1a2ZySDdrQ01sZnd2N20yRmpYL1grNGJJN2g5QlhhY05Mc2RKUkRNbUhVLytzNjdBKwo1TWM1VjFJU0VqYWNMeG5PZmZrNTNzVGg0U282WkNrQmkrcDlqVmxTN1hCdi9nV2doaU1odEVpLzJoaUIybE9iCmplcCsrTWVKVDRCQVJGNjI3OTRnV0w3eklNc2t3a1ZhYkxKYUlubWtWSVZLOTJqOEdTaVJDa0VNcUJJNFZqRDgKQnVRa09pN3pHZWxlQ3RDMTRMTW01T21NMkx3Vi9ERktpY2hBRjBuVkxLRG9TWUNtZmVIalNLeUVVVk54d01lagpsM3ZDYjRjOTRRcCtXWlZFQSt4UE91T2p1eFIrUkFmN2p0bVl4dXd2c0M4Q0F3RUFBYU1qTUNFd0RnWURWUjBQCkFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGcWYKZU1HZEdVellycHFIT3BnTzhvdzFEai9GZzA5dG5yd2dVQUZwd0NmQ1lheXdxUFJYUmdtMHBVUngrTFl2UnI1SQpSbWEvYzI4LzhjbVdXUG5ReEFHOUYyWVFnZU5TbExZNWY0U2NJemFzaDVQMWZIQlVRWUlWUzNkWnB1QXJrR1hOCkpySkszbFBHY0hWNm5ZRys5ZUs3b1FWS0Nrd1IvWnMvQ21FSHNOTGdPcDZRSFhPS0hTcWtlMmpwNHZmSG1TdC8Kb0R4VFRVSFIzN3g2cTEyTTRoYVhYM2NTWCthYkNRYzFZUU5kcUxiZHhYbjZ6OE14dVptRkI5YnJnSkVoU2N2Rgp2VHh6TnRpUnkzVHNkZUxROForWjF0cHNiR1NvSmFPUFJUcit3ZWNzM3RETEpuR1hwaGc0NHc2UW0zNnZkNENtCjhoaTN6ODczbEdSNmVGL05ZN1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K", "cert": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURSVENDQWkyZ0F3SUJBZ0lCQXpBTkJna3Foa2lHOXcwQkFRVUZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpZd01sb1hEVEU1TURZd09URTJNell3TWxvdwpSakVRTUE0R0ExVUVDZ3dIVEc5bloybHVaekVTTUJBR0ExVUVDd3dKVDNCbGJsTm9hV1owTVI0d0hBWURWUVFECkRCVnplWE4wWlcwdWJHOW5aMmx1Wnk1cmFXSmhibUV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXcKZ2dFS0FvSUJBUURTMzJYZXRPeDdjU3AyR3J6bWhYQ3Yxb2tOL0hnSnVDLzQ2a2dwSFNhNERzSGs5VjB2N3NDUQpCbE5mcTl0NEhxMkFIUDg5blU3Q09QUTRnSjZVa1VnVEV4WHYrQ1B3SDB2SFlmWTVmMWZPNlFTdmluMVpZQTVnCkpTdXYydm9UMmxsRzNhdHY5dktXb3pzTE9IUjVJM1JpaGQ0NWdsempXTDM0U1JiQjB0VG1jclUzTENsVWwyZW0KSWtyMUhnQ2hqZkowYWpLQWJBU2h2Wk5MODlDeEpaS3NvL2dPSmlJQ0cxNkJPUGdMekFtbjRPTUhjb1daeWx4RgpuUEt3UmtENkVjL2FJWUJnUDVmODVmZ2U3RkJDYWRzQi9yd1U1WjRYK0lRaG5wUzRUdkpLWDEvVnBPMUFCc3pSCnFGUmhwZlpiTWRXeFBsTlN5TERsazBjQWwwVTJxQUVUQWdNQkFBR2paakJrTUE0R0ExVWREd0VCL3dRRUF3SUYKb0RBSkJnTlZIUk1FQWpBQU1CMEdBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFkQmdOVgpIUTRFRmdRVU8yeXNpVTdTbjNja1p0b2FMSlUxWXhJck5CMHdDUVlEVlIwakJBSXdBREFOQmdrcWhraUc5dzBCCkFRVUZBQU9DQVFFQUc5VFhKTUlGalphZy9VTnMyaGJob0tTZHdEa3FwTHJ1WWpUVEQ2a2VDOEdkWGtKdXUxQUYKZi9zMUVLcXBqRTdnMHN5NlI4dmd4a2hsbG1OTFEwQktrQkFlM1NWRlRYWlJ2UjFUc2tiZGJQTTJqOUxYbFpCRQpBSUE2SWk2SkhxTFp0YlJWV043UW9nSDFtUnU1d0laS0RqaDcxZThSUmtic1FFZEV4TmRBbFNyYVpKNXJ3ME1tCjIyd3VNSFJrTmlObkNKOW81SktWOEhOVFFNUFBlRkdxdFl2VE4veUNTNmtRb0Y5Qng2ZDZBZVVIdjBYdXUwcXQKakdsTHZRc3NRK2N0THFybVh6SFlBVUdzTm1yblRVSWdsYjRBcHE3R1hqT1cweHRuOHF0U2ZWbGhZZnYrUDE4OApSSE9rYlJWRlZoUVVxUlNSUVN3bzVhcExTTnllaWw1RlBBPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", "key": "LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2Z0lCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktnd2dnU2tBZ0VBQW9JQkFRRFMzMlhldE94N2NTcDIKR3J6bWhYQ3Yxb2tOL0hnSnVDLzQ2a2dwSFNhNERzSGs5VjB2N3NDUUJsTmZxOXQ0SHEyQUhQODluVTdDT1BRNApnSjZVa1VnVEV4WHYrQ1B3SDB2SFlmWTVmMWZPNlFTdmluMVpZQTVnSlN1djJ2b1QybGxHM2F0djl2S1dvenNMCk9IUjVJM1JpaGQ0NWdsempXTDM0U1JiQjB0VG1jclUzTENsVWwyZW1Ja3IxSGdDaGpmSjBhaktBYkFTaHZaTkwKODlDeEpaS3NvL2dPSmlJQ0cxNkJPUGdMekFtbjRPTUhjb1daeWx4Rm5QS3dSa0Q2RWMvYUlZQmdQNWY4NWZnZQo3RkJDYWRzQi9yd1U1WjRYK0lRaG5wUzRUdkpLWDEvVnBPMUFCc3pScUZSaHBmWmJNZFd4UGxOU3lMRGxrMGNBCmwwVTJxQUVUQWdNQkFBRUNnZ0VCQUs1UG1JZldDcjJudUMvVXZsM1g1R1NWOU1rbzc1a3BOZzZPcTRQQTR5YTIKbkRURlZuaXU1V0tvdC9hWHUxMjZEaldld3FhT0NIS29YY2p2R1NjNlNOZnJxUENxc0JwbjVGNDJiakR0UVFwNgp1RExiU2U2OVptUzNsVHRTeWZIclJtMDhBK0VUdzBnZU9rcDJLbUdRbXo1dDVscW5sbDBnR0RqTFFoS0tVdzhDCk52T0lVbUtaMFJsUDdWNkprQkJRQ2d0RkpYQVBKMTlFTWNFZ1JhUjE5akd2MHRVZU9XejcrQ3kvSXdzdVNXZDUKTm9qaGc3ZExLUzkxMGErdzBxQUV4SEVBL2ZMYktUeFBVSjBaa1RZUzlZTmhLTHp4U05QbkIydUxxM2RDV0p2QgpYQndsRUNUNzh0YnJPNmZYVnNFc3lvVjJJUEFHN1Jyd3h0aHRKTEdvN3NFQ2dZRUE3UEhYSnFodHNTR2FST0FJClgydy92VFZTa09rMXhsQnpaTGhkMGpyN0gyR21Xejgzblg0UDNVNGNxaDZSWC92LzhEVGZoSFB3NlI2WmJuVkQKOUFSUEF2cmhic2R1NFJ3WXBnbEI0OW0wcWxhTFpUM21ETTlUOEdvUHhkd2lnWVNiR3FIWlRrTW04QXZDWUNJWQpycVBxTE1BbkEzU3IwanFOaHVacytrYkZzRkVDZ1lFQTQ5VEs2LzdCS0Z3RTlvL0RqZHpRcGo4SHJLNDBGUExLCmZtVEdYRmc2RWQrV3o2M0RWOEFCVDUyOWhXSkJSQ3NpUmlSeXdUZWZaanJiTEpNVXFaaFNUWEhua0pkVGhDTlkKR0YyQWkyUWtZaWRac0VaQmRnOG5KOWNjRThYR2lrZkdjZGVFKzVIaEFiM0o5NFhPWGRhZkprZTA1KzhXWS9kSwo5aGNhWm5iUEJpTUNnWUVBcjl5bnRiZVpBcmxRQ1FMRzlCd2lDc3NDNXk5UWpFcTErdkdrSVI5SnQvY0FSQVZkCnYvVUt6VlRtalpWMmpPazdBd3diUWZIVHd4TEZDbWphOE1ncDVLcFVtNlpHTTZwbzJRTlI5azFsZkhBVHNpY0oKRWtUbnBpN3lXZ2FUVGRJZ1ZObi9zY2pCbzdUMnBlZG9ybVpXcnJSSHdKYmQySUZBL2ZrVXRZWEZ3S0VDZ1lBZwowZ205ZjRDUjhOZWlNVW1waVJMaHNCako3bnFBUGcyZnNhTEZzTXdFd1NIM252K1NCNDNveXo5L0ZJMTBaWVpWCmRuQzFsdTYyaW1vQ2ZObXdLTXBLYjhzOUVCK3UzR3lYYmZqVG04U0dBMGN2VnArZGcwZFB2UFJGZ09zK0JtTDMKQ1dZVjI0ai9lRGk5R0tDRzhpRVJJYWxVOUNOVW1VNUtMei9WWE80cjR3S0JnQnc2aXdZcm1Hc0JNQ2xCNmM5Qwp6ZDdWR3ZKdjF2MVpVUFd6RkFjYk54eUVqRGh6VW5TSzhWU0h0TDhkVjI1Y084YXh1aFZCck5JNlFkNG0yNHZGCkhUVzZHZlBoWnRucWZFend2aXlHSHhlcGZtRGxDVUlqdXVOdnVtUk1uNnJtbnFnV1lNeVJsUUp2eGtaeXg0emkKak1seUpTMXhBRmhMSVB2eU45Rk1LTEp5Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K" }, "kind": "Secret", "metadata": { "creationTimestamp": null, "name": "logging-kibana" }, "type": "Opaque" }, "state": "present" } TASK [openshift_logging_kibana : Set Kibana Proxy secret] ********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:195 ok: [openshift] => { "changed": false, "results": { "apiVersion": "v1", "data": { "oauth-secret": "RDBhTVcwUTVoQXBwbnB0N0gyQ1NkWFJzVlBqelFJNnRmVjVSMVpmMWZycUhkeVRZckFLZ3djZk92a0lRNlA2TA==", "server-cert": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURUakNDQWphZ0F3SUJBZ0lCQWpBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpVMU9Gb1hEVEU1TURZd09URTJNelUxT1ZvdwpGakVVTUJJR0ExVUVBeE1MSUd0cFltRnVZUzF2Y0hNd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3CmdnRUtBb0lCQVFDbmlUZ2FTVFh4RzB1NTAyUThMTitVblRscXFhUVNjSEJ2Z2EwYmdQRWJVcXZtbWo5Zm0wSjIKR2FpUjRXbUlTUFhnSW0zUlBpU1gvNzVTenVqblZXT05kQTFPWThGUjlhc0VpS2RmeTZJSjJnUDZOVXlZd2laQQpsRUhlNHN6NU8wOWt2RmhtM2t5ajZCYXJQMW4wT2JneXUvZFJLSkNJbkgwVWhjQ1QxaWlYTkNKWVZxOWZXWlljCjJjMVkvMmxTYVprQ2o2M0U1ZDRFZVBBWGRBOVVMYldHamg5T0xYTXdDZ3RtNnhOQ3h0MWVWai9yNkpvVVd3NlcKcHpMQTV5ZFAzVXQvOWJ1dytSTzl0cWxSTk44c3krNk1WZHVzNEN3MS9nUEhqL2lrN3dlbDJyVzk1S3BrNWQzWApQUjh1MUJjV05mZkg4QlBLOGc0VkdvbkN2ZXh2RDE1MUFnTUJBQUdqZ1o0d2dac3dEZ1lEVlIwUEFRSC9CQVFECkFnV2dNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUZCd01CTUF3R0ExVWRFd0VCL3dRQ01BQXdaZ1lEVlIwUkJGOHcKWFlJTElHdHBZbUZ1WVMxdmNIT0NMQ0JyYVdKaGJtRXRiM0J6TG5KdmRYUmxjaTVrWldaaGRXeDBMbk4yWXk1agpiSFZ6ZEdWeUxteHZZMkZzZ2hnZ2EybGlZVzVoTGpFeU55NHdMakF1TVM1NGFYQXVhVytDQm10cFltRnVZVEFOCkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQXV1eFZ3S3QrcDNCdVpWQVd4VktmZGUwM1BVWksxRm0xZnZ5c0xIOWMKL0dJcnB6UC8rVlR6YUQydlpWbGtQd2FzRDkzbVRoNGFiQ1htZWtyZld6YmZjdmNhTFBwVHYvc2hISUVsczNNcwpyT05USGJlSmxyK3dMTkJMREFVdSt6NFZTaGlJcEhlQTIzWW5wUnByMlcyUnhoQ0hzNGZlUGlyanl4OEFiRklCCjZUVUxWZC9jWDhwZHVVME9BR3dYVGIzN1Zma1dQbHBKNFdYeGMvQWFxWmxnVEZWWVFZY2VDT3B6SHp6bzJsWlQKMkgzYzRCYnhrekpybVVMU09qV1BTdmhiV2FkTkV6L3BOaDdINnB3cm1jT0ZQRVl2RWQ0NmNoNjV6MkNnMjJwcAppRWZKWmxvTWl4d3FUYjVzQkxrSVdaa05hbkVUdjYwa2JGYzd1Uis2Z2V1ck5nPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJQzJqQ0NBY0tnQXdJQkFnSUJBVEFOQmdrcWhraUc5dzBCQVFzRkFEQWVNUnd3R2dZRFZRUURFeE5zYjJkbgphVzVuTFhOcFoyNWxjaTEwWlhOME1CNFhEVEUzTURZd09URTJNelUxTmxvWERUSXlNRFl3T0RFMk16VTFOMW93CkhqRWNNQm9HQTFVRUF4TVRiRzluWjJsdVp5MXphV2R1WlhJdGRHVnpkRENDQVNJd0RRWUpLb1pJaHZjTkFRRUIKQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUhEMlVhZ2ZHNVhCK1FidGRxNzhNdGZRYzFFWTJ0dFplV3M0d0dMaEhFQwpHVXMwUWlkRXNnSTdyajVrZnJIN2tDTWxmd3Y3bTJGalgvWCs0Ykk3aDlCWGFjTkxzZEpSRE1tSFUvK3M2N0ErCjVNYzVWMUlTRWphY0x4bk9mZms1M3NUaDRTbzZaQ2tCaStwOWpWbFM3WEJ2L2dXZ2hpTWh0RWkvMmhpQjJsT2IKamVwKytNZUpUNEJBUkY2Mjc5NGdXTDd6SU1za3drVmFiTEphSW5ta1ZJVks5Mmo4R1NpUkNrRU1xQkk0VmpEOApCdVFrT2k3ekdlbGVDdEMxNExNbTVPbU0yTHdWL0RGS2ljaEFGMG5WTEtEb1NZQ21mZUhqU0t5RVVWTnh3TWVqCmwzdkNiNGM5NFFwK1daVkVBK3hQT3VPanV4UitSQWY3anRtWXh1d3ZzQzhDQXdFQUFhTWpNQ0V3RGdZRFZSMFAKQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUZxZgplTUdkR1V6WXJwcUhPcGdPOG93MURqL0ZnMDl0bnJ3Z1VBRnB3Q2ZDWWF5d3FQUlhSZ20wcFVSeCtMWXZScjVJClJtYS9jMjgvOGNtV1dQblF4QUc5RjJZUWdlTlNsTFk1ZjRTY0l6YXNoNVAxZkhCVVFZSVZTM2RacHVBcmtHWE4KSnJKSzNsUEdjSFY2bllHKzllSzdvUVZLQ2t3Ui9acy9DbUVIc05MZ09wNlFIWE9LSFNxa2UyanA0dmZIbVN0LwpvRHhUVFVIUjM3eDZxMTJNNGhhWFgzY1NYK2FiQ1FjMVlRTmRxTGJkeFhuNno4TXh1Wm1GQjlicmdKRWhTY3ZGCnZUeHpOdGlSeTNUc2RlTFE4WitaMXRwc2JHU29KYU9QUlRyK3dlY3MzdERMSm5HWHBoZzQ0dzZRbTM2dmQ0Q20KOGhpM3o4NzNsR1I2ZUYvTlk3VT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", "server-key": "LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBcDRrNEdrazE4UnRMdWROa1BDemZsSjA1YXFta0VuQndiNEd0RzREeEcxS3I1cG8vClg1dENkaG1va2VGcGlFajE0Q0p0MFQ0a2wvKytVczdvNTFWampYUU5UbVBCVWZXckJJaW5YOHVpQ2RvRCtqVk0KbU1JbVFKUkIzdUxNK1R0UFpMeFladDVNbytnV3F6OVo5RG00TXJ2M1VTaVFpSng5RklYQWs5WW9selFpV0ZhdgpYMW1XSE5uTldQOXBVbW1aQW8rdHhPWGVCSGp3RjNRUFZDMjFobzRmVGkxek1Bb0xadXNUUXNiZFhsWS82K2lhCkZGc09scWN5d09jblQ5MUxmL1c3c1BrVHZiYXBVVFRmTE12dWpGWGJyT0FzTmY0RHg0LzRwTzhIcGRxMXZlU3EKWk9YZDF6MGZMdFFYRmpYM3gvQVR5dklPRlJxSndyM3NidzllZFFJREFRQUJBb0lCQUZ6eHJXUTMxV3NyWDNsNgpJeE9YT1hwek54VkxnN0NQSEhGZ0dhODJlNHhTOHZiOS84YUVsZng3QTUvYTAvYzlzdHBqZjFVa0NwMGd6bStXClJuT0FWYmgxOTZQVGg3NTJjaWdUdi9kUE8xZ0gwVWtRSkt4aW1XSWVYZVRQSWpQcWlZTXpVai9QZCtrK3FzVGEKWW9jeE5sVS9IbGV5RDIwYWsvRzh1amZLQ2I3SDBabi9kYkhIUFdFdGJVQ0ZiRGp6QTRCdXBZWUdva1NvdHBUZQovdk8wRTJrbXVGZi9oNXhHdHZMd3JwMGFxQ29PRjVPNlNRd0Y2a0NvdTBaU24raFNiQlU4Umltc2xnZFpXWE9iCjVNaWYvZ21WUGIwY1h3VkhjRzVyL1lCVXBEaWwvSGdYRG1GdmxKTU9VdVNFWmlMSktjcklhU0Zzek83Q0k2T2QKU3luU2NVa0NnWUVBMVc1a0ErMzhpNDBoanBmcXpCRFhoS1V4SjY2cDByazczT0NNNkl4c2N0WXJZc215YUdVcQpTRDFxNGRCNC9sK2JKOU5GbDhxdFlrL1FCZjJBUVVxOHd3RzJJMTBMYnZZL21PNDczVjdza2o4dXhXZnJKMnFjClhYNk5FV05HdFlxQmRQbVNzaW9yRU5ZcXJyQ2RVdTd2TUQ1bE9oK2RsV1BydDNiOUU4bExVYk1DZ1lFQXlQTjEKdzAyc01HVzA5Z0dMUS9qMFJYeStTRzRwbjE5KzlQUnNobXZxOHpOVndaVC8reEJsUUxpbUU4NFZPVDRrSHlONwpJd2JXYk1NVXFIdGZUbjAwR3NSblFUT2xpNFFrZnBuaU83K3NGU2xESkxxeWpNWkVNUkRyUmlQVWwxcWF0cFB2CjZYNzQ0NW9ESllONHVQcFlYVEJpKy9YbFJxbEdkbkNIaHJ6WmF6Y0NnWUVBbTNkUHp6VDVzc1ZZM2o5MnZySzkKNmxkU0dPKzNzRkZSYzdEMXN5Y21qL2ZGbnE0QkVONHdFSEw4ZWdTYlBkOXlhMjRQMFdQSExFZXE1ZElrTEdzKwpXR1hVd0ttei9GM1BNWnpIYXBjTWgvTVNKS1Z2dTBTREplOWEzb0dNTGd6NEE3Y1B6a3U5RndTS0RybUR2N2RmCjc2cGpuaFVOZXBaUVMxR1hRMlJyUXRVQ2dZRUF4dmhoWXpjMlJsbCtXNU5kekxqbFRTUmQyS1BUYksyR3JJNlUKSW1rYW95SHlTSkg3QUNIVXo4ZkZFWmtkWktlRjVwc2dJcWR6aXcrQ0ZlektpcEhvRkpUaDFDcUtRT0ZmclVjNwpQWVBrcExOTmUwR3M1enl0YXBQaEVYc0sxYTV0d0ZnczFkcldINjV3YlNXSFZFajFVNWhFZWt5WWIyWE9raG1MCjU3cDM0UE1DZ1lFQWg2a3NLUWtaTmQ1czh1RG12YW9PZ1NrblVoRmVacDI2YTM3OVdUdGM2TWNOZ21hWWN5UnQKcU5NdUY1bWp6b2tYSXJOTUxYQ2kwcUl1cmhsa0N6YmV4TXJTOFFzQ2xqdUYxRzJjZTlMNGVTNnRpTjZoaTE2aQpNSk1zN2NaWTVtME43VzIyUWNOdDFNeDZpb2hnRkpwRkJPMW9GVU92V09uZmNKbHBPbjZQV1VrPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=", "server-tls.json": "Ly8gU2VlIGZvciBhdmFpbGFibGUgb3B0aW9uczogaHR0cHM6Ly9ub2RlanMub3JnL2FwaS90bHMuaHRtbCN0bHNfdGxzX2NyZWF0ZXNlcnZlcl9vcHRpb25zX3NlY3VyZWNvbm5lY3Rpb25saXN0ZW5lcgp0bHNfb3B0aW9ucyA9IHsKCWNpcGhlcnM6ICdrRUVDREg6K2tFRUNESCtTSEE6a0VESDora0VESCtTSEE6K2tFREgrQ0FNRUxMSUE6a0VDREg6K2tFQ0RIK1NIQTprUlNBOitrUlNBK1NIQTora1JTQStDQU1FTExJQTohYU5VTEw6IWVOVUxMOiFTU0x2MjohUkM0OiFERVM6IUVYUDohU0VFRDohSURFQTorM0RFUycsCglob25vckNpcGhlck9yZGVyOiB0cnVlCn0K", "session-secret": "dWJ3bzJDNUxGVXdjMU03SkVmcExhWDJCWG15bHBtaWthMzhEaERyRnJPaXNJY2FOZ2JLaE9YZlQ5eTEzZE1HZXVKa0RMR1d5WjBlVVhseEZzdG4xZlRSdzhvS3pxV2dFYkhhN0NTeDVQMk0yZWNwdWduNWdDSVk4THJCb2xqM1hkdzhSdDk5Q1RUc3l3Vmo5TVNUdE1FRlA5QVk4YVV4Sk1SV1dOelZoYWpEcEtvcFJ3RmZXT2VTOTg0VzJVSUlBZWNsUEVsUnE=" }, "kind": "Secret", "metadata": { "creationTimestamp": null, "name": "logging-kibana-proxy" }, "type": "Opaque" }, "state": "present" } TASK [openshift_logging_kibana : Generate Kibana DC template] ****************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:221 changed: [openshift] => { "changed": true, "checksum": "38be8830d08f849a469d733ed00d5469e7c3de3a", "dest": "/tmp/openshift-logging-ansible-gvBLkA/templates/kibana-dc.yaml", "gid": 0, "group": "root", "md5sum": "ab1fa1dda7de48784fbc118a5e1d2fda", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 3763, "src": "/root/.ansible/tmp/ansible-tmp-1497026226.04-156667358573009/source", "state": "file", "uid": 0 } TASK [openshift_logging_kibana : Set Kibana DC] ******************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:240 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get dc logging-kibana-ops -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "DeploymentConfig", "metadata": { "creationTimestamp": "2017-06-09T16:37:07Z", "generation": 2, "labels": { "component": "kibana-ops", "logging-infra": "kibana", "provider": "openshift" }, "name": "logging-kibana-ops", "namespace": "logging", "resourceVersion": "1489", "selfLink": "/oapi/v1/namespaces/logging/deploymentconfigs/logging-kibana-ops", "uid": "e093e489-4d31-11e7-8061-0ed679898f5a" }, "spec": { "replicas": 1, "selector": { "component": "kibana-ops", "logging-infra": "kibana", "provider": "openshift" }, "strategy": { "activeDeadlineSeconds": 21600, "resources": {}, "rollingParams": { "intervalSeconds": 1, "maxSurge": "25%", "maxUnavailable": "25%", "timeoutSeconds": 600, "updatePeriodSeconds": 1 }, "type": "Rolling" }, "template": { "metadata": { "creationTimestamp": null, "labels": { "component": "kibana-ops", "logging-infra": "kibana", "provider": "openshift" }, "name": "logging-kibana-ops" }, "spec": { "containers": [ { "env": [ { "name": "ES_HOST", "value": "logging-es-ops" }, { "name": "ES_PORT", "value": "9200" }, { "name": "KIBANA_MEMORY_LIMIT", "valueFrom": { "resourceFieldRef": { "containerName": "kibana", "divisor": "0", "resource": "limits.memory" } } } ], "image": "172.30.180.71:5000/logging/logging-kibana:latest", "imagePullPolicy": "Always", "name": "kibana", "readinessProbe": { "exec": { "command": [ "/usr/share/kibana/probe/readiness.sh" ] }, "failureThreshold": 3, "initialDelaySeconds": 5, "periodSeconds": 5, "successThreshold": 1, "timeoutSeconds": 4 }, "resources": { "limits": { "memory": "736Mi" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/etc/kibana/keys", "name": "kibana", "readOnly": true } ] }, { "env": [ { "name": "OAP_BACKEND_URL", "value": "http://localhost:5601" }, { "name": "OAP_AUTH_MODE", "value": "oauth2" }, { "name": "OAP_TRANSFORM", "value": "user_header,token_header" }, { "name": "OAP_OAUTH_ID", "value": "kibana-proxy" }, { "name": "OAP_MASTER_URL", "value": "https://kubernetes.default.svc.cluster.local" }, { "name": "OAP_PUBLIC_MASTER_URL", "value": "https://172.18.12.108:8443" }, { "name": "OAP_LOGOUT_REDIRECT", "value": "https://172.18.12.108:8443/console/logout" }, { "name": "OAP_MASTER_CA_FILE", "value": "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" }, { "name": "OAP_DEBUG", "value": "False" }, { "name": "OAP_OAUTH_SECRET_FILE", "value": "/secret/oauth-secret" }, { "name": "OAP_SERVER_CERT_FILE", "value": "/secret/server-cert" }, { "name": "OAP_SERVER_KEY_FILE", "value": "/secret/server-key" }, { "name": "OAP_SERVER_TLS_FILE", "value": "/secret/server-tls.json" }, { "name": "OAP_SESSION_SECRET_FILE", "value": "/secret/session-secret" }, { "name": "OCP_AUTH_PROXY_MEMORY_LIMIT", "valueFrom": { "resourceFieldRef": { "containerName": "kibana-proxy", "divisor": "0", "resource": "limits.memory" } } } ], "image": "172.30.180.71:5000/logging/logging-auth-proxy:latest", "imagePullPolicy": "Always", "name": "kibana-proxy", "ports": [ { "containerPort": 3000, "name": "oaproxy", "protocol": "TCP" } ], "resources": { "limits": { "memory": "96Mi" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/secret", "name": "kibana-proxy", "readOnly": true } ] } ], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "aggregated-logging-kibana", "serviceAccountName": "aggregated-logging-kibana", "terminationGracePeriodSeconds": 30, "volumes": [ { "name": "kibana", "secret": { "defaultMode": 420, "secretName": "logging-kibana" } }, { "name": "kibana-proxy", "secret": { "defaultMode": 420, "secretName": "logging-kibana-proxy" } } ] } }, "test": false, "triggers": [ { "type": "ConfigChange" } ] }, "status": { "availableReplicas": 0, "conditions": [ { "lastTransitionTime": "2017-06-09T16:37:07Z", "lastUpdateTime": "2017-06-09T16:37:07Z", "message": "Deployment config does not have minimum availability.", "status": "False", "type": "Available" }, { "lastTransitionTime": "2017-06-09T16:37:07Z", "lastUpdateTime": "2017-06-09T16:37:07Z", "message": "replication controller \"logging-kibana-ops-1\" is waiting for pod \"logging-kibana-ops-1-deploy\" to run", "status": "Unknown", "type": "Progressing" } ], "details": { "causes": [ { "type": "ConfigChange" } ], "message": "config change" }, "latestVersion": 1, "observedGeneration": 2, "replicas": 0, "unavailableReplicas": 0, "updatedReplicas": 0 } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_kibana : Delete temp directory] ************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_kibana/tasks/main.yaml:252 ok: [openshift] => { "changed": false, "path": "/tmp/openshift-logging-ansible-gvBLkA", "state": "absent" } TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:195 statically included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml TASK [openshift_logging_curator : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml:3 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml:7 ok: [openshift] => { "ansible_facts": { "curator_version": "3_5" }, "changed": false } TASK [openshift_logging_curator : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml:15 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : Create temp directory for doing work in] ***** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:5 ok: [openshift] => { "changed": false, "cmd": [ "mktemp", "-d", "/tmp/openshift-logging-ansible-XXXXXX" ], "delta": "0:00:00.005902", "end": "2017-06-09 12:37:09.534909", "rc": 0, "start": "2017-06-09 12:37:09.529007" } STDOUT: /tmp/openshift-logging-ansible-99h6AJ TASK [openshift_logging_curator : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:10 ok: [openshift] => { "ansible_facts": { "tempdir": "/tmp/openshift-logging-ansible-99h6AJ" }, "changed": false } TASK [openshift_logging_curator : Create templates subdirectory] *************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:14 ok: [openshift] => { "changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/tmp/openshift-logging-ansible-99h6AJ/templates", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 6, "state": "directory", "uid": 0 } TASK [openshift_logging_curator : Create Curator service account] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:24 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : Create Curator service account] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:32 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get sa aggregated-logging-curator -o json -n logging", "results": [ { "apiVersion": "v1", "imagePullSecrets": [ { "name": "aggregated-logging-curator-dockercfg-rsv0j" } ], "kind": "ServiceAccount", "metadata": { "creationTimestamp": "2017-06-09T16:37:10Z", "name": "aggregated-logging-curator", "namespace": "logging", "resourceVersion": "1501", "selfLink": "/api/v1/namespaces/logging/serviceaccounts/aggregated-logging-curator", "uid": "e2594458-4d31-11e7-8061-0ed679898f5a" }, "secrets": [ { "name": "aggregated-logging-curator-token-4xg8m" }, { "name": "aggregated-logging-curator-dockercfg-rsv0j" } ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_curator : copy] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:41 ok: [openshift] => { "changed": false, "checksum": "9008efd9a8892dcc42c28c6dfb6708527880a6d8", "dest": "/tmp/openshift-logging-ansible-99h6AJ/curator.yml", "gid": 0, "group": "root", "md5sum": "5498c5fd98f3dd06e34b20eb1f55dc12", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 320, "src": "/root/.ansible/tmp/ansible-tmp-1497026230.99-25942318222518/source", "state": "file", "uid": 0 } TASK [openshift_logging_curator : copy] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:47 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : Set Curator configmap] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:53 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get configmap logging-curator -o json -n logging", "results": [ { "apiVersion": "v1", "data": { "config.yaml": "# Logging example curator config file\n\n# uncomment and use this to override the defaults from env vars\n#.defaults:\n# delete:\n# days: 30\n# runhour: 0\n# runminute: 0\n\n# to keep ops logs for a different duration:\n#.operations:\n# delete:\n# weeks: 8\n\n# example for a normal project\n#myapp:\n# delete:\n# weeks: 1\n" }, "kind": "ConfigMap", "metadata": { "creationTimestamp": "2017-06-09T16:37:12Z", "name": "logging-curator", "namespace": "logging", "resourceVersion": "1518", "selfLink": "/api/v1/namespaces/logging/configmaps/logging-curator", "uid": "e3adb29f-4d31-11e7-8061-0ed679898f5a" } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_curator : Set Curator secret] ************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:62 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc secrets new logging-curator ca=/etc/origin/logging/ca.crt key=/etc/origin/logging/system.logging.curator.key cert=/etc/origin/logging/system.logging.curator.crt -n logging", "results": "", "returncode": 0 }, "state": "present" } TASK [openshift_logging_curator : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:75 ok: [openshift] => { "ansible_facts": { "curator_component": "curator", "curator_name": "logging-curator" }, "changed": false } TASK [openshift_logging_curator : Generate Curator deploymentconfig] *********** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:81 ok: [openshift] => { "changed": false, "checksum": "05b4981fda122d80e388a1f7493a3fbf00d26fe3", "dest": "/tmp/openshift-logging-ansible-99h6AJ/templates/curator-dc.yaml", "gid": 0, "group": "root", "md5sum": "8ac5452ba16cf770e81ffdd9471c1c8e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 2340, "src": "/root/.ansible/tmp/ansible-tmp-1497026233.71-112934955792782/source", "state": "file", "uid": 0 } TASK [openshift_logging_curator : Set Curator DC] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:99 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get dc logging-curator -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "DeploymentConfig", "metadata": { "creationTimestamp": "2017-06-09T16:37:14Z", "generation": 2, "labels": { "component": "curator", "logging-infra": "curator", "provider": "openshift" }, "name": "logging-curator", "namespace": "logging", "resourceVersion": "1536", "selfLink": "/oapi/v1/namespaces/logging/deploymentconfigs/logging-curator", "uid": "e4d3a677-4d31-11e7-8061-0ed679898f5a" }, "spec": { "replicas": 1, "selector": { "component": "curator", "logging-infra": "curator", "provider": "openshift" }, "strategy": { "activeDeadlineSeconds": 21600, "recreateParams": { "timeoutSeconds": 600 }, "resources": {}, "rollingParams": { "intervalSeconds": 1, "maxSurge": "25%", "maxUnavailable": "25%", "timeoutSeconds": 600, "updatePeriodSeconds": 1 }, "type": "Recreate" }, "template": { "metadata": { "creationTimestamp": null, "labels": { "component": "curator", "logging-infra": "curator", "provider": "openshift" }, "name": "logging-curator" }, "spec": { "containers": [ { "env": [ { "name": "K8S_HOST_URL", "value": "https://kubernetes.default.svc.cluster.local" }, { "name": "ES_HOST", "value": "logging-es" }, { "name": "ES_PORT", "value": "9200" }, { "name": "ES_CLIENT_CERT", "value": "/etc/curator/keys/cert" }, { "name": "ES_CLIENT_KEY", "value": "/etc/curator/keys/key" }, { "name": "ES_CA", "value": "/etc/curator/keys/ca" }, { "name": "CURATOR_DEFAULT_DAYS", "value": "30" }, { "name": "CURATOR_RUN_HOUR", "value": "0" }, { "name": "CURATOR_RUN_MINUTE", "value": "0" }, { "name": "CURATOR_RUN_TIMEZONE", "value": "UTC" }, { "name": "CURATOR_SCRIPT_LOG_LEVEL", "value": "INFO" }, { "name": "CURATOR_LOG_LEVEL", "value": "ERROR" } ], "image": "172.30.180.71:5000/logging/logging-curator:latest", "imagePullPolicy": "Always", "name": "curator", "resources": { "limits": { "cpu": "100m" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/etc/curator/keys", "name": "certs", "readOnly": true }, { "mountPath": "/etc/curator/settings", "name": "config", "readOnly": true } ] } ], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "aggregated-logging-curator", "serviceAccountName": "aggregated-logging-curator", "terminationGracePeriodSeconds": 30, "volumes": [ { "name": "certs", "secret": { "defaultMode": 420, "secretName": "logging-curator" } }, { "configMap": { "defaultMode": 420, "name": "logging-curator" }, "name": "config" } ] } }, "test": false, "triggers": [ { "type": "ConfigChange" } ] }, "status": { "availableReplicas": 0, "conditions": [ { "lastTransitionTime": "2017-06-09T16:37:14Z", "lastUpdateTime": "2017-06-09T16:37:14Z", "message": "Deployment config does not have minimum availability.", "status": "False", "type": "Available" }, { "lastTransitionTime": "2017-06-09T16:37:14Z", "lastUpdateTime": "2017-06-09T16:37:14Z", "message": "replication controller \"logging-curator-1\" is waiting for pod \"logging-curator-1-deploy\" to run", "status": "Unknown", "type": "Progressing" } ], "details": { "causes": [ { "type": "ConfigChange" } ], "message": "config change" }, "latestVersion": 1, "observedGeneration": 2, "replicas": 0, "unavailableReplicas": 0, "updatedReplicas": 0 } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_curator : Delete temp directory] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:109 ok: [openshift] => { "changed": false, "path": "/tmp/openshift-logging-ansible-99h6AJ", "state": "absent" } TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:207 statically included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml TASK [openshift_logging_curator : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml:3 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml:7 ok: [openshift] => { "ansible_facts": { "curator_version": "3_5" }, "changed": false } TASK [openshift_logging_curator : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/determine_version.yaml:15 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : Create temp directory for doing work in] ***** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:5 ok: [openshift] => { "changed": false, "cmd": [ "mktemp", "-d", "/tmp/openshift-logging-ansible-XXXXXX" ], "delta": "0:00:00.002503", "end": "2017-06-09 12:37:17.269258", "rc": 0, "start": "2017-06-09 12:37:17.266755" } STDOUT: /tmp/openshift-logging-ansible-cb13rT TASK [openshift_logging_curator : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:10 ok: [openshift] => { "ansible_facts": { "tempdir": "/tmp/openshift-logging-ansible-cb13rT" }, "changed": false } TASK [openshift_logging_curator : Create templates subdirectory] *************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:14 ok: [openshift] => { "changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/tmp/openshift-logging-ansible-cb13rT/templates", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 6, "state": "directory", "uid": 0 } TASK [openshift_logging_curator : Create Curator service account] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:24 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : Create Curator service account] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:32 ok: [openshift] => { "changed": false, "results": { "cmd": "/bin/oc get sa aggregated-logging-curator -o json -n logging", "results": [ { "apiVersion": "v1", "imagePullSecrets": [ { "name": "aggregated-logging-curator-dockercfg-rsv0j" } ], "kind": "ServiceAccount", "metadata": { "creationTimestamp": "2017-06-09T16:37:10Z", "name": "aggregated-logging-curator", "namespace": "logging", "resourceVersion": "1501", "selfLink": "/api/v1/namespaces/logging/serviceaccounts/aggregated-logging-curator", "uid": "e2594458-4d31-11e7-8061-0ed679898f5a" }, "secrets": [ { "name": "aggregated-logging-curator-token-4xg8m" }, { "name": "aggregated-logging-curator-dockercfg-rsv0j" } ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_curator : copy] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:41 ok: [openshift] => { "changed": false, "checksum": "9008efd9a8892dcc42c28c6dfb6708527880a6d8", "dest": "/tmp/openshift-logging-ansible-cb13rT/curator.yml", "gid": 0, "group": "root", "md5sum": "5498c5fd98f3dd06e34b20eb1f55dc12", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 320, "src": "/root/.ansible/tmp/ansible-tmp-1497026238.06-60658083755193/source", "state": "file", "uid": 0 } TASK [openshift_logging_curator : copy] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:47 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_curator : Set Curator configmap] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:53 ok: [openshift] => { "changed": false, "results": { "cmd": "/bin/oc get configmap logging-curator -o json -n logging", "results": [ { "apiVersion": "v1", "data": { "config.yaml": "# Logging example curator config file\n\n# uncomment and use this to override the defaults from env vars\n#.defaults:\n# delete:\n# days: 30\n# runhour: 0\n# runminute: 0\n\n# to keep ops logs for a different duration:\n#.operations:\n# delete:\n# weeks: 8\n\n# example for a normal project\n#myapp:\n# delete:\n# weeks: 1\n" }, "kind": "ConfigMap", "metadata": { "creationTimestamp": "2017-06-09T16:37:12Z", "name": "logging-curator", "namespace": "logging", "resourceVersion": "1518", "selfLink": "/api/v1/namespaces/logging/configmaps/logging-curator", "uid": "e3adb29f-4d31-11e7-8061-0ed679898f5a" } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_curator : Set Curator secret] ************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:62 ok: [openshift] => { "changed": false, "results": { "apiVersion": "v1", "data": { "ca": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMyakNDQWNLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpVMU5sb1hEVEl5TURZd09ERTJNelUxTjFvdwpIakVjTUJvR0ExVUVBeE1UYkc5bloybHVaeTF6YVdkdVpYSXRkR1Z6ZERDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1IRDJVYWdmRzVYQitRYnRkcTc4TXRmUWMxRVkydHRaZVdzNHdHTGhIRUMKR1VzMFFpZEVzZ0k3cmo1a2ZySDdrQ01sZnd2N20yRmpYL1grNGJJN2g5QlhhY05Mc2RKUkRNbUhVLytzNjdBKwo1TWM1VjFJU0VqYWNMeG5PZmZrNTNzVGg0U282WkNrQmkrcDlqVmxTN1hCdi9nV2doaU1odEVpLzJoaUIybE9iCmplcCsrTWVKVDRCQVJGNjI3OTRnV0w3eklNc2t3a1ZhYkxKYUlubWtWSVZLOTJqOEdTaVJDa0VNcUJJNFZqRDgKQnVRa09pN3pHZWxlQ3RDMTRMTW01T21NMkx3Vi9ERktpY2hBRjBuVkxLRG9TWUNtZmVIalNLeUVVVk54d01lagpsM3ZDYjRjOTRRcCtXWlZFQSt4UE91T2p1eFIrUkFmN2p0bVl4dXd2c0M4Q0F3RUFBYU1qTUNFd0RnWURWUjBQCkFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFGcWYKZU1HZEdVellycHFIT3BnTzhvdzFEai9GZzA5dG5yd2dVQUZwd0NmQ1lheXdxUFJYUmdtMHBVUngrTFl2UnI1SQpSbWEvYzI4LzhjbVdXUG5ReEFHOUYyWVFnZU5TbExZNWY0U2NJemFzaDVQMWZIQlVRWUlWUzNkWnB1QXJrR1hOCkpySkszbFBHY0hWNm5ZRys5ZUs3b1FWS0Nrd1IvWnMvQ21FSHNOTGdPcDZRSFhPS0hTcWtlMmpwNHZmSG1TdC8Kb0R4VFRVSFIzN3g2cTEyTTRoYVhYM2NTWCthYkNRYzFZUU5kcUxiZHhYbjZ6OE14dVptRkI5YnJnSkVoU2N2Rgp2VHh6TnRpUnkzVHNkZUxROForWjF0cHNiR1NvSmFPUFJUcit3ZWNzM3RETEpuR1hwaGc0NHc2UW0zNnZkNENtCjhoaTN6ODczbEdSNmVGL05ZN1U9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K", "cert": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURSakNDQWk2Z0F3SUJBZ0lCQkRBTkJna3Foa2lHOXcwQkFRVUZBREFlTVJ3d0dnWURWUVFERXhOc2IyZG4KYVc1bkxYTnBaMjVsY2kxMFpYTjBNQjRYRFRFM01EWXdPVEUyTXpZd00xb1hEVEU1TURZd09URTJNell3TTFvdwpSekVRTUE0R0ExVUVDZ3dIVEc5bloybHVaekVTTUJBR0ExVUVDd3dKVDNCbGJsTm9hV1owTVI4d0hRWURWUVFECkRCWnplWE4wWlcwdWJHOW5aMmx1Wnk1amRYSmhkRzl5TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEEKTUlJQkNnS0NBUUVBdGhTUWhTTEgrTUtjcFZObGxlaEp0MnJoU0Q3RlJNNVpNcW14UzBmUW9jNk5BcExCQlJlLwoxWi9HeVlEYTdkQnBqNkRsT2hjaDBUK0tua3FUcDh3WisvT3lKVzNKcUVDUk1sbjdxNTBHa0tEdktmRjh5N0toCkFLR0JjcWxXejI3RTlSM0xweFh1U1pXQkVWaURpbExwcWVNWTVQeUZySUR6TExWZS9uVW9RbnZBN0R0SzExMXIKVlJvZ3RIYWtlZVBoQ3NKZkJZRThabTFHcFc0Z24wWGw5OWZ4b3YvNW5JSXE5VUt2cDhyblBRdVNZZmN5ZDN0ZQpkY3h6anA0UVB4dU1mK2hwREJBS1BlL05pMWI1L3N1ZndPaGxMcnZ1VDhGTFhQSGt3SEh0UTlWQ0R2ZWlpb1RXCmJjcmtHdUROemFhS3pEaGxQWi9nNkNKdFJaUWkxeDU5bFFJREFRQUJvMll3WkRBT0JnTlZIUThCQWY4RUJBTUMKQmFBd0NRWURWUjBUQkFJd0FEQWRCZ05WSFNVRUZqQVVCZ2dyQmdFRkJRY0RBUVlJS3dZQkJRVUhBd0l3SFFZRApWUjBPQkJZRUZDV3E0MVFIaWEwa3NUWm5lWFpETG9ReWR1bHBNQWtHQTFVZEl3UUNNQUF3RFFZSktvWklodmNOCkFRRUZCUUFEZ2dFQkFDcUtjM0hXTW1hbW5uUVBzcUtzN1ZHaUVXN2RIc1h6RitvaUNEajgvZUlWZGhoRlNhYkgKcWhqcDlZQ2x0YVJRMjhPdmdqdkZ4dHpPVnZaeG01eGlLTEYxL25qQUtsYk5Rb3RLcVh4SDB3dDVsdHpkVjJ6QQpGRnR4VGlSN1JId0RlSTRQNG1LTFc4ZEZROU1pcU9USG51cjR1bFdadWlEQjMxSnY0OHhldElzcE1vWERIUUZrCkVZa3NCcmFqdHhzMVFuTjIxbXZZRm5Lbzc2Mk4yVjdLMjFsOHNuR2VZZGltQ01MQ3o0eWpyTG0zQVRWMXBTRmEKVjVyRDAxakRTMzNjWFJVTTFNSXM5Q2N2dkpQUUI2dnEvVU10NytIdEF5c3hwMnJYYXA1SDl2M2tVdzBJbnJEMQpaRW1uTGVTcCtPNWxEVUNmeHlEU0k0d2pBOXJhemZJM1NYaz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=", "key": "LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzJGSkNGSXNmNHdweWwKVTJXVjZFbTNhdUZJUHNWRXpsa3lxYkZMUjlDaHpvMENrc0VGRjcvVm44YkpnTnJ0MEdtUG9PVTZGeUhSUDRxZQpTcE9uekJuNzg3SWxiY21vUUpFeVdmdXJuUWFRb084cDhYekxzcUVBb1lGeXFWYlBic1QxSGN1bkZlNUpsWUVSCldJT0tVdW1wNHhqay9JV3NnUE1zdFY3K2RTaENlOERzTzByWFhXdFZHaUMwZHFSNTQrRUt3bDhGZ1R4bWJVYWwKYmlDZlJlWDMxL0dpLy9tY2dpcjFRcStueXVjOUM1Smg5ekozZTE1MXpIT09uaEEvRzR4LzZHa01FQW85NzgyTApWdm4reTUvQTZHVXV1KzVQd1V0YzhlVEFjZTFEMVVJTzk2S0toTlp0eXVRYTRNM05wb3JNT0dVOW4rRG9JbTFGCmxDTFhIbjJWQWdNQkFBRUNnZ0VBQzU1b1VuYVR1Q1lEQlBHN2E4TjA0TVZyTnNiNm81ZGZUY0pTVS9Gek9NQUwKYzN5VjRKTlJaSHBibndRZjlFajJVM242NXFnUE9tRUpvV3cwMitKRVRqV1F5cDZrM1oyeEpqdGp1ZUc5UjRGZApnWm10VXZwRkY5czJmWENjMzRPOEFjQXkvS2lEbWlKMkovc3orYUdzMDNCd2FEeE04Mk1QSVcwNWNKZGRTRkN0CmxXcGNSWXVSMjF0R1ZJdDRLVUhOUHVYYlVabGI5SWNYN09HbG9mSk5yK2pzN3Q0Ni9GVFB0NDkyK1RqbitXbzIKdmlFSWdHN0g1YTZVQ0NFb2ROTlV1ejd1OWN3UFdHcnhlWVFUbXFXeWpsYUNtTFI1VlVLTmJwR1EwcTBXOXVKTQpUQ0x5d0V5THQ3SVprcGRxdW1HMU5VNEZwcDh6L2UvbUU0VlZSSkxqQVFLQmdRRHg1WDRZMGFXdEpWSHVuTzZmCk1PRVlYVm8wOW02M2UzNFVUYkxsS2xybFRoQWYyc0FNOUMwZ2lObitzUVNlK1ZtL05XN0YzNll5OTRnYmVGQ0IKSlNZZTFlZjh1M28wUXo3Y3drZlpSNGVRVG9nYU0xaWM5TGZyT3FYeEhZWkNTYzI0VjlnY3BreWtFK2Y2V2xJQgpPcHh1TTFhR1c3aHR1cURURmpzS2FoOUJkUUtCZ1FEQXNrUWh2cmZtNWlIMHBMUnJXTTU1eWp1V01va081eE1wCnhqQmRvN0NnVEZSUkh4cDRVSzlrQk9JN0pROXIvb0t1RG8wcEMzTTB4UUNKSTFTQWFTNy9VZDk2R3NhODJodDEKOWVWUGorTG1ia1ZOSElBMWRUNVlXaStWMXpRTHNwTm5Db3MwNDZhQkxsQUs0Mk8rMytZRkpyUG1yYkptZjJzVApEdkM4dTFlbm9RS0JnRkJvZ2Q4TUtXK0IzVmRkRVRFWGxXdDg5SlJ3YW1OUmlKcUZMWGRoK2U4SllQREZ6VnRBCis1TTZDb1RRSVFJYlR3cnhYdzRsaHVDZ1NiMGY1RnJabDI5ZW1ZdjRZZW5uaElqcmgraWhEMTRYek1RdzF6N3QKVmdtNmFCamJRSnJlWDZUc2hta1htcGhjQ1p1OTFFNldSaE9YdXlaVWJOMEUwWnI1WTh6VmMraFZBb0dCQUxaUApzd0JPUmdZU1U2djNySzhuZldwaStNOVpEcTFuMkl5Y2N5UVUrOFlka1dTU3plTnBTeGVSTzAyRCtieE9OdXRnCjN4TW5QeDk5QjgyMy9IbmxpazZMOHFpKzdxcThuR0NWZmhneFpSWm8yKzVoK2JjYUx4cytjR2tQQzdISmlZWmkKeG1RS3duelVDV3pUeUlwTm9YM2NLVGNsdlhVV05vTDRqckd4ZWZRaEFvR0FjRHd4WWdnNkxwc01uNG5kblRBegpFV2lkS0dHdnBxZ2VCaHJlSE5FcC9HQmZ5YnNmbk1yWWdzdTlVVHEwaSt2Z0N6YWtabkdRTGVhUStsYndkajgyCnFjc3g1c3hEM1g0dEl0WXRLVEd3UVhoR0FlbDlFc2lwaHoveHl1SG1GYVhkVEZOMEZ5NkZra1I3dU44ak1mTGgKZ1lvTFBLeUdNSEtHdnJwL05ic3BNZTg9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K" }, "kind": "Secret", "metadata": { "creationTimestamp": null, "name": "logging-curator" }, "type": "Opaque" }, "state": "present" } TASK [openshift_logging_curator : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:75 ok: [openshift] => { "ansible_facts": { "curator_component": "curator-ops", "curator_name": "logging-curator-ops" }, "changed": false } TASK [openshift_logging_curator : Generate Curator deploymentconfig] *********** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:81 ok: [openshift] => { "changed": false, "checksum": "e5fecbf56665462ecd330358b03177886060e9a4", "dest": "/tmp/openshift-logging-ansible-cb13rT/templates/curator-dc.yaml", "gid": 0, "group": "root", "md5sum": "7472f3a9640421adee1869d8e011e4a7", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 2364, "src": "/root/.ansible/tmp/ansible-tmp-1497026239.54-242758577743628/source", "state": "file", "uid": 0 } TASK [openshift_logging_curator : Set Curator DC] ****************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:99 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get dc logging-curator-ops -o json -n logging", "results": [ { "apiVersion": "v1", "kind": "DeploymentConfig", "metadata": { "creationTimestamp": "2017-06-09T16:37:20Z", "generation": 2, "labels": { "component": "curator-ops", "logging-infra": "curator", "provider": "openshift" }, "name": "logging-curator-ops", "namespace": "logging", "resourceVersion": "1558", "selfLink": "/oapi/v1/namespaces/logging/deploymentconfigs/logging-curator-ops", "uid": "e8465547-4d31-11e7-8061-0ed679898f5a" }, "spec": { "replicas": 1, "selector": { "component": "curator-ops", "logging-infra": "curator", "provider": "openshift" }, "strategy": { "activeDeadlineSeconds": 21600, "recreateParams": { "timeoutSeconds": 600 }, "resources": {}, "rollingParams": { "intervalSeconds": 1, "maxSurge": "25%", "maxUnavailable": "25%", "timeoutSeconds": 600, "updatePeriodSeconds": 1 }, "type": "Recreate" }, "template": { "metadata": { "creationTimestamp": null, "labels": { "component": "curator-ops", "logging-infra": "curator", "provider": "openshift" }, "name": "logging-curator-ops" }, "spec": { "containers": [ { "env": [ { "name": "K8S_HOST_URL", "value": "https://kubernetes.default.svc.cluster.local" }, { "name": "ES_HOST", "value": "logging-es-ops" }, { "name": "ES_PORT", "value": "9200" }, { "name": "ES_CLIENT_CERT", "value": "/etc/curator/keys/cert" }, { "name": "ES_CLIENT_KEY", "value": "/etc/curator/keys/key" }, { "name": "ES_CA", "value": "/etc/curator/keys/ca" }, { "name": "CURATOR_DEFAULT_DAYS", "value": "30" }, { "name": "CURATOR_RUN_HOUR", "value": "0" }, { "name": "CURATOR_RUN_MINUTE", "value": "0" }, { "name": "CURATOR_RUN_TIMEZONE", "value": "UTC" }, { "name": "CURATOR_SCRIPT_LOG_LEVEL", "value": "INFO" }, { "name": "CURATOR_LOG_LEVEL", "value": "ERROR" } ], "image": "172.30.180.71:5000/logging/logging-curator:latest", "imagePullPolicy": "Always", "name": "curator", "resources": { "limits": { "cpu": "100m" } }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/etc/curator/keys", "name": "certs", "readOnly": true }, { "mountPath": "/etc/curator/settings", "name": "config", "readOnly": true } ] } ], "dnsPolicy": "ClusterFirst", "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "aggregated-logging-curator", "serviceAccountName": "aggregated-logging-curator", "terminationGracePeriodSeconds": 30, "volumes": [ { "name": "certs", "secret": { "defaultMode": 420, "secretName": "logging-curator" } }, { "configMap": { "defaultMode": 420, "name": "logging-curator" }, "name": "config" } ] } }, "test": false, "triggers": [ { "type": "ConfigChange" } ] }, "status": { "availableReplicas": 0, "conditions": [ { "lastTransitionTime": "2017-06-09T16:37:20Z", "lastUpdateTime": "2017-06-09T16:37:20Z", "message": "Deployment config does not have minimum availability.", "status": "False", "type": "Available" }, { "lastTransitionTime": "2017-06-09T16:37:20Z", "lastUpdateTime": "2017-06-09T16:37:20Z", "message": "replication controller \"logging-curator-ops-1\" is waiting for pod \"logging-curator-ops-1-deploy\" to run", "status": "Unknown", "type": "Progressing" } ], "details": { "causes": [ { "type": "ConfigChange" } ], "message": "config change" }, "latestVersion": 1, "observedGeneration": 2, "replicas": 0, "unavailableReplicas": 0, "updatedReplicas": 0 } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_curator : Delete temp directory] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_curator/tasks/main.yaml:109 ok: [openshift] => { "changed": false, "path": "/tmp/openshift-logging-ansible-cb13rT", "state": "absent" } TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:226 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : include_role] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:241 statically included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/determine_version.yaml TASK [openshift_logging_fluentd : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:2 [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ openshift_logging_fluentd_nodeselector.keys() | count }} > 1 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:6 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:10 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:14 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/determine_version.yaml:3 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/determine_version.yaml:7 ok: [openshift] => { "ansible_facts": { "fluentd_version": "3_5" }, "changed": false } TASK [openshift_logging_fluentd : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/determine_version.yaml:12 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : fail] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/determine_version.yaml:15 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:20 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:26 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : Create temp directory for doing work in] ***** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:33 ok: [openshift] => { "changed": false, "cmd": [ "mktemp", "-d", "/tmp/openshift-logging-ansible-XXXXXX" ], "delta": "0:00:00.002728", "end": "2017-06-09 12:37:24.252191", "rc": 0, "start": "2017-06-09 12:37:24.249463" } STDOUT: /tmp/openshift-logging-ansible-x88aJU TASK [openshift_logging_fluentd : set_fact] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:38 ok: [openshift] => { "ansible_facts": { "tempdir": "/tmp/openshift-logging-ansible-x88aJU" }, "changed": false } TASK [openshift_logging_fluentd : Create templates subdirectory] *************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:41 ok: [openshift] => { "changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/tmp/openshift-logging-ansible-x88aJU/templates", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 6, "state": "directory", "uid": 0 } TASK [openshift_logging_fluentd : Create Fluentd service account] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:51 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : Create Fluentd service account] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:59 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get sa aggregated-logging-fluentd -o json -n logging", "results": [ { "apiVersion": "v1", "imagePullSecrets": [ { "name": "aggregated-logging-fluentd-dockercfg-tbhns" } ], "kind": "ServiceAccount", "metadata": { "creationTimestamp": "2017-06-09T16:37:25Z", "name": "aggregated-logging-fluentd", "namespace": "logging", "resourceVersion": "1583", "selfLink": "/api/v1/namespaces/logging/serviceaccounts/aggregated-logging-fluentd", "uid": "eb3324e1-4d31-11e7-8061-0ed679898f5a" }, "secrets": [ { "name": "aggregated-logging-fluentd-dockercfg-tbhns" }, { "name": "aggregated-logging-fluentd-token-9qm1d" } ] } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_fluentd : Set privileged permissions for Fluentd] ****** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:68 changed: [openshift] => { "changed": true, "present": "present", "results": { "cmd": "/bin/oc adm policy add-scc-to-user privileged system:serviceaccount:logging:aggregated-logging-fluentd -n logging", "results": "", "returncode": 0 } } TASK [openshift_logging_fluentd : Set cluster-reader permissions for Fluentd] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:77 changed: [openshift] => { "changed": true, "present": "present", "results": { "cmd": "/bin/oc adm policy add-cluster-role-to-user cluster-reader system:serviceaccount:logging:aggregated-logging-fluentd -n logging", "results": "", "returncode": 0 } } TASK [openshift_logging_fluentd : template] ************************************ task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:86 ok: [openshift] => { "changed": false, "checksum": "a8c8596f5fc2c5dd7c8d33d244af17a2555be086", "dest": "/tmp/openshift-logging-ansible-x88aJU/fluent.conf", "gid": 0, "group": "root", "md5sum": "579698b48ffce6276ee0e8d5ac71a338", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 1301, "src": "/root/.ansible/tmp/ansible-tmp-1497026247.07-280120135852772/source", "state": "file", "uid": 0 } TASK [openshift_logging_fluentd : copy] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:94 ok: [openshift] => { "changed": false, "checksum": "b3e75eddc4a0765edc77da092384c0c6f95440e1", "dest": "/tmp/openshift-logging-ansible-x88aJU/fluentd-throttle-config.yaml", "gid": 0, "group": "root", "md5sum": "25871b8e0a9bedc166a6029872a6c336", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 133, "src": "/root/.ansible/tmp/ansible-tmp-1497026247.54-242975930420781/source", "state": "file", "uid": 0 } TASK [openshift_logging_fluentd : copy] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:100 ok: [openshift] => { "changed": false, "checksum": "a3aa36da13f3108aa4ad5b98d4866007b44e9798", "dest": "/tmp/openshift-logging-ansible-x88aJU/secure-forward.conf", "gid": 0, "group": "root", "md5sum": "1084b00c427f4fa48dfc66d6ad6555d4", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 563, "src": "/root/.ansible/tmp/ansible-tmp-1497026247.84-138139448879089/source", "state": "file", "uid": 0 } TASK [openshift_logging_fluentd : copy] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:107 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : copy] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:113 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : copy] **************************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:119 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging_fluentd : Set Fluentd configmap] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:125 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get configmap logging-fluentd -o json -n logging", "results": [ { "apiVersion": "v1", "data": { "fluent.conf": "# This file is the fluentd configuration entrypoint. Edit with care.\n\n@include configs.d/openshift/system.conf\n\n# In each section below, pre- and post- includes don't include anything initially;\n# they exist to enable future additions to openshift conf as needed.\n\n## sources\n## ordered so that syslog always runs last...\n@include configs.d/openshift/input-pre-*.conf\n@include configs.d/dynamic/input-docker-*.conf\n@include configs.d/dynamic/input-syslog-*.conf\n@include configs.d/openshift/input-post-*.conf\n##\n\n<label @INGRESS>\n## filters\n @include configs.d/openshift/filter-pre-*.conf\n @include configs.d/openshift/filter-retag-journal.conf\n @include configs.d/openshift/filter-k8s-meta.conf\n @include configs.d/openshift/filter-kibana-transform.conf\n @include configs.d/openshift/filter-k8s-flatten-hash.conf\n @include configs.d/openshift/filter-k8s-record-transform.conf\n @include configs.d/openshift/filter-syslog-record-transform.conf\n @include configs.d/openshift/filter-viaq-data-model.conf\n @include configs.d/openshift/filter-post-*.conf\n##\n\n## matches\n @include configs.d/openshift/output-pre-*.conf\n @include configs.d/openshift/output-operations.conf\n @include configs.d/openshift/output-applications.conf\n # no post - applications.conf matches everything left\n##\n</label>\n", "secure-forward.conf": "# @type secure_forward\n\n# self_hostname ${HOSTNAME}\n# shared_key <SECRET_STRING>\n\n# secure yes\n# enable_strict_verification yes\n\n# ca_cert_path /etc/fluent/keys/your_ca_cert\n# ca_private_key_path /etc/fluent/keys/your_private_key\n # for private CA secret key\n# ca_private_key_passphrase passphrase\n\n# <server>\n # or IP\n# host server.fqdn.example.com\n# port 24284\n# </server>\n# <server>\n # ip address to connect\n# host 203.0.113.8\n # specify hostlabel for FQDN verification if ipaddress is used for host\n# hostlabel server.fqdn.example.com\n# </server>\n", "throttle-config.yaml": "# Logging example fluentd throttling config file\n\n#example-project:\n# read_lines_limit: 10\n#\n#.operations:\n# read_lines_limit: 100\n" }, "kind": "ConfigMap", "metadata": { "creationTimestamp": "2017-06-09T16:37:28Z", "name": "logging-fluentd", "namespace": "logging", "resourceVersion": "1594", "selfLink": "/api/v1/namespaces/logging/configmaps/logging-fluentd", "uid": "ed3fa722-4d31-11e7-8061-0ed679898f5a" } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_fluentd : Set logging-fluentd secret] ****************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:137 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc secrets new logging-fluentd ca=/etc/origin/logging/ca.crt key=/etc/origin/logging/system.logging.fluentd.key cert=/etc/origin/logging/system.logging.fluentd.crt -n logging", "results": "", "returncode": 0 }, "state": "present" } TASK [openshift_logging_fluentd : Generate logging-fluentd daemonset definition] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:154 ok: [openshift] => { "changed": false, "checksum": "4afd779147618e7347626d410d32280290fa3c76", "dest": "/tmp/openshift-logging-ansible-x88aJU/templates/logging-fluentd.yaml", "gid": 0, "group": "root", "md5sum": "77ad5c303464ea244ac6680b2ce876a8", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 3414, "src": "/root/.ansible/tmp/ansible-tmp-1497026249.71-157684544104832/source", "state": "file", "uid": 0 } TASK [openshift_logging_fluentd : Set logging-fluentd daemonset] *************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:172 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc get daemonset logging-fluentd -o json -n logging", "results": [ { "apiVersion": "extensions/v1beta1", "kind": "DaemonSet", "metadata": { "creationTimestamp": "2017-06-09T16:37:30Z", "generation": 1, "labels": { "component": "fluentd", "logging-infra": "fluentd", "provider": "openshift" }, "name": "logging-fluentd", "namespace": "logging", "resourceVersion": "1598", "selfLink": "/apis/extensions/v1beta1/namespaces/logging/daemonsets/logging-fluentd", "uid": "ee654694-4d31-11e7-8061-0ed679898f5a" }, "spec": { "selector": { "matchLabels": { "component": "fluentd", "provider": "openshift" } }, "template": { "metadata": { "creationTimestamp": null, "labels": { "component": "fluentd", "logging-infra": "fluentd", "provider": "openshift" }, "name": "fluentd-elasticsearch" }, "spec": { "containers": [ { "env": [ { "name": "K8S_HOST_URL", "value": "https://kubernetes.default.svc.cluster.local" }, { "name": "ES_HOST", "value": "logging-es" }, { "name": "ES_PORT", "value": "9200" }, { "name": "ES_CLIENT_CERT", "value": "/etc/fluent/keys/cert" }, { "name": "ES_CLIENT_KEY", "value": "/etc/fluent/keys/key" }, { "name": "ES_CA", "value": "/etc/fluent/keys/ca" }, { "name": "OPS_HOST", "value": "logging-es-ops" }, { "name": "OPS_PORT", "value": "9200" }, { "name": "OPS_CLIENT_CERT", "value": "/etc/fluent/keys/cert" }, { "name": "OPS_CLIENT_KEY", "value": "/etc/fluent/keys/key" }, { "name": "OPS_CA", "value": "/etc/fluent/keys/ca" }, { "name": "ES_COPY", "value": "false" }, { "name": "USE_JOURNAL", "value": "true" }, { "name": "JOURNAL_SOURCE" }, { "name": "JOURNAL_READ_FROM_HEAD", "value": "false" } ], "image": "172.30.180.71:5000/logging/logging-fluentd:latest", "imagePullPolicy": "Always", "name": "fluentd-elasticsearch", "resources": { "limits": { "cpu": "100m", "memory": "512Mi" } }, "securityContext": { "privileged": true }, "terminationMessagePath": "/dev/termination-log", "terminationMessagePolicy": "File", "volumeMounts": [ { "mountPath": "/run/log/journal", "name": "runlogjournal" }, { "mountPath": "/var/log", "name": "varlog" }, { "mountPath": "/var/lib/docker/containers", "name": "varlibdockercontainers", "readOnly": true }, { "mountPath": "/etc/fluent/configs.d/user", "name": "config", "readOnly": true }, { "mountPath": "/etc/fluent/keys", "name": "certs", "readOnly": true }, { "mountPath": "/etc/docker-hostname", "name": "dockerhostname", "readOnly": true }, { "mountPath": "/etc/localtime", "name": "localtime", "readOnly": true }, { "mountPath": "/etc/sysconfig/docker", "name": "dockercfg", "readOnly": true }, { "mountPath": "/etc/docker", "name": "dockerdaemoncfg", "readOnly": true } ] } ], "dnsPolicy": "ClusterFirst", "nodeSelector": { "logging-infra-fluentd": "true" }, "restartPolicy": "Always", "schedulerName": "default-scheduler", "securityContext": {}, "serviceAccount": "aggregated-logging-fluentd", "serviceAccountName": "aggregated-logging-fluentd", "terminationGracePeriodSeconds": 30, "volumes": [ { "hostPath": { "path": "/run/log/journal" }, "name": "runlogjournal" }, { "hostPath": { "path": "/var/log" }, "name": "varlog" }, { "hostPath": { "path": "/var/lib/docker/containers" }, "name": "varlibdockercontainers" }, { "configMap": { "defaultMode": 420, "name": "logging-fluentd" }, "name": "config" }, { "name": "certs", "secret": { "defaultMode": 420, "secretName": "logging-fluentd" } }, { "hostPath": { "path": "/etc/hostname" }, "name": "dockerhostname" }, { "hostPath": { "path": "/etc/localtime" }, "name": "localtime" }, { "hostPath": { "path": "/etc/sysconfig/docker" }, "name": "dockercfg" }, { "hostPath": { "path": "/etc/docker" }, "name": "dockerdaemoncfg" } ] } }, "templateGeneration": 1, "updateStrategy": { "rollingUpdate": { "maxUnavailable": 1 }, "type": "RollingUpdate" } }, "status": { "currentNumberScheduled": 0, "desiredNumberScheduled": 0, "numberMisscheduled": 0, "numberReady": 0, "observedGeneration": 1 } } ], "returncode": 0 }, "state": "present" } TASK [openshift_logging_fluentd : Retrieve list of Fluentd hosts] ************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:183 ok: [openshift] => { "changed": false, "results": { "cmd": "/bin/oc get node -o json -n default", "results": [ { "apiVersion": "v1", "items": [ { "apiVersion": "v1", "kind": "Node", "metadata": { "annotations": { "volumes.kubernetes.io/controller-managed-attach-detach": "true" }, "creationTimestamp": "2017-06-09T16:22:11Z", "labels": { "beta.kubernetes.io/arch": "amd64", "beta.kubernetes.io/os": "linux", "kubernetes.io/hostname": "172.18.12.108" }, "name": "172.18.12.108", "namespace": "", "resourceVersion": "1593", "selfLink": "/api/v1/nodes/172.18.12.108", "uid": "ca4dd811-4d2f-11e7-8061-0ed679898f5a" }, "spec": { "externalID": "172.18.12.108", "providerID": "aws:////i-08ba2905c8dea9d7e" }, "status": { "addresses": [ { "address": "172.18.12.108", "type": "LegacyHostIP" }, { "address": "172.18.12.108", "type": "InternalIP" }, { "address": "172.18.12.108", "type": "Hostname" } ], "allocatable": { "cpu": "4", "memory": "7129288Ki", "pods": "40" }, "capacity": { "cpu": "4", "memory": "7231688Ki", "pods": "40" }, "conditions": [ { "lastHeartbeatTime": "2017-06-09T16:37:28Z", "lastTransitionTime": "2017-06-09T16:22:11Z", "message": "kubelet has sufficient disk space available", "reason": "KubeletHasSufficientDisk", "status": "False", "type": "OutOfDisk" }, { "lastHeartbeatTime": "2017-06-09T16:37:28Z", "lastTransitionTime": "2017-06-09T16:22:11Z", "message": "kubelet has sufficient memory available", "reason": "KubeletHasSufficientMemory", "status": "False", "type": "MemoryPressure" }, { "lastHeartbeatTime": "2017-06-09T16:37:28Z", "lastTransitionTime": "2017-06-09T16:22:11Z", "message": "kubelet has no disk pressure", "reason": "KubeletHasNoDiskPressure", "status": "False", "type": "DiskPressure" }, { "lastHeartbeatTime": "2017-06-09T16:37:28Z", "lastTransitionTime": "2017-06-09T16:22:11Z", "message": "kubelet is posting ready status", "reason": "KubeletReady", "status": "True", "type": "Ready" } ], "daemonEndpoints": { "kubeletEndpoint": { "Port": 10250 } }, "images": [ { "names": [ "openshift/origin-federation:6acabdc", "openshift/origin-federation:latest" ], "sizeBytes": 1205885664 }, { "names": [ "docker.io/openshift/origin-docker-registry@sha256:7934d28a00d728e3841980d772acd0f593426d3c4f731547d4d80917ef371b4a", "docker.io/openshift/origin-docker-registry:latest" ], "sizeBytes": 1100571644 }, { "names": [ "openshift/origin-docker-registry:6acabdc", "openshift/origin-docker-registry:latest" ], "sizeBytes": 1100164272 }, { "names": [ "openshift/openvswitch:6acabdc", "openshift/openvswitch:latest" ], "sizeBytes": 1053403667 }, { "names": [ "openshift/node:6acabdc", "openshift/node:latest" ], "sizeBytes": 1051721928 }, { "names": [ "openshift/origin-keepalived-ipfailover:6acabdc", "openshift/origin-keepalived-ipfailover:latest" ], "sizeBytes": 1028529711 }, { "names": [ "openshift/origin-haproxy-router:6acabdc", "openshift/origin-haproxy-router:latest" ], "sizeBytes": 1022758742 }, { "names": [ "openshift/origin-f5-router:6acabdc", "openshift/origin-f5-router:latest" ], "sizeBytes": 1001728427 }, { "names": [ "openshift/origin-recycler:6acabdc", "openshift/origin-recycler:latest" ], "sizeBytes": 1001728427 }, { "names": [ "openshift/origin:6acabdc", "openshift/origin:latest" ], "sizeBytes": 1001728427 }, { "names": [ "openshift/origin-sti-builder:6acabdc", "openshift/origin-sti-builder:latest" ], "sizeBytes": 1001728427 }, { "names": [ "openshift/origin-deployer:6acabdc", "openshift/origin-deployer:latest" ], "sizeBytes": 1001728427 }, { "names": [ "openshift/origin-docker-builder:6acabdc", "openshift/origin-docker-builder:latest" ], "sizeBytes": 1001728427 }, { "names": [ "rhel7.1:latest" ], "sizeBytes": 765301508 }, { "names": [ "openshift/dind-master:latest" ], "sizeBytes": 731456758 }, { "names": [ "openshift/dind-node:latest" ], "sizeBytes": 731453034 }, { "names": [ "172.30.180.71:5000/logging/logging-auth-proxy@sha256:533806737412976cc767f4f6f8289ab5e1a5cbfcace714cde3ea385caa39bea8", "172.30.180.71:5000/logging/logging-auth-proxy:latest" ], "sizeBytes": 715536101 }, { "names": [ "<none>@<none>", "<none>:<none>" ], "sizeBytes": 709532011 }, { "names": [ "docker.io/node@sha256:46db0dd19955beb87b841c30a6b9812ba626473283e84117d1c016deee5949a9", "docker.io/node:0.10.36" ], "sizeBytes": 697128386 }, { "names": [ "172.30.180.71:5000/logging/logging-kibana@sha256:6d17dee0e59993e46c9432d758d36a5fba15e43632cc972228b5862791e3a785", "172.30.180.71:5000/logging/logging-kibana:latest" ], "sizeBytes": 682851525 }, { "names": [ "docker.io/openshift/origin-logging-kibana@sha256:950568237cc7d0ff14ea9fe22c3967d888996db70c66181421ad68caeb5ba75f", "docker.io/openshift/origin-logging-kibana:latest" ], "sizeBytes": 682851513 }, { "names": [ "openshift/dind:latest" ], "sizeBytes": 640650210 }, { "names": [ "172.30.180.71:5000/logging/logging-elasticsearch@sha256:19fd59f29904d98103635555cb58b58f19b312326c93cb2be9aaa949f276a14d", "172.30.180.71:5000/logging/logging-elasticsearch:latest" ], "sizeBytes": 623513082 }, { "names": [ "172.30.180.71:5000/logging/logging-fluentd@sha256:ec7b38ca9da5869ecb694f92b64a39061218c3f917d724ffecb89e6c36aca3b7", "172.30.180.71:5000/logging/logging-fluentd:latest" ], "sizeBytes": 472184954 }, { "names": [ "docker.io/openshift/origin-logging-elasticsearch@sha256:6296f1719676e970438cac4d912542b35ac786c14a15df892507007c4ecbe490", "docker.io/openshift/origin-logging-elasticsearch:latest" ], "sizeBytes": 425567196 }, { "names": [ "172.30.180.71:5000/logging/logging-curator@sha256:b7664d5f3da1bee95880c962e59e1a9b9c8deccba233c7e44f6235ca5fa460d0", "172.30.180.71:5000/logging/logging-curator:latest" ], "sizeBytes": 418288033 }, { "names": [ "docker.io/openshift/base-centos7@sha256:aea292a3bddba020cde0ee83e6a45807931eb607c164ec6a3674f67039d8cd7c", "docker.io/openshift/base-centos7:latest" ], "sizeBytes": 383049978 }, { "names": [ "rhel7.2:latest" ], "sizeBytes": 377493597 }, { "names": [ "openshift/origin-base:latest" ], "sizeBytes": 363070172 }, { "names": [ "docker.io/openshift/origin-logging-fluentd@sha256:cae7c21c9f111d4f5b481c14a65c597c67e715a8ffe3aee4c483100ee77296d7", "docker.io/openshift/origin-logging-fluentd:latest" ], "sizeBytes": 359223728 }, { "names": [ "docker.io/fedora@sha256:69281ddd7b2600e5f2b17f1e12d7fba25207f459204fb2d15884f8432c479136", "docker.io/fedora:25" ], "sizeBytes": 230864375 }, { "names": [ "rhel7.3:latest", "rhel7:latest" ], "sizeBytes": 219121266 }, { "names": [ "openshift/origin-pod:6acabdc", "openshift/origin-pod:latest" ], "sizeBytes": 213199843 }, { "names": [ "registry.access.redhat.com/rhel7.2@sha256:98e6ca5d226c26e31a95cd67716afe22833c943e1926a21daf1a030906a02249", "registry.access.redhat.com/rhel7.2:latest" ], "sizeBytes": 201376319 }, { "names": [ "registry.access.redhat.com/rhel7.3@sha256:1e232401d8e0ba53b36b757b4712fbcbd1dab9c21db039c45a84871a74e89e68", "registry.access.redhat.com/rhel7.3:latest" ], "sizeBytes": 192693772 }, { "names": [ "docker.io/centos@sha256:bba1de7c9d900a898e3cadbae040dfe8a633c06bc104a0df76ae24483e03c077" ], "sizeBytes": 192548999 }, { "names": [ "openshift/origin-source:latest" ], "sizeBytes": 192548894 }, { "names": [ "registry.access.redhat.com/rhel7.1@sha256:1bc5a4c43bbb29a5a96a61896ff696933be3502e2f5fdc4cde02d9e101731fdd", "registry.access.redhat.com/rhel7.1:latest" ], "sizeBytes": 158229901 } ], "nodeInfo": { "architecture": "amd64", "bootID": "64a1913d-5a6f-485c-8e07-024fe2b25e0b", "containerRuntimeVersion": "docker://1.12.6", "kernelVersion": "3.10.0-327.22.2.el7.x86_64", "kubeProxyVersion": "v1.6.1+5115d708d7", "kubeletVersion": "v1.6.1+5115d708d7", "machineID": "f9370ed252a14f73b014c1301a9b6d1b", "operatingSystem": "linux", "osImage": "Red Hat Enterprise Linux Server 7.3 (Maipo)", "systemUUID": "EC2E3595-6DF8-F87D-D6D8-1B4169ECFD13" } } } ], "kind": "List", "metadata": {}, "resourceVersion": "", "selfLink": "" } ], "returncode": 0 }, "state": "list" } TASK [openshift_logging_fluentd : Set openshift_logging_fluentd_hosts] ********* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:190 ok: [openshift] => { "ansible_facts": { "openshift_logging_fluentd_hosts": [ "172.18.12.108" ] }, "changed": false } TASK [openshift_logging_fluentd : include] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:195 included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/label_and_wait.yaml for openshift TASK [openshift_logging_fluentd : Label 172.18.12.108 for Fluentd deployment] *** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/label_and_wait.yaml:2 changed: [openshift] => { "changed": true, "results": { "cmd": "/bin/oc label node 172.18.12.108 logging-infra-fluentd=true --overwrite", "results": "", "returncode": 0 }, "state": "add" } TASK [openshift_logging_fluentd : command] ************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/label_and_wait.yaml:10 changed: [openshift -> 127.0.0.1] => { "changed": true, "cmd": [ "sleep", "0.5" ], "delta": "0:00:00.503549", "end": "2017-06-09 12:37:33.195407", "rc": 0, "start": "2017-06-09 12:37:32.691858" } TASK [openshift_logging_fluentd : Delete temp directory] *********************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging_fluentd/tasks/main.yaml:202 ok: [openshift] => { "changed": false, "path": "/tmp/openshift-logging-ansible-x88aJU", "state": "absent" } TASK [openshift_logging : include] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/install_logging.yaml:253 included: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/update_master_config.yaml for openshift TASK [openshift_logging : include] ********************************************* task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/main.yaml:36 skipping: [openshift] => { "changed": false, "skip_reason": "Conditional result was False", "skipped": true } TASK [openshift_logging : Cleaning up local temp dir] ************************** task path: /tmp/tmp.yPQutins2A/openhift-ansible/roles/openshift_logging/tasks/main.yaml:40 ok: [openshift -> 127.0.0.1] => { "changed": false, "path": "/tmp/openshift-logging-ansible-YJJ6SM", "state": "absent" } META: ran handlers META: ran handlers PLAY [Update Master configs] *************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=0 openshift : ok=213 changed=71 unreachable=0 failed=0 /data/src/github.com/openshift/origin-aggregated-logging Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:170: executing 'oc get pods -l component=es' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s... SUCCESS after 0.343s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:170: executing 'oc get pods -l component=es' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s Standard output from the command: NAME READY STATUS RESTARTS AGE logging-es-data-master-03l0u89m-1-p747h 1/1 Running 0 1m There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:171: executing 'oc get pods -l component=kibana' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s... SUCCESS after 0.226s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:171: executing 'oc get pods -l component=kibana' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s Standard output from the command: NAME READY STATUS RESTARTS AGE logging-kibana-1-20vz2 2/2 Running 0 36s There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:172: executing 'oc get pods -l component=curator' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s... SUCCESS after 6.459s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:172: executing 'oc get pods -l component=curator' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s Standard output from the command: NAME READY STATUS RESTARTS AGE logging-curator-1-1h9p5 0/1 ContainerCreating 0 11s NAME READY STATUS RESTARTS AGE logging-curator-1-1h9p5 0/1 ContainerCreating 0 12s ... repeated 2 times NAME READY STATUS RESTARTS AGE logging-curator-1-1h9p5 0/1 ContainerCreating 0 13s ... repeated 2 times NAME READY STATUS RESTARTS AGE logging-curator-1-1h9p5 0/1 ContainerCreating 0 14s ... repeated 3 times NAME READY STATUS RESTARTS AGE logging-curator-1-1h9p5 0/1 ContainerCreating 0 15s ... repeated 2 times NAME READY STATUS RESTARTS AGE logging-curator-1-1h9p5 0/1 ContainerCreating 0 16s ... repeated 2 times NAME READY STATUS RESTARTS AGE logging-curator-1-1h9p5 0/1 ContainerCreating 0 17s ... repeated 2 times NAME READY STATUS RESTARTS AGE logging-curator-1-1h9p5 1/1 Running 0 18s Standard error from the command: Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:175: executing 'oc get pods -l component=es-ops' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s... SUCCESS after 0.231s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:175: executing 'oc get pods -l component=es-ops' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s Standard output from the command: NAME READY STATUS RESTARTS AGE logging-es-ops-data-master-kq3j3au5-1-8bzmq 1/1 Running 0 58s There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:176: executing 'oc get pods -l component=kibana-ops' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s... SUCCESS after 0.291s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:176: executing 'oc get pods -l component=kibana-ops' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s Standard output from the command: NAME READY STATUS RESTARTS AGE logging-kibana-ops-1-91x0s 2/2 Running 0 30s There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:177: executing 'oc get pods -l component=curator-ops' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s... SUCCESS after 3.416s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:177: executing 'oc get pods -l component=curator-ops' expecting any result and text 'Running'; re-trying every 0.2s until completion or 180.000s Standard output from the command: NAME READY STATUS RESTARTS AGE logging-curator-ops-1-jbk9s 0/1 ContainerCreating 0 11s ... repeated 3 times NAME READY STATUS RESTARTS AGE logging-curator-ops-1-jbk9s 0/1 ContainerCreating 0 12s ... repeated 2 times NAME READY STATUS RESTARTS AGE logging-curator-ops-1-jbk9s 0/1 ContainerCreating 0 13s ... repeated 2 times NAME READY STATUS RESTARTS AGE logging-curator-ops-1-jbk9s 1/1 Running 0 14s Standard error from the command: Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:185: executing 'oc project logging > /dev/null' expecting success... SUCCESS after 0.887s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:185: executing 'oc project logging > /dev/null' expecting success There was no output from the command. There was no error output from the command. /data/src/github.com/openshift/origin-aggregated-logging/hack/testing /data/src/github.com/openshift/origin-aggregated-logging --> Deploying template "logging/logging-fluentd-template-maker" for "-" to project logging logging-fluentd-template-maker --------- Template to create template for fluentd * With parameters: * MASTER_URL=https://kubernetes.default.svc.cluster.local * ES_HOST=logging-es * ES_PORT=9200 * ES_CLIENT_CERT=/etc/fluent/keys/cert * ES_CLIENT_KEY=/etc/fluent/keys/key * ES_CA=/etc/fluent/keys/ca * OPS_HOST=logging-es-ops * OPS_PORT=9200 * OPS_CLIENT_CERT=/etc/fluent/keys/cert * OPS_CLIENT_KEY=/etc/fluent/keys/key * OPS_CA=/etc/fluent/keys/ca * ES_COPY=false * ES_COPY_HOST= * ES_COPY_PORT= * ES_COPY_SCHEME=https * ES_COPY_CLIENT_CERT= * ES_COPY_CLIENT_KEY= * ES_COPY_CA= * ES_COPY_USERNAME= * ES_COPY_PASSWORD= * OPS_COPY_HOST= * OPS_COPY_PORT= * OPS_COPY_SCHEME=https * OPS_COPY_CLIENT_CERT= * OPS_COPY_CLIENT_KEY= * OPS_COPY_CA= * OPS_COPY_USERNAME= * OPS_COPY_PASSWORD= * IMAGE_PREFIX_DEFAULT=172.30.180.71:5000/logging/ * IMAGE_VERSION_DEFAULT=latest * USE_JOURNAL= * JOURNAL_SOURCE= * JOURNAL_READ_FROM_HEAD=false * USE_MUX=false * USE_MUX_CLIENT=false * MUX_ALLOW_EXTERNAL=false * BUFFER_QUEUE_LIMIT=1024 * BUFFER_SIZE_LIMIT=16777216 --> Creating resources ... template "logging-fluentd-template" created --> Success Run 'oc status' to view your app. START wait_for_fluentd_to_catch_up at 2017-06-09 16:37:57.828796461+00:00 added es message f7835dcf-afda-488c-bf62-aa4e10e6b69b added es-ops message 37b327b1-bd4f-407b-a730-8f7fba0865eb good - wait_for_fluentd_to_catch_up: found 1 record project logging for f7835dcf-afda-488c-bf62-aa4e10e6b69b good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 37b327b1-bd4f-407b-a730-8f7fba0865eb END wait_for_fluentd_to_catch_up took 13 seconds at 2017-06-09 16:38:10.201939407+00:00 Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:223: executing 'oc login --username=admin --password=admin' expecting success... SUCCESS after 0.235s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:223: executing 'oc login --username=admin --password=admin' expecting success Standard output from the command: Login successful. You don't have any projects. You can try to create a new project, by running oc new-project <projectname> There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:224: executing 'oc login --username=system:admin' expecting success... SUCCESS after 0.264s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:224: executing 'oc login --username=system:admin' expecting success Standard output from the command: Logged into "https://172.18.12.108:8443" as "system:admin" using existing credentials. You have access to the following projects and can switch between them with 'oc project <projectname>': * default kube-public kube-system logging openshift openshift-infra Using project "default". There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:225: executing 'oadm policy add-cluster-role-to-user cluster-admin admin' expecting success... SUCCESS after 0.488s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:225: executing 'oadm policy add-cluster-role-to-user cluster-admin admin' expecting success Standard output from the command: cluster role "cluster-admin" added: "admin" There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:226: executing 'oc login --username=loguser --password=loguser' expecting success... SUCCESS after 0.396s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:226: executing 'oc login --username=loguser --password=loguser' expecting success Standard output from the command: Login successful. You don't have any projects. You can try to create a new project, by running oc new-project <projectname> There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:227: executing 'oc login --username=system:admin' expecting success... SUCCESS after 0.248s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:227: executing 'oc login --username=system:admin' expecting success Standard output from the command: Logged into "https://172.18.12.108:8443" as "system:admin" using existing credentials. You have access to the following projects and can switch between them with 'oc project <projectname>': * default kube-public kube-system logging openshift openshift-infra Using project "default". There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:228: executing 'oc project logging > /dev/null' expecting success... SUCCESS after 0.480s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:228: executing 'oc project logging > /dev/null' expecting success There was no output from the command. There was no error output from the command. Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:229: executing 'oadm policy add-role-to-user view loguser' expecting success... SUCCESS after 0.228s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:229: executing 'oadm policy add-role-to-user view loguser' expecting success Standard output from the command: role "view" added: "loguser" There was no error output from the command. Checking if Elasticsearch logging-es-data-master-03l0u89m-1-p747h is ready { "_id": "0", "_index": ".searchguard.logging-es-data-master-03l0u89m-1-p747h", "_shards": { "failed": 0, "successful": 1, "total": 1 }, "_type": "rolesmapping", "_version": 2, "created": false } Checking if Elasticsearch logging-es-ops-data-master-kq3j3au5-1-8bzmq is ready { "_id": "0", "_index": ".searchguard.logging-es-ops-data-master-kq3j3au5-1-8bzmq", "_shards": { "failed": 0, "successful": 1, "total": 1 }, "_type": "rolesmapping", "_version": 2, "created": false } ------------------------------------------ Test 'admin' user can access cluster stats ------------------------------------------ Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:265: executing 'test 200 = 200' expecting success... SUCCESS after 0.010s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:265: executing 'test 200 = 200' expecting success There was no output from the command. There was no error output from the command. ------------------------------------------ Test 'admin' user can access cluster stats for OPS cluster ------------------------------------------ Running /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:274: executing 'test 200 = 200' expecting success... SUCCESS after 0.016s: /data/src/github.com/openshift/origin-aggregated-logging/logging.sh:274: executing 'test 200 = 200' expecting success There was no output from the command. There was no error output from the command. Running e2e tests Checking installation of the EFK stack... Running test/cluster/rollout.sh:20: executing 'oc project logging' expecting success... SUCCESS after 0.239s: test/cluster/rollout.sh:20: executing 'oc project logging' expecting success Standard output from the command: Already on project "logging" on server "https://172.18.12.108:8443". There was no error output from the command. [INFO] Checking for DeploymentConfigurations... Running test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-kibana' expecting success... SUCCESS after 0.214s: test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-kibana' expecting success Standard output from the command: NAME REVISION DESIRED CURRENT TRIGGERED BY logging-kibana 1 1 1 config There was no error output from the command. Running test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-kibana' expecting success... SUCCESS after 0.209s: test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-kibana' expecting success Standard output from the command: replication controller "logging-kibana-1" successfully rolled out There was no error output from the command. Running test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-curator' expecting success... SUCCESS after 0.247s: test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-curator' expecting success Standard output from the command: NAME REVISION DESIRED CURRENT TRIGGERED BY logging-curator 1 1 1 config There was no error output from the command. Running test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-curator' expecting success... SUCCESS after 0.215s: test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-curator' expecting success Standard output from the command: replication controller "logging-curator-1" successfully rolled out There was no error output from the command. Running test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-kibana-ops' expecting success... SUCCESS after 0.211s: test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-kibana-ops' expecting success Standard output from the command: NAME REVISION DESIRED CURRENT TRIGGERED BY logging-kibana-ops 1 1 1 config There was no error output from the command. Running test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-kibana-ops' expecting success... SUCCESS after 0.212s: test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-kibana-ops' expecting success Standard output from the command: replication controller "logging-kibana-ops-1" successfully rolled out There was no error output from the command. Running test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-curator-ops' expecting success... SUCCESS after 0.300s: test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-curator-ops' expecting success Standard output from the command: NAME REVISION DESIRED CURRENT TRIGGERED BY logging-curator-ops 1 1 1 config There was no error output from the command. Running test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-curator-ops' expecting success... SUCCESS after 0.219s: test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-curator-ops' expecting success Standard output from the command: replication controller "logging-curator-ops-1" successfully rolled out There was no error output from the command. Running test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-es-data-master-03l0u89m' expecting success... SUCCESS after 0.214s: test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-es-data-master-03l0u89m' expecting success Standard output from the command: NAME REVISION DESIRED CURRENT TRIGGERED BY logging-es-data-master-03l0u89m 1 1 1 config There was no error output from the command. Running test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-es-data-master-03l0u89m' expecting success... SUCCESS after 0.209s: test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-es-data-master-03l0u89m' expecting success Standard output from the command: replication controller "logging-es-data-master-03l0u89m-1" successfully rolled out There was no error output from the command. Running test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-es-ops-data-master-kq3j3au5' expecting success... SUCCESS after 0.212s: test/cluster/rollout.sh:24: executing 'oc get deploymentconfig logging-es-ops-data-master-kq3j3au5' expecting success Standard output from the command: NAME REVISION DESIRED CURRENT TRIGGERED BY logging-es-ops-data-master-kq3j3au5 1 1 1 config There was no error output from the command. Running test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-es-ops-data-master-kq3j3au5' expecting success... SUCCESS after 0.211s: test/cluster/rollout.sh:25: executing 'oc rollout status deploymentconfig/logging-es-ops-data-master-kq3j3au5' expecting success Standard output from the command: replication controller "logging-es-ops-data-master-kq3j3au5-1" successfully rolled out There was no error output from the command. [INFO] Checking for Routes... Running test/cluster/rollout.sh:30: executing 'oc get route logging-kibana' expecting success... SUCCESS after 0.230s: test/cluster/rollout.sh:30: executing 'oc get route logging-kibana' expecting success Standard output from the command: NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD logging-kibana kibana.router.default.svc.cluster.local logging-kibana <all> reencrypt/Redirect None There was no error output from the command. Running test/cluster/rollout.sh:30: executing 'oc get route logging-kibana-ops' expecting success... SUCCESS after 0.211s: test/cluster/rollout.sh:30: executing 'oc get route logging-kibana-ops' expecting success Standard output from the command: NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD logging-kibana-ops kibana-ops.router.default.svc.cluster.local logging-kibana-ops <all> reencrypt/Redirect None There was no error output from the command. [INFO] Checking for Services... Running test/cluster/rollout.sh:35: executing 'oc get service logging-es' expecting success... SUCCESS after 0.231s: test/cluster/rollout.sh:35: executing 'oc get service logging-es' expecting success Standard output from the command: NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE logging-es 172.30.128.77 <none> 9200/TCP 2m There was no error output from the command. Running test/cluster/rollout.sh:35: executing 'oc get service logging-es-cluster' expecting success... SUCCESS after 0.222s: test/cluster/rollout.sh:35: executing 'oc get service logging-es-cluster' expecting success Standard output from the command: NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE logging-es-cluster 172.30.206.91 <none> 9300/TCP 2m There was no error output from the command. Running test/cluster/rollout.sh:35: executing 'oc get service logging-kibana' expecting success... SUCCESS after 0.231s: test/cluster/rollout.sh:35: executing 'oc get service logging-kibana' expecting success Standard output from the command: NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE logging-kibana 172.30.59.132 <none> 443/TCP 1m There was no error output from the command. Running test/cluster/rollout.sh:35: executing 'oc get service logging-es-ops' expecting success... SUCCESS after 0.222s: test/cluster/rollout.sh:35: executing 'oc get service logging-es-ops' expecting success Standard output from the command: NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE logging-es-ops 172.30.189.53 <none> 9200/TCP 1m There was no error output from the command. Running test/cluster/rollout.sh:35: executing 'oc get service logging-es-ops-cluster' expecting success... SUCCESS after 0.266s: test/cluster/rollout.sh:35: executing 'oc get service logging-es-ops-cluster' expecting success Standard output from the command: NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE logging-es-ops-cluster 172.30.230.239 <none> 9300/TCP 1m There was no error output from the command. Running test/cluster/rollout.sh:35: executing 'oc get service logging-kibana-ops' expecting success... SUCCESS after 0.303s: test/cluster/rollout.sh:35: executing 'oc get service logging-kibana-ops' expecting success Standard output from the command: NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE logging-kibana-ops 172.30.46.235 <none> 443/TCP 1m There was no error output from the command. [INFO] Checking for OAuthClients... Running test/cluster/rollout.sh:40: executing 'oc get oauthclient kibana-proxy' expecting success... SUCCESS after 0.219s: test/cluster/rollout.sh:40: executing 'oc get oauthclient kibana-proxy' expecting success Standard output from the command: NAME SECRET WWW-CHALLENGE REDIRECT URIS kibana-proxy D0aMW0Q5hAppnpt7H2CSdXRsVPjzQI6tfV5R1Zf1frqHdyTYrAKgwcfOvkIQ6P6L FALSE https://kibana.router.default.svc.cluster.local,https://kibana-ops.router.default.svc.cluster.local There was no error output from the command. [INFO] Checking for DaemonSets... Running test/cluster/rollout.sh:45: executing 'oc get daemonset logging-fluentd' expecting success... SUCCESS after 0.217s: test/cluster/rollout.sh:45: executing 'oc get daemonset logging-fluentd' expecting success Standard output from the command: NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE logging-fluentd 1 1 1 1 1 logging-infra-fluentd=true 1m There was no error output from the command. Running test/cluster/rollout.sh:47: executing 'oc get daemonset logging-fluentd -o jsonpath='{ .status.numberReady }'' expecting any result and text '1'; re-trying every 0.2s until completion or 60.000s... SUCCESS after 0.217s: test/cluster/rollout.sh:47: executing 'oc get daemonset logging-fluentd -o jsonpath='{ .status.numberReady }'' expecting any result and text '1'; re-trying every 0.2s until completion or 60.000s Standard output from the command: 1 There was no error output from the command. Checking for log entry matches between ES and their sources... WARNING: bridge-nf-call-ip6tables is disabled Running test/cluster/functionality.sh:40: executing 'oc login --username=admin --password=admin' expecting success... SUCCESS after 0.264s: test/cluster/functionality.sh:40: executing 'oc login --username=admin --password=admin' expecting success Standard output from the command: Login successful. You have access to the following projects and can switch between them with 'oc project <projectname>': default kube-public kube-system * logging openshift openshift-infra Using project "logging". There was no error output from the command. Running test/cluster/functionality.sh:44: executing 'oc login --username=system:admin' expecting success... SUCCESS after 0.231s: test/cluster/functionality.sh:44: executing 'oc login --username=system:admin' expecting success Standard output from the command: Logged into "https://172.18.12.108:8443" as "system:admin" using existing credentials. You have access to the following projects and can switch between them with 'oc project <projectname>': default kube-public kube-system * logging openshift openshift-infra Using project "logging". There was no error output from the command. Running test/cluster/functionality.sh:45: executing 'oc project logging' expecting success... SUCCESS after 0.234s: test/cluster/functionality.sh:45: executing 'oc project logging' expecting success Standard output from the command: Already on project "logging" on server "https://172.18.12.108:8443". There was no error output from the command. [INFO] Testing Kibana pod logging-kibana-1-20vz2 for a successful start... Running test/cluster/functionality.sh:52: executing 'oc exec logging-kibana-1-20vz2 -c kibana -- curl -s --request HEAD --write-out '%{response_code}' http://localhost:5601/' expecting any result and text '200'; re-trying every 0.2s until completion or 600.000s... SUCCESS after 120.294s: test/cluster/functionality.sh:52: executing 'oc exec logging-kibana-1-20vz2 -c kibana -- curl -s --request HEAD --write-out '%{response_code}' http://localhost:5601/' expecting any result and text '200'; re-trying every 0.2s until completion or 600.000s Standard output from the command: 200 There was no error output from the command. Running test/cluster/functionality.sh:53: executing 'oc get pod logging-kibana-1-20vz2 -o jsonpath='{ .status.containerStatuses[?(@.name=="kibana")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s... SUCCESS after 0.218s: test/cluster/functionality.sh:53: executing 'oc get pod logging-kibana-1-20vz2 -o jsonpath='{ .status.containerStatuses[?(@.name=="kibana")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s Standard output from the command: true There was no error output from the command. Running test/cluster/functionality.sh:54: executing 'oc get pod logging-kibana-1-20vz2 -o jsonpath='{ .status.containerStatuses[?(@.name=="kibana-proxy")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s... SUCCESS after 0.228s: test/cluster/functionality.sh:54: executing 'oc get pod logging-kibana-1-20vz2 -o jsonpath='{ .status.containerStatuses[?(@.name=="kibana-proxy")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s Standard output from the command: true There was no error output from the command. [INFO] Testing Elasticsearch pod logging-es-data-master-03l0u89m-1-p747h for a successful start... Running test/cluster/functionality.sh:59: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/' -X HEAD -w '%{response_code}'' expecting any result and text '200'; re-trying every 0.2s until completion or 600.000s... SUCCESS after 0.380s: test/cluster/functionality.sh:59: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/' -X HEAD -w '%{response_code}'' expecting any result and text '200'; re-trying every 0.2s until completion or 600.000s Standard output from the command: 200 There was no error output from the command. Running test/cluster/functionality.sh:60: executing 'oc get pod logging-es-data-master-03l0u89m-1-p747h -o jsonpath='{ .status.containerStatuses[?(@.name=="elasticsearch")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s... SUCCESS after 0.229s: test/cluster/functionality.sh:60: executing 'oc get pod logging-es-data-master-03l0u89m-1-p747h -o jsonpath='{ .status.containerStatuses[?(@.name=="elasticsearch")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s Standard output from the command: true There was no error output from the command. [INFO] Checking that Elasticsearch pod logging-es-data-master-03l0u89m-1-p747h recovered its indices after starting... Running test/cluster/functionality.sh:63: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_cluster/state/master_node' -w '%{response_code}'' expecting any result and text '}200$'; re-trying every 0.2s until completion or 600.000s... SUCCESS after 0.460s: test/cluster/functionality.sh:63: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_cluster/state/master_node' -w '%{response_code}'' expecting any result and text '}200$'; re-trying every 0.2s until completion or 600.000s Standard output from the command: {"cluster_name":"logging-es","master_node":"4Gx1PicFTjujr3aeunSDPA"}200 There was no error output from the command. [INFO] Elasticsearch pod logging-es-data-master-03l0u89m-1-p747h is the master [INFO] Checking that Elasticsearch pod logging-es-data-master-03l0u89m-1-p747h has persisted indices created by Fluentd... Running test/cluster/functionality.sh:76: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_cat/indices?h=index'' expecting any result and text '^(project|\.operations)\.'; re-trying every 0.2s until completion or 600.000s... SUCCESS after 0.503s: test/cluster/functionality.sh:76: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_cat/indices?h=index'' expecting any result and text '^(project|\.operations)\.'; re-trying every 0.2s until completion or 600.000s Standard output from the command: .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a.2017.06.09 .searchguard.logging-es-data-master-03l0u89m-1-p747h .kibana project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a.2017.06.09 There was no error output from the command. [INFO] Cheking for index project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a with Kibana pod logging-kibana-1-20vz2... Running test/cluster/functionality.sh:100: executing 'sudo -E VERBOSE=true go run '/data/src/github.com/openshift/origin-aggregated-logging/hack/testing/check-logs.go' 'logging-kibana-1-20vz2' 'logging-es:9200' 'project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a' '/var/log/containers/*_c76629dc-4d2f-11e7-8061-0ed679898f5a_*.log' '500' 'admin' 'FSMfUJaX1ld2PsWR-jW1831YjqzmlwjbOwxgP2Mq4AY' '127.0.0.1'' expecting success... SUCCESS after 8.726s: test/cluster/functionality.sh:100: executing 'sudo -E VERBOSE=true go run '/data/src/github.com/openshift/origin-aggregated-logging/hack/testing/check-logs.go' 'logging-kibana-1-20vz2' 'logging-es:9200' 'project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a' '/var/log/containers/*_c76629dc-4d2f-11e7-8061-0ed679898f5a_*.log' '500' 'admin' 'FSMfUJaX1ld2PsWR-jW1831YjqzmlwjbOwxgP2Mq4AY' '127.0.0.1'' expecting success Standard output from the command: Executing command [oc exec logging-kibana-1-20vz2 -- curl -s --key /etc/kibana/keys/key --cert /etc/kibana/keys/cert --cacert /etc/kibana/keys/ca -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer FSMfUJaX1ld2PsWR-jW1831YjqzmlwjbOwxgP2Mq4AY' -H 'X-Forwarded-For: 127.0.0.1' -XGET "https://logging-es:9200/project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a.*/_search?q=hostname:ip-172-18-12-108&fields=message&size=500"] Failure - no log entries found in Elasticsearch logging-es:9200 for index project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a There was no error output from the command. [INFO] Cheking for index project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a with Kibana pod logging-kibana-1-20vz2... Running test/cluster/functionality.sh:100: executing 'sudo -E VERBOSE=true go run '/data/src/github.com/openshift/origin-aggregated-logging/hack/testing/check-logs.go' 'logging-kibana-1-20vz2' 'logging-es:9200' 'project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a' '/var/log/containers/*_cb9bf79f-4d2f-11e7-8061-0ed679898f5a_*.log' '500' 'admin' 'FSMfUJaX1ld2PsWR-jW1831YjqzmlwjbOwxgP2Mq4AY' '127.0.0.1'' expecting success... SUCCESS after 0.555s: test/cluster/functionality.sh:100: executing 'sudo -E VERBOSE=true go run '/data/src/github.com/openshift/origin-aggregated-logging/hack/testing/check-logs.go' 'logging-kibana-1-20vz2' 'logging-es:9200' 'project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a' '/var/log/containers/*_cb9bf79f-4d2f-11e7-8061-0ed679898f5a_*.log' '500' 'admin' 'FSMfUJaX1ld2PsWR-jW1831YjqzmlwjbOwxgP2Mq4AY' '127.0.0.1'' expecting success Standard output from the command: Executing command [oc exec logging-kibana-1-20vz2 -- curl -s --key /etc/kibana/keys/key --cert /etc/kibana/keys/cert --cacert /etc/kibana/keys/ca -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer FSMfUJaX1ld2PsWR-jW1831YjqzmlwjbOwxgP2Mq4AY' -H 'X-Forwarded-For: 127.0.0.1' -XGET "https://logging-es:9200/project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a.*/_search?q=hostname:ip-172-18-12-108&fields=message&size=500"] Failure - no log entries found in Elasticsearch logging-es:9200 for index project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a There was no error output from the command. [INFO] Checking that Elasticsearch pod logging-es-data-master-03l0u89m-1-p747h contains common data model index templates... Running test/cluster/functionality.sh:105: executing 'oc exec logging-es-data-master-03l0u89m-1-p747h -- ls -1 /usr/share/elasticsearch/index_templates' expecting success... SUCCESS after 0.280s: test/cluster/functionality.sh:105: executing 'oc exec logging-es-data-master-03l0u89m-1-p747h -- ls -1 /usr/share/elasticsearch/index_templates' expecting success Standard output from the command: com.redhat.viaq-openshift-operations.template.json com.redhat.viaq-openshift-project.template.json org.ovirt.viaq-collectd.template.json There was no error output from the command. Running test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_template/com.redhat.viaq-openshift-operations.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200'... SUCCESS after 0.383s: test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_template/com.redhat.viaq-openshift-operations.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200' Standard output from the command: 200 There was no error output from the command. Running test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_template/com.redhat.viaq-openshift-project.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200'... SUCCESS after 0.363s: test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_template/com.redhat.viaq-openshift-project.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200' Standard output from the command: 200 There was no error output from the command. Running test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_template/org.ovirt.viaq-collectd.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200'... SUCCESS after 0.369s: test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-data-master-03l0u89m-1-p747h' '/_template/org.ovirt.viaq-collectd.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200' Standard output from the command: 200 There was no error output from the command. Running test/cluster/functionality.sh:40: executing 'oc login --username=admin --password=admin' expecting success... SUCCESS after 0.264s: test/cluster/functionality.sh:40: executing 'oc login --username=admin --password=admin' expecting success Standard output from the command: Login successful. You have access to the following projects and can switch between them with 'oc project <projectname>': default kube-public kube-system * logging openshift openshift-infra Using project "logging". There was no error output from the command. Running test/cluster/functionality.sh:44: executing 'oc login --username=system:admin' expecting success... SUCCESS after 0.234s: test/cluster/functionality.sh:44: executing 'oc login --username=system:admin' expecting success Standard output from the command: Logged into "https://172.18.12.108:8443" as "system:admin" using existing credentials. You have access to the following projects and can switch between them with 'oc project <projectname>': default kube-public kube-system * logging openshift openshift-infra Using project "logging". There was no error output from the command. Running test/cluster/functionality.sh:45: executing 'oc project logging' expecting success... SUCCESS after 0.239s: test/cluster/functionality.sh:45: executing 'oc project logging' expecting success Standard output from the command: Already on project "logging" on server "https://172.18.12.108:8443". There was no error output from the command. [INFO] Testing Kibana pod logging-kibana-ops-1-91x0s for a successful start... Running test/cluster/functionality.sh:52: executing 'oc exec logging-kibana-ops-1-91x0s -c kibana -- curl -s --request HEAD --write-out '%{response_code}' http://localhost:5601/' expecting any result and text '200'; re-trying every 0.2s until completion or 600.000s... SUCCESS after 120.339s: test/cluster/functionality.sh:52: executing 'oc exec logging-kibana-ops-1-91x0s -c kibana -- curl -s --request HEAD --write-out '%{response_code}' http://localhost:5601/' expecting any result and text '200'; re-trying every 0.2s until completion or 600.000s Standard output from the command: 200 There was no error output from the command. Running test/cluster/functionality.sh:53: executing 'oc get pod logging-kibana-ops-1-91x0s -o jsonpath='{ .status.containerStatuses[?(@.name=="kibana")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s... SUCCESS after 0.212s: test/cluster/functionality.sh:53: executing 'oc get pod logging-kibana-ops-1-91x0s -o jsonpath='{ .status.containerStatuses[?(@.name=="kibana")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s Standard output from the command: true There was no error output from the command. Running test/cluster/functionality.sh:54: executing 'oc get pod logging-kibana-ops-1-91x0s -o jsonpath='{ .status.containerStatuses[?(@.name=="kibana-proxy")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s... SUCCESS after 0.226s: test/cluster/functionality.sh:54: executing 'oc get pod logging-kibana-ops-1-91x0s -o jsonpath='{ .status.containerStatuses[?(@.name=="kibana-proxy")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s Standard output from the command: true There was no error output from the command. [INFO] Testing Elasticsearch pod logging-es-ops-data-master-kq3j3au5-1-8bzmq for a successful start... Running test/cluster/functionality.sh:59: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/' -X HEAD -w '%{response_code}'' expecting any result and text '200'; re-trying every 0.2s until completion or 600.000s... SUCCESS after 0.369s: test/cluster/functionality.sh:59: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/' -X HEAD -w '%{response_code}'' expecting any result and text '200'; re-trying every 0.2s until completion or 600.000s Standard output from the command: 200 There was no error output from the command. Running test/cluster/functionality.sh:60: executing 'oc get pod logging-es-ops-data-master-kq3j3au5-1-8bzmq -o jsonpath='{ .status.containerStatuses[?(@.name=="elasticsearch")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s... SUCCESS after 0.218s: test/cluster/functionality.sh:60: executing 'oc get pod logging-es-ops-data-master-kq3j3au5-1-8bzmq -o jsonpath='{ .status.containerStatuses[?(@.name=="elasticsearch")].ready }'' expecting any result and text 'true'; re-trying every 0.2s until completion or 60.000s Standard output from the command: true There was no error output from the command. [INFO] Checking that Elasticsearch pod logging-es-ops-data-master-kq3j3au5-1-8bzmq recovered its indices after starting... Running test/cluster/functionality.sh:63: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_cluster/state/master_node' -w '%{response_code}'' expecting any result and text '}200$'; re-trying every 0.2s until completion or 600.000s... SUCCESS after 0.380s: test/cluster/functionality.sh:63: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_cluster/state/master_node' -w '%{response_code}'' expecting any result and text '}200$'; re-trying every 0.2s until completion or 600.000s Standard output from the command: {"cluster_name":"logging-es-ops","master_node":"uNJVSWF2SzSKXMIPwh8uMQ"}200 There was no error output from the command. [INFO] Elasticsearch pod logging-es-ops-data-master-kq3j3au5-1-8bzmq is the master [INFO] Checking that Elasticsearch pod logging-es-ops-data-master-kq3j3au5-1-8bzmq has persisted indices created by Fluentd... Running test/cluster/functionality.sh:76: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_cat/indices?h=index'' expecting any result and text '^(project|\.operations)\.'; re-trying every 0.2s until completion or 600.000s... SUCCESS after 0.405s: test/cluster/functionality.sh:76: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_cat/indices?h=index'' expecting any result and text '^(project|\.operations)\.'; re-trying every 0.2s until completion or 600.000s Standard output from the command: .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 .operations.2017.06.09 .kibana .searchguard.logging-es-ops-data-master-kq3j3au5-1-8bzmq There was no error output from the command. [INFO] Cheking for index .operations with Kibana pod logging-kibana-ops-1-91x0s... Running test/cluster/functionality.sh:100: executing 'sudo -E VERBOSE=true go run '/data/src/github.com/openshift/origin-aggregated-logging/hack/testing/check-logs.go' 'logging-kibana-ops-1-91x0s' 'logging-es-ops:9200' '.operations' '/var/log/messages' '500' 'admin' 'yIHLA6MHKrv3nawCCqf1B6Kd_ylN-puFJd3QicknZyM' '127.0.0.1'' expecting success... SUCCESS after 0.892s: test/cluster/functionality.sh:100: executing 'sudo -E VERBOSE=true go run '/data/src/github.com/openshift/origin-aggregated-logging/hack/testing/check-logs.go' 'logging-kibana-ops-1-91x0s' 'logging-es-ops:9200' '.operations' '/var/log/messages' '500' 'admin' 'yIHLA6MHKrv3nawCCqf1B6Kd_ylN-puFJd3QicknZyM' '127.0.0.1'' expecting success Standard output from the command: Executing command [oc exec logging-kibana-ops-1-91x0s -- curl -s --key /etc/kibana/keys/key --cert /etc/kibana/keys/cert --cacert /etc/kibana/keys/ca -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer yIHLA6MHKrv3nawCCqf1B6Kd_ylN-puFJd3QicknZyM' -H 'X-Forwarded-For: 127.0.0.1' -XGET "https://logging-es-ops:9200/.operations.*/_search?q=hostname:ip-172-18-12-108&fields=message&size=500"] Failure - no log entries found in Elasticsearch logging-es-ops:9200 for index .operations There was no error output from the command. [INFO] Checking that Elasticsearch pod logging-es-ops-data-master-kq3j3au5-1-8bzmq contains common data model index templates... Running test/cluster/functionality.sh:105: executing 'oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- ls -1 /usr/share/elasticsearch/index_templates' expecting success... SUCCESS after 0.355s: test/cluster/functionality.sh:105: executing 'oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- ls -1 /usr/share/elasticsearch/index_templates' expecting success Standard output from the command: com.redhat.viaq-openshift-operations.template.json com.redhat.viaq-openshift-project.template.json org.ovirt.viaq-collectd.template.json There was no error output from the command. Running test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_template/com.redhat.viaq-openshift-operations.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200'... SUCCESS after 0.411s: test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_template/com.redhat.viaq-openshift-operations.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200' Standard output from the command: 200 There was no error output from the command. Running test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_template/com.redhat.viaq-openshift-project.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200'... SUCCESS after 0.380s: test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_template/com.redhat.viaq-openshift-project.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200' Standard output from the command: 200 There was no error output from the command. Running test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_template/org.ovirt.viaq-collectd.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200'... SUCCESS after 0.402s: test/cluster/functionality.sh:107: executing 'curl_es 'logging-es-ops-data-master-kq3j3au5-1-8bzmq' '/_template/org.ovirt.viaq-collectd.template.json' -X HEAD -w '%{response_code}'' expecting success and text '200' Standard output from the command: 200 There was no error output from the command. running test test-curator.sh configmap "logging-curator" deleted configmap "logging-curator" created deploymentconfig "logging-curator" scaled deploymentconfig "logging-curator" scaled configmap "logging-curator" deleted configmap "logging-curator" created deploymentconfig "logging-curator" scaled deploymentconfig "logging-curator" scaled configmap "logging-curator" deleted configmap "logging-curator" created deploymentconfig "logging-curator" scaled deploymentconfig "logging-curator" scaled current indices before 1st deletion are: .kibana .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 .operations.curatortest.2017.03.31 .operations.curatortest.2017.06.09 .searchguard.logging-es-data-master-03l0u89m-1-p747h default-index.curatortest.2017.05.09 default-index.curatortest.2017.06.09 project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a.2017.06.09 project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a.2017.06.09 project.project-dev.curatortest.2017.06.08 project.project-dev.curatortest.2017.06.09 project.project-prod.curatortest.2017.05.12 project.project-prod.curatortest.2017.06.09 project.project-qe.curatortest.2017.06.02 project.project-qe.curatortest.2017.06.09 project.project2-qe.curatortest.2017.06.02 project.project2-qe.curatortest.2017.06.09 project.project3-qe.curatortest.2017.06.02 project.project3-qe.curatortest.2017.06.09 Fri Jun 9 16:45:08 UTC 2017 configmap "logging-curator" deleted configmap "logging-curator" created deploymentconfig "logging-curator" scaled deploymentconfig "logging-curator" scaled current indices after 1st deletion are: .kibana .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 .operations.curatortest.2017.06.09 .searchguard.logging-es-data-master-03l0u89m-1-p747h default-index.curatortest.2017.06.09 project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a.2017.06.09 project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a.2017.06.09 project.project-dev.curatortest.2017.06.09 project.project-prod.curatortest.2017.06.09 project.project-qe.curatortest.2017.06.09 project.project2-qe.curatortest.2017.06.09 project.project3-qe.curatortest.2017.06.09 good - index project.project-dev.curatortest.2017.06.09 is present good - index project.project-dev.curatortest.2017.06.08 is missing good - index project.project-qe.curatortest.2017.06.09 is present good - index project.project-qe.curatortest.2017.06.02 is missing good - index project.project-prod.curatortest.2017.06.09 is present good - index project.project-prod.curatortest.2017.05.12 is missing good - index .operations.curatortest.2017.06.09 is present good - index .operations.curatortest.2017.03.31 is missing good - index default-index.curatortest.2017.06.09 is present good - index default-index.curatortest.2017.05.09 is missing good - index project.project2-qe.curatortest.2017.06.09 is present good - index project.project2-qe.curatortest.2017.06.02 is missing good - index project.project3-qe.curatortest.2017.06.09 is present good - index project.project3-qe.curatortest.2017.06.02 is missing current indices before 2nd deletion are: .kibana .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 .operations.curatortest.2017.03.31 .operations.curatortest.2017.06.09 .searchguard.logging-es-data-master-03l0u89m-1-p747h default-index.curatortest.2017.05.09 default-index.curatortest.2017.06.09 project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a.2017.06.09 project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a.2017.06.09 project.project-dev.curatortest.2017.06.08 project.project-dev.curatortest.2017.06.09 project.project-prod.curatortest.2017.05.12 project.project-prod.curatortest.2017.06.09 project.project-qe.curatortest.2017.06.02 project.project-qe.curatortest.2017.06.09 project.project2-qe.curatortest.2017.06.02 project.project2-qe.curatortest.2017.06.09 project.project3-qe.curatortest.2017.06.02 project.project3-qe.curatortest.2017.06.09 sleeping 221 seconds to see if runhour and runminute are working . . . verify indices deletion again current indices after 2nd deletion are: .kibana .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 .operations.curatortest.2017.06.09 .searchguard.logging-es-data-master-03l0u89m-1-p747h default-index.curatortest.2017.06.09 project.default.c76629dc-4d2f-11e7-8061-0ed679898f5a.2017.06.09 project.logging.cb9bf79f-4d2f-11e7-8061-0ed679898f5a.2017.06.09 project.project-dev.curatortest.2017.06.09 project.project-prod.curatortest.2017.06.09 project.project-qe.curatortest.2017.06.09 project.project2-qe.curatortest.2017.06.09 project.project3-qe.curatortest.2017.06.09 good - index project.project-dev.curatortest.2017.06.09 is present good - index project.project-dev.curatortest.2017.06.08 is missing good - index project.project-qe.curatortest.2017.06.09 is present good - index project.project-qe.curatortest.2017.06.02 is missing good - index project.project-prod.curatortest.2017.06.09 is present good - index project.project-prod.curatortest.2017.05.12 is missing good - index .operations.curatortest.2017.06.09 is present good - index .operations.curatortest.2017.03.31 is missing good - index default-index.curatortest.2017.06.09 is present good - index default-index.curatortest.2017.05.09 is missing good - index project.project2-qe.curatortest.2017.06.09 is present good - index project.project2-qe.curatortest.2017.06.02 is missing good - index project.project3-qe.curatortest.2017.06.09 is present good - index project.project3-qe.curatortest.2017.06.02 is missing current indices before 1st deletion are: .kibana .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 .operations.2017.06.09 .operations.curatortest.2017.03.31 .operations.curatortest.2017.06.09 .searchguard.logging-es-ops-data-master-kq3j3au5-1-8bzmq default-index.curatortest.2017.05.09 default-index.curatortest.2017.06.09 project.project-dev.curatortest.2017.06.08 project.project-dev.curatortest.2017.06.09 project.project-prod.curatortest.2017.05.12 project.project-prod.curatortest.2017.06.09 project.project-qe.curatortest.2017.06.02 project.project-qe.curatortest.2017.06.09 project.project2-qe.curatortest.2017.06.02 project.project2-qe.curatortest.2017.06.09 project.project3-qe.curatortest.2017.06.02 project.project3-qe.curatortest.2017.06.09 Fri Jun 9 16:50:33 UTC 2017 configmap "logging-curator" deleted configmap "logging-curator" created deploymentconfig "logging-curator-ops" scaled deploymentconfig "logging-curator-ops" scaled current indices after 1st deletion are: .kibana .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 .operations.2017.06.09 .operations.curatortest.2017.06.09 .searchguard.logging-es-ops-data-master-kq3j3au5-1-8bzmq default-index.curatortest.2017.06.09 project.project-dev.curatortest.2017.06.09 project.project-prod.curatortest.2017.06.09 project.project-qe.curatortest.2017.06.09 project.project2-qe.curatortest.2017.06.09 project.project3-qe.curatortest.2017.06.09 good - index project.project-dev.curatortest.2017.06.09 is present good - index project.project-dev.curatortest.2017.06.08 is missing good - index project.project-qe.curatortest.2017.06.09 is present good - index project.project-qe.curatortest.2017.06.02 is missing good - index project.project-prod.curatortest.2017.06.09 is present good - index project.project-prod.curatortest.2017.05.12 is missing good - index .operations.curatortest.2017.06.09 is present good - index .operations.curatortest.2017.03.31 is missing good - index default-index.curatortest.2017.06.09 is present good - index default-index.curatortest.2017.05.09 is missing good - index project.project2-qe.curatortest.2017.06.09 is present good - index project.project2-qe.curatortest.2017.06.02 is missing good - index project.project3-qe.curatortest.2017.06.09 is present good - index project.project3-qe.curatortest.2017.06.02 is missing current indices before 2nd deletion are: .kibana .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 .operations.2017.06.09 .operations.curatortest.2017.03.31 .operations.curatortest.2017.06.09 .searchguard.logging-es-ops-data-master-kq3j3au5-1-8bzmq default-index.curatortest.2017.05.09 default-index.curatortest.2017.06.09 project.project-dev.curatortest.2017.06.08 project.project-dev.curatortest.2017.06.09 project.project-prod.curatortest.2017.05.12 project.project-prod.curatortest.2017.06.09 project.project-qe.curatortest.2017.06.02 project.project-qe.curatortest.2017.06.09 project.project2-qe.curatortest.2017.06.02 project.project2-qe.curatortest.2017.06.09 project.project3-qe.curatortest.2017.06.02 project.project3-qe.curatortest.2017.06.09 sleeping 220 seconds to see if runhour and runminute are working . . . verify indices deletion again current indices after 2nd deletion are: .kibana .kibana.d033e22ae348aeb5660fc2140aec35850c4da997 .operations.2017.06.09 .operations.curatortest.2017.06.09 .searchguard.logging-es-ops-data-master-kq3j3au5-1-8bzmq default-index.curatortest.2017.06.09 project.project-dev.curatortest.2017.06.09 project.project-prod.curatortest.2017.06.09 project.project-qe.curatortest.2017.06.09 project.project2-qe.curatortest.2017.06.09 project.project3-qe.curatortest.2017.06.09 good - index project.project-dev.curatortest.2017.06.09 is present good - index project.project-dev.curatortest.2017.06.08 is missing good - index project.project-qe.curatortest.2017.06.09 is present good - index project.project-qe.curatortest.2017.06.02 is missing good - index project.project-prod.curatortest.2017.06.09 is present good - index project.project-prod.curatortest.2017.05.12 is missing good - index .operations.curatortest.2017.06.09 is present good - index .operations.curatortest.2017.03.31 is missing good - index default-index.curatortest.2017.06.09 is present good - index default-index.curatortest.2017.05.09 is missing good - index project.project2-qe.curatortest.2017.06.09 is present good - index project.project2-qe.curatortest.2017.06.02 is missing good - index project.project3-qe.curatortest.2017.06.09 is present good - index project.project3-qe.curatortest.2017.06.02 is missing curator running [5] jobs curator run finish curator running [5] jobs curator run finish configmap "logging-curator" deleted configmap "logging-curator" created deploymentconfig "logging-curator-ops" scaled deploymentconfig "logging-curator-ops" scaled running test test-datetime-future.sh ++ set -o nounset ++ set -o pipefail ++ type get_running_pod ++ [[ 1 -ne 1 ]] ++ [[ true = \f\a\l\s\e ]] ++ CLUSTER=true ++ ops=-ops ++ INDEX_PREFIX= ++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts ++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']' ++ get_test_user_token ++ local current_project +++ oc project -q ++ current_project=logging ++ oc login --username=admin --password=admin +++ oc whoami -t ++ test_token=YGfeK7SmT9wAKTLIlCtptB-i2Ed9hxwNlFRvxuCnis0 +++ oc whoami ++ test_name=admin ++ test_ip=127.0.0.1 ++ oc login --username=system:admin ++ oc project logging ++ TEST_DIVIDER=------------------------------------------ +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-x2vxq +++ date +%z ++ nodetz=-0400 +++ oc exec logging-fluentd-x2vxq -- date +%z ++ podtz=-0400 ++ '[' x-0400 = x-0400 ']' +++ date +%Z ++ echo Good - node timezone -0400 EDT is equal to the fluentd pod timezone Good - node timezone -0400 EDT is equal to the fluentd pod timezone ++ docker_uses_journal ++ type -p docker ++ sudo docker info ++ grep -q 'Logging Driver: journald' WARNING: bridge-nf-call-ip6tables is disabled ++ return 0 ++ echo The rest of the test is not applicable when using the journal - skipping The rest of the test is not applicable when using the journal - skipping ++ exit 0 running test test-es-copy.sh ++ set -o nounset ++ set -o pipefail ++ type get_running_pod ++ [[ 1 -ne 1 ]] ++ [[ true = \f\a\l\s\e ]] ++ CLUSTER=true ++ ops=-ops ++ INDEX_PREFIX= ++ PROJ_PREFIX=project. ++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts ++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']' ++ get_test_user_token ++ local current_project +++ oc project -q ++ current_project=logging ++ oc login --username=admin --password=admin +++ oc whoami -t ++ test_token=KmcdDEMRmEjcIHMhyRTDVjD5vY_1YhXukG-Jn1qndMc +++ oc whoami ++ test_name=admin ++ test_ip=127.0.0.1 ++ oc login --username=system:admin ++ oc project logging ++ TEST_DIVIDER=------------------------------------------ +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-x2vxq +++ mktemp ++ cfg=/tmp/tmp.ikDYVA1kf0 ++ oc replace -f - ++ sed '/- name: ES_COPY/,/value:/ s/value: .*$/value: "false"/' ++ oc get template logging-fluentd-template -o yaml template "logging-fluentd-template" replaced ++ restart_fluentd ++ oc delete daemonset logging-fluentd daemonset "logging-fluentd" deleted ++ wait_for_pod_ACTION stop logging-fluentd-x2vxq ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-x2vxq ++ '[' -z logging-fluentd-x2vxq -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-x2vxq ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc process logging-fluentd-template ++ oc create -f - daemonset "logging-fluentd" created ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' ++ echo pod for component=fluentd not running yet pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-8vxss ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-8vxss ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-8vxss +++ mktemp ++ origconfig=/tmp/tmp.NDUV7cqzkd ++ oc get template logging-fluentd-template -o yaml ++ write_and_verify_logs 1 ++ rc=0 ++ wait_for_fluentd_to_catch_up '' '' 1 +++ date +%s ++ local starttime=1497027669 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:01:09.329699509+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:01:09.329699509+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=e21bd202-9917-49e5-ad98-6a38b1093626 +++ uuidgen ++ local uuid_es_ops=d79aa0a0-e01a-4b0f-9375-ec79e8073336 ++ local expected=1 ++ local timeout=300 ++ add_test_message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/e21bd202-9917-49e5-ad98-6a38b1093626 ++ echo added es message e21bd202-9917-49e5-ad98-6a38b1093626 added es message e21bd202-9917-49e5-ad98-6a38b1093626 ++ logger -i -p local6.info -t d79aa0a0-e01a-4b0f-9375-ec79e8073336 d79aa0a0-e01a-4b0f-9375-ec79e8073336 added es-ops message d79aa0a0-e01a-4b0f-9375-ec79e8073336 ++ echo added es-ops message d79aa0a0-e01a-4b0f-9375-ec79e8073336 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=e21bd202-9917-49e5-ad98-6a38b1093626 ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 290 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 289 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 288 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message e21bd202-9917-49e5-ad98-6a38b1093626 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:e21bd202-9917-49e5-ad98-6a38b1093626' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 288 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project logging for e21bd202-9917-49e5-ad98-6a38b1093626 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for e21bd202-9917-49e5-ad98-6a38b1093626 ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=d79aa0a0-e01a-4b0f-9375-ec79e8073336 ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER d79aa0a0-e01a-4b0f-9375-ec79e8073336 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d79aa0a0-e01a-4b0f-9375-ec79e8073336' --connect-timeout 1 +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d79aa0a0-e01a-4b0f-9375-ec79e8073336' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:d79aa0a0-e01a-4b0f-9375-ec79e8073336' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 300 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for d79aa0a0-e01a-4b0f-9375-ec79e8073336 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for d79aa0a0-e01a-4b0f-9375-ec79e8073336 ++ '[' -n '' ']' ++ '[' -n '' ']' +++ date +%s ++ local endtime=1497027687 +++ expr 1497027687 - 1497027669 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 18 seconds at 2017-06-09 17:01:27.910863085+00:00 ++ echo END wait_for_fluentd_to_catch_up took 18 seconds at 2017-06-09 17:01:27.910863085+00:00 ++ return 0 ++ '[' 0 -ne 0 ']' ++ return 0 ++ trap cleanup INT TERM EXIT +++ mktemp ++ nocopy=/tmp/tmp.P8TIkB5ZcM ++ sed /_COPY/,/value/d /tmp/tmp.NDUV7cqzkd +++ mktemp ++ envpatch=/tmp/tmp.Si7AASb9LV ++ sed -n '/^ - env:/,/^ image:/ { /^ image:/d /^ - env:/d /name: K8S_HOST_URL/,/value/d s/ES_/ES_COPY_/ s/OPS_/OPS_COPY_/ p }' /tmp/tmp.P8TIkB5ZcM ++ cat ++ cat /tmp/tmp.P8TIkB5ZcM ++ sed '/^ - env:/r /tmp/tmp.Si7AASb9LV' ++ oc replace -f - template "logging-fluentd-template" replaced ++ rm -f /tmp/tmp.Si7AASb9LV /tmp/tmp.P8TIkB5ZcM ++ restart_fluentd ++ oc delete daemonset logging-fluentd daemonset "logging-fluentd" deleted ++ wait_for_pod_ACTION stop logging-fluentd-8vxss ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-8vxss ++ '[' -z logging-fluentd-8vxss -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-8vxss ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc process logging-fluentd-template ++ oc create -f - daemonset "logging-fluentd" created ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' pod for component=fluentd not running yet ++ echo pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-t3tgm ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-t3tgm ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-t3tgm ++ write_and_verify_logs 2 ++ rc=0 ++ wait_for_fluentd_to_catch_up '' '' 2 +++ date +%s ++ local starttime=1497027712 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:01:52.511852823+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:01:52.511852823+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ uuidgen ++ local uuid_es_ops=db9f0430-8686-4cb9-ab90-86c6b9ccaf75 ++ local expected=2 ++ local timeout=300 ++ add_test_message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/b3bd1bc6-761c-4cb0-9311-e7a37d32d35a ++ echo added es message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a added es message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a ++ logger -i -p local6.info -t db9f0430-8686-4cb9-ab90-86c6b9ccaf75 db9f0430-8686-4cb9-ab90-86c6b9ccaf75 added es-ops message db9f0430-8686-4cb9-ab90-86c6b9ccaf75 ++ echo added es-ops message db9f0430-8686-4cb9-ab90-86c6b9ccaf75 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=b3bd1bc6-761c-4cb0-9311-e7a37d32d35a ++ expected=2 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 290 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 289 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 288 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=0 ++ test 0 = 2 ++ sleep 1 ++ let ii=ii-1 ++ '[' 287 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b3bd1bc6-761c-4cb0-9311-e7a37d32d35a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b3bd1bc6-761c-4cb0-9311-e7a37d32d35a' ++ local nrecs=2 ++ test 2 = 2 ++ break ++ '[' 287 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 2 record project logging for b3bd1bc6-761c-4cb0-9311-e7a37d32d35a ++ echo good - wait_for_fluentd_to_catch_up: found 2 record project logging for b3bd1bc6-761c-4cb0-9311-e7a37d32d35a ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=db9f0430-8686-4cb9-ab90-86c6b9ccaf75 ++ expected=2 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER db9f0430-8686-4cb9-ab90-86c6b9ccaf75 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:db9f0430-8686-4cb9-ab90-86c6b9ccaf75' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:db9f0430-8686-4cb9-ab90-86c6b9ccaf75' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:db9f0430-8686-4cb9-ab90-86c6b9ccaf75' ++ local nrecs=2 ++ test 2 = 2 ++ break ++ '[' 300 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 2 record project .operations for db9f0430-8686-4cb9-ab90-86c6b9ccaf75 ++ echo good - wait_for_fluentd_to_catch_up: found 2 record project .operations for db9f0430-8686-4cb9-ab90-86c6b9ccaf75 ++ '[' -n '' ']' ++ '[' -n '' ']' +++ date +%s ++ local endtime=1497027732 +++ expr 1497027732 - 1497027712 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-09 17:02:12.357958070+00:00 ++ echo END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-09 17:02:12.357958070+00:00 ++ return 0 ++ '[' 0 -ne 0 ']' ++ return 0 ++ oc replace --force -f /tmp/tmp.NDUV7cqzkd template "logging-fluentd-template" deleted template "logging-fluentd-template" replaced ++ rm -f /tmp/tmp.NDUV7cqzkd ++ restart_fluentd ++ oc delete daemonset logging-fluentd daemonset "logging-fluentd" deleted ++ wait_for_pod_ACTION stop logging-fluentd-t3tgm ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-t3tgm ++ '[' -z logging-fluentd-t3tgm -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-t3tgm ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc process logging-fluentd-template ++ oc create -f - daemonset "logging-fluentd" created ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' ++ echo pod for component=fluentd not running yet pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-26stf ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-26stf ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-26stf ++ write_and_verify_logs 1 ++ rc=0 ++ wait_for_fluentd_to_catch_up '' '' 1 +++ date +%s ++ local starttime=1497027753 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:02:33.027983666+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:02:33.027983666+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=6a55892b-1814-487e-8066-cc884362511a +++ uuidgen ++ local uuid_es_ops=9875c71c-da49-459f-bb66-5f2b52112a54 ++ local expected=1 ++ local timeout=300 ++ add_test_message 6a55892b-1814-487e-8066-cc884362511a +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/6a55892b-1814-487e-8066-cc884362511a added es message 6a55892b-1814-487e-8066-cc884362511a ++ echo added es message 6a55892b-1814-487e-8066-cc884362511a ++ logger -i -p local6.info -t 9875c71c-da49-459f-bb66-5f2b52112a54 9875c71c-da49-459f-bb66-5f2b52112a54 added es-ops message 9875c71c-da49-459f-bb66-5f2b52112a54 ++ echo added es-ops message 9875c71c-da49-459f-bb66-5f2b52112a54 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=6a55892b-1814-487e-8066-cc884362511a ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ get_count_from_json +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 290 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 289 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 288 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 287 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 6a55892b-1814-487e-8066-cc884362511a +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:6a55892b-1814-487e-8066-cc884362511a' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 287 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project logging for 6a55892b-1814-487e-8066-cc884362511a ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 6a55892b-1814-487e-8066-cc884362511a ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=9875c71c-da49-459f-bb66-5f2b52112a54 ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER 9875c71c-da49-459f-bb66-5f2b52112a54 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:9875c71c-da49-459f-bb66-5f2b52112a54' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:9875c71c-da49-459f-bb66-5f2b52112a54' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:9875c71c-da49-459f-bb66-5f2b52112a54' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 300 -le 0 ']' ++ return 0 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 9875c71c-da49-459f-bb66-5f2b52112a54 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 9875c71c-da49-459f-bb66-5f2b52112a54 ++ '[' -n '' ']' ++ '[' -n '' ']' +++ date +%s ++ local endtime=1497027773 +++ expr 1497027773 - 1497027753 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-09 17:02:53.059856372+00:00 ++ echo END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-09 17:02:53.059856372+00:00 ++ return 0 ++ '[' 0 -ne 0 ']' ++ return 0 ++ cleanup ++ '[' '!' -f /tmp/tmp.NDUV7cqzkd ']' ++ return 0 running test test-fluentd-forward.sh ++ set -o nounset ++ set -o pipefail ++ type get_running_pod ++ [[ 1 -ne 1 ]] ++ [[ true = \f\a\l\s\e ]] ++ CLUSTER=true ++ ops=-ops ++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts ++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']' ++ PROJ_PREFIX=project. ++ get_test_user_token ++ local current_project +++ oc project -q ++ current_project=logging ++ oc login --username=admin --password=admin +++ oc whoami -t ++ test_token=MK-OCOlFXIEzBjWM6blYSCJo-_YiwtrglDxkbKDH-sg +++ oc whoami ++ test_name=admin ++ test_ip=127.0.0.1 ++ oc login --username=system:admin ++ oc project logging ++ TEST_DIVIDER=------------------------------------------ +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-26stf ++ write_and_verify_logs 1 ++ expected=1 ++ rc=0 ++ wait_for_fluentd_to_catch_up '' '' +++ date +%s ++ local starttime=1497027774 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:02:54.467739884+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:02:54.467739884+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=37cff620-b594-439a-bb47-39396f0a5bda +++ uuidgen ++ local uuid_es_ops=df5d2238-b8b7-4ae3-8ea3-6065d40b716d ++ local expected=1 ++ local timeout=300 ++ add_test_message 37cff620-b594-439a-bb47-39396f0a5bda +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/37cff620-b594-439a-bb47-39396f0a5bda ++ echo added es message 37cff620-b594-439a-bb47-39396f0a5bda added es message 37cff620-b594-439a-bb47-39396f0a5bda ++ logger -i -p local6.info -t df5d2238-b8b7-4ae3-8ea3-6065d40b716d df5d2238-b8b7-4ae3-8ea3-6065d40b716d added es-ops message df5d2238-b8b7-4ae3-8ea3-6065d40b716d ++ echo added es-ops message df5d2238-b8b7-4ae3-8ea3-6065d40b716d ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=37cff620-b594-439a-bb47-39396f0a5bda ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 37cff620-b594-439a-bb47-39396f0a5bda +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ get_count_from_json +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 37cff620-b594-439a-bb47-39396f0a5bda +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 37cff620-b594-439a-bb47-39396f0a5bda +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 37cff620-b594-439a-bb47-39396f0a5bda +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' --connect-timeout 1 +++ get_count_from_json +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' +++ shift +++ shift +++ args=("${@:-}") +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 37cff620-b594-439a-bb47-39396f0a5bda +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' +++ shift +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 37cff620-b594-439a-bb47-39396f0a5bda +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:37cff620-b594-439a-bb47-39396f0a5bda' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 295 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project logging for 37cff620-b594-439a-bb47-39396f0a5bda ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 37cff620-b594-439a-bb47-39396f0a5bda ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=df5d2238-b8b7-4ae3-8ea3-6065d40b716d ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER df5d2238-b8b7-4ae3-8ea3-6065d40b716d +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:df5d2238-b8b7-4ae3-8ea3-6065d40b716d' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:df5d2238-b8b7-4ae3-8ea3-6065d40b716d' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:df5d2238-b8b7-4ae3-8ea3-6065d40b716d' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 300 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for df5d2238-b8b7-4ae3-8ea3-6065d40b716d ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for df5d2238-b8b7-4ae3-8ea3-6065d40b716d ++ '[' -n '' ']' ++ '[' -n '' ']' +++ date +%s ++ local endtime=1497027783 +++ expr 1497027783 - 1497027774 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 9 seconds at 2017-06-09 17:03:03.077423281+00:00 ++ echo END wait_for_fluentd_to_catch_up took 9 seconds at 2017-06-09 17:03:03.077423281+00:00 ++ return 0 ++ return 0 ++ trap cleanup INT TERM EXIT ++ create_forwarding_fluentd ++ oc create configmap logging-forward-fluentd --from-file=fluent.conf=../templates/forward-fluent.conf configmap "logging-forward-fluentd" created ++ oc get template/logging-fluentd-template -o yaml ++ sed -e 's/logging-infra-fluentd: "true"/logging-infra-forward-fluentd: "true"/' -e 's/name: logging-fluentd/name: logging-forward-fluentd/' -e 's/ fluentd/ forward-fluentd/' -e '/image:/ a \ ports: \ - containerPort: 24284' ++ oc new-app -f - --> Deploying template "logging/logging-forward-fluentd-template" for "-" to project logging logging-forward-fluentd-template --------- Template for logging forward-fluentd deployment. * With parameters: * IMAGE_PREFIX=172.30.180.71:5000/logging/ * IMAGE_VERSION=latest --> Creating resources ... daemonset "logging-forward-fluentd" created --> Success Run 'oc status' to view your app. ++ oc label node --all logging-infra-forward-fluentd=true node "172.18.12.108" labeled ++ wait_for_pod_ACTION start forward-fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod forward-fluentd +++ oc get pods -l component=forward-fluentd +++ awk -v sel=forward-fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' pod for component=forward-fluentd not running yet ++ echo pod for component=forward-fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod forward-fluentd +++ oc get pods -l component=forward-fluentd +++ awk -v sel=forward-fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-forward-fluentd-8zx87 ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-forward-fluentd-8zx87 ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 ++ update_current_fluentd ++ oc label node --all logging-infra-fluentd- node "172.18.12.108" labeled ++ wait_for_pod_ACTION stop logging-fluentd-26stf ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-26stf ++ '[' -z logging-fluentd-26stf -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-26stf pod logging-fluentd-26stf still running ++ '[' -n 1 ']' ++ echo pod logging-fluentd-26stf still running ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' stop = start ']' ++ '[' 110 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-26stf ++ '[' stop = start ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 ++ oc get configmap/logging-fluentd -o yaml ++ sed '/## matches/ a\ <match **>\ @include configs.d/user/secure-forward.conf\ </match>' ++ oc replace -f - configmap "logging-fluentd" replaced +++ oc get pods -l component=forward-fluentd -o name ++ POD=pods/logging-forward-fluentd-8zx87 +++ oc get pods/logging-forward-fluentd-8zx87 '--template={{.status.podIP}}' ++ FLUENTD_FORWARD=172.17.0.10 ++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\ @type secure_forward\n\ self_hostname forwarding-${HOSTNAME}\n\ shared_key aggregated_logging_ci_testing\n\ secure no\n\ buffer_queue_limit \"#{ENV['\''BUFFER_QUEUE_LIMIT'\'']}\"\n\ buffer_chunk_limit \"#{ENV['\''BUFFER_SIZE_LIMIT'\'']}\"\n\ <server>\n\ host 172.17.0.10\n\ port 24284\n\ </server>"}]' configmap "logging-fluentd" patched ++ oc label node --all logging-infra-fluentd=true node "172.18.12.108" labeled ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' pod for component=fluentd not running yet ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' ++ echo pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-vtgg1 ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-vtgg1 ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 ++ write_and_verify_logs 1 ++ expected=1 ++ rc=0 ++ wait_for_fluentd_to_catch_up '' '' +++ date +%s ++ local starttime=1497027816 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:03:36.592040716+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:03:36.592040716+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ uuidgen ++ local uuid_es_ops=84d2fe66-bcd9-4562-b0bc-58b37e179a59 ++ local expected=1 ++ local timeout=300 ++ add_test_message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd added es message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd ++ echo added es message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd ++ logger -i -p local6.info -t 84d2fe66-bcd9-4562-b0bc-58b37e179a59 84d2fe66-bcd9-4562-b0bc-58b37e179a59 added es-ops message 84d2fe66-bcd9-4562-b0bc-58b37e179a59 ++ echo added es-ops message 84d2fe66-bcd9-4562-b0bc-58b37e179a59 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ get_count_from_json +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 290 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 289 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 288 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 287 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 286 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 285 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 284 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 283 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 282 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 281 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 280 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 279 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 278 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 277 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 276 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 275 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ get_count_from_json +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 274 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 273 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 272 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 271 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 270 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 269 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 268 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 267 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 266 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 265 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 264 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 263 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 262 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 261 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 260 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 259 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 258 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 257 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 256 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 255 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 254 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 253 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 252 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 251 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 250 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 249 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 248 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 247 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 246 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 246 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project logging for 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 4ef1dfb3-fc3a-4e4e-91fb-7f73974f83cd ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=84d2fe66-bcd9-4562-b0bc-58b37e179a59 ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER 84d2fe66-bcd9-4562-b0bc-58b37e179a59 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:84d2fe66-bcd9-4562-b0bc-58b37e179a59' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:84d2fe66-bcd9-4562-b0bc-58b37e179a59' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:84d2fe66-bcd9-4562-b0bc-58b37e179a59' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 300 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 84d2fe66-bcd9-4562-b0bc-58b37e179a59 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 84d2fe66-bcd9-4562-b0bc-58b37e179a59 ++ '[' -n '' ']' ++ '[' -n '' ']' +++ date +%s ++ local endtime=1497027893 +++ expr 1497027893 - 1497027816 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 77 seconds at 2017-06-09 17:04:53.580799825+00:00 ++ echo END wait_for_fluentd_to_catch_up took 77 seconds at 2017-06-09 17:04:53.580799825+00:00 ++ return 0 ++ return 0 ++ cleanup ++ cleanup_forward ++ oc label node --all logging-infra-fluentd- node "172.18.12.108" labeled ++ wait_for_pod_ACTION stop logging-fluentd-26stf ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-26stf ++ '[' -z logging-fluentd-26stf -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-26stf ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc delete daemonset/logging-forward-fluentd daemonset "logging-forward-fluentd" deleted +++ grep '<match \*\*>' +++ oc get configmap/logging-fluentd -o yaml ++ '[' -n ' <match **>' ']' ++ oc get configmap/logging-fluentd -o yaml ++ sed -e '/<match \*\*>/ d' -e '/@include configs\.d\/user\/secure-forward\.conf/ d' -e '/<\/match>/ d' ++ oc replace -f - configmap "logging-fluentd" replaced ++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\ # @type secure_forward\n\ # self_hostname forwarding-${HOSTNAME}\n\ # shared_key aggregated_logging_ci_testing\n\ # secure no\n\ # <server>\n\ # host ${FLUENTD_FORWARD}\n\ # port 24284\n\ # </server>"}]' configmap "logging-fluentd" patched ++ oc label node --all logging-infra-fluentd=true node "172.18.12.108" labeled ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' pod for component=fluentd not running yet ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' ++ echo pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-fptx6 ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-fptx6 ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-fptx6 ++ oc get events -o yaml ++ write_and_verify_logs 1 ++ expected=1 ++ rc=0 ++ wait_for_fluentd_to_catch_up '' '' +++ date +%s ++ local starttime=1497027913 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:05:13.282831831+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:05:13.282831831+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=14a14546-3450-4ca1-8ebc-4db5642c2524 +++ uuidgen ++ local uuid_es_ops=b31455dc-7230-4a2f-8181-be6fb791edd7 ++ local expected=1 ++ local timeout=300 ++ add_test_message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/14a14546-3450-4ca1-8ebc-4db5642c2524 ++ echo added es message 14a14546-3450-4ca1-8ebc-4db5642c2524 added es message 14a14546-3450-4ca1-8ebc-4db5642c2524 ++ logger -i -p local6.info -t b31455dc-7230-4a2f-8181-be6fb791edd7 b31455dc-7230-4a2f-8181-be6fb791edd7 added es-ops message b31455dc-7230-4a2f-8181-be6fb791edd7 ++ echo added es-ops message b31455dc-7230-4a2f-8181-be6fb791edd7 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=14a14546-3450-4ca1-8ebc-4db5642c2524 ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 290 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 289 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 288 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 287 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 14a14546-3450-4ca1-8ebc-4db5642c2524 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:14a14546-3450-4ca1-8ebc-4db5642c2524' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 287 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project logging for 14a14546-3450-4ca1-8ebc-4db5642c2524 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 14a14546-3450-4ca1-8ebc-4db5642c2524 ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=b31455dc-7230-4a2f-8181-be6fb791edd7 ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER b31455dc-7230-4a2f-8181-be6fb791edd7 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:b31455dc-7230-4a2f-8181-be6fb791edd7' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:b31455dc-7230-4a2f-8181-be6fb791edd7' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:b31455dc-7230-4a2f-8181-be6fb791edd7' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 300 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for b31455dc-7230-4a2f-8181-be6fb791edd7 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for b31455dc-7230-4a2f-8181-be6fb791edd7 ++ '[' -n '' ']' ++ '[' -n '' ']' +++ date +%s ++ local endtime=1497027933 +++ expr 1497027933 - 1497027913 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-09 17:05:33.008995954+00:00 ++ echo END wait_for_fluentd_to_catch_up took 20 seconds at 2017-06-09 17:05:33.008995954+00:00 ++ return 0 ++ return 0 ++ cleanup ++ cleanup_forward ++ oc label node --all logging-infra-fluentd- node "172.18.12.108" labeled ++ wait_for_pod_ACTION stop logging-fluentd-fptx6 ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-fptx6 ++ '[' -z logging-fluentd-fptx6 -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-fptx6 ++ '[' -n 1 ']' pod logging-fluentd-fptx6 still running ++ echo pod logging-fluentd-fptx6 still running ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' stop = start ']' ++ '[' 110 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-fptx6 ++ '[' stop = start ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 ++ oc delete daemonset/logging-forward-fluentd Error from server (NotFound): daemonsets.extensions "logging-forward-fluentd" not found ++ : +++ oc get configmap/logging-fluentd -o yaml +++ grep '<match \*\*>' ++ '[' -n '' ']' ++ oc patch configmap/logging-fluentd --type=json --patch '[{ "op": "replace", "path": "/data/secure-forward.conf", "value": "\ # @type secure_forward\n\ # self_hostname forwarding-${HOSTNAME}\n\ # shared_key aggregated_logging_ci_testing\n\ # secure no\n\ # <server>\n\ # host ${FLUENTD_FORWARD}\n\ # port 24284\n\ # </server>"}]' configmap "logging-fluentd" not patched ++ oc label node --all logging-infra-fluentd=true node "172.18.12.108" labeled ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' ++ echo pod for component=fluentd not running yet pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-wlspv ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-wlspv ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' +++ oc get pods -l component=fluentd ++ fpod=logging-fluentd-wlspv ++ oc get events -o yaml running test test-json-parsing.sh ++ set -o nounset ++ set -o pipefail ++ type get_running_pod ++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts ++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']' +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ es_pod=logging-es-data-master-03l0u89m-1-p747h +++ uuidgen ++ uuid_es=f0bdebdd-82e5-4aa7-a864-6190b9b2e389 ++ echo Adding test message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 to Kibana . . . Adding test message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 to Kibana . . . ++ add_test_message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/f0bdebdd-82e5-4aa7-a864-6190b9b2e389 ++ rc=0 ++ timeout=600 ++ echo Waiting 600 seconds for f0bdebdd-82e5-4aa7-a864-6190b9b2e389 to show up in Elasticsearch . . . Waiting 600 seconds for f0bdebdd-82e5-4aa7-a864-6190b9b2e389 to show up in Elasticsearch . . . ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging. ++ mymessage=f0bdebdd-82e5-4aa7-a864-6190b9b2e389 ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 600 ++ let ii=600 ++ local interval=1 ++ '[' 600 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 599 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 598 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 597 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 596 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 595 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 594 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 593 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 592 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 591 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 590 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 589 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _count message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_count?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 589 -le 0 ']' ++ return 0 good - ./logging.sh: found 1 record project logging for f0bdebdd-82e5-4aa7-a864-6190b9b2e389 ++ echo good - ./logging.sh: found 1 record project logging for f0bdebdd-82e5-4aa7-a864-6190b9b2e389 Testing if record is in correct format . . . ++ echo Testing if record is in correct format . . . ++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging. _search message f0bdebdd-82e5-4aa7-a864-6190b9b2e389 ++ python test-json-parsing.py f0bdebdd-82e5-4aa7-a864-6190b9b2e389 ++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging.*/_search?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' --connect-timeout 1 ++ local pod=logging-es-data-master-03l0u89m-1-p747h ++ local 'endpoint=/project.logging.*/_search?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' ++ shift ++ shift ++ args=("${@:-}") ++ local args ++ local secret_dir=/etc/elasticsearch/secret/ ++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging.*/_search?q=message:f0bdebdd-82e5-4aa7-a864-6190b9b2e389' Success: record contains all of the expected fields/values Success: ./logging.sh passed ++ echo Success: ./logging.sh passed ++ exit 0 running test test-mux.sh ++ set -o nounset ++ set -o pipefail ++ type get_running_pod ++ '[' false == false -o false == false ']' Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true. ++ echo 'Skipping -- This test requires both USE_MUX_CLIENT and MUX_ALLOW_EXTERNAL are true.' ++ exit 0 SKIPPING upgrade test for now running test test-viaq-data-model.sh ++ set -o nounset ++ set -o pipefail ++ type get_running_pod ++ [[ 1 -ne 1 ]] ++ [[ true = \f\a\l\s\e ]] ++ CLUSTER=true ++ ops=-ops ++ INDEX_PREFIX= ++ PROJ_PREFIX=project. ++ ARTIFACT_DIR=/tmp/origin-aggregated-logging/artifacts ++ '[' '!' -d /tmp/origin-aggregated-logging/artifacts ']' ++ get_test_user_token ++ local current_project +++ oc project -q ++ current_project=logging ++ oc login --username=admin --password=admin +++ oc whoami -t ++ test_token=gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA +++ oc whoami ++ test_name=admin ++ test_ip=127.0.0.1 ++ oc login --username=system:admin ++ oc project logging ++ TEST_DIVIDER=------------------------------------------ +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-wlspv ++ remove_test_volume ++ oc get template logging-fluentd-template -o json ++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)' ++ oc replace -f - template "logging-fluentd-template" replaced +++ mktemp ++ cfg=/tmp/tmp.RDolQxko64 ++ cat ++ add_test_volume /tmp/tmp.RDolQxko64 ++ oc get template logging-fluentd-template -o json ++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"].append({"name": "cdmtest", "mountPath": "/etc/fluent/configs.d/openshift/filter-pre-cdm-test.conf", "readOnly": True}); obj["objects"][0]["spec"]["template"]["spec"]["volumes"].append({"name": "cdmtest", "hostPath": {"path": "/tmp/tmp.RDolQxko64"}}); print json.dumps(obj, indent=2)' ++ oc replace -f - template "logging-fluentd-template" replaced ++ trap cleanup INT TERM EXIT ++ restart_fluentd ++ oc delete daemonset logging-fluentd daemonset "logging-fluentd" deleted ++ wait_for_pod_ACTION stop logging-fluentd-wlspv ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-wlspv ++ '[' -z logging-fluentd-wlspv -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-wlspv ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc process logging-fluentd-template ++ oc create -f - daemonset "logging-fluentd" created ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' ++ echo pod for component=fluentd not running yet pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-jp70j ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-jp70j ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-jp70j ++ keep_fields=method,statusCode,type,@timestamp,req,res ++ write_and_verify_logs test1 ++ expected=1 ++ rc=0 ++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2 +++ date +%s ++ local starttime=1497027992 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:06:32.398679961+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:06:32.398679961+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ uuidgen ++ local uuid_es_ops=7168732c-49e6-4d5b-bb43-4d2f46ace509 ++ local expected=1 ++ local timeout=300 ++ add_test_message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/65ebea43-3d53-4af4-b80d-5d23fe1a4e27 added es message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 ++ echo added es message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 ++ logger -i -p local6.info -t 7168732c-49e6-4d5b-bb43-4d2f46ace509 7168732c-49e6-4d5b-bb43-4d2f46ace509 added es-ops message 7168732c-49e6-4d5b-bb43-4d2f46ace509 ++ echo added es-ops message 7168732c-49e6-4d5b-bb43-4d2f46ace509 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=65ebea43-3d53-4af4-b80d-5d23fe1a4e27 ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 290 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 290 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project logging for 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=7168732c-49e6-4d5b-bb43-4d2f46ace509 ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER 7168732c-49e6-4d5b-bb43-4d2f46ace509 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:7168732c-49e6-4d5b-bb43-4d2f46ace509' --connect-timeout 1 +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:7168732c-49e6-4d5b-bb43-4d2f46ace509' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:7168732c-49e6-4d5b-bb43-4d2f46ace509' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 300 -le 0 ']' ++ return 0 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 7168732c-49e6-4d5b-bb43-4d2f46ace509 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 7168732c-49e6-4d5b-bb43-4d2f46ace509 ++ '[' -n get_logmessage ']' ++ get_logmessage 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 ++ logmessage=65ebea43-3d53-4af4-b80d-5d23fe1a4e27 ++ '[' -n get_logmessage2 ']' ++ get_logmessage2 7168732c-49e6-4d5b-bb43-4d2f46ace509 ++ logmessage2=7168732c-49e6-4d5b-bb43-4d2f46ace509 +++ date +%s ++ local endtime=1497028007 +++ expr 1497028007 - 1497027992 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-09 17:06:47.988584526+00:00 ++ echo END wait_for_fluentd_to_catch_up took 15 seconds at 2017-06-09 17:06:47.988584526+00:00 ++ return 0 +++ get_running_pod kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' +++ oc get pods -l component=kibana ++ kpod=logging-kibana-1-20vz2 ++ '[' 0 = 0 ']' ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es project.logging _search message 65ebea43-3d53-4af4-b80d-5d23fe1a4e27 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:65ebea43-3d53-4af4-b80d-5d23fe1a4e27' ++ python test-viaq-data-model.py test1 ++ : ++ '[' 0 = 0 ']' ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es-ops .operations _search message 7168732c-49e6-4d5b-bb43-4d2f46ace509 ++ python test-viaq-data-model.py test1 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:7168732c-49e6-4d5b-bb43-4d2f46ace509' ++ : ++ '[' 0 '!=' 0 ']' ++ return 0 ++ add_cdm_env_var_val CDM_USE_UNDEFINED '"true"' +++ mktemp ++ junk=/tmp/tmp.lfhSKXtVxx ++ cat ++ oc get template logging-fluentd-template -o yaml ++ sed '/env:/r /tmp/tmp.lfhSKXtVxx' ++ oc replace -f - template "logging-fluentd-template" replaced ++ rm -f /tmp/tmp.lfhSKXtVxx ++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS method,statusCode,type,@timestamp,req,res +++ mktemp ++ junk=/tmp/tmp.Zd72FTurmG ++ cat ++ oc get template logging-fluentd-template -o yaml ++ sed '/env:/r /tmp/tmp.Zd72FTurmG' ++ oc replace -f - template "logging-fluentd-template" replaced ++ rm -f /tmp/tmp.Zd72FTurmG ++ restart_fluentd ++ oc delete daemonset logging-fluentd daemonset "logging-fluentd" deleted ++ wait_for_pod_ACTION stop logging-fluentd-jp70j ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-jp70j ++ '[' -z logging-fluentd-jp70j -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-jp70j ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc process logging-fluentd-template ++ oc create -f - daemonset "logging-fluentd" created ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' pod for component=fluentd not running yet ++ echo pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-2hzht ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-2hzht ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-2hzht ++ write_and_verify_logs test2 ++ expected=1 ++ rc=0 ++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2 +++ date +%s ++ local starttime=1497028032 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:07:12.747955447+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:07:12.747955447+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=b389a17a-eeb3-4979-a15d-723888f2bb65 +++ uuidgen ++ local uuid_es_ops=c1eb0d52-5f45-4312-b34f-1e9983ce8530 ++ local expected=1 ++ local timeout=300 ++ add_test_message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/b389a17a-eeb3-4979-a15d-723888f2bb65 added es message b389a17a-eeb3-4979-a15d-723888f2bb65 ++ echo added es message b389a17a-eeb3-4979-a15d-723888f2bb65 ++ logger -i -p local6.info -t c1eb0d52-5f45-4312-b34f-1e9983ce8530 c1eb0d52-5f45-4312-b34f-1e9983ce8530 added es-ops message c1eb0d52-5f45-4312-b34f-1e9983ce8530 ++ echo added es-ops message c1eb0d52-5f45-4312-b34f-1e9983ce8530 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=b389a17a-eeb3-4979-a15d-723888f2bb65 ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 290 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 289 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 288 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 287 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message b389a17a-eeb3-4979-a15d-723888f2bb65 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 287 -le 0 ']' ++ return 0 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for b389a17a-eeb3-4979-a15d-723888f2bb65 good - wait_for_fluentd_to_catch_up: found 1 record project logging for b389a17a-eeb3-4979-a15d-723888f2bb65 ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=c1eb0d52-5f45-4312-b34f-1e9983ce8530 ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER c1eb0d52-5f45-4312-b34f-1e9983ce8530 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:c1eb0d52-5f45-4312-b34f-1e9983ce8530' --connect-timeout 1 +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:c1eb0d52-5f45-4312-b34f-1e9983ce8530' +++ shift +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:c1eb0d52-5f45-4312-b34f-1e9983ce8530' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 300 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for c1eb0d52-5f45-4312-b34f-1e9983ce8530 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for c1eb0d52-5f45-4312-b34f-1e9983ce8530 ++ '[' -n get_logmessage ']' ++ get_logmessage b389a17a-eeb3-4979-a15d-723888f2bb65 ++ logmessage=b389a17a-eeb3-4979-a15d-723888f2bb65 ++ '[' -n get_logmessage2 ']' ++ get_logmessage2 c1eb0d52-5f45-4312-b34f-1e9983ce8530 ++ logmessage2=c1eb0d52-5f45-4312-b34f-1e9983ce8530 +++ date +%s ++ local endtime=1497028053 +++ expr 1497028053 - 1497028032 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 21 seconds at 2017-06-09 17:07:33.032434180+00:00 ++ echo END wait_for_fluentd_to_catch_up took 21 seconds at 2017-06-09 17:07:33.032434180+00:00 ++ return 0 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ kpod=logging-kibana-1-20vz2 ++ '[' 0 = 0 ']' ++ python test-viaq-data-model.py test2 ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es project.logging _search message b389a17a-eeb3-4979-a15d-723888f2bb65 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:b389a17a-eeb3-4979-a15d-723888f2bb65' ++ : ++ '[' 0 = 0 ']' ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es-ops .operations _search message c1eb0d52-5f45-4312-b34f-1e9983ce8530 ++ python test-viaq-data-model.py test2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:c1eb0d52-5f45-4312-b34f-1e9983ce8530' ++ : ++ '[' 0 '!=' 0 ']' ++ return 0 ++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS ++ oc get template logging-fluentd-template -o yaml ++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d' ++ oc replace -f - template "logging-fluentd-template" replaced ++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,method,statusCode,type,@timestamp,req,res +++ mktemp ++ junk=/tmp/tmp.iybK4F5SSr ++ cat ++ oc get template logging-fluentd-template -o yaml ++ sed '/env:/r /tmp/tmp.iybK4F5SSr' ++ oc replace -f - template "logging-fluentd-template" replaced ++ rm -f /tmp/tmp.iybK4F5SSr ++ restart_fluentd ++ oc delete daemonset logging-fluentd daemonset "logging-fluentd" deleted ++ wait_for_pod_ACTION stop logging-fluentd-2hzht ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-2hzht ++ '[' -z logging-fluentd-2hzht -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-2hzht ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc process logging-fluentd-template ++ oc create -f - daemonset "logging-fluentd" created ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' pod for component=fluentd not running yet ++ echo pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-8kn33 ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-8kn33 ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-8kn33 ++ write_and_verify_logs test3 ++ expected=1 ++ rc=0 ++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2 +++ date +%s ++ local starttime=1497028083 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:08:03.093894993+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:08:03.093894993+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=3969390e-78ab-451c-b193-049a8b1dc50f +++ uuidgen ++ local uuid_es_ops=4852e26f-a8d7-410f-81fc-696a168be4b0 ++ local expected=1 ++ local timeout=300 ++ add_test_message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/3969390e-78ab-451c-b193-049a8b1dc50f added es message 3969390e-78ab-451c-b193-049a8b1dc50f ++ echo added es message 3969390e-78ab-451c-b193-049a8b1dc50f ++ logger -i -p local6.info -t 4852e26f-a8d7-410f-81fc-696a168be4b0 4852e26f-a8d7-410f-81fc-696a168be4b0 added es-ops message 4852e26f-a8d7-410f-81fc-696a168be4b0 ++ echo added es-ops message 4852e26f-a8d7-410f-81fc-696a168be4b0 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=3969390e-78ab-451c-b193-049a8b1dc50f ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ shift +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ get_count_from_json +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message 3969390e-78ab-451c-b193-049a8b1dc50f +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 291 -le 0 ']' ++ return 0 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for 3969390e-78ab-451c-b193-049a8b1dc50f good - wait_for_fluentd_to_catch_up: found 1 record project logging for 3969390e-78ab-451c-b193-049a8b1dc50f ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=4852e26f-a8d7-410f-81fc-696a168be4b0 ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER 4852e26f-a8d7-410f-81fc-696a168be4b0 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER 4852e26f-a8d7-410f-81fc-696a168be4b0 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER 4852e26f-a8d7-410f-81fc-696a168be4b0 +++ get_count_from_json +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' --connect-timeout 1 +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER 4852e26f-a8d7-410f-81fc-696a168be4b0 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:4852e26f-a8d7-410f-81fc-696a168be4b0' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 297 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 4852e26f-a8d7-410f-81fc-696a168be4b0 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 4852e26f-a8d7-410f-81fc-696a168be4b0 ++ '[' -n get_logmessage ']' ++ get_logmessage 3969390e-78ab-451c-b193-049a8b1dc50f ++ logmessage=3969390e-78ab-451c-b193-049a8b1dc50f ++ '[' -n get_logmessage2 ']' ++ get_logmessage2 4852e26f-a8d7-410f-81fc-696a168be4b0 ++ logmessage2=4852e26f-a8d7-410f-81fc-696a168be4b0 +++ date +%s ++ local endtime=1497028101 +++ expr 1497028101 - 1497028083 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 18 seconds at 2017-06-09 17:08:21.778546655+00:00 ++ echo END wait_for_fluentd_to_catch_up took 18 seconds at 2017-06-09 17:08:21.778546655+00:00 ++ return 0 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ kpod=logging-kibana-1-20vz2 ++ '[' 0 = 0 ']' ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es project.logging _search message 3969390e-78ab-451c-b193-049a8b1dc50f ++ python test-viaq-data-model.py test3 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:3969390e-78ab-451c-b193-049a8b1dc50f' ++ : ++ '[' 0 = 0 ']' ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es-ops .operations _search message 4852e26f-a8d7-410f-81fc-696a168be4b0 ++ python test-viaq-data-model.py test3 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:4852e26f-a8d7-410f-81fc-696a168be4b0' ++ : ++ '[' 0 '!=' 0 ']' ++ return 0 ++ add_cdm_env_var_val CDM_UNDEFINED_NAME myname +++ mktemp ++ junk=/tmp/tmp.45FCBqG641 ++ cat ++ oc get template logging-fluentd-template -o yaml ++ sed '/env:/r /tmp/tmp.45FCBqG641' ++ oc replace -f - template "logging-fluentd-template" replaced ++ rm -f /tmp/tmp.45FCBqG641 ++ restart_fluentd ++ oc delete daemonset logging-fluentd daemonset "logging-fluentd" deleted ++ wait_for_pod_ACTION stop logging-fluentd-8kn33 ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-8kn33 ++ '[' -z logging-fluentd-8kn33 -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-8kn33 ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc process logging-fluentd-template ++ oc create -f - daemonset "logging-fluentd" created ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' pod for component=fluentd not running yet ++ echo pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-j07sl ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-j07sl ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-j07sl ++ write_and_verify_logs test4 ++ expected=1 ++ rc=0 ++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2 +++ date +%s ++ local starttime=1497028122 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:08:42.870313947+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:08:42.870313947+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=bc03c63b-5593-4717-b304-cc81085a12d5 +++ uuidgen ++ local uuid_es_ops=337a6519-d385-4768-a421-599d6dc5c462 ++ local expected=1 ++ local timeout=300 ++ add_test_message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/bc03c63b-5593-4717-b304-cc81085a12d5 added es message bc03c63b-5593-4717-b304-cc81085a12d5 ++ echo added es message bc03c63b-5593-4717-b304-cc81085a12d5 ++ logger -i -p local6.info -t 337a6519-d385-4768-a421-599d6dc5c462 337a6519-d385-4768-a421-599d6dc5c462 added es-ops message 337a6519-d385-4768-a421-599d6dc5c462 ++ echo added es-ops message 337a6519-d385-4768-a421-599d6dc5c462 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=bc03c63b-5593-4717-b304-cc81085a12d5 ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ shift +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 290 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message bc03c63b-5593-4717-b304-cc81085a12d5 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 290 -le 0 ']' ++ return 0 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for bc03c63b-5593-4717-b304-cc81085a12d5 good - wait_for_fluentd_to_catch_up: found 1 record project logging for bc03c63b-5593-4717-b304-cc81085a12d5 ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=337a6519-d385-4768-a421-599d6dc5c462 ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER 337a6519-d385-4768-a421-599d6dc5c462 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:337a6519-d385-4768-a421-599d6dc5c462' --connect-timeout 1 +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:337a6519-d385-4768-a421-599d6dc5c462' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:337a6519-d385-4768-a421-599d6dc5c462' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 300 -le 0 ']' ++ return 0 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 337a6519-d385-4768-a421-599d6dc5c462 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for 337a6519-d385-4768-a421-599d6dc5c462 ++ '[' -n get_logmessage ']' ++ get_logmessage bc03c63b-5593-4717-b304-cc81085a12d5 ++ logmessage=bc03c63b-5593-4717-b304-cc81085a12d5 ++ '[' -n get_logmessage2 ']' ++ get_logmessage2 337a6519-d385-4768-a421-599d6dc5c462 ++ logmessage2=337a6519-d385-4768-a421-599d6dc5c462 +++ date +%s ++ local endtime=1497028138 +++ expr 1497028138 - 1497028122 +++ date -u --rfc-3339=ns END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-09 17:08:58.688754825+00:00 ++ echo END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-09 17:08:58.688754825+00:00 ++ return 0 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ kpod=logging-kibana-1-20vz2 ++ '[' 0 = 0 ']' ++ python test-viaq-data-model.py test4 ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es project.logging _search message bc03c63b-5593-4717-b304-cc81085a12d5 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:bc03c63b-5593-4717-b304-cc81085a12d5' ++ : ++ '[' 0 = 0 ']' ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es-ops .operations _search message 337a6519-d385-4768-a421-599d6dc5c462 ++ python test-viaq-data-model.py test4 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:337a6519-d385-4768-a421-599d6dc5c462' ++ : ++ '[' 0 '!=' 0 ']' ++ return 0 ++ del_cdm_env_var CDM_EXTRA_KEEP_FIELDS ++ oc get template logging-fluentd-template -o yaml ++ sed '/- name: CDM_EXTRA_KEEP_FIELDS$/,/value:/d' ++ oc replace -f - template "logging-fluentd-template" replaced ++ add_cdm_env_var_val CDM_EXTRA_KEEP_FIELDS undefined4,undefined5,empty1,undefined3,method,statusCode,type,@timestamp,req,res +++ mktemp ++ junk=/tmp/tmp.W4kUBEefWm ++ cat ++ oc get template logging-fluentd-template -o yaml ++ sed '/env:/r /tmp/tmp.W4kUBEefWm' ++ oc replace -f - template "logging-fluentd-template" replaced ++ rm -f /tmp/tmp.W4kUBEefWm ++ add_cdm_env_var_val CDM_KEEP_EMPTY_FIELDS undefined4,undefined5,empty1,undefined3 +++ mktemp ++ junk=/tmp/tmp.0ezaoNyQq7 ++ cat ++ oc get template logging-fluentd-template -o yaml ++ sed '/env:/r /tmp/tmp.0ezaoNyQq7' ++ oc replace -f - template "logging-fluentd-template" replaced ++ rm -f /tmp/tmp.0ezaoNyQq7 ++ restart_fluentd ++ oc delete daemonset logging-fluentd daemonset "logging-fluentd" deleted ++ wait_for_pod_ACTION stop logging-fluentd-j07sl ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-j07sl ++ '[' -z logging-fluentd-j07sl -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-j07sl ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc process logging-fluentd-template ++ oc create -f - daemonset "logging-fluentd" created ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' pod for component=fluentd not running yet ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' ++ echo pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-dhzkd ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-dhzkd ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ fpod=logging-fluentd-dhzkd ++ write_and_verify_logs test5 allow_empty ++ expected=1 ++ rc=0 ++ wait_for_fluentd_to_catch_up get_logmessage get_logmessage2 +++ date +%s ++ local starttime=1497028156 +++ date -u --rfc-3339=ns START wait_for_fluentd_to_catch_up at 2017-06-09 17:09:16.974250432+00:00 ++ echo START wait_for_fluentd_to_catch_up at 2017-06-09 17:09:16.974250432+00:00 +++ get_running_pod es +++ oc get pods -l component=es +++ awk -v sel=es '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_pod=logging-es-data-master-03l0u89m-1-p747h +++ get_running_pod es-ops +++ oc get pods -l component=es-ops +++ awk -v sel=es-ops '$1 ~ sel && $3 == "Running" {print $1}' ++ local es_ops_pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ '[' -z logging-es-ops-data-master-kq3j3au5-1-8bzmq ']' +++ uuidgen ++ local uuid_es=caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ uuidgen ++ local uuid_es_ops=f99e259d-842c-403d-9fbe-051095012a63 ++ local expected=1 ++ local timeout=300 ++ add_test_message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ local kib_pod=logging-kibana-1-20vz2 ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s http://localhost:5601/caf17305-fa8f-4372-a286-5d9bc60eb0c2 ++ echo added es message caf17305-fa8f-4372-a286-5d9bc60eb0c2 added es message caf17305-fa8f-4372-a286-5d9bc60eb0c2 ++ logger -i -p local6.info -t f99e259d-842c-403d-9fbe-051095012a63 f99e259d-842c-403d-9fbe-051095012a63 added es-ops message f99e259d-842c-403d-9fbe-051095012a63 ++ echo added es-ops message f99e259d-842c-403d-9fbe-051095012a63 ++ local rc=0 ++ espod=logging-es-data-master-03l0u89m-1-p747h ++ myproject=project.logging ++ mymessage=caf17305-fa8f-4372-a286-5d9bc60eb0c2 ++ expected=1 ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 299 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 298 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 297 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 296 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 295 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 294 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 293 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 292 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ args=("${@:-}") +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 291 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=0 ++ test 0 = 1 ++ sleep 1 ++ let ii=ii-1 ++ '[' 290 -gt 0 ']' ++ test_count_expected ++ myfield=message +++ query_es_from_es logging-es-data-master-03l0u89m-1-p747h project.logging _count message caf17305-fa8f-4372-a286-5d9bc60eb0c2 +++ get_count_from_json +++ curl_es logging-es-data-master-03l0u89m-1-p747h '/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' --connect-timeout 1 +++ local pod=logging-es-data-master-03l0u89m-1-p747h +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local 'endpoint=/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-data-master-03l0u89m-1-p747h -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/project.logging*/_count?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 290 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project logging for caf17305-fa8f-4372-a286-5d9bc60eb0c2 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project logging for caf17305-fa8f-4372-a286-5d9bc60eb0c2 ++ espod=logging-es-ops-data-master-kq3j3au5-1-8bzmq ++ myproject=.operations ++ mymessage=f99e259d-842c-403d-9fbe-051095012a63 ++ expected=1 ++ myfield=systemd.u.SYSLOG_IDENTIFIER ++ wait_until_cmd_or_err test_count_expected test_count_err 300 ++ let ii=300 ++ local interval=1 ++ '[' 300 -gt 0 ']' ++ test_count_expected ++ myfield=systemd.u.SYSLOG_IDENTIFIER +++ query_es_from_es logging-es-ops-data-master-kq3j3au5-1-8bzmq .operations _count systemd.u.SYSLOG_IDENTIFIER f99e259d-842c-403d-9fbe-051095012a63 +++ get_count_from_json +++ curl_es logging-es-ops-data-master-kq3j3au5-1-8bzmq '/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:f99e259d-842c-403d-9fbe-051095012a63' --connect-timeout 1 +++ python -c 'import json, sys; print json.loads(sys.stdin.read()).get("count", 0)' +++ local pod=logging-es-ops-data-master-kq3j3au5-1-8bzmq +++ local 'endpoint=/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:f99e259d-842c-403d-9fbe-051095012a63' +++ shift +++ shift +++ args=("${@:-}") +++ local args +++ local secret_dir=/etc/elasticsearch/secret/ +++ oc exec logging-es-ops-data-master-kq3j3au5-1-8bzmq -- curl --silent --insecure --connect-timeout 1 --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert 'https://localhost:9200/.operations*/_count?q=systemd.u.SYSLOG_IDENTIFIER:f99e259d-842c-403d-9fbe-051095012a63' ++ local nrecs=1 ++ test 1 = 1 ++ break ++ '[' 300 -le 0 ']' ++ return 0 good - wait_for_fluentd_to_catch_up: found 1 record project .operations for f99e259d-842c-403d-9fbe-051095012a63 ++ echo good - wait_for_fluentd_to_catch_up: found 1 record project .operations for f99e259d-842c-403d-9fbe-051095012a63 ++ '[' -n get_logmessage ']' ++ get_logmessage caf17305-fa8f-4372-a286-5d9bc60eb0c2 ++ logmessage=caf17305-fa8f-4372-a286-5d9bc60eb0c2 ++ '[' -n get_logmessage2 ']' ++ get_logmessage2 f99e259d-842c-403d-9fbe-051095012a63 ++ logmessage2=f99e259d-842c-403d-9fbe-051095012a63 +++ date +%s ++ local endtime=1497028172 +++ expr 1497028172 - 1497028156 +++ date -u --rfc-3339=ns ++ echo END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-09 17:09:32.876357075+00:00 END wait_for_fluentd_to_catch_up took 16 seconds at 2017-06-09 17:09:32.876357075+00:00 ++ return 0 +++ get_running_pod kibana +++ oc get pods -l component=kibana +++ awk -v sel=kibana '$1 ~ sel && $3 == "Running" {print $1}' ++ kpod=logging-kibana-1-20vz2 ++ '[' 0 = 0 ']' ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es project.logging _search message caf17305-fa8f-4372-a286-5d9bc60eb0c2 ++ python test-viaq-data-model.py test5 allow_empty ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es:9200/project.logging*/_search?q=message:caf17305-fa8f-4372-a286-5d9bc60eb0c2' ++ : ++ '[' 0 = 0 ']' ++ curl_es_from_kibana logging-kibana-1-20vz2 logging-es-ops .operations _search message f99e259d-842c-403d-9fbe-051095012a63 ++ python test-viaq-data-model.py test5 allow_empty ++ oc exec logging-kibana-1-20vz2 -c kibana -- curl --connect-timeout 1 -s -k --cert /etc/kibana/keys/cert --key /etc/kibana/keys/key -H 'X-Proxy-Remote-User: admin' -H 'Authorization: Bearer gq0SLvitkDNWYLytx7I6FpfYNPl8zvvHEE_r36Gr3lA' -H 'X-Forwarded-For: 127.0.0.1' 'https://logging-es-ops:9200/.operations*/_search?q=message:f99e259d-842c-403d-9fbe-051095012a63' ++ : ++ '[' 0 '!=' 0 ']' ++ return 0 ++ cleanup ++ remove_test_volume ++ oc get template logging-fluentd-template -o json ++ oc replace -f - ++ python -c 'import json, sys; obj = json.loads(sys.stdin.read()); vm = obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"]; obj["objects"][0]["spec"]["template"]["spec"]["containers"][0]["volumeMounts"] = [xx for xx in vm if xx["name"] != "cdmtest"]; vs = obj["objects"][0]["spec"]["template"]["spec"]["volumes"]; obj["objects"][0]["spec"]["template"]["spec"]["volumes"] = [xx for xx in vs if xx["name"] != "cdmtest"]; print json.dumps(obj, indent=2)' template "logging-fluentd-template" replaced ++ remove_cdm_env ++ oc get template logging-fluentd-template -o yaml ++ sed '/- name: CDM_/,/value:/d' ++ oc replace -f - template "logging-fluentd-template" replaced ++ rm -f /tmp/tmp.RDolQxko64 ++ restart_fluentd ++ oc delete daemonset logging-fluentd daemonset "logging-fluentd" deleted ++ wait_for_pod_ACTION stop logging-fluentd-dhzkd ++ local ii=120 ++ local incr=10 ++ '[' stop = start ']' ++ curpod=logging-fluentd-dhzkd ++ '[' -z logging-fluentd-dhzkd -a -n '' ']' ++ '[' 120 -gt 0 ']' ++ '[' stop = stop ']' ++ oc describe pod/logging-fluentd-dhzkd ++ '[' stop = start ']' ++ break ++ '[' 120 -le 0 ']' ++ return 0 ++ oc process logging-fluentd-template ++ oc create -f - daemonset "logging-fluentd" created ++ wait_for_pod_ACTION start fluentd ++ local ii=120 ++ local incr=10 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod= ++ '[' 120 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z '' ']' ++ '[' -n '' ']' ++ '[' -n 1 ']' pod for component=fluentd not running yet ++ echo pod for component=fluentd not running yet ++ sleep 10 +++ expr 120 - 10 ++ ii=110 ++ '[' start = start ']' +++ get_running_pod fluentd +++ oc get pods -l component=fluentd +++ awk -v sel=fluentd '$1 ~ sel && $3 == "Running" {print $1}' ++ curpod=logging-fluentd-fmh08 ++ '[' 110 -gt 0 ']' ++ '[' start = stop ']' ++ '[' start = start ']' ++ '[' -z logging-fluentd-fmh08 ']' ++ break ++ '[' 110 -le 0 ']' ++ return 0 SKIPPING reinstall test for now /data/src/github.com/openshift/origin-aggregated-logging/hack/lib/log/system.sh: line 31: 4631 Terminated sar -A -o "${binary_logfile}" 1 86400 > /dev/null 2> "${stderr_logfile}" (wd: /data/src/github.com/openshift/origin-aggregated-logging) [INFO] [CLEANUP] Beginning cleanup routines... [INFO] [CLEANUP] Dumping cluster events to /tmp/origin-aggregated-logging/artifacts/events.txt [INFO] [CLEANUP] Dumping etcd contents to /tmp/origin-aggregated-logging/artifacts/etcd [WARNING] No compiled `etcdhelper` binary was found. Attempting to build one using: [WARNING] $ hack/build-go.sh tools/etcdhelper ++ Building go targets for linux/amd64: tools/etcdhelper /data/src/github.com/openshift/origin-aggregated-logging/../origin/hack/build-go.sh took 133 seconds 2017-06-09 13:12:18.673371 I | warning: ignoring ServerName for user-provided CA for backwards compatibility is deprecated [INFO] [CLEANUP] Dumping container logs to /tmp/origin-aggregated-logging/logs/containers [INFO] [CLEANUP] Truncating log files over 200M [INFO] [CLEANUP] Stopping docker containers [INFO] [CLEANUP] Removing docker containers Error: No such image, container or task: af4778152aa4 json: cannot unmarshal array into Go value of type types.ContainerJSON Error: No such image, container or task: 76b2d93db69b json: cannot unmarshal array into Go value of type types.ContainerJSON Error response from daemon: You cannot remove a running container f91738558655a00ecd7de4497135a14b2f4b868606448f06983e84d1b66bb26f. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container a4b670b064c30c1bcce7c2776d5cc7c927f7e3cb0bc4290eb7d678823d601895. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container 2d54c6eff8dd4b518516f2f6197e6b2b7bafa7331f93693034c2eea34810c50d. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container 411809a267306f2a925027dc9d7a330d36cce5729d66907c7b03a29fc7c69352. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container 2c0e6027dac52c14ae351399fcfbc60675dc7c6c9acf5b7608388f9468bc0b10. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container 8fcb129873c9d26bd0c07db7924e946a39058738b575ce2ed4cc85687fdaca53. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container d20e7831b671cb055f288b5c3a7cb174aa80cd805b00dc597a4044405dff295d. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container fb382d6e75dc6f3e92614706e94cec64778f05ae0d3f7356750275af0378fe0f. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container 5c6b1b7cc2e8ba148366572119dab27b4ca2faa5f03217608d107db5ca8d643e. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container 7489b531745811128aaf9659dbd34d7bdcaa3d994f01e3141f4b09f8c600487a. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container fe009e665b24961c987136d61de85c7215d78ec851fdfa74a85dc782aaec891e. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container 15af1cc5e3aac3330feae7dd61bc152c453eda17feed58712dd4404bbe78c222. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container 8112e9a370cd1d458fb9b4229091195f0a68dc80a19d934079068e4367dee57b. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container 92c24ab653aba274a5c90d14babb7a20c88ca6a23bbb6f3d2a98bef7eacb470e. Stop the container before attempting removal or use -f Error response from daemon: You cannot remove a running container fb1a675e918f0106f9747502faacca78204be1695f0c4862fff95ccdd8642c51. Stop the container before attempting removal or use -f [INFO] [CLEANUP] Killing child processes [INFO] [CLEANUP] Pruning etcd data directory [INFO] /data/src/github.com/openshift/origin-aggregated-logging/logging.sh exited with code 0 after 00h 58m 24s real 58m24.084s user 5m32.952s sys 1m5.280s Finished GIT_URL=https://github.com/openshift/origin-aggregated-logging GIT_BRANCH=master O_A_L_DIR=/data/src/github.com/openshift/origin-aggregated-logging OS_ROOT=/data/src/github.com/openshift/origin ENABLE_OPS_CLUSTER=true USE_LOCAL_SOURCE=true TEST_PERF=false VERBOSE=1 OS_ANSIBLE_REPO=https://github.com/openshift/openshift-ansible OS_ANSIBLE_BRANCH=master ./logging.sh *************************************************** ==> openshiftdev: Downloading logs ==> openshiftdev: Downloading artifacts from '/var/log/yum.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/yum.log' ==> openshiftdev: Downloading artifacts from '/var/log/secure' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/secure' ==> openshiftdev: Downloading artifacts from '/var/log/audit/audit.log' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts/audit.log' ==> openshiftdev: Downloading artifacts from '/tmp/origin-aggregated-logging/' to '/var/lib/jenkins/jobs/test-origin-aggregated-logging/workspace@2/origin/artifacts' + false + false + test_pull_requests --mark_test_success --repo origin-aggregated-logging --config /var/lib/jenkins/.test_pull_requests_logging.json Rate limit remaining: 3509 Marking SUCCESS for pull request #--repo in repo '' /usr/share/ruby/net/http/response.rb:119:in `error!': 404 "Not Found" (Net::HTTPServerException) from /bin/test_pull_requests:543:in `block in get_comments' from /bin/test_pull_requests:535:in `each' from /bin/test_pull_requests:535:in `get_comments' from /bin/test_pull_requests:1074:in `get_comment_matching_regex' from /bin/test_pull_requests:1090:in `get_comment_with_prefix' from /bin/test_pull_requests:779:in `mark_test_success' from /bin/test_pull_requests:2361:in `<main>' + true [description-setter] Could not determine description. [PostBuildScript] - Execution post build scripts. [workspace@2] $ /bin/sh -xe /tmp/hudson8019278742756084700.sh + INSTANCE_NAME=origin_logging-rhel7-1655 + pushd origin ~/jobs/test-origin-aggregated-logging/workspace@2/origin ~/jobs/test-origin-aggregated-logging/workspace@2 + rc=0 + '[' -f .vagrant-openshift.json ']' ++ /usr/bin/vagrant ssh -c 'sudo ausearch -m avc' + ausearchresult='<no matches>' + rc=1 + '[' '<no matches>' = '<no matches>' ']' + rc=0 + /usr/bin/vagrant destroy -f ==> openshiftdev: Terminating the instance... ==> openshiftdev: Running cleanup tasks for 'shell' provisioner... + popd ~/jobs/test-origin-aggregated-logging/workspace@2 + exit 0 Finished: SUCCESS