{"action":"created","issue":{"url":"https://api.github.com/repos/kata-containers/runtime/issues/702","repository_url":"https://api.github.com/repos/kata-containers/runtime","labels_url":"https://api.github.com/repos/kata-containers/runtime/issues/702/labels{/name}","comments_url":"https://api.github.com/repos/kata-containers/runtime/issues/702/comments","events_url":"https://api.github.com/repos/kata-containers/runtime/issues/702/events","html_url":"https://github.com/kata-containers/runtime/issues/702","id":357380354,"node_id":"MDU6SXNzdWUzNTczODAzNTQ=","number":702,"title":"Kata fails to start","user":{"login":"ydjainopensource","id":33664673,"node_id":"MDQ6VXNlcjMzNjY0Njcz","avatar_url":"https://avatars0.githubusercontent.com/u/33664673?v=4","gravatar_id":"","url":"https://api.github.com/users/ydjainopensource","html_url":"https://github.com/ydjainopensource","followers_url":"https://api.github.com/users/ydjainopensource/followers","following_url":"https://api.github.com/users/ydjainopensource/following{/other_user}","gists_url":"https://api.github.com/users/ydjainopensource/gists{/gist_id}","starred_url":"https://api.github.com/users/ydjainopensource/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ydjainopensource/subscriptions","organizations_url":"https://api.github.com/users/ydjainopensource/orgs","repos_url":"https://api.github.com/users/ydjainopensource/repos","events_url":"https://api.github.com/users/ydjainopensource/events{/privacy}","received_events_url":"https://api.github.com/users/ydjainopensource/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2018-09-05T19:58:38Z","updated_at":"2018-09-24T08:47:16Z","closed_at":null,"author_association":"NONE","body":"# Description of problem\r\n\r\nI was trying to port runtime to Z. It passes all the unit tests as of now.  I used the following command to run `kata-runtime --log /dev/stdout create -b ../osbuilder/ubuntu_rootfs test`\r\n\r\nHowever, I got `Failed to check if grpc server is working: rpc error: code = DeadlineExceeded desc = context deadline exceeded`\r\n\r\n\r\n\r\n---\r\n# Meta details\r\n\r\nRunning `kata-collect-data.sh` version `1.2.0 (commit 1820047c4e028314a2a19d1773d1aa26aa42e1f8-dirty)` at `2018-09-05.15:29:54.876826586-0400`.\r\n\r\n---\r\n\r\nRuntime is `/usr/local/bin/kata-runtime`.\r\n\r\n# `kata-env`\r\n\r\nOutput of \"`/usr/local/bin/kata-runtime kata-env`\":\r\n```\r\ncannot find model field in file /proc/cpuinfo\r\n```\r\n\r\n---\r\n\r\n\r\n# Runtime config files\r\n\r\n\r\n## Runtime default config files\r\n\r\n```\r\n/etc/kata-containers/configuration.toml\r\n/usr/share/defaults/kata-containers/configuration.toml\r\n```\r\n\r\n## Runtime config file contents\r\n\r\nConfig file `/etc/kata-containers/configuration.toml` not found\r\nOutput of \"`cat \"/usr/share/defaults/kata-containers/configuration.toml\"`\":\r\n```\r\n# Copyright (c) 2017-2018 Intel Corporation\r\n#\r\n# SPDX-License-Identifier: Apache-2.0\r\n#\r\n\r\n# XXX: WARNING: this file is auto-generated.\r\n# XXX:\r\n# XXX: Source file: \"cli/config/configuration.toml.in\"\r\n# XXX: Project:\r\n# XXX:   Name: Kata Containers\r\n# XXX:   Type: kata\r\n\r\n[hypervisor.qemu]\r\npath = \"/usr/bin/qemu-system-s390x\"\r\nkernel = \"/usr/share/kata-containers/vmlinuz.container\"\r\ninitrd = \"/usr/share/kata-containers/kata-containers-initrd.img\"\r\n#image = \"/usr/share/kata-containers/kata-containers.img\"\r\nmachine_type = \"s390-ccw-virtio \"\r\n\r\n# Optional space-separated list of options to pass to the guest kernel.\r\n# For example, use `kernel_params = \"vsyscall=emulate\"` if you are having\r\n# trouble running pre-2.15 glibc.\r\n#\r\n# WARNING: - any parameter specified here will take priority over the default\r\n# parameter value of the same name used to start the virtual machine.\r\n# Do not set values here unless you understand the impact of doing so as you\r\n# may stop the virtual machine from booting.\r\n# To see the list of default parameters, enable hypervisor debug, create a\r\n# container and look for 'default-kernel-parameters' log entries.\r\nkernel_params = \"\"\r\n\r\n# Path to the firmware.\r\n# If you want that qemu uses the default firmware leave this option empty\r\nfirmware = \"\"\r\n\r\n# Machine accelerators\r\n# comma-separated list of machine accelerators to pass to the hypervisor.\r\n# For example, `machine_accelerators = \"nosmm,nosmbus,nosata,nopit,static-prt,nofw\"`\r\nmachine_accelerators=\"\"\r\n\r\n# Default number of vCPUs per SB/VM:\r\n# unspecified or 0                --> will be set to 1\r\n# < 0                             --> will be set to the actual number of physical cores\r\n# > 0 <= number of physical cores --> will be set to the specified number\r\n# > number of physical cores      --> will be set to the actual number of physical cores\r\ndefault_vcpus = 1\r\n\r\n# Default maximum number of vCPUs per SB/VM:\r\n# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number\r\n#                                     of vCPUs supported by KVM if that number is exceeded\r\n# > 0 <= number of physical cores --> will be set to the specified number\r\n# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number\r\n#                                     of vCPUs supported by KVM if that number is exceeded\r\n# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when\r\n# the actual number of physical cores is greater than it.\r\n# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU\r\n# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs\r\n# can be added to a SB/VM, but the memory footprint will be big. Another example, with\r\n# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of\r\n# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,\r\n# unless you know what are you doing.\r\ndefault_maxvcpus = 0\r\n\r\n# Bridges can be used to hot plug devices.\r\n# Limitations:\r\n# * Currently only pci bridges are supported\r\n# * Until 30 devices per bridge can be hot plugged.\r\n# * Until 5 PCI bridges can be cold plugged per VM.\r\n#   This limitation could be a bug in qemu or in the kernel\r\n# Default number of bridges per SB/VM:\r\n# unspecified or 0   --> will be set to 1\r\n# > 1 <= 5           --> will be set to the specified number\r\n# > 5                --> will be set to 5\r\ndefault_bridges = 1\r\n\r\n# Default memory size in MiB for SB/VM.\r\n# If unspecified then it will be set 2048 MiB.\r\n#default_memory = 2048\r\n\r\n# Disable block device from being used for a container's rootfs.\r\n# In case of a storage driver like devicemapper where a container's \r\n# root file system is backed by a block device, the block device is passed\r\n# directly to the hypervisor for performance reasons. \r\n# This flag prevents the block device from being passed to the hypervisor, \r\n# 9pfs is used instead to pass the rootfs.\r\ndisable_block_device_use = false\r\n\r\n# Block storage driver to be used for the hypervisor in case the container\r\n# rootfs is backed by a block device. This is either virtio-scsi or \r\n# virtio-blk.\r\nblock_device_driver = \"virtio-scsi\"\r\n\r\n# Enable iothreads (data-plane) to be used. This causes IO to be\r\n# handled in a separate IO thread. This is currently only implemented\r\n# for SCSI.\r\n#\r\nenable_iothreads = false\r\n\r\n# Enable pre allocation of VM RAM, default false\r\n# Enabling this will result in lower container density\r\n# as all of the memory will be allocated and locked\r\n# This is useful when you want to reserve all the memory\r\n# upfront or in the cases where you want memory latencies\r\n# to be very predictable\r\n# Default false\r\n#enable_mem_prealloc = true\r\n\r\n# Enable huge pages for VM RAM, default false\r\n# Enabling this will result in the VM memory\r\n# being allocated using huge pages.\r\n# This is useful when you want to use vhost-user network\r\n# stacks within the container. This will automatically \r\n# result in memory pre allocation\r\n#enable_hugepages = true\r\n\r\n# Enable swap of vm memory. Default false.\r\n# The behaviour is undefined if mem_prealloc is also set to true\r\n#enable_swap = true\r\n\r\n# This option changes the default hypervisor and kernel parameters\r\n# to enable debug output where available. This extra output is added\r\n# to the proxy logs, but only when proxy debug is also enabled.\r\n# \r\n# Default false\r\n#enable_debug = true\r\n\r\n# Disable the customizations done in the runtime when it detects\r\n# that it is running on top a VMM. This will result in the runtime\r\n# behaving as it would when running on bare metal.\r\n# \r\n#disable_nesting_checks = true\r\n\r\n# This is the msize used for 9p shares. It is the number of bytes \r\n# used for 9p packet payload.\r\n#msize_9p = 8192\r\n\r\n# If true and vsocks are supported, use vsocks to communicate directly\r\n# with the agent and no proxy is started, otherwise use unix\r\n# sockets and start a proxy to communicate with the agent.\r\n# Default false\r\n#use_vsock = true\r\n\r\n# VFIO devices are hotplugged on a bridge by default. \r\n# Enable hotplugging on root bus. This may be required for devices with\r\n# a large PCI bar, as this is a current limitation with hotplugging on \r\n# a bridge. This value is valid for \"pc\" machine type.\r\n# Default false\r\n#hotplug_vfio_on_root_bus = true\r\n\r\n[factory]\r\n# VM templating support. Once enabled, new VMs are created from template\r\n# using vm cloning. They will share the same initial kernel, initramfs and\r\n# agent memory by mapping it readonly. It helps speeding up new container\r\n# creation and saves a lot of memory if there are many kata containers running\r\n# on the same host.\r\n#\r\n# When disabled, new VMs are created from scratch.\r\n#\r\n# Default false\r\n#enable_template = true\r\n\r\n[proxy.kata]\r\npath = \"/usr/libexec/kata-containers/kata-proxy\"\r\n\r\n# If enabled, proxy messages will be sent to the system log\r\n# (default: disabled)\r\n#enable_debug = true\r\n\r\n[shim.kata]\r\npath = \"/usr/libexec/kata-containers/kata-shim\"\r\n\r\n# If enabled, shim messages will be sent to the system log\r\n# (default: disabled)\r\n#enable_debug = true\r\n\r\n[agent.kata]\r\n# There is no field for this section. The goal is only to be able to\r\n# specify which type of agent the user wants to use.\r\n\r\n[runtime]\r\n# If enabled, the runtime will log additional debug messages to the\r\n# system log\r\n# (default: disabled)\r\n#enable_debug = true\r\n#\r\n# Internetworking model\r\n# Determines how the VM should be connected to the\r\n# the container network interface\r\n# Options:\r\n#\r\n#   - bridged\r\n#     Uses a linux bridge to interconnect the container interface to\r\n#     the VM. Works for most cases except macvlan and ipvlan.\r\n#\r\n#   - macvtap\r\n#     Used when the Container network interface can be bridged using\r\n#     macvtap.\r\ninternetworking_model=\"macvtap\"\r\n\r\n# If enabled, the runtime will create opentracing.io traces and spans.\r\n# (See https://www.jaegertracing.io/docs/getting-started).\r\n# (default: disabled)\r\n#enable_tracing = true\r\n```\r\n\r\n---\r\n\r\n\r\n# Image details\r\n\r\nNo image\r\n\r\n---\r\n\r\n\r\n# Initrd details\r\n\r\nNo initrd\r\n\r\n---\r\n\r\n\r\n# Logfiles\r\n\r\n\r\n## Runtime logs\r\n\r\nRecent runtime problems found in system journal:\r\n```\r\ntime=\"2018-09-05T14:56:28.027073996-04:00\" level=error msg=\"#\\t0x80039809\\truntime.allocm+0x1a9\\t\\t\\t/usr/local/go/src/runtime/proc.go:1516\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027073996-04:00\" level=error msg=\"#\\t0x80039809\\truntime.allocm+0x1a9\\t\\t\\t/usr/local/go/src/runtime/proc.go:1516\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027073996-04:00\" level=error msg=\"#\\t0x80039809\\truntime.allocm+0x1a9\\t\\t\\t/usr/local/go/src/runtime/proc.go:1516\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027073996-04:00\" level=error msg=\"#\\t0x80039809\\truntime.allocm+0x1a9\\t\\t\\t/usr/local/go/src/runtime/proc.go:1516\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027135241-04:00\" level=error msg=\"#\\t0x8003a0bf\\truntime.newm+0x2f\\t\\t\\t/usr/local/go/src/runtime/proc.go:1830\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027135241-04:00\" level=error msg=\"#\\t0x8003a0bf\\truntime.newm+0x2f\\t\\t\\t/usr/local/go/src/runtime/proc.go:1830\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027135241-04:00\" level=error msg=\"#\\t0x8003a0bf\\truntime.newm+0x2f\\t\\t\\t/usr/local/go/src/runtime/proc.go:1830\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027135241-04:00\" level=error msg=\"#\\t0x8003a0bf\\truntime.newm+0x2f\\t\\t\\t/usr/local/go/src/runtime/proc.go:1830\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027135241-04:00\" level=error msg=\"#\\t0x8003a0bf\\truntime.newm+0x2f\\t\\t\\t/usr/local/go/src/runtime/proc.go:1830\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027135241-04:00\" level=error msg=\"#\\t0x8003a0bf\\truntime.newm+0x2f\\t\\t\\t/usr/local/go/src/runtime/proc.go:1830\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027135241-04:00\" level=error msg=\"#\\t0x8003a0bf\\truntime.newm+0x2f\\t\\t\\t/usr/local/go/src/runtime/proc.go:1830\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027201984-04:00\" level=error msg=\"#\\t0x8003a3e9\\truntime.startTemplateThread+0x59\\t/usr/local/go/src/runtime/proc.go:1891\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027201984-04:00\" level=error msg=\"#\\t0x8003a3e9\\truntime.startTemplateThread+0x59\\t/usr/local/go/src/runtime/proc.go:1891\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027201984-04:00\" level=error msg=\"#\\t0x8003a3e9\\truntime.startTemplateThread+0x59\\t/usr/local/go/src/runtime/proc.go:1891\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027201984-04:00\" level=error msg=\"#\\t0x8003a3e9\\truntime.startTemplateThread+0x59\\t/usr/local/go/src/runtime/proc.go:1891\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027201984-04:00\" level=error msg=\"#\\t0x8003a3e9\\truntime.startTemplateThread+0x59\\t/usr/local/go/src/runtime/proc.go:1891\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027201984-04:00\" level=error msg=\"#\\t0x8003a3e9\\truntime.startTemplateThread+0x59\\t/usr/local/go/src/runtime/proc.go:1891\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027201984-04:00\" level=error msg=\"#\\t0x8003a3e9\\truntime.startTemplateThread+0x59\\t/usr/local/go/src/runtime/proc.go:1891\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.0272788-04:00\" level=error msg=\"#\\t0x800354fb\\truntime.main+0x1bb\\t\\t\\t/usr/local/go/src/runtime/proc.go:181\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.0272788-04:00\" level=error msg=\"#\\t0x800354fb\\truntime.main+0x1bb\\t\\t\\t/usr/local/go/src/runtime/proc.go:181\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.0272788-04:00\" level=error msg=\"#\\t0x800354fb\\truntime.main+0x1bb\\t\\t\\t/usr/local/go/src/runtime/proc.go:181\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.0272788-04:00\" level=error msg=\"#\\t0x800354fb\\truntime.main+0x1bb\\t\\t\\t/usr/local/go/src/runtime/proc.go:181\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.0272788-04:00\" level=error msg=\"#\\t0x800354fb\\truntime.main+0x1bb\\t\\t\\t/usr/local/go/src/runtime/proc.go:181\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.0272788-04:00\" level=error msg=\"#\\t0x800354fb\\truntime.main+0x1bb\\t\\t\\t/usr/local/go/src/runtime/proc.go:181\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.0272788-04:00\" level=error msg=\"#\\t0x800354fb\\truntime.main+0x1bb\\t\\t\\t/usr/local/go/src/runtime/proc.go:181\" arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027363046-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027363046-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027363046-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027363046-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027363046-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027363046-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027363046-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027426045-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027426045-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027426045-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027426045-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027426045-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027426045-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.027426045-04:00\" level=error arch=s390x container=1 name=kata-runtime pid=41263 sandbox=1 source=runtime\r\ntime=\"2018-09-05T14:56:28.030204632-04:00\" level=error msg=\"Annotations[com.github.containers.virtcontainers.pkg.oci.container_type] not found, cannot determine the container type\" arch=s390x container= name=kata-runtime pid=41263 sandbox=99999999-9999-9999-99999999999999999 source=virtcontainers subsystem=oci\r\ntime=\"2018-09-05T14:56:28.030204632-04:00\" level=error msg=\"Annotations[com.github.containers.virtcontainers.pkg.oci.container_type] not found, cannot determine the container type\" arch=s390x container= name=kata-runtime pid=41263 sandbox=99999999-9999-9999-99999999999999999 source=virtcontainers subsystem=oci\r\ntime=\"2018-09-05T14:56:28.030204632-04:00\" level=error msg=\"Annotations[com.github.containers.virtcontainers.pkg.oci.container_type] not found, cannot determine the container type\" arch=s390x container= name=kata-runtime pid=41263 sandbox=99999999-9999-9999-99999999999999999 source=virtcontainers subsystem=oci\r\ntime=\"2018-09-05T14:56:28.030204632-04:00\" level=error msg=\"Annotations[com.github.containers.virtcontainers.pkg.oci.container_type] not found, cannot determine the container type\" arch=s390x container= name=kata-runtime pid=41263 sandbox=99999999-9999-9999-99999999999999999 source=virtcontainers subsystem=oci\r\ntime=\"2018-09-05T14:56:28.030204632-04:00\" level=error msg=\"Annotations[com.github.containers.virtcontainers.pkg.oci.container_type] not found, cannot determine the container type\" arch=s390x container= name=kata-runtime pid=41263 sandbox=99999999-9999-9999-99999999999999999 source=virtcontainers subsystem=oci\r\ntime=\"2018-09-05T14:56:28.030204632-04:00\" level=error msg=\"Annotations[com.github.containers.virtcontainers.pkg.oci.container_type] not found, cannot determine the container type\" arch=s390x container= name=kata-runtime pid=41263 sandbox=99999999-9999-9999-99999999999999999 source=virtcontainers subsystem=oci\r\ntime=\"2018-09-05T14:56:28.030204632-04:00\" level=error msg=\"Annotations[com.github.containers.virtcontainers.pkg.oci.container_type] not found, cannot determine the container type\" arch=s390x container= name=kata-runtime pid=41263 sandbox=99999999-9999-9999-99999999999999999 source=virtcontainers subsystem=oci\r\ntime=\"2018-09-05T14:57:05.008232156-04:00\" level=error msg=\"/usr/share/defaults/kata-containers/configuration.toml: EvalSymlinks: too many links\" arch=s390x command=create name=kata-runtime pid=41954 source=runtime\r\ntime=\"2018-09-05T15:15:41.392836552-04:00\" level=error msg=\"/usr/share/defaults/kata-containers/configuration.toml: EvalSymlinks: too many links\" arch=s390x command=create name=kata-runtime pid=6207 source=runtime\r\ntime=\"2018-09-05T15:16:11.280441792-04:00\" level=error msg=\"/usr/share/defaults/kata-containers/configuration.toml: EvalSymlinks: too many links\" arch=s390x command=create name=kata-runtime pid=6266 source=runtime\r\ntime=\"2018-09-05T15:16:15.286036822-04:00\" level=error msg=\"/usr/share/defaults/kata-containers/configuration.toml: EvalSymlinks: too many links\" arch=s390x command=create name=kata-runtime pid=6319 source=runtime\r\n```\r\n\r\n## Proxy logs\r\n\r\nRecent proxy problems found in system journal:\r\n```\r\ntime=\"2018-09-05T15:21:04.304446926-04:00\" level=fatal msg=\"channel error\" error=\"session shutdown\" name=kata-proxy pid=6845 sandbox=test source=proxy\r\ntime=\"2018-09-05T15:21:04.306412868-04:00\" level=fatal msg=\"failed to dial channel\" channel=/run/vc/vm/test/kata.sock error=\"dial unix /run/vc/vm/test/kata.sock: connect: no such file or directory\" name=kata-proxy pid=6927 sandbox=test source=proxy\r\ntime=\"2018-09-05T15:29:20.104046063-04:00\" level=fatal msg=\"failed to dial channel\" channel=/run/vc/vm/test/kata.sock error=\"dial unix /run/vc/vm/test/kata.sock: connect: no such file or directory\" name=kata-proxy pid=7711 sandbox=test source=proxy\r\n```\r\n\r\n## Shim logs\r\n\r\nNo recent shim problems found in system journal.\r\n\r\n---\r\n\r\n\r\n# Container manager details\r\n\r\nHave `docker`\r\n\r\n## Docker\r\n\r\nOutput of \"`docker version`\":\r\n```\r\nClient:\r\n Version:           18.06.1-ce\r\n API version:       1.38\r\n Go version:        go1.10.3\r\n Git commit:        e68fc7a\r\n Built:             Tue Aug 21 17:24:34 2018\r\n OS/Arch:           linux/s390x\r\n Experimental:      false\r\n\r\nServer:\r\n Engine:\r\n  Version:          18.06.1-ce\r\n  API version:      1.38 (minimum version 1.12)\r\n  Go version:       go1.10.3\r\n  Git commit:       e68fc7a\r\n  Built:            Tue Aug 21 17:23:34 2018\r\n  OS/Arch:          linux/s390x\r\n  Experimental:     false\r\n```\r\nOutput of \"`docker info`\":\r\n```\r\nContainers: 3\r\n Running: 0\r\n Paused: 0\r\n Stopped: 3\r\nImages: 18\r\nServer Version: 18.06.1-ce\r\nStorage Driver: overlay2\r\n Backing Filesystem: extfs\r\n Supports d_type: true\r\n Native Overlay Diff: true\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local\r\n Network: bridge host macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog\r\nSwarm: inactive\r\nRuntimes: runc kata-runtime\r\nDefault Runtime: runc\r\nInit Binary: docker-init\r\ncontainerd version: 468a545b9edcd5932818eb9de8e72413e616e86e\r\nrunc version: 69663f0bd4b60df09991c08812a60108003fa340\r\ninit version: fec3683\r\nSecurity Options:\r\n apparmor\r\n seccomp\r\n  Profile: default\r\nKernel Version: 4.15.0-33-generic\r\nOperating System: Ubuntu 16.04.5 LTS\r\nOSType: linux\r\nArchitecture: s390x\r\nCPUs: 4\r\nTotal Memory: 15.51GiB\r\nName: lxkarta1\r\nID: MJF7:7SBK:MKRD:N4UV:2EMB:OYOR:HSTJ:GGUD:H7SS:GSAB:36K5:PNLY\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): true\r\n File Descriptors: 23\r\n Goroutines: 44\r\n System Time: 2018-09-05T15:29:55.668590841-04:00\r\n EventsListeners: 0\r\nRegistry: https://index.docker.io/v1/\r\nLabels:\r\nExperimental: false\r\nInsecure Registries:\r\n 127.0.0.0/8\r\nLive Restore Enabled: false\r\n\r\nWARNING: No swap limit support\r\n```\r\nOutput of \"`systemctl show docker`\":\r\n```\r\nType=simple\r\nRestart=on-failure\r\nNotifyAccess=none\r\nRestartUSec=100ms\r\nTimeoutStartUSec=infinity\r\nTimeoutStopUSec=1min 30s\r\nRuntimeMaxUSec=infinity\r\nWatchdogUSec=0\r\nWatchdogTimestamp=Thu 2018-08-30 07:19:28 EDT\r\nWatchdogTimestampMonotonic=6510414\r\nFailureAction=none\r\nPermissionsStartOnly=no\r\nRootDirectoryStartOnly=no\r\nRemainAfterExit=no\r\nGuessMainPID=yes\r\nMainPID=1314\r\nControlPID=0\r\nFileDescriptorStoreMax=0\r\nNFileDescriptorStore=0\r\nStatusErrno=0\r\nResult=success\r\nExecMainStartTimestamp=Thu 2018-08-30 07:19:28 EDT\r\nExecMainStartTimestampMonotonic=6510396\r\nExecMainExitTimestampMonotonic=0\r\nExecMainPID=1314\r\nExecMainCode=0\r\nExecMainStatus=0\r\nExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -D --default-runtime runc --add-runtime kata-runtime=/usr/local/bin/kata-runtime ; ignore_errors=no ; start_time=[Thu 2018-08-30 07:19:28 EDT] ; stop_time=[n/a] ; pid=1314 ; code=(null) ; status=0/0 }\r\nExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }\r\nSlice=system.slice\r\nControlGroup=/system.slice/docker.service\r\nMemoryCurrent=2280456192\r\nCPUUsageNSec=1377781180720\r\nTasksCurrent=45\r\nDelegate=yes\r\nCPUAccounting=no\r\nCPUShares=18446744073709551615\r\nStartupCPUShares=18446744073709551615\r\nCPUQuotaPerSecUSec=infinity\r\nBlockIOAccounting=no\r\nBlockIOWeight=18446744073709551615\r\nStartupBlockIOWeight=18446744073709551615\r\nMemoryAccounting=no\r\nMemoryLimit=18446744073709551615\r\nDevicePolicy=auto\r\nTasksAccounting=no\r\nTasksMax=18446744073709551615\r\nUMask=0022\r\nLimitCPU=18446744073709551615\r\nLimitCPUSoft=18446744073709551615\r\nLimitFSIZE=18446744073709551615\r\nLimitFSIZESoft=18446744073709551615\r\nLimitDATA=18446744073709551615\r\nLimitDATASoft=18446744073709551615\r\nLimitSTACK=18446744073709551615\r\nLimitSTACKSoft=8388608\r\nLimitCORE=18446744073709551615\r\nLimitCORESoft=18446744073709551615\r\nLimitRSS=18446744073709551615\r\nLimitRSSSoft=18446744073709551615\r\nLimitNOFILE=1048576\r\nLimitNOFILESoft=1048576\r\nLimitAS=18446744073709551615\r\nLimitASSoft=18446744073709551615\r\nLimitNPROC=18446744073709551615\r\nLimitNPROCSoft=18446744073709551615\r\nLimitMEMLOCK=65536\r\nLimitMEMLOCKSoft=65536\r\nLimitLOCKS=18446744073709551615\r\nLimitLOCKSSoft=18446744073709551615\r\nLimitSIGPENDING=63445\r\nLimitSIGPENDINGSoft=63445\r\nLimitMSGQUEUE=819200\r\nLimitMSGQUEUESoft=819200\r\nLimitNICE=0\r\nLimitNICESoft=0\r\nLimitRTPRIO=0\r\nLimitRTPRIOSoft=0\r\nLimitRTTIME=18446744073709551615\r\nLimitRTTIMESoft=18446744073709551615\r\nOOMScoreAdjust=0\r\nNice=0\r\nIOScheduling=0\r\nCPUSchedulingPolicy=0\r\nCPUSchedulingPriority=0\r\nTimerSlackNSec=50000\r\nCPUSchedulingResetOnFork=no\r\nNonBlocking=no\r\nStandardInput=null\r\nStandardOutput=journal\r\nStandardError=inherit\r\nTTYReset=no\r\nTTYVHangup=no\r\nTTYVTDisallocate=no\r\nSyslogPriority=30\r\nSyslogLevelPrefix=yes\r\nSyslogLevel=6\r\nSyslogFacility=3\r\nSecureBits=0\r\nCapabilityBoundingSet=18446744073709551615\r\nAmbientCapabilities=0\r\nMountFlags=0\r\nPrivateTmp=no\r\nPrivateNetwork=no\r\nPrivateDevices=no\r\nProtectHome=no\r\nProtectSystem=no\r\nSameProcessGroup=no\r\nUtmpMode=init\r\nIgnoreSIGPIPE=yes\r\nNoNewPrivileges=no\r\nSystemCallErrorNumber=0\r\nRuntimeDirectoryMode=0755\r\nKillMode=process\r\nKillSignal=15\r\nSendSIGKILL=yes\r\nSendSIGHUP=no\r\nId=docker.service\r\nNames=docker.service\r\nRequires=docker.socket system.slice sysinit.target\r\nWants=network-online.target\r\nWantedBy=multi-user.target\r\nConsistsOf=docker.socket\r\nConflicts=shutdown.target\r\nBefore=shutdown.target multi-user.target\r\nAfter=network-online.target firewalld.service sysinit.target basic.target system.slice docker.socket systemd-journald.socket\r\nTriggeredBy=docker.socket\r\nDocumentation=https://docs.docker.com\r\nDescription=Docker Application Container Engine\r\nLoadState=loaded\r\nActiveState=active\r\nSubState=running\r\nFragmentPath=/lib/systemd/system/docker.service\r\nDropInPaths=/etc/systemd/system/docker.service.d/kata-containers.conf\r\nUnitFileState=enabled\r\nUnitFilePreset=enabled\r\nStateChangeTimestamp=Thu 2018-08-30 07:19:28 EDT\r\nStateChangeTimestampMonotonic=6510414\r\nInactiveExitTimestamp=Thu 2018-08-30 07:19:28 EDT\r\nInactiveExitTimestampMonotonic=6510414\r\nActiveEnterTimestamp=Thu 2018-08-30 07:19:28 EDT\r\nActiveEnterTimestampMonotonic=6510414\r\nActiveExitTimestampMonotonic=0\r\nInactiveEnterTimestampMonotonic=0\r\nCanStart=yes\r\nCanStop=yes\r\nCanReload=yes\r\nCanIsolate=no\r\nStopWhenUnneeded=no\r\nRefuseManualStart=no\r\nRefuseManualStop=no\r\nAllowIsolate=no\r\nDefaultDependencies=yes\r\nOnFailureJobMode=replace\r\nIgnoreOnIsolate=no\r\nNeedDaemonReload=no\r\nJobTimeoutUSec=infinity\r\nJobTimeoutAction=none\r\nConditionResult=yes\r\nAssertResult=yes\r\nConditionTimestamp=Thu 2018-08-30 07:19:28 EDT\r\nConditionTimestampMonotonic=6509668\r\nAssertTimestamp=Thu 2018-08-30 07:19:28 EDT\r\nAssertTimestampMonotonic=6509668\r\nTransient=no\r\nStartLimitInterval=60000000\r\nStartLimitBurst=3\r\nStartLimitAction=none\r\n```\r\nNo `kubectl`\r\n\r\n---\r\n\r\n\r\n# Packages\r\n\r\nHave `dpkg`\r\nOutput of \"`dpkg -l|egrep \"(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-containers-image|linux-container|qemu-)\"`\":\r\n```\r\nii  ipxe-qemu-256k-compat-efi-roms      1.0.0+git-20150424.a25a16d-0ubuntu2~cloud0  all          PXE boot firmware - Compat EFI ROM images for qemu\r\nii  qemu-block-extra:s390x              1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        extra block backend modules for qemu-system and qemu-utils\r\nii  qemu-efi                            0~20160408.ffea0a2c-2                       all          UEFI firmware for virtual machines\r\nii  qemu-kvm                            1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU Full virtualization on x86 hardware\r\nii  qemu-launcher                       1.7.4-1ubuntu2                              all          GTK+ front-end to QEMU computer emulator\r\nii  qemu-slof                           20151103+dfsg-1ubuntu1.1                    all          Slimline Open Firmware -- QEMU PowerPC version\r\nii  qemu-system                         1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU full system emulation binaries\r\nii  qemu-system-arm                     1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU full system emulation binaries (arm)\r\nii  qemu-system-common                  1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU full system emulation binaries (common files)\r\nii  qemu-system-mips                    1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU full system emulation binaries (mips)\r\nii  qemu-system-misc                    1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU full system emulation binaries (miscellaneous)\r\nii  qemu-system-ppc                     1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU full system emulation binaries (ppc)\r\nii  qemu-system-s390x                   1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU full system emulation binaries (s390x)\r\nii  qemu-system-sparc                   1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU full system emulation binaries (sparc)\r\nii  qemu-system-x86                     1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU full system emulation binaries (x86)\r\nii  qemu-user                           1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU user mode emulation binaries\r\nii  qemu-user-binfmt                    1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU user mode binfmt registration for qemu-user\r\nii  qemu-utils                          1:2.11+dfsg-1ubuntu7.4~cloud0               s390x        QEMU utilities\r\n```\r\nNo `rpm`\r\n\r\n---\r\n"},"comment":{"url":"https://api.github.com/repos/kata-containers/runtime/issues/comments/423909994","html_url":"https://github.com/kata-containers/runtime/issues/702#issuecomment-423909994","issue_url":"https://api.github.com/repos/kata-containers/runtime/issues/702","id":423909994,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzkwOTk5NA==","user":{"login":"jodh-intel","id":20315601,"node_id":"MDQ6VXNlcjIwMzE1NjAx","avatar_url":"https://avatars1.githubusercontent.com/u/20315601?v=4","gravatar_id":"","url":"https://api.github.com/users/jodh-intel","html_url":"https://github.com/jodh-intel","followers_url":"https://api.github.com/users/jodh-intel/followers","following_url":"https://api.github.com/users/jodh-intel/following{/other_user}","gists_url":"https://api.github.com/users/jodh-intel/gists{/gist_id}","starred_url":"https://api.github.com/users/jodh-intel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jodh-intel/subscriptions","organizations_url":"https://api.github.com/users/jodh-intel/orgs","repos_url":"https://api.github.com/users/jodh-intel/repos","events_url":"https://api.github.com/users/jodh-intel/events{/privacy}","received_events_url":"https://api.github.com/users/jodh-intel/received_events","type":"User","site_admin":false},"created_at":"2018-09-24T08:47:16Z","updated_at":"2018-09-24T08:47:16Z","author_association":"CONTRIBUTOR","body":"Hi @fredbcode - thanks for posting this. Please could you attach the full proxy log:\r\n\r\n```\r\n$ sudo journalctl -q -o cat -a -t kata-proxy > ./proxy.log && gzip -9 ./proxy.log\r\n```"}}
{"actor":{"display_login":"jodh-intel"}}