{"action":"created","issue":{"url":"https://api.github.com/repos/lxc/lxd/issues/5197","repository_url":"https://api.github.com/repos/lxc/lxd","labels_url":"https://api.github.com/repos/lxc/lxd/issues/5197/labels{/name}","comments_url":"https://api.github.com/repos/lxc/lxd/issues/5197/comments","events_url":"https://api.github.com/repos/lxc/lxd/issues/5197/events","html_url":"https://github.com/lxc/lxd/issues/5197","id":372807224,"node_id":"MDU6SXNzdWUzNzI4MDcyMjQ=","number":5197,"title":"Help on server getting stuck","user":{"login":"laralar","id":3006419,"node_id":"MDQ6VXNlcjMwMDY0MTk=","avatar_url":"https://avatars2.githubusercontent.com/u/3006419?v=4","gravatar_id":"","url":"https://api.github.com/users/laralar","html_url":"https://github.com/laralar","followers_url":"https://api.github.com/users/laralar/followers","following_url":"https://api.github.com/users/laralar/following{/other_user}","gists_url":"https://api.github.com/users/laralar/gists{/gist_id}","starred_url":"https://api.github.com/users/laralar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laralar/subscriptions","organizations_url":"https://api.github.com/users/laralar/orgs","repos_url":"https://api.github.com/users/laralar/repos","events_url":"https://api.github.com/users/laralar/events{/privacy}","received_events_url":"https://api.github.com/users/laralar/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-10-23T04:51:45Z","updated_at":"2018-10-23T16:03:27Z","closed_at":"2018-10-23T15:41:39Z","author_association":"NONE","body":"<!--\r\nGithub issues are used for bug reports. For support questions, please use [our forum](https://discuss.linuxcontainers.org).\r\n\r\nPlease fill the template below as it will greatly help us track down your issue and reproduce it on our side.  \r\nFeel free to remove anything which doesn't apply to you and add more information where it makes sense.\r\n-->\r\n\r\n# Required information\r\n\r\n * Distribution: ubuntu\r\n * Distribution version: 18.04.1\r\n * The output of \"lxc info\" or if that fails:\r\n```\r\nroot@node33:~# lxc info\r\nconfig:\r\n  core.https_address: '[::]:8443'\r\n  core.trust_password: true\r\n  maas.api.key: rVFh7j9tUQ5x8aZ8NK:J9fETgjvLG5behn9br:D8vxhRFbGRx6zw4xtHMrXA77JL3gHxH9\r\n  maas.api.url: http://10.3.4.10:5240/MAAS\r\napi_extensions:\r\n- storage_zfs_remove_snapshots\r\n- container_host_shutdown_timeout\r\n- container_stop_priority\r\n- container_syscall_filtering\r\n- auth_pki\r\n- container_last_used_at\r\n- etag\r\n- patch\r\n- usb_devices\r\n- https_allowed_credentials\r\n- image_compression_algorithm\r\n- directory_manipulation\r\n- container_cpu_time\r\n- storage_zfs_use_refquota\r\n- storage_lvm_mount_options\r\n- network\r\n- profile_usedby\r\n- container_push\r\n- container_exec_recording\r\n- certificate_update\r\n- container_exec_signal_handling\r\n- gpu_devices\r\n- container_image_properties\r\n- migration_progress\r\n- id_map\r\n- network_firewall_filtering\r\n- network_routes\r\n- storage\r\n- file_delete\r\n- file_append\r\n- network_dhcp_expiry\r\n- storage_lvm_vg_rename\r\n- storage_lvm_thinpool_rename\r\n- network_vlan\r\n- image_create_aliases\r\n- container_stateless_copy\r\n- container_only_migration\r\n- storage_zfs_clone_copy\r\n- unix_device_rename\r\n- storage_lvm_use_thinpool\r\n- storage_rsync_bwlimit\r\n- network_vxlan_interface\r\n- storage_btrfs_mount_options\r\n- entity_description\r\n- image_force_refresh\r\n- storage_lvm_lv_resizing\r\n- id_map_base\r\n- file_symlinks\r\n- container_push_target\r\n- network_vlan_physical\r\n- storage_images_delete\r\n- container_edit_metadata\r\n- container_snapshot_stateful_migration\r\n- storage_driver_ceph\r\n- storage_ceph_user_name\r\n- resource_limits\r\n- storage_volatile_initial_source\r\n- storage_ceph_force_osd_reuse\r\n- storage_block_filesystem_btrfs\r\n- resources\r\n- kernel_limits\r\n- storage_api_volume_rename\r\n- macaroon_authentication\r\n- network_sriov\r\n- console\r\n- restrict_devlxd\r\n- migration_pre_copy\r\n- infiniband\r\n- maas_network\r\n- devlxd_events\r\n- proxy\r\n- network_dhcp_gateway\r\n- file_get_symlink\r\n- network_leases\r\n- unix_device_hotplug\r\n- storage_api_local_volume_handling\r\n- operation_description\r\n- clustering\r\n- event_lifecycle\r\n- storage_api_remote_volume_handling\r\n- nvidia_runtime\r\n- container_mount_propagation\r\n- container_backup\r\n- devlxd_images\r\n- container_local_cross_pool_handling\r\n- proxy_unix\r\n- proxy_udp\r\n- clustering_join\r\n- proxy_tcp_udp_multi_port_handling\r\n- network_state\r\n- proxy_unix_dac_properties\r\n- container_protection_delete\r\n- unix_priv_drop\r\n- pprof_http\r\n- proxy_haproxy_protocol\r\n- network_hwaddr\r\n- proxy_nat\r\n- network_nat_order\r\n- container_full\r\n- candid_authentication\r\n- backup_compression\r\n- candid_config\r\n- nvidia_runtime_config\r\n- storage_api_volume_snapshots\r\n- storage_unmapped\r\n- projects\r\n- candid_config_key\r\napi_status: stable\r\napi_version: \"1.0\"\r\nauth: trusted\r\npublic: false\r\nauth_methods:\r\n- tls\r\nenvironment:\r\n  addresses:\r\n  - 10.3.4.33:8443\r\n  - 10.3.5.233:8443\r\n  - 192.168.122.1:8443\r\n  architectures:\r\n  - x86_64\r\n  - i686\r\n  certificate: |\r\n    -----BEGIN CERTIFICATE-----\r\n    MIIFRzCCAy+gAwIBAgIRALyIp8eIXIP4p1Oc/AOIV9kwDQYJKoZIhvcNAQELBQAw\r\n    NDEcMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEUMBIGA1UEAwwLcm9vdEBu\r\n    b2RlMzMwHhcNMTgwNjEzMDQxMDQ0WhcNMjgwNjEwMDQxMDQ0WjA0MRwwGgYDVQQK\r\n    ExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QG5vZGUzMzCCAiIw\r\n    DQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANJRN2ESwgLOUMyMKJWsL62b1LtE\r\n    l+xczFN+RBQ2ctk0a08PXWbdbOSGfV9EgxyVlf/0MRl3Rrey/xvtc4KQix++n7b+\r\n    g6yYo6HNq4WV2S+BmNtpe293UKTqzizb0Ie9ynalnACsm027hXDsNjaRzbGLR+z9\r\n    gxqFj4tR9gLf3/NTSdXrFRpHTdpciIHIdwDgphkEgSVImX1rJvaaI9ODtWjAWxSq\r\n    UYslnLKW7nXXM5ep/Tf/VzLH+4BhNnvccqCLIxes8yJbUczbxe/wLPiEK+aewB/7\r\n    J/f0NfrbDWlfjWCu9BjxoFwjSznlLxC+G4BARh/7gxJAIDaDI9Xa1SX+tPCpU14q\r\n    tQqVcb48D/IBF+b5lVojCgXBAWGEJyvRJxY5dfqG2y4WRm3fHg+6szAtve6FEAhI\r\n    FAufpxF8u/esgpcITUZO134/tfrmFeMy8l+dE+3tD4AOWs46SsjEO8gsyPtj/3a3\r\n    O5gTffzmucKKmEDHd+3Q2UE09HYhgMtW2+QL6uHFU1WOnalR1I5KFqiEqUcs3VvG\r\n    R98JdLWzd2r8Q38ROVtlllL2TI+/f5QE8YsF2pVKjd8gcTmeUuEEzYf+eMyoeJuN\r\n    DLwSaxwyujpORsRqOlcLTAZ1hxkKcSkbj03GzjC/BvilAh7uXNjdCunagb7QMI55\r\n    DL3VKLJWJ0qAkWKDAgMBAAGjVDBSMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAK\r\n    BggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdEQQWMBSCBm5vZGUzM4cECgME\r\n    IYcEwKh6ATANBgkqhkiG9w0BAQsFAAOCAgEAc3c4D6DxnUqmQdRm8H3S0gdJFU/5\r\n    JSDZiCobelXQ/YrAc9ODlbe7aI+5HZ0/U/JqNyApjXAuxafmT2T8YClopFovK5TG\r\n    Os8SpxDFEK2yYzq+XAt1Q4IXXbD2Ou1dvGgLnILmjhcCnYttXKFmAL+7yQLQA59G\r\n    usZtY6YVnLP9yZuPbb/9oKmqTn7mQcEMwHOPh+YLvMmcLlHACuZ3bRuPqZ1uqv2/\r\n    TWLXNhEjjLjZFoJBSCXU6/EoNtjHrVHcolE2ABTgexF8MiA+spLt5/2JxQuHRER1\r\n    sMPH+SpOPtMrWEQZCupr6wsJwV6CutJKJee6I9DmJvVvmR6vBVjd8QbWTcHiZPvD\r\n    Nb0h4irA0iWJ2rco7bENU8eVKhNt7mj60pTzIJz9bsUPUEkxRjVFPCzGqa4KZt1n\r\n    ohadH2PFUe7QRqz/mmT52EQ7nvG/Gd5e15cWQW1Mv7wTa8vmpO1gRLbQMz6BMP7b\r\n    kD87x7+6WepnrC/yK5tMMrogMkc+XnLtSEC7jF09rvlQlM0iQrFpXm2LkVqQEgFk\r\n    fxafvvBhD8PkwkIi0MtOohdwqYvXPC8jYWXpZguLdwUwd2SF5kgXmh/1sAXbV0aB\r\n    sDgRUry95PKDkxs3wIeycw8RQWy3AQ9gnQMnoeXKDGrd+auRJlhESyTcyqrg07o2\r\n    v+MMAXOAocNFz50=\r\n    -----END CERTIFICATE-----\r\n  certificate_fingerprint: 38779b656fd9a6945b75c68c15e02f9bcd005308c9eb1d5626f9aad4e3b1cfaa\r\n  driver: lxc\r\n  driver_version: 3.0.2\r\n  kernel: Linux\r\n  kernel_architecture: x86_64\r\n  kernel_version: 4.15.0-36-generic\r\n  server: lxd\r\n  server_pid: 4159\r\n  server_version: \"3.6\"\r\n  storage: zfs | btrfs\r\n  storage_version: 0.7.5-1ubuntu16.3 | 4.4\r\n  server_clustered: false\r\n  server_name: node33\r\n  project: default\r\nroot@node33:~#\r\n```\r\n# Issue description\r\n\r\nI have a few hosts as LXD hosts, around 40 total\r\n\r\nI have been updating them to the latest upgrades regularly, but since I upgrded them to Ubuntu 18.04, from time to time the server gets stucked, I have to login to the remote console using iLO/iDRAC and reboot the server since it doesn't respond to even keystrokes to be able to do a console login\r\n\r\nOn the console there are no messages\r\n\r\non /var/log/syslog there is no relevant message, except the last one before getting stucked and the first one after reboot\r\n\r\nFor example:\r\n```\r\n\r\nOct 22 17:17:01 node33 CRON[21497]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)\r\nOct 22 18:17:01 node33 CRON[5452]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)\r\nOct 22 18:33:39 node33 systemd[1]: Starting Daily apt download activities...\r\nOct 22 18:33:40 node33 systemd[1]: Started Daily apt download activities.\r\nOct 22 19:17:01 node33 CRON[48541]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)\r\nOct 22 19:28:34 node33 snapd[42014]: storehelpers.go:398: cannot refresh: snap has no updates available: \"core\", \"lxd\"\r\nOct 22 19:28:34 node33 snapd[42014]: autorefresh.go:387: auto-refresh: all snaps are up-to-date\r\nOct 23 09:58:22 node33 systemd-modules-load[793]: Inserted module 'iscsi_tcp'\r\nOct 23 09:58:22 node33 systemd-modules-load[793]: Inserted module 'ib_iser'\r\nOct 23 09:58:22 node33 systemd[1]: Started Create list of required static device nodes for the current kernel.\r\nOct 23 09:58:22 node33 systemd[1]: Started Load Kernel Modules.\r\nOct 23 09:58:22 node33 systemd[1]: Started LVM2 metadata daemon.\r\n```\r\n\r\nThere is nothing else installed or running on the hosts except LXD and a few containers in each hosts.  It happens randomly (so far I haven't been able to identify the cause) and if I am not mistaen it started happening when I upgraded from 16.04 to 18.04\r\n\r\nAt first I thought there was some kernel issue and there were some messages in the console regarding some clues about this.  But now there is nothing.\r\n\r\nWhat else could I try to see in which logs to try to get a clue of what is happening?  I am not very used to explore system/kernel logs.  That is why I ask for help.\r\n\r\nLXD log doesn't show anything relevant either. These are the last messages of previous log file lxd.log.1 :\r\n```\r\nlvl=info msg=\"Done updating instance types\" t=2018-10-22T01:54:31+0530\r\nlvl=info msg=\"Updating images\" t=2018-10-22T07:54:22+0530\r\nlvl=info msg=\"Done updating images\" t=2018-10-22T07:54:22+0530\r\nlvl=info msg=\"Updating images\" t=2018-10-22T13:54:22+0530\r\nlvl=info msg=\"Done updating images\" t=2018-10-22T13:54:22+0530\r\n```\r\n\r\ndmesg, if I am not incorrect, it shows only the information since booting\r\n\r\nThe isue is happenign on different hardware, I have HP ProLiant DL380p from 24GB of ram to 48GB of ram, Dell servers from 16GB of ram to 128GB of ram, blade servers with 4GB of RAM\r\n\r\nIt seems to happen more often in some servers that may be overloaded in regards to memory assigned for the running LXD continers (after all the respective containers are up there is hardly any memory left on the host.  Could it be that SWAP is finished?\r\n\r\nbut last night it happened on a server that on average has a few GB free and no major CPU consuming container is running there.  \r\n\r\nAs I mention, only LXD containers I have in the hosts, nothing else.  in some I have a few KVMs, but hag ups happen mostly on servers where there is none of these.\r\n\r\nAny help would be appreciated\r\n\r\nThanks\r\n\r\n\r\n# Steps to reproduce\r\n\r\n NOT being able so far\r\n\r\n# Information to attach\r\n\r\n - [ ] Any relevant kernel output (`dmesg`)\r\n - [ ] Container log (`lxc info NAME --show-log`)\r\n - [ ] Container configuration (`lxc config show NAME --expanded`)\r\n - [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)\r\n - [ ] Output of the client with --debug\r\n - [ ] Output of the daemon with --debug (alternatively output of `lxc monitor` while reproducing the issue)\r\n\r\nThis is the only software that I have installed on the hosts:\r\n```\r\npdsh -R ssh -w ubuntu@10.3.4.[33] sudo apt install -y zfsutils-linux iperf speedtest-cli \\\r\n     arp-scan cpu-checker qemu qemu-kvm libvirt-bin bridge-utils zram-config tree pdsh snapd ncdu ntp \\\r\n     ntpdate nfs-common criu\r\npdsh -R ssh -w ubuntu@10.3.4.[48] sudo apt purge -y lxd lxd-client\r\nnohup pdsh -R ssh -w ubuntu@10.3.4.[48] sudo snap install lxd &\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/lxc/lxd/issues/comments/432310556","html_url":"https://github.com/lxc/lxd/issues/5197#issuecomment-432310556","issue_url":"https://api.github.com/repos/lxc/lxd/issues/5197","id":432310556,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMjMxMDU1Ng==","user":{"login":"stgraber","id":2230599,"node_id":"MDQ6VXNlcjIyMzA1OTk=","avatar_url":"https://avatars0.githubusercontent.com/u/2230599?v=4","gravatar_id":"","url":"https://api.github.com/users/stgraber","html_url":"https://github.com/stgraber","followers_url":"https://api.github.com/users/stgraber/followers","following_url":"https://api.github.com/users/stgraber/following{/other_user}","gists_url":"https://api.github.com/users/stgraber/gists{/gist_id}","starred_url":"https://api.github.com/users/stgraber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stgraber/subscriptions","organizations_url":"https://api.github.com/users/stgraber/orgs","repos_url":"https://api.github.com/users/stgraber/repos","events_url":"https://api.github.com/users/stgraber/events{/privacy}","received_events_url":"https://api.github.com/users/stgraber/received_events","type":"User","site_admin":false},"created_at":"2018-10-23T16:03:27Z","updated_at":"2018-10-23T16:03:27Z","author_association":"MEMBER","body":"@laralar https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1799497"}}
{"actor":{"display_login":"stgraber"}}