{"action":"created","issue":{"url":"https://api.github.com/repos/gregoryolsen/lxc-devuan/issues/1","repository_url":"https://api.github.com/repos/gregoryolsen/lxc-devuan","labels_url":"https://api.github.com/repos/gregoryolsen/lxc-devuan/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/gregoryolsen/lxc-devuan/issues/1/comments","events_url":"https://api.github.com/repos/gregoryolsen/lxc-devuan/issues/1/events","html_url":"https://github.com/gregoryolsen/lxc-devuan/issues/1","id":158845812,"number":1,"title":"DHCP not working - Transmit (Tx) checksum offload problem","user":{"login":"gregoryolsen","id":13941338,"avatar_url":"https://avatars.githubusercontent.com/u/13941338?v=3","gravatar_id":"","url":"https://api.github.com/users/gregoryolsen","html_url":"https://github.com/gregoryolsen","followers_url":"https://api.github.com/users/gregoryolsen/followers","following_url":"https://api.github.com/users/gregoryolsen/following{/other_user}","gists_url":"https://api.github.com/users/gregoryolsen/gists{/gist_id}","starred_url":"https://api.github.com/users/gregoryolsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gregoryolsen/subscriptions","organizations_url":"https://api.github.com/users/gregoryolsen/orgs","repos_url":"https://api.github.com/users/gregoryolsen/repos","events_url":"https://api.github.com/users/gregoryolsen/events{/privacy}","received_events_url":"https://api.github.com/users/gregoryolsen/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"gregoryolsen","id":13941338,"avatar_url":"https://avatars.githubusercontent.com/u/13941338?v=3","gravatar_id":"","url":"https://api.github.com/users/gregoryolsen","html_url":"https://github.com/gregoryolsen","followers_url":"https://api.github.com/users/gregoryolsen/followers","following_url":"https://api.github.com/users/gregoryolsen/following{/other_user}","gists_url":"https://api.github.com/users/gregoryolsen/gists{/gist_id}","starred_url":"https://api.github.com/users/gregoryolsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gregoryolsen/subscriptions","organizations_url":"https://api.github.com/users/gregoryolsen/orgs","repos_url":"https://api.github.com/users/gregoryolsen/repos","events_url":"https://api.github.com/users/gregoryolsen/events{/privacy}","received_events_url":"https://api.github.com/users/gregoryolsen/received_events","type":"User","site_admin":false},"milestone":null,"comments":1,"created_at":"2016-06-07T06:04:52Z","updated_at":"2016-06-07T06:07:07Z","closed_at":"2016-06-07T06:07:07Z","body":"#### Synopsis: Container can't get DHCP assigned IP address\r\n\r\nThe failed sequence looks like this:  \r\n- Client DHCPDISCOVER(lxcbr0), followed by Server DHCPOFFER(lxcbr0)\r\n- Client does **not** follow with DHCPREQUEST as required per protocol\r\n- Consequently neither will DHCPACK be sent from server to acknowledge\r\n\r\nDon't blame Devuan for this.\r\n\r\nIt's an old issue related to Linux transmit (Tx) checksum offload \r\nhandling for virtual devices such as:  \r\n- bridges, veth devices, etc.  \r\n\r\nFor a more in-depth explanation see **_Root Cause_** below.\r\n\r\n#### Solution\r\n\r\nHere's the solution I've used reliably for many years with my Xen VMs, \r\nand more recently with LXC:\r\n\r\n1. Install ethtool:\r\n\r\n    ```shell\r\n    $ sudo apt-get install ethtool\r\n    ```\r\n\r\n2. Turn off Tx checksum offloading on the LXC bridge:\r\n\r\n    There's more than one way to do this, but the cleanest way\r\n    I've found is to simply add an \"up\" command to the bridge\r\n    interface definition.\r\n\r\n    Example 'up' command ($IFACE = bridge interface):  \r\n\r\n    ```shell\r\n    up /sbin/ethtool -K $IFACE tx off  # <== TURN OFF TX CHECKSUM OFFLOAD\r\n    ```\r\n\r\n    I manually define all my bridges.\r\n\r\n    Example bridge definition in `/etc/network/interfaces`:\r\n\r\n    ```\r\n    auto lxcbr0\r\n    iface lxcbr0 inet static\r\n            pre-up    brctl addbr $IFACE\r\n            address   10.0.0.1\r\n            netmask   255.255.0.0\r\n            network   10.0.0.0\r\n            broadcast 10.0.255.255\r\n            bridge_stp off                    # disable Spanning Tree Protocol\r\n            bridge_waitport 0                 # no delay before a port becomes available\r\n            bridge_fd 0                       # no forwarding delay\r\n            up        ip link set $IFACE up\r\n            up        /sbin/ethtool -K $IFACE tx off  # <== TURN OFF TX CHECKSUM OFFLOAD\r\n            down      ip link set $IFACE down\r\n            post-down brctl delbr $IFACE\r\n    ```\r\n\r\n    *CentOS/RHEL/SUSE* example (**untested**):\r\n\r\n    Add to interface config `/etc/sysconfig/network/ifcfg-lxcbr0`:\r\n\r\n    ```shell\r\n    ETHTOOL_OPTIONS='-K iface tx off'\r\n    ```\r\n\r\nOther solutions include, setting an iptables rule on either the \r\nPOSTROUTING or OUTPUT chains of the *mangle* table. At this time \r\nI don't do this, therefore I don't have an example to provide.\r\n\r\nIMHO, the iptables rule solution might be prone to breakage as \r\nsubsequent rule changes can lose the checksum rule.\r\n\r\nHowever if maximum performance on a prod server is important, \r\nI suggest using the **OUTPUT chain** of the *mangle* table to \r\ncalculate checksums for DHCP.\r\n\r\n#### Root Cause\r\n\r\nThe root cause of the issue is a missing or incorrect checksum for packets \r\ntransmitted from virtual devices, and a *DHCLIENT* that rejects packets with \r\nbad checksums.\r\n\r\nThere's a lot of confusion about the cause.\r\n\r\nFirst and foremost, know this is **not** a problem with Devuan, \r\nand nor is it a problem with LXC.\r\n\r\nIt's an old issue related to Linux transmit (Tx) checksum *offload* handling \r\nfor virtual devices:  \r\n- bridge's, veth's, etc.  \r\n\r\nThe kernel Linux defers calculating checksums until packet *egress* on \r\n**physical devices** only.\r\n\r\nTherefore this is not a kernel bug, but a design decision.\r\n\r\nAs I understand it, it's because there's no need to calculate checksum's \r\nwhen packets transmit via in-memory copy, as is the case for virtual devices. \r\nMakes sense. Why consume CPU performing the calculation when there's no \r\nphysical network?\r\n\r\nThere's also the *partial checksum offload* feature which was introduced \r\nas a performance optimization. I don't fully understand it. Apparently it \r\nhas resulted in incorrect checksums that can cause DHCP to reject packets.\r\n\r\nDisabling Tx checksum *offload* circumvents the issue.\r\n\r\nOn the surface it may seem like there's a problem with the *DHCLIENT*, especially \r\nif other VMs/Containers work fine on the same bridge. However all this \r\nimplies is some patching has been done to accept packets with bad checksums.\r\nDepending on ones perspective this is either good, or it's bad. IMHO, both \r\nperspectives have some merit.\r\n\r\n#### Related links\r\n[Fix Debian base boxes networking issue #153](https://github.com/fgrehm/vagrant-lxc/issues/153)\r\n[Bug #1244589 VM can't get DHCP lease due packet has no checksum](https://bugs.launchpad.net/neutron/+bug/1244589)\r\n[Don't rely on setting `ethtool tx off` on guest interfaces #1255](https://github.com/weaveworks/weave/issues/1255)\r\n[DHCP not working due to bad udp checksum - TX offloading problem #40](https://github.com/projectcalico/calico/issues/40)\r\n[dhcp client not working properly in Xen domU due to partial checksum offload](https://bugzilla.novell.com/show_bug.cgi?id=668194)\r\n[Always fill UDP checksums in DHCP replies](https://review.openstack.org/#/c/148718/)\r\n[Add an iptables mangle rule per-bridge for DHCP](https://review.openstack.org/#/c/18336/4)\r\n[KVM guests networking issues with no virbr0 and with vhost_net kernel modules loaded](https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1029430)\r\n[dhcp3-server reports many bad udp checksums to syslog using virtio NIC](https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/930962)  \r\n"},"comment":{"url":"https://api.github.com/repos/gregoryolsen/lxc-devuan/issues/comments/224187994","html_url":"https://github.com/gregoryolsen/lxc-devuan/issues/1#issuecomment-224187994","issue_url":"https://api.github.com/repos/gregoryolsen/lxc-devuan/issues/1","id":224187994,"user":{"login":"gregoryolsen","id":13941338,"avatar_url":"https://avatars.githubusercontent.com/u/13941338?v=3","gravatar_id":"","url":"https://api.github.com/users/gregoryolsen","html_url":"https://github.com/gregoryolsen","followers_url":"https://api.github.com/users/gregoryolsen/followers","following_url":"https://api.github.com/users/gregoryolsen/following{/other_user}","gists_url":"https://api.github.com/users/gregoryolsen/gists{/gist_id}","starred_url":"https://api.github.com/users/gregoryolsen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gregoryolsen/subscriptions","organizations_url":"https://api.github.com/users/gregoryolsen/orgs","repos_url":"https://api.github.com/users/gregoryolsen/repos","events_url":"https://api.github.com/users/gregoryolsen/events{/privacy}","received_events_url":"https://api.github.com/users/gregoryolsen/received_events","type":"User","site_admin":false},"created_at":"2016-06-07T06:07:07Z","updated_at":"2016-06-07T06:07:07Z","body":"See Solution."}}
{"actor":{"display_login":"gregoryolsen"}}