{"action":"created","issue":{"url":"https://api.github.com/repos/joyent/illumos-joyent/issues/180","repository_url":"https://api.github.com/repos/joyent/illumos-joyent","labels_url":"https://api.github.com/repos/joyent/illumos-joyent/issues/180/labels{/name}","comments_url":"https://api.github.com/repos/joyent/illumos-joyent/issues/180/comments","events_url":"https://api.github.com/repos/joyent/illumos-joyent/issues/180/events","html_url":"https://github.com/joyent/illumos-joyent/issues/180","id":358143184,"node_id":"MDU6SXNzdWUzNTgxNDMxODQ=","number":180,"title":"i40e driver: sporadic issue","user":{"login":"youzhongyang","id":15167055,"node_id":"MDQ6VXNlcjE1MTY3MDU1","avatar_url":"https://avatars3.githubusercontent.com/u/15167055?v=4","gravatar_id":"","url":"https://api.github.com/users/youzhongyang","html_url":"https://github.com/youzhongyang","followers_url":"https://api.github.com/users/youzhongyang/followers","following_url":"https://api.github.com/users/youzhongyang/following{/other_user}","gists_url":"https://api.github.com/users/youzhongyang/gists{/gist_id}","starred_url":"https://api.github.com/users/youzhongyang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/youzhongyang/subscriptions","organizations_url":"https://api.github.com/users/youzhongyang/orgs","repos_url":"https://api.github.com/users/youzhongyang/repos","events_url":"https://api.github.com/users/youzhongyang/events{/privacy}","received_events_url":"https://api.github.com/users/youzhongyang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"rzezeski","id":208153,"node_id":"MDQ6VXNlcjIwODE1Mw==","avatar_url":"https://avatars0.githubusercontent.com/u/208153?v=4","gravatar_id":"","url":"https://api.github.com/users/rzezeski","html_url":"https://github.com/rzezeski","followers_url":"https://api.github.com/users/rzezeski/followers","following_url":"https://api.github.com/users/rzezeski/following{/other_user}","gists_url":"https://api.github.com/users/rzezeski/gists{/gist_id}","starred_url":"https://api.github.com/users/rzezeski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rzezeski/subscriptions","organizations_url":"https://api.github.com/users/rzezeski/orgs","repos_url":"https://api.github.com/users/rzezeski/repos","events_url":"https://api.github.com/users/rzezeski/events{/privacy}","received_events_url":"https://api.github.com/users/rzezeski/received_events","type":"User","site_admin":false},"assignees":[{"login":"rzezeski","id":208153,"node_id":"MDQ6VXNlcjIwODE1Mw==","avatar_url":"https://avatars0.githubusercontent.com/u/208153?v=4","gravatar_id":"","url":"https://api.github.com/users/rzezeski","html_url":"https://github.com/rzezeski","followers_url":"https://api.github.com/users/rzezeski/followers","following_url":"https://api.github.com/users/rzezeski/following{/other_user}","gists_url":"https://api.github.com/users/rzezeski/gists{/gist_id}","starred_url":"https://api.github.com/users/rzezeski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rzezeski/subscriptions","organizations_url":"https://api.github.com/users/rzezeski/orgs","repos_url":"https://api.github.com/users/rzezeski/repos","events_url":"https://api.github.com/users/rzezeski/events{/privacy}","received_events_url":"https://api.github.com/users/rzezeski/received_events","type":"User","site_admin":false}],"milestone":null,"comments":29,"created_at":"2018-09-07T17:02:49Z","updated_at":"2018-12-19T20:19:14Z","closed_at":null,"author_association":"NONE","body":"Recently we evaluated a new supermicro unit which comes with X710 cards:\r\n```\r\n            pci8086,7 (pciex8086,1572) [Intel Corporation Ethernet Controller X710 for 10GbE SFP+], instance #0 (driver name: i40e)\r\n            pci8086,0 (pciex8086,1572) [Intel Corporation Ethernet Controller X710 for 10GbE SFP+], instance #1 (driver name: i40e)\r\n            pci8086,7 (pciex8086,1572) [Intel Corporation Ethernet Controller X710 for 10GbE SFP+], instance #2 (driver name: i40e)\r\n            pci8086,0 (pciex8086,1572) [Intel Corporation Ethernet Controller X710 for 10GbE SFP+], instance #3 (driver name: i40e)\r\n```\r\nThere's sporadic issue when running the following perl script which connects to a mysql db and then disconnects repeatedly:\r\n```\r\n#!/usr/bin/perl -w\r\n\r\nuse DBI;\r\nuse POSIX qw(strftime);\r\nuse Time::HiRes qw(usleep);\r\n\r\nmy $dbh;\r\n\r\nwhile (1) {\r\n   eval {\r\n       $dbh = DBI->connect('DBI:mysql:dbname;host=dbhost', 'user', 'password', { RaiseError => 1 });\r\n       $dbh->disconnect();\r\n   } or print strftime(\"%H:%M:%S\", localtime) . \" => \" . $@;\r\n   usleep(500000);\r\n}\r\n```\r\nWe first thought the issue might be on the network switch side, but clearly the network switch (Juniper) report command indicated there's problem on the host side:\r\n```\r\n> show lacp statistics interfaces\r\nAggregated interface: ae5\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      et-0/0/50            6528201     6525933            0            0\r\n      et-0/0/51             261738      261627            0            0\r\n\r\nAggregated interface: ae10\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/1             6561283     6526648            0            0\r\n\r\nAggregated interface: ae11\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/2             6561066     6526597            0            0\r\n\r\nAggregated interface: ae12\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/3             6561214     6526649            0            0\r\n\r\nAggregated interface: ae13\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/4             6561388     6526649            0            0\r\n\r\nAggregated interface: ae14\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/5             6561804     6527083            0            0\r\n\r\nAggregated interface: ae16\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/0             6555587     6524326            0            0\r\n\r\nAggregated interface: ae17\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/8             6560536     6526656            0            0\r\n\r\nAggregated interface: ae18\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/9             6560478     6526657            0            0\r\n\r\nAggregated interface: ae19\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/10            6549335     6526670            0            0\r\n\r\nAggregated interface: ae20\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/11            6560332     6526543            0            0\r\n\r\nAggregated interface: ae21\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/12            6039869     6008958            0            0\r\n\r\nAggregated interface: ae22\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/13            6560535     6526668            0            0\r\n\r\nAggregated interface: ae15\r\n    LACP Statistics:       LACP Rx     LACP Tx   Unknown Rx   Illegal Rx\r\n      xe-0/0/6               76177       86909            0            0\r\n      xe-0/0/7               76177       86903            0            0\r\n```\r\n\r\nI then loaded an old SmartOS image which was built around Oct 26, 2017. Everything works flawlessly!\r\n\r\nI haven't gotten a chance to figure out which commit to i40e driver might have caused the issue.\r\n\r\nThanks for your attention.\r\n"},"comment":{"url":"https://api.github.com/repos/joyent/illumos-joyent/issues/comments/448730531","html_url":"https://github.com/joyent/illumos-joyent/issues/180#issuecomment-448730531","issue_url":"https://api.github.com/repos/joyent/illumos-joyent/issues/180","id":448730531,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0ODczMDUzMQ==","user":{"login":"goekesmi","id":786913,"node_id":"MDQ6VXNlcjc4NjkxMw==","avatar_url":"https://avatars3.githubusercontent.com/u/786913?v=4","gravatar_id":"","url":"https://api.github.com/users/goekesmi","html_url":"https://github.com/goekesmi","followers_url":"https://api.github.com/users/goekesmi/followers","following_url":"https://api.github.com/users/goekesmi/following{/other_user}","gists_url":"https://api.github.com/users/goekesmi/gists{/gist_id}","starred_url":"https://api.github.com/users/goekesmi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/goekesmi/subscriptions","organizations_url":"https://api.github.com/users/goekesmi/orgs","repos_url":"https://api.github.com/users/goekesmi/repos","events_url":"https://api.github.com/users/goekesmi/events{/privacy}","received_events_url":"https://api.github.com/users/goekesmi/received_events","type":"User","site_admin":false},"created_at":"2018-12-19T20:19:14Z","updated_at":"2018-12-19T20:19:14Z","author_association":"NONE","body":"One crash dump coming right up.\r\n```\r\n- SmartOS (build: 20181121T181952Z)\r\nYou have new mail.\r\n[root@AgentAsh (us-elns-1) ~]# mdb -k\r\nLoading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba xhci fctl stmf_sbd stmf zfs sd mm lofs idm sata i40e crypto fcp random cpc logindmux ptm kvm sppp nsmb smbsrv nfs ]\r\n> ::walk mac_impl_cache | ::printf \"0x%p %s\\n\" mac_impl_t . mi_name ! grep i40e\r\n0xfffffdf2c7c2c288 i40e1\r\n0xfffffdf2c7c2f848 i40e0\r\n> 0xfffffdf2c7c2f848::print mac_impl_t mi_driver | ::print i40e_t i40e_trqpairs | ::array i40e_trqpair_t 7 | ::printf \"%u %u %u %u (%u)\\n\" i40e_trqpair_t itrq_txstat.itxs_packets.value.ui64 itrq_txstat.itxs_descriptors.value.ui64 itrq_txstat.itxs_recycled.value.ui64 itrq_txstat.itxs_err_nodescs.value.ui64 itrq_desc_free\r\n27874 27874 27874 0 (1024)\r\n0 0 0 0 (1024)\r\n0 0 0 0 (1024)\r\n0 0 0 0 (1024)\r\n3671 7302 6285 1036 (7)\r\n981 1698 875 0 (201)\r\n1310 3945 3565 0 (644)\r\n```\r\n\r\nI'm going to just assume, @rzezeski , that you have access to the JPC Manta on nonpublic objects, or can get access to it.\r\n\r\nhttps://us-east.manta.joyent.com/goekesmi/stor/crashdumps/us-elns-1/AgentAsh.vmdump.2018-1219.i40e0\r\n\r\nEnjoy?"}}
{"actor":{"display_login":"goekesmi"}}