{"action":"created","issue":{"url":"https://api.github.com/repos/cloudius-systems/osv/issues/726","repository_url":"https://api.github.com/repos/cloudius-systems/osv","labels_url":"https://api.github.com/repos/cloudius-systems/osv/issues/726/labels{/name}","comments_url":"https://api.github.com/repos/cloudius-systems/osv/issues/726/comments","events_url":"https://api.github.com/repos/cloudius-systems/osv/issues/726/events","html_url":"https://github.com/cloudius-systems/osv/issues/726","id":133220760,"number":726,"title":"tcp_do_segment abort (NFS related?)","user":{"login":"justinc1","id":587971,"avatar_url":"https://avatars.githubusercontent.com/u/587971?v=3","gravatar_id":"","url":"https://api.github.com/users/justinc1","html_url":"https://github.com/justinc1","followers_url":"https://api.github.com/users/justinc1/followers","following_url":"https://api.github.com/users/justinc1/following{/other_user}","gists_url":"https://api.github.com/users/justinc1/gists{/gist_id}","starred_url":"https://api.github.com/users/justinc1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/justinc1/subscriptions","organizations_url":"https://api.github.com/users/justinc1/orgs","repos_url":"https://api.github.com/users/justinc1/repos","events_url":"https://api.github.com/users/justinc1/events{/privacy}","received_events_url":"https://api.github.com/users/justinc1/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-02-12T11:47:08Z","updated_at":"2016-02-16T14:58:53Z","closed_at":null,"body":"Crash:\r\n```\r\nsudo ./scripts/run.py -d -V -n -v -e '--ip=eth0,192.168.122.202,255.255.255.0 --defaultgw=192.168.122.1 --nameserver=192.168.122.1 /tools/mount-nfs.so nfs://192.168.122.1/ggg/?uid=0 /fff; tst-execve.so 30 /tst-file-read.so 30' --novnc -i build/debug/usr.img -c8 1>log-723-b/issue723-$ii 2>&1;\r\n\r\nPP 7 tst-file-read.c ii=29/30\r\nPP 14 tst-file-read.c ii=29/30\r\nPP 9 fn=/fff/openfoam/etc/cellModels\r\nPP 7 fn=/fff/openfoam/etc/cellModels\r\nPP 14 fn=/fff/openfoam/etc/cellModels\r\nPP 3 tst-file-read.c main EXIT ret=0\r\nAssertion failed: tp->get_state() > 1 (bsd/sys/netinet/tcp_input.cc: tcp_do_segment: 1076)\r\n\r\n[backtrace]\r\n0x000000000022d8ea <abort(char const*, ...)+249>\r\n0x000000000022d94b <__assert_fail+58>\r\nPP 23 tst-file-read.c main EXIT ret=0\r\nPP 16 tst-file-read.c ii=29/30\r\nPP 16 fn=/fff/openfoam/etc/cellModels\r\n0x00000000002a7206 <???+2781702>\r\n0x00000000002abc3e <???+2800702>\r\n0x00000000002abd19 <???+2800921>\r\n0x00000000002ac6b7 <???+2803383>\r\n0x00000000006313f8 <std::function<void (mbuf*)>::operator()(mbuf*) const+72>\r\n0x0000000000630299 <net_channel::process_queue()+57>\r\n0x0000000000244bb1 <???+2378673>\r\n0x0000000000244ed2 <soclose+797>\r\n0x0000000000254c77 <socket_file::close()+47>\r\n0x0000000000675770 <fdrop+147>\r\n0x000000000067529d <fdclose(int)+163>\r\n0x000000000064cfae <close+35>\r\n0x000000000070afbe <rpc_destroy_context+318>\r\n0xffff8000091d903f <???+152932415>\r\n```\r\n\r\nRunning on fedora 23 host, code is master@c93ebf9d140f + :\r\n```\r\n__cxa_atexit/finalize: be thread-safe\r\nAdd test for osv_execve.\r\nAdd osv_execve, osv_waittid\r\n```\r\n\r\ngdb:\r\n```\r\n(gdb) where\r\n#0  processor::cli_hlt () at arch/x64/processor.hh:248\r\n#1  0x0000000000209c4e in arch::halt_no_interrupts () at arch/x64/arch.hh:48\r\n#2  0x000000000049d36b in osv::halt () at arch/x64/power.cc:24\r\n#3  0x000000000022d911 in abort (fmt=0xa36f50 \"Assertion failed: %s (%s: %s: %d)\\n\") at runtime.cc:132\r\n#4  0x000000000022d94c in __assert_fail (expr=0xa4997e \"tp->get_state() > 1\", \r\n    file=0xa49644 \"bsd/sys/netinet/tcp_input.cc\", line=1076, \r\n    func=0xa49da0 <tcp_do_segment(mbuf*, tcphdr*, socket*, tcpcb*, int, int, unsigned char, int, bool&)::__func__> \"tcp_do_segment\") at runtime.cc:139\r\n#5  0x00000000002a7207 in tcp_do_segment (m=0xffffa0000cfe6f00, th=0xffff80000c71e02e, so=0xffffa00009b4b600, \r\n    tp=0xffffa00002e8ec00, drop_hdrlen=66, tlen=732, iptos=0 '\\000', ti_locked=2, want_close=@0x200000dffa4e: false)\r\n    at bsd/sys/netinet/tcp_input.cc:1075\r\n#6  0x00000000002abc3f in tcp_net_channel_packet (tp=0xffffa00002e8ec00, m=0xffffa0000cfe6f00)\r\n    at bsd/sys/netinet/tcp_input.cc:3210\r\n#7  0x00000000002abd1a in <lambda(mbuf*)>::operator()(mbuf *) const (__closure=0xffff9000043d5000, \r\n    m=0xffffa0000cfe6f00) at bsd/sys/netinet/tcp_input.cc:3229\r\n#8  0x00000000002ac6b8 in std::_Function_handler<void(mbuf*), tcp_setup_net_channel(tcpcb*, ifnet*)::<lambda(mbuf*)> >::_M_invoke(const std::_Any_data &, <unknown type in /home/justin_cinkelj/devel/mikelangelo/osv/build/debug.x64/loader.elf, CU 0x77dfa7, DIE 0x7ad6c5>) (__functor=..., \r\n    __args#0=<unknown type in /home/justin_cinkelj/devel/mikelangelo/osv/build/debug.x64/loader.elf, CU 0x77dfa7, DIE 0x7ad6c5>) at /usr/include/c++/5.3.1/functional:1871\r\n#9  0x00000000006313f9 in std::function<void (mbuf*)>::operator()(mbuf*) const (this=0xffff9000043d5000, \r\n    __args#0=0xffffa0000cfe6f00) at /usr/include/c++/5.3.1/functional:2271\r\n#10 0x000000000063029a in net_channel::process_queue (this=0xffff9000043d5000) at core/net_channel.cc:37\r\n#11 0x0000000000244bb2 in flush_net_channel (so=0xffffa00009b4b600) at bsd/sys/kern/uipc_socket.cc:587\r\n#12 0x0000000000244ed3 in soclose (so=0xffffa00009b4b600) at bsd/sys/kern/uipc_socket.cc:657\r\n#13 0x0000000000254c78 in socket_file::close (this=0xffffa00009802700) at bsd/sys/kern/sys_socket.cc:383\r\n#14 0x0000000000675771 in fdrop (fp=0xffffa00009802700) at fs/vfs/kern_descrip.cc:211\r\n#15 0x000000000067529e in fdclose (fd=10) at fs/vfs/kern_descrip.cc:94\r\n#16 0x000000000064cfaf in close (fd=10) at fs/vfs/main.cc:235\r\n#17 0x000000000070afbf in rpc_destroy_context (rpc=0xffff800009046040) at init.c:284\r\n#18 0x0000000000701c13 in nfs_destroy_context (nfs=0xffffa00008fdf480) at libnfs.c:445\r\n#19 0x00000000006f6fa1 in std::unique_ptr<nfs_context, void (*)(nfs_context*)>::~unique_ptr (\r\n    this=0xffffa00008fc6128, __in_chrg=<optimized out>) at /usr/include/c++/5.3.1/bits/unique_ptr.h:236\r\n---Type <return> to continue, or q <return> to quit---\r\n#20 0x00000000006f7c1a in mount_context::~mount_context (this=0xffffa00008fc6100, __in_chrg=<optimized out>)\r\n    at fs/nfs/nfs.hh:32\r\n#21 0x00000000006f7c50 in std::default_delete<mount_context>::operator() (this=0xffffa00006e94be8, \r\n    __ptr=0xffffa00008fc6100) at /usr/include/c++/5.3.1/bits/unique_ptr.h:76\r\n#22 0x00000000006f75c3 in std::unique_ptr<mount_context, std::default_delete<mount_context> >::~unique_ptr (\r\n    this=0xffffa00006e94be8, __in_chrg=<optimized out>) at /usr/include/c++/5.3.1/bits/unique_ptr.h:236\r\n#23 0x00000000006f90e2 in std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > >::~pair (this=0xffffa00006e94bc8, \r\n    __in_chrg=<optimized out>) at /usr/include/c++/5.3.1/bits/stl_pair.h:96\r\n#24 0x00000000006f910e in __gnu_cxx::new_allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > > > (this=0x200000dffe17, __p=0xffffa00006e94bc8)\r\n    at /usr/include/c++/5.3.1/ext/new_allocator.h:124\r\n#25 0x00000000006f8f99 in std::allocator_traits<std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > > > >::_S_destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > > > (__a=..., __p=0xffffa00006e94bc8)\r\n    at /usr/include/c++/5.3.1/bits/alloc_traits.h:285\r\n#26 0x00000000006f8bce in std::allocator_traits<std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > > > (__a=..., __p=0xffffa00006e94bc8)\r\n    at /usr/include/c++/5.3.1/bits/alloc_traits.h:414\r\n#27 0x00000000006f83df in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > >, true> > >::_M_deallocate_node (this=0xffff900007563678, __n=0xffffa00006e94bc0)\r\n    at /usr/include/c++/5.3.1/bits/hashtable_policy.h:1973\r\n#28 0x00000000006f7d74 in std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > >, true> > >::_M_deallocate_nodes (this=0xffff900007563678, __n=0x0)\r\n    at /usr/include/c++/5.3.1/bits/hashtable_policy.h:1986\r\n#29 0x00000000006f7934 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<cha---Type <return> to continue, or q <return> to quit---\r\nr> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::clear (this=0xffff900007563678)\r\n    at /usr/include/c++/5.3.1/bits/hashtable.h:1914\r\n#30 0x00000000006f7348 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable (this=0xffff900007563678, __in_chrg=<optimized out>)\r\n    at /usr/include/c++/5.3.1/bits/hashtable.h:1225\r\n#31 0x00000000006f9302 in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unique_ptr<mount_context, std::default_delete<mount_context> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unique_ptr<mount_context, std::default_delete<mount_context> > > > >::~unordered_map (this=0xffff900007563678, __in_chrg=<optimized out>) at /usr/include/c++/5.3.1/bits/unordered_map.h:98\r\n#32 0x00000000006b6384 in <lambda()>::operator()(void) const (__closure=0xffff900002155010)\r\n    at libc/cxa_thread_atexit.cc:69\r\n#33 0x00000000006b6505 in std::_Function_handler<void(), register_call_destructor()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/5.3.1/functional:1871\r\n#34 0x000000000044f924 in std::function<void ()>::operator()() const (this=0xffff900002155010)\r\n    at /usr/include/c++/5.3.1/functional:2271\r\n#35 0x00000000005c149d in sched::run_exit_notifiers () at core/sched.cc:864\r\n#36 0x00000000005c2051 in sched::thread::complete (this=0xffff800006e21050) at core/sched.cc:1081\r\n#37 0x00000000005be772 in sched::thread_main_c (t=0xffff800006e21050) at arch/x64/arch-switch.hh:165\r\n#38 0x000000000048ceb6 in thread_main () at arch/x64/entry.S:113\r\n(gdb) \r\n\r\n```\r\n\r\n30 test clients are started, and abort happens when they start to exit. NFS ~mount_context is part of trace.\r\n\r\nSo far, it happened once in 80 runs."},"comment":{"url":"https://api.github.com/repos/cloudius-systems/osv/issues/comments/184715658","html_url":"https://github.com/cloudius-systems/osv/issues/726#issuecomment-184715658","issue_url":"https://api.github.com/repos/cloudius-systems/osv/issues/726","id":184715658,"user":{"login":"nyh","id":584227,"avatar_url":"https://avatars.githubusercontent.com/u/584227?v=3","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"created_at":"2016-02-16T14:58:53Z","updated_at":"2016-02-16T14:58:53Z","body":"On first glance, it seems to me that there's nothing that the NFS code could do to cause this bug - even calling close() twice in parallel should have worked (protected by a mutex). So it appears it is more likely to be a bug in our TCP/IP implementation. We should probably investigate (I don't remember...) what this \"tp->get_state() > 1\" assertion is about. It seems to me (but it's been ages since I looked at this code) that we are calling code which assumes the socket is not closed, on a closed socket.\r\n\r\nWould be nice if we could somehow reproduce this bug without NFS or OpenMPI - perhaps by some test program which opens many TCP connections in parallel and then closes all of them in parallel (after, optionally, writing/reading/? on each connection). I don't really know if it would reproduce this bug, but might be worth trying if reproducing Justin's original scenario is too difficult."}}