{"action":"created","issue":{"url":"https://api.github.com/repos/camlistore/camlistore/issues/878","repository_url":"https://api.github.com/repos/camlistore/camlistore","labels_url":"https://api.github.com/repos/camlistore/camlistore/issues/878/labels{/name}","comments_url":"https://api.github.com/repos/camlistore/camlistore/issues/878/comments","events_url":"https://api.github.com/repos/camlistore/camlistore/issues/878/events","html_url":"https://github.com/camlistore/camlistore/issues/878","id":187528995,"number":878,"title":"deadlock between indexing & search (via importers)","user":{"login":"bradfitz","id":2621,"avatar_url":"https://avatars.githubusercontent.com/u/2621?v=3","gravatar_id":"","url":"https://api.github.com/users/bradfitz","html_url":"https://github.com/bradfitz","followers_url":"https://api.github.com/users/bradfitz/followers","following_url":"https://api.github.com/users/bradfitz/following{/other_user}","gists_url":"https://api.github.com/users/bradfitz/gists{/gist_id}","starred_url":"https://api.github.com/users/bradfitz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bradfitz/subscriptions","organizations_url":"https://api.github.com/users/bradfitz/orgs","repos_url":"https://api.github.com/users/bradfitz/repos","events_url":"https://api.github.com/users/bradfitz/events{/privacy}","received_events_url":"https://api.github.com/users/bradfitz/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-11-05T22:48:11Z","updated_at":"2016-11-08T00:10:36Z","closed_at":null,"body":"I keep getting camlistored deadlocks trying to reindex.\r\n\r\nNo web browser is open. (No websocket), which was a problem in the past.\r\n\r\nWhat I see in the stacks is:\r\n\r\n```\r\ngoroutine 2732318 [semacquire, 10 minutes]:\r\nsync.runtime_Semacquire(0xc420a803a4)\r\n        /home/bradfitz/go1.7/src/runtime/sema.go:47 +0x30\r\nsync.(*RWMutex).RLock(0xc420a80398)\r\n        /home/bradfitz/go1.7/src/sync/rwmutex.go:43 +0x5d\r\ncamlistore.org/pkg/index.(*Index).RLock(0xc420a80360)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/index/index.go:88 +0x31\r\ncamlistore.org/pkg/search.(*Handler).Query(0xc420200540, 0xc458246000, 0x0, 0x0, 0x0)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/search/query.go:867 +0x19c\r\ncamlistore.org/pkg/importer.(*importer).Node(0xc4206a13b0, 0x0, 0x0, 0x0)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/importer/importer.go:948 +0x32b\r\ncamlistore.org/pkg/importer.(*importer).credentials(0xc4206a13b0, 0xa1, 0xa1, 0xa1, 0x0, 0xa1, 0xa)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/importer/importer.go:792 +0x40\r\ncamlistore.org/pkg/importer.(*RunContext).Credentials(0xc45cbab500, 0xc455063000, 0xa1, 0x412498, 0xc480f37d60, 0x8, 0x0)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/importer/importer.go:331 +0x32\r\ncamlistore.org/pkg/importer/twitter.(*imp).Run(0x1555d60, 0xc45cbab500, 0x0, 0x0)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/importer/twitter/twitter.go:167 +0x96\r\ncamlistore.org/pkg/importer.(*importerAcct).start.func1(0xc46c30a000, 0xc45cbab500)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/importer/importer.go:1137 +0x159\r\ncreated by camlistore.org/pkg/importer.(*importerAcct).start\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/importer/importer.go:1150 +0x2a0\r\n```\r\n\r\nAnd tons of:\r\n\r\n```\r\ngoroutine 2731853 [semacquire, 10 minutes]:\r\nsync.runtime_Semacquire(0xc420a8039c)\r\n        /home/bradfitz/go1.7/src/runtime/sema.go:47 +0x30\r\nsync.(*Mutex).Lock(0xc420a80398)\r\n        /home/bradfitz/go1.7/src/sync/mutex.go:85 +0xd0\r\nsync.(*RWMutex).Lock(0xc420a80398)\r\n        /home/bradfitz/go1.7/src/sync/rwmutex.go:86 +0x31\r\ncamlistore.org/pkg/index.(*Index).Lock(0xc420a80360)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/index/index.go:86 +0x31\r\ncamlistore.org/pkg/index.(*Index).ReceiveBlob(0xc420a80360, 0x1503c20, 0xc449cab300, 0x14f83a0, 0xc45d23d340, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/index/receive.go:195 +0xa2\r\ncamlistore.org/pkg/serverinit.(*storageAndConfig).ReceiveBlob(0xc426316d40, 0x1503c20, 0xc449cab300, 0x14f83a0, 0xc45d23d340, 0x41127f, 0x153ddc0, 0x7fc462408b00, 0xd76fe0, 0x779911)\r\n        <autogenerated>:56 +0x78\r\ncamlistore.org/pkg/blobserver.receive(0x7fc462408b00, 0xc426316d40, 0x1503c20, 0xc449cab300, 0x14f83a0, 0xc45d23d340, 0xc481de1801, 0x4128bf, 0xd306c0, 0xd9c520, ...)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/blobserver/receive.go:50 +0xd0\r\ncamlistore.org/pkg/blobserver.Receive(0x7fc462408b00, 0xc426316d40, 0x1503c20, 0xc449cab300, 0x14f8860, 0xc449cab360, 0x0, 0x41492e, 0xc480f37900, 0x556b0b, ...)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/blobserver/receive.go:38 +0x72\r\ncamlistore.org/pkg/blobserver/handlers.handleMultiPartUpload(0x1503360, 0xc45f228c60, 0xc4361ce870, 0x7fc462408aa8, 0xc426316d40)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/blobserver/handlers/upload.go:229 +0x4cd\r\ncamlistore.org/pkg/blobserver/handlers.CreateBatchUploadHandler.func1(0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/blobserver/handlers/upload.go:44 +0x51\r\nnet/http.HandlerFunc.ServeHTTP(0xc449ca6a00, 0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/go1.7/src/net/http/server.go:1726 +0x44\r\ncamlistore.org/pkg/auth.RequireAuth.func1(0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/auth/auth.go:410 +0x85\r\nnet/http.HandlerFunc.ServeHTTP(0xc449ca6a20, 0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/go1.7/src/net/http/server.go:1726 +0x44\r\ncamlistore.org/pkg/serverinit.makeCamliHandler.func1(0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/serverinit/serverinit.go:194 +0x217\r\nnet/http.HandlerFunc.ServeHTTP(0xc426316d60, 0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/go1.7/src/net/http/server.go:1726 +0x44\r\nnet/http.(*ServeMux).ServeHTTP(0xc420944b40, 0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/go1.7/src/net/http/server.go:2022 +0x7f\r\ncamlistore.org/pkg/webserver.(*Server).ServeHTTP(0xc42000ed80, 0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/src/camlistore.org/tmp/build-gopath-nosqlite/src/camlistore.org/pkg/webserver/webserver.go:127 +0xaa\r\nnet/http.serverHandler.ServeHTTP(0xc46c228200, 0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/go1.7/src/net/http/server.go:2202 +0x7d\r\nnet/http.initNPNRequest.ServeHTTP(0xc469ff0a80, 0xc46c228200, 0x1503360, 0xc45f228c60, 0xc4361ce870)\r\n        /home/bradfitz/go1.7/src/net/http/server.go:2670 +0x93\r\n```"},"comment":{"url":"https://api.github.com/repos/camlistore/camlistore/issues/comments/259005137","html_url":"https://github.com/camlistore/camlistore/issues/878#issuecomment-259005137","issue_url":"https://api.github.com/repos/camlistore/camlistore/issues/878","id":259005137,"user":{"login":"mpl","id":626071,"avatar_url":"https://avatars.githubusercontent.com/u/626071?v=3","gravatar_id":"","url":"https://api.github.com/users/mpl","html_url":"https://github.com/mpl","followers_url":"https://api.github.com/users/mpl/followers","following_url":"https://api.github.com/users/mpl/following{/other_user}","gists_url":"https://api.github.com/users/mpl/gists{/gist_id}","starred_url":"https://api.github.com/users/mpl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpl/subscriptions","organizations_url":"https://api.github.com/users/mpl/orgs","repos_url":"https://api.github.com/users/mpl/repos","events_url":"https://api.github.com/users/mpl/events{/privacy}","received_events_url":"https://api.github.com/users/mpl/received_events","type":"User","site_admin":false},"created_at":"2016-11-08T00:10:36Z","updated_at":"2016-11-08T00:10:36Z","body":"Just from your logs it looks like (from the very last one you pasted), that one of the uploads is blocked trying to read the blob from the source (pkg/blobserver/receive.go:72), so it never finishes indexing that blob, and never relinquishes the mutex? hence all the other ones deadlocked?\r\n"}}
{"actor":{"display_login":"mpl"}}