{"action":"created","issue":{"url":"https://api.github.com/repos/jhillyerd/inbucket/issues/77","repository_url":"https://api.github.com/repos/jhillyerd/inbucket","labels_url":"https://api.github.com/repos/jhillyerd/inbucket/issues/77/labels{/name}","comments_url":"https://api.github.com/repos/jhillyerd/inbucket/issues/77/comments","events_url":"https://api.github.com/repos/jhillyerd/inbucket/issues/77/events","html_url":"https://github.com/jhillyerd/inbucket/issues/77","id":301688132,"number":77,"title":"Concurrency issues","user":{"login":"makarchuk","id":13999738,"avatar_url":"https://avatars1.githubusercontent.com/u/13999738?v=4","gravatar_id":"","url":"https://api.github.com/users/makarchuk","html_url":"https://github.com/makarchuk","followers_url":"https://api.github.com/users/makarchuk/followers","following_url":"https://api.github.com/users/makarchuk/following{/other_user}","gists_url":"https://api.github.com/users/makarchuk/gists{/gist_id}","starred_url":"https://api.github.com/users/makarchuk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/makarchuk/subscriptions","organizations_url":"https://api.github.com/users/makarchuk/orgs","repos_url":"https://api.github.com/users/makarchuk/repos","events_url":"https://api.github.com/users/makarchuk/events{/privacy}","received_events_url":"https://api.github.com/users/makarchuk/received_events","type":"User","site_admin":false},"labels":[{"id":15774256,"url":"https://api.github.com/repos/jhillyerd/inbucket/labels/bug","name":"bug","color":"e99695","default":true}],"state":"open","locked":false,"assignee":{"login":"makarchuk","id":13999738,"avatar_url":"https://avatars1.githubusercontent.com/u/13999738?v=4","gravatar_id":"","url":"https://api.github.com/users/makarchuk","html_url":"https://github.com/makarchuk","followers_url":"https://api.github.com/users/makarchuk/followers","following_url":"https://api.github.com/users/makarchuk/following{/other_user}","gists_url":"https://api.github.com/users/makarchuk/gists{/gist_id}","starred_url":"https://api.github.com/users/makarchuk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/makarchuk/subscriptions","organizations_url":"https://api.github.com/users/makarchuk/orgs","repos_url":"https://api.github.com/users/makarchuk/repos","events_url":"https://api.github.com/users/makarchuk/events{/privacy}","received_events_url":"https://api.github.com/users/makarchuk/received_events","type":"User","site_admin":false},"assignees":[{"login":"makarchuk","id":13999738,"avatar_url":"https://avatars1.githubusercontent.com/u/13999738?v=4","gravatar_id":"","url":"https://api.github.com/users/makarchuk","html_url":"https://github.com/makarchuk","followers_url":"https://api.github.com/users/makarchuk/followers","following_url":"https://api.github.com/users/makarchuk/following{/other_user}","gists_url":"https://api.github.com/users/makarchuk/gists{/gist_id}","starred_url":"https://api.github.com/users/makarchuk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/makarchuk/subscriptions","organizations_url":"https://api.github.com/users/makarchuk/orgs","repos_url":"https://api.github.com/users/makarchuk/repos","events_url":"https://api.github.com/users/makarchuk/events{/privacy}","received_events_url":"https://api.github.com/users/makarchuk/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/jhillyerd/inbucket/milestones/9","html_url":"https://github.com/jhillyerd/inbucket/milestone/9","labels_url":"https://api.github.com/repos/jhillyerd/inbucket/milestones/9/labels","id":3156526,"number":9,"title":"v1.3.1","description":"","creator":{"login":"jhillyerd","id":2502736,"avatar_url":"https://avatars3.githubusercontent.com/u/2502736?v=4","gravatar_id":"","url":"https://api.github.com/users/jhillyerd","html_url":"https://github.com/jhillyerd","followers_url":"https://api.github.com/users/jhillyerd/followers","following_url":"https://api.github.com/users/jhillyerd/following{/other_user}","gists_url":"https://api.github.com/users/jhillyerd/gists{/gist_id}","starred_url":"https://api.github.com/users/jhillyerd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhillyerd/subscriptions","organizations_url":"https://api.github.com/users/jhillyerd/orgs","repos_url":"https://api.github.com/users/jhillyerd/repos","events_url":"https://api.github.com/users/jhillyerd/events{/privacy}","received_events_url":"https://api.github.com/users/jhillyerd/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":0,"state":"open","created_at":"2018-03-02T15:59:18Z","updated_at":"2018-03-02T15:59:42Z","due_on":null,"closed_at":null},"comments":8,"created_at":"2018-03-02T08:27:06Z","updated_at":"2018-03-08T15:00:06Z","closed_at":null,"author_association":"NONE","body":"Inbucket seems to handle poorly multiple emails receiving:\r\n\r\nmy test python script: \r\n```\r\nfrom multiprocessing.pool import ThreadPool\r\nimport requests\r\nimport time\r\n\r\np = ThreadPool(500)\r\n\r\neml = \"\"\"\r\nTo: test@test.com\r\nFrom: Empty <empty@example.net>\r\nSubject: Empty message\r\nMessage-ID: <7ced65ff-46e2-9ae2-e589-237ab9201078@example.net>\r\nDate: Fri, 10 Nov 2017 15:09:34 +0300\r\nX-Mozilla-Draft-Info: internal/draft; vcard=0; receipt=0; DSN=0; uuencode=0;\r\n attachmentreminder=0; deliveryformat=4\r\nUser-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\r\n Thunderbird/52.4.0\r\nMIME-Version: 1.0\r\nContent-Type: text/plain; charset=utf-8; format=flowed\r\nContent-Language: en-US\r\nContent-Transfer-Encoding: 7bit\r\n\r\nSome text\r\n\"\"\"\r\n\r\ndef send(i):   \r\n    import smtplib                                                       \r\n    cli = smtplib.SMTP('localhost', port=10025)                     \r\n    cli.sendmail('test@test.com', ['test@test.com'], eml)\r\n\r\nrequests.delete('http://localhost:10080/api/v1/mailbox/test')\r\nprint(\"mailbox size after purge: {}\".format(len(requests.get('http://localhost:10080/api/v1/mailbox/test').json())))\r\ntime.sleep(1)\r\np.map(send, range(1000))\r\ntime.sleep(1)\r\nprint(\"mailbox size after send: {}\".format(len(requests.get('http://localhost:10080/api/v1/mailbox/test').json())))\r\n```\r\n\r\nExcpected: result:\r\n```\r\nmailbox size after purge: 0\r\nmailbox size after send: 1000\r\n```\r\n\r\nActual result:\r\n```\r\nmailbox size after purge: 0\r\nmailbox size after send: %MUCH LOWER NUMBER%\r\n```\r\nNumber of emails I see is usually around 20-50, but it differs from run to run\r\n\r\n\r\nMy inbucket config: \r\n```\r\n# inbucket.conf\r\n# Configuration for Inbucket inside of Docker\r\n#\r\n# These should be reasonable defaults for a production install of Inbucket\r\n\r\n#############################################################################\r\n[DEFAULT]\r\n\r\n# Not used directly, but is typically referenced below in %()s format.\r\ninstall.dir=/opt/inbucket\r\ndefault.domain=inbucket.local\r\n\r\n#############################################################################\r\n[logging]\r\n\r\n# Options from least to most verbose: ERROR, WARN, INFO, TRACE\r\nlevel=INFO\r\n\r\n#############################################################################\r\n[smtp]\r\n\r\n# IPv4 address to listen for SMTP connections on.\r\nip4.address=0.0.0.0\r\n\r\n# IPv4 port to listen for SMTP connections on.\r\nip4.port=10025\r\n\r\n# used in SMTP greeting\r\ndomain=%(default.domain)s\r\n\r\n# optional: mail sent to accounts at this domain will not be stored,\r\n# for mixed use (content and load testing)\r\ndomain.nostore=bitbucket.local\r\n\r\n# Maximum number of RCPT TO: addresses we allow from clients, the SMTP\r\n# RFC recommends this be at least 100.\r\nmax.recipients=100\r\n\r\n# How long we allow a network connection to be idle before hanging up on the\r\n# client, SMTP RFC recommends at least 5 minutes (300 seconds).\r\nmax.idle.seconds=300\r\n\r\n# Maximum allowable size of message body in bytes (including attachments)\r\nmax.message.bytes=20480000\r\n\r\n# Should we place messages into the datastore, or just throw them away\r\n# (for load testing): true or false\r\nstore.messages=true\r\n\r\n#############################################################################\r\n[pop3]\r\n\r\n# IPv4 address to listen for POP3 connections on.\r\nip4.address=0.0.0.0\r\n\r\n# IPv4 port to listen for POP3 connections on.\r\nip4.port=10110\r\n\r\n# used in POP3 greeting\r\ndomain=%(default.domain)s\r\n\r\n# How long we allow a network connection to be idle before hanging up on the\r\n# client, POP3 RFC requires at least 10 minutes (600 seconds).\r\nmax.idle.seconds=600\r\n\r\n#############################################################################\r\n[web]\r\n\r\n# IPv4 address to serve HTTP web interface on\r\nip4.address=0.0.0.0\r\n\r\n# IPv4 port to serve HTTP web interface on\r\nip4.port=10080\r\n\r\n# Name of web theme to use\r\ntheme=bootstrap\r\n\r\n# Prompt displayed between the mailbox entry field and View button. Leave\r\n# empty or comment out to hide the prompt.\r\nmailbox.prompt=@inbucket\r\n\r\n# Path to the selected themes template files\r\ntemplate.dir=%(install.dir)s/themes/%(theme)s/templates\r\n\r\n# Should we cache parsed templates (set to false during theme dev)\r\ntemplate.cache=true\r\n\r\n# Path to the selected themes public (static) files\r\npublic.dir=%(install.dir)s/themes/%(theme)s/public\r\n\r\n# Path to the greeting HTML displayed on front page, can be moved out of\r\n# installation dir for customization\r\ngreeting.file=/con/configuration/greeting.html\r\n\r\n# Key used to sign session cookie data so that it cannot be tampered with.\r\n# If this is left unset, Inbucket will generate a random key at startup\r\n# and previous sessions will be invalidated.\r\n#cookie.auth.key=secret-inbucket-session-cookie-key\r\n\r\n# Enable or disable the live message monitor tab for the web UI. This will let\r\n# anybody see all messages delivered to Inbucket.  This setting has no impact\r\n# on the availability of the underlying WebSocket.\r\nmonitor.visible=true\r\n\r\n# How many historical message headers should be cached for display by new\r\n# monitor connections.  It does not limit the number of messages displayed by\r\n# the browser once the monitor is open; all freshly received messages will be\r\n# appended to the on screen list.  This setting also affects the underlying\r\n# API/WebSocket.\r\nmonitor.history=30\r\n\r\n#############################################################################\r\n[datastore]\r\n\r\n# Path to the datastore, mail will be written into subdirectories\r\npath=/con/data\r\n\r\n# How many minutes after receipt should a message be stored until it's\r\n# automatically purged.  To retain messages until manually deleted, set this\r\n# to 0\r\nretention.minutes=4320\r\n\r\n# How many milliseconds to sleep after purging messages from a mailbox.\r\n# This should help reduce disk I/O when there are a large number of messages\r\n# to purge.\r\nretention.sleep.millis=100\r\n\r\n# Maximum number of messages we will store in a single mailbox. If this\r\n# number is exceeded, the oldest message in the box will be deleted each\r\n# time a new message is received for it.\r\nmailbox.message.cap=3000\r\n```\r\n\r\nAll emails seem to be present in `/con/data/mail/a94/a94a8f/a94a8fe5ccb19ba61c4c0873d391e987982fbbd3/` folder they are just missing in index.\r\n\r\nAny help is much appreciated"},"comment":{"url":"https://api.github.com/repos/jhillyerd/inbucket/issues/comments/371512218","html_url":"https://github.com/jhillyerd/inbucket/issues/77#issuecomment-371512218","issue_url":"https://api.github.com/repos/jhillyerd/inbucket/issues/77","id":371512218,"user":{"login":"jhillyerd","id":2502736,"avatar_url":"https://avatars3.githubusercontent.com/u/2502736?v=4","gravatar_id":"","url":"https://api.github.com/users/jhillyerd","html_url":"https://github.com/jhillyerd","followers_url":"https://api.github.com/users/jhillyerd/followers","following_url":"https://api.github.com/users/jhillyerd/following{/other_user}","gists_url":"https://api.github.com/users/jhillyerd/gists{/gist_id}","starred_url":"https://api.github.com/users/jhillyerd/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jhillyerd/subscriptions","organizations_url":"https://api.github.com/users/jhillyerd/orgs","repos_url":"https://api.github.com/users/jhillyerd/repos","events_url":"https://api.github.com/users/jhillyerd/events{/privacy}","received_events_url":"https://api.github.com/users/jhillyerd/received_events","type":"User","site_admin":false},"created_at":"2018-03-08T15:00:06Z","updated_at":"2018-03-08T15:00:06Z","author_association":"OWNER","body":"Yes, I think this is a good idea.  Getting rid of Mailbox, and requiring all mutation calls to be made on the central Store interface would allow the Store implementation to hide any required locking or caching from client code.  We should continue this in #69.\r\n\r\nI think using a keymutex style lock at message delivery and message delete is the quick and dirty fix for this bug.\r\n"}}
{"actor":{"display_login":"jhillyerd"}}