{"action":"created","issue":{"url":"https://api.github.com/repos/mitmproxy/mitmproxy/issues/1117","repository_url":"https://api.github.com/repos/mitmproxy/mitmproxy","labels_url":"https://api.github.com/repos/mitmproxy/mitmproxy/issues/1117/labels{/name}","comments_url":"https://api.github.com/repos/mitmproxy/mitmproxy/issues/1117/comments","events_url":"https://api.github.com/repos/mitmproxy/mitmproxy/issues/1117/events","html_url":"https://github.com/mitmproxy/mitmproxy/issues/1117","id":153506114,"number":1117,"title":"Interaction between -r and -a","user":{"login":"thiagoarrais","id":3014,"avatar_url":"https://avatars.githubusercontent.com/u/3014?v=3","gravatar_id":"","url":"https://api.github.com/users/thiagoarrais","html_url":"https://github.com/thiagoarrais","followers_url":"https://api.github.com/users/thiagoarrais/followers","following_url":"https://api.github.com/users/thiagoarrais/following{/other_user}","gists_url":"https://api.github.com/users/thiagoarrais/gists{/gist_id}","starred_url":"https://api.github.com/users/thiagoarrais/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thiagoarrais/subscriptions","organizations_url":"https://api.github.com/users/thiagoarrais/orgs","repos_url":"https://api.github.com/users/thiagoarrais/repos","events_url":"https://api.github.com/users/thiagoarrais/events{/privacy}","received_events_url":"https://api.github.com/users/thiagoarrais/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-06T18:02:13Z","updated_at":"2016-05-06T19:37:11Z","closed_at":"2016-05-06T19:37:11Z","body":"I don't know what is the expected behavior here, but I've found out that doing this results in _very_ large flow files:\r\n\r\n1. Start some webserver\r\n`python -m SimpleHTTPServer 8000`\r\n\r\n2. Start a reverse proxy to that server. Append flows to some (non-existent) file\r\n`mitmproxy -a /tmp/somewhere.flows -p 8080 -R http://localhost:8000`\r\n\r\n3. Make some requests to said proxy:\r\n`curl -s http://localhost:8080 > /dev/null`\r\n\r\n4. Stop the proxy\r\n\r\n5. Check the size of the flows file:\r\n`stat -c \"%s %n\" /tmp/somewhere.flows` => `23059 /tmp/somewhere.flows`\r\n\r\n6. Start a new proxy instance. Append flows to same file. But, this time, try to read from it too.\r\n`mitmproxy -r /tmp/somewhere.flows -a /tmp/somewhere.flows -R http://localhost:8000`\r\n\r\n7. Do not issue any requests. Just wait for a few seconds (the process hangs)\r\n8. Kill the proxy process\r\n\r\n9. Check the size of the flows file again:\r\n`stat -c \"%s %n\" /tmp/somewhere.flows` => `21942730 /tmp/somewhere.flows`\r\n\r\n10. Wow!\r\n\r\nI did this expecting to be able to work with the previous captured flows (filter them, repeat requests, etc.) while still recording the new interactions. Do you think this is a valid use case?\r\n\r\nTested with mitmproxy 0.17 and master."},"comment":{"url":"https://api.github.com/repos/mitmproxy/mitmproxy/issues/comments/217539083","html_url":"https://github.com/mitmproxy/mitmproxy/issues/1117#issuecomment-217539083","issue_url":"https://api.github.com/repos/mitmproxy/mitmproxy/issues/1117","id":217539083,"user":{"login":"mhils","id":1019198,"avatar_url":"https://avatars.githubusercontent.com/u/1019198?v=3","gravatar_id":"","url":"https://api.github.com/users/mhils","html_url":"https://github.com/mhils","followers_url":"https://api.github.com/users/mhils/followers","following_url":"https://api.github.com/users/mhils/following{/other_user}","gists_url":"https://api.github.com/users/mhils/gists{/gist_id}","starred_url":"https://api.github.com/users/mhils/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mhils/subscriptions","organizations_url":"https://api.github.com/users/mhils/orgs","repos_url":"https://api.github.com/users/mhils/repos","events_url":"https://api.github.com/users/mhils/events{/privacy}","received_events_url":"https://api.github.com/users/mhils/received_events","type":"User","site_admin":false},"created_at":"2016-05-06T19:37:11Z","updated_at":"2016-05-06T19:37:11Z","body":"I think this expected behaviour. We don't want to have all flows in memory, so we read iteratively. If you append one flow for each flow that has been read, you end up in an infinite loop. \r\nIf you just want to record new interactions, `-a` is append only, you don't need to re-read the existing ones. If you want to filter an existing file, `-r existing -w new` does the job just fine. If you want to concat two files, you want to use `-r foo -a all`."}}