{"action":"created","issue":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/257","repository_url":"https://api.github.com/repos/cgiffard/node-simplecrawler","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/257/labels{/name}","comments_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/257/comments","events_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/257/events","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/257","id":144177475,"number":257,"title":"Asynchronously save queue items","user":{"login":"fredrikekelund","id":1101677,"avatar_url":"https://avatars.githubusercontent.com/u/1101677?v=3","gravatar_id":"","url":"https://api.github.com/users/fredrikekelund","html_url":"https://github.com/fredrikekelund","followers_url":"https://api.github.com/users/fredrikekelund/followers","following_url":"https://api.github.com/users/fredrikekelund/following{/other_user}","gists_url":"https://api.github.com/users/fredrikekelund/gists{/gist_id}","starred_url":"https://api.github.com/users/fredrikekelund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredrikekelund/subscriptions","organizations_url":"https://api.github.com/users/fredrikekelund/orgs","repos_url":"https://api.github.com/users/fredrikekelund/repos","events_url":"https://api.github.com/users/fredrikekelund/events{/privacy}","received_events_url":"https://api.github.com/users/fredrikekelund/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/labels/New%20Feature","name":"New Feature","color":"02e10c"},{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/labels/Underway","name":"Underway","color":"fbca04"}],"state":"open","locked":false,"assignee":{"login":"fredrikekelund","id":1101677,"avatar_url":"https://avatars.githubusercontent.com/u/1101677?v=3","gravatar_id":"","url":"https://api.github.com/users/fredrikekelund","html_url":"https://github.com/fredrikekelund","followers_url":"https://api.github.com/users/fredrikekelund/followers","following_url":"https://api.github.com/users/fredrikekelund/following{/other_user}","gists_url":"https://api.github.com/users/fredrikekelund/gists{/gist_id}","starred_url":"https://api.github.com/users/fredrikekelund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredrikekelund/subscriptions","organizations_url":"https://api.github.com/users/fredrikekelund/orgs","repos_url":"https://api.github.com/users/fredrikekelund/repos","events_url":"https://api.github.com/users/fredrikekelund/events{/privacy}","received_events_url":"https://api.github.com/users/fredrikekelund/received_events","type":"User","site_admin":false},"assignees":[{"login":"fredrikekelund","id":1101677,"avatar_url":"https://avatars.githubusercontent.com/u/1101677?v=3","gravatar_id":"","url":"https://api.github.com/users/fredrikekelund","html_url":"https://github.com/fredrikekelund","followers_url":"https://api.github.com/users/fredrikekelund/followers","following_url":"https://api.github.com/users/fredrikekelund/following{/other_user}","gists_url":"https://api.github.com/users/fredrikekelund/gists{/gist_id}","starred_url":"https://api.github.com/users/fredrikekelund/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fredrikekelund/subscriptions","organizations_url":"https://api.github.com/users/fredrikekelund/orgs","repos_url":"https://api.github.com/users/fredrikekelund/repos","events_url":"https://api.github.com/users/fredrikekelund/events{/privacy}","received_events_url":"https://api.github.com/users/fredrikekelund/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/milestones/1","html_url":"https://github.com/cgiffard/node-simplecrawler/milestones/1.0","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/milestones/1/labels","id":1645384,"number":1,"title":"1.0","description":"Aiming for:\r\n\r\n* Cleaned up API for fetchConditions and the addition of downloadConditions\r\n* The crawler should use this same API internally\r\n* CLI\r\n* Robots.txt\r\n* Better phantom integration support\r\n* No more hanging!\r\n* Contributor guide\r\n* Troubleshooting guide\r\n* Some good debugging tools","creator":{"login":"cgiffard","id":453993,"avatar_url":"https://avatars.githubusercontent.com/u/453993?v=3","gravatar_id":"","url":"https://api.github.com/users/cgiffard","html_url":"https://github.com/cgiffard","followers_url":"https://api.github.com/users/cgiffard/followers","following_url":"https://api.github.com/users/cgiffard/following{/other_user}","gists_url":"https://api.github.com/users/cgiffard/gists{/gist_id}","starred_url":"https://api.github.com/users/cgiffard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgiffard/subscriptions","organizations_url":"https://api.github.com/users/cgiffard/orgs","repos_url":"https://api.github.com/users/cgiffard/repos","events_url":"https://api.github.com/users/cgiffard/events{/privacy}","received_events_url":"https://api.github.com/users/cgiffard/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":15,"state":"open","created_at":"2016-03-15T11:46:02Z","updated_at":"2016-07-17T19:50:47Z","due_on":null,"closed_at":null},"comments":11,"created_at":"2016-03-29T07:09:47Z","updated_at":"2016-07-21T03:14:38Z","closed_at":null,"body":"Building on #253, I'd like to look into how we mutate queue items. Currently it's all done synchronously, and I'd like to save them asynchronously after they're mutated, perhaps via an upsert method on the queue that takes two arguments, `id` and a `queueItem`. With this, we'd truly enable different queue backing stores, and even make it really easy to run the crawler in a distributed fashion."},"comment":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/comments/234145644","html_url":"https://github.com/cgiffard/node-simplecrawler/issues/257#issuecomment-234145644","issue_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/257","id":234145644,"user":{"login":"jtlapp","id":138133,"avatar_url":"https://avatars.githubusercontent.com/u/138133?v=3","gravatar_id":"","url":"https://api.github.com/users/jtlapp","html_url":"https://github.com/jtlapp","followers_url":"https://api.github.com/users/jtlapp/followers","following_url":"https://api.github.com/users/jtlapp/following{/other_user}","gists_url":"https://api.github.com/users/jtlapp/gists{/gist_id}","starred_url":"https://api.github.com/users/jtlapp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jtlapp/subscriptions","organizations_url":"https://api.github.com/users/jtlapp/orgs","repos_url":"https://api.github.com/users/jtlapp/repos","events_url":"https://api.github.com/users/jtlapp/events{/privacy}","received_events_url":"https://api.github.com/users/jtlapp/received_events","type":"User","site_admin":false},"created_at":"2016-07-21T03:14:38Z","updated_at":"2016-07-21T03:14:38Z","body":"I suspect there might be a more modular way to separate concerns, so that queue-smart people can write queue wrappers, and application writers can just pick a queue and separately write the application logic. I fear that this interface may require wrapping the queue in an application layer.\r\n\r\nI think we want to be able to say that a queue must pass the queue test suite in order to assure that it works with the crawler. Anything that isn't strictly necessary for queuing might be hung off of a different component, which could be either application-specific or generic.\r\n\r\nI'll have a closer look. But the queue interface does seem to impose unnecessary requirements."}}
{"actor":{"display_login":"jtlapp"}}