{"action":"opened","issue":{"url":"https://api.github.com/repos/Norconex/collector-http/issues/256","repository_url":"https://api.github.com/repos/Norconex/collector-http","labels_url":"https://api.github.com/repos/Norconex/collector-http/issues/256/labels{/name}","comments_url":"https://api.github.com/repos/Norconex/collector-http/issues/256/comments","events_url":"https://api.github.com/repos/Norconex/collector-http/issues/256/events","html_url":"https://github.com/Norconex/collector-http/issues/256","id":160188628,"number":256,"title":"Dynamically adding pages to the \"seeds\" (startup pages)&\"frontier\" (list of pages to crawl)","user":{"login":"liar666","id":3216927,"avatar_url":"https://avatars.githubusercontent.com/u/3216927?v=3","gravatar_id":"","url":"https://api.github.com/users/liar666","html_url":"https://github.com/liar666","followers_url":"https://api.github.com/users/liar666/followers","following_url":"https://api.github.com/users/liar666/following{/other_user}","gists_url":"https://api.github.com/users/liar666/gists{/gist_id}","starred_url":"https://api.github.com/users/liar666/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liar666/subscriptions","organizations_url":"https://api.github.com/users/liar666/orgs","repos_url":"https://api.github.com/users/liar666/repos","events_url":"https://api.github.com/users/liar666/events{/privacy}","received_events_url":"https://api.github.com/users/liar666/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-06-14T13:59:27Z","updated_at":"2016-06-14T13:59:27Z","closed_at":null,"body":"Hi,\r\n\r\nSince I got very good answers to my previous questions, I'll add some :)\r\n\r\nI'm faced with another problem: I have a site that acts like a modern search engine, with dynamic generation of the HTML results-pages from JSON response from an API. What the sites does behind the scene, and that you have to reproduce to crawl it, is:\r\n- go to an url like: http://api.domain.com/api/script?query=plasma&xxx\r\n- get as a response a JSON objects, like\r\n{\r\n\"status\": \"success\",\r\n\"count\": 3, \r\n\"results\": [{..., \"homepage\": \"http://...\", ...},{..., \"homepage\": \"http://...\", ...},{..., \"homepage\": \"http://...\", ...}},\r\n...\r\n}\r\n- the pages of interest are the \"homepage\" ones. These are the ones that are presented on the actual website, as the results-pages, and that you have to crawl to get the interesting info.\r\n\r\nMy questions with respect to crawling such a site are:\r\n1. Is it possible to dynamically generate the startup pages? I know it is possible to read them from a file, but I would prefer to create a Generator that would read a file with a bunch of queries, then generate on-the-fly the seeds/startURLs, like http://api.domain.com/api/script?query=<item1>&xxx\r\nWhat is the class I should overload in this case?\r\n2. Is there a native JSON parser/importer to help me extract the \"homepage\"s attributes of the returned JSON object, or should I write one? (according to what I see here: https://www.norconex.com/collectors/importer/latest/apidocs/ there is none...)\r\n3. Is it possible to re-introduce the extracted \"homepage\"s into the list of URLs still-to-be-crawled?\r\n4. Is it possible to create such an interlinked 2-step crawler, 1rst-step being the crawling & extraction from JSON results of \"search-pages\", resulting in the insertion of URLs to the 2nd-step crawler that consists in crawling & extracting from normal HTML pages?"}}
{"actor":{"display_login":"liar666"}}