{"action":"created","issue":{"url":"https://api.github.com/repos/indix/web-auto-extractor/issues/10","repository_url":"https://api.github.com/repos/indix/web-auto-extractor","labels_url":"https://api.github.com/repos/indix/web-auto-extractor/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/indix/web-auto-extractor/issues/10/comments","events_url":"https://api.github.com/repos/indix/web-auto-extractor/issues/10/events","html_url":"https://github.com/indix/web-auto-extractor/issues/10","id":184823236,"number":10,"title":"Title, redirects, and user-agent","user":{"login":"gpthome","id":5378257,"avatar_url":"https://avatars.githubusercontent.com/u/5378257?v=3","gravatar_id":"","url":"https://api.github.com/users/gpthome","html_url":"https://github.com/gpthome","followers_url":"https://api.github.com/users/gpthome/followers","following_url":"https://api.github.com/users/gpthome/following{/other_user}","gists_url":"https://api.github.com/users/gpthome/gists{/gist_id}","starred_url":"https://api.github.com/users/gpthome/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gpthome/subscriptions","organizations_url":"https://api.github.com/users/gpthome/orgs","repos_url":"https://api.github.com/users/gpthome/repos","events_url":"https://api.github.com/users/gpthome/events{/privacy}","received_events_url":"https://api.github.com/users/gpthome/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-24T12:11:00Z","updated_at":"2016-11-24T14:49:26Z","closed_at":null,"body":"My team is comparing this package, web-auto-extractor, with node-metainspector and meta-extractor.\n\nTITLE - web-auto-extractor seems to return all the meta tags including og tags, along with the LRMI tags.  Great.  But, it doesn't seem to return some basic HTML elements like 'Title', which is actually important.  Was this intentional?  Perhaps I'm overlooking something.  Perhaps the intent of this package is different than what I'm trying to do.\n\nREDIRECTS - node-metainspector will also follow redirects up to 5 redirects I believe.  This is nice and many sites do actually redirect as their main page changes.  Does web-auto-extractor have the ability to follow redirects?\n\nUSER-AGENT - Both other packages allow the user-agent to be set.  Does web-auto-extractor have this ability too?\n\nThanks.\n"},"comment":{"url":"https://api.github.com/repos/indix/web-auto-extractor/issues/comments/262792885","html_url":"https://github.com/indix/web-auto-extractor/issues/10#issuecomment-262792885","issue_url":"https://api.github.com/repos/indix/web-auto-extractor/issues/10","id":262792885,"user":{"login":"addnab","id":1926126,"avatar_url":"https://avatars.githubusercontent.com/u/1926126?v=3","gravatar_id":"","url":"https://api.github.com/users/addnab","html_url":"https://github.com/addnab","followers_url":"https://api.github.com/users/addnab/followers","following_url":"https://api.github.com/users/addnab/following{/other_user}","gists_url":"https://api.github.com/users/addnab/gists{/gist_id}","starred_url":"https://api.github.com/users/addnab/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/addnab/subscriptions","organizations_url":"https://api.github.com/users/addnab/orgs","repos_url":"https://api.github.com/users/addnab/repos","events_url":"https://api.github.com/users/addnab/events{/privacy}","received_events_url":"https://api.github.com/users/addnab/received_events","type":"User","site_admin":false},"created_at":"2016-11-24T14:49:26Z","updated_at":"2016-11-24T14:49:26Z","body":"Sorry for the late reply. I guess you must have already found your favorite package by now. I'll answer your questions anyway.\r\n\r\n> TITLE - web-auto-extractor seems to return all the meta tags including og tags, along with the LRMI tags. Great. But, it doesn't seem to return some basic HTML elements like 'Title', which is actually important. Was this intentional? Perhaps I'm overlooking something. Perhaps the intent of this package is different than what I'm trying to do.\r\n\r\nYou are right, capturing `title` is important. When we decided to write this, our main focus was on its ability to parse Schema.org information.\r\n\r\n> REDIRECTS - node-metainspector will also follow redirects up to 5 redirects I believe. This is nice and many sites do actually redirect as their main page changes. Does web-auto-extractor have the ability to follow redirects?\r\n\r\n> USER-AGENT - Both other packages allow the user-agent to be set. Does web-auto-extractor have this ability too?\r\n\r\nweb-auto-extractor only takes the HTML as input and doesn't do any fetching. It is fully synchronous. The decision on how you'd want to fetch the page is up to you."}}
{"actor":{"display_login":"addnab"}}