{"action":"created","issue":{"url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/169","labels_url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/169/labels{/name}","comments_url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/169/comments","events_url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/169/events","html_url":"https://github.com/jaimeiniesta/metainspector/pull/169","id":128157564,"number":169,"title":"Improve parsing of image content","user":{"login":"trotzig","id":578277,"avatar_url":"https://avatars.githubusercontent.com/u/578277?v=3","gravatar_id":"","url":"https://api.github.com/users/trotzig","html_url":"https://github.com/trotzig","followers_url":"https://api.github.com/users/trotzig/followers","following_url":"https://api.github.com/users/trotzig/following{/other_user}","gists_url":"https://api.github.com/users/trotzig/gists{/gist_id}","starred_url":"https://api.github.com/users/trotzig/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/trotzig/subscriptions","organizations_url":"https://api.github.com/users/trotzig/orgs","repos_url":"https://api.github.com/users/trotzig/repos","events_url":"https://api.github.com/users/trotzig/events{/privacy}","received_events_url":"https://api.github.com/users/trotzig/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-22T13:31:57Z","updated_at":"2016-01-22T15:31:00Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/jaimeiniesta/metainspector/pulls/169","html_url":"https://github.com/jaimeiniesta/metainspector/pull/169","diff_url":"https://github.com/jaimeiniesta/metainspector/pull/169.diff","patch_url":"https://github.com/jaimeiniesta/metainspector/pull/169.patch"},"body":"I was tracking down a bug in our app where users would paste a url to an\r\nimage in a comment field and see a garbled summary of the contents\r\nbehind that url. We use metainspector in the backend to scrape urls, and\r\nafter digging around a bit I discovered that any url with content type\r\n`image/*` would result in bad descriptions.\r\n\r\nSo I filed an issue: #168\r\n\r\nAnd then I read the docs (as you do after first filing an issue...),\r\nonly to discover that scraping images wasn't really part of the\r\nresponsibility of metainspector [1]. Which I agree with, though the\r\nproblem for me as a consumer of the gem is that in order to really\r\nfigure out if an url serves an image, I have to first fetch it. That\r\nfeels like a lot of work, and so I started thinking about ways to make\r\nmetainspector work decently when fetching image urls. This commit is a\r\nresult of that:\r\n\r\n - Return `nil` for text descriptions, titles and such\r\n - Return the image url as the only image found\r\n\r\nTo work around an issue with grabbing content type for documents where a\r\n`document: '<html/>'` option was passed in, I inlined the `content_type`\r\nmethod instead of delegating it, and added a nil-check.\r\n\r\n[Fixes #168]\r\n\r\n[1]: https://github.com/jaimeiniesta/metainspector#html-content-only"},"comment":{"url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/comments/173951291","html_url":"https://github.com/jaimeiniesta/metainspector/pull/169#issuecomment-173951291","issue_url":"https://api.github.com/repos/jaimeiniesta/metainspector/issues/169","id":173951291,"user":{"login":"jaimeiniesta","id":2629,"avatar_url":"https://avatars.githubusercontent.com/u/2629?v=3","gravatar_id":"","url":"https://api.github.com/users/jaimeiniesta","html_url":"https://github.com/jaimeiniesta","followers_url":"https://api.github.com/users/jaimeiniesta/followers","following_url":"https://api.github.com/users/jaimeiniesta/following{/other_user}","gists_url":"https://api.github.com/users/jaimeiniesta/gists{/gist_id}","starred_url":"https://api.github.com/users/jaimeiniesta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaimeiniesta/subscriptions","organizations_url":"https://api.github.com/users/jaimeiniesta/orgs","repos_url":"https://api.github.com/users/jaimeiniesta/repos","events_url":"https://api.github.com/users/jaimeiniesta/events{/privacy}","received_events_url":"https://api.github.com/users/jaimeiniesta/received_events","type":"User","site_admin":false},"created_at":"2016-01-22T15:31:00Z","updated_at":"2016-01-22T15:31:00Z","body":"Thanks for the PR, but as MetaInspector is for scraping HTML, we shouldn't try to scrape any other content. In my opinion, if we try to scrape an image, we should raise an exception, like this:\r\n\r\nhttps://metainspectordemo.herokuapp.com/scrape?url=https%3A%2F%2Fcnngps.files.wordpress.com%2F2011%2F12%2Fputin.jpg\r\n\r\nI think that's the best way to express the result, like, \"we attempted to parse this web page but, guess what, it's not a web page! It's an image!!!\" :)\r\n\r\nInstead of that, I think we should get totally rid of the `html_content_only` option. We should always fail if we try to scrape an image, pdf, mp3, or any other non-html content.\r\n\r\nThis `html_content_only` option is there because in the past (before 5.0.0) we were much more permissive with exceptions, we had an `ExceptionLog` where we would silently store all exceptions, but after 5.0.0. we prefer to raise them and have the user decide what to do with it. [Full conversation here](https://github.com/jaimeiniesta/metainspector/issues/154). We should have removed this option on 5.0.0, we didn't, but we can review this - it probably means bumping the version to 6.0.0 as it's a major change, though :disappointed: \r\n\r\n@trotzig what do you think? In the case of your application, I think you could catch this exception and notify the users if needed: you could not get the description or whatever, because it's not a web page, it's an image/pdf/mp3.\r\n\r\n "}}