{"action":"created","issue":{"url":"https://api.github.com/repos/lintool/warcbase/issues/232","repository_url":"https://api.github.com/repos/lintool/warcbase","labels_url":"https://api.github.com/repos/lintool/warcbase/issues/232/labels{/name}","comments_url":"https://api.github.com/repos/lintool/warcbase/issues/232/comments","events_url":"https://api.github.com/repos/lintool/warcbase/issues/232/events","html_url":"https://github.com/lintool/warcbase/issues/232","id":156553175,"number":232,"title":"Built-in Image URL building from wayback","user":{"login":"greebie","id":111091,"avatar_url":"https://avatars.githubusercontent.com/u/111091?v=3","gravatar_id":"","url":"https://api.github.com/users/greebie","html_url":"https://github.com/greebie","followers_url":"https://api.github.com/users/greebie/followers","following_url":"https://api.github.com/users/greebie/following{/other_user}","gists_url":"https://api.github.com/users/greebie/gists{/gist_id}","starred_url":"https://api.github.com/users/greebie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/greebie/subscriptions","organizations_url":"https://api.github.com/users/greebie/orgs","repos_url":"https://api.github.com/users/greebie/repos","events_url":"https://api.github.com/users/greebie/events{/privacy}","received_events_url":"https://api.github.com/users/greebie/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/lintool/warcbase/labels/feature","name":"feature","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-05-24T16:32:58Z","updated_at":"2016-07-26T20:06:59Z","closed_at":null,"body":"The image URL process is excellent.  I would like to suggest adding another object to Warcbase that gives you pre-processed urls to get the images from the wayback machine.   (Suggested code at bottom).  Basically, it works the same as the regular ExtractImageLinks, except there is one additional parameter for a date string.  Output is a Seq of urls that can be flatMapped and used to scrape images from wayback for research purposes.\r\n\r\nPossible additional development could be to include an image cloud for the collection (eg. counts determine image size).\r\n\r\nRyan. . ..\r\n\r\n\r\n```\r\npackage org.warcbase.spark.matchbox\r\nimport java.io.IOException\r\nimport org.jsoup.Jsoup\r\nimport org.jsoup.select.Elements\r\nimport scala.collection.mutable\r\n\r\nobject ExtractImageLinksWBFormatted {\r\n  /**\r\n    * @param src the src link.\r\n    * @param date a date for the content (eg. a crawl date). Way back will resolve truncated dates.\r\n    * @param html the content from which links are to be extracted.\r\n    *\r\n    * Returns a sequence of image links\r\n    */\r\n  def apply(src: String, date: String, html: String): Seq[String] = {\r\n    if (html.isEmpty || date.isEmpty) return Nil\r\n    try {\r\n\r\n      val output = mutable.MutableList[String]()\r\n      val doc = Jsoup.parse(html)\r\n      val links: Elements = doc.select(\"img[src]\")\r\n      val it = links.iterator()\r\n      while (it.hasNext) {\r\n        val link = it.next()\r\n        val prefix: String = \"http://wayback.archive-it.org/227/\"\r\n        val suffix: String = \"im_/\"\r\n        link.setBaseUri(src)\r\n        val target =  link.attr(\"abs:src\")\r\n        output += (prefix+date+suffix+target)\r\n      }\r\n      output\r\n    } catch {\r\n      case e: Exception =>\r\n        throw new IOException(\"Caught exception processing input \", e);\r\n    }\r\n  }\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/lintool/warcbase/issues/comments/235388513","html_url":"https://github.com/lintool/warcbase/issues/232#issuecomment-235388513","issue_url":"https://api.github.com/repos/lintool/warcbase/issues/232","id":235388513,"user":{"login":"greebie","id":111091,"avatar_url":"https://avatars.githubusercontent.com/u/111091?v=3","gravatar_id":"","url":"https://api.github.com/users/greebie","html_url":"https://github.com/greebie","followers_url":"https://api.github.com/users/greebie/followers","following_url":"https://api.github.com/users/greebie/following{/other_user}","gists_url":"https://api.github.com/users/greebie/gists{/gist_id}","starred_url":"https://api.github.com/users/greebie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/greebie/subscriptions","organizations_url":"https://api.github.com/users/greebie/orgs","repos_url":"https://api.github.com/users/greebie/repos","events_url":"https://api.github.com/users/greebie/events{/privacy}","received_events_url":"https://api.github.com/users/greebie/received_events","type":"User","site_admin":false},"created_at":"2016-07-26T20:06:59Z","updated_at":"2016-07-26T20:06:59Z","body":"Main purpose has been to run them through some kind of object detection api (eg. detect major politicians in cpp, male vs female etc.).  Watson has some ability to do some interesting things in that direction too.  I suppose integration with the api would be better than capturing the photos?  I agree it could be dangerous.  Grabbing the image links might be safer, if less complete."}}
{"actor":{"display_login":"greebie"}}