{"action":"created","issue":{"url":"https://api.github.com/repos/archivesunleashed/aut/issues/243","repository_url":"https://api.github.com/repos/archivesunleashed/aut","labels_url":"https://api.github.com/repos/archivesunleashed/aut/issues/243/labels{/name}","comments_url":"https://api.github.com/repos/archivesunleashed/aut/issues/243/comments","events_url":"https://api.github.com/repos/archivesunleashed/aut/issues/243/events","html_url":"https://github.com/archivesunleashed/aut/issues/243","id":343136152,"node_id":"MDU6SXNzdWUzNDMxMzYxNTI=","number":243,"title":"Replace hashing of unique ids with .zipWithUniqueId()","user":{"login":"greebie","id":111091,"node_id":"MDQ6VXNlcjExMTA5MQ==","avatar_url":"https://avatars0.githubusercontent.com/u/111091?v=4","gravatar_id":"","url":"https://api.github.com/users/greebie","html_url":"https://github.com/greebie","followers_url":"https://api.github.com/users/greebie/followers","following_url":"https://api.github.com/users/greebie/following{/other_user}","gists_url":"https://api.github.com/users/greebie/gists{/gist_id}","starred_url":"https://api.github.com/users/greebie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/greebie/subscriptions","organizations_url":"https://api.github.com/users/greebie/orgs","repos_url":"https://api.github.com/users/greebie/repos","events_url":"https://api.github.com/users/greebie/events{/privacy}","received_events_url":"https://api.github.com/users/greebie/received_events","type":"User","site_admin":false},"labels":[{"id":641287329,"node_id":"MDU6TGFiZWw2NDEyODczMjk=","url":"https://api.github.com/repos/archivesunleashed/aut/labels/enhancement","name":"enhancement","color":"31638e","default":true}],"state":"open","locked":false,"assignee":{"login":"greebie","id":111091,"node_id":"MDQ6VXNlcjExMTA5MQ==","avatar_url":"https://avatars0.githubusercontent.com/u/111091?v=4","gravatar_id":"","url":"https://api.github.com/users/greebie","html_url":"https://github.com/greebie","followers_url":"https://api.github.com/users/greebie/followers","following_url":"https://api.github.com/users/greebie/following{/other_user}","gists_url":"https://api.github.com/users/greebie/gists{/gist_id}","starred_url":"https://api.github.com/users/greebie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/greebie/subscriptions","organizations_url":"https://api.github.com/users/greebie/orgs","repos_url":"https://api.github.com/users/greebie/repos","events_url":"https://api.github.com/users/greebie/events{/privacy}","received_events_url":"https://api.github.com/users/greebie/received_events","type":"User","site_admin":false},"assignees":[{"login":"greebie","id":111091,"node_id":"MDQ6VXNlcjExMTA5MQ==","avatar_url":"https://avatars0.githubusercontent.com/u/111091?v=4","gravatar_id":"","url":"https://api.github.com/users/greebie","html_url":"https://github.com/greebie","followers_url":"https://api.github.com/users/greebie/followers","following_url":"https://api.github.com/users/greebie/following{/other_user}","gists_url":"https://api.github.com/users/greebie/gists{/gist_id}","starred_url":"https://api.github.com/users/greebie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/greebie/subscriptions","organizations_url":"https://api.github.com/users/greebie/orgs","repos_url":"https://api.github.com/users/greebie/repos","events_url":"https://api.github.com/users/greebie/events{/privacy}","received_events_url":"https://api.github.com/users/greebie/received_events","type":"User","site_admin":false}],"milestone":null,"comments":14,"created_at":"2018-07-20T14:47:32Z","updated_at":"2018-11-07T19:55:33Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"**Describe the Enhancement**\r\n\r\nAUT uses hash values to create unique ids, which can leave us duplicates of the same url in a network graph when hashes collide.\r\n\r\n**To Reproduce**\r\nSteps to reproduce the behavior (e.g.):\r\nRun a Domain Graph Extractor with a large number of network nodes (websites).\r\nRun in Gephi.\r\nDiscover duplicate websites in graph.\r\n\r\n**Expected behavior**\r\nAll network nodes should be unique.\r\n\r\n**Screenshots**\r\nN/A\r\n\r\n**Additional context**\r\nThe .zipWithIndex() feature in Apache Spark would be a better approach. http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.zipWithIndex\r\n\r\n.zipWithUniqueId() does not call another context so it could be faster.\r\n\r\nSee also #228 \r\n"},"comment":{"url":"https://api.github.com/repos/archivesunleashed/aut/issues/comments/436756319","html_url":"https://github.com/archivesunleashed/aut/issues/243#issuecomment-436756319","issue_url":"https://api.github.com/repos/archivesunleashed/aut/issues/243","id":436756319,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNjc1NjMxOQ==","user":{"login":"greebie","id":111091,"node_id":"MDQ6VXNlcjExMTA5MQ==","avatar_url":"https://avatars0.githubusercontent.com/u/111091?v=4","gravatar_id":"","url":"https://api.github.com/users/greebie","html_url":"https://github.com/greebie","followers_url":"https://api.github.com/users/greebie/followers","following_url":"https://api.github.com/users/greebie/following{/other_user}","gists_url":"https://api.github.com/users/greebie/gists{/gist_id}","starred_url":"https://api.github.com/users/greebie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/greebie/subscriptions","organizations_url":"https://api.github.com/users/greebie/orgs","repos_url":"https://api.github.com/users/greebie/repos","events_url":"https://api.github.com/users/greebie/events{/privacy}","received_events_url":"https://api.github.com/users/greebie/received_events","type":"User","site_admin":false},"created_at":"2018-11-07T19:55:33Z","updated_at":"2018-11-07T19:55:33Z","author_association":"CONTRIBUTOR","body":"@ruebot I realize I failed to answer your last questions.  The graphs produced by graphpass include metadata about how to display the graph in Gephi or Sigma (ie. how big should the dots be, what color and where they should be positioned in the visualisation).\r\n\r\nAuk-produced graphs, unfortunately, just provide the raw network data with no visualization metadata.\r\n\r\nCurrently, the best we have in auk right now is ExtractGraphX which produces metadata for node sizes and some things can offer a fair way to reduce the size of large graphs for visualization, but it would increase the amount of time it would take to produce derivatives for small graphs. When we accepted that udf into the repo, we decided it might be good for the toolkit, but it's not quite ready to help auk."}}
{"actor":{"display_login":"greebie"}}