{"action":"opened","issue":{"url":"https://api.github.com/repos/Beej126/BeejBlog/issues/1","repository_url":"https://api.github.com/repos/Beej126/BeejBlog","labels_url":"https://api.github.com/repos/Beej126/BeejBlog/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/Beej126/BeejBlog/issues/1/comments","events_url":"https://api.github.com/repos/Beej126/BeejBlog/issues/1/events","html_url":"https://github.com/Beej126/BeejBlog/issues/1","id":185840183,"number":1,"title":"Scrape Web page to API","user":{"login":"Beej126","id":6301228,"avatar_url":"https://avatars.githubusercontent.com/u/6301228?v=3","gravatar_id":"","url":"https://api.github.com/users/Beej126","html_url":"https://github.com/Beej126","followers_url":"https://api.github.com/users/Beej126/followers","following_url":"https://api.github.com/users/Beej126/following{/other_user}","gists_url":"https://api.github.com/users/Beej126/gists{/gist_id}","starred_url":"https://api.github.com/users/Beej126/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Beej126/subscriptions","organizations_url":"https://api.github.com/users/Beej126/orgs","repos_url":"https://api.github.com/users/Beej126/repos","events_url":"https://api.github.com/users/Beej126/events{/privacy}","received_events_url":"https://api.github.com/users/Beej126/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-10-28T05:11:43Z","updated_at":"2016-10-28T05:11:43Z","closed_at":null,"body":"\r\n## scraping GPS coords out of tracking device service provider since the don't provide a real API (hee hee)...\r\nthis handily boils it down to true API convenience ready to be leveraged behind our web and mobile clients\r\n``` C#\r\n// this is \"script\" C# syntax... \r\n// run interactively from command line like a dos batch file for trial n' error convenience (aka [REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop))...\r\n// but then also ready to be dropped into a real webApi project when working\r\n\r\n#r \"Newtonsoft.Json.9.0.1\\lib\\netstandard1.0\\Newtonsoft.Json.dll\"\r\n#load \"secrets.csx\" //load email/password, hidden in a file that doesn't get checked into source control\r\n#load \"WebClientEx.csx\" //custom WebClient that tracks cookies for Session/login support\r\n\r\nusing System;\r\nusing System.Collections.Specialized;\r\nusing System.Text.RegularExpressions;\r\nusing Newtonsoft.Json;\r\nusing Newtonsoft.Json.Linq;\r\n\r\npublic class Device { public int Id { get; set; } public string Name { get; set; } public double? Lat { get; set; } public double? Lng { get; set; } }\r\n\r\nvar client = new WebClientEx();\r\n\r\n//login - this registers their _session_id cookie in the customized WebClient instance above...\r\n//  *** so that subsequent requests for data are authorized ***\r\nvar fields = new NameValueCollection();\r\nfields.Add(\"user[email]\", email);\r\nfields.Add(\"user[password]\", password);\r\nfields.Add(\"subdomain\", subdomain);\r\nclient.UploadValues($\"{baseUrl}/users/sign_in\", fields);\r\nfields.Clear();\r\n\r\nvar resultCodeTables = client.DownloadString($\"{baseUrl}/mydash\");\r\n//in one fell swoop this RegEx sucks the device objects out of inline <script> in raw html\r\n//     ____  ____  ______  _____    __  __   __   __\r\n//    / __ )/ __ \\/ __ \\ \\/ /   |  / / / /  / /  / /\r\n//   / __  / / / / / / /\\  / /| | / /_/ /  / /  / / \r\n//  / /_/ / /_/ / /_/ / / / ___ |/ __  /  /_/  /_/  \r\n// /_____/\\____/\\____/ /_/_/  |_/_/ /_/  (_)  (_)   \r\n//(we also have Device-to-Division assignment data available here)\r\nvar devices = JsonConvert.DeserializeObject<List<Device>>($\"[{String.Join(\",\", Regex.Matches(resultCodeTables, @\"devices.push\\(({id.*?})\").Cast<Match>().Select(m => m.Groups[1].Value))}]\");\r\n\r\n//this is the actual \"API\" request for GPS data on each device\r\nvar devicesQueryString = string.Join(\"\", devices.Select(d => d.Id.ToString($\"&device[]={0}\")));\r\nvar resultLocations = client.UploadString($\"{baseUrl}/mydash/get_location?widget=365821{devicesQueryString}\",\"\");\r\nvar gps = JObject.Parse(resultLocations);\r\n\r\n//spit out the lat/lngs coords for each device!\r\nConsole.WriteLine(string.Join(\"\", gps.GetValue(\"locations\").Where(l => l[\"lat\"] != null).Select(l => $\"{l[\"id\"]}: {l[\"lat\"]}, {l[\"lng\"]}\\n\")));\r\n```\r\n\r\n## example output\r\n![image](https://cloud.githubusercontent.com/assets/6301228/19795389/ec38b3e0-9c8f-11e6-8df5-5f4842a3d964.png)\r\n"}}
{"actor":{"display_login":"Beej126"}}