{"push_id":2327060221,"size":26,"distinct_size":12,"ref":"refs/heads/etl-master-process","head":"1e7685545274922d689a8cbec5e9cf450ba18f8f","before":"9169b8bed263fc7b589ea82855d9051211a8d81f","commits":[{"sha":"033690878bcb3fe155f954984527b250265aa423","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Fix loading JSON from Content Store with Job\n\nIt seems that in #482 the jobs that are created to retrieve the JSON\nis not the right one. ETL::Items was calling `ImportItemJob` when it \nshould be invoking `ImportContentDetailsJob`","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/033690878bcb3fe155f954984527b250265aa423"},{"sha":"d616db6e77828fc31953c97756e73761e48ccfb7","author":{"name":"Kyle MacPherson","email":"7010261b0a605d28a235abdd0a50bfb8600c261f@gmail.com"},"message":"Merge pull request #487 from alphagov/publishing-api-update\n\nFix loading JSON from Content Store with Job","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/d616db6e77828fc31953c97756e73761e48ccfb7"},{"sha":"2818a4bbb01cf551ae9539174b5995ca2f0df9c0","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Update schema.rb\n\nThe latest version of schema.rb is not matching the current state of\nthe database. This commit fixes this problem.","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/2818a4bbb01cf551ae9539174b5995ca2f0df9c0"},{"sha":"321bc693773bac470de9000c0e6ec65e3b2843cb","author":{"name":"Andrew Garner","email":"6b7c097d961f68685fc978cf91eeae17be6cbb45@andrewgarner.com"},"message":"Add a method to manufacture a date dimensions\n\nAdds a helper method that will find or manufacture a date dimension\nfor a given date.","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/321bc693773bac470de9000c0e6ec65e3b2843cb"},{"sha":"56906cb492a3cfd6a15f843ccc0b15a715cf793d","author":{"name":"Andrew Garner","email":"6b7c097d961f68685fc978cf91eeae17be6cbb45@andrewgarner.com"},"message":"Don't duplicate Google API constant","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/56906cb492a3cfd6a15f843ccc0b15a715cf793d"},{"sha":"b1a879a39223ed94a8b75567758ee695cc27ff83","author":{"name":"Andrew Garner","email":"6b7c097d961f68685fc978cf91eeae17be6cbb45@andrewgarner.com"},"message":"Add an ETL source for Google Analytics report data\n\nThis will fetch pageview and unique pageview metrics by page path\nfor a given day from the Google Analytics Reporting API.\n\nThe data is returned as an enumerable collection so that it can be\ntraversed, searched, filtered etc as part of a pipelines of steps.\n\nThe data returned is not filtered so some values may need to be\ndiscarded in a future pipeline step. This is the optimal way to\nuse the Reporting API as it produces a  computationally inexpensive\nquery.\n\nThe API client will automatically page through the response data\nreturned by the API. We fetch 10,000 rows of data in each query to\nthe API, this is the maximum supported and ensures we use our daily\nquota of API requests economically.","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/b1a879a39223ed94a8b75567758ee695cc27ff83"},{"sha":"9998df8d102a3d3427699f9599b43229231406e7","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Move into GA Service the extraction of metrics\n\nIn the same way that the Items extraction is isolated in the services\nlayer, I am following the same pattern here. I have added a method\nto return the metrics for a particular day in batches of 10_000.\n\nThe method should grow through time; we should add parameters there like\nthe metrics, so we don’t couple this class with the object that use them,\nbut as of today this is not a real need.","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/9998df8d102a3d3427699f9599b43229231406e7"},{"sha":"485f2238758576305cedde4b1ad0cbb478482f45","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Add `date` to the metrics result\n\nThe date is a core attribute of the results, and we should have self\ncontained information in each item returned by the GA service.\n\nWe are moving from\n\n```\n{\n  \"ga:pagePath\" => \"/path1\",\n  \"ga:pageviews\" => 1,\n  \"ga:uniquePageviews\" => 1,\n},\n```\n\nto\n\n```\n{\n  \"ga:pagePath\" => \"/path1\",\n  \"ga:pageviews\" => 1,\n  \"ga:uniquePageviews\" => 1,\n  \"date\" => \"2018-02-20\",\n},\n```","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/485f2238758576305cedde4b1ad0cbb478482f45"},{"sha":"a95b5915b327bc37e912c0d9ade036f23a314cf4","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Remove GA dependency from service\n\nWe don’t need to expose to all the application that the source of\ndata is Google Analytics. It is better to isolate the data from the\nsource.","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/a95b5915b327bc37e912c0d9ade036f23a314cf4"},{"sha":"9b84511605de05a4b9c86788eefe5a453fc2b836","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Store Google Analytics events for processing\n\nWe have changed the approach of pulling Google Analytics events. We now\ngo through all the visits for a particular day and fetch GA metrics for\neach path.\n\nThe Content Items base_path sometimes does not match the path on GA.\nThis happens with Guides, for example, so we won’t be able to find\ndirectly a match between a Content Item and an Event from GA.\n\nFor this reason, I have decided to store the list of events in the\ndatabase so we can do:\n\n1. A quick update of the content items with a matching path\n2. A batch processing of the links that do not match.","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/9b84511605de05a4b9c86788eefe5a453fc2b836"},{"sha":"168c7007e96bc257797a5052fb1c60ab49b1ce8d","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Load GA events in Facts Table\n\nI have split the process in two steps. The first one will import all\nthe events from GA in batches of 10_000 events. I have chosen this number\nbecause it matches the maximum number of events in each GA request.\nAll the events are stored in GA::Events table to be process in the second\nstep.\n\nOn the second step, in a single query, we update the facts metrics table.\nI have chosen to write custom SQL as the amount of records that we are\nhandling is very big (around 600K), and loading into AR for processing,\nis taking almost 2 hours with a single process. The time is not a problem\nas we don’t have performance requirements so far, but we should’t have\ntransactions that take so much time.\n\nBy running the update query we reduce this process to 7-8 seconds, so we\nhave a small transaction, and the database does all the heavy lifting,\nnot the application.","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/168c7007e96bc257797a5052fb1c60ab49b1ce8d"},{"sha":"eae8753777e85528026591ec7f0d00392dd22aba","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Delete GA events that has been copied to Facts table\n\nOnce the events has been copied to the facts table we no longer need them\nIn case we have to download them again, we just need to use the API.\n\nI would rather storing events only when there is no other way to\nget them.","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/eae8753777e85528026591ec7f0d00392dd22aba"},{"sha":"d75fca647695a13a7be68b67bcda3f754ad6236f","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Do not stub/mock the framework\n\nIt is not a good practice to mock the framework. We should only be testing\nthe code that we own, so changes in the framework do not affect the\napplication.\n\nIn this code, we are mocking Active Record, which is not recommended.","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/d75fca647695a13a7be68b67bcda3f754ad6236f"},{"sha":"d001b977c126a1a3506a4b77b65ce55f62832091","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Merge pull request #486 from alphagov/google-analytics-batch-import\n\nImport Google Analytics metrics in the Facts Table","distinct":false,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/d001b977c126a1a3506a4b77b65ce55f62832091"},{"sha":"320ce850fa1662bb35757c6f47d725aeb1ae1bc8","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Rename ETL process to `master`\n\nI am planning to use this process as the director to populate the facts\ntable. In this commit I am renaming it to `master` as a preparatory\nwork to incoming commits.","distinct":true,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/320ce850fa1662bb35757c6f47d725aeb1ae1bc8"},{"sha":"e0865404e5b7592cbdd080e9d174ca8741b01665","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Prevent Facts Table from being deleted in each process\n\nWe no longer want to delete the facts table when importing. If we need\nto delete the table we will do it manually (this would be a task that is\nout of the normal procedure)","distinct":true,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/e0865404e5b7592cbdd080e9d174ca8741b01665"},{"sha":"626a66e6dfa9e767276f1555b0fc9f2e09c90ad5","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Remove tests that depends on Organisations\n\nWe have already removed the Dimensions for Organisations, so this tests\nshould have been removed as well.","distinct":true,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/626a66e6dfa9e767276f1555b0fc9f2e09c90ad5"},{"sha":"d663a5f9efe377b88dc1f42cf6db7ffe038be646","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Remove ETL::Items dependency\n\nWe no longer populates the Content Items on a daily basis, as we are\ngoing to grow the Content Items Dimensions via a pub/sub patter with\nthe publishing-api.","distinct":true,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/d663a5f9efe377b88dc1f42cf6db7ffe038be646"},{"sha":"9e6e32512d62649cb0c39036152022960fbad578","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Rename to reveal intention: initialize_facts_table\n\n`create_metrics` does not represent what this process is doing. We are\npopulating the Facts table with a place holder to later on store the\nmetrics.","distinct":true,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/9e6e32512d62649cb0c39036152022960fbad578"},{"sha":"ab7902eddc8033427ed2ad3b828ca54b71ec981b","author":{"name":"Pablo Manrubia","email":"0135dcad04957dc37e942d031bc793dd210bbc84@gmail.com"},"message":"Remove unused parameter: `organisation_id`","distinct":true,"url":"https://api.github.com/repos/alphagov/content-performance-manager/commits/ab7902eddc8033427ed2ad3b828ca54b71ec981b"}]}
{"actor":{"display_login":"pmanrubia"}}