{"action":"created","comment":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/comments/54032503","id":54032503,"diff_hunk":"@@ -0,0 +1,113 @@\n+-- Copyright (C) 2014-2015 LinkedIn Corp. All rights reserved.\n+--\n+-- Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use\n+-- this file except in compliance with the License. You may obtain a copy of the\n+-- License at  http://www.apache.org/licenses/LICENSE-2.0\n+--\n+-- Unless required by applicable law or agreed to in writing, software distributed\n+-- under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR\n+-- CONDITIONS OF ANY KIND, either express or implied.\n+\n+--\n+-- Version: 1.0.1\n+--\n+-- Helper stored procedures that enable idempotent schema modifications in MySql.\n+-- The following stored procedures are being added:\n+--   1. add_index_if_not_exists: check if the specified index exists on the\n+--                                  given table and create it if it does not.\n+--   2. drop_index_if_exists: check if the specified index exists on the\n+--                            given table and delete it if it does.\n+--   3. add_column_if_not_exists: check if the specified columns exists on the\n+--                                give table and create it if it does not.\n+--   4. modify_column_if_exists: check if the specified column exists on the\n+--                               given table and modifies it if it does.\n+--\n+\n+DELIMITER //","path":"gobblin-metastore/src/main/resources/db/migration/V1_0_1__Add_schema_modification_sprocs.sql","position":26,"original_position":26,"commit_id":"f6d8b5027cacce04e71b474dc2c7edebaa2ebbdf","original_commit_id":"f6d8b5027cacce04e71b474dc2c7edebaa2ebbdf","user":{"login":"eogren","id":5913279,"avatar_url":"https://avatars.githubusercontent.com/u/5913279?v=3","gravatar_id":"","url":"https://api.github.com/users/eogren","html_url":"https://github.com/eogren","followers_url":"https://api.github.com/users/eogren/followers","following_url":"https://api.github.com/users/eogren/following{/other_user}","gists_url":"https://api.github.com/users/eogren/gists{/gist_id}","starred_url":"https://api.github.com/users/eogren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eogren/subscriptions","organizations_url":"https://api.github.com/users/eogren/orgs","repos_url":"https://api.github.com/users/eogren/repos","events_url":"https://api.github.com/users/eogren/events{/privacy}","received_events_url":"https://api.github.com/users/eogren/received_events","type":"User","site_admin":false},"body":"Although more tedious, I'm wondering if this logic can be built into the actual migrations ourselves. We tend to not give most database users rights to create/drop stored procedures, so would have to run the migration as a user with broader powers. Could affect other users as well.","created_at":"2016-02-25T00:38:58Z","updated_at":"2016-02-25T00:38:58Z","html_url":"https://github.com/linkedin/gobblin/pull/705#discussion_r54032503","pull_request_url":"https://api.github.com/repos/linkedin/gobblin/pulls/705","_links":{"self":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/comments/54032503"},"html":{"href":"https://github.com/linkedin/gobblin/pull/705#discussion_r54032503"},"pull_request":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/705"}}},"pull_request":{"url":"https://api.github.com/repos/linkedin/gobblin/pulls/705","id":59211368,"html_url":"https://github.com/linkedin/gobblin/pull/705","diff_url":"https://github.com/linkedin/gobblin/pull/705.diff","patch_url":"https://github.com/linkedin/gobblin/pull/705.patch","issue_url":"https://api.github.com/repos/linkedin/gobblin/issues/705","number":705,"state":"open","locked":false,"title":"Improve database history store performance","user":{"login":"kadaan","id":1904898,"avatar_url":"https://avatars.githubusercontent.com/u/1904898?v=3","gravatar_id":"","url":"https://api.github.com/users/kadaan","html_url":"https://github.com/kadaan","followers_url":"https://api.github.com/users/kadaan/followers","following_url":"https://api.github.com/users/kadaan/following{/other_user}","gists_url":"https://api.github.com/users/kadaan/gists{/gist_id}","starred_url":"https://api.github.com/users/kadaan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kadaan/subscriptions","organizations_url":"https://api.github.com/users/kadaan/orgs","repos_url":"https://api.github.com/users/kadaan/repos","events_url":"https://api.github.com/users/kadaan/events{/privacy}","received_events_url":"https://api.github.com/users/kadaan/received_events","type":"User","site_admin":false},"body":"# Changes:\r\n\r\n1. Switched database scripts to use [flyway](https://flywaydb.org).  This enables us to keep track of migration scripts, ensure that users can upgrade, and verify that the correct version is installed.\r\n2. Changed tests from using derby to using embedded mysql.  This allows our tests to verify the same code, migrations, etc. as will be used in production.\r\n3. Added a check in `DatabaseJobHistoryStore` that verifies that the DB is the correct version.  If it is not, a warning will be logged and all operations on the store will become no-ops. This ensures that we don't break if users upgrade the code, but forget to upgrade the database.\r\n4. Changed travis build to have dependencies needed to run embedded mysql, split the jdk7/jdk8 build jobs, enabled caching.  This allows easier identification of problems, improves performance, etc.\r\n5. Changed `DatabaseJobHistoryStore` to use batching and upserts to greatly increase performance.  Also stopped sending in epoch for startTime/endTime which are not specified.  Lastly, removed job properties from task properties to reduce data suplication.  All this is to improve performance and data correctness.\r\n6. Changed the admin UI to show '-' for startTime/endTime when not set\r\n7. Added step at end of travis build to output test failures to console.  The facilitates the ability to determine why the build failed.\r\n8. Enabled support for compression in the JobExecutionInfoServer\r\n9. Reduced the amount of data requested by Admin UI\r\n10. Rewrote the `DatabaseJobHistoryStore` querying code to greatly reduce the number of queries\r\n\r\n# Job History Store Management\r\n**Migrate an unmanaged database (First Time)**\r\n```bash\r\n./historystore-manager.sh migrate -Durl=\"jdbc:mysql://<SERVER>/gobblin\" -Duser=\"<USER>\" -Dpassword=\"<PASSWORD>\" -DbaselineOnMigrate=true\r\n```\r\n\r\n**Migrate a database (Subsequent Time)**\r\n```bash\r\n./historystore-manager.sh migrate -Durl=\"jdbc:mysql://<SERVER>/gobblin\" -Duser=\"<USER>\" -Dpassword=\"<PASSWORD>\"\r\n```\r\n\r\n**Get database migration status**\r\n```bash\r\n./historystore-manager.sh info -Durl=\"jdbc:mysql://<SERVER>/gobblin\" -Duser=\"<USER>\" -Dpassword=\"<PASSWORD>\"\r\n```","created_at":"2016-02-12T18:25:07Z","updated_at":"2016-02-25T00:38:58Z","closed_at":null,"merged_at":null,"merge_commit_sha":"aad2e8f16f83b9af8cc9a769928e9ff2fad9a8e1","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/linkedin/gobblin/pulls/705/commits","review_comments_url":"https://api.github.com/repos/linkedin/gobblin/pulls/705/comments","review_comment_url":"https://api.github.com/repos/linkedin/gobblin/pulls/comments{/number}","comments_url":"https://api.github.com/repos/linkedin/gobblin/issues/705/comments","statuses_url":"https://api.github.com/repos/linkedin/gobblin/statuses/f6d8b5027cacce04e71b474dc2c7edebaa2ebbdf","head":{"label":"kadaan:ImproveDatabaseHistoryStorePerformance","ref":"ImproveDatabaseHistoryStorePerformance","sha":"f6d8b5027cacce04e71b474dc2c7edebaa2ebbdf","user":{"login":"kadaan","id":1904898,"avatar_url":"https://avatars.githubusercontent.com/u/1904898?v=3","gravatar_id":"","url":"https://api.github.com/users/kadaan","html_url":"https://github.com/kadaan","followers_url":"https://api.github.com/users/kadaan/followers","following_url":"https://api.github.com/users/kadaan/following{/other_user}","gists_url":"https://api.github.com/users/kadaan/gists{/gist_id}","starred_url":"https://api.github.com/users/kadaan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kadaan/subscriptions","organizations_url":"https://api.github.com/users/kadaan/orgs","repos_url":"https://api.github.com/users/kadaan/repos","events_url":"https://api.github.com/users/kadaan/events{/privacy}","received_events_url":"https://api.github.com/users/kadaan/received_events","type":"User","site_admin":false},"repo":{"id":47274294,"name":"gobblin","full_name":"kadaan/gobblin","owner":{"login":"kadaan","id":1904898,"avatar_url":"https://avatars.githubusercontent.com/u/1904898?v=3","gravatar_id":"","url":"https://api.github.com/users/kadaan","html_url":"https://github.com/kadaan","followers_url":"https://api.github.com/users/kadaan/followers","following_url":"https://api.github.com/users/kadaan/following{/other_user}","gists_url":"https://api.github.com/users/kadaan/gists{/gist_id}","starred_url":"https://api.github.com/users/kadaan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kadaan/subscriptions","organizations_url":"https://api.github.com/users/kadaan/orgs","repos_url":"https://api.github.com/users/kadaan/repos","events_url":"https://api.github.com/users/kadaan/events{/privacy}","received_events_url":"https://api.github.com/users/kadaan/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/kadaan/gobblin","description":"Universal data ingestion framework for Hadoop.","fork":true,"url":"https://api.github.com/repos/kadaan/gobblin","forks_url":"https://api.github.com/repos/kadaan/gobblin/forks","keys_url":"https://api.github.com/repos/kadaan/gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kadaan/gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kadaan/gobblin/teams","hooks_url":"https://api.github.com/repos/kadaan/gobblin/hooks","issue_events_url":"https://api.github.com/repos/kadaan/gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/kadaan/gobblin/events","assignees_url":"https://api.github.com/repos/kadaan/gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/kadaan/gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/kadaan/gobblin/tags","blobs_url":"https://api.github.com/repos/kadaan/gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kadaan/gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kadaan/gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/kadaan/gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kadaan/gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/kadaan/gobblin/languages","stargazers_url":"https://api.github.com/repos/kadaan/gobblin/stargazers","contributors_url":"https://api.github.com/repos/kadaan/gobblin/contributors","subscribers_url":"https://api.github.com/repos/kadaan/gobblin/subscribers","subscription_url":"https://api.github.com/repos/kadaan/gobblin/subscription","commits_url":"https://api.github.com/repos/kadaan/gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/kadaan/gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/kadaan/gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/kadaan/gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/kadaan/gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/kadaan/gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kadaan/gobblin/merges","archive_url":"https://api.github.com/repos/kadaan/gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kadaan/gobblin/downloads","issues_url":"https://api.github.com/repos/kadaan/gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/kadaan/gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/kadaan/gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/kadaan/gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kadaan/gobblin/labels{/name}","releases_url":"https://api.github.com/repos/kadaan/gobblin/releases{/id}","deployments_url":"https://api.github.com/repos/kadaan/gobblin/deployments","created_at":"2015-12-02T16:40:10Z","updated_at":"2016-01-11T23:41:37Z","pushed_at":"2016-02-24T19:38:40Z","git_url":"git://github.com/kadaan/gobblin.git","ssh_url":"git@github.com:kadaan/gobblin.git","clone_url":"https://github.com/kadaan/gobblin.git","svn_url":"https://github.com/kadaan/gobblin","homepage":"https://github.com/linkedin/gobblin/wiki","size":83189,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":3,"forks":2,"open_issues":3,"watchers":0,"default_branch":"master"}},"base":{"label":"linkedin:master","ref":"master","sha":"a35ad9517eef3280f97c6c0da56849f487087e0a","user":{"login":"linkedin","id":357098,"avatar_url":"https://avatars.githubusercontent.com/u/357098?v=3","gravatar_id":"","url":"https://api.github.com/users/linkedin","html_url":"https://github.com/linkedin","followers_url":"https://api.github.com/users/linkedin/followers","following_url":"https://api.github.com/users/linkedin/following{/other_user}","gists_url":"https://api.github.com/users/linkedin/gists{/gist_id}","starred_url":"https://api.github.com/users/linkedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linkedin/subscriptions","organizations_url":"https://api.github.com/users/linkedin/orgs","repos_url":"https://api.github.com/users/linkedin/repos","events_url":"https://api.github.com/users/linkedin/events{/privacy}","received_events_url":"https://api.github.com/users/linkedin/received_events","type":"Organization","site_admin":false},"repo":{"id":27392543,"name":"gobblin","full_name":"linkedin/gobblin","owner":{"login":"linkedin","id":357098,"avatar_url":"https://avatars.githubusercontent.com/u/357098?v=3","gravatar_id":"","url":"https://api.github.com/users/linkedin","html_url":"https://github.com/linkedin","followers_url":"https://api.github.com/users/linkedin/followers","following_url":"https://api.github.com/users/linkedin/following{/other_user}","gists_url":"https://api.github.com/users/linkedin/gists{/gist_id}","starred_url":"https://api.github.com/users/linkedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linkedin/subscriptions","organizations_url":"https://api.github.com/users/linkedin/orgs","repos_url":"https://api.github.com/users/linkedin/repos","events_url":"https://api.github.com/users/linkedin/events{/privacy}","received_events_url":"https://api.github.com/users/linkedin/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/linkedin/gobblin","description":"Universal data ingestion framework for Hadoop.","fork":false,"url":"https://api.github.com/repos/linkedin/gobblin","forks_url":"https://api.github.com/repos/linkedin/gobblin/forks","keys_url":"https://api.github.com/repos/linkedin/gobblin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/linkedin/gobblin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/linkedin/gobblin/teams","hooks_url":"https://api.github.com/repos/linkedin/gobblin/hooks","issue_events_url":"https://api.github.com/repos/linkedin/gobblin/issues/events{/number}","events_url":"https://api.github.com/repos/linkedin/gobblin/events","assignees_url":"https://api.github.com/repos/linkedin/gobblin/assignees{/user}","branches_url":"https://api.github.com/repos/linkedin/gobblin/branches{/branch}","tags_url":"https://api.github.com/repos/linkedin/gobblin/tags","blobs_url":"https://api.github.com/repos/linkedin/gobblin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/linkedin/gobblin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/linkedin/gobblin/git/refs{/sha}","trees_url":"https://api.github.com/repos/linkedin/gobblin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/linkedin/gobblin/statuses/{sha}","languages_url":"https://api.github.com/repos/linkedin/gobblin/languages","stargazers_url":"https://api.github.com/repos/linkedin/gobblin/stargazers","contributors_url":"https://api.github.com/repos/linkedin/gobblin/contributors","subscribers_url":"https://api.github.com/repos/linkedin/gobblin/subscribers","subscription_url":"https://api.github.com/repos/linkedin/gobblin/subscription","commits_url":"https://api.github.com/repos/linkedin/gobblin/commits{/sha}","git_commits_url":"https://api.github.com/repos/linkedin/gobblin/git/commits{/sha}","comments_url":"https://api.github.com/repos/linkedin/gobblin/comments{/number}","issue_comment_url":"https://api.github.com/repos/linkedin/gobblin/issues/comments{/number}","contents_url":"https://api.github.com/repos/linkedin/gobblin/contents/{+path}","compare_url":"https://api.github.com/repos/linkedin/gobblin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/linkedin/gobblin/merges","archive_url":"https://api.github.com/repos/linkedin/gobblin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/linkedin/gobblin/downloads","issues_url":"https://api.github.com/repos/linkedin/gobblin/issues{/number}","pulls_url":"https://api.github.com/repos/linkedin/gobblin/pulls{/number}","milestones_url":"https://api.github.com/repos/linkedin/gobblin/milestones{/number}","notifications_url":"https://api.github.com/repos/linkedin/gobblin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/linkedin/gobblin/labels{/name}","releases_url":"https://api.github.com/repos/linkedin/gobblin/releases{/id}","deployments_url":"https://api.github.com/repos/linkedin/gobblin/deployments","created_at":"2014-12-01T18:10:50Z","updated_at":"2016-02-23T18:31:12Z","pushed_at":"2016-02-25T00:35:03Z","git_url":"git://github.com/linkedin/gobblin.git","ssh_url":"git@github.com:linkedin/gobblin.git","clone_url":"https://github.com/linkedin/gobblin.git","svn_url":"https://github.com/linkedin/gobblin","homepage":"https://github.com/linkedin/gobblin/wiki","size":164087,"stargazers_count":445,"watchers_count":445,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":173,"mirror_url":null,"open_issues_count":59,"forks":173,"open_issues":59,"watchers":445,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/705"},"html":{"href":"https://github.com/linkedin/gobblin/pull/705"},"issue":{"href":"https://api.github.com/repos/linkedin/gobblin/issues/705"},"comments":{"href":"https://api.github.com/repos/linkedin/gobblin/issues/705/comments"},"review_comments":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/705/comments"},"review_comment":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/linkedin/gobblin/pulls/705/commits"},"statuses":{"href":"https://api.github.com/repos/linkedin/gobblin/statuses/f6d8b5027cacce04e71b474dc2c7edebaa2ebbdf"}}}}