{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/3429","repository_url":"https://api.github.com/repos/elastic/logstash","labels_url":"https://api.github.com/repos/elastic/logstash/issues/3429/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/3429/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/3429/events","html_url":"https://github.com/elastic/logstash/issues/3429","id":87435981,"number":3429,"title":"JDBC/SQL Plugin Proposal","user":{"login":"talevy","id":388837,"avatar_url":"https://avatars.githubusercontent.com/u/388837?v=3","gravatar_id":"","url":"https://api.github.com/users/talevy","html_url":"https://github.com/talevy","followers_url":"https://api.github.com/users/talevy/followers","following_url":"https://api.github.com/users/talevy/following{/other_user}","gists_url":"https://api.github.com/users/talevy/gists{/gist_id}","starred_url":"https://api.github.com/users/talevy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/talevy/subscriptions","organizations_url":"https://api.github.com/users/talevy/orgs","repos_url":"https://api.github.com/users/talevy/repos","events_url":"https://api.github.com/users/talevy/events{/privacy}","received_events_url":"https://api.github.com/users/talevy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/logstash/labels/discuss","name":"discuss","color":"cc317c"},{"url":"https://api.github.com/repos/elastic/logstash/labels/new%20plugin","name":"new plugin","color":"bfd4f2"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":29,"created_at":"2015-06-11T18:15:32Z","updated_at":"2016-09-20T12:19:29Z","closed_at":"2015-07-16T17:38:05Z","body":"Link to Plugin: [here](https://github.com/logstash-plugins/logstash-input-jdbc)\r\n#### Purpose of the issue\r\n\r\nUnderstand any features or databases to provide support for with the jdbc plugin for logstash as \r\nthe community [jdbc-river](https://github.com/jprante/elasticsearch-jdbc) becomes deprecated.\r\n\r\n---\r\n\r\n### What is the purpose of logstash-input-jdbc?\r\n\r\n1. Generic plugin for supporting all databases as input which have jdbc connectors. To be leveraged \r\nfor creating specific plugins for databases (for example: [logstash-input-mysql](https://github.com/talevy/logstash-input-mysql)\r\n2. Integration point between Logstash and popular databases like MySQL, Postgres, Oracle, Mongodb\r\n\r\n### What does logstash-input-jdbc do?\r\n\r\nIt runs a sql query and generates input events for each resulting row. The option to schedule the query is also available.\r\n\r\n### Important Fields\r\n1. `statement`\r\n  - Any valid SQL statement will work.\r\n2. `parameters`\r\n   - You are able to specify parameter values for use in the sql statement using the `:<fieldname>` reference syntax\r\n3. `schedule`\r\n   - a valid cron schedule is accepted here for when to execute the statement.\r\n\r\n### Special Parameters\r\n\r\nFor now the only special parameter is `::sql_last_start` which is the time in which the last \r\nquery was executed. This can be used within the `statement` to limit the rows being processed to only new rows as queries are executed under their specified schedule.\r\n\r\n### How to Use\r\n\r\n#### Installation\r\n```\r\nbin/plugin install logstash-input-jdbc\r\n```\r\n\r\nI will show how this is used with `MySQL`.\r\n\r\ndownload jdbc driver library from [here](http://dev.mysql.com/downloads/connector/j/) then point to \r\nlibrary in logstash config\r\n```\r\n# logstash config\r\ninput {\r\n    jdbc {\r\n      \"jdbc_driver_class\" => \"com.mysql.jdbc.Driver\"\r\n      \"jdbc_driver_library\" => \"/path/to/mysql_jdbc_connector.jar\"\r\n      \"jdbc_connection_string\" => \"jdbc:mysql://localhost:3306/foodb\"\r\n      \"jdbc_user\" => \"myuser\"\r\n      \"jdbc_password\" => \"mypassword\"\r\n      \"statement\" => \"select * from foo where created_at > :sql_last_time\r\n      \"parameters\" => { hello => 4}\r\n      \"schedule\" => \"* * * * *\"\r\n    }\r\n}\r\n```\r\n\r\nFor many of the popular databases, we can write wrapper plugins on top of `logstash-input-jdbc` as was done for `logstash-input-mysql`, which would make configuration more straightforward.\r\n\r\ne.g.\r\n\r\n```\r\n# mysql logstash config\r\ninput {\r\n  mysql {\r\n      host => localhost,\r\n      port => 3306,\r\n      database => \"testdb\"\r\n      user => \"logstash\"\r\n      password => \"logstash\"\r\n      statement => \"SELECT * FROM foo where created_at >= :sql_last_start\"\r\n  }\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/248284746","html_url":"https://github.com/elastic/logstash/issues/3429#issuecomment-248284746","issue_url":"https://api.github.com/repos/elastic/logstash/issues/3429","id":248284746,"user":{"login":"pranmitt","id":9164647,"avatar_url":"https://avatars.githubusercontent.com/u/9164647?v=3","gravatar_id":"","url":"https://api.github.com/users/pranmitt","html_url":"https://github.com/pranmitt","followers_url":"https://api.github.com/users/pranmitt/followers","following_url":"https://api.github.com/users/pranmitt/following{/other_user}","gists_url":"https://api.github.com/users/pranmitt/gists{/gist_id}","starred_url":"https://api.github.com/users/pranmitt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pranmitt/subscriptions","organizations_url":"https://api.github.com/users/pranmitt/orgs","repos_url":"https://api.github.com/users/pranmitt/repos","events_url":"https://api.github.com/users/pranmitt/events{/privacy}","received_events_url":"https://api.github.com/users/pranmitt/received_events","type":"User","site_admin":false},"created_at":"2016-09-20T12:19:29Z","updated_at":"2016-09-20T12:19:29Z","body":"Is there any way we can apply logic over that data(means massage the sql output) before indexing into elasticsearch?"}}
{"actor":{"display_login":"pranmitt"}}