{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/osquery/issues/4735","repository_url":"https://api.github.com/repos/facebook/osquery","labels_url":"https://api.github.com/repos/facebook/osquery/issues/4735/labels{/name}","comments_url":"https://api.github.com/repos/facebook/osquery/issues/4735/comments","events_url":"https://api.github.com/repos/facebook/osquery/issues/4735/events","html_url":"https://github.com/facebook/osquery/issues/4735","id":342797173,"node_id":"MDU6SXNzdWUzNDI3OTcxNzM=","number":4735,"title":"Kinesis appears non-functioning in osquery 3.2.8","user":{"login":"rustybrooks","id":8725156,"node_id":"MDQ6VXNlcjg3MjUxNTY=","avatar_url":"https://avatars2.githubusercontent.com/u/8725156?v=4","gravatar_id":"","url":"https://api.github.com/users/rustybrooks","html_url":"https://github.com/rustybrooks","followers_url":"https://api.github.com/users/rustybrooks/followers","following_url":"https://api.github.com/users/rustybrooks/following{/other_user}","gists_url":"https://api.github.com/users/rustybrooks/gists{/gist_id}","starred_url":"https://api.github.com/users/rustybrooks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rustybrooks/subscriptions","organizations_url":"https://api.github.com/users/rustybrooks/orgs","repos_url":"https://api.github.com/users/rustybrooks/repos","events_url":"https://api.github.com/users/rustybrooks/events{/privacy}","received_events_url":"https://api.github.com/users/rustybrooks/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-07-19T16:15:06Z","updated_at":"2018-09-20T16:16:29Z","closed_at":null,"author_association":"NONE","body":"### What version of `osquery` are you using? What operating system. Be specific.\r\n\r\nAll of my testing has been on ubuntu14.04.  I'm compiling from scratch because I'm trying to use 3.2.8, which is not available as a stable release yet.\r\n\r\nThe specific version(s) are outlined below - essentially I tried 3.2.6 (which worked) and 3.2.8 (which didn't) and a lot of git revisions in between.\r\n\r\n### What have you already tried?\r\n\r\nI have a simple osquery config that I have used to test whether osquery can send to kinesis (see below).  I first verified that it worked up to 3.2.6 and failed in 3.2.8, and then started a git bisect between the two.  I finally nailed it down to this commit:\r\nhttps://github.com/facebook/osquery/commit/9e98cefd22d3399582b016d484506ff110cd33a0\r\nUnsurprisingly, the commit message is\r\n\r\n\"deps: Update aws-sdk-cpp to version 1.4.55 (#4437)\"\r\n\r\nIf you look at this change, it's only changing some tests, and updating the AWS SDK dependency.  Apparently, something in this upgrade has broken kinesis.\r\n\r\n### What did you expect to see?\r\n\r\nWhen using my test config, I have a scheduled query running every few seconds, and it's configured to log via kinesis.  When it succeeds, you'll see messages like\r\n\r\n```\r\n I0719 01:43:55.058900  1567 aws_log_forwarder.h:222] aws_kinesis: Successfully sent 3 out of 3 log records\r\n```\r\n\r\n### What did you see instead?\r\n\r\nIn versions where kinesis is non functional, instead, I will see this\r\n\r\n```\r\nE0719 12:02:17.132575  1559 aws_log_forwarder.h:247] aws_kinesis: Complete request failure: Unable to connect to endpoint\r\n```\r\n\r\nIt takes 10 seconds or so to start seeing these error messages - I assume it's exhasting a number of retries before complaining\r\n\r\n\r\nHere is the script I use to run the test:\r\n\r\n```\r\n#!/bin/bash\r\n\r\nmkdir -p /var/log/osquery\r\n\r\ntouch /tmp/flagfile\r\n\r\n/srv/osquery/build/xenial/osquery/osqueryi \\\r\n    --tls_server_certs=/tmp/osquery_packaging/prefix/usr/share/osquery/certs/certs.pem \\\r\n    --disable-enrollment \\\r\n    --flagfile=/tmp/flagfile \\\r\n    --config_path=/srv/patch/osquery.conf \\\r\n    --watchdog_memory_limit=350 \\\r\n    --verbose \\\r\n    --logger_path=/tmp\r\n```\r\n\r\nAnd here is the config file I use\r\n\r\n```\r\n\r\n{\r\n    \"schedule\": {\r\n        \"test\": {\r\n            \"interval\": 2,\r\n            \"query\": \"SELECT strftime('%s','now')\"\r\n        }\r\n    },\r\n    \"options\": {\r\n        \"logger_path\": \"/var/log/osquery/\",\r\n        \"disable_logging\": false,\r\n        \"aws_access_key_id\": \"XXX\",\r\n        \"logger_event_type\": true,\r\n        \"aws_region\": \"us-east-1\",\r\n        \"aws_kinesis_stream\": \"my-stream-name\",\r\n        \"enable_monitor\": true,\r\n        \"aws_kinesis_random_partition_key\": true,\r\n        \"logger_plugin\": \"aws_kinesis,filesystem\",\r\n        \"logger_min_status\": 1,\r\n        \"aws_secret_access_key\": \"YYY\",\r\n        \"logger_snapshot_event_type\": true\r\n    }\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/facebook/osquery/issues/comments/423243932","html_url":"https://github.com/facebook/osquery/issues/4735#issuecomment-423243932","issue_url":"https://api.github.com/repos/facebook/osquery/issues/4735","id":423243932,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzI0MzkzMg==","user":{"login":"rustybrooks","id":8725156,"node_id":"MDQ6VXNlcjg3MjUxNTY=","avatar_url":"https://avatars2.githubusercontent.com/u/8725156?v=4","gravatar_id":"","url":"https://api.github.com/users/rustybrooks","html_url":"https://github.com/rustybrooks","followers_url":"https://api.github.com/users/rustybrooks/followers","following_url":"https://api.github.com/users/rustybrooks/following{/other_user}","gists_url":"https://api.github.com/users/rustybrooks/gists{/gist_id}","starred_url":"https://api.github.com/users/rustybrooks/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rustybrooks/subscriptions","organizations_url":"https://api.github.com/users/rustybrooks/orgs","repos_url":"https://api.github.com/users/rustybrooks/repos","events_url":"https://api.github.com/users/rustybrooks/events{/privacy}","received_events_url":"https://api.github.com/users/rustybrooks/received_events","type":"User","site_admin":false},"created_at":"2018-09-20T16:16:29Z","updated_at":"2018-09-20T16:16:29Z","author_association":"NONE","body":"I believe it's this commit:\r\nhttps://github.com/facebook/osquery/commit/9e98cefd22d3399582b016d484506ff110cd33a0\r\n\r\nUndoing the change to aws-sdk-cpp.rb \"fixed\" the problem for me, but obviously that means I'd still be  using the older AWS SDK library.\r\n\r\nI'm not currently using a 3.x version, because of this problem and some others (such as, syslog doesn't work on linux now), so I couldn't actually say if reverting the AWS version upgrade would cause other problems."}}
{"actor":{"display_login":"rustybrooks"}}