{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/dynomite-manager/issues/9","repository_url":"https://api.github.com/repos/Netflix/dynomite-manager","labels_url":"https://api.github.com/repos/Netflix/dynomite-manager/issues/9/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/dynomite-manager/issues/9/comments","events_url":"https://api.github.com/repos/Netflix/dynomite-manager/issues/9/events","html_url":"https://github.com/Netflix/dynomite-manager/pull/9","id":162241262,"number":9,"title":"Dynomite-Manager working with Amazon VPC","user":{"login":"diegopacheco","id":121278,"avatar_url":"https://avatars.githubusercontent.com/u/121278?v=3","gravatar_id":"","url":"https://api.github.com/users/diegopacheco","html_url":"https://github.com/diegopacheco","followers_url":"https://api.github.com/users/diegopacheco/followers","following_url":"https://api.github.com/users/diegopacheco/following{/other_user}","gists_url":"https://api.github.com/users/diegopacheco/gists{/gist_id}","starred_url":"https://api.github.com/users/diegopacheco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diegopacheco/subscriptions","organizations_url":"https://api.github.com/users/diegopacheco/orgs","repos_url":"https://api.github.com/users/diegopacheco/repos","events_url":"https://api.github.com/users/diegopacheco/events{/privacy}","received_events_url":"https://api.github.com/users/diegopacheco/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Netflix/dynomite-manager/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"ipapapa","id":4562887,"avatar_url":"https://avatars.githubusercontent.com/u/4562887?v=3","gravatar_id":"","url":"https://api.github.com/users/ipapapa","html_url":"https://github.com/ipapapa","followers_url":"https://api.github.com/users/ipapapa/followers","following_url":"https://api.github.com/users/ipapapa/following{/other_user}","gists_url":"https://api.github.com/users/ipapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ipapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ipapapa/subscriptions","organizations_url":"https://api.github.com/users/ipapapa/orgs","repos_url":"https://api.github.com/users/ipapapa/repos","events_url":"https://api.github.com/users/ipapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ipapapa/received_events","type":"User","site_admin":false},"assignees":[{"login":"ipapapa","id":4562887,"avatar_url":"https://avatars.githubusercontent.com/u/4562887?v=3","gravatar_id":"","url":"https://api.github.com/users/ipapapa","html_url":"https://github.com/ipapapa","followers_url":"https://api.github.com/users/ipapapa/followers","following_url":"https://api.github.com/users/ipapapa/following{/other_user}","gists_url":"https://api.github.com/users/ipapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ipapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ipapapa/subscriptions","organizations_url":"https://api.github.com/users/ipapapa/orgs","repos_url":"https://api.github.com/users/ipapapa/repos","events_url":"https://api.github.com/users/ipapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ipapapa/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-06-24T22:20:02Z","updated_at":"2016-06-25T19:14:11Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Netflix/dynomite-manager/pulls/9","html_url":"https://github.com/Netflix/dynomite-manager/pull/9","diff_url":"https://github.com/Netflix/dynomite-manager/pull/9.diff","patch_url":"https://github.com/Netflix/dynomite-manager/pull/9.patch"},"body":"@ipapapa \r\n\r\nI made some very small changes to make Dynomite-Manager work on AWS VPC.\r\nI also added a scripts folder with some startup script for people who use CentOS / Amazon Linux.\r\n\r\nAs you can see if fully working for :-)\r\n```bash\r\n[ec2-user@ip-172-31-6-76 dynomite-manager-1]$ /etc/init.d/dynomite-manager start\r\nStarting Dynomite Manager...\r\n[ec2-user@ip-172-31-6-76 dynomite-manager-1]$ dmlog \r\nInferred project: dynomite-manager, version: 0.1.0-SNAPSHOT\r\nThe testJar task is deprecated.  Please place common test harness code in its own project and publish separately.\r\nPublication nebula not found in project :.\r\n[buildinfo] Not using buildInfo properties file for this build.\r\nPublication named 'nebula' does not exist for project ':' in task ':artifactoryPublish'.\r\n:dynomitemanager:compileJava UP-TO-DATE\r\n:dynomitemanager:processResources UP-TO-DATE\r\n:dynomitemanager:classes UP-TO-DATE\r\n:dynomitemanager:writeManifestProperties UP-TO-DATE\r\n:dynomitemanager:jar UP-TO-DATE\r\n:dynomitemanager-web:compileJava UP-TO-DATE\r\n:dynomitemanager-web:processResources UP-TO-DATE\r\n:dynomitemanager-web:classes UP-TO-DATE\r\n:dynomitemanager-web:jettyRunSLF4J: Class path contains multiple SLF4J bindings.\r\nSLF4J: Found binding in [jar:file:/home/ec2-user/.gradle/wrapper/dists/gradle-2.12-bin/avhnk0p45wmm16bas931at19r/gradle-2.12/lib/gradle-core-2.12.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: Found binding in [jar:file:/home/ec2-user/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-log4j12/1.7.2/7539c264413b9b1ff9841cd00058c974b7cd1ec9/slf4j-log4j12-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]\r\nSLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.\r\nSLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]\r\n\r\n2016-06-24 21:57:51 INFO  InjectedWebListener:112 - **Binding OSS Config classes.\r\n2016-06-24 21:57:51 WARN  URLConfigurationSource:120 - No URLs will be polled as dynamic configuration sources.\r\n2016-06-24 21:57:51 INFO  URLConfigurationSource:121 - To enable URLs as dynamic configuration sources, define System property archaius.configurationSource.additionalUrls or make config.properties available on classpath.\r\n2016-06-24 21:57:51 INFO  DynamicPropertyFactory:281 - DynamicPropertyFactory is initialized with configuration sources: com.netflix.config.ConcurrentCompositeConfiguration@1b897ffb\r\n2016-06-24 21:57:51 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/placement/availability-zone returns: us-west-2c\r\n2016-06-24 21:57:51 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/public-hostname returns: ec2-52-41-100-19.us-west-2.compute.amazonaws.com\r\n2016-06-24 21:57:51 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/public-ipv4 returns: 52.41.100.19\r\n2016-06-24 21:57:51 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/instance-id returns: i-5404bd89\r\n2016-06-24 21:57:51 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/instance-type returns: m4.large\r\n2016-06-24 21:57:51 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/network/interfaces/macs/ returns: 0a:54:fb:54:bc:31/\r\n2016-06-24 21:57:51 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/network/interfaces/macs/ returns: 0a:54:fb:54:bc:31/\r\n2016-06-24 21:57:51 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/network/interfaces/macs/0a:54:fb:54:bc:31/vpc-id returns: vpc-aeeb50cb\r\n2016-06-24 21:57:51 INFO  DynomitemanagerConfiguration:245 - vpc id for running instance: vpc-aeeb50cb\r\n2016-06-24 21:57:51 INFO  DynomitemanagerConfiguration:278 - Setting up Environmental Variables\r\n2016-06-24 21:57:51 INFO  DynomitemanagerConfiguration:286 - REGION set to us-west-2, ASG Name set to asg_dynomite\r\n2016-06-24 21:57:51 INFO  PropertiesConfigSource:83 - No Dynomitemanager.properties. Ignore!\r\n2016-06-24 21:57:52 INFO  PropertiesConfigSource:83 - No Dynomitemanager.properties. Ignore!\r\n2016-06-24 21:57:53 INFO  SimpleThreadPool:267 - Job execution threads will use class loader of thread: main\r\n2016-06-24 21:57:53 INFO  SchedulerSignalerImpl:60 - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl\r\n2016-06-24 21:57:53 INFO  QuartzScheduler:220 - Quartz Scheduler v.1.7.3 created.\r\n2016-06-24 21:57:53 INFO  RAMJobStore:139 - RAMJobStore initialized.\r\n2016-06-24 21:57:53 INFO  StdSchedulerFactory:1240 - Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'\r\n2016-06-24 21:57:53 INFO  StdSchedulerFactory:1244 - Quartz scheduler version: 1.7.3\r\n2016-06-24 21:57:53 INFO  QuartzScheduler:2075 - JobFactory set to: com.netflix.dynomitemanager.sidecore.scheduler.GuiceJobFactory@7314f4cb\r\n2016-06-24 21:57:53 INFO  InstanceDataDAOCassandra:351 - BOOT_CLUSTER = cass_dyno, KS_NAME = dyno_bootstrap\r\n2016-06-24 21:57:53 INFO  CountingConnectionPoolMonitor:194 - AddHost: 127.0.0.1\r\n2016-06-24 21:57:53 INFO  ConnectionPoolMBeanManager:53 - Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=MyConnectionPool,ServiceType=connectionpool\r\n2016-06-24 21:57:53 INFO  UpdateChecker:86 - New update(s) found: 1.8.5 [http://www.terracotta.org/kit/reflector?kitID=default&pageID=QuartzChangeLog]\r\n2016-06-24 21:57:54 INFO  AWSMembership:92 - Querying Amazon returned following instance in the ASG: asg_dynomite --> i-5404bd89,i-5504bd88,i-adff4770\r\n2016-06-24 21:58:07 INFO  InstanceIdentity:163 - Found dead instances: i-981ba245\r\n2016-06-24 21:58:07 INFO  InstanceIdentity:171 - Trying to grab slot 1383429731 with availability zone us-west-2c\r\n2016-06-24 21:58:07 INFO  InstanceDataDAOCassandra:125 - *** Creating New Instance Entry ***\r\n2016-06-24 21:58:07 INFO  InstanceDataDAOCassandra:130 - Key already exists: dynomite_demo1_asg_dynomite_1383429731\r\n2016-06-24 21:58:07 INFO  InstanceIdentity:136 - My token: 1383429731\r\n2016-06-24 21:58:08 INFO  FloridaServer:79 - Initializing Florida Server now ...\r\n2016-06-24 21:58:08 INFO  AWSMembership:269 - Fetch current permissions for vpc env of running instance\r\n2016-06-24 21:58:08 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/network/interfaces/macs/ returns: 0a:54:fb:54:bc:31/\r\n2016-06-24 21:58:08 INFO  SystemUtils:62 - Calling URL API: http://169.254.169.254/latest/meta-data/network/interfaces/macs/0a:54:fb:54:bc:31//security-groups returns: sg_asg_dynomite_florida\r\n2016-06-24 21:58:08 INFO  AWSMembership:154 - Done adding ACL to vpc: 50.112.72.203/32\r\n2016-06-24 21:58:08 INFO  FloridaServer:87 - Sleeping 130 seconds -> a node is replaced or token is pregenerated.\r\n2016-06-24 22:00:18 INFO  FloridaServer:103 - Running TuneTask and updating configuration.\r\n2016-06-24 22:00:18 INFO  FloridaStandardTuner:136 - dyn_o_mite:\r\n  dyn_listen: 0.0.0.0:8101\r\n  data_store: 0\r\n  listen: 0.0.0.0:8102\r\n  dyn_seed_provider: dynomitemanager_provider\r\n  servers:\r\n  - 127.0.0.1:22122:1\r\n  tokens: '1383429731'\r\n  auto_eject_hosts: true\r\n  rack: asg_dynomite\r\n  distribution: vnode\r\n  gos_interval: 10000\r\n  hash: murmur\r\n  preconnect: true\r\n  server_retry_timeout: 30000\r\n  timeout: 5000\r\n  secure_server_option: datacenter\r\n  datacenter: us-west-2\r\n  read_consistency: DC_ONE\r\n  write_consistency: DC_ONE\r\n  pem_key_file: /apps/dynomite/conf/dynomite.pem\r\n  dyn_seeds:\r\n  - ec2-50-112-72-203.us-west-2.compute.amazonaws.com:8101:asg_dynomite:us-west-2:1383429731\r\n\r\n2016-06-24 22:00:18 INFO  FloridaStandardTuner:250 - totalMem:8178632 Setting Redis storage max mem to 6081480\r\n2016-06-24 22:00:18 INFO  FloridaStandardTuner:164 - Updating Redis conf: /apps/nfredis/conf/redis.conf\r\n2016-06-24 22:00:18 INFO  FloridaServer:112 - Restore is disabled.\r\n2016-06-24 22:00:18 INFO  FloridaServer:121 - Cold bootstraping, launching dynomite and storage process.\r\n2016-06-24 22:00:18 INFO  FloridaProcessManager:74 - Starting dynomite server joinRing:true\r\n2016-06-24 22:00:23 INFO  FloridaProcessManager:96 - Dynomite server has been started\r\n2016-06-24 22:00:23 INFO  FloridaProcessManager:124 - std_out: root      2577     1  0 20:03 ?        00:00:00 /apps/dynomite/bin/dynomite -d -c /apps/dynomite/conf/dynomite.yml -m16384 -M200000 --output=/logs/system/dynomite/dynomite.log\r\nec2-user: Dynomite already running, no need to restart it again.\r\n\r\n2016-06-24 22:00:23 INFO  FloridaProcessManager:125 - std_err: \r\n2016-06-24 22:00:24 INFO  RedisStorageProxy:351 - Checking if Redis needs to be resetted to master\r\n2016-06-24 22:00:24 INFO  ProxyAndStorageResetTask:74 - Checking Dynomite's status\r\n2016-06-24 22:00:24 INFO  ProxyAndStorageResetTask:94 - Dynomite is up and running\r\n2016-06-24 22:00:24 INFO  FloridaServer:142 - Starting task scheduler\r\n```\r\n\r\nAll REST operations are working 100% for me with no errors!\r\n```bash\r\ncurl -v http://localhost:8080/dynomitemanager-web/REST/v1/admin/takesnapshot\r\ncurl -v http://localhost:8080/dynomitemanager-web/REST/v1/admin/start\r\ncurl -v http://localhost:8080/dynomitemanager-web/REST/v1/admin/stop\r\ncurl -v http://localhost:8080/dynomitemanager-web/REST/v1/admin/cluster_describe\r\ncurl -v http://localhost:8080/dynomitemanager-web/REST/v1/admin/startstorageprocess\r\ncurl -v http://localhost:8080/dynomitemanager-web/REST/v1/admin/stopstorageprocess\r\ncurl -v http://localhost:8080/dynomitemanager-web/REST/v1/admin/s3backup\r\ncurl -v http://localhost:8080/dynomitemanager-web/REST/v1/admin/s3restore\r\ncurl -v http://localhost:8080/dynomitemanager-web/REST/v1/admin/get_seeds\r\n```\r\n\r\nSome improvements for the documentation:\r\n\r\n1. You need make sure you expose this OS_ENV vars:\r\n```bash\r\nexport ASG_NAME=\"asg_dynomite\"\r\nexport AUTO_SCALE_GROUP=\"asg_dynomite\"\r\nexport EC2_REGION=\"us-west-2\"\r\n```\r\n\r\n2. Here is the dynomite.yml (located at: /apps/dynomite/conf/dynomite.yml) file I'm using it.\r\n```yaml\r\ndyn_listen: 0.0.0.0:8101\r\ndata_store: 0\r\nlisten: 0.0.0.0:8102\r\ndyn_seed_provider: dynomitemanager_provider\r\nservers:\r\n  - 127.0.0.1:22122:1\r\ntokens: '1383429731'\r\nauto_eject_hosts: true\r\nrack: null\r\ndistribution: vnode\r\ngos_interval: 10000\r\nhash: murmur\r\npreconnect: true\r\nserver_retry_timeout: 30000\r\ntimeout: 5000\r\nsecure_server_option: datacenter\r\ndatacenter: us-west-2\r\nread_consistency: DC_ONE\r\nwrite_consistency: DC_ONE\r\npem_key_file: /apps/dynomite/conf/dynomite.pem\r\n```\r\n\r\n3. I added this rules on my SG(Allow this ports) - but I'm not sure I need it because looks like DM create permissions for us :D  \r\n```bash\r\n22\r\n8080\r\n8000 - 8100\r\n8101 - 8102\r\n```\r\n\r\n4. For S3 you need crate a AMI ROLE(the same role you use for the Auto-Scale Groups and you need crate the bucket: dynomite-backup and make sure you add permission for the dynomte AMI role there. This also can be configure at: DynomitemanagerConfiguration.java on DEFAULT_BUCKET_NAME.\r\n\r\n5. In order to make Redis Backup work(snapshots and backup/restore) you need make sure this folder has write access for DM -> /mnt/data/ and also this DIR is configure on the redis conf file. \r\n\r\n6. IF you have issues with the S3Backup configuration beware of the Date Diff. You might not have a backup to restore :D \r\n\r\n7. You need make sure this folder exist and they have the right files and permissions\r\n```bash\r\n\r\nsudo mkdir -p /apps/dynomite/conf/\r\nsudo chmod -R 777 /apps/\r\nsudo mkdir -p /apps/dynomite/bin/\r\nsudo chmod -R 777 /apps/\r\ncp /home/ec2-user/dynomite/bin/kill_dynomite.sh /apps/dynomite/bin/\r\ncp /home/ec2-user/dynomite/bin/launch_dynomite.sh /apps/dynomite/bin/\r\nsudo mkdir -p /mnt/data/nfredis/\r\nsudo chmod -R 777 /mnt/data/\r\nsudo mkdir -p /logs/system/\r\nsudo chmod -R 777 /logs/\r\ncp ~/dynomite/src/dynomite /apps/dynomite/bin/\r\ncp ~/dynomite/src/*.* /apps/dynomite/bin/\r\nsudo cp ~/dynomite/conf/dynomite.pem /apps/dynomite/conf/dynomite.pem\r\nsudo mkdir -p /apps/nfredis/bin/\r\nsudo chmod -R 777 /apps/\r\n\r\nsudo touch /apps/nfredis/bin/launch_nfredis.sh\r\nsudo chmod +x /apps/nfredis/bin/launch_nfredis.sh\r\nsudo chmod +x /apps/nfredis/bin/kill_redis.sh\r\n\r\nChange [redis.conf] -> sudo vim /etc/redis/6379.conf and sudo vim /apps/nfredis/conf/redis.conf\r\n#dir /var/lib/redis/6379\r\ndir /mnt/data/nfredis/\r\n\r\nsudo chmod -R 777 /mnt/data/\r\nsudo mkdir /logs/system/dynomite-manager/\r\nsudo chmod -R 777 /logs/system/dynomite-manager/\r\nsudo chmod -R 777 /apps/\r\nsudo chmod -R 777 /mnt/data/\r\n\r\n```\r\n\r\nSorry If is too many information. I will do a blog post showing how to configure and run DM step-by-step. Thanks so much @ipapapa @timiblossom @ckalantzis Dynomite Manager and Dynomite are Rock Solid! You Guys Are AWESOME!\r\n\r\nCheers,\r\nDiego Pacheco\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/Netflix/dynomite-manager/issues/comments/228565787","html_url":"https://github.com/Netflix/dynomite-manager/pull/9#issuecomment-228565787","issue_url":"https://api.github.com/repos/Netflix/dynomite-manager/issues/9","id":228565787,"user":{"login":"diegopacheco","id":121278,"avatar_url":"https://avatars.githubusercontent.com/u/121278?v=3","gravatar_id":"","url":"https://api.github.com/users/diegopacheco","html_url":"https://github.com/diegopacheco","followers_url":"https://api.github.com/users/diegopacheco/followers","following_url":"https://api.github.com/users/diegopacheco/following{/other_user}","gists_url":"https://api.github.com/users/diegopacheco/gists{/gist_id}","starred_url":"https://api.github.com/users/diegopacheco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/diegopacheco/subscriptions","organizations_url":"https://api.github.com/users/diegopacheco/orgs","repos_url":"https://api.github.com/users/diegopacheco/repos","events_url":"https://api.github.com/users/diegopacheco/events{/privacy}","received_events_url":"https://api.github.com/users/diegopacheco/received_events","type":"User","site_admin":false},"created_at":"2016-06-25T19:14:11Z","updated_at":"2016-06-25T19:14:11Z","body":"Thanks @ipapapa I will check what you said by Monday and let you know. \r\n\r\nCheers,\r\nDiego Pacheco"}}
{"actor":{"display_login":"diegopacheco"}}