{"action":"created","issue":{"url":"https://api.github.com/repos/juditacs/wordcount/issues/73","repository_url":"https://api.github.com/repos/juditacs/wordcount","labels_url":"https://api.github.com/repos/juditacs/wordcount/issues/73/labels{/name}","comments_url":"https://api.github.com/repos/juditacs/wordcount/issues/73/comments","events_url":"https://api.github.com/repos/juditacs/wordcount/issues/73/events","html_url":"https://github.com/juditacs/wordcount/issues/73","id":145219945,"number":73,"title":"AWS test environment, thoughts?","user":{"login":"svanoort","id":5400948,"avatar_url":"https://avatars.githubusercontent.com/u/5400948?v=3","gravatar_id":"","url":"https://api.github.com/users/svanoort","html_url":"https://github.com/svanoort","followers_url":"https://api.github.com/users/svanoort/followers","following_url":"https://api.github.com/users/svanoort/following{/other_user}","gists_url":"https://api.github.com/users/svanoort/gists{/gist_id}","starred_url":"https://api.github.com/users/svanoort/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/svanoort/subscriptions","organizations_url":"https://api.github.com/users/svanoort/orgs","repos_url":"https://api.github.com/users/svanoort/repos","events_url":"https://api.github.com/users/svanoort/events{/privacy}","received_events_url":"https://api.github.com/users/svanoort/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-04-01T16:08:56Z","updated_at":"2016-04-05T18:28:48Z","closed_at":null,"body":"I am looking at setting up an AWS environment (spun up on demand only) that will run tests in a fast and automated fashion, using my personal Jenkins host to trigger it when commits are pushed.  \r\n\r\n**Work progress:**\r\n- [x] Create an r3.large instance with ephemeral storage and assigned benchmarking-specific IAM role\r\n- [x] Create setup script that installs docker, xz, git, starts docker, and pulls the ubuntu-16 allthelanguages docker image\r\n- [x] Create and attach policy to IAM role for benchmarking that allows read of the dataset S3 bucket, and write of the results bucket\r\n- [x] Compress huwiki, huwikisource, cleaned huwiki with xz -9 (smallest size) and upload to new S3 buckets (data is a private bucket, benchmark results is initially private, but later public). \r\n- [x] Add script commands to setup script that will download data from S3 and decompress it\r\n- [x] Set the AWS host to use ephemeral (instance) storage for /tmp folder\r\n- [x] Run benchmark using docker\r\n- [x] Upload first result to S3 - [available here](http://codeablereason-benchmark-results.s3-website-us-east-1.amazonaws.com/clean_results.txt)\r\n- [x] Create scripting to grab instance + package info to metadata file\r\n  - Git hash used in build\r\n  - Timestamp\r\n  - Host type, from aws cli\r\n  - Hash of input file\r\n- [ ] Timeouts and resource limits on individual runs (Node.js for example hung on the instance, and needed to be manually killed, another one ran out of RAM and broke the Docker session)\r\n- [ ] Create scripting to name results by run/host info individually\r\n- [ ] Jenkins - role or similar to allow control of benchmarking host?\r\n- [ ] Jenkins - job to fire benchmarks (github triggering)\r\n**Hardware/specs:**\r\n\r\n- Storage: use SSD instance storage to benchmark (limits instance types).  General purpose EBS SSD storage is generally slower and would run out of I/O credits after 1/2 hour (benchmarks need several hours). \r\n- Memory:  either 7 GB (small datasets or where memory is not needed) or 15 GB (large or high-memory datasets).\r\n- CPUs: 2 or 4 core. \r\n- Instance types: m3.large (2-core, 7. 5GB RAM) for the small datasets, and r3.large (2-core, 15.25 GB ram) for big.  If we do lots of parallelized implementations, add m3.xlarge (4-core, 15 GB RAM). \r\n- Cost:  I am not spending more than $10-15/month on it, beyond my existing Jenkins host (reserved t2.micro) and domain/S3 hosting.  Instances will be created to run a set of benchmarks and then terminated, with frequency to keep costs within limits.\r\n\r\n**Architecture:**\r\n\r\n- Instances are spun up by my Jenkins host, with an appropriate IAM role or credentials to do this in a limited way. \r\n- Benchmark datasets will be self-hosted to not hit their sources hard.  They won't be fully public unless small.\r\n- Instance gets an IAM role that allows uploading to a public (?) S3 results bucket.\r\n- Instance runs benchmarks on instance storage\r\n- Instance will upload each result to the S3 bucket as it completes, stamped with the git commit hash, timestamp run, language, etc. \r\n- All testing will use a reasonable timeout for both individual tests and the whole test set, if it hangs it is killed or skipped.\r\n- All testing uses the docker image, for reproducibility across hardware.\r\n\r\n**Two options for how to set it up:**\r\n\r\n- EBS based & on-demand instaces:  \r\n    - Use an EBS volume containing benchmark data and preconfigured system, and just start/stop the instance.  \r\n    - When run, the git repo is cloned, the dataset is copied to the data folder, and tests are run & uploaded.  \r\n    - Easier to set up and run, but more expensive.\r\n- S3 based/spot instances:  \r\n    - cheaper (1/4 the instance price) but more maintenance.  \r\n    - Submit spot bids, instances are configured using the \"user data\" field to submit a startup script which sets up and runs benchmarks.\r\n    - Private S3 buckets host compressed corpus data, these are fetched and decompressed. \r\n\r\n**Open questions:**\r\n\r\n- What to use for controlling instances?   \r\n   - AWS CLI is easy\r\n   - Jenkins AWS EC2 plugin will spin jenkins agents in EC2 (far easier to generate and report results from this), but comes with performance overheads\r\n   - Ansible is kind of amazing and easy to work with\r\n \r\nYesterday I had good results tinkering with a spot-purchased c3.large instance for benchmarking, doing all I/O to the /media/ephemeral0 instance store.    Pricing was only  about $0.04/hour for the spot buy (purchased at 2x current spot price to prevent it being terminated after exceeding the price).  "},"comment":{"url":"https://api.github.com/repos/juditacs/wordcount/issues/comments/205933397","html_url":"https://github.com/juditacs/wordcount/issues/73#issuecomment-205933397","issue_url":"https://api.github.com/repos/juditacs/wordcount/issues/73","id":205933397,"user":{"login":"juditacs","id":1611053,"avatar_url":"https://avatars.githubusercontent.com/u/1611053?v=3","gravatar_id":"","url":"https://api.github.com/users/juditacs","html_url":"https://github.com/juditacs","followers_url":"https://api.github.com/users/juditacs/followers","following_url":"https://api.github.com/users/juditacs/following{/other_user}","gists_url":"https://api.github.com/users/juditacs/gists{/gist_id}","starred_url":"https://api.github.com/users/juditacs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/juditacs/subscriptions","organizations_url":"https://api.github.com/users/juditacs/orgs","repos_url":"https://api.github.com/users/juditacs/repos","events_url":"https://api.github.com/users/juditacs/events{/privacy}","received_events_url":"https://api.github.com/users/juditacs/received_events","type":"User","site_admin":false},"created_at":"2016-04-05T18:28:48Z","updated_at":"2016-04-05T18:28:48Z","body":"> Perhaps if we added a stipulation that multibyte encodings of normally single-byte characters must be converted to a single byte for counting, that would remove the ability to just use raw bytes and look at if they're above a certain value?\r\n\r\nWhat do you mean by 'normally single-byte characters'? Codepoints under 256?\r\nCharacter counting in utf-8 is pretty easy, the first byte of a characters tells how many mores come."}}