{"action":"created","comment":{"url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/comments/53693826","id":53693826,"diff_hunk":"@@ -0,0 +1,131 @@\n+# Whole Test Suite Generation\n+**Article:** *G. Fraser and A. Arcuri.* Whole Test Suite Generation. IEEE Transactions on Software Engineering, vol. 39, iss. 2, pp. 276-291, 2013.\n+\n+## Research Goal\n+Software testing is an essential component of software development. \n+Over the years, smart programmers have designed techniques that automatically produce test cases, and we can now generate entire test suites with high coverage. \n+However, one problem remains, known as the *oracle problem*. \n+Automatic oracles that assert what the outcome of a test should be are possible when it comes to explicit events such as 'the program should not crash'. \n+But when dealing with more complex outcomes, human intervention is necessary to specify oracles.\n+For this to be feasible, test generation must aim not only at high code coverage, but also at small test suites that make manual oracle generation as easy as possible. \n+The developer who needs to specify an oracle does not have to go through thousands of lines of code to manually add assertions, but through a set limit for the generation. \n+The goal of this paper is to therefore make such technology possible by achieving the following:\n+- High code coverage\n+- Small yet representative test sets\n+- Whole test suite generation\n+\n+This technology was made available through the EvoSuite framework. \n+\n+## Background Information\n+To generate effective test cases, the guiding medium used most commonly is the code coverage criterion, which represents a finite set of coverage goals (e.g. branch coverage, path coverage, etc). \n+Many techniques (such as the one mentioned in the previous papers) are already being used by software engineers, but they all have certain drawbacks.  \n+One example of such drawback is that when applying the work mentioned in Paper 2, handling the growth/reduction of generated test cases requires quite a bit of effort. \n+Additionally, there is mention of a technique that also takes into account collateral coverage, which represents the amount of collateral targets that are accidentally covered. \n+However, no particular heuristic exists to help cover these targets.  \n+The approaches mentioned so far all target a single test goal at a time, which is the predominant method. \n+But there is one that uses a single sequence of function calls to maximize the number of covered branches while watching out for the length of such a test case. \n+A drawback of such approach is that there can be conflicting testing goals and it might be impossible to cover all of them with a single test sequence, regardless of its length.  \n+Another method proposed optimizes entire test suites with a search algorithm with respect to mutation analysis. \n+However, the strong limitation is having to manually choose and fix the length of the test cases, which does not evolve during the search.  \n+To deal with these drawbacks, the paper therefore approaches a new method with the intent to satisfy the chosen coverage criterion with the smallest possible test suite, which makes use of a search algorithm applied on a population of test suites. \n+## Test Suite Optimization\n+To execute whole test suite generation, the paper describes the use of a Genetic Algorithm (GA) as search algorithm, on a population of test suites. It also focuses on *branch coverage* as test criterion. \n+\n+### Genetic Algorithm\n+This algorithm starts with a random population of test suites, and evolves this population until a solution is found that fulfills the coverage criterion, or the set resources such as time or number of fitness evaluations have been used up. \n+For each generation, the new population is initialised with the best individuals of the last generation. \n+The new generation is then filled up with individuals produced by rank selection, crossover, and mutation. \n+Either the offspring or the parents are added to the new generation, depending on fitness and length constraints.   \n+\n+### Representation\n+To apply this search algorithm, the problem needs to be defined, and a solution for that problem needs to be represented accurately. \n+The paper declares the solution to be a *test suite*, which is represented as a set of test cases. \n+A test case is represented by a sequence of statements, and the length of the test suite is defined as the sum of the lengths of its test cases. \n+Test cases contain statements, and each statement has a value and a type. This paper identifies five types of statements:\n+- **Primitive** statements, which represent Boolean, String, and enumeration variables, but also arrays of any type.\n+- **Constructor** statements, which generate new instances of any given class.\n+- **Field** statements, which access public member variables of objects. \n+- **Method** statements, which invoke methods on objects or call static methods.\n+- **Assignment** statements, which assign values to array indices or to public member variables of objects, but do not define new values.\n+\n+The *test cluster* for a given system under test defines the set of available classes, their public constructors, methods, and fields. \n+This chosen representation has a variable size for the test suite for one good reason: for a new software to test, the optimal number of test cases and their respective optimal length are not known, and need to be searched for. ","path":"test-suite-genetation.md","position":52,"original_position":52,"commit_id":"a3aeaa6bbec2ead8205db73d32a060373167cec8","original_commit_id":"a3aeaa6bbec2ead8205db73d32a060373167cec8","user":{"login":"ataneva","id":14241662,"avatar_url":"https://avatars.githubusercontent.com/u/14241662?v=3","gravatar_id":"","url":"https://api.github.com/users/ataneva","html_url":"https://github.com/ataneva","followers_url":"https://api.github.com/users/ataneva/followers","following_url":"https://api.github.com/users/ataneva/following{/other_user}","gists_url":"https://api.github.com/users/ataneva/gists{/gist_id}","starred_url":"https://api.github.com/users/ataneva/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ataneva/subscriptions","organizations_url":"https://api.github.com/users/ataneva/orgs","repos_url":"https://api.github.com/users/ataneva/repos","events_url":"https://api.github.com/users/ataneva/events{/privacy}","received_events_url":"https://api.github.com/users/ataneva/received_events","type":"User","site_admin":false},"body":"statements, and each  - no comma\r\nAnd better to put a stop here ...to test. The optimal","created_at":"2016-02-22T21:14:01Z","updated_at":"2016-02-22T21:14:01Z","html_url":"https://github.com/TUDelft-CS4110/2016-0x90/pull/4#discussion_r53693826","pull_request_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/4","_links":{"self":{"href":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/comments/53693826"},"html":{"href":"https://github.com/TUDelft-CS4110/2016-0x90/pull/4#discussion_r53693826"},"pull_request":{"href":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/4"}}},"pull_request":{"url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/4","id":60078758,"html_url":"https://github.com/TUDelft-CS4110/2016-0x90/pull/4","diff_url":"https://github.com/TUDelft-CS4110/2016-0x90/pull/4.diff","patch_url":"https://github.com/TUDelft-CS4110/2016-0x90/pull/4.patch","issue_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues/4","number":4,"state":"open","locked":false,"title":"Whole Test Suite Generation","user":{"login":"ValMai","id":13940591,"avatar_url":"https://avatars.githubusercontent.com/u/13940591?v=3","gravatar_id":"","url":"https://api.github.com/users/ValMai","html_url":"https://github.com/ValMai","followers_url":"https://api.github.com/users/ValMai/followers","following_url":"https://api.github.com/users/ValMai/following{/other_user}","gists_url":"https://api.github.com/users/ValMai/gists{/gist_id}","starred_url":"https://api.github.com/users/ValMai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ValMai/subscriptions","organizations_url":"https://api.github.com/users/ValMai/orgs","repos_url":"https://api.github.com/users/ValMai/repos","events_url":"https://api.github.com/users/ValMai/events{/privacy}","received_events_url":"https://api.github.com/users/ValMai/received_events","type":"User","site_admin":false},"body":"**Ready for review.**","created_at":"2016-02-21T08:13:52Z","updated_at":"2016-02-22T21:14:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ad21253853ac75376a1c0bdd6291e1a0ebecaa2e","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/4/commits","review_comments_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/4/comments","review_comment_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/comments{/number}","comments_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues/4/comments","statuses_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/statuses/a3aeaa6bbec2ead8205db73d32a060373167cec8","head":{"label":"TUDelft-CS4110:test-suite-generation","ref":"test-suite-generation","sha":"a3aeaa6bbec2ead8205db73d32a060373167cec8","user":{"login":"TUDelft-CS4110","id":16717479,"avatar_url":"https://avatars.githubusercontent.com/u/16717479?v=3","gravatar_id":"","url":"https://api.github.com/users/TUDelft-CS4110","html_url":"https://github.com/TUDelft-CS4110","followers_url":"https://api.github.com/users/TUDelft-CS4110/followers","following_url":"https://api.github.com/users/TUDelft-CS4110/following{/other_user}","gists_url":"https://api.github.com/users/TUDelft-CS4110/gists{/gist_id}","starred_url":"https://api.github.com/users/TUDelft-CS4110/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TUDelft-CS4110/subscriptions","organizations_url":"https://api.github.com/users/TUDelft-CS4110/orgs","repos_url":"https://api.github.com/users/TUDelft-CS4110/repos","events_url":"https://api.github.com/users/TUDelft-CS4110/events{/privacy}","received_events_url":"https://api.github.com/users/TUDelft-CS4110/received_events","type":"Organization","site_admin":false},"repo":{"id":51511489,"name":"2016-0x90","full_name":"TUDelft-CS4110/2016-0x90","owner":{"login":"TUDelft-CS4110","id":16717479,"avatar_url":"https://avatars.githubusercontent.com/u/16717479?v=3","gravatar_id":"","url":"https://api.github.com/users/TUDelft-CS4110","html_url":"https://github.com/TUDelft-CS4110","followers_url":"https://api.github.com/users/TUDelft-CS4110/followers","following_url":"https://api.github.com/users/TUDelft-CS4110/following{/other_user}","gists_url":"https://api.github.com/users/TUDelft-CS4110/gists{/gist_id}","starred_url":"https://api.github.com/users/TUDelft-CS4110/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TUDelft-CS4110/subscriptions","organizations_url":"https://api.github.com/users/TUDelft-CS4110/orgs","repos_url":"https://api.github.com/users/TUDelft-CS4110/repos","events_url":"https://api.github.com/users/TUDelft-CS4110/events{/privacy}","received_events_url":"https://api.github.com/users/TUDelft-CS4110/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/TUDelft-CS4110/2016-0x90","description":"","fork":false,"url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90","forks_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/forks","keys_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/keys{/key_id}","collaborators_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/teams","hooks_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/hooks","issue_events_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues/events{/number}","events_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/events","assignees_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/assignees{/user}","branches_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/branches{/branch}","tags_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/tags","blobs_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/refs{/sha}","trees_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/trees{/sha}","statuses_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/statuses/{sha}","languages_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/languages","stargazers_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/stargazers","contributors_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/contributors","subscribers_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/subscribers","subscription_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/subscription","commits_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/commits{/sha}","git_commits_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/commits{/sha}","comments_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/comments{/number}","issue_comment_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues/comments{/number}","contents_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/contents/{+path}","compare_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/compare/{base}...{head}","merges_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/merges","archive_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/downloads","issues_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues{/number}","pulls_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls{/number}","milestones_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/milestones{/number}","notifications_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/labels{/name}","releases_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/releases{/id}","deployments_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/deployments","created_at":"2016-02-11T11:52:10Z","updated_at":"2016-02-11T16:18:07Z","pushed_at":"2016-02-22T20:46:14Z","git_url":"git://github.com/TUDelft-CS4110/2016-0x90.git","ssh_url":"git@github.com:TUDelft-CS4110/2016-0x90.git","clone_url":"https://github.com/TUDelft-CS4110/2016-0x90.git","svn_url":"https://github.com/TUDelft-CS4110/2016-0x90","homepage":null,"size":52,"stargazers_count":1,"watchers_count":1,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":3,"forks":0,"open_issues":3,"watchers":1,"default_branch":"master"}},"base":{"label":"TUDelft-CS4110:master","ref":"master","sha":"16a6488acc84e28b36e16cfa49b1abd2265301a6","user":{"login":"TUDelft-CS4110","id":16717479,"avatar_url":"https://avatars.githubusercontent.com/u/16717479?v=3","gravatar_id":"","url":"https://api.github.com/users/TUDelft-CS4110","html_url":"https://github.com/TUDelft-CS4110","followers_url":"https://api.github.com/users/TUDelft-CS4110/followers","following_url":"https://api.github.com/users/TUDelft-CS4110/following{/other_user}","gists_url":"https://api.github.com/users/TUDelft-CS4110/gists{/gist_id}","starred_url":"https://api.github.com/users/TUDelft-CS4110/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TUDelft-CS4110/subscriptions","organizations_url":"https://api.github.com/users/TUDelft-CS4110/orgs","repos_url":"https://api.github.com/users/TUDelft-CS4110/repos","events_url":"https://api.github.com/users/TUDelft-CS4110/events{/privacy}","received_events_url":"https://api.github.com/users/TUDelft-CS4110/received_events","type":"Organization","site_admin":false},"repo":{"id":51511489,"name":"2016-0x90","full_name":"TUDelft-CS4110/2016-0x90","owner":{"login":"TUDelft-CS4110","id":16717479,"avatar_url":"https://avatars.githubusercontent.com/u/16717479?v=3","gravatar_id":"","url":"https://api.github.com/users/TUDelft-CS4110","html_url":"https://github.com/TUDelft-CS4110","followers_url":"https://api.github.com/users/TUDelft-CS4110/followers","following_url":"https://api.github.com/users/TUDelft-CS4110/following{/other_user}","gists_url":"https://api.github.com/users/TUDelft-CS4110/gists{/gist_id}","starred_url":"https://api.github.com/users/TUDelft-CS4110/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TUDelft-CS4110/subscriptions","organizations_url":"https://api.github.com/users/TUDelft-CS4110/orgs","repos_url":"https://api.github.com/users/TUDelft-CS4110/repos","events_url":"https://api.github.com/users/TUDelft-CS4110/events{/privacy}","received_events_url":"https://api.github.com/users/TUDelft-CS4110/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/TUDelft-CS4110/2016-0x90","description":"","fork":false,"url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90","forks_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/forks","keys_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/keys{/key_id}","collaborators_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/teams","hooks_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/hooks","issue_events_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues/events{/number}","events_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/events","assignees_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/assignees{/user}","branches_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/branches{/branch}","tags_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/tags","blobs_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/refs{/sha}","trees_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/trees{/sha}","statuses_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/statuses/{sha}","languages_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/languages","stargazers_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/stargazers","contributors_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/contributors","subscribers_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/subscribers","subscription_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/subscription","commits_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/commits{/sha}","git_commits_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/git/commits{/sha}","comments_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/comments{/number}","issue_comment_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues/comments{/number}","contents_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/contents/{+path}","compare_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/compare/{base}...{head}","merges_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/merges","archive_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/downloads","issues_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues{/number}","pulls_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls{/number}","milestones_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/milestones{/number}","notifications_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/labels{/name}","releases_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/releases{/id}","deployments_url":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/deployments","created_at":"2016-02-11T11:52:10Z","updated_at":"2016-02-11T16:18:07Z","pushed_at":"2016-02-22T20:46:14Z","git_url":"git://github.com/TUDelft-CS4110/2016-0x90.git","ssh_url":"git@github.com:TUDelft-CS4110/2016-0x90.git","clone_url":"https://github.com/TUDelft-CS4110/2016-0x90.git","svn_url":"https://github.com/TUDelft-CS4110/2016-0x90","homepage":null,"size":52,"stargazers_count":1,"watchers_count":1,"language":null,"has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":3,"forks":0,"open_issues":3,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/4"},"html":{"href":"https://github.com/TUDelft-CS4110/2016-0x90/pull/4"},"issue":{"href":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues/4"},"comments":{"href":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/issues/4/comments"},"review_comments":{"href":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/4/comments"},"review_comment":{"href":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/pulls/4/commits"},"statuses":{"href":"https://api.github.com/repos/TUDelft-CS4110/2016-0x90/statuses/a3aeaa6bbec2ead8205db73d32a060373167cec8"}}}}