{"action":"created","issue":{"url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/issues/463","repository_url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs","labels_url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/issues/463/labels{/name}","comments_url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/issues/463/comments","events_url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/issues/463/events","html_url":"https://github.com/SpiderLabs/owasp-modsecurity-crs/pull/463","id":168513728,"number":463,"title":"Improve setup.conf","user":{"login":"lifeforms","id":3464056,"avatar_url":"https://avatars.githubusercontent.com/u/3464056?v=3","gravatar_id":"","url":"https://api.github.com/users/lifeforms","html_url":"https://github.com/lifeforms","followers_url":"https://api.github.com/users/lifeforms/followers","following_url":"https://api.github.com/users/lifeforms/following{/other_user}","gists_url":"https://api.github.com/users/lifeforms/gists{/gist_id}","starred_url":"https://api.github.com/users/lifeforms/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lifeforms/subscriptions","organizations_url":"https://api.github.com/users/lifeforms/orgs","repos_url":"https://api.github.com/users/lifeforms/repos","events_url":"https://api.github.com/users/lifeforms/events{/privacy}","received_events_url":"https://api.github.com/users/lifeforms/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/labels/V3.0.0-rc1%20Development","name":"V3.0.0-rc1 Development","color":"bfe5bf"},{"url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/labels/V3.0.0-rc1%20Testing","name":"V3.0.0-rc1 Testing","color":"c5def5"}],"state":"open","locked":false,"assignee":{"login":"lifeforms","id":3464056,"avatar_url":"https://avatars.githubusercontent.com/u/3464056?v=3","gravatar_id":"","url":"https://api.github.com/users/lifeforms","html_url":"https://github.com/lifeforms","followers_url":"https://api.github.com/users/lifeforms/followers","following_url":"https://api.github.com/users/lifeforms/following{/other_user}","gists_url":"https://api.github.com/users/lifeforms/gists{/gist_id}","starred_url":"https://api.github.com/users/lifeforms/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lifeforms/subscriptions","organizations_url":"https://api.github.com/users/lifeforms/orgs","repos_url":"https://api.github.com/users/lifeforms/repos","events_url":"https://api.github.com/users/lifeforms/events{/privacy}","received_events_url":"https://api.github.com/users/lifeforms/received_events","type":"User","site_admin":false},"assignees":[{"login":"lifeforms","id":3464056,"avatar_url":"https://avatars.githubusercontent.com/u/3464056?v=3","gravatar_id":"","url":"https://api.github.com/users/lifeforms","html_url":"https://github.com/lifeforms","followers_url":"https://api.github.com/users/lifeforms/followers","following_url":"https://api.github.com/users/lifeforms/following{/other_user}","gists_url":"https://api.github.com/users/lifeforms/gists{/gist_id}","starred_url":"https://api.github.com/users/lifeforms/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lifeforms/subscriptions","organizations_url":"https://api.github.com/users/lifeforms/orgs","repos_url":"https://api.github.com/users/lifeforms/repos","events_url":"https://api.github.com/users/lifeforms/events{/privacy}","received_events_url":"https://api.github.com/users/lifeforms/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/milestones/3","html_url":"https://github.com/SpiderLabs/owasp-modsecurity-crs/milestones/CRS%20v3.0.0%20RC1","labels_url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/milestones/3/labels","id":458025,"number":3,"title":"CRS v3.0.0 RC1","description":"","creator":{"login":"rcbarnett","id":621352,"avatar_url":"https://avatars.githubusercontent.com/u/621352?v=3","gravatar_id":"","url":"https://api.github.com/users/rcbarnett","html_url":"https://github.com/rcbarnett","followers_url":"https://api.github.com/users/rcbarnett/followers","following_url":"https://api.github.com/users/rcbarnett/following{/other_user}","gists_url":"https://api.github.com/users/rcbarnett/gists{/gist_id}","starred_url":"https://api.github.com/users/rcbarnett/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rcbarnett/subscriptions","organizations_url":"https://api.github.com/users/rcbarnett/orgs","repos_url":"https://api.github.com/users/rcbarnett/repos","events_url":"https://api.github.com/users/rcbarnett/events{/privacy}","received_events_url":"https://api.github.com/users/rcbarnett/received_events","type":"User","site_admin":false},"open_issues":11,"closed_issues":119,"state":"open","created_at":"2013-10-17T21:30:28Z","updated_at":"2016-08-05T18:33:32Z","due_on":"2016-08-08T22:00:00Z","closed_at":null},"comments":25,"created_at":"2016-07-31T11:32:51Z","updated_at":"2016-08-06T21:54:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/pulls/463","html_url":"https://github.com/SpiderLabs/owasp-modsecurity-crs/pull/463","diff_url":"https://github.com/SpiderLabs/owasp-modsecurity-crs/pull/463.diff","patch_url":"https://github.com/SpiderLabs/owasp-modsecurity-crs/pull/463.patch"},"body":"I've done a thorough review of the setup.conf.example file. This fixes known problems in issues #368 and #384. As a user I've always found the setup file pretty intimidating, and whenever I tried experimenting, it would almost never do what I expected. My main mission is to simplify, lower cognitive load, potential for error, and remove noise:\r\n\r\n- A setting should do the obvious thing\r\n- Any non salient/inactive parts should be commented out, so novice users can feel confident skipping over them\r\n- Set good defaults\r\n- CRS internal initialization and implementation details should not be in this file\r\n- Remove all vestigial remains of past CRS2 functionality\r\n- Remove duplicated functionality\r\n- Review and update comments and examples\r\n- Make the CRS work great in the default configuration with no single tweak needed to setup.conf\r\n- Move important settings to the top of the file\r\n\r\nThis is a big PR, but all changes were done in isolation, so controversial parts can easily be improved.\r\n\r\nPlease review and share your opinion.\r\n\r\n### Metrics:\r\n\r\n- The setup.conf file has been reduced from 662 to 580 lines.\r\n- The NON-COMMENTED lines have been reduced from 212 to 44 lines!\r\n\r\n### List of changes:\r\n\r\n***Move CRS internal initialization out of the setup***\r\n\r\n- Move `SecComponentSignature` to REQUEST-901-INITIALIZATION.conf\r\n- Move tx scores initialiation (was 900002) to REQUEST-901-INITIALIZATION.conf\r\n- Move tx.sql_error_match initialization (was 900024) to REQUEST-901-INITIALIZATION.conf\r\n- Move collection initialization (was 900018-900021) to REQUEST-901-INITIALIZATION.conf\r\n\r\nThis decreases the scariness of the setup. Also, moving these rules to a versioned file means we can improve them with an update.\r\n\r\n***Lower user error rate and gracefully handle problems***\r\n\r\n- Check if setup is loaded at CRS init (rule 901001). At the bottom of setup.conf, set variable `tx.setup_done=1`. Check if set. If not, pass, but log a warning. This will help people having misconfigurations.\r\n- Add various complete examples for anomaly/self-contained mode, and ensure that their configuration can be done from one place as much as possible\r\n- Improve the anomaly mode disruptive action examples in RESPONSE-999-EXCEPTIONS adding outbound response cases\r\n- Add defaults for severity levels & block duration, and comment them out in the setup\r\n- Add defaults for threshold levels, and comment them out in the setup\r\n\r\n***Make policies apply consistently***\r\n\r\nI experimented with HTTP policy settings (900006-900012) and found the effects very unpredictable. I tried to make these policy settings consistent and easy. Basically, if the user sets a policy or limit, the CRS must enforce it at all times.\r\n\r\n- Enforce `restricted_extensions` (900012) HTTP policy in PL1. This setting was only honored in PL2 and higher (rule 920440). This was undocumented. It was also highly inconsistent, because the other HTTP policy settings like `allowed_methods`, `allowed_http_versions`, `allowed_request_content_type` ARE active in PL1! I find the linkage with PL2 very hard to understand for the user. It forces the user to migrate to PL2 in order to use this policy feature, which they might not want or need. So, I have moved 920440 to PL1, but I am open to discussion. If we should feel that the extension list is too restrictive for the default user, we have more options like clearing or restricting the default extensions and adding more examples, or increase the list in PL2. (I personally think the feature is fine for PL1, since many users will benefit hugely from not leaking .bak / .sql files etc., and a user can tweak this list relatively easily.)\r\n- Enforce `max_num_args` HTTP limits (900006) in PL1, and make the rule consistent with other HTTP limits (90007-900011). This is a similar story. The treatment of this rule was inconsistent: rule 900006 was enabled in setup.conf while all the other HTTP limits were commented out. Then, probably to lower FP, the checking rule for `max_num_args` (920380) was moved to PL2, while all the other HTTP limits were still checked in PL1. Not clear. If we want to prevent FP in the default setting, which we should, we can just unset this default. So, all HTTP limits are now unset by default, but if a user enables a limit, it will always be honored, no matter the PL.\r\n- Fix `allowed_http_versions` check (920430) having CRITICAL severity but accumulating only a WARNING score. If a user does not allow a HTTP version, it should be critically blocked (the definition of disallow!). So fix the anomaly score.\r\n\r\nIn the interest of efficiency I have addressed all of these inconsistencies in the PR. I know we have worked on these rules earlier, so if I am overriding an earlier decision please discuss.\r\n\r\n***Improve usability***\r\n\r\n- In file header, refer to minimum required ModSecurity version, file ordering, and the INSTALL file for complete instructions\r\n- Move the most important, most-often changed settings higher up in the file\r\n- Add comments for paranoia level feature\r\n- Clarify anomaly blocking, move 999-EXCEPTION instructions to that file instead of duplicating it here\r\n- Consistently use 'anomaly mode' instead of its synonym 'collaborative blocking'\r\n- When a rule is commented out, comment out *all* its lines and not only the `SecAction` line. This makes much clearer to see from a distance that those lines are inactive, especially when using an editor with syntax highlighting.\r\n- Split off block duration to a separate rule\r\n- Comment out GeoDB/country rules since they are inactive by default\r\n- Add Include instructions to regression test mode, and rename the regression rule file to modern 9xx convention\r\n\r\n***Remove retired and unnecessary settings***\r\n\r\n- Remove `tx.anomaly_score_blocking` variable which has no function anymore. It was checked in the 949/959 rules, but once the client ends up here, we always know that the request isn't blocked by self-contained mode, so we do not need to check it.\r\n- Remove `tx.brute_force` settings, this was a CRS2 experimental rule which is gone\r\n- Remove `tx.csp_report` settings, this was a CRS2 experimental rule which is gone\r\n\r\n***Text and style changes***\r\n\r\n- Improve wording, shorten and simplify many comments\r\n- Fit lines in 80 characters, always use 2 newlines between sections\r\n- Replace references to old rule file names, use new 9xx variants\r\n- Add notes and warnings for dangerous features\r\n- Fix typos, copyright year, LICENCE file references\r\n\r\n***Recommended further removals (NOT yet carried out, please ack or discuss)***\r\n\r\nNote that I have NOT yet performed the following changes yet, since I don't know if there are any uses for these settings. I would strongly recommend removing them, however I will wait for an ACK first since they involve rule surgery:\r\n\r\n- `tx.allowed_html_urls` (rule 900023): This setting just disables XSS rules (file 941) for some list of configured URLs. It's basically an inferior duplicate subsystem just for excluding XSS. Using exclusions with tags is better: this setting excludes all args, but in an exclusion rule, you can exclude a specific target (like ARGS:bad). Making the user learn two systems is counterproductive. Removing this system would also reduce rule complexity. Strongly recommend to REMOVE this setting and this functionality.\r\n- `tx.msaccess`, `tx.mysql`, `tx.oracle` etc. (rule 900024): Same story. These settings disable some rules in file 951. It's basically another nonstandard subsystem to perform exclusions. Earlier, we decided to not expand this system to further applications, but instead use tagging. I now think we should apply this decision consistently. Strongly recommend to REMOVE this setting and this functionality from the 951 rules. Instead, the relevant rules can be tagged with `application-msaccess`, `application-mysql`, et cetera, and users can apply an exclusion rule if they please. Performance is not an issue -- these variables are only used in 951 after a pmf pre-check. Removing them simplifies setup, exclusions, and rule complexity.\r\n\r\nCulling these two duplicate systems would lower the setup.conf length from 580 to 520 lines.\r\n\r\nIt would also lower the NON-COMMENTED lines from 44 to 22 lines.\r\n\r\nMeaning, after these improvements, the whole ACTIVE, non-commented part of the setup.conf would only remain:\r\n\r\n```\r\n# cat modsecurity_crs_10_setup.conf.example |grep -v '^#'|grep -v '^$'\r\nSecDefaultAction \"phase:1,log,auditlog,pass\"\r\nSecDefaultAction \"phase:2,log,auditlog,pass\"\r\nSecAction \\\r\n \"id:900012,\\\r\n  phase:1,\\\r\n  nolog,\\\r\n  pass,\\\r\n  t:none,\\\r\n  setvar:'tx.allowed_methods=GET HEAD POST OPTIONS',\\\r\n  setvar:'tx.static_resources=.jpg/ .jpeg/ .png/ .gif/ .js/ .css/ .ico/ .svg/ .webp/',\\\r\n  setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded|multipart/form-data|text/xml|application/xml|application/x-amf|application/json|text/plain',\\\r\n  setvar:'tx.allowed_http_versions=HTTP/1.0 HTTP/1.1 HTTP/2',\\\r\n  setvar:'tx.restricted_extensions=.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .resources/ .resx/ .sql/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/',\\\r\n  setvar:'tx.restricted_headers=proxy lock-token content-range translate if'\"\r\nSecCollectionTimeout 600\r\nSecAction \\\r\n \"id:'900099',\\\r\n  phase:1,\\\r\n  nolog,\\\r\n  pass,\\\r\n  t:none,\\\r\n  setvar:tx.setup_done=1\"\r\n```\r\n\r\nThis makes the setup extremely nimble.\r\n\r\nAs a sidenote, I note that the setup also still has an old-fashioned CRS2-style file name `modsecurity_crs_10_setup.conf.example`. Rename it now to something better? We haven't cared about case ordering/sorting, so just `crs.conf.example`?\r\n\r\n### Pending items:\r\n\r\n- [ ] code review\r\n- [x] discuss `tx.allowed_html_urls` (900023) setting removal\r\n- [x] discuss `tx.msaccess`, `tx.mysql`, `tx.oracle` etc. (900024) setting removal\r\n- [x] if #464 (upgrade.py) is merged, update GeoIP line and instructions\n- [ ] discuss config file rename, rename file, search/replace name, add to .gitignore\r\n"},"comment":{"url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/issues/comments/238050855","html_url":"https://github.com/SpiderLabs/owasp-modsecurity-crs/pull/463#issuecomment-238050855","issue_url":"https://api.github.com/repos/SpiderLabs/owasp-modsecurity-crs/issues/463","id":238050855,"user":{"login":"lifeforms","id":3464056,"avatar_url":"https://avatars.githubusercontent.com/u/3464056?v=3","gravatar_id":"","url":"https://api.github.com/users/lifeforms","html_url":"https://github.com/lifeforms","followers_url":"https://api.github.com/users/lifeforms/followers","following_url":"https://api.github.com/users/lifeforms/following{/other_user}","gists_url":"https://api.github.com/users/lifeforms/gists{/gist_id}","starred_url":"https://api.github.com/users/lifeforms/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lifeforms/subscriptions","organizations_url":"https://api.github.com/users/lifeforms/orgs","repos_url":"https://api.github.com/users/lifeforms/repos","events_url":"https://api.github.com/users/lifeforms/events{/privacy}","received_events_url":"https://api.github.com/users/lifeforms/received_events","type":"User","site_admin":false},"created_at":"2016-08-06T21:54:17Z","updated_at":"2016-08-06T21:54:17Z","body":"Cheers!\r\n\r\n@dune73 What file name would you propose for the setup?"}}
{"actor":{"display_login":"lifeforms"}}