{"action":"created","issue":{"url":"https://api.github.com/repos/Unidata/thredds/issues/513","repository_url":"https://api.github.com/repos/Unidata/thredds","labels_url":"https://api.github.com/repos/Unidata/thredds/issues/513/labels{/name}","comments_url":"https://api.github.com/repos/Unidata/thredds/issues/513/comments","events_url":"https://api.github.com/repos/Unidata/thredds/issues/513/events","html_url":"https://github.com/Unidata/thredds/pull/513","id":145560024,"number":513,"title":"Fix Gihub issue 496","user":{"login":"DennisHeimbigner","id":905179,"avatar_url":"https://avatars.githubusercontent.com/u/905179?v=3","gravatar_id":"","url":"https://api.github.com/users/DennisHeimbigner","html_url":"https://github.com/DennisHeimbigner","followers_url":"https://api.github.com/users/DennisHeimbigner/followers","following_url":"https://api.github.com/users/DennisHeimbigner/following{/other_user}","gists_url":"https://api.github.com/users/DennisHeimbigner/gists{/gist_id}","starred_url":"https://api.github.com/users/DennisHeimbigner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DennisHeimbigner/subscriptions","organizations_url":"https://api.github.com/users/DennisHeimbigner/orgs","repos_url":"https://api.github.com/users/DennisHeimbigner/repos","events_url":"https://api.github.com/users/DennisHeimbigner/events{/privacy}","received_events_url":"https://api.github.com/users/DennisHeimbigner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-04-04T01:06:12Z","updated_at":"2016-04-05T20:19:04Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Unidata/thredds/pulls/513","html_url":"https://github.com/Unidata/thredds/pull/513","diff_url":"https://github.com/Unidata/thredds/pull/513.diff","patch_url":"https://github.com/Unidata/thredds/pull/513.patch"},"body":"(https://github.com/Unidata/thredds/issues/496)\r\nThe primary problem to be addressed was the observation by John Caron\r\nthat Tdm was getting hung if ran with more than one thread.\r\n\r\nSecondarily, there was a problem with 'UrlAuthenticatorDialog' not being\r\nable to handle bad inputs; it did not ask again when they failed.  Note\r\nthat this error typically was seen when accessing a password protected\r\ndataset using 'ToolsUI'.\r\n\r\n## Solution WRT to the threading problem:\r\n\r\nFirst, I created a testcase to show the problem.  This is called\r\n'TestThreading'. If run under original version 5.0.0, it hangs when the\r\nthread level is above a certain level.\r\n\r\nSecond, I centralized all of the connection management code into a new\r\nclass called 'HTTPConnections'.  I then carefully synchronized it.  I\r\nthen modified other httpservices module code to utilize this class.\r\n\r\nThis seems to solve the problem in that TestThreading works with large\r\n(~100) numbers of threads. Unfortunately, I cannot pin-point what\r\nchanged that fixed the problem.\r\n\r\n## Solution WRT to the 'UrlAuthenticatorDialog' problem.\r\n\r\nFirst, I created an alternate and simplified Dialog box for testing\r\npurposes. This is called 'Login'.  I then added a test to 'TestAuth'\r\n(now renamed to 'TestTomcatAuth') to utilize this dialog box to show the\r\nproblem.\r\n\r\nThe issue was an interaction with Apache httpclient credentials\r\ncaching. I was keeping an internal (to UrlAuthenticationDialog) database\r\nmapping AuthScope->Credentials. My mistake was in storing the results of\r\nthe interactive dialog in the internal database rather than letting the\r\nApache httpclient cache store it. The result, for bad credentials, was\r\nan inconsistency between the internal database and the apache library\r\ncache.\r\n\r\nThe reason I had put the internal database into 'UrlAuthenticationDialog'\r\nis that, ideally, a CredentialsProvider object should contain all the\r\ncredentials required for responding to all authorization requests from\r\nthe client to the ultimate server. This is the purpose of the\r\nCredentialsProvider#setCredentials() method.  The reason this may be\r\nneeded is that as the client wends its way to the server, it may\r\nencounter, for, example, password protected proxies, and/or\r\nredirection-based authentication (e.g. OAUTH2) before it even gets to\r\nthe server. The server itself may then require additional authentication\r\nbased on spring-security for accessing specific datasets.\r\n\r\nIn any case, I reverted 'UrlAuthenticatorDialog' to its original form\r\nthat has no internal credentials database. I left it in the 'Login'\r\ndialog, however, for testing purposes. The net result is that if\r\n'UrlAuthenticationDialog' is given a bad set of credentials, it\r\nwill report this properly. \r\n\r\nThere is a potential problem down the road, however, in that \r\n'UrlAuthenticationDialog' as it stands is not well suited to handle\r\ncalls that require multiple authorizations along the path from client\r\nto server. So, at some point, this will have to fixed.\r\n\r\n## Other Changes\r\n\r\n* The original 'TestAuth' was failing because Caron introduced\r\n  spring-security to control access to datasets in the new class called\r\n  'TomcatAuthorizer'. So, I renamed 'TestAuth' to 'TestTomcatAuth' and\r\n  modified web.xml and catalog.xml so that it tested only using\r\n  container-based authorization and by-passed spring altogether. This\r\n  change also required adding a new root called 'containerauth' to the\r\n  catalog.  This caused some unforseen failures in some other tests\r\n  because the number of roots in the catalog changed from 3 to 4.  The\r\n  affected tests were 'TestConfigCatalogBuilder' and 'TestDatasetScan'.\r\n\r\n* In the process of revamping the Dap4 parsers, I rebuilt\r\n  'Dap4ParserBody' and 'CEParserBody'. Also I removed the\r\n  copies in dap4/grammars from github because they can\r\n  are generated files.\r\n\r\n* Removed hold-over reference to thredds-dev in 'TestDir'.\r\n\r\n* Misc. changes to the 'HTTPMethod#getURI' to return the URL object\r\n  instead of the .toString() result.  this introduced some back\r\n  incompatibilities, although I doubt any external clients are using the\r\n  affected method.\r\n\r\n* Certain methods in httpservices are public purely for testing access.\r\n  I modified them to require that HTTPSession.TESTING be set to true\r\n  in order to access those methods.\r\n\r\n* Added a new documentation file called 'ssh.adoc' to better\r\n   describe how ssh support in tomcat should be handled.\r\n"},"comment":{"url":"https://api.github.com/repos/Unidata/thredds/issues/comments/205974570","html_url":"https://github.com/Unidata/thredds/pull/513#issuecomment-205974570","issue_url":"https://api.github.com/repos/Unidata/thredds/issues/513","id":205974570,"user":{"login":"DennisHeimbigner","id":905179,"avatar_url":"https://avatars.githubusercontent.com/u/905179?v=3","gravatar_id":"","url":"https://api.github.com/users/DennisHeimbigner","html_url":"https://github.com/DennisHeimbigner","followers_url":"https://api.github.com/users/DennisHeimbigner/followers","following_url":"https://api.github.com/users/DennisHeimbigner/following{/other_user}","gists_url":"https://api.github.com/users/DennisHeimbigner/gists{/gist_id}","starred_url":"https://api.github.com/users/DennisHeimbigner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DennisHeimbigner/subscriptions","organizations_url":"https://api.github.com/users/DennisHeimbigner/orgs","repos_url":"https://api.github.com/users/DennisHeimbigner/repos","events_url":"https://api.github.com/users/DennisHeimbigner/events{/privacy}","received_events_url":"https://api.github.com/users/DennisHeimbigner/received_events","type":"User","site_admin":false},"created_at":"2016-04-05T20:19:04Z","updated_at":"2016-04-05T20:19:04Z","body":"> So this PR will fix TDM's hanging problem? Can we test it on thredds-test?\r\nTo be precise, it fixed the bug shown by TestThreading. Whether or not if fixed\r\nthe Tdm problem with nthreads > 1, I have no idea and cannot test.\r\n"}}