{"action":"created","issue":{"url":"https://api.github.com/repos/isetbio/UnitTestToolbox/issues/14","repository_url":"https://api.github.com/repos/isetbio/UnitTestToolbox","labels_url":"https://api.github.com/repos/isetbio/UnitTestToolbox/issues/14/labels{/name}","comments_url":"https://api.github.com/repos/isetbio/UnitTestToolbox/issues/14/comments","events_url":"https://api.github.com/repos/isetbio/UnitTestToolbox/issues/14/events","html_url":"https://github.com/isetbio/UnitTestToolbox/issues/14","id":189114515,"number":14,"title":"Fast validations and custom tolerances","user":{"login":"DavidBrainard","id":1158757,"avatar_url":"https://avatars.githubusercontent.com/u/1158757?v=3","gravatar_id":"","url":"https://api.github.com/users/DavidBrainard","html_url":"https://github.com/DavidBrainard","followers_url":"https://api.github.com/users/DavidBrainard/followers","following_url":"https://api.github.com/users/DavidBrainard/following{/other_user}","gists_url":"https://api.github.com/users/DavidBrainard/gists{/gist_id}","starred_url":"https://api.github.com/users/DavidBrainard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DavidBrainard/subscriptions","organizations_url":"https://api.github.com/users/DavidBrainard/orgs","repos_url":"https://api.github.com/users/DavidBrainard/repos","events_url":"https://api.github.com/users/DavidBrainard/events{/privacy}","received_events_url":"https://api.github.com/users/DavidBrainard/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-14T13:39:18Z","updated_at":"2016-11-14T17:20:42Z","closed_at":null,"body":"When we use a custom tolerance, the Fast validation hash should respect it, right?\r\n\r\nThis does not seem to be happening for a validation in our temporalFittingEngine code, see output below.  The Full passes (with custom tolerance) but the Fast fails.  The validation files appear to have been generated together.\r\n\r\n[  1] 'v_crossValidateFits'\r\n\tInternal validation  : PASSED\r\n \tRun-time status      : no exception raised\r\n\tGround truth  url    : http://52.32.77.154/repository/temporalFittingEngine/validation/fast/CrossValidationRoutines/crossValidateFits/run00001/crossValidateFits-run00001-mat.mat\r\n\tGround truth  localP : /Users/aguirre/.gradle/caches/modules-2/files-2.1/validation.fast.CrossValidationRoutines/crossValidateFits/run00001/75e5df9a577878df758636c08ed1fec0b4548125/crossValidateFits-run00001-mat.mat\r\n\tFast validation      : FAILED against ground truth data of 12-Nov-2016 12:49:40.\r\n\t > Ground truth info :       eagleray.psych.upenn.edu / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'dhb'\r\n\t > Local host info   :             GKA-MacBook.local\r\n / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'aguirre'\r\n\tDataHash-this run    : 9f6e4cbd61f2690c18e95222874028fd93508bb41d06c88819e55877b39282e2\r\n\tDataHash-groundTruth : 2d716f3b535f969099d256975002de9fe73e9d904ed77fcae646dc0fbe00918b\r\n\t---------------------------------------------------------------------------------------------------------------------------------\r\n\tGround truth  url    : http://52.32.77.154/repository/temporalFittingEngine/validation/full/CrossValidationRoutines/crossValidateFits/run00001/crossValidateFits-run00001-mat.mat\r\n\tGround truth  localP : /Users/aguirre/.gradle/caches/modules-2/files-2.1/validation.full.CrossValidationRoutines/crossValidateFits/run00001/28e619906dbd3b5c51f5d43b9e4090f0d36aed91/crossValidateFits-run00001-mat.mat\r\n\tValidating using custom tolerance for field validationData1(1).paramMainMatrix: 0.0005\r\n\tValidating using custom tolerance for field validationData1(1).testfVals: 1e-05\r\n\tValidating using custom tolerance for field validationData1(1).trainfVals: 1e-05\r\n\tFull validation      : PASSED using custom tolerance for fields: {'validationData1'}  against ground truth data of 12-Nov-2016 12:49:52.\r\n\t > Ground truth info :       eagleray.psych.upenn.edu / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'dhb'\r\n\t > Local host info   :             GKA-MacBook.local\r\n / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'aguirre'\r\n\tNote (*)             : script does not store any extra data.\r\n\tValidation report    : ***** v_crossValidateFits *****\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/isetbio/UnitTestToolbox/issues/comments/260399802","html_url":"https://github.com/isetbio/UnitTestToolbox/issues/14#issuecomment-260399802","issue_url":"https://api.github.com/repos/isetbio/UnitTestToolbox/issues/14","id":260399802,"user":{"login":"npcottaris","id":3444212,"avatar_url":"https://avatars.githubusercontent.com/u/3444212?v=3","gravatar_id":"","url":"https://api.github.com/users/npcottaris","html_url":"https://github.com/npcottaris","followers_url":"https://api.github.com/users/npcottaris/followers","following_url":"https://api.github.com/users/npcottaris/following{/other_user}","gists_url":"https://api.github.com/users/npcottaris/gists{/gist_id}","starred_url":"https://api.github.com/users/npcottaris/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/npcottaris/subscriptions","organizations_url":"https://api.github.com/users/npcottaris/orgs","repos_url":"https://api.github.com/users/npcottaris/repos","events_url":"https://api.github.com/users/npcottaris/events{/privacy}","received_events_url":"https://api.github.com/users/npcottaris/received_events","type":"User","site_admin":false},"created_at":"2016-11-14T17:20:42Z","updated_at":"2016-11-14T17:20:42Z","body":"Hmmm... The above passes both FAST and FULL on my iMac.\r\nI think that this is probably a precision issue with GKA-MacBook. \r\nWe can try lowering the custom tolerance until it passes validation on the GKA, but this will be a trial an error approach.\r\n\r\n\r\nOutput from my iMac shown below:\r\n~~~\r\n------------------------------------------------------------------------------------------------------------\r\nRunning in 'FULL' mode with 'catchExceptionAndContinue' runtime hehavior and verbosity level = 'high'.\r\nWill validate 1 scripts.\r\n------------------------------------------------------------------------------------------------------------\r\n\r\n[  1] 'v_crossValidateFits'\r\n\tInternal validation  : PASSED\r\n \tRun-time status      : no exception raised\r\n\tGround truth  url    : http://52.32.77.154/repository/temporalFittingEngine/validation/fast/CrossValidationRoutines/crossValidateFits/run00001/crossValidateFits-run00001-mat.mat\r\n\tGround truth  localP : /Users/nicolas/.gradle/caches/modules-2/files-2.1/validation.fast.CrossValidationRoutines/crossValidateFits/run00001/75e5df9a577878df758636c08ed1fec0b4548125/crossValidateFits-run00001-mat.mat\r\n\tFast validation      : PASSED against ground truth data of 12-Nov-2016 12:49:40.\r\n\t > Ground truth info :       eagleray.psych.upenn.edu / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'dhb'\r\n\t > Local host info   :          manta.PSYCH.UPENN.EDU / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'nicolas'\r\n\tData hash key        : 2d716f3b535f969099d256975002de9fe73e9d904ed77fcae646dc0fbe00918b\r\n\t---------------------------------------------------------------------------------------------------------------------------------\r\n\tGround truth  url    : http://52.32.77.154/repository/temporalFittingEngine/validation/full/CrossValidationRoutines/crossValidateFits/run00001/crossValidateFits-run00001-mat.mat\r\n\tGround truth  localP : /Users/nicolas/.gradle/caches/modules-2/files-2.1/validation.full.CrossValidationRoutines/crossValidateFits/run00001/28e619906dbd3b5c51f5d43b9e4090f0d36aed91/crossValidateFits-run00001-mat.mat\r\n\tValidating using custom tolerance for field validationData1(1).paramMainMatrix: 0.0005\r\n\tValidating using custom tolerance for field validationData1(1).testfVals: 1e-05\r\n\tValidating using custom tolerance for field validationData1(1).trainfVals: 1e-05\r\n\tFull validation      : PASSED using custom tolerance for fields: {'validationData1'}  against ground truth data of 12-Nov-2016 12:49:52.\r\n\t > Ground truth info :       eagleray.psych.upenn.edu / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'dhb'\r\n\t > Local host info   :          manta.PSYCH.UPENN.EDU / MACI64, MATLAB 9.0.0.341360 (R2016a) by 'nicolas'\r\n\tNote (*)             : script does not store any extra data.\r\n\tValidation report    : ***** v_crossValidateFits *****\r\n\r\n\r\n SUMMARY REPORT \r\n[  1] 'v_crossValidateFits': ......................................... Internal validation: PASSED ; Runtime status: NO EXCEPTION RAISED ; Fast validation: PASSED ; Full validation: PASSED ; (using custom tolerances for fields: {'validationData1'}); \r\n~~~"}}
{"actor":{"display_login":"npcottaris"}}