{"action":"created","issue":{"url":"https://api.github.com/repos/radiasoft/zgoubi/issues/4","repository_url":"https://api.github.com/repos/radiasoft/zgoubi","labels_url":"https://api.github.com/repos/radiasoft/zgoubi/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/radiasoft/zgoubi/issues/4/comments","events_url":"https://api.github.com/repos/radiasoft/zgoubi/issues/4/events","html_url":"https://github.com/radiasoft/zgoubi/issues/4","id":328242401,"node_id":"MDU6SXNzdWUzMjgyNDI0MDE=","number":4,"title":"Profile with TAU","user":{"login":"rouson","id":13108868,"node_id":"MDQ6VXNlcjEzMTA4ODY4","avatar_url":"https://avatars2.githubusercontent.com/u/13108868?v=4","gravatar_id":"","url":"https://api.github.com/users/rouson","html_url":"https://github.com/rouson","followers_url":"https://api.github.com/users/rouson/followers","following_url":"https://api.github.com/users/rouson/following{/other_user}","gists_url":"https://api.github.com/users/rouson/gists{/gist_id}","starred_url":"https://api.github.com/users/rouson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rouson/subscriptions","organizations_url":"https://api.github.com/users/rouson/orgs","repos_url":"https://api.github.com/users/rouson/repos","events_url":"https://api.github.com/users/rouson/events{/privacy}","received_events_url":"https://api.github.com/users/rouson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/radiasoft/zgoubi/milestones/1","html_url":"https://github.com/radiasoft/zgoubi/milestone/1","labels_url":"https://api.github.com/repos/radiasoft/zgoubi/milestones/1/labels","id":3400506,"node_id":"MDk6TWlsZXN0b25lMzQwMDUwNg==","number":1,"title":"SOW 1","description":"I suggest focusing SOW 1 primarily around an \"active\" code review designed to increase our knowledge of the code and confidence in running and refactoring it.   By \"active,\" I'm referring to tasks that produce useful deliverables while exposing code structure.\r\n\r\nI'm choosing a due date that allows for pre-existing commitments on the part of Sourcery staff and contracts and also allows time for staffing the project.  Work will commence as soon as this SOW is approved by Radiasoft, and it's possible this iteration could finish sooner than the due date if our work mix and staffing allows.","creator":{"login":"rouson","id":13108868,"node_id":"MDQ6VXNlcjEzMTA4ODY4","avatar_url":"https://avatars2.githubusercontent.com/u/13108868?v=4","gravatar_id":"","url":"https://api.github.com/users/rouson","html_url":"https://github.com/rouson","followers_url":"https://api.github.com/users/rouson/followers","following_url":"https://api.github.com/users/rouson/following{/other_user}","gists_url":"https://api.github.com/users/rouson/gists{/gist_id}","starred_url":"https://api.github.com/users/rouson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rouson/subscriptions","organizations_url":"https://api.github.com/users/rouson/orgs","repos_url":"https://api.github.com/users/rouson/repos","events_url":"https://api.github.com/users/rouson/events{/privacy}","received_events_url":"https://api.github.com/users/rouson/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":5,"state":"open","created_at":"2018-06-04T23:38:36Z","updated_at":"2018-07-19T00:19:34Z","due_on":"2018-07-13T07:00:00Z","closed_at":null},"comments":4,"created_at":"2018-05-31T18:09:29Z","updated_at":"2018-07-19T01:34:56Z","closed_at":null,"author_association":"COLLABORATOR","body":"1. Lay the groundwork for parallel performance analysis by profiling with the [Tuning and Analysis Utilities]  (TAU). Comparing the profiling information that TAU produces from the existing serial code with the previous profiles will validate the initial TAU output but, more importantly, will provide a straightforward path to parallel performance analysis.\r\n2. Identify bottlenecks and estimate the ideal speedup that can be achieved though one or more performance-enhancement strategies, including coarray parallelization.\r\n\r\n[Tuning and Analysis Utilities]: https://tau.uoregon.edu"},"comment":{"url":"https://api.github.com/repos/radiasoft/zgoubi/issues/comments/406126256","html_url":"https://github.com/radiasoft/zgoubi/issues/4#issuecomment-406126256","issue_url":"https://api.github.com/repos/radiasoft/zgoubi/issues/4","id":406126256,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNjEyNjI1Ng==","user":{"login":"zbeekman","id":279612,"node_id":"MDQ6VXNlcjI3OTYxMg==","avatar_url":"https://avatars3.githubusercontent.com/u/279612?v=4","gravatar_id":"","url":"https://api.github.com/users/zbeekman","html_url":"https://github.com/zbeekman","followers_url":"https://api.github.com/users/zbeekman/followers","following_url":"https://api.github.com/users/zbeekman/following{/other_user}","gists_url":"https://api.github.com/users/zbeekman/gists{/gist_id}","starred_url":"https://api.github.com/users/zbeekman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zbeekman/subscriptions","organizations_url":"https://api.github.com/users/zbeekman/orgs","repos_url":"https://api.github.com/users/zbeekman/repos","events_url":"https://api.github.com/users/zbeekman/events{/privacy}","received_events_url":"https://api.github.com/users/zbeekman/received_events","type":"User","site_admin":false},"created_at":"2018-07-19T01:34:56Z","updated_at":"2018-07-19T01:34:56Z","author_association":"COLLABORATOR","body":"# RE: caf/cafrun\r\n\r\n> I didn't see the caf/cafrun support in the develop branch. Is that checked in?\r\n\r\nNo, this is trivial and is essentially a system/compiler property, not something that should be tracked in the repository. For compilation, the quick and dirty hack is just to say\r\n\r\n```\r\nexport FC=caf\r\nexport CC=mpicc # gcc would probably be fine here too, since the Fortran \"compiler\" is used to invoke the linker\r\n```\r\n\r\nafter installing OpenCoarrays. There is recent CMake support for compiler wrappers/tools, that we can look into implementing, if so desired.\r\n\r\nTo run: `cafrun -np 1 ./zgoubi` etc.\r\n\r\n# RE: IO & profiling\r\n\r\n> I would like to understand why I/O comprises 50% of the run-time.\r\n\r\nThe answer to this question is a bit involved. I try to be concise below and provide more or less an executive summary. I *highly* doubt that this profile is representative of production zgoubi usage (but this is speculation).\r\n\r\n## TL;DR;\r\n\r\nThis is not intended to be a comprehensive performance analysis. Rather, the purpose is to verify that TAU/TAU Commander are compatible with zgoubi, and demonstrate the utility and potential for applying TAU to zgoubi.\r\n\r\nThe reason why so much time is spent in IO was not investigated in detail, however, the most likely reasons are:\r\n\r\n 1. The test case we ran here was not computationally intensive enough. Per word, the number of clock cycles wasted on IO is at least an order of magnitude larger than moving data from RAM to cache/register, which is at least an order of magnitude larger than the clock cycles per operation on a word in CPU register. Reading files (especially ascii files) is very time consuming and dependent on the system hardware.\r\n 2. The tests were conducted in a virtual machine, so there may have been additional IO slow-down/overhead associated with that\r\n\r\n## Longer answer\r\n\r\nSaying ~50% of runtime is in IO, is not 100% accurate. We observed that the only other significant time spent was in the GFortran runtime, most of which appeared to have to do with IO. We pruned sample and summary statistics from the profiles for code regions outside of those in zgoubi itself, because, they are somewhat confusing and not necessarily indicative of zgoubi's runtime performance. The generated profiles were using event based sampling with TAU & TAU Commander. The time not accounted for directly in zgoubi was either:\r\n\r\n1. In libgfortran (Fortran runtime) performing mostly what appear to be IO operations\r\n2. In system calls doing IO\r\n3. Unresolved\r\n\r\nUnlike other tools, like gprof, TAU tries to avoid time dilation artifacts caused by instrumentation/profiling. After all, what good is a profile if you're just measuring how long it takes to start and stop timers? There are different ways that TAU can be configured to collect different types of data, but by far the easiest data to collect is using event-based sampling. zgoubi looks like it has many, many very short, near-trivial procedures which may not contain a lot of computation. With event based sampling on a short, small test case, it is possible that too few, or no samples were collected in these very short procedures.\r\n\r\nIn addition, the test case run when these profiles were collected was a short one. It's more than possible that there simply is not enough computational work in the test case for the computations to take significantly longer than the IO performed in reading input files, input data and writing results. The tests were performed inside of a linux VM, which may also introduce bottlenecks associated with IO, etc. that might not be observed in a real system.\r\n\r\nFurthermore, these tests were run using OpenCoarrays with GFortran, so it is likely that there is some additional runtime overhead in libgfortran when setting up the parallel runtime. This may also account for some of the observed time in libgfortran.\r\n\r\nThe purpose of this, was not to develop a detailed performance analysis of zgoubi, but rather to ensure that profiling data could be collected with TAU/TAU Commander and show the utility and potential of these tools when working to modernize, parallelize and improve zgoubi.\r\n\r\nTo obtain a more useful and accurate profile, a test case representative of real workflows should be used, on production hardware or representative hardware. The next step is typically to use event based sampling to get a decent idea of where time is being spent. Some parameters can be tuned if the defaults are not adequate, including the sampling rate, etc. If event based sampling still appears insufficient in terms of detail and accuracy, the next step is source instrumentation. This is achieved with automatic or compiler based source instrumentation. This process has TAU & PDT go in and insert timing calls in the source code (automatically), or uses compiler options/interfaces to trigger calls to TAU's timers. This is more accurate than sampling, with the caveat that, you can cause artificial time dilation if instrumentation is indiscriminately added everywhere. TAU has throttling capabilities that help (it turns off instrumentation for procedures called too frequently, with too little time spent per call) but using an event based sampling profile, it can also generate a selective instrumentation file that will cause the automatic or compiler based instrumentation to be restricted to the important areas.\r\n\r\n# RE: Parallelism\r\n\r\n> I'm a little confused about the parallelism. What did you modify to get the embarrassingly parallel execution?\r\n\r\nWe simply compiled the code with OpenCoarray's `caf` compiler wrapper script and then ran it with `cafrun -np 4 ./zgoubi ...`. (No algorithms in zgoubi were parallelized, it is still a serial code AFAIK.) This has the effect of the GFortran runtime calling MPI_Init and MPI_Finalize to setup the behind the scenes MPI transport layer for coarray fortran. The zgoubi code itself required no modifications, it was just built with OpenCoarrays. At runtime the only real difference to spawning 4 identical zgoubi processes at the same time is the initial coordination to setup the parallel runtime and then the cleanup when each process finishes. So, essentially, the same problem is simultaneously being solved by N instances of zgoubi. "}}
{"actor":{"display_login":"zbeekman"}}