{"action":"closed","issue":{"url":"https://api.github.com/repos/ai-se/phaseDelay/issues/28","repository_url":"https://api.github.com/repos/ai-se/phaseDelay","labels_url":"https://api.github.com/repos/ai-se/phaseDelay/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/ai-se/phaseDelay/issues/28/comments","events_url":"https://api.github.com/repos/ai-se/phaseDelay/issues/28/events","html_url":"https://github.com/ai-se/phaseDelay/issues/28","id":158819364,"number":28,"title":"reply to emse","user":{"login":"timm","id":29195,"avatar_url":"https://avatars.githubusercontent.com/u/29195?v=3","gravatar_id":"","url":"https://api.github.com/users/timm","html_url":"https://github.com/timm","followers_url":"https://api.github.com/users/timm/followers","following_url":"https://api.github.com/users/timm/following{/other_user}","gists_url":"https://api.github.com/users/timm/gists{/gist_id}","starred_url":"https://api.github.com/users/timm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timm/subscriptions","organizations_url":"https://api.github.com/users/timm/orgs","repos_url":"https://api.github.com/users/timm/repos","events_url":"https://api.github.com/users/timm/events{/privacy}","received_events_url":"https://api.github.com/users/timm/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-06-07T02:31:15Z","updated_at":"2016-09-07T21:52:30Z","closed_at":"2016-09-07T21:52:30Z","body":"**DEAR AUTHORS-- THIS TEXT NOW MOVED TO PAPER**\r\n \r\n``` \r\nDear authors,\r\n\r\nThree knowledgable reviewers have scrutinized the manuscript\r\nand found it interesting and relevant,  although in need for\r\ncertain improvements. Therefore, the authors are\r\n invited to submit a revised version for a new review, where\r\n the reviewers' comments\r\nare adressed. Particularly, this holds for:\r\n\r\n- the connection to activities of theory building in ESE\r\n(reviewers 1).  \r\n```\r\n\r\n- [x ] TM\r\n\r\nThis new draft  now maps our work into the SE literature on theory building.\r\n\r\n``` \r\n- clarification about the data origin and\r\ncharacteristics (reviewer 1). \r\n```\r\n\r\n- [x ] TM\r\n\r\nThis new draft  now maps not contains extensive notes on the origin and characteristics of our data.\r\n\r\n```\r\n- use of statistical analysis (reviewers 2 and 3) \r\n```\r\n\r\n- [ x] TM  \r\n\r\nAll our results are now augmented with statistical significance tests as well as effect size tests. Those tests greatly strength the overall message of this paper.\r\n\r\n```\r\n- writing style, that would benefit form being more\r\nprecise (reviewer 3)\r\n```\r\n-   [ ] TM >\r\n\r\n```\r\nReviewer #1: PAPER SUMMARY:\r\n\r\nOne of the most widespread beliefs in software engineering\r\nis that the sooner you identify and resolve  an issue the\r\nless effort it requires (the authors call it DIE).  This is\r\nalso being taught in software engineering education; fix\r\nsomething in the r equirements phase and it is a quick fix,\r\nwait until testing and you will have a to debug source code,\r\nmodify design, update documentation, and then change the\r\nrequirement - a much more complex task. This rule-of-thumb\r\nwas expressed in the 80s, and much has changed in the\r\nsoftware business since, e.g., programming languages,\r\neditors, agile development and stack overflow. To what\r\nextent does DIE actually exist in modern software\r\nengineering projects?\r\n\r\nThe authors investigate the DIE phenomenon quantitatively by\r\nanalyzing data from 171 completed projects. All included\r\nprojects used Team Software Process (TSP), a development\r\nmethodology developed by the Software Engineering Institute\r\nat Carnegie Mellon University. A central component in TSP is\r\ndetailed time logging, i.e., a TSP developer reports how\r\nmuch time he spends on various tasks including issue\r\nresolution. Thanks to access to a large dataset of completed\r\nTSP-driven projects, the authors have been able to analyze\r\neffort spent on issues in different development phases.\r\n\r\nThe results show that DIE does not apply to all projects,\r\nand the effect is not as strong as previous publications\r\nsuggest. The authors thus conclude that DIE cannot be\r\nassumed to always hold.  Furthermore, they present five\r\nexplanations of the observed lack-of-effect: i) DIE is a\r\nhistoric relic, ii) DIE is intermittent, iii) DIE applies\r\nonly to very large systems, iv) DIE has been mitigated by\r\nmodern development methodologies, and v) DIE has been\r\nmitigated by modern development tools. Finally, the authors\r\nstate the relevant question: how could DIE become such a\r\nwidespread idea in the literature, despite limited empirical\r\nevidence?\r\n\r\n\r\nREVIEW SUMMARY: + The paper investigates whether a widely\r\naccepted \"rule\" in software engineering holds by looking at\r\na large set of empirical data. The research topic is highly\r\nrelevant and the results are interesting.  + The discussion\r\non reassessing old truism fits the scope of the journal very\r\nwell.  + The language in the paper is excellent, and the\r\nstructure is (although non-standard) easy to follow.\r\n```\r\n\r\n- [ ] TM Thank you for those comments.\r\n\r\n```\r\n- It is not easy for the reader to understand what parts\r\nof the \"life of an issue\" the authors claim are affected by\r\nDIE, and it is also not too easy to understand what data the\r\nauthors have available. I believe adding another figure\r\ncould address both these aspects.  ```\r\n```\r\n\r\n-  [X] @llayman: We have added an example of the DIE in the Introduction, and more precisely defined how we measure this effect Section 6. \r\n\r\n```\r\n- The different projects studied are all grouped together. I\r\nbelieve studying clusters of development contexts would be\r\nhighly interesting, but unfortunately the current\r\ncharacterization of projects is inadequate.  \r\n```\r\n\r\n- [ ] @williamNichols: need some way to group the projects\r\n\r\n```\r\n- The authors do not relate their reassessment of an\r\n\"old truism\" to the growing set of papers on theory building\r\nin software engineering. With access to such large amounts\r\nof empirical evidence, it should be possible to take some\r\nsteps toward improved DIE theories. \r\n```\r\n\r\n[ ] @fshull need words on this\r\n\r\n```\r\nDETAILED REVIEW: The authors should expand on what they\r\ninclude in the \"delayed issue effect\" (DIE), in particular\r\nin relation to standard milestones in the life of an issue\r\nand the corresponding timespans. I would suggest adding a\r\nfigure to point out important issue milestones on a\r\ntimeline, e.g., 1) injected, 2) found/reported, 3) work on\r\nresolution started, 4) work on resolution ended, 5) V&V\r\ncompleted, 6) resolution deployed/integrated. I believe the\r\nauthors refer to an increase in time between 3) and 4) as\r\nthe DIE, but I'm not really sure. I'm particularly\r\ninterested in whether all \"indirect\" issue resolution steps\r\nare covered, e.g., change impact analysis, updating\r\ndocumentation, and regression testing. The effort involved\r\nin these steps clearly depends on the development context of\r\nthe specific project. The short a-c) list on page 14:50\r\nsuggest that the issues studied are restricted to minor\r\nfixes, i.e., no updates to architecture, changed hardware,\r\nrecertification, updated user manuals etc.\r\n\r\nA figure with issue milestones could also help the reader\r\nunderstand what data are actually available for the\r\nanalysis. Section 6.3 describes the data, but some aspects\r\nshould be further explained. Logging interruption time\r\nsounds very useful, but I wonder how carefully the\r\ndevelopers actually did this -\r\n ``` \r\n\r\n- [X] @llayman: We not provide a more explicit definition of what constitutes a defect (our study of issues is limited to defects) and the \"defect lifecycle\" phases we measure in Section 6.2.2. \"a defect is any change to a product, after its construction, that is necessary to make the product correct\". This includes both minor issues (typography) and severe issues (architecture changes, requirements errors). Time on a defect includes a) time to investigate/analyze a defect once discovered, b) time to craft and implement a fix, and c) time to verify and close the defect.\r\n\r\n```\r\n it must be really hard to keep up the discipline required.\r\n If you are interrupted as a developer (e.g., phone calls,\r\n urgent mail, someone asks a question) I don't think the\r\n first thing you do is to stop a timer on your computer.\r\n Moreover, developers often work on several defects in\r\n parallel, and might interweave bug fixing with new\r\n development. I don't think the \"interruption time\" captures\r\n all such multi-tasking, and it should at least be properly\r\n discussed in Section 7 \"Threats to validity\".\r\n```\r\n\r\n- [X] @llayman Yes, there is undoubtedly some accuracy in the time-tracking. We provide some evidence on the TSP data set's integrity in Section 6.3, and a more extensive discussion on the issue of time-tracking and defect logging accuracy in Section 7.1 on Conclusion Validity.\r\n\r\n```\r\nThe discussion on reassessing old truisms (Section 3) is\r\ninteresting, but it should be complemented by the\r\nperspective of theory building in software engineering - An\r\nactive research topic lately, with a dedicated workshop\r\nseries (GTSE). I suggest looking into the following\r\nreferences for a start: Sjøberg et al. (2008) \"Building\r\nTheories in Software Engineering\", Smolander and Päivärinta\r\n(2013) \"Theorizing about software development practices\",\r\nand Stol and Fitzgerald (2015) \"Theory-oriented software\r\nengineering\". Considering new empirical evidence is\r\nobviously critical to theory building, and discussing your\r\nnew results in the light of an theory creation would be\r\nvaluable. The authors mention that DIE appears to occur\r\nintermittently in certain kinds of projects - maybe the\r\nauthors could elaborate on this idea and present an improved\r\nDIE theory based on what they now know? I believe the\r\nauthors have the best available data to do so, and I would\r\nexpect the paper to go beyond simply questioning the \"old\r\ntruth\". \r\n ```\r\n\r\n- [ ] @fshull:   is this material you know? can you do a\r\npage or two casting this work in terms of t: Sjøberg et al.\r\n(2008) \"Building Theories in Software Engineering\",\r\nSmolander and Päivärinta (2013) \"Theorizing about software\r\ndevelopment practices\", and Stol and Fitzgerald (2015) \"T\r\n\r\n```\r\nThe authors study three claims in this paper, and the\r\nthird claim is the central one: \"delayed issues are not\r\nharder to resolve\". To study whether issues require longer\r\nresolution times in later phases, the authors analyze a\r\nlarge set of issue data from historical projects. Thus the\r\nmanuscript reports from an observational study rather than\r\nan experiment with a controlled delivery of treatments.\r\nWhile I believe the authors' approach is practical, I would\r\nlike to see a critical discussion on threats to validity of\r\nobservational studies (e.g., Madigan et al., A Systematic\r\nStatistical Approach to Evaluating Evidence from\r\nObservational Studies, Annu. Rev. Stat. Appl. 2014. 1:11-39\r\nand Carlson and Morison, Study Design, Precision, and\r\nValidity in Observational Studies, J Palliat Med. 2009 Jan;\r\n12(1): 77-82). An alternative study design (although\r\ndifficult to realize in a system of industrial size) would\r\nbe to let different developers resolve the same issues for\r\nthe same software system, i.e., one group resolves an issue\r\nduring design, another during implementation, and a third\r\nduring testing - some discussion along these lines would\r\nstrengthen the validity section. \r\n```\r\n\r\n- [X] @llayman We have reviewed and cited the mentioned papers. We have incorporated many of the concerns raised in these papers into our expanded Section 7.4 on External Validity.\r\n\r\n```\r\nThe first and second claims are studied with much less\r\nrigor. \"DIE is a commonly held belief\" is studied using a\r\nsurvey of software engineers, both practitioners and\r\nresearchers. According to Fig. 2 (actually a table), the\r\nnumber of respondents is 16 and 30 for practitioners and\r\nresearchers, respectively. Sixteen respondents from industry\r\nrepresent a tiny survey of a very general claim that any\r\nsoftware engineer could respond to. Why were not more\r\nanswers collected? The authors do not have much evidence to\r\ndefend the first claim, and there is no discussion of the\r\ncorresponding validity threats. The second claim, \"DIE is\r\npoorly documented\", is studied using a literature review.\r\nUnfortunately, the method underlying the literature review\r\nis not presented. Although it doesn't need to be an SLR of\r\nthe most rigorous kind, the authors should report how the\r\npapers were identified. I suspect the terminology used to\r\ndescribe the DIE phenomenon is highly diverse, thus it would\r\nstrengthen the paper if the authors reported how they\r\nreviewed the literature. According to Page 19:34 only eight\r\npublications were identified. \r\n```\r\n\r\n- [ ] @llayman need to show that these were the only empirical\r\nresults for DIE in tow\r\n\r\n```\r\nThe iterative fashion of modern software development,\r\nwith agile at the extreme end, is not fully discussed (the\r\nshort discussion section could be extended). The phases of\r\nthe linear development of the 80s, (such as in Fig. 1)\r\nprobably don't exist in many of the projects in the TSP\r\ndataset, still the authors discuss the DIE-effect from the\r\nperspective of the 80s. Page 16:31 states \"DIE has been used\r\nto justify many changes to early lifecycle software\r\nengineering\" - does this mean the agile movement\r\nsuccessfully mitigated DIE? This possibility is not fully\r\nconsidered in the paper.\r\nHow many of the 171 projects were\r\n(more or less) agile? This appears to be an important\r\ncharacteristic of the included projects - very important to\r\ndescribe! \r\n```\r\n\r\n- [X] @llayman An astute observation. We provide more information on the types of projects included in our sample in Section 6.4. More to the point -- does agile help mitigate DIE? The answer is - we don't know. A strong argument can be made that all iterative&incremental methodolgies are meant to mitigate the DIE. Further, huge advances in software engineering technology and process have occurred since DIE was first observed, many of which have precisely targeted the DIE (static analysis, test-frist development, automated build). Perhaps our analysis provides evidence that DIE can be defeated, at a large scale across multiple varied projects, using modern technologies. We do not yet know the causes here, but certainly it appears the DIE should not be treated as a truism any more, but rather a variable that can be controlled. We discuss this in Section 8.\r\n\r\n- [ ] @WilliamNichols : need a quesstimate of how many of the\r\nprojects were \"agile\"\r\n\r\n```\r\nConcerning characterization of the 171 projects, the\r\npaper needs to report much more detail. I would expect to\r\nsee some descriptive statistics. On page 17:18 the authors\r\nsay \"perhaps we should split our SE principles into two sets\r\n/---/\" - of course SE practices need to be adapted to the\r\ndevelopment context, and also two sets of principles is a\r\ntoo simple split. The paper does not report much\r\ncharacterization of the 171 projects. I strongly suggest the\r\nauthors to dig deeper into the data, and analyzing for which\r\ntypes of projects the findings hold. What patterns are there\r\nto discover? I suggest Petersen and Wohlin, \"Context in\r\nindustrial software engineering research\" (ESEM2009) for\r\ndetails on how to characterize development contexts.\r\nMoreover, I would really like to see what practitioners from\r\nthe 171 projects think of your findings - an interesting\r\noption for a qualitative angle on your study. \r\n```\r\n\r\n- [ ]@WilliamNichols : can you look at p402 and 403 of\r\nhttp://www.cse.chalmers.se/~feldt/courses/sple/papers/petersen_2009_context_in_industrial_research.pdf. do you have any statistics on our projects broken out according to the terms of that paper?\r\n\r\n```\r\nFig 1: \"Operation\" dwarfs everything in Boehm's diagram.\r\nSince you do not study anything post release in this paper,\r\nI think this figure skews the reader's mental picture. If\r\nyou remove the rightmost bar and rescale accordingly, the\r\nplot better matches the findings you report in Fig. 10. You\r\nstill identify an interesting result though, but the\r\npresentation turns fairer. \r\n```\r\n\r\n- [ ] @timm will do\r\n\r\n```\r\nMINOR COMMENTS: Keywords: I believe the keywords could be\r\nimproved to help readers find this work.  \r\n```\r\n\r\n- [ ] @llayman\r\n\r\n```\r\nSeveral figures are copied from previous work. Are all copyrights properly\r\nmanaged?  \r\n```\r\n\r\n- [ ] @llayman\r\n\r\nFig 10: Black and red is not a good pair for\r\ngrayscale printouts. Please use black and gray instead.\r\n```\r\n\r\n- [ ] @timm \r\n\r\n```\r\nSome figures are actually tables, thus their captions should\r\nbe replaced accordingly.  \r\n```\r\n\r\n- [ ] @llayman\r\n\r\n```\r\nSome figures should be resized to better match the page width.  \r\n```\r\n\r\n- [ ] @llayman\r\n\r\n```\r\nPage 3:14 - \"The above argument\" Which argument? Could be precisely specified.\r\n```\r\n\r\n- [X]\r\n\r\n```\r\nPage 3:22 - Please provide the full link to the dataset.\r\n```\r\n\r\n- [ ] @timm \r\n\r\n```\r\nPage 4:8 - \"More difficult\" but \"harder\" is used in claim3. Why this inconsistency?\r\n```\r\n- [ ] \r\n\r\n```\r\nPage 18:26 - First sentence: \"Unexpected results such as this one\". Ambiguous reference,\r\nplease be specific.  \r\n```\r\n\r\n- [ ]\r\n\r\nPage 18:27 - \"We also survey the state of SW dev. practice /---/\": given the size of the survey,\r\nthis statement feels a bit bold.\r\n\r\n- [ ] \r\n\r\n```\r\nTYPOS ETC.  \r\nSec 1, §1: [3] and [30] have been swapped?  \r\nPage 14:46 - Spell out IV&V the first (and only) time it appears.\r\nPage 3:24 - missing verb (is) Page 16:43 - \"Did this study failed\" \r\nPage 18:44 - Appache \r\nPage 19:19 - Al Ref 26 - oo --> OO\r\n```\r\n\r\n- [X]\r\n_____________________________________________\r\n\r\n```\r\nReviewer #2: The topic of this paper is the empirical\r\nevaluation of a largely admitted claim in software\r\nengineering, i.e. the exponential cost of correcting errors\r\naccording to the phase in which the errors are discovered.\r\nThe claim is first confirmed by surveying and interviewing\r\npractitioners and experts. Next, the authors use a large set\r\nof data from projects in SEI database. The results show\r\nthat, strictly speaking, the claim does not hold (although\r\nsome effects of delayed corrections can be noticed).\r\n\r\nGlobally, the paper is well written and the authors develop\r\na convincing demonstration. The topic of the paper is highly\r\nrelevant for both practitioners and people from academia.\r\nAmong many, I personally believed in this claim and had\r\nregularly taught it in my courses. Thus, reading this paper\r\noffers a refreshing perspective on our understandings of\r\nsoftware engineering background and theoretical knowledge. I\r\nalso like the suggested posture that insists on the\r\nnecessary skepticism that we should have against such\r\nprevailing claims.\r\n\r\nBeyond this positive global impression, I have some concerns\r\nabout the study reporting and analysis: -  It seems to me\r\nthat more descriptive statistics about the sample would\r\ncontribute to a better understanding of the scope of the\r\nstudy. \r\n```\r\n\r\n- [ ] @timm more  stats\r\n\r\n```\r\nI think in particular about the size (lines of codes and/or\r\n number of software components); a duration histogram would\r\n also be relevant. Accordingly, the formula for calculating\r\n variable \"Total effort\" in Fig. 7 could be explained, and\r\n Figure 9 made more readable (bigger size).\r\n```\r\n\r\n- [X] @llayman We have added more description of the projects to Section 6.4. We have also added more precise definitions of our measures in Section 6.2. We have adjusted the size of figures throughout the paper to be more readable.\r\n\r\n```\r\nIf certain attributes for the issues and errors are\r\navailable in the data (e.g. severity, priority, etc.), they\r\nshould also be brought to the readers knowledge.  \r\n```\r\n\r\n- [ ] @WilliamNichols: got any stats on these\r\n\r\n```\r\nFigure 10 is central to the paper's demonstration; its\r\nexpressiveness could be enhanced: i) the reader tends to\r\nthink that the \"right hand side bars\" are different from the\r\nBLACK and RED bars, ii) the formula for calculating the 50th\r\nand 95 ratio could be provided, iii) the unit of column\r\n\"Percentile\" could be mentioned. \r\n```\r\n\r\n- [ ] @timm split into two\r\n\r\n``` \r\nI like the discussion section, in particular, the idea\r\nof software architectures that could be a contributing\r\nfactor for enhancing software evolution and reducing the\r\ncost of issues and errors fixing. Is there any available\r\nknowledge in the data set concerning any architectural\r\ndesign choices made in each project? If so, would it make\r\nsense to seek any correlation between these architectural\r\nchoices and how expensive it was to solve issues?  \r\n```\r\n\r\n- [ ] @WilliamNichols: anything on architectural style? or should\r\nwe jsut say \"many and varied and hard to get a precise\r\npicture of it all\r\n\r\n``` \r\nIn the same trend of ideas, I think what has\r\nfundamentally changed since early days of SE is the\r\ndevelopment of requirements engineering. In actual SE\r\npractices, problem and the solution spaces are\r\nsystematically explored thanks to RE techniques; together\r\nwith architecture, this could also explain why things have\r\nchanged since the 70'. It could be, for example, that people\r\ntend to make less severe errors in early software project\r\nphases (thanks to RE techniques); a similar phenomenon was\r\nobserved when SE practices become more mature (see Harter et\r\nal. 2012).  \r\n\r\nReferences Harter D. E., Kemerer C. F., Slaughter S. (2012).\r\nDoes Software Process Improvement Reduce the Severity of\r\nDefects? A Longitudinal Field Study. IEEE Transactions on\r\nSoftware Engineering, vol. 38, n° 4, p. 810-827.\r\n```\r\n\r\n- [ ] @timm reference harter\r\n- [X] @llayman Absolutely - there are potentially many explanations for why the DIE was NOT observed in our dataset. We discuss some more of the potential causes in Section 8.\r\n\r\n``` \r\nIn the conclusion section, I feel uncomfortable with\r\nthe assertion \"That data held no trace of the delayed issued\r\neffect\" (line 17). As mentioned elsewhere in the paper, the\r\ndelayed issue is not absent; it is much less significant and\r\nsystematic than what is usually claimed. Moreover, this\r\nreduction has been demonstrated for medium sized projects;\r\nwe cannot generalize to larger projects.  \r\n```\r\n\r\n- [ ] @timm  \r\n\r\n``` \r\nMinor remarks: - p.9, line 29 : \" … reported by Shull [52], found that the cost to find certain non-critical classes of defects …\" => is it the \"cost to find\" or the \"cost to FIX\"?  \r\n```\r\n\r\n- [X] @llayman It should be \"cost to fix\". We have corrected this.\r\n\r\n```\r\n- p.10, line 21: duplicate word \"there\" -\r\np.12, line19: missing word \"One of THE guiding …\" - p.16,\r\nline 33: \"distinguish\" instead of \"distinguished\"?  - p.18,\r\nline 42: explain acronym MEAN - p.20, line 20: useless\r\nenumeration A1?\r\n```\r\n\r\n- [X] \r\n_______________________\r\n\r\n```\r\nReviewer #3: I like the idea of the paper and I certainly\r\nwould like to see more empirical studies that periodically\r\ncheck if our beliefs about software engineering practices\r\nstill hold (or even if they have ever held). The authors\r\naddress one of the beliefs that are more entrenched in the\r\npopulation of researchers and practitioners, and a very\r\nimportant one too.\r\n\r\nAt the same time, I think that the paper needs to be\r\nimproved before it can be published.\r\n\r\nThe empirical analysis (Section 6) is a bit of a let down. I\r\nwould like to see some sort of more robust and rigorous\r\nstatistical analysis, based on statistical tests. Instead,\r\nthe paper only provides qualitative comparisons between the\r\nresults obtained at the 50-th and 95-percentiles. This is\r\nnot to say that the results or the discussions are\r\nincorrect, but only that they need to be better supported.\r\nThe lack of this kind of statistical analysis should have\r\nleast be mentioned in the Threats to Validity. I understand\r\nthat you say that your \"claims\" (including Claim 3) should\r\nnot be considered \"hypotheses\" in the statistical sense and\r\nI agree with you on Claim 1 and Claim 2. However, Claim 3\r\nlends itself to a statistical analysis, without which your\r\nresults become a bit more anecdotal, and this is what you\r\nrightly criticize in the previous claims that DIE actually\r\nexists. In addition, you may want to analyze some additional\r\nand somewhat unexpected results that can be derived from\r\nyour data (see Detail Comments below).\r\n```\r\n\r\n- [ ] do the stats\r\n\r\n```\r\nThe paper seems to be written in a somewhat casual style,\r\nwhich makes the paper more pleasant to read, but less clear\r\nin some parts (see Detail Comments below).\r\n\r\n\r\n\r\nDETAIL COMMENTS\r\n\r\nSection 2\r\n\r\nI'm not really sure what the authors mean by \"We say that a\r\nmeasure collected in phase 1, ., i, .. j is very much more\r\nwhen that measure at phase j is larger than the sum of that\r\nmeasure in earlier phases 1 <= i < j.\" You are defining a\r\nproperty of a measure, but, the way this definition is\r\nwritten, it seems as if you are defining the property of a\r\nmeasure of \"being much more,\" without any further\r\nqualifications.  I can guess that you mean that a measure\r\n$m$ (collected in phases, so you can denote by $m(i)$ the\r\nvalue of $m$ in phase $i$) has this property *in phase $j$*\r\n(so, it's a property m has in a specific phase and not a\r\ngeneral property of the measure) if $\\sum_{1 <= i < i} m(i)\r\n< m(j)$. Then, it's up to you to make this a property of the\r\nmeasure, for example by using an existential \"policy\" ($m$\r\nhas this property if there exists one phase $j$ where\r\n$\\sum_{1 <= i < i} m(i) < m(j)$) or a universal one ($m$ has\r\nthis property if for all phases $1<j$, $\\sum_{1 <= i < i}\r\nm(i) < m(j)$).\r\n``` \r\n\r\n- [ ] @timm just do that\r\n\r\n```\r\nThe real problem with this definition, however, is that it\r\ndoesn't seem to be used anywhere in the paper. The only\r\npoint where it might be used is in the discussion of Figure\r\n1 (which precedes it anyway). So, you may want to remove\r\nthis property altogether.\r\n```\r\n\r\n- [ ] @timm your right. the real thing here is that here is\r\nstatistically singicicat increase \r\n\r\n```\r\nYour definition of \"difficult issue\" is \"Issues are more\r\ndifficult when their resolution takes more time or costs\r\nmore (e.g. needs expensive debugging tools or the skills of\r\nexpensive developers).\" That's not a very precise\r\ndefinition, though, because it has two different\r\ninterpretations, one in terms of time and the other in terms\r\nof effort. The reasons why you introduce this definition\r\nbecome clearer only in Section 7.2, in the Threats to\r\nValidity, and that's too late. You should move some of the\r\ndiscussion of Section 7.2 here.\r\n```\r\n\r\n- [ ] @timm roger\r\n\r\n```\r\nWhy would the term \"delayed issue effect\" be a\r\ngeneralization of the rule \"requirements errors are hardest\r\nto fix\"? \"Delayed\" seems to be quite specific as it appears\r\nto refer to time, while \"hardest to fix\" may refer to other\r\nvariables, like effort or cost.\r\n```\r\n\r\n- [ ] @timm text\r\n\r\n```` \r\nIn Claim 3, \"very much more harder\" sounds like a\r\nlittle bit too much ... (\"more harder\"?)\r\n```\r\n\r\n- [ ] \r\n\r\n```\r\nSection 4\r\n\r\nA very minor issue: why is Figure 2 a ... figure, instead of\r\na table? Same for all other tables ...\r\n```\r\n\r\n- [ ]\r\n\r\n```\r\nSection 5.1\r\n\r\nThis is not a complete sentence \"All the literature\r\ndescribed above that reports the onset of DIE prior to\r\ndelivery.\"\r\n```\r\n\r\n- [X] \r\n\r\n```\r\nSection 6.3\r\n\r\nYou should rephrase your definition \"a defect is any change,\" since a defect is what existed before the change was made and not the change itself.\r\n```\r\n\r\n- [X] @llayman We now provide a more precise definition of defect in Section 6.2.\r\n\r\n```\r\nYou do not explain that \"QualTest\" in Fig. 8 is.\r\n```\r\n\r\n- [ ]\r\n\r\n```\r\nThe definition of \"time per defect - The total # of defects found in a plan item during a removal phase divided by the total time spent on that plan item in that phase.\" is not correct as it is, as this would basically be the number of defects per unit time, instead. The roles of time and\r\ndefects should be reversed.\r\n```\r\n\r\n- [X] We now provide a more precise definition of time measurement related to defects in Section 6.2.\r\n\r\n\r\n```\r\nSection 6.4\r\n\r\nBesides adding a more rigorous statistical analysis, you\r\nneed improve your explanations.\r\n\r\nFigure 9. The caption says \"Distribution of defects found\r\nand fixed by phase.\" Is that the distribution of defects\r\nfound in some phase and fixed in that phase? The text says\r\n\"The distribution of defects found and fixed per phase in\r\nour data is shown in Figure 9. A high percentage of defects\r\n(44%) were found and fixed in the early phases, i.e.,\r\nrequirements, high level design, and design reviews and\r\ninspections.\" which doesn't add much. If that's the\r\ndistribution of defects introduced in a phase and fixed\r\nimmediately in that phase, it is not clear why you introduce\r\nit, though.\r\n```\r\n\r\n- [ ]\r\n\r\n```\r\nFigure 10. The explanation of the data and histograms in the\r\nfigure is, at best, confusing. The caption says \"50th and\r\n95th percentiles of issue resolution times\" and the opening\r\nsentence of Section 6.4 says \"Figure 10 shows the 50th and\r\n95th percentile of the time spent resolving issues ...\"\r\nHowever, these are not \"times,\" because you write \"expressed\r\nas ratios of resolution time in the phase where they were\r\nfirst injected\" a couple of paragraphs below. This seems to\r\nbe the right interpretation, but right after that you write\r\n\"The BLACK and RED bars show the increases for the 50th\r\n(median) and 95th percentile values, respectively,\" which\r\nwould be a third interpretation of the data in Figure 10.\r\n```\r\n\r\n- [ ]\r\n\r\n```\r\nYou should provide the value of the mean, in addition to the\r\n50-th percentile (i.e., the median).\r\n```\r\n\r\n- [ ]\r\n\r\n```\r\nYou need to at least mention (or, better, discuss) some of\r\nyour results, because they appear to be \"counterintuitive,\"\r\nsince they show that, for example, median resolution times\r\ncan even decrease if issues are not fixed immediately, but\r\nin later phases. For example, looking at the Reqts section,\r\nmost percentages are way below 1, which seems to indicate\r\nthat some issues are actually much easier to fix at later\r\nstages (maybe because more or better information about the\r\nsoftware system is available only later). That would be a\r\nvery interesting result by itself, especially if you can\r\nprovide some statistical support for it.\r\n```\r\n\r\n- [ ] @timm really, no statisticall difference\r\n\r\n```\r\nSection 7.1\r\n\r\nMaybe you could analyze the projects developed in a\r\n\"traditional\" way and check if there is some sort of DIE\r\nthere. For example, if you look at the projects developed\r\nwith the waterfall life cycle, maybe you will find larger\r\nDIE than for the other life cycles, which might partially\r\njustify the claims of previous researchers about the\r\nexistence of DIE. This would also show that newer kinds of\r\nlife cycles (like Agile ones) help solve DIE. You do provide\r\nhints about this in Section 8 (\"We also note that other\r\ndevelopment practices have changed in ways that could\r\nmitigate the delayed issued effect\") and Section 9, but\r\nthat's clearly not enough.\r\n```\r\n\r\n- [ ] @williamNichols: can we separate out the agile and the trad?\r\n\r\n```\r\nSection 8\r\n\r\nI'm not sure what you are really trying to discuss in this\r\nsection, because you are not discussing the results of the\r\npaper. It sounds like you are saying that maybe it's the\r\nnewer software development approaches that made DIE\r\ndisappear?\r\n```\r\n\r\n- [ ]\r\n\r\n\r\n- [ ] Verify that the Section #'s in our responses are accurate prior to submission.\r\n"}}
{"actor":{"display_login":"llayman"}}