Reinventing software quality
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Adzic, Gojko
The needs to go faster to market and cut costs, dominating the industry lately, are significantly changing the landscape of software quality. Many pillars of successful quality strategies in the past are just slowing down delivery too much. We will have to find better ways to provide build in quality into our systems, track and assure it.Join Gojko Adzic to learn how you can reinvent your approach to software quality and scale delivery processes by engaging business users better, enhancing visibility, collaboration and knowledge transfer. This is a talk for anyone serious about software delivery, regardless of the role in their process.
 Yeah, fantastic. Okay. Hello. Thanks for coming. What I want to talk about now is a bunch of questions I have. I don't really have any answers to these things. It's kind of talking to lots of different people and doing conference talks over the last few years. I've started kind of thinking about that we got software quality completely wrong. One of the things that was really interesting is that I got a call two years ago to do a keynote at an internal company talk by one of the secretaries called me and I started trying to understand what they really want. She said, well, I'll put in touch with this guy who is the director for that thing. I talked to him and essentially the guy told me that he saw me speak at some other conference and I was able to make people laugh. So they scheduled me after lunch and if I can make people wake up after lunch. He didn't really care about what I was talking about. So I said, okay, I'll come up with some good material. I started trying to kind of make fun of how we approach software quality. When I really started talking about all these jokes that I wrote for them, I couldn't kind of disprove most of it when I started seriously thinking about it. So that got me thinking about this and this is part of my kind of reinventing software quality tour. I've submitted this talk to almost every conference I'm speaking at this year and I'm refining it as I go with the idea that I'll make myself clean up things. So kind of halfway it was changed from the last time I did the talk. And I said, I still do not have any answers to these things but hopefully I'll provoke some of your thinking and then if I do, let me know if you think you have answers to these things. And software is everywhere now. Absolutely everywhere. Everything relies on software now. And we are so used to software being wrong that we don't even think about it anymore. Like at this railway station where one of the displays is asking us to put a disk drive in C while the other is showing us that we should have a train arriving on time. And another example that's really interesting is kind of this hero of mine who was able to rewire an ATM to play Solitaire on it. I was recently in Italy and I had to take out some cash from a cash machine and I had a situation very similar to this where I've put in my PIN code and the machine started validating my credit card, whatever, and then a Windows exception popped up and I said, oh shit. Five seconds later there was a message in German that I kind of used my phone to translate to something like we've taken your card, call your bank, and I was like, oh no, no, no, no, no, no, no. You know, my wife and two kids are waiting there. I have to take cash to pay for a taxi. What's going on? As I was trying to call my bank, I saw a dialogue OK cancel. So at this point, if you've ever done any software testing, you have to press the cancel button. So I pressed the cancel button and then I heard 300 euros came out. My card came out. All I could say is thank you, Microsoft Windows. But transaction has never showed up on my bank account. I was really, really happy at that point. So software quality has lots of different aspects for lots of different people. For me, that was one of the best qualities of Windows I've ever seen in my life. For somebody else, they were not that happy. One thing that's really, really puzzling is that there are certain things that are now going on in the industry and I think we'll have to change a lot of how we approach software. And in particular, our thinking about software quality will need to change because it's no longer going to be applicable. One of the trends that's happening in the industry over, at least from my experience, perhaps it was before, I don't know, is that the phases of software development, the phases of software releases are shortening all the time. 10 years ago, most of the teams I worked with were maybe releasing every six months. The delivery phases of two years were relatively normal. Most people I worked with these days want to release two weeks to a month. Two weeks seems to be the default scum, whatever that means today. All teams I worked with will release several times a day. So there's a general shift towards reducing the phases. Now, when we start reducing the phases, certain things that used to apply no longer apply. For example, one of the most accepted best practices today is using a satna while we drive. One of the things that we do is we get a lot of people who don't even drive without a satna. This is where we get a lot of these insane reports where somebody's car ended up in a river because the satna told them to go left. Although, arguably, there was a river left and they didn't care about that. So I was on a holiday in France last year and we spent some time in Provence. It was beautiful countryside, lots of really nice views, very nice villas and some hilltop castles. We got into the car to go and see this castle that was five kilometers away. We have this stupid tom-tom that for the first 10 minutes when you turn it on, has an identity crisis. Is it really a satna or is it just a stupid box? So as we were driving, the tom-tom is still looking for satellites. I'm like, okay, they're fucking there. So we see this castle that we wanted to see visibly passing. We're passing it. So my wife decides to take a risk and drive without a satna and go into the road left visually, navigating towards the castle. One minute later, the road goes narrower, narrower, narrower and about two minutes later, we end up in a ditch like this. And I got out and started to push the car out. You can imagine a scene where I kind of just mud everywhere. I'm pushing the car out. My wife is swearing. She's in full reverse. The kids are screaming and tom-tom wakes up at that point and says, turn around when possible. So at that point, I did something that's very, very difficult to explain because I'm from Serbia and we are known for our tolerance and understanding. So I got out of the ditch, got the tom-tom and started smashing it like crazy. And I did this for two main reasons. Reason number one is I don't really need a satna to tell me when I'm lost. I need the satna to prevent me from being lost in the first place. Reason number two, the people who are married in this audience, you can vouch for this being true that in a situation like that, it is a lot easier to smash the satna than blame your wife for not knowing how to drive. So what I'm trying to say is that satna is generally a best practice, but when your journey is five minutes long, that best practice no longer applies and can even get you in a worse situation than you would be if you wouldn't use that. And I think the way we are shrinking our delivery phase is going to make us rethink really hard about everything we do in terms of software quality and whether these things really still apply. So the second important trend in the industry is that everybody's talking about cost reductions. Every single team I talk to, most of what they're focused on is reducing cost, which means that if we need to reduce cost, we have even less money and kind of time is money to spend on things we want that are not really essential. So stuff like, okay, going back and refactoring or cleaning up code, it's even more difficult to justify than before. And stuff like, okay, let's redo our automation or let's do this or let's spend a lot of time rewriting a test. It's very, very difficult to justify today. The third thing that's happening that I think has already happened in a sense, but we don't really see that yet that much is that software quality is really information art. What I mean by that is sometime ago we had this idea that we could do quality control and then we realized that's not going to work. And then people had this idea, okay, we can do quality assurance. But that didn't turn out to work as well. And I think kind of what a lot of people are moving to now is that software quality is about distributing information, informing the right decisions. Elizabeth Hendricks on a Tegel testing days 2009, I think, said that people who ask for control genuinely want visibility. That was an interesting statement, so I decided to put that to test. And I worked with one of the biggest media companies in the UK where they were complaining about this horrible, horrible, horrible publishing system for their website. Lots of problems all the time. They wanted to rewrite it. They wanted to redo the automation and everything. And I said, okay, let's think about this whole quality problem. You're complaining that it's very bad quality. How would we visualize this? How do we show this problem before even starting to solve it? What does this mean? And after a few weeks of trying to negotiate that, we established that one of the biggest problems the way people perceive quality is that content publishers, because of all the bugs in the system and some business rules that are obscure and all things like that, sometimes they publish an article, it shows up immediately. Sometimes they publish an article, it doesn't show up in 10 minutes time, so they publish it again. And then it doesn't show up in five minutes time, they publish it again, and then 45 minutes from then you have the same article published on three places on the homepage, which is embarrassing. And sometimes it doesn't go out at all. Sometimes it was stuck because of a bug and sometimes it was because of genuine business rules because they don't want to overload their website with stuff that stays there for five seconds. So if you publish lots of articles, the system is going to delay that. And this was the essence of what people complained against, and they wanted to rewrite the whole system because of that. And they said, okay, that's the problem. How do we visualize this problem? And one of the business users said, well, if you can tell me the time, how long it's going to take for this article to be published, that's a good visualization for me. And we talked to developers, could we estimate that? And I said, well, we could estimate maybe the business rules, but then again, lots of bugs. And it's going to be very difficult to estimate that maybe we can show the last article that got published. We have that in the database. We have the timestamps. So is that good enough? And we said, I don't know. Let's try it. So we bought two big monitors. One was in the development room. One was in the content publishers room. And it was continuously refreshing a single web page with a single SQL query, looking at the difference between the last timestamp article when it got published and when it was queued. All of a sudden, all the complaints stopped. The system didn't get cleaned up. It was still the same bad system. It still had the same problems. People stopped complaining. The quality of the system jumped incredibly. What had happened is that the content publishers would look at this and say, okay, 15 minutes, I'm not going to complain about it until 15 minutes pass. I know how long it's going to take. The developers would look at the monitor and say, oh, 15 minutes, something's gone wrong. So they would massage it through so that people don't really notice any problems. And then they would try and understand why that happened and clean it up so it doesn't happen next time. And just by visualizing this thing, we were able to increase the quality as kind of any measure they had, it increased without really changing anything. So that got me started thinking about, well, was a rewrite of their system and an investment in huge test automation really the right thing to do or not? Because they could have invested months and months in doing good test automation or doing kind of better code architecture or doing everything else, but was that the right investment to make? Because we solved the problem. When we talk about information, then the question is really, are we distributing the right kind of information? And for a lot of teams now, the test team or testers or whoever does quality, they're basically a mirror. A mirror in terms of is our architecture correct, is our code correct, is my design good, are all the business functions there, that's kind of a mirror. And when we talk about mirrors, one of the things that I always remember is my friend Nick from high school. So when I was in high school, we were going out together as a group, there was 20 of us, and my friend Nick was always the last person to arrive. Now this was before mobile phones, and again, technology software is now everywhere. Before mobile phones, you couldn't really call him to shout at him and see why he's late. So you had to wait, and we waited and waited and waited, and Nick was always the last person to come. Sometimes he was 45 minutes late. And the reason why he was late all the time is he was changing shoes. So Nick held the whole wardrobe full of shoes and kind of had this small mirror where for, I don't know how long, every time he wanted to go out, he would kind of spend deciding what shade of black to wear. And the idea was that if he only had the right type of shoes, he would get lucky tonight. He never, ever, ever did. The reason why he didn't do that was that the choice of shoes was completely relevant, because Nick was exactly as a cave troll. So kind of this is what we do, what a lot of organizations that I worked with do. They point a very small mirror to a place that they like, and then they optimize that incredibly. But they leave the big, ugly troll phase that's still the biggest problem there. So I recently worked with one very big software development company in Germany, and they said, well, we do three-week scrums. It's okay. What do you do inside those three-week scrums? And they said, okay, there's good stuff going on there, but they were still complaining about lots of quality problems, business people were unhappy. And I said, okay, let's kind of, this three-week scrum looks very development-oriented. Everything there is around development. How long does it take you from when a business person says, I want this till it goes live? And they said, well, we don't know. So, well, can we find out? And we could look at the requirement system where can they keep these things before they become user stories, and we could possibly look at the deployment date for something. Let's do that. Two and a half years. So there's crum of three weeks that they've optimized, and they do lots of really good stuff that doesn't really matter. Because they have two years and one month of business stuff before that, and they have three months of testing after that, where kind of the business people keep the software to marinate a bit so that it becomes stable enough to kind of go live. And this is not an isolated case. Forrester Research published a paper called Waters Crum Fall in July 2011, where that's the result of their research on how people actually do Agile. So you know, choose whether you want to believe it or not, but lots of organizations I worked with have a very similar process where kind of the business people do all the important business decisions, and then you children can go and play your scrum, and then kind of we'll still look at the software later, and kind of they call that Waters Crum Fall. So we need to be, I think, much better in terms of positioning this mirror somewhere else or getting a bigger mirror if we really want to influence software quality in the future. And kind of the third thing is kind of, if we're talking about information art, I kind of like to think about it as a balance of three things. It needs to be accurate, it needs to be relevant, and it needs to be timely. And at the moment, I think we're overly focused on only accuracy. So the way most of the teams I get engaged with approach quality is, well, we have this 77.92 coverage in this module, incredibly accurate, fantastically precise. We visualize it using those green bars, red bars, and stuff like that, 97.62. And then because this is kind of lots of information, we combine it with even more information. We have these bug tracking tools that are incredibly accurate as well. We have 6,572 issues that are in this status and these new issues and lots of, lots of accuracy there. And because this is too much information, we then kind of add more information. That is how, you know, what's a cyclomatic co-complexity and where is my, you know, matrix of paint.net against paint.net complexity and things like that. Lots of, lots of really good stuff, very, very accurate. But is it really relevant? And kind of, we try to make it more relevant by creating dashboards like this, where we have a pie chart of technical debt that is at 11.6%, incredibly accurate. All these things really remind me of this. So I'll make a couple of really broad assumptions at this point. So one assumption is I've read somewhere that Heathrow, that's one of the busiest airports in the UK, in Europe, did, had about 70 million passengers last year. So I'll make a broad assumption that on kind of a European level, that's one of the busiest hubs, but it's not the only one. So let's say that on a European level, there were 100 million passengers last year. These screens have been around for at least 10 years. And let's, again, make a broad assumption that over the last 10 years, one billion people have seen this thing. Now Doug Hubbard in his book about business metrics has a very interesting statement. He says that the value of a metric is directly a value of the decision it informs. And it's interesting to kind of look at this and say out of one billion people who have seen this, how many have actually made a decision based on this? Exactly one. There was a girl, there is a girl called, or a lady called Sandrin Helen Selie, who on the Hong Kong to Brisbane flight in 2005 looked at this thing and said minus 10. It's not that cold. I can go and have a smoke. She opened the door and two minutes later the entire cabin crew jumped on her to prevent her from decompressing the plane. So out of one billion people that have seen this, there was only one person who made the decision and even that decision wasn't that good. Now really ask yourselves, when's the last time anybody made a decision based on this? Or this? And what kind of a decision do we make? The decisions we make based on these things are should they fix it? The answer is probably, and the decision we make based on these things is should I write more tests? The answer is probably, so I can kind of replace these two things with a paper that says probably. And save a lot of time of kind of collecting this information, visualizing it, storing it and things like that. And the question is really, how do we inform better decisions? Can we instead of wasting that money and time on collecting things that don't give us good decisions, can we invest that money and time in informing some better decisions? And in particular, I think one of the problems we really have is that all the decisions that we inform are very technical. And we can probably make these decisions cheaper and faster. So in terms of timeliness, what most of people do, reminding of this guy who tweets World War II 60 years later, that's kind of interesting but too late to prevent the Germans from invading. So a lot of this information we collect, a lot of these dashboards, they're interesting but they're too late to do anything serious. And kind of if you go back to this thing, the real explanation behind this tells me that I'm not important enough for somebody to play a movie, so here are some numbers that move to keep me occupied. And kind of this is exactly that. You're not important enough to spend a lot of time and money doing something useful but here are some numbers that move and some pie charts that tell you something interesting and be amused. So kind of how did we get there? And how did we get to the point where all the decisions we inform that kind of have anything to do with quality are purely technical and kind of in that sense when you go and talk to business people about okay we need some time to restructure code, we need some time to improve quality, we need some time to kind of do better test automation or do better reporting, they tell you well basically blah blah blah blah blah blah blah blah money. And all they hear is cost. We almost told the business people I talked to over the last couple of years, quality is just cost. And cost is something to be minimized, cost is something to be outsourced to India, cost is something to be sold and kind of avoided if possible. And that's kind of a position in all these stuff as cost to the business people. That's what they look at it is kind of why would they pay for refactoring, why would we spend time doing better coding, why would we do this, it's just cost. We need to move away from that but we need to move away from that in a way that we redefine quality to position it as value. So that's the real thing that I don't have an answer to and that kind of bothers me quite a lot how do we reposition software quality so that we provide value. And one way of doing that would be to start informing some decisions that business people care about if we're really thinking about information art. And I was thinking how we got there and then I read a really interesting book about the change in the organization. And in the book there was quite an interesting study about a physician called Joe Bogan. Joe Bogan was a physician in 1960s who was treating people from severe epilepsy. And they realized that very, very bad epileptic attacks start in one part of the brain but they're not that bad until they move over to the other part of the brain as well. This is where the really difficult stuff starts happening. And very much like a military commander trying to prevent the enemy from following them or crossing the bridges and stuff like that and burning the bridges, they were playing around with the idea of separating parts of the brain. So kind of they were looking at monkeys for example and they've experimented with monkeys by splitting their brains and then try to see if anything important happens because people really didn't understand all those connections between the brains, what do they do? Monkeys didn't really seem to be bothered about it so they started experimenting on humans. And they've gotten to some interesting results where these people were no longer suffering from really bad epileptic attacks. And then a guy called Michael Gazzaniga got charged of actually investigating what are the consequences of this because nobody knew. And he did a lot of research with people who were operated on and out of that research comes a lot of what we now know in pop psychologists left brainers, right brainers because before that we didn't really know what was the responsibility of different parts of the brains. One of the more interesting experiments that they've done is they were trying to show pictures of a chicken to people by showing it to only one of their eyes because eyes, the left eye is connected to the right brain, the right eye is connected to the left brain. So what they did is they showed a picture of a chicken while closing the left eye of somebody and they saw it with only the right brain and said, what are you thinking about? And they said, well, I'm thinking about the chicken. Why are you thinking about the chicken? Well, you just showed me a picture of a chicken. Okay. Then they took another group of people, they closed their right eyes. So they showed them a picture of a chicken, the left eye connected to the right brain. They said, what are you thinking about? Well, I'm thinking about the chicken. Why are you thinking about the chicken? Well, last summer I ran a chicken over with a car so we had to kind of bury it and it was embarrassing or kind of two days ago I had chicken soup. And they realized that kind of if you show people partial information, the left brain doesn't get full information, left brain only gets partial information. There's a part in the left brain that invents stuff. It's called the left brain interpreter in psychology. And kind of again, this is where kind of the left brain and right brain I think comes out, but what's interesting from the whole thing is that given partial information, we invent stuff to invent a plausible excuse. And I think that's what we did with software quality. We invented stuff based on some very, very early epileptic projects in the 60s. To prevent epileptic attacks, what we did is we split the brain. We split the brain into business and delivery and that wasn't enough. So somewhere in the 80s, we split the brain into BA, Dev and QA and we give ourselves partial information. And we give ourselves partial information, the left brain invents stuff. And this is where we invented bugs. Bugs are one of the things that we invented. It was us, the technical people. Bugs don't exist in the real world. They do exist, but not the kinds of bugs that we're tracking and we're talking about. One of the more interesting things is that the absence of bugs has nothing to do with the presence of quality. Yet we still measure that we focus on that. The absence of bugs is an indicator. And it's not a particularly good indicator. So for example, blood pressure is an indicator of human health. If I have high blood pressure, that's probably a sign that I'm not really healthy. I can reduce blood pressure incredibly well by chopping my head off. Blood pressure zero. Have I increased the quality of my life? Probably not. So lots of teams are overly obsessed with bugs. Bugs are a good indicator. Bugs are an indicator, not that good. A high number of bugs probably means the system is not good. But should we try and kill everything? Should we try and reduce it? I don't know. One really interesting thing that happened to me was while I was working on bridging the communication gap, I did everything myself in terms of publishing. And I automated the process completely. That's kind of the developer approach to publishing. I had DocBook. DocBook is XML. It was under CI. Every time I changed it and committed to the version control, the CI would build a new PDF. I had lots of checks to make sure that everything is in the right place. I was including source code into that so I don't have the problem of source code that's wrong. I had lots of stuff there. And even my copy editor worked on XML. I explained to my copy editor how to check in and check out of SVN. She would change XML. The SVN would tell her you've broken XML formatting. Go and fix that. CI would tell her that. So from the point when I started writing the book, the first letter there, till it appeared on Amazon, six months. And when I looked at it, there was maybe one or two typos, not that important. When I did spec by example, because I worked with a traditional publisher, their process was completely different. So their process was more, I did stuff in XML because they published HTML pages out of that as well. Then to my horror, somebody manually copied pieces of that XML into Microsoft Word. And then they printed that off. Then somebody kind of looked at that fixed grammar and spelling. Then somebody entered that into Word. And then they sent it to me and I said, this is wrong, this is wrong, this is wrong. You've changed the meaning. This is wrong, this is horrible. Change it. And then somebody took that, entered it from my emails into Microsoft Word. Then somebody manually took that Word and copied it into FrameMaker where they laid it out in PDF, put the images there and everything. Printed that off, somebody went to the printed page and said, okay, this is there. And over the course of about nine months of going back and forth and fixing these things, they said I have no trust and no confidence this is ever going to go out anymore. We're spending a longer time fixing this than I published the whole book. So if you don't get it out by Monday, we're canceling the contract. And this is no, no, no, no, no, no, no, it's done, it's done, it's done. It's out Monday. Not Monday, but kind of the other Monday. Is that okay? Okay, just get it out. So when the book finally got out, on page number three, I talked about how I proposed to talk about specifications and specifying with examples instead of acceptance tests. What you find in this part here is that the word acceptance stops and a whole other sentence starts. So if the purpose of a book is to be read, then this is kind of a priority one problem. You can't read half of that sentence. It's not there. And on page number seven, I found something similar. I found an image covering a part of a word. And I sent an email saying this is horrible. This is like, this release is like two priority one bugs five seconds after release, you're crazy. What have you done? And they said, okay, we'll fix it. And then I started reading it and I found more stuff and I started getting really annoyed and I sent another email. I said, this is, you know, let's fix this and reprint. And I said, don't worry about it. I said, what do you mean don't worry about it? It's my reputation online. People will read this. They'll say, you know, Gweiko doesn't know any English, which is kind of yes, true, but embarrassing. So I'll get, you know, negative number reviews on Amazon. So what are you talking about? I said, don't worry about this and I said, okay, escalate. I want to talk to somebody more serious. And then I talked to somebody more serious and they said, don't worry about this. I said, okay, I want to escalate and kind of, you know, I talked to somebody even more serious and I said, look, don't worry about this. I said, look, okay, this is insane. So in order to prove a point, what I did is I spent about a month and a half reading the book very, very carefully for a few hours a day. And I've cataloged about 27 places where you cannot read a part of a word. So priority one, the purpose of the book is not there. I found 74 places where they've changed the meaning of what I wanted to say. For example, there's a sentence there that looks like I'm arguing against selenium, which I'm not. I'm just talking about how people misuse it. And they've changed the meaning of that. I didn't catch that because word is not a good version control system. So there are 58 places where they've not really changed the meaning, but they've made it more difficult to read. So kind of priority three things. So this is my bug catalog. I've sent a really long email and I said, this is horribly embarrassing. And I got an email back saying, don't worry about this. I said, look, you're really pissing me off. And this is horrible. The book came out horrible. It's wrong to do it. I don't mind how much it costs. Let's reprint it. Let's fix it and reprint it. And I said, look at Amazon. And on Amazon, there were 21, five-star reviews at that point. Nobody was complaining that they didn't speak English. So I invented what quality meant there. I invented what priority one is, what priority two and priority three is. I spent a month and a half of a couple of hours a day of my life cataloging these bugs and tracking them. And some of them got fixed in the next printing. Some of them didn't because it wasn't that important. The book was selling quite fine and people liked it. So this got me thinking and I really started looking for this when I worked with teams. Who really decides what priority one is? Who decides what priority two is? How do we know that the stuff that we think is important is really important? Most cases, the testers who don't have any financial stake in anything, who sometimes are not even domain experts, and they decide what priority one is. They catalog it. They track it. And this is where I kind of going back to the idea that as we shrink development phases, we'll have to start changing stuff. So are we really investing our time and money and effort in tracking the right things? Are we investing our time and money and effort in reporting the right things? Or have we invented it like I invented priority one here? And would I be better off by doing something much more productive with my life for those months and a half, two hours a day? Would the teams that I worked with be better off doing something much more productive with their time instead of populating 700 G-R items with screenshots and stuff like that? Are we really faking it? Or are we doing the right thing? And as we start collapsing time, as we start moving from a two-week delivery phase to a one-week delivery phase, are we really investing our time wisely because we have even less time? Several teams I worked with now release several times a day. When you do that, you no longer have time for an overnight test run because there is no overnight. And then people really start questioning, are we maintaining the right tests? Are we investing the time in what we do right? So I think this is coming because 10 years ago when you told people, okay, let's try and release every two weeks, they would tell you're crazy. Now everybody does that. So you might think that releasing several times a day is crazy, but in five years time the industry is going to move there. All your competitors are going to be doing it and your business is going to be starting pushing you to do that. And at that point, a lot of what we do, a lot of what we approach in terms of quality, might be a waste of time. So the key thing that I was thinking about is what's the more useful thing to report on? If I say 7,800 tests executed, this coverage and seven failed, does that inform better decisions or well with this user story, you expected the user registrations to rise by 7% actually the increase by two? Which of these things informs better decisions? If we have to choose where to invest money, which of these reports is better in terms of telling business people what they're interested in, where they can make some good decisions. And I think kind of something like this tells us where to invest money next. It tells us whether we have to fix the system. It tells us whether we have bugs, but kind of this thing might be the same kind of tests we execute. We just report it differently. Or maybe this thing is just five tests out of these 7,000 and then should we really maintain the remaining 7,000 or not? So kind of the, I was trying to explain this to one of my clients and we ended up talking about how software quality really has lots of different things on lots of different levels for lots of different people and how bugs are not a good thing, but we still like looking at them, we still like reducing. How quality is still important. It's not unimportant. It doesn't inform any business decisions, but it is an important thing to have. So that reminded me of Maslov's hierarchy of human needs because Abraham Maslov was a psychologist who was trying to understand why people spend a lot of time in their life trying to earn a lot of money. And then when they earn a lot of money, they're still not happy. And they were trying to model kind of human life quality and human happiness. And the end result of that research was this pyramid of human needs where he says that on the basic level humans have physiological needs. So we need air to breathe. We need water. We need food. But kind of these things, we just need enough of them. More water than I can drink is irrelevant. More food than I need gives me obesity. So it's even bad. It's kind of more of that is not necessarily good. Then kind of on top of that, there are safety needs. Like I need to be, I need to have physical safety and financial security and things like that. On top of that, there's love and belonging. That's after a certain point more important than money. Then on top of that, there's self-esteem and on top of that, there's self-actualization. So what's interesting about this model is that kind of from bottom up, good enough is good enough. And they said that people who lack something on a bottom level are going to disregard upper level stuff. So if you don't have enough food, you will not care about self-esteem. But if you have enough food and you have enough money, then you're going to start caring about other things more. So we're trying to kind of come up with a model similar like this for software. So I'm not saying this is universally applicable. This was for them and it worked for them. But it might make you think how to model this thing in your situation. So we said, what would be the physiological needs of our software? So one need is for this thing to be deployable. If it's not deployable, it doesn't matter. Everything else does not matter because there is no software. So another thing is that it needs to be functionally OK. So it needs to have enough functionality. It needs the functionality needs to be correct. But more functionality than necessary is not necessarily good. Look at Microsoft Word, for example. Over the last 10 years, it's built, it's grown by, I don't know how many thousands of new features, but nobody uses anything that wasn't there in 1995. So more functionality in that case means more time, more tests, more analysis, more code to maintain, but not necessarily good. So the other thing on top of that was, well, what would be the equivalent of security needs? Well, we said, OK, obviously, once the software works, it needs to work well. So things like code quality, things like good architecture, things like performance, things like security, things, all these kind of non-functional things are on that level. Again, if the software is not deployable, I don't care about the architecture. If the software is deployable, the architecture becomes important. Is it scalable? Is it performance and things like that? And then we said, OK, but then more performance than we need is not necessarily that important. I wasted a lot of my own money with the startup that I co-founded in 2009. We built a multiplayer system that could handle about 10 million players concurrently. In 2009, we had 1,000 players at the same time, peak time. We had a 10 million capacity. In 2010, we had 10,000 players concurrently in peak time. We had a 10 million capacity. So at the moment there at about, I think, 500,000 players peak time, it's still over-engineered. And the question is really, should we in 2009 have spent that time creating an architecture that scales to 10 million concurrent players? Or should we have built something that scales to 10,000 players concurrently and built something that brings us money? And on top of this thing, we were trying to look at love and belonging. What would that correspond to? So love and belonging would correspond to usability. And do my users love my software? And I think usability then is, at least for these guys, goes on top of their non-functional stuff. I think a very good example of where usability is more important than performance just enough is just enough, and it's good enough is Twitter, where they're famous for the fail-well. It falls over all the time. Their performance is just good enough. And then they increase it when they need. But people love it. It's usable. It's useful. People use it. That's why seeing an occasional fail-well is not that big of a problem. So what's also interesting is when you look at what most teams do with testing and with quality, we spend a huge amount of time investing in functional tests and looking at the system functionally. Teams do performance testing and things that are a bit less. Very few teams actually have a good strategy around doing performance testing and security testing. And then usability is something very, very few teams actually do. And even those that do that do it less than the other things. So maybe approaching quality like that, we should be trying to figure out what's good enough in terms of the bottom two level and then investing more time in other things. But what's also interesting is that this pyramid has two more levels. And this is nothing new so far. But if you look at, for example, the whole usability thing, this whole thing about design thinking at the moment, that's a new trend, design thinking was started by IDEO, that is a famous interaction design agency in the States. And they started applying the way they approach design to the business. And that now moved into business. There's a Stanford School of Business that applies design thinking, there's kind of, it's been around for a while. And one of the key books about design thinking, that's kind of how to apply creative techniques on managing business, things like that, is called Change by Design by Tim Brown. And he wrote that about five years ago. One of the most important case studies in his book is Nokia Ovi. Does anybody remember that? You've never heard of that. And that's good. Nokia Ovi was this completely misguided attempt to put usability on top of Nokia. They've redesigned everything to be kind of design thinking usable and things like that. Tim Brown has a fantastic sentence that in only one and a half years of applying design thinking, Nokia has reinvented itself as a service provider, no longer a technology company. And this is a massive success of design thinking. It is, it's going bankrupt now. Massive success. Fantastic. So Nokia kind of has reinvented themselves as this kind of Ovi thing and then they've completely dumped it because it was wrong. And that was overly focused on usability and design and kind of things like that. Well, there's one level on top of that. Software can be usable, but if nobody wants to use it, it doesn't matter. So there's a level on top of that. Is it really useful? So for example, you know, I can imagine that Nokia Lumia phones are technically much better than this Android shit that I have. This thing is restarting every three days. I say I can assume that Nokia Lumia phones are better than that because in order to do, to measure that somebody would have to buy a Nokia Lumia phone, which is never going to happen. So yes, it's usable. It's fantastic. But if it's not useful, it doesn't matter. So I think we need to start thinking about measuring usefulness much more. And then again, there's this famous study of a website that did redesign and they've made the website much more usable and they've tracked where the kind of people are using it that way. And they had a purchase process where they removed 10 steps of checking out and made the one click checkout and things like that. People loved it. Absolutely. The problem with that is their revenues went like that because most of the revenues came from advertising and by removing 10 pages, they've reduced the amount of advertisement they were displaying by 10 times. So kind of there's another level on top of this. So it can be useful like that cash machine that gave me 300 euros. Really useful. But is it successful? Is it successful from achieving whatever it needs to achieve? So lots of people go around talking about mapping everything to money. Not everything can be mapped to money. And I like Chris Mats's idea that you should be able to save for any piece of software, how he's going to save money, protect money, and earn money. But even if not, we should be able to say what we actually expect out of a piece of software to do from a business perspective. Now this is a really interesting question. Since I started asking this, I got really, really confused answers from people. Nobody knows that. I worked with this team and they had a 30 page requirements document and I said, okay, do you, does anybody really know what this is supposed to achieve from a business perspective? Over what time? By how much? And this is the area we know. So what? Well, I don't know personally. But those guys in the solutions department know. And then we went to the solutions departments and said, well, what do you want the software to do? Well, we want the software to work as specified. But forget that. What do you want it to do? Nobody knows. That's not an isolated case. I've worked with a few other clients where really it's very difficult to figure out what this piece is supposed to achieve from a business perspective. Very few people know that and it's not communicated. And if that's not communicated, how do we ensure that that thing happens? We focus on specs. We focus on bugs according to specs or user stories. Some kind of user stories, lots of stuff that I see is too vague. As a stakeholder in order to do something, I want the software to do something else. But not by how much and over what time. If we say we want a piece of software to increase the number of users, by how much? I mean, increasing the number of users by five is a completely different solution than increasing the number of users by five million. Do we know that up front? If not, how do we know that we've actually succeeded? So I think of those are those are the, it's an interesting thing to start looking at different levels of quality and then thinking about when are we good enough? So kind of the other thing that I was thinking about is this whole informing the decision thing. And if we're going to produce any quality reports, if we're going to spend time aggregating any data or putting in any metrics for anything, what are the decisions that we can inform? And I think we can split this into four different decision types. There are some decisions we can make while we are developing software that are useful. There are some decisions we can make after we've developed software or after we've released something. And there are some decisions we can inform from a technical perspective or a delivery perspective. There are some decisions we can inform from a business perspective. So I think kind of the typical thing we talk about is, well, should I release this software? I have seven priority one bugs. I have this and that should I release it? And we are pretty good in doing that now. And we are not that good in informing the decision of should I build this thing at all? Can I argue against this? Maybe we can build something differently. I've recently worked with a bank that has offices all over Scandinavia. And they were going to go on a one and a half year project to rewrite one of their systems by actually thinking about what they want to achieve from a business perspective and how to measure that. We were able to restructure the same project to be about a month long. Because we removed all the scope creep, we focused it much better. We were able to argue against should we build this thing at all? So that's quite an important decision we can inform. The other decision that we can inform is should we really invest in maintaining this? Should we invest in further development of this or not? And I think that's quite an important business decision that we should start informing. And the other thing is how do we improve the process? So at the moment again, process improvements, Kanban boards, scum boards, lots of other things, we're good with that diagonal. We're not that good with the other one. And maybe we should start informing some decisions on the other diagonal. So one thing that I think is kind of some reports that we could do is first of all, can we be better reporting risk, not bugs, not kind of priority one, priority two items? Can we show people how much risk there is in software? The other thing is can we give them feedback on assumptions they have? Because all software is based on assumptions. If we don't know what assumptions people are making, how do we know that we're building the right thing? How do we know that we're investing in the right thing? And kind of on the bottom part there, I think kind of decisions we can, the reports we can do are kind of informing better specifications and informing about the pipeline. So kind of I don't have time to explain these things. And again, I don't have all the answers here, but here are two interesting tools to investigate. One thing that's really interesting is called the attribute component capability matrix. It was invented in Google and they use that to report risk. They use that to capture a shared understanding of how much risk their software carries from a business perspective. And then you can talk about an agreement with the business that if our software doesn't have any area that is risk level five or more, we can release it. So on this diagram here, the depth is actually the risk a certain piece carries. And this was a real time monitor that kind of they integrate all the testing into that that actually reports it on something that business people care about. So I've done this with several companies so far and the results are magical where you ever shared the agreement among business people first what's risky. So they don't scream at the team that this is wrong or this is right. And then given that shared understanding of where the risk is, we can approach kind of how do we invest in, where do we invest in testing? Where do we do risk stuff? And also we can have a good agreement with the businesses. When is the software releasable? We're not going to be able to debug and de-risk everything. But what kind of risk is acceptable? So that helps us do this conversation so you can find more on that on this URL. The other thing that's really, really useful over the last two years is a technique called effect mapping. Effect mapping was invented by a Swedish interaction design agency and it's a good way to visualize assumptions on a project. I've used that to visualize assumptions that kind of different business people have about different scope and argue about is this the right thing to develop or are there better things to develop, easier things to develop? And it's one of these things where when you start visualizing assumptions you realize how much useful information that gives you because you can start testing these assumptions, not spend seven months developing software based on assumption, spend a week trying to prove or disprove that assumption and then invest in developing. So you can find a lot more about this technique on this URL. And as kind of some final idea said, I don't really have any answers to these questions, but here are things to think about. Can you move quality from being a cost center to being a value add? Because if so then people will want to invest more money into that. The more you invest in something that generates value, the better. And can we kind of connect the whole brain again, not have partial information but distribute useful information to everybody, find some information that's globally useful? Another thing to think about is can we start informing business decisions, not just technical stuff. I like and depend, I love and depend, it generates a graph of complexity, I can improve my classes, nobody else cares about that. Can I invest the same time in something that informs business decisions because then what I do becomes more important? And another thing is, are there ways we can measure business success? Are there ways we can measure the top of the pyramid? Is that more useful than measuring stuff lower down? So kind of here's some books to kind of read later. One book is called Effect Managing IT, that explains effect mapping. The other book is called How Google Test Software. You'll find an explanation of the attribute component capability matrix there. The third book that's interesting is How to Measure Anything by Doug Hub, but he talks about business metrics and good business indicators and how to measure business stuff. The fourth book is my book, Spec by Example, that kind of gives us a good way of giving people feedback on specifications. And that's about it. I hope I've at least provoked some thinking. And I hope this was useful. If you have any ideas about this, I said I really do not have any answers to these things, but if you have any ideas of this, if you've tried out these techniques or if you know something else that is useful in this area, please let me know. This is how to get in touch with me. I think I've overrun a bit, so I'll let you go and have lunch, but I'll be around for a while. So thank you very much. I hope this was useful. Thank you. I hope that everyone has a hug. You You
 Yeah, fantastic.  Okay.  Hello.  Thanks for coming.  What I want to talk about now is a bunch of questions I have.  I don't really have any answers to these things.  It's kind of talking to lots of different people and doing conference talks over the  last few years.  I've started kind of thinking about that we got software quality completely wrong.  One of the things that was really interesting is that I got a call two years ago to do a  keynote at an internal company talk by one of the secretaries called me and I started  trying to understand what they really want.  She said, well, I'll put in touch with this guy who is the director for that thing.  I talked to him and essentially the guy told me that he saw me speak at some other conference  and I was able to make people laugh.  So they scheduled me after lunch and if I can make people wake up after lunch.  He didn't really care about what I was talking about.  So I said, okay, I'll come up with some good material.  I started trying to kind of make fun of how we approach software quality.  When I really started talking about all these jokes that I wrote for them, I couldn't kind  of disprove most of it when I started seriously thinking about it.  So that got me thinking about this and this is part of my kind of reinventing software  quality tour.  I've submitted this talk to almost every conference I'm speaking at this year and I'm refining  it as I go with the idea that I'll make myself clean up things.  So kind of halfway it was changed from the last time I did the talk.  And I said, I still do not have any answers to these things but hopefully I'll provoke  some of your thinking and then if I do, let me know if you think you have answers to these  things.  And software is everywhere now.  Absolutely everywhere.  Everything relies on software now.  And we are so used to software being wrong that we don't even think about it anymore.  Like at this railway station where one of the displays is asking us to put a disk drive  in C while the other is showing us that we should have a train arriving on time.  And another example that's really interesting is kind of this hero of mine who was able  to rewire an ATM to play Solitaire on it.  I was recently in Italy and I had to take out some cash from a cash machine and I had  a situation very similar to this where I've put in my PIN code and the machine started  validating my credit card, whatever, and then a Windows exception popped up and I said,  oh shit.  Five seconds later there was a message in German that I kind of used my phone to translate  to something like we've taken your card, call your bank, and I was like, oh no, no, no,  no, no, no, no.  You know, my wife and two kids are waiting there.  I have to take cash to pay for a taxi.  What's going on?  As I was trying to call my bank, I saw a dialogue OK cancel.  So at this point, if you've ever done any software testing, you have to press the cancel  button.  So I pressed the cancel button and then I heard 300 euros came out.  My card came out.  All I could say is thank you, Microsoft Windows.  But transaction has never showed up on my bank account.  I was really, really happy at that point.  So software quality has lots of different aspects for lots of different people.  For me, that was one of the best qualities of Windows I've ever seen in my life.  For somebody else, they were not that happy.  One thing that's really, really puzzling is that there are certain things that are now  going on in the industry and I think we'll have to change a lot of how we approach software.  And in particular, our thinking about software quality will need to change because it's no  longer going to be applicable.  One of the trends that's happening in the industry over, at least from my experience,  perhaps it was before, I don't know, is that the phases of software development, the phases  of software releases are shortening all the time.  10 years ago, most of the teams I worked with were maybe releasing every six months.  The delivery phases of two years were relatively normal.  Most people I worked with these days want to release two weeks to a month.  Two weeks seems to be the default scum, whatever that means today.  All teams I worked with will release several times a day.  So there's a general shift towards reducing the phases.  Now, when we start reducing the phases, certain things that used to apply no longer apply.  For example, one of the most accepted best practices today is using a satna while we  drive.  One of the things that we do is we get a lot of people who don't even drive without a satna.  This is where we get a lot of these insane reports where somebody's car ended up in a  river because the satna told them to go left.  Although, arguably, there was a river left and they didn't care about that.  So I was on a holiday in France last year and we spent some time in Provence.  It was beautiful countryside, lots of really nice views, very nice villas and some hilltop  castles.  We got into the car to go and see this castle that was five kilometers away.  We have this stupid tom-tom that for the first 10 minutes when you turn it on, has an identity  crisis.  Is it really a satna or is it just a stupid box?  So as we were driving, the tom-tom is still looking for satellites.  I'm like, okay, they're fucking there.  So we see this castle that we wanted to see visibly passing.  We're passing it.  So my wife decides to take a risk and drive without a satna and go into the road left  visually, navigating towards the castle.  One minute later, the road goes narrower, narrower, narrower and about two minutes later, we end  up in a ditch like this.  And I got out and started to push the car out.  You can imagine a scene where I kind of just mud everywhere.  I'm pushing the car out.  My wife is swearing.  She's in full reverse.  The kids are screaming and tom-tom wakes up at that point and says, turn around when possible.  So at that point, I did something that's very, very difficult to explain because I'm from  Serbia and we are known for our tolerance and understanding.  So I got out of the ditch, got the tom-tom and started smashing it like crazy.  And I did this for two main reasons.  Reason number one is I don't really need a satna to tell me when I'm lost.  I need the satna to prevent me from being lost in the first place.  Reason number two, the people who are married in this audience, you can vouch for this being  true that in a situation like that, it is a lot easier to smash the satna than blame  your wife for not knowing how to drive.  So what I'm trying to say is that satna is generally a best practice, but when your journey  is five minutes long, that best practice no longer applies and can even get you in a worse  situation than you would be if you wouldn't use that.  And I think the way we are shrinking our delivery phase is going to make us rethink really hard  about everything we do in terms of software quality and whether these things really still  apply.  So the second important trend in the industry is that everybody's talking about cost reductions.  Every single team I talk to, most of what they're focused on is reducing cost, which  means that if we need to reduce cost, we have even less money and kind of time is money  to spend on things we want that are not really essential.  So stuff like, okay, going back and refactoring or cleaning up code, it's even more difficult  to justify than before.  And stuff like, okay, let's redo our automation or let's do this or let's spend a lot of  time rewriting a test.  It's very, very difficult to justify today.  The third thing that's happening that I think has already happened in a sense, but we don't  really see that yet that much is that software quality is really information art.  What I mean by that is sometime ago we had this idea that we could do quality control  and then we realized that's not going to work.  And then people had this idea, okay, we can do quality assurance.  But that didn't turn out to work as well.  And I think kind of what a lot of people are moving to now is that software quality is  about distributing information, informing the right decisions.  Elizabeth Hendricks on a Tegel testing days 2009, I think, said that people who ask for  control genuinely want visibility.  That was an interesting statement, so I decided to put that to test.  And I worked with one of the biggest media companies in the UK where they were complaining  about this horrible, horrible, horrible publishing system for their website.  Lots of problems all the time.  They wanted to rewrite it.  They wanted to redo the automation and everything.  And I said, okay, let's think about this whole quality problem.  You're complaining that it's very bad quality.  How would we visualize this?  How do we show this problem before even starting to solve it?  What does this mean?  And after a few weeks of trying to negotiate that, we established that one of the biggest  problems the way people perceive quality is that content publishers, because of all the  bugs in the system and some business rules that are obscure and all things like that,  sometimes they publish an article, it shows up immediately.  Sometimes they publish an article, it doesn't show up in 10 minutes time, so they publish  it again.  And then it doesn't show up in five minutes time, they publish it again, and then 45 minutes  from then you have the same article published on three places on the homepage, which is  embarrassing.  And sometimes it doesn't go out at all.  Sometimes it was stuck because of a bug and sometimes it was because of genuine business  rules because they don't want to overload their website with stuff that stays there for five  seconds.  So if you publish lots of articles, the system is going to delay that.  And this was the essence of what people complained against, and they wanted to rewrite the whole  system because of that.  And they said, okay, that's the problem.  How do we visualize this problem?  And one of the business users said, well, if you can tell me the time, how long it's  going to take for this article to be published, that's a good visualization for me.  And we talked to developers, could we estimate that?  And I said, well, we could estimate maybe the business rules, but then again, lots of  bugs.  And it's going to be very difficult to estimate that maybe we can show the last article that  got published.  We have that in the database.  We have the timestamps.  So is that good enough?  And we said, I don't know.  Let's try it.  So we bought two big monitors.  One was in the development room.  One was in the content publishers room.  And it was continuously refreshing a single web page with a single SQL query, looking  at the difference between the last timestamp article when it got published and when it  was queued.  All of a sudden, all the complaints stopped.  The system didn't get cleaned up.  It was still the same bad system.  It still had the same problems.  People stopped complaining.  The quality of the system jumped incredibly.  What had happened is that the content publishers would look at this and say, okay, 15 minutes,  I'm not going to complain about it until 15 minutes pass.  I know how long it's going to take.  The developers would look at the monitor and say, oh, 15 minutes, something's gone wrong.  So they would massage it through so that people don't really notice any problems.  And then they would try and understand why that happened and clean it up so it doesn't  happen next time.  And just by visualizing this thing, we were able to increase the quality as kind of any  measure they had, it increased without really changing anything.  So that got me started thinking about, well, was a rewrite of their system and an investment  in huge test automation really the right thing to do or not?  Because they could have invested months and months in doing good test automation or doing  kind of better code architecture or doing everything else, but was that the right investment  to make?  Because we solved the problem.  When we talk about information, then the question is really, are we distributing the right kind  of information?  And for a lot of teams now, the test team or testers or whoever does quality, they're  basically a mirror.  A mirror in terms of is our architecture correct, is our code correct, is my design good, are  all the business functions there, that's kind of a mirror.  And when we talk about mirrors, one of the things that I always remember is my friend  Nick from high school.  So when I was in high school, we were going out together as a group, there was 20 of us,  and my friend Nick was always the last person to arrive.  Now this was before mobile phones, and again, technology software is now everywhere.  Before mobile phones, you couldn't really call him to shout at him and see why he's  late.  So you had to wait, and we waited and waited and waited, and Nick was always the last person  to come.  Sometimes he was 45 minutes late.  And the reason why he was late all the time is he was changing shoes.  So Nick held the whole wardrobe full of shoes and kind of had this small mirror where for,  I don't know how long, every time he wanted to go out, he would kind of spend deciding  what shade of black to wear.  And the idea was that if he only had the right type of shoes, he would get lucky tonight.  He never, ever, ever did.  The reason why he didn't do that was that the choice of shoes was completely relevant,  because Nick was exactly as a cave troll.  So kind of this is what we do, what a lot of organizations that I worked with do.  They point a very small mirror to a place that they like, and then they optimize that  incredibly.  But they leave the big, ugly troll phase that's still the biggest problem there.  So I recently worked with one very big software development company in Germany, and they said,  well, we do three-week scrums.  It's okay.  What do you do inside those three-week scrums?  And they said, okay, there's good stuff going on there, but they were still complaining  about lots of quality problems, business people were unhappy.  And I said, okay, let's kind of, this three-week scrum looks very development-oriented.  Everything there is around development.  How long does it take you from when a business person says, I want this till it goes live?  And they said, well, we don't know.  So, well, can we find out?  And we could look at the requirement system where can they keep these things before they  become user stories, and we could possibly look at the deployment date for something.  Let's do that.  Two and a half years.  So there's crum of three weeks that they've optimized, and they do lots of really good  stuff that doesn't really matter.  Because they have two years and one month of business stuff before that, and they have  three months of testing after that, where kind of the business people keep the software  to marinate a bit so that it becomes stable enough to kind of go live.  And this is not an isolated case.  Forrester Research published a paper called Waters Crum Fall in July 2011, where that's  the result of their research on how people actually do Agile.  So you know, choose whether you want to believe it or not, but lots of organizations I worked  with have a very similar process where kind of the business people do all the important  business decisions, and then you children can go and play your scrum, and then kind  of we'll still look at the software later, and kind of they call that Waters Crum Fall.  So we need to be, I think, much better in terms of positioning this mirror somewhere  else or getting a bigger mirror if we really want to influence software quality in the  future.  And kind of the third thing is kind of, if we're talking about information art, I kind  of like to think about it as a balance of three things.  It needs to be accurate, it needs to be relevant, and it needs to be timely.  And at the moment, I think we're overly focused on only accuracy.  So the way most of the teams I get engaged with approach quality is, well, we have this  77.92 coverage in this module, incredibly accurate, fantastically precise.  We visualize it using those green bars, red bars, and stuff like that, 97.62.  And then because this is kind of lots of information, we combine it with even more  information.  We have these bug tracking tools that are incredibly accurate as well.  We have 6,572 issues that are in this status and these new issues and lots of, lots of  accuracy there.  And because this is too much information, we then kind of add more information.  That is how, you know, what's a cyclomatic co-complexity and where is my, you know, matrix  of paint.net against paint.net complexity and things like that.  Lots of, lots of really good stuff, very, very accurate.  But is it really relevant?  And kind of, we try to make it more relevant by creating dashboards like this, where we  have a pie chart of technical debt that is at 11.6%, incredibly accurate.  All these things really remind me of this.  So I'll make a couple of really broad assumptions at this point.  So one assumption is I've read somewhere that Heathrow, that's one of the busiest airports  in the UK, in Europe, did, had about 70 million passengers last year.  So I'll make a broad assumption that on kind of a European level, that's one of the busiest  hubs, but it's not the only one.  So let's say that on a European level, there were 100 million passengers last year.  These screens have been around for at least 10 years.  And let's, again, make a broad assumption that over the last 10 years, one billion people  have seen this thing.  Now Doug Hubbard in his book about business metrics has a very interesting statement.  He says that the value of a metric is directly a value of the decision it informs.  And it's interesting to kind of look at this and say out of one billion people who have  seen this, how many have actually made a decision based on this?  Exactly one.  There was a girl, there is a girl called, or a lady called Sandrin Helen Selie, who on  the Hong Kong to Brisbane flight in 2005 looked at this thing and said minus 10.  It's not that cold.  I can go and have a smoke.  She opened the door and two minutes later the entire cabin crew jumped on her to prevent  her from decompressing the plane.  So out of one billion people that have seen this, there was only one person who made the  decision and even that decision wasn't that good.  Now really ask yourselves, when's the last time anybody made a decision based on this?  Or this?  And what kind of a decision do we make?  The decisions we make based on these things are should they fix it?  The answer is probably, and the decision we make based on these things is should I write  more tests?  The answer is probably, so I can kind of replace these two things with a paper that says probably.  And save a lot of time of kind of collecting this information, visualizing it, storing  it and things like that.  And the question is really, how do we inform better decisions?  Can we instead of wasting that money and time on collecting things that don't give us good  decisions, can we invest that money and time in informing some better decisions?  And in particular, I think one of the problems we really have is that all the decisions that  we inform are very technical.  And we can probably make these decisions cheaper and faster.  So in terms of timeliness, what most of people do, reminding of this guy who tweets World  War II 60 years later, that's kind of interesting but too late to prevent the Germans from invading.  So a lot of this information we collect, a lot of these dashboards, they're interesting  but they're too late to do anything serious.  And kind of if you go back to this thing, the real explanation behind this tells me  that I'm not important enough for somebody to play a movie, so here are some numbers  that move to keep me occupied.  And kind of this is exactly that.  You're not important enough to spend a lot of time and money doing something useful but  here are some numbers that move and some pie charts that tell you something interesting  and be amused.  So kind of how did we get there?  And how did we get to the point where all the decisions we inform that kind of have  anything to do with quality are purely technical and kind of in that sense when you go and  talk to business people about okay we need some time to restructure code, we need some  time to improve quality, we need some time to kind of do better test automation or do  better reporting, they tell you well basically blah blah blah blah blah blah blah blah  money.  And all they hear is cost.  We almost told the business people I talked to over the last couple of years, quality  is just cost.  And cost is something to be minimized, cost is something to be outsourced to India, cost  is something to be sold and kind of avoided if possible.  And that's kind of a position in all these stuff as cost to the business people.  That's what they look at it is kind of why would they pay for refactoring, why would  we spend time doing better coding, why would we do this, it's just cost.  We need to move away from that but we need to move away from that in a way that we redefine  quality to position it as value.  So that's the real thing that I don't have an answer to and that kind of bothers me quite  a lot how do we reposition software quality so that we provide value.  And one way of doing that would be to start informing some decisions that business people  care about if we're really thinking about information art.  And I was thinking how we got there and then I read a really interesting book about the  change in the organization.  And in the book there was quite an interesting study about a physician called Joe Bogan.  Joe Bogan was a physician in 1960s who was treating people from severe epilepsy.  And they realized that very, very bad epileptic attacks start in one part of the brain but  they're not that bad until they move over to the other part of the brain as well.  This is where the really difficult stuff starts happening.  And very much like a military commander trying to prevent the enemy from following them or  crossing the bridges and stuff like that and burning the bridges, they were playing around  with the idea of separating parts of the brain.  So kind of they were looking at monkeys for example and they've experimented with monkeys  by splitting their brains and then try to see if anything important happens because people  really didn't understand all those connections between the brains, what do they do?  Monkeys didn't really seem to be bothered about it so they started experimenting on humans.  And they've gotten to some interesting results where these people were no longer suffering  from really bad epileptic attacks.  And then a guy called Michael Gazzaniga got charged of actually investigating what are  the consequences of this because nobody knew.  And he did a lot of research with people who were operated on and out of that research  comes a lot of what we now know in pop psychologists left brainers, right brainers because before  that we didn't really know what was the responsibility of different parts of the brains.  One of the more interesting experiments that they've done is they were trying to show pictures  of a chicken to people by showing it to only one of their eyes because eyes, the left eye  is connected to the right brain, the right eye is connected to the left brain.  So what they did is they showed a picture of a chicken while closing the left eye of  somebody and they saw it with only the right brain and said, what are you thinking about?  And they said, well, I'm thinking about the chicken.  Why are you thinking about the chicken?  Well, you just showed me a picture of a chicken.  Okay.  Then they took another group of people, they closed their right eyes.  So they showed them a picture of a chicken, the left eye connected to the right brain.  They said, what are you thinking about?  Well, I'm thinking about the chicken.  Why are you thinking about the chicken?  Well, last summer I ran a chicken over with a car so we had to kind of bury it and it  was embarrassing or kind of two days ago I had chicken soup.  And they realized that kind of if you show people partial information, the left brain  doesn't get full information, left brain only gets partial information.  There's a part in the left brain that invents stuff.  It's called the left brain interpreter in psychology.  And kind of again, this is where kind of the left brain and right brain I think comes out,  but what's interesting from the whole thing is that given partial information, we invent  stuff to invent a plausible excuse.  And I think that's what we did with software quality.  We invented stuff based on some very, very early epileptic projects in the 60s.  To prevent epileptic attacks, what we did is we split the brain.  We split the brain into business and delivery and that wasn't enough.  So somewhere in the 80s, we split the brain into BA, Dev and QA and we give ourselves  partial information.  And we give ourselves partial information, the left brain invents stuff.  And this is where we invented bugs.  Bugs are one of the things that we invented.  It was us, the technical people.  Bugs don't exist in the real world.  They do exist, but not the kinds of bugs that we're tracking and we're talking about.  One of the more interesting things is that the absence of bugs has nothing to do with  the presence of quality.  Yet we still measure that we focus on that.  The absence of bugs is an indicator.  And it's not a particularly good indicator.  So for example, blood pressure is an indicator of human health.  If I have high blood pressure, that's probably a sign that I'm not really healthy.  I can reduce blood pressure incredibly well by chopping my head off.  Blood pressure zero.  Have I increased the quality of my life?  Probably not.  So lots of teams are overly obsessed with bugs.  Bugs are a good indicator.  Bugs are an indicator, not that good.  A high number of bugs probably means the system is not good.  But should we try and kill everything?  Should we try and reduce it?  I don't know.  One really interesting thing that happened to me was while I was working on bridging  the communication gap, I did everything myself in terms of publishing.  And I automated the process completely.  That's kind of the developer approach to publishing.  I had DocBook.  DocBook is XML.  It was under CI.  Every time I changed it and committed to the version control, the CI would build a new  PDF.  I had lots of checks to make sure that everything is in the right place.  I was including source code into that so I don't have the problem of source code that's  wrong.  I had lots of stuff there.  And even my copy editor worked on XML.  I explained to my copy editor how to check in and check out of SVN.  She would change XML.  The SVN would tell her you've broken XML formatting.  Go and fix that.  CI would tell her that.  So from the point when I started writing the book, the first letter there, till it appeared  on Amazon, six months.  And when I looked at it, there was maybe one or two typos, not that important.  When I did spec by example, because I worked with a traditional publisher, their process  was completely different.  So their process was more, I did stuff in XML because they published HTML pages out of that  as well.  Then to my horror, somebody manually copied pieces of that XML into Microsoft Word.  And then they printed that off.  Then somebody kind of looked at that fixed grammar and spelling.  Then somebody entered that into Word.  And then they sent it to me and I said, this is wrong, this is wrong, this is wrong.  You've changed the meaning.  This is wrong, this is horrible.  Change it.  And then somebody took that, entered it from my emails into Microsoft Word.  Then somebody manually took that Word and copied it into FrameMaker where they laid  it out in PDF, put the images there and everything.  Printed that off, somebody went to the printed page and said, okay, this is there.  And over the course of about nine months of going back and forth and fixing these things,  they said I have no trust and no confidence this is ever going to go out anymore.  We're spending a longer time fixing this than I published the whole book.  So if you don't get it out by Monday, we're canceling the contract.  And this is no, no, no, no, no, no, no, it's done, it's done, it's done.  It's out Monday.  Not Monday, but kind of the other Monday.  Is that okay?  Okay, just get it out.  So when the book finally got out, on page number three, I talked about how I proposed  to talk about specifications and specifying with examples instead of acceptance tests.  What you find in this part here is that the word acceptance stops and a whole other sentence  starts.  So if the purpose of a book is to be read, then this is kind of a priority one problem.  You can't read half of that sentence.  It's not there.  And on page number seven, I found something similar.  I found an image covering a part of a word.  And I sent an email saying this is horrible.  This is like, this release is like two priority one bugs five seconds after release, you're  crazy.  What have you done?  And they said, okay, we'll fix it.  And then I started reading it and I found more stuff and I started getting really annoyed  and I sent another email.  I said, this is, you know, let's fix this and reprint.  And I said, don't worry about it.  I said, what do you mean don't worry about it?  It's my reputation online.  People will read this.  They'll say, you know, Gweiko doesn't know any English, which is kind of yes, true, but  embarrassing.  So I'll get, you know, negative number reviews on Amazon.  So what are you talking about?  I said, don't worry about this and I said, okay, escalate.  I want to talk to somebody more serious.  And then I talked to somebody more serious and they said, don't worry about this.  I said, okay, I want to escalate and kind of, you know, I talked to somebody even more  serious and I said, look, don't worry about this.  I said, look, okay, this is insane.  So in order to prove a point, what I did is I spent about a month and a half reading  the book very, very carefully for a few hours a day.  And I've cataloged about 27 places where you cannot read a part of a word.  So priority one, the purpose of the book is not there.  I found 74 places where they've changed the meaning of what I wanted to say.  For example, there's a sentence there that looks like I'm arguing against selenium,  which I'm not.  I'm just talking about how people misuse it.  And they've changed the meaning of that.  I didn't catch that because word is not a good version control system.  So there are 58 places where they've not really changed the meaning, but they've made it more  difficult to read.  So kind of priority three things.  So this is my bug catalog.  I've sent a really long email and I said, this is horribly embarrassing.  And I got an email back saying, don't worry about this.  I said, look, you're really pissing me off.  And this is horrible.  The book came out horrible.  It's wrong to do it.  I don't mind how much it costs.  Let's reprint it.  Let's fix it and reprint it.  And I said, look at Amazon.  And on Amazon, there were 21, five-star reviews at that point.  Nobody was complaining that they didn't speak English.  So I invented what quality meant there.  I invented what priority one is, what priority two and priority three is.  I spent a month and a half of a couple of hours a day of my life cataloging these bugs  and tracking them.  And some of them got fixed in the next printing.  Some of them didn't because it wasn't that important.  The book was selling quite fine and people liked it.  So this got me thinking and I really started looking for this when I worked with teams.  Who really decides what priority one is?  Who decides what priority two is?  How do we know that the stuff that we think is important is really important?  Most cases, the testers who don't have any financial stake in anything, who sometimes  are not even domain experts, and they decide what priority one is.  They catalog it.  They track it.  And this is where I kind of going back to the idea that as we shrink development phases,  we'll have to start changing stuff.  So are we really investing our time and money and effort in tracking the right things?  Are we investing our time and money and effort in reporting the right things?  Or have we invented it like I invented priority one here?  And would I be better off by doing something much more productive with my life for those  months and a half, two hours a day?  Would the teams that I worked with be better off doing something much more productive with  their time instead of populating 700 G-R items with screenshots and stuff like that?  Are we really faking it?  Or are we doing the right thing?  And as we start collapsing time, as we start moving from a two-week delivery phase to a  one-week delivery phase, are we really investing our time wisely because we have even less  time?  Several teams I worked with now release several times a day.  When you do that, you no longer have time for an overnight test run because there is  no overnight.  And then people really start questioning, are we maintaining the right tests?  Are we investing the time in what we do right?  So I think this is coming because 10 years ago when you told people, okay, let's try  and release every two weeks, they would tell you're crazy.  Now everybody does that.  So you might think that releasing several times a day is crazy, but in five years time  the industry is going to move there.  All your competitors are going to be doing it and your business is going to be starting  pushing you to do that.  And at that point, a lot of what we do, a lot of what we approach in terms of quality,  might be a waste of time.  So the key thing that I was thinking about is what's the more useful thing to report  on?  If I say 7,800 tests executed, this coverage and seven failed, does that inform better  decisions or well with this user story, you expected the user registrations to rise by  7% actually the increase by two?  Which of these things informs better decisions?  If we have to choose where to invest money, which of these reports is better in terms  of telling business people what they're interested in, where they can make some good decisions.  And I think kind of something like this tells us where to invest money next.  It tells us whether we have to fix the system.  It tells us whether we have bugs, but kind of this thing might be the same kind of tests  we execute.  We just report it differently.  Or maybe this thing is just five tests out of these 7,000 and then should we really maintain  the remaining 7,000 or not?  So kind of the, I was trying to explain this to one of my clients and we ended up talking  about how software quality really has lots of different things on lots of different levels  for lots of different people and how bugs are not a good thing, but we still like looking  at them, we still like reducing.  How quality is still important.  It's not unimportant.  It doesn't inform any business decisions, but it is an important thing to have.  So that reminded me of Maslov's hierarchy of human needs because Abraham Maslov was a  psychologist who was trying to understand why people spend a lot of time in their life  trying to earn a lot of money.  And then when they earn a lot of money, they're still not happy.  And they were trying to model kind of human life quality and human happiness.  And the end result of that research was this pyramid of human needs where he says that on  the basic level humans have physiological needs.  So we need air to breathe.  We need water.  We need food.  But kind of these things, we just need enough of them.  More water than I can drink is irrelevant.  More food than I need gives me obesity.  So it's even bad.  It's kind of more of that is not necessarily good.  Then kind of on top of that, there are safety needs.  Like I need to be, I need to have physical safety and financial security and things like  that.  On top of that, there's love and belonging.  That's after a certain point more important than money.  Then on top of that, there's self-esteem and on top of that, there's self-actualization.  So what's interesting about this model is that kind of from bottom up, good enough is  good enough.  And they said that people who lack something on a bottom level are going to disregard upper  level stuff.  So if you don't have enough food, you will not care about self-esteem.  But if you have enough food and you have enough money, then you're going to start caring about  other things more.  So we're trying to kind of come up with a model similar like this for software.  So I'm not saying this is universally applicable.  This was for them and it worked for them.  But it might make you think how to model this thing in your situation.  So we said, what would be the physiological needs of our software?  So one need is for this thing to be deployable.  If it's not deployable, it doesn't matter.  Everything else does not matter because there is no software.  So another thing is that it needs to be functionally OK.  So it needs to have enough functionality.  It needs the functionality needs to be correct.  But more functionality than necessary is not necessarily good.  Look at Microsoft Word, for example.  Over the last 10 years, it's built, it's grown by, I don't know how many thousands of new  features, but nobody uses anything that wasn't there in 1995.  So more functionality in that case means more time, more tests, more analysis, more code  to maintain, but not necessarily good.  So the other thing on top of that was, well, what would be the equivalent of security needs?  Well, we said, OK, obviously, once the software works, it needs to work well.  So things like code quality, things like good architecture, things like performance, things  like security, things, all these kind of non-functional things are on that level.  Again, if the software is not deployable, I don't care about the architecture.  If the software is deployable, the architecture becomes important.  Is it scalable?  Is it performance and things like that?  And then we said, OK, but then more performance than we need is not necessarily that important.  I wasted a lot of my own money with the startup that I co-founded in 2009.  We built a multiplayer system that could handle about 10 million players concurrently.  In 2009, we had 1,000 players at the same time, peak time.  We had a 10 million capacity.  In 2010, we had 10,000 players concurrently in peak time.  We had a 10 million capacity.  So at the moment there at about, I think, 500,000 players peak time, it's still over-engineered.  And the question is really, should we in 2009 have spent that time creating an architecture  that scales to 10 million concurrent players?  Or should we have built something that scales to 10,000 players concurrently and built something  that brings us money?  And on top of this thing, we were trying to look at love and belonging.  What would that correspond to?  So love and belonging would correspond to usability.  And do my users love my software?  And I think usability then is, at least for these guys, goes on top of their non-functional  stuff.  I think a very good example of where usability is more important than performance just enough  is just enough, and it's good enough is Twitter, where they're famous for the fail-well.  It falls over all the time.  Their performance is just good enough.  And then they increase it when they need.  But people love it.  It's usable.  It's useful.  People use it.  That's why seeing an occasional fail-well is not that big of a problem.  So what's also interesting is when you look at what most teams do with testing and with  quality, we spend a huge amount of time investing in functional tests and looking at the system  functionally.  Teams do performance testing and things that are a bit less.  Very few teams actually have a good strategy around doing performance testing and security  testing.  And then usability is something very, very few teams actually do.  And even those that do that do it less than the other things.  So maybe approaching quality like that, we should be trying to figure out what's good  enough in terms of the bottom two level and then investing more time in other things.  But what's also interesting is that this pyramid has two more levels.  And this is nothing new so far.  But if you look at, for example, the whole usability thing, this whole thing about design  thinking at the moment, that's a new trend, design thinking was started by IDEO, that  is a famous interaction design agency in the States.  And they started applying the way they approach design to the business.  And that now moved into business.  There's a Stanford School of Business that applies design thinking, there's kind of,  it's been around for a while.  And one of the key books about design thinking, that's kind of how to apply creative techniques  on managing business, things like that, is called Change by Design by Tim Brown.  And he wrote that about five years ago.  One of the most important case studies in his book is Nokia Ovi.  Does anybody remember that?  You've never heard of that.  And that's good.  Nokia Ovi was this completely misguided attempt to put usability on top of Nokia.  They've redesigned everything to be kind of design thinking usable and things like that.  Tim Brown has a fantastic sentence that in only one and a half years of applying design  thinking, Nokia has reinvented itself as a service provider, no longer a technology company.  And this is a massive success of design thinking.  It is, it's going bankrupt now.  Massive success.  Fantastic.  So Nokia kind of has reinvented themselves as this kind of Ovi thing and then they've  completely dumped it because it was wrong.  And that was overly focused on usability and design and kind of things like that.  Well, there's one level on top of that.  Software can be usable, but if nobody wants to use it, it doesn't matter.  So there's a level on top of that.  Is it really useful?  So for example, you know, I can imagine that Nokia Lumia phones are technically much better  than this Android shit that I have.  This thing is restarting every three days.  I say I can assume that Nokia Lumia phones are better than that because in order to do,  to measure that somebody would have to buy a Nokia Lumia phone, which is never going to  happen.  So yes, it's usable.  It's fantastic.  But if it's not useful, it doesn't matter.  So I think we need to start thinking about measuring usefulness much more.  And then again, there's this famous study of a website that did redesign and they've made  the website much more usable and they've tracked where the kind of people are using  it that way.  And they had a purchase process where they removed 10 steps of checking out and made  the one click checkout and things like that.  People loved it.  Absolutely.  The problem with that is their revenues went like that because most of the revenues came  from advertising and by removing 10 pages, they've reduced the amount of advertisement  they were displaying by 10 times.  So kind of there's another level on top of this.  So it can be useful like that cash machine that gave me 300 euros.  Really useful.  But is it successful?  Is it successful from achieving whatever it needs to achieve?  So lots of people go around talking about mapping everything to money.  Not everything can be mapped to money.  And I like Chris Mats's idea that you should be able to save for any piece of software,  how he's going to save money, protect money, and earn money.  But even if not, we should be able to say what we actually expect out of a piece of  software to do from a business perspective.  Now this is a really interesting question.  Since I started asking this, I got really, really confused answers from people.  Nobody knows that.  I worked with this team and they had a 30 page requirements document and I said, okay,  do you, does anybody really know what this is supposed to achieve from a business perspective?  Over what time?  By how much?  And this is the area we know.  So what?  Well, I don't know personally.  But those guys in the solutions department know.  And then we went to the solutions departments and said, well, what do you want the software  to do?  Well, we want the software to work as specified.  But forget that.  What do you want it to do?  Nobody knows.  That's not an isolated case.  I've worked with a few other clients where really it's very difficult to figure out what  this piece is supposed to achieve from a business perspective.  Very few people know that and it's not communicated.  And if that's not communicated, how do we ensure that that thing happens?  We focus on specs.  We focus on bugs according to specs or user stories.  Some kind of user stories, lots of stuff that I see is too vague.  As a stakeholder in order to do something, I want the software to do something else.  But not by how much and over what time.  If we say we want a piece of software to increase the number of users, by how much?  I mean, increasing the number of users by five is a completely different solution than  increasing the number of users by five million.  Do we know that up front?  If not, how do we know that we've actually succeeded?  So I think of those are those are the, it's an interesting thing to start looking at different  levels of quality and then thinking about when are we good enough?  So kind of the other thing that I was thinking about is this whole informing the decision  thing.  And if we're going to produce any quality reports, if we're going to spend time aggregating  any data or putting in any metrics for anything, what are the decisions that we can inform?  And I think we can split this into four different decision types.  There are some decisions we can make while we are developing software that are useful.  There are some decisions we can make after we've developed software or after we've released  something.  And there are some decisions we can inform from a technical perspective or a delivery  perspective.  There are some decisions we can inform from a business perspective.  So I think kind of the typical thing we talk about is, well, should I release this software?  I have seven priority one bugs.  I have this and that should I release it?  And we are pretty good in doing that now.  And we are not that good in informing the decision of should I build this thing at all?  Can I argue against this?  Maybe we can build something differently.  I've recently worked with a bank that has offices all over Scandinavia.  And they were going to go on a one and a half year project to rewrite one of their systems  by actually thinking about what they want to achieve from a business perspective and  how to measure that.  We were able to restructure the same project to be about a month long.  Because we removed all the scope creep, we focused it much better.  We were able to argue against should we build this thing at all?  So that's quite an important decision we can inform.  The other decision that we can inform is should we really invest in maintaining this?  Should we invest in further development of this or not?  And I think that's quite an important business decision that we should start informing.  And the other thing is how do we improve the process?  So at the moment again, process improvements, Kanban boards, scum boards, lots of other  things, we're good with that diagonal.  We're not that good with the other one.  And maybe we should start informing some decisions on the other diagonal.  So one thing that I think is kind of some reports that we could do is first of all, can  we be better reporting risk, not bugs, not kind of priority one, priority two items?  Can we show people how much risk there is in software?  The other thing is can we give them feedback on assumptions they have?  Because all software is based on assumptions.  If we don't know what assumptions people are making, how do we know that we're building  the right thing?  How do we know that we're investing in the right thing?  And kind of on the bottom part there, I think kind of decisions we can, the reports we can  do are kind of informing better specifications and informing about the pipeline.  So kind of I don't have time to explain these things.  And again, I don't have all the answers here, but here are two interesting tools to investigate.  One thing that's really interesting is called the attribute component capability matrix.  It was invented in Google and they use that to report risk.  They use that to capture a shared understanding of how much risk their software carries from  a business perspective.  And then you can talk about an agreement with the business that if our software doesn't  have any area that is risk level five or more, we can release it.  So on this diagram here, the depth is actually the risk a certain piece carries.  And this was a real time monitor that kind of they integrate all the testing into that  that actually reports it on something that business people care about.  So I've done this with several companies so far and the results are magical where you  ever shared the agreement among business people first what's risky.  So they don't scream at the team that this is wrong or this is right.  And then given that shared understanding of where the risk is, we can approach kind of  how do we invest in, where do we invest in testing?  Where do we do risk stuff?  And also we can have a good agreement with the businesses.  When is the software releasable?  We're not going to be able to debug and de-risk everything.  But what kind of risk is acceptable?  So that helps us do this conversation so you can find more on that on this URL.  The other thing that's really, really useful over the last two years is a technique called  effect mapping.  Effect mapping was invented by a Swedish interaction design agency and it's a good way to visualize  assumptions on a project.  I've used that to visualize assumptions that kind of different business people have about  different scope and argue about is this the right thing to develop or are there better  things to develop, easier things to develop?  And it's one of these things where when you start visualizing assumptions you realize  how much useful information that gives you because you can start testing these assumptions,  not spend seven months developing software based on assumption, spend a week trying  to prove or disprove that assumption and then invest in developing.  So you can find a lot more about this technique on this URL.  And as kind of some final idea said, I don't really have any answers to these questions,  but here are things to think about.  Can you move quality from being a cost center to being a value add?  Because if so then people will want to invest more money into that.  The more you invest in something that generates value, the better.  And can we kind of connect the whole brain again, not have partial information but distribute  useful information to everybody, find some information that's globally useful?  Another thing to think about is can we start informing business decisions, not just technical  stuff.  I like and depend, I love and depend, it generates a graph of complexity, I can improve my classes,  nobody else cares about that.  Can I invest the same time in something that informs business decisions because then what  I do becomes more important?  And another thing is, are there ways we can measure business success?  Are there ways we can measure the top of the pyramid?  Is that more useful than measuring stuff lower down?  So kind of here's some books to kind of read later.  One book is called Effect Managing IT, that explains effect mapping.  The other book is called How Google Test Software.  You'll find an explanation of the attribute component capability matrix there.  The third book that's interesting is How to Measure Anything by Doug Hub, but he talks  about business metrics and good business indicators and how to measure business stuff.  The fourth book is my book, Spec by Example, that kind of gives us a good way of giving  people feedback on specifications.  And that's about it.  I hope I've at least provoked some thinking.  And I hope this was useful.  If you have any ideas about this, I said I really do not have any answers to these things,  but if you have any ideas of this, if you've tried out these techniques or if you know  something else that is useful in this area, please let me know.  This is how to get in touch with me.  I think I've overrun a bit, so I'll let you go and have lunch, but I'll be around for  a while.  So thank you very much.  I hope this was useful.  Thank you.  I hope that everyone has a hug.  You  You
Gojko_Adzic_-_Reinventing_software_quality_0_851.png Gojko_Adzic_-_Reinventing_software_quality_852_1031.png Gojko_Adzic_-_Reinventing_software_quality_1032_1607.png Gojko_Adzic_-_Reinventing_software_quality_1608_1847.png Gojko_Adzic_-_Reinventing_software_quality_1848_2399.png Gojko_Adzic_-_Reinventing_software_quality_2400_2879.png Gojko_Adzic_-_Reinventing_software_quality_2880_3047.png Gojko_Adzic_-_Reinventing_software_quality_3048_3743.png Gojko_Adzic_-_Reinventing_software_quality_3744_4139.png Gojko_Adzic_-_Reinventing_software_quality_4140_6755.png Gojko_Adzic_-_Reinventing_software_quality_6756_13367.png Gojko_Adzic_-_Reinventing_software_quality_13368_14327.png Gojko_Adzic_-_Reinventing_software_quality_14328_22967.png Gojko_Adzic_-_Reinventing_software_quality_22968_23615.png Gojko_Adzic_-_Reinventing_software_quality_23616_27455.png Gojko_Adzic_-_Reinventing_software_quality_27456_28139.png Gojko_Adzic_-_Reinventing_software_quality_28140_28715.png Gojko_Adzic_-_Reinventing_software_quality_28716_29123.png Gojko_Adzic_-_Reinventing_software_quality_29124_29807.png Gojko_Adzic_-_Reinventing_software_quality_29808_30107.png Gojko_Adzic_-_Reinventing_software_quality_30108_32699.png Gojko_Adzic_-_Reinventing_software_quality_32700_32843.png Gojko_Adzic_-_Reinventing_software_quality_32844_32999.png Gojko_Adzic_-_Reinventing_software_quality_33000_33047.png Gojko_Adzic_-_Reinventing_software_quality_33048_33287.png Gojko_Adzic_-_Reinventing_software_quality_33288_33767.png Gojko_Adzic_-_Reinventing_software_quality_33768_33827.png Gojko_Adzic_-_Reinventing_software_quality_33828_34439.png Gojko_Adzic_-_Reinventing_software_quality_34440_35099.png Gojko_Adzic_-_Reinventing_software_quality_35100_35543.png Gojko_Adzic_-_Reinventing_software_quality_35544_35915.png Gojko_Adzic_-_Reinventing_software_quality_35916_36575.png Gojko_Adzic_-_Reinventing_software_quality_36576_39527.png Gojko_Adzic_-_Reinventing_software_quality_39528_39707.png Gojko_Adzic_-_Reinventing_software_quality_39708_43523.png Gojko_Adzic_-_Reinventing_software_quality_43524_43751.png Gojko_Adzic_-_Reinventing_software_quality_43752_44435.png Gojko_Adzic_-_Reinventing_software_quality_44436_50243.png Gojko_Adzic_-_Reinventing_software_quality_50244_50507.png Gojko_Adzic_-_Reinventing_software_quality_50508_52871.png Gojko_Adzic_-_Reinventing_software_quality_52872_54563.png Gojko_Adzic_-_Reinventing_software_quality_54564_59015.png Gojko_Adzic_-_Reinventing_software_quality_59016_59447.png Gojko_Adzic_-_Reinventing_software_quality_59448_59615.png Gojko_Adzic_-_Reinventing_software_quality_59616_60251.png Gojko_Adzic_-_Reinventing_software_quality_60252_60359.png Gojko_Adzic_-_Reinventing_software_quality_60360_60539.png Gojko_Adzic_-_Reinventing_software_quality_60540_60803.png Gojko_Adzic_-_Reinventing_software_quality_60804_64679.png Gojko_Adzic_-_Reinventing_software_quality_64680_66323.png Gojko_Adzic_-_Reinventing_software_quality_66324_68567.png Gojko_Adzic_-_Reinventing_software_quality_68568_73907.png Gojko_Adzic_-_Reinventing_software_quality_73908_76307.png Gojko_Adzic_-_Reinventing_software_quality_76308_80051.png Gojko_Adzic_-_Reinventing_software_quality_80052_83399.png Gojko_Adzic_-_Reinventing_software_quality_83400_84527.png Gojko_Adzic_-_Reinventing_software_quality_84528_86855.png Gojko_Adzic_-_Reinventing_software_quality_86856_88163.png Gojko_Adzic_-_Reinventing_software_quality_88164_89867.png Gojko_Adzic_-_Reinventing_software_quality_89868_90239.png Gojko_Adzic_-_Reinventing_software_quality_90240_90683.png Gojko_Adzic_-_Reinventing_software_quality_90684_91151.png Gojko_Adzic_-_Reinventing_software_quality_91152_91319.png Gojko_Adzic_-_Reinventing_software_quality_91320_91439.png Gojko_Adzic_-_Reinventing_software_quality_91440_91883.png Gojko_Adzic_-_Reinventing_software_quality_91884_93352.png