What could help fix belief problems?

[cross-post from Medium]

Last part (of 4) from a talk I gave about hacking belief systems.

Who can help?

My favourite quote is “ignore everything they say, watch everything they do”. It was originally advice to girls about men, but it works equally well with both humanity and machines.

If you want to understand what is happening behind a belief-based system, you’re going to need a contact point with reality. Be aware of what people are saying, but also watch their actions; follow the money, and follow the data, because everything leaves a trace somewhere if you know how to look for it (looking may be best done as a group, not just to split the load, but also because diversity of viewpoint and opinion will be helpful).

Verification and validation (V&V) become important. Verification means going there. For most of us, verification is something we might do up front, but rarely do as a continuing practice. Which, apart from making people easy to phish, also makes us vulnerable to deliberate misinformation. Want to believe stuff? You need to do the leg-work of cross-checking that the source is real, finding alternate sources, or getting someone to physically go look at something and send photos (groups like findyr still do this). Validation is asking whether this is a true representation of what’s happening. Both are important; both can be hard.

One way to start V&V is to find people who are already doing this, and learn from them. Crisismappers used a bunch of V&V techniques, including checking how someone’s social media profile had grown, looking at activity patterns, tracking and contacting sources, and not releasing a datapoint until at least 3 messages had come in about it. We also used data proxies and crowdsourcing from things like satellite images (spending one painful Christmas counting and marking all the buildings in Somalia’s Afgooye region), automating that crowdsourcing where we could.

Another group that’s used to V&V is the intelligence community, e.g. intelligence analysts (IAs). As Heuer puts it in the Psychology of Intelligence Analysis (see the CIA’s online library), “Conflicting information of uncertain reliability is endemic to intelligence analysis, as is the need to make rapid judgments on current events even before all the evidence is in”. Both of these groups (IAs and crisismappers) have been making rapid judgements about what to believe, how to handle deliberate misinformation, what to share and how to present uncertainty in it for years now. And they both have toolsets.

Is there a toolset?

Graphs from a session on statistical significance, AB testing and friends. We can ‘clearly’ see when the two distributions are separate or nearly the same, but need tools to work out where the boundary between these two states are.

The granddaddy of belief tools is statistics. Statistics helps you deal with and code for uncertainty; it also makes me uncomfortable, and makes a lot of other people uncomfortable. My personal theory on that is it’s because statistics tries to describe uncertainty, and there’s always that niggling feeling that there’s something else we haven’t quite considered when we use do this.

Statistics is nice once you’ve pinned down your problem and got it into a nicely quantified form. But most data on belief isn’t like that. And that’s where some of the more interesting qualitative tools come in, including structured analytics (many of which are described in Heuer’s book “structured analytic techniques”). I’ll talk more about these in a later post, but for now I’ll share my favourite technique: Analysis of Competing Hypotheses (ACH).

ACH is about comparing multiple hypotheses; these might be problem hypotheses or solution hypotheses at any level of the data science ladder (e.g. from explanation to prediction and learning). The genius of ACH isn’t that it encourages people to come up with multiple competing explanations for something; it’s in the way that a ‘winning’ explanation is selected, i.e. by collecting evidence *against* each hypothesis and choosing the one with the least disproof, rather than trying to find evidence to support the “strongest” one. The basic steps (Heuer) are:

  • Hypothesis. Create a set of potential hypotheses. This is similar to the hypothesis generation used in hypothesis-driven development (HDD), but generally has graver potential consequences than how a system’s users are likely to respond to a new website, and is generally encouraged to be wilder and more creative.
  • Evidence. List evidence and arguments for each hypothesis.
  • Diagnostics. List evidence against each hypothesis; use that evidence to compare the relative likelihood of different hypotheses.
  • Refinement. Review findings so far, find gaps in knowledge, collect more evidence (especially evidence that can remove hypotheses).
  • Inconsistency. Estimate the relative likelihood of each hypothesis; remove the weakest hypotheses.
  • Sensitivity. Run sensitivity analysis on evidence and arguments.
  • Conclusions and evidence. Present most likely explanation, and reasons why other explanations were rejected.

This is a beautiful thing, and one that I suspect several of my uncertainty in AI colleagues were busy working on back in the 80s. It could be extended (and has been); for example, DHCP creativity theory might come in useful in hypothesis generation.

What can help us change beliefs?

Boyd’s original OODA loop diagram

Which us brings us back to one of the original questions. What could help with changing beliefs? My short answer is “other people”, and that the beliefs of both humans and human networks are adjustable. The long answer is that we’re all individuals (it helps if you say this in a Monty Python voice), but if we view humans as systems, then we have a number of pressure points, places that we can be disrupted, some of which are nicely illustrated in Boyd’s original OODA loop diagram (but do this gently: go too quickly with belief change, and you get shut down by cognitive dissonance). We can also be disrupted as a network, borrowing ideas from biostatistics, and creating idea ‘infections’ across groups. Some of the meta-level things that we could do are:

  • Teaching people about disinformation and how to question (this works best on the young; we old ‘uns can get a bit set in our ways)
  • Making belief differences visible
  • Breaking down belief ‘silos’
  • Building credibility standards
  • Creating new belief carriers (e.g. meme wars)

What goes wrong in belief-based models?

[cross-post from Medium]

This is part 3 of a talk I gave recently on hacking belief systems using old AI techniques. Parts 1 and 2 looked at human and computational belief, and the internet and computer systems development as belief-based systems. Now we look at the things that can (and do) go wrong with belief-based systems.

Machine Belief

Machines have a belief system either because we hard-wire that system to give a known output for a known input (all those if-then statements), or because they learn it from data. We touched on machine learning in part 1 of this series (remember the racist chatbot?), and on how it’s missing the things like culture, error-correction and questioning that we install in small humans. But before we get to how we might build those in, let’s look at the basic mechanics of machine learning.

At the heart of machine learning are algorithms: series of steps that, repeated, create the ability to cluster similar things together, classify objects, and build inference chains. Algorithms are hackable too, and three big things tend to go wrong with them: inputs, models and willful abuse. Bias and missing data are issues on the input side: for instance, if an algorithm is fed racially-biased cause-to-effect inputs, it’s likely to produce a similarly-biased system, and although we can interpolate between data points, if there’s a demographic missing from our training data, we’re likely to misrepresent them.

And although we may appear to have algorithms that automatically turn training data into models, there are almost always human design decisions being used to tune them, from the interpretations that we make of inputs (e.g. mouseclicks are equivalent to interest in a website, or that we make assumptions like 1 phone per person in a country means that everyone has a phone, when it could mean, as is common in West Africa, that some people have multiple phones and others have none) to the underlying mathematical models that they use.

Back in the day, we used to try to build models of the world by hand, with assertions like “all birds can fly”, “birds have wings”, “fred is a bird so fred can fly”, and many arguments over what to do about penguins. That took time and created some very small and fragile systems, so now we automate a lot of that, by giving algorithms examples of input and output pairs, and leaving them to learn enough of a model that we get sensible results when we feed in an input it hasn’t seen before. But sometimes we build systems whose models don’t fit the underlying mechanics of the world that they’re trying to capture, either because we’re using the wrong type of model (e.g. trying to fit straight lines to a curve), the input data doesn’t cover the things that we need it to learn, the world has changed since we built the model, or any of a dozen other things have gone wrong.

Machines can also be lied to. When we build models from data, we usually assume that the data, although inexact, isn’t deliberately wrong. That’s not true any more: systems can be ‘gamed’ with bad data (either misinformation, e.g. accidentally, or disinformation, e.g. deliberately), with machine learning from data on the internet being especially susceptible to this.

And issues in machine belief aren’t abstract: as more systems are used to make decisions about people, Lazar’s catalogue of algorithm risks become important to consider in building any algorithm (these risks include manipulation, bias, censorship, privacy violations, social discrimination, property rights violations, market power abuses, cognitive effects and heteronomy, e.g. individuals no longer having agency over the algorithms influencing their lives).

Human Belief

Humans and machines can go wrong in similar ways (missing inputs, incorrect assumptions, models that don’t explain the input data etc), but the way that humans are wired adds some more interesting errors.

When we’re shown a new fact (e.g. I crashed your car), our brains initially load in that fact as true, before we consciously decide whether it’s true or not. That is itself a belief, but it’s a belief that’s consistent with human behaviour. And it has some big implications: if we believe, even momentarily, that something false is true, then by the time we reject it, it’s a) left a trace of that belief in our brain, and b) our brain has built a set of linked beliefs (e.g. I’m an unsafe driver) that it doesn’t remove with it. Algorithms can have similar memory trace problems; for instance, removing an input-output pair from a support vector machine is unlikely to remove its effect from the model, and unpicking bad inputs from deep learning systems is hard.

Humans have imperfect recall, especially when something fits their unconscious bias (try asking a US conservative about inauguration numbers). And humans also suffer from confirmation bias: they’re more likely to believe something if it fits their existing belief framework. Humans also have mental immune systems, where new fact that are far enough away from their existing belief systems are immediately rejected. I saw this a lot when I worked as an innovations manager (we called this the corporate immune system): if we didn’t introduce ideas very carefully and/or gradually, the cognitive dissonance in decision-makers’ heads was too great, and we watched them shut those ideas out.

We’re complicated. We also suffer from the familiarity backfire effect, as demonstrated by advertising: the more someone tries to persuade us that a myth is wrong, the more familiar we become with that myth, and more likely we are to believe it (familiarity and belief are linked). At the same time, we can also suffer from the overkill backfire effect, as demonstrated by every teenager on the planet ever, where the more evidence we are given against a myth, and the more complex that evidence is compared to a simple myth, the more resistant we are to believing it (don’t despair: the answer is to avoid focussing on the myth whilst focussing on more simple alternative explanations).

And we’re social: even when presented with evidence that could make us change our minds, our beliefs are often normative (the beliefs that we think other people expect us to believe, often held so we can continue to be part of a group), and resistance to new beliefs is part of holding onto our in-group identity. Counters to this are likely to include looking at group dynamics and trust, and working out how to incrementally change beliefs not just at the individual but also at the group level.

Combined belief

 house has-a
Image: http://users.cs.cf.ac.uk/Dave.Marshall/AI2/node145.html

Human and machine beliefs have some common issues and characteristics. First, they have to deal with the frame problem: you can’t measure the whole world, so there are always parts of your system potentially affected by things you haven’t measured. This can be a problem; we spent a lot of the early 80s reasoning about children’s building blocks (Winston’s blocks world) and worrying about things like the “naughty baby problem”, where sometime during our careful reasoning about the world, an outside force (the “naughty baby”) could come into the frame and move all the block orders around.

We also have input issues, e.g. unknown data and uncertainty about the world. Some of that uncertainty is in the data itself; some of it is in the sources (e.g. other people) and channels (e.g. how facts are communicated); a useful trick from intelligence system design is to assess and score those 3 sources of uncertainty separately. Some uncertainty can be reduced by averaging: for instance, if you ask several farmers for the weight of a bull, each individual guess is likely to be wrong, but an average of the guesses is usually close to the real weight (this is how several cool tricks in statistics work).

And then there’s human bias. A long time ago, if you wanted to study AI, you inevitably ended up reading up on cognitive psychology. Big data didn’t exist (it did, but not applied to everything: it was mainly in things like real-time sonar processing systems), the Cyc system was a Sisyphean attempt to gather enough ‘facts’ to make a large expert system, and most of our efforts revolved around how to make sense of the world given a limited number of ‘facts’ about it (and also how to make sense of the world, given a limited number of uncertain data points and connections between them). And then the internet, and wikipedia, and internet 2.0 where we all became content providers, and suddenly we had so much data that we could just point machine learning algorithms at it, and start finding useful patterns.

We relied on two things: having a lot of data, and not needing to be specific: all we cared about was that, on average, the patterns worked. So we could sell more goods because the patterns that we used to target the types of goods to the types of people most likely to buy them, with the types of prompts that they were most likely to respond to. But we forgot the third thing: that almost all data collection has human bias in it — either in the collection, the models used or the assumptions made when using them. And that is starting to be a real problem for both humans and machines trying to make sense of the world from internet inputs.

And sometimes, beliefs are wrong


Science is cool — not just because blowing things up is acceptable, but also because good science knows that it’s based on models, and actively encourages the possibility that those models are wrong. This leads to something called the Kuhn Cycle: most of the time we’re building out a model of the world that we’ve collectively agreed is the best one available (e.g. Newtonian mechanics), but then at some point the evidence that we’re collecting begins to break that collective agreement, and a new stronger model (e.g. relativity theory) takes over, with a strong change in collective belief. It usually takes an intelligent group with much humility (and not a little arguing) to do this, so it might not apply to all groups.

RedCross t-shirt after 2010 Chile earthquake

And sometimes people lie to you. The t-shirt above was printed to raise funds for the Red Cross after the Chile 2010 earthquake. It was part of a set of tweets sent pretending to be from a trapped mother and child, and was believed by the local fire brigade, who instead found an unaffected elderly couple at that address. Most crisismapping deployments have very strong verification teams to catch this sort of thing, and have been dealing with deliberate misinformation like this during crises since at least 2010. There’s also been a lot of work on veracity in data science (which added Veracity as a fourth “V” to “Volume, Velocity, Variety”).

Fox News visualisation: not exactly untrue, but not emphasising the truth either

And sometimes the person lying is you: if you’re doing data science, be careful not to deceive people by mistake, e.g. by not starting your axes at zero (see above), you can give readers a false impression of how significant differences between numbers are.

Why should technologists care about belief?

[cross-posted from Medium]

This is part of a talk on hacking belief systems using old AI techniques. The first part covered human beliefs, machine beliefs, and deep learning and the internet as belief-based systems.

Lean and Agile are based on belief

The build-measure-learn cycle. Image from https://steveblank.com/2015/05/06/build-measure-learn-throw-things-against-the-wall-and-see-if-they-work/

Most system development I see now uses agile principles: short, iterative cycles of development, and an emphasis on learning, preferably alongside the system’s users, and adapting their systems to increase some measure of customer or user value. Lean is a similar idea at the company level, based around learning quickly about what works through prototyping and measurement (aka the “build-measure-learn” cycle). Lean Startup is the small-scale version of this; Lean Enterprise is the large-organisation one.

I’m going to focus on Lean Enterprise because it’s more complex. In an established organization, Lean’s usually a combination of learning and adapting existing business, and exploring new business ideas (with innovations people bridging across these). Core ideas in this include purpose, customer value and incremental learning.

  • Purpose is the ‘why’ of an organization, and acts as the common vision that everything in the organisation works towards (Daniel Pink’s videodescribes this well).
  • Customer value is what the customer needs instead of the things we think they’d like to have. Concentrating on customer value reduces waste in the organisation.
  • Iterative learning is about learning in small increments so we can adapt quickly when markets or needs change, and don’t bet too many resources on an idea that might not work out.

The build-measure learn cycle is at the heart of this. We start with ideas, build the smallest thing that we can learn something useful about them from (this might be code, but it might also be wireframes, screenshots, a pitch, a talk), test it, measure the results of our tests, learn from them, adapt our idea and iterate again.

And at the heart of this cycle are value hypotheses: testable guesses about the problems that we think we’re trying to solve (“our mailing lists are too general”), and the solutions that we think could solve them (“more people will vote for us if we target our advertisements to them”). We make each guess, often framed as a “null hypothesis” (a statement that what we’re seeing in our results is probably due to random chance rather than the effect that we were hoping to see), then try to prove or disprove it.

There’s a lot of design thinking behind hypothesis generation, and there’s a whole different post to write on how hypotheses breed smaller hypotheses and form hierarchies and structure, but the point here is that a hypothesis is an expression of a belief, and that belief and its management is at the core of Lean.

Lean value trees link beliefs together

Mock-up Lean Value Tree for Bannon’s brain

A lean value tree (LVT) links an organisation’s purpose to its beliefs, metrics (the ways it measures itself), hypotheses and experiments. The layers in the tree build on each other, and are:

  • The organisation’s purpose (aka mission): who is it, and what its vision is for itself. This is a stable belief, owned by the organisation’s executive.
  • Goals that support the purpose, framed as desired, stable and measurable outcomes.
  • Strategic bets on those goals: different ways that we could satisfy a goal, with metrics associated with them. Stable in the medium term, owned below executive level.
  • Promises of value: value-driven initiatives against those bets. Stable in the medium-term, owned by the product team.
  • Hypotheses: measurable, testable guesses that support or refute the promises of value. Experiments on these are very short term (often quick and dirty) ways to learn about the problem space and potential solutions, and are owned by the development team.

Beliefs (hypotheses) are built into the tree, but belief is also built into how we manage the tree. One layer is called “strategic bets” because organisations rarely have the resources (teams, time, money) to do everything on their LVTs, and need to prioritise where they put those resources, by allocating them to the more ‘important’ things on the tree, where “important” is belief-based: a combination of beliefs in how big the market connected to a bet is, and how much value (financial or otherwise) is in that market.

HDD card (from Barry O’Reilly post on HDD)

Hypothesis driven development is an extension of the arc from test driven development to behavior driven development: its purpose is to learn, and to experiment quickly with ideas. Before a team tests a HDD hypothesis, it fills out a card (like the one above) describing the work done, the changes that we expect from that work, and a success condition (a “signal”) that the team believes will happen if the experiment succeeds. Writing the success condition (e.g. “1% increase in donations”) before testing is important because it a) forces the team to concentrate on what they would accept as an outcome, and b) stops them fitting the test to the results they obtained (which could perhaps be called “optimism driven development”).

LVTs are useful models for talking not only about what we want within our own organisations, but also for testing our assumptions and beliefs about the goals and actions of other organisations, including organisations competing with us in the same markets.

We’re really optimising on other peoples’ beliefs

Lean practitioners talk a lot about customer value, and about including users in its feedback loops (either directly, or by tracking the changes in user behavior after changing the system). This optimizes systems not on an organisation’s beliefs, but on its beliefs about other peoples’ beliefs, filtered through those other peoples’ behavior. This is why things like good survey design and execution are difficult, with many types of observer effect (e.g. people doing and saying what they think you want to hear, responding differently to different ways of asking etc) that need to be allowed for.

Image: http://www.rosebt.com/blog/descriptive-diagnostic-predictive-prescriptive-analytics

But this isn’t just about understanding customer beliefs; it’s also about using that knowledge to predict and adjust customer behavior. Data scientists have a diagram for that, and talk about their work as a continuum from traditional statistics summarising data about events that have already happened (descriptive and diagnostic analytics), to predicting what might happen and how we could influence that (predictive and prescriptive analytics). But there is another box at the top of this descriptive-diagnostic-predictive-prescriptive diagram, describing a feedback loop where we learn and adapt our environment given that learning. This box is where the belief hackers, the roboticists and the lean enterprise people live.

Mixing human and computational belief

[cross-post from Medium]





You don’t escape the machine





First technique: uncertain reasoning


Image: wikipedia article on belief networks



Image: wikipedia article on belief networks




Learning = forming beliefs





Beliefs can be shared








Beliefs can be hacked




Image: wikipedia page on the Muller-Lyer illusion



The Internet is made of belief

websearch result for “fake news”


