The present Wikipedia article on the Scientific method, section "Elements of the scientific method" states:


  • Experiments (tests of all of the above)

[Emphasis mine. Markup referring to several wikilinks and references omitted. Notably, there was no specific reference attached to the emphasized phrase.]

In this quote I find the final parenthetical emphasized phrase questionable:
because apparently it implies that "characterizations (observations, definitions, measurements)" are subject to experimental tests.

In contrast, I'd say that

  • definitions of notions and terminology by which to express particular hypotheses and predictions endure any tests and possibly resulting refutation of those hypotheses and predictions,

  • definitions of notions and terminology by which to express particular measurement operations endure any application of those measurement operations to given observational data regardless of resulting measurement values, and

  • observational data collected in previous trials and derived measured values pertaining to those previous trials endure the possible collection of observational data and derivation of measurements in any subsequent trials.

Focussing especially on definitions, therefore I'd like to know:

Is there anyone besides Wikipedia known to claim explicitly that definitions can be experimentally tested?

Is there anyone known who would follow my claim instead, as sketched above, that definitions cannot be experimentally tested?

Is there any possible resolution of this conflict, for instance by considering different notions of "definition" and/or of "experimental test"?

  • Comments are not for extended discussion; this conversation has been moved to chat. – stoicfury Dec 12 '14 at 16:29

You can test whether a definition is a good one by assessing whether it parsimoniously divides things where distinctions need to be made.

For example, let's say a restaurant is "swell" if it has a rating in the top half (of all ratings of restaurants), and it is "lame" if it is not. It will turn out that there are a huge number of restaurants that are essentially indistinguishable in quality (those that are about average) but are labeled "swell" or "lame". This means our definitions are--at least for the purposes of distinguishing restaurants we wish to go to from those we do not--poor. It's not false, since you can define anything you want (as long as it's not logically inconsistent), but it's wrong in that it doesn't map onto the structure of the world the way we (at least implicitly) want it to.

Also, some definitions make implicit truth-claims, e.g. if you have procedural vs. declarative memory, you are implying that a memory cannot simultaneously be both--and that implicit claim is subject to test, and if a memory can be both, then the definition needs at least to be amended.

Whether a definition is good or not depends also on whether it is good for us even if it isn't universal. For example, we say that yellow and violet are complementary colors not because of the properties of the electromagnetic spectrum at wavelengths that appear to us to be yellow and violet, but because of how our three-receptor color vision works. It is true that yellow and violet mixed together appears uncolored (gray) when mixed together in appropriate proportions, while if you mix yellow and blue in various proportions you will get a spectrum of non-gray colors. The definition of complementary colors therefore can be tested and found to be sound (i.e. there is actually an interesting phenomenon there that it captures, even if it is specific to human perception of color).

  • Rex Kerr: A considerate and helpful answer, thanks, +1. I take from it the following: "an experimental test of a consistent, correctly understood definition, in a particular trial" is by itself still not sensible and meaningful, but requires some specific additional measure of "what we want (utility)". We may then ask whether or not the given definition "provided what we want, in the trial under consideration". (This seems very different from "an experimental test of a specific hypothesis or prediction, in a particular trial" which appears rather unambiguous as it stands.) – user12262 Dec 10 '14 at 6:18
  • 1
    @user12262 - Since the "what we want" is the same thing as why we are presumably doing science in the first place, I don't think it's actually "very different". You do have to be aware of it when interpreting the test, though, which is not always necessary when determining whether the prediction of a model is accurate or not. – Rex Kerr Dec 10 '14 at 6:59
  • Rex Kerr: Let your "swell/lame" definition be based on annual ratings. Now suppose the mayor of city X says: "People think there's always more lame restaurants going out of business than swell ones. Can you check that, for this year, and the past five?". This seems an unambiguous assignment. But OTOH, being asked to "test your definition (for city X, and years 2009 - 2015)", you may need clarification: "Testing it for what??: Usefulness for Tourists? Gourmets? Students? City planners? ...". Doesn't that illustrate the difference between testing hypotheses/predictions vs. "testing definitions"? – user12262 Dec 10 '14 at 15:44
  • @user12262 - The mayor's request is unambiguous, but ambiguity is not the problem. The problem is that the distinction between two classes has been drawn at the point where the least distinction exists. So by almost any measure of utility, this is a bad place to draw a distinction, and thus one of the least helpful definitions to make. – Rex Kerr Dec 10 '14 at 20:12
  • Rex Kerr: I understand and agree that you meant the "swell/lame" scale definition to be less distinctive than the underlying "given ratings"; which may in turn be less distinctive than the judgements noted in the "log book" of each restaurant tester. The "swell/lame" scale may be quite useless to gourmets, and unfair to the best "lame" restaurants. (Nevertheless: convenient for tourists, for instance.) These are features of your example. But my question was general; and the "ambiguity" (even: "ill-formedness") shown above seems a general problem. Thanks again for helping me to pinpoint it. – user12262 Dec 11 '14 at 7:02

There is a scene in Lakatos's Proofs and Refutations that goes something like this:

"We define a polyhedron to be a three dimensional solid with only straight edges."

Someone draws a cube with another cube inside it -- "This is a polyhedron?"

"Oh, I guess we need to add the fact that the object is connected."

"But this is a cube with a cube cut out of the middle of it. It is connected."

"I mean the edges must be connected when they are drawn out, We need to add an edge that connects the outer cube to the inner cube."

"In what way is that an edge? It lies along the side of no face."

"Well, then we will reduce the definition of a polyhedron for our purposes to be a connected drawing of a network of polygons."

"Well we just had trouble agreeing on what an edge was, so we should define a polygon."

"A polygon is a connected, closed sequence of non-intersecting line segments embedded in a plane."

(Several attempts to define a polygon fail.)

In what way is this not experimental evidence disproving a succession of definitions?

You might consider drawing or building geometrical objects to be just thought experiments, and not real experiments. But either thought-experiment data is data or Relativity was not science until the 40's. And constructions or direct representations are awfully concrete for thought-experiments.

Further, in practice these objections do not arise a priori like this, at the very start of a theory. The counterexamples are found when predictions fail and we realize the flaw was in an overly-aggressive definition that is in fact not properly fit to do the job of defining.

So, yes, data tests definitions, there is not a nice neat waterfall that goes define -> predict -> evaluate -> refine -> next theory.

(I would suggest you read the actual Proofs and Refutations just because it is simple, short, entertaining, and makes some very good points.)

Another approach is to look at something from a weaker science like psychology. Look at intelligence or personality theory.

Every definition of intelligence begs the question as to whether it is stable enough to mean anything. Half the theoretical literature in the field for about 25 years, was using experimental data to prove or disprove the well-foundedness and non-circularity of a given definition of intelligence. (If a definition is not well-founded, or is ultimately circular, then is it a definition, or just a logical structure that will hopefully converge upon one?)

(You cannot give up and claim you have a test that measures success on the test, without discarding the usefulness of the test. That is simply defensive nonsense. The test is constructed from a theory, with specific defined attributes dictated by the theory.)

Either none of this is science, or a whole lot of empirical data may be needed to establish whether a definition actually defines anything. This cannot be done with mere thought-experiments.

Likewise, every personality theory has a totally different definition of personality, and proceeds to continually refine that definition based upon observations.

Are the Jungian types: Introverted vs Extroverted, Sensing vs Intuitive, Thinking vs Feeling, defined? You can read the definitions, from back in the 1930's, but they won't really help unless you run some experiments, or you are Jung. The definitions are observations of what kinds of communication techniques address different kinds of people in different states. So is there no definition here? Are these undefined, yet useful distinctions? To what degree does that leave meaning to the word definition?

  • jobermark: "a scene in Lakatos's Proofs and Refutations [...] "Oh, I guess we need to add the fact [...]"" -- This supposed "need" doesn't seem to arise from the (initial) definition itself, nor from any or all imaginable examples/instances (which may be expressed by the available terminology) themselves but ... from some extraneous "what we want" (to use Rex Kerr's phrase). Unfortunately I'm not yet fluent in applicable "technical terms" (as far as any have been coined already); but at least I had read about "Lakatos" already. So reading "P&R", for once, seems good advice. Thus +1. – user12262 Dec 10 '14 at 22:19
  • 1
    How is what we intended to define extraneous? And did you consider any of the rest of the post, or just stop there? – user9166 Dec 11 '14 at 0:03
  • jobermark: "How is what we intended to define extraneous?" -- Well, you name it ("intent"). Can any one specific definition (e.g. in terms of "points", "edges", and, along with "straigthness", perhaps a full-fledged metric space) not be considered and used without regard of intent? (Even: of intents of whoever might be interested?) Doesn't science (especially experimental) rather subscribe to "impartiality"? ... "And did you consider any of the rest of the post" -- Only with growing incomprehension/displeasure; sorry. – user12262 Dec 11 '14 at 6:18
  • @user12262 That is our divergence. Definitions of long standing may be impartial, but they are never so when initially proposed -- they seek some goal or capture some individual's intuition, so they are biased. Science may work toward impartiality, but it cannot start there. There is no purely objective observation, rule or choice of words, but in combining enough you can get balance and thus impartiality. So we need to freely fix definitions, moving toward including more perspectives in our understanding, rather than imagining they originate with some special warrant against error. – user9166 Dec 11 '14 at 10:40
  • That is why the post contains the examples of IQ and personality traits. You can watch the history slowly wind in on the definitions. From the actual literature you have to see it change over time. If you like harder science, consider the process by which birds became dinosaurs, or Pluto stopped being a planet. The definitions got better. – user9166 Dec 11 '14 at 10:44

Lets try another route. Yes, there are definitely different kinds of definitions, in mathematics, where such things tend to be most restrictive, there are at least three:

  1. As compressive notation: We can define the real numbers as distances along a chosen line. This is, in the end, completely circular and does not define anything. We already had the notion of the real numbers, or we would not be measuring things, but it makes a new way of saying things that is more compact when we are focussed on points rather than measures.

    An example from outside math of this kind of definition is defining "work" as the product of force and distance, just so we can refer to it efficiently.

  2. Via distinction and equivalence: We can define the real numbers as the equivalence classes of convergent sequences of rational numbers the unions of which still converge. This brings up background questions. We have to establish that the condition on the equivalence satisfies the requirements of an equivalence relation. That can be work. If it failed, we would not have a definition. We would not have accidentally defined something else, other than the ordinary reals, we would in fact have a statement that if treated as a definition, would give us false proofs.

    An example from outside of math is the definition of "species". What defines the word is the ability to describe how to tell whether two animals are of the same species, and just that. (I don't mean the standards for each species, but the idea that those standards exist generally.) It is important that the distinguishing criterion not get too vague or become ambivalent about a too many pairs of animals, or we have to start questioning the concept of species itself.

  3. Via models of axioms: We can define real numbers as the minimal ordered field with consistent least upper bounds. Besides being oddly abstruse, this also pushes a horde of assumptions. It is the minimal model of this set of axioms, there are not thousands of unrelated models that all behave differently -- they all contain this one, and that takes a lot of proving.

    An example of this kind of definition from outside math is that of 'the alpha of a troupe'. It picks out a specific kind of animal from a group by its behavior and that of those around it. The individual models an archetype or a stereotype made up of general rules. If we find different applications of the same checklist of features leads to disagreement too often, we have to discard the definition, and perhaps question the applicability of the concept itself.

All three of these would be considered genuine definitions by almost any mathematician. So anything you say about all definitions must apply to all of them, or you need to go scold the entire field of math for not knowing what a definition is.

Logical positivists like to imagine that all definitions really take the first form, at root, in some indirect way, and that all of language goes back to symbolic logic. But they are wrong: their enterprise fell apart under the weight of formal proof and observational data long ago. Even to the degree mathematics can be reduced to symbolic logic, the axioms of set theory are not strictly unique and therefore purely logical.

Even if that were somehow true, it is not what the practitioners mean when they use the word 'definition'. They do not unwind every case into set theory. Instead, they prove the internal consistency and non-circularity of their definition as a statement, the same way they would prove any other theorem. In most higher mathematics, the very first thing you have to do after giving a definition is prove it is a definition.

When these approaches are used in an experimental science, those proofs of applicability need not be formal mathematics, but are usually made on the basis of observations, instead. I know when I am defining a new species of bird, not because I have formally plumbed the separation criteria between animals and the equivalence classes created by enumerating traits, but because I can look at my clade diagram and see where the overlaps with other species are ruled out. But all of that on the chart is data, not math, and any part might easily have been misinterpreted.

Then when the observation offered instead of formal proof is challenged by new data the definition is tested by experimental evidence.

  • jobermark: "there are definitely different kinds of definitions" -- That's certainly addressing the final point of my OP: +1. "1. As compressive notation: [...] the product of force and distance, just so we can refer to it efficiently." -- That's pretty much my perspective. However: in "sufficiently complicated contexts" there may be several overtly/formally distinct "definientia" which turn out (with effort by "practitioners") to be equivalent; thus correctly all being compressed/denoted as the same "definiendum term". – user12262 Dec 15 '14 at 19:18
  • jobermark: "defining a new species of bird, [...] I can look at my clade diagram and see where the overlaps with other species are ruled out. But all of that on the chart is data" -- It's evaluated observational data: measured "trait" values. Which requires some undelying, stable, applicable definition for "how to". "So anything you say about all definitions must apply to all of them, or you need to go scold the entire field of [...]" -- As I tried to say in the OP: foremost I'd scold Wikipedia; aiming to improve it. – user12262 Dec 15 '14 at 19:19
  • Did you read "Logical positivists like to imagine that all definitions really take the first form, at root, in some indirect way, and that all of language goes back to symbolic logic. But they are wrong: their enterprise fell apart under the weight of formal proof and observational data long ago. Even to the degree mathematics can be reduced to symbolic logic, the axioms of set theory are not strictly unique and therefore purely logical." Don't scold Wikipedia, because it is right. You, on the other hand, are subscribing to a philosophy of definition that fails. – user9166 Dec 15 '14 at 19:49
  • Since not all definition really takes the form of compressive notation, but it is instead part of a massive feedback loop, just like all interpretation, these two other forms of definition are real and distinct from that. When you make a definition, instead of just relying on the (unattainable) precision of language, you must prove it will obey substitution criteria to an acceptable degree. That proof can be formal or experimental. – user9166 Dec 15 '14 at 19:54
  • Yes, all experimental evidence is partially circular, and incorporates prior evaluations, yet it is used as proof. And yes, there is an underlying definition of "species" that allows one to order the separate definition of each species. But those definitions are both held up by observation, not mere language conventions. – user9166 Dec 15 '14 at 20:06

Your Answer

