
To start with, no methodology or form of reasoning is infallible.  So
there's a question of how much certainty we are willing to pay for in a
given context.  Insistence on too much rigor bogs science down completely
and makes progress impossible.  (Expenditure of sufficiently large sums
of money and amounts of time can sometimes overcome this.)  On the other
hand, with too little rigor much is lost by basing work on results which
eventually turn out to be false.  There is a morass of studies
contradicting other studies and outsiders start saying  "You people call
THIS science?"   (My opinion, for what it's worth, is that one sees both
these phenomena happening simultaneously in some parts of psychology.)  

Some subjective judgement is required to decide on the level of rigor
appropriate for a particular investigation.  I don't believe it is 
ever possible to banish subjective judgement from science.  


My second point, though, is that highly capable people can often make
extremely reliable judgements about scientific validity even when using
methodology considered inadequate by the usual standards.  I think this
is true of many scientists and I think it is true of many who approach
their discipline in a way that is not generally recognized as scientific.

Within mathematics, I think there are several examples, especially before
the twentieth century.  One conspicuous case is that of Riemann, who is
famous for many theorems he stated but did not prove.  (Later 
mathematicians did prove them, of course.)  

I think that for a good scientist, empirical investigation is often not
so much a matter of determining what is true and what's not  as it is a 
matter of convincing other people.  (People have proposed lots of 
incompatible definitions of science here, but I think the ability to 
objectively convince others of the validity of one's results is an
essential element.  Not that one can necessarily do that at every step 
of the scientific process, but I think that if one is not moving toward 
that goal then one is not doing science.)

When a person other than a scientist is quite good at what he does and
seems to be very successful at it, I think that his judgements are also
worthy of respect and that his assertions are well worth further
investigation.  


Admittedly, my question was not at all well posed.  A considerable
amount of effort in a "serious scholarly investigation" such as I
suggested would be required simply to formulate an appropriately 
specific question to try and answer.  

The "science" I was thinking of in my question is the actual science 
currently practiced now in the last decade of the twentieth century.  
I certainly wasn't thinking of some idealized science or the mere use 
of "reason and observation."

One thing I had in mind in my suggestion was the question as to whether
in many cases the subjective judgements of skilled and experienced
practitioners might be more reliable than statistical studies.  

Since Russell Turpin seems to be much more familiar than I am with
the study of scientific methodology, perhaps he can tell us if there 
is any existing research related to this question.  

--
In the arguments between behaviorists and cognitivists, psychology seems 
less like a science than a collection of competing religious sects.   