Following on from this comment from Sven Yargs, I think we should consider making it clear to users what research we expect.

I often see a poorly researched question and assume the research would be easy. Sometimes I try it and find out it's not. In order to save me doing the same thing the OP did, it'd be better if they stated exactly what they tried.

As per Sven Yargs's comment, I think we can make it easier by telling users who have their questions closed for lack of research, what exactly constitutes tell us the research you've done. Too often they edit in something generic: e.g. "I searched for it and didn't find anything". What's "it"? Did they include quote marks in your search to make sure google knew it was a phrase? Did they check reputable dictionaries? Did they only check US dictionaries or only UK dictionaries? As far as they're concerned they've done research and proved it, and thus met our "show your research" criterion; as far as I'm concerned they've done minimal research and proved nothing and not met my interpretation of the criterion.

I'd suggest something like this (based on Sven Yargs's wording):

If you can't find any information, please identify your search term(s) and at least three references you checked that were not helpful.

| |
  • I prefaced my suggestion with the words "If EL&U's position is that users who can't find any information relevant to their question on their own should specify the sources they checked, ..." But instead of making the research requirement clearer and more stringent, I think EL&U should replace the "show research" close reason with "too localized"—for the reasons given in my answer to "What is the best way of dealing with the questions that are too localised?" – Sven Yargs May 20 '19 at 16:53
  • @SvenYargs - Apologies if you feel I've mischaracterised your comment. I wasn't sure if your comment was a bit sarcastic, but my position is definitely that users should specify the sources they've checked, hence I created this meta post. – AndyT May 21 '19 at 10:45
  • I think your position is a reasonable one and deserves consideration on Meta. I just wanted to make clear that my own take on the issue of the "show research" close reason is that it is inherently flawed because it doesn't tell people who post bad questions what is actually bad about their questions—and would still be bad regardless of how much research they showed. – Sven Yargs May 22 '19 at 4:33
  • @Mari-LouA - What does "warn users they must post a sample sentence" have to do with "replace Show your Research with clearer wording"? – AndyT May 23 '19 at 9:05
  • Placing on hold or closing questions for lack of research is often the excuse or legitimate reason chosen by close voters. Your question is a variation of the same theme, fixing or explaining to “new contributors” how to improve their low quality questions. We're just going round round in circles. LQQs are here to stay because anyone in the world can post a question. Sounds lovely and democratic in theory but the reality tells us differently. – Mari-Lou A May 23 '19 at 9:16
  • But you're right the two are not duplicates, so I shall withdraw my close vote. It was also something like 4 am in the morning when I commented. Apologies. – Mari-Lou A May 23 '19 at 9:21
  • @Mari-LouA - I agree that it's a variation on a theme. But when we close a question due to "lack of research", and the OP then edits in the phrase "I googled it and found nothing", then I'm not going to reopen it. As far as they're concerned they've done research and proved it, and thus met our "show your research" criterion; as far as I'm concerned they've done minimal research and proved nothing and not met my interpretation of the criterion. This proposal is aimed at preventing that specific case. I'll edit this in as I didn't explain it properly first time. – AndyT May 23 '19 at 9:24
  • Users saying they did research are wrong because the guideline clearly says "show" meaning display, i.e. showing physical proof not just saying "done that". Too bad if they don't understand or deliberately refuse to follow simple instructions, their questions should be closed (and maybe even downvoted). – Mari-Lou A May 23 '19 at 9:34

I agree with most of what @Tonepoet said, but I would like to try a simpler answer.

Anything more complicated than a "clear threshold limit" of non-zero will be too complicated to define and enforce, and the sensibleness of it will vary from type of question to type of question.

Many questions simply go away before ever being asked if the OP does a quick search; I have often thought I had an interesting question to find that a few minutes on Google answered it. So I didn't post the Q.

We would like more such trivial questions to be self-aborted by the OPs. But we don't want all questions or even most questions to be self-aborted, and many will be with the suggested three references rule. So, I would edit the proposed comment of the OP to:

If you can't find any information, please identify your search term(s) and the reference(s) you checked that were not helpful.

| |

Here's a still admittedly somewhat lengthy attempt to summarize my more detailed thoughts:

  1. I agree with some of AndyT's concerns, but questions which are difficult to research should be answered, in order to achieve Stack Exchange's goal of disseminating information.

  2. Proving a negative is harder than proving a positive. It is at least hypothetically possible for somebody to try and do research and have absolutely nothing to show for it, or at least nothing worth showing, which leaves them with little to no recourse for reopening the question, even if it is worth answering. This is most particularly the case for when people who do cursory research with a search engine and find no reference websites with their search, and other forms of research can be harder.

  3. I doubt delimiting strict expectations will do much to help us. It would be one thing if the clarification could raise compliance with our standards, but we have enough difficulty getting people to do much of anything without asking even more of them than what we presently enforce, and any amount of shown research shows that they did the least they could do for the people bothered by that. We would probably only see more questions closed and fewer questions reopened, because it will require more work to keep them open or mend them.

  4. If the main purpose of the research requirement is to reduce the number of answers that solely rest on an appeal to authority from a readily accessible resource, on the basis that questions which can be answered that way do not merit our personalized assistance, then one reference usually suffices to highlight the uselessness of answers that just duplicate co-equal resources, since the veracity of information of one co-equal authority like can not be trusted more than another like it without further proof or explanation.

  5. Unfortunately, in cases where no productive results are found on a search engine, the questioner providing us with their search terms probably does little to nothing to help us, even under unlikely optimal circumstances. They will not always reliably produce the same results that the questioners and other visitors may find, even in the best case scenario, so we gain little to no insight regarding the nature of the problem. It also requires us to manually double check anyway, and if we are going to bother doing that, then I trust we are capable of devising our own and evaluate if answering the question is needful for ourselves.

I would like to add that there may be ways to improve the phrasing of our current closure reason, but I doubt delimiting stricter standards as suggested will help for these reasons.

Look, I know that strictly speaking the present research standards endorse checking multiple resources, but we have never actually gone so far as to enforce it to my personal recollection, and it may just be too high of a threshold to be reasonable, so I do not think we should start.

Firstly, I have difficulty seeing much of a practical benefit, if the of our research standard goal is simply to bar questions that are too simple. If a user checks even just one authoritative reference work, then I think it does enough to solve the problem of an overly simple question because who's to say,for example that Merriam-Webster's 11th Collegiate Dictionary defines a word better than The Oxford Dictionary of English 3rd edition, or vice versa, without a more thorough examination of the circumstances and facts that can be used as a basis to explain which one did a better job? Moreover, if all of the dictionaries are in perfect agreement, then checking one is more or less effectively the same as checking them all anyway. My opinion is that checking even just one reference work should be enough, but we often have difficulty persuading new users to even do just that and I think the community would be elated if we found a way to encourage even just that. Referencing even just one authority does enough to prevent merely making an appeal to an equivalent authority.

Moreover, we often have difficulty persuading people to edit questions to add any any amount of research, and I think we would be very happy if they did. We should keep in mind that if the research standard is too high that we may start losing questions that could have been answered very well to other resources.

However, more importantly, I doubt making the standards clearer and stricter solve the problem you are experiencing anyway. I think the root of the problem may be that they are too clear, and too strict for us, because they inadvertently cause a problem. Questions are just closed whenever they lack research, without any effort on our part to determine if research would have much practical benefit. Now do not get me wrong: I am of the opinion that a question with research is better than a question without it, but consider this:

Showing an absence of evidence is much harder than proving a positive. How should people even show just one reference to a question which doesn't have an answer, let alone three? Do they give us a link to a nonexistent webpage? Do they read an entire book that might be expected to have the answer so they report that it does not?

Originally the research standards were established as a preventative measure for questions that are too easy to answer with "commonly available resources", but the way it is now, where we needn't necessarily check for ourselves if such is the case may also inadvertently bar questions which are too hard, except I do not even think that was ever the intention of it. That does not mean that there are not some questions which are too hard to answer, but we have separate guidelines for those which are handled by different closure reasons specifically designed to ensure that we only close questions that are too hard to answer when answering them is not productive, which are Too Broad, Primarily Opinion Based, and Unclear What You Are Asking.

Now it could be argued that those questions are P.O.B. for lack of reference, but I think the guidelines for P.O.B. are more sensitive and fine tuned than that, since other forms of corroboration can be used in lieu of direct reference. The way the research standard is set up now seems like we may only answer questions if an answer exists in an easily found reference work that somehow fails to explain the matter sufficiently. I think there are many more questions of interest that we should be addressing than that.

Fixing that problem may actually require experienced and privileged users to exercise more responsible discretion with the vote to close options, and actually check to see if a commonly available resource actually even exists more often, rather than just assuming that it does not whenever the questioner fails to provide research pursuant to Jeff Atwood's guidance in Basic Questions Are Not So Basic.

The initial closure of the Emoji/Emoticon comparison question would have been averted if we did that for example, and that's not the only time it has been a thorn in my side, as somebody who has not as of yet even asked a question, and usually tries to write lengthy in-depth answers. Sometimes I think we need to concern ourselves more with answer potential than question quality if we want this to be the best website it can be. (Which is not to say that question quality should be ignored, but again, I doubt that raising standards even higher than they are now will help much with this particular problem.)

There is another half of the proposal, regarding sharing search terms. Unfortunately, irrespective of what our research standards are I do not think that will help us very much, because even if we know all of that, it might not help us very much. The problem is that they are inconsistent.

One way they can be inconsistent is from search engine to search engine, since they are all competing to deliver better results than the rest.

Even on the same search engine, if they are personalized according to location or search history. Google, which is the most popular search engine right now, does personalize search results somewhat. The C.N.B.C. tech article We sat in on an internal Google meeting where they talked about changing the search algorithm—here’s what we learned, written by Jillian D’Onfro and published on September 17th 2018 states the following:

Other times, ideas for algorithm changes come from broad company directives or priorities. For example, some employees have long argued that Google search results should be more personalized, Nayak said. Right now, there is very little search personalization and what exists is focused on a user’s location or immediate context from a prior search. (If you Googled something related to baseball followed by “The Giants,” the results wouldn’t surface the football team, for example.)

Even if we take this statement at face value it has significant consequences. For the purposes of verifying research, it is problematic enough that somebody in the United States of America might not get the same results as somebody in the United Kingdom for dialectical questions. It also seems possible that we might get different results for people who study punctuation more as opposed to people who study syntax. Ellipsis is often used as a syntactic term which refers to the practice of dropping words out of sentences, but then there is the ellipsis as a punctuation mark that represents the omission of words from quotation.

Even if we take steps to reduce or eliminate the amount of personalization in our search engine by changing Google's user settings or using Duckduckgo instead, the results are also transient. They change with time as search engine optimization improves and websites pop in or out of existence. Over time, some questions which are likely to be general reference today might not be general reference a year from now, or questions which need answering today might be general reference a year from now. In the latter case, I believe it is better if we get to the question first: General reference often offers sparse explanation and rests on the authority of the source, and we can give questions much more thorough treatment and share our evidence if we get to them first, so every time we are beaten to the punch represents a wasted opportunity to optimize the utility of our answers not only through quality, but through earn greater exposure and offer better timeliness than more slowly updated references.

Another thing we should keep in mind is that one of the founding principles of Stack Exchange is to make the internet a better place by making it easier to find information through search engines, so we should probably keep an eye out for when we can serve that purpose.

Since search engine results can not reliably communicate to us what the questioner did or did not see, they don't prove to us that the questioner couldn't find anything useful. Our audit of those search terms also won't be able to prove that they could have easily found something useful, or what they are likely to have seen. I will admit that it is perhaps somewhat more useful than nothing at all, but again, I fail to see much practical benefit because it does not offer much more information for achieving our practical goal.

The only thing it really does is give us a starting point for an audit, but if we are going to go through the effort of doing an audit, we can probably do nearly as well through some cursory guesswork, and decide how to handle the question from there.

| |
  • 2
    You seem to have missed the first bold point in the linked guidance: "if you researched and didn't find anything, at least show where you looked." That provides the opportunity to show absence of evidence. or possibly an inability or misunderstanding of how to search or what to search for. – Andrew Leach May 20 '19 at 21:28
  • 1
    @AndrewLeach Not entirely, but I suppose I forgot to mention that under present guidelines citing a search engine isn't usually considered enough for a number of reasons, such as not being primary resources, inconsistant user experience and transiency of results, so somebody who does a search and finds no relevant website seemingly has no recourse against a research required closure, even if there's nothing to be found that way. Filling out those results to make information easier to find is part of S.E's. purpose, so I think we need to solve that problem. – Tonepoet May 20 '19 at 22:04
  • 5
    Argh - I agree with a lot of what you wrote here, but it's so long that I'm not sure what your actual answer is. Could you add a Tl;Dr? Or is this a discussion piece without any specific conclusion/recommendation on a course of action? – AndyT May 21 '19 at 10:51
  • @AndyT The way things are right now is that almost any amount of research suffices to keep a question open, which is a clear enough threshold for users to comply with if they wanted to do so, and raising the standards in the way you suggest will make things harder on them for little to no practical benefit to us and only lead to more closed questions that won't be reopened, and that we should consider keeping questions of the sort you described open, even if they aren't researched, because in this case they pose no practical problem and answering them would be a net positive to everybody. – Tonepoet May 21 '19 at 11:52
  • 3
    I think I agree with what you said, but the answer is so long, I can't be sure. I certainly agree that requiring three references is pointless overkill if the first reference is, e.g., Merriam-Webster. I upvoted your answer, but hope you will edit in a summary at the beginning. – ab2 May 21 '19 at 19:46
  • @ab2 Well, since seemingly everybody wants me to do so, I'll try to see what I can do about it, but it'll take time for me to think about how to do it, and I can't promise any results. I went over quite a lot of information I didn't wish to gloss over, because our research standards are such a complicated and sensitive issue. – Tonepoet May 21 '19 at 22:17
  • @Ab2 I tried. Hopefully that clears things up for you. Tonepoet targets Mimic Daravon on self to go to sleep. – Tonepoet May 22 '19 at 19:05
  • Can we have a TL;DR for the first point "I concur that we have a problem dismissing questions that might be useful to address in accordance to Stack Exchange's founding purpose of disseminating obscure information, but I think the solution involves us using our discretion and expertise to determine which questions need leniency to best facilitate that goal." I can do that for you if you like :P – Mari-Lou A May 23 '19 at 9:37
  • @Mari-LouA I like the way I originally wrote it, but I tried shortening it a little. I'd say "our goal" instead of "stack exchange's" goal, but I'm not sure as to if the E.L.U. community has the same goals as S.E., so the disambiguation is necessary. Otherwise, if you believe the sentiment and know how to better iterate it, go ahead and re-write it. If I think how you wrote it misrepresents my sentiments, then I can always just change it back to how it was originally. I thank you for your proofreading by the way. – Tonepoet May 23 '19 at 9:59
  • How about 1. I largely agree with the OP's concerns but questions that cannot be resolved by Googling 5-10 minutes deserve leniency. – Mari-Lou A May 23 '19 at 10:01
  • @Mari-LouA I do not think that works very well. I doubt anybody cares what I think, and even if they did it is not just my opinion for one thing: Joel made official statements regarding how he expects the website to work. It may also trivialize the concern to quantify the research effort it in such terms without proving how much research is reasonable to expect before consultation is the better option, and I do not even know for sure how much time that is anyway. It also implicitly emphasizes individual effort over typical propensity to find, when the latter might be more important. – Tonepoet May 23 '19 at 10:37
  • In other words you don't like my paraphrasing. That's fine, I didn't expect you would. But you are verbose, which is a pity because in that thicket of words you build, you say some valid stuff. – Mari-Lou A May 23 '19 at 10:46
  • @Mari-LouA To say I disliked it is not entirely accurate. I suppose that I do dislike the word deserve in this case though, as an afterthought. It is a very effective word when used appropriately, but also very detrimental to a cause when misused even if just slightly. I appreciate your feedback and hope this is a satisfactory compromise. – Tonepoet May 23 '19 at 12:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .