A Fox, a Henhouse, and Custodial Self-Collection

Judge Scheindlin just issued an opinion in the Freedom of Information Act (FOIA) case National Day Laborer Organizing Network et al. v. United States Immigration and Customs Enforcement Agency, et al. 2012 U.S. Dist. Lexis 97863 (SDNY, July 13, 2012). This dispute focuses on plaintiffs’ attempts to obtain information from several US government agencies including the Federal Bureau of Investigation, the Immigration and Customs Enforcement Agency,   and the Department of Homeland Security. Specifically, the plaintiffs have sought information regarding “Secure Communities”, a federal immigration enforcement program launched in 2008.

In December 2010, after the defendants failed to comply with their obligations under the agreement, Judge Scheindlin ordered them to produce the records on a new “drop dead date”. With the new date in mind, the defendants’ searched hundreds of employees expending thousands of hours and resulted in the production of tens of thousands of responsive records.

The plaintiffs argued the searches had been insufficient i.e. that the agencies failed to conduct any searches of the files of certain custodians who were likely to possess responsive records. Another complaint was that the defendants had not established that the searches that they did conduct were adequate.

On the issue of relying on custodians to “self-collect” i.e., conduct appropriate and legally defensible searches themselves, she writes:

“There are two answers to defendants’ question. First, custodians cannot ‘be trusted to run effective searches,’ without providing a detailed description of those searches, because FOIA places a burden on defendants to establish that they have conducted adequate searches; FOIA permits agencies to do so by submitting affidavits that ‘contain reasonable specificity of detail rather than merely conclusory statements.’ Defendants’ counsel recognize that, for over twenty years, courts have required that these affidavits ‘set [ ] forth the search terms and the type of search performed.’ But, somehow, DHS, ICE, and the FBI have not gotten the message. So it bears repetition: the government will not be able to establish the adequacy of its FOIA searches if it does not record and report the search terms that it used, how it combined them, and whether it searched the full text of documents.”

“The second answer to defendants’ question has emerged from scholarship and case law only in recent years: most custodians cannot be ‘trusted’ to run effective searches because designing legally sufficient electronic searches in the discovery or FOIA contexts is not part of their daily responsibilities. Searching for an answer on Google (or Westlaw or Lexis) is very different from searching for all responsive documents in the FOIA or e-discovery context.”

“Simple keyword searching is often not enough: ‘Even in the simplest case requiring a search of on-line e-mail, there is no guarantee that using keywords will always prove sufficient.’ There is increasingly strong evidence that ‘[k]eyword search[ing] is not nearly as effective at identifying relevant information as many lawyers would like to believe.’ As Judge Andrew Peck — one of this Court’s experts in e-discovery — recently put it: ‘In too many cases, however, the way lawyers choose keywords is the equivalent of the child’s game of ‘Go Fish’ … keyword searches usually are not very effective.'”

Custodial self-discovery has been falling out of favor with some Judges for several reasons. First, the defense attorney should be overseeing the discovery process to ensure correctness and completeness. In many courts, the attorney has to certify that the discovery process was done correctly… and what attorney wants to do that if they didn’t really manage it?

In a recent Law.com article written by Ralph Losey, Ralph pointed out that custodial self-discovery was “equivalent to the fox guarding the hen house”.


Early Case Assessment and Concept Search

There has been an ongoing argument as to the validity of concept search verses keyword search in discovery searches. The main arguments I have seen are:

  1. Keyword searches tend to miss relevant documents and are under-inclusive in their search results.
  2. Concept searches tend to produce too many non-responsive documents and are considered over-inclusive in their search results.
  3. The other argument against concept searches for eDiscovery is that concept searches are a “black box” and are therefore very hard to explain to the court as to their validity.

I have not been able to find any cases where the eDiscovery response was conducted via a concept search.

While at LegalTech 2010 in New York, I spoke to several conceptual search/clustering vendors that were positioning conceptual search as the next big thing…that keyword search was falling in favor. I don’t believe that to be the case but I am curious whether conceptual search technology has a future.

I do believe there is an interesting possibility to use conceptual search capabilities in the area of Early Case Assessments (ECA). For ECA, the discoveree wants to “data mine” potentially responsive ESI to determine their going forward strategy; should we settle or should we fight? To make the best decision about legal strategy, I believe having access to the most complete and relevant data set is a top priority. One of the wraps against concept search is it is over-inclusive; a benefit in making sure you have reviewed all potentially responsive ESI when performing ECA.