title: How to handle "private" data in Elastic Search
body: <p>I need some input from anyone who might have experience/knowledge about how ElasticSearch works and API's..
  I have a (very large) database with a lot of data for a lot of different items.</p>
  
  <p>I need to make all of this data searchable through a public API, so that anyone can use it and query the API about data for specific items. I already have ElasticSearch up &amp; running, and have populated an index in ElasticSearch with all of the data from the database. ElasticSearch is working fine and so is the API.</p>
  
  <p>The challenge I now face is that some of the data in our database is "private" data which must not be publicly searchable. At the same time this private data must be searchable internally, which means that I need to make the API run in both at public mode and a private mode (user authenticated). When a client that has not been authenticated queries the API for some data the client should only get the public items, whereas the private (user authenticated) client should get all possible results.
  I don't have a problem with the items where all the data for one item must not be publicly available. I can simply mark them with a flag and make sure that when I return data to the client through the API they are not returned by ElasticSearch.</p>
  
  <p>The challenge occurs when there is data for an item where part of the data is private and part of the data is public. I have thought about peeling off the private data before returning the data to the (public) client. This way the private data is not available directly through the API, but it will be indirectly/implicitly. If for instance the client have searched for some data which is of a private nature and in which case I will "strip" the private data from the search result before returning it to the user, then the client will get the document returned, indicating that the document was a "hit" for the specific query. However the specific query string from the client is nowhere to be found in the document that I return, thus indicating that the query string is somehow associated with the document and that the association is of a sensitive/private nature.</p>
  
  <p>I have thought about creating two different indices. One that has all the data for all the objects (the private index) and one that has only the publicly available data (where I have stripped all the documents for the data parts that are of a sensitive nature). This would work and would be a fairly easy solution to implement, but the downside is that I now have duplicated data in two indices.</p>
  
  <p>Any ideas?</p>