Beyond the basics with Elasticsearch
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Král, Honza
Honza Král - Beyond the basics with Elasticsearch Elasticsearch has many use cases, some of them fairly obvious and widely used, like plain searching through documents or analytics. In this talk I would like to go through some of the more advanced scenarios we have seen in the wild. Some examples of what we will cover: Trend detection - how you can use the aggregation framework to go beyond simple "counting" and make use of the full-text properties of Elasticsearch. Percolator - percolator is reversed search and many people use it as such to drive alerts or "stored search" functionality for their website, let's look at how we can use it to detect languages, geo locations or drive live search. If we end up with some time to spare we can explore some other ideas about how we can utilize the features of a search engine to drive non- trivial data analysis including Geo-enabled search with relevancy.
 So, hello everyone and good morning. So I'm here to talk about what's beyond the basics with Elasticsearch. I work for Elastic, the company behind it, so we've seen a lot of use cases and some of them actually surprised us and definitely surprised many people that are familiar with Elasticsearch as sort of the full-text search solution. But before we get beyond the basics, we first need to know what the basics are. So super quickly, for us, that's where we come from. We are a search product. It's an open-source search product. And search is not a new thing. Like it's been around for a while, for a long while. And the basic theory, the really down-to-earth basics haven't changed that much since those times. We still use the same data structures. We still use the same data structure that you find in any book at the end, the index, especially the inverted index, which looks something like this. It looks the same in a book as it does in a computer. It is a list of words that actually exist somewhere in our dataset. Notice that they're sorted. And for each of these words, we have, again, sorted, the list of documents or files or pages when it's a book, where actually these words exist. And we have some additional information stored there, too. For example, how many files does actually contain the word Python? Or how many times is it present in file one? And on what positions? And stuff like that. Those information, those statistics will be very important for us as we go on through the talks. So this is the data structure that we use. So how does search work, then? Well, it's super simple. If we're looking for Python and Django, it's the same search that you would do if you were looking for those things in a book. You locate the line mentioning Django and the line mentioning Python. You can do that effectively, both as a computer and as a person, because, again, it's sorted. And then you just walk the list, and if you find a file or a document that is actually present in both lists, that's your result. Naturally, if you want to do an OR search instead of AND, you just take everything from both lists. And that's not enough, because this will give you the information what matches, but it doesn't give you the most important thing for us. And that is the information how well does it match. What is the difference between the Django book that talks specifically about Python and Django? And the biography of Django Reinhardt when it mentions in one passage that he had an encounter with Python, the snake. Obviously there is a big difference between those two books. And the difference is in relevancy. It is a numerical value, a score, essentially, saying you how well does a given document match a given query? And a lot of research has gone into how best to calculate the score. And again, it hasn't changed that much since the beginning. At the core of it, there is still the TFIDF formula. Those are fancy words, fancy shortcuts. It's a term frequency and inverse document frequency. It essentially represents how rare a word we are looking for, and how many times have we found it in the document. So this essentially represents that if you found the word the in a document, that doesn't really mean much. Like every document in the world, if we are talking in English, will have the word the. That's not a good information. Because the IDF, the inverse document frequency, that's the part. It will tell you that this is not a specific word. It's almost in every document. If you, however, find the word framework or something like that, that is fairly specific. So that's the IDF part. And the TF part is just how many times did you find it there? If it's only mentioned once in a book, doesn't mean much. But if it's there 100 times, that probably means more. And we can keep building on top of that. So for Lucine, for example, adds another factor to it, which is a normalization for the length of the field. That's essentially the equivalent of saying that, yeah, there is a fish somewhere in the ocean. Probably true. Not really that relevant or surprising. But if you have a bucket with water and you say there is a fish in it, that is much more actionable information. So that's the second part of it, the normalization for the field length. If you find something in a super big field, okay, if you find it in a much shorter field, for example, title compared to body, that probably means much more. So already we have a formula that's baked into Lucine, it's baked into Elasticsearch that does very well for text and for search. But sometimes even that is not enough. For example, you're not dealing with text, but with numerical information. Or you have some additional information that Elasticsearch is not aware of. For example, you have the quality of a document. You have some user-contributed value, or even somebody paid you to promote this piece of content or something. Or you want to penalize or favor things based on a distance, let's say, from a geolocation or distance from some numerical range. So how do you do that? We have a few ways of expressing that. And the best way to show it is on an example. So this is a standard query for Elasticsearch. And it's using the function score query type. The function score query type takes the regular query. So normally we are looking for a hotel and we are looking for a hotel that's called the Grand Hotel. So far so good. And then we want that hotel to have a balcony. We want our balcony in our room. But we don't want to just filter just the hotels that have balconies. Because then we would be robbing ourselves of the opportunity to discover something else. But if a hotel has a balcony, we want to favorize it. We will just add two to the score. So all the hotels with balconies will be towards the top. Then we want the hotel to be in central London within one kilometer of the center. If it's within one kilometer, it's a perfect match. The further away from it that it gets, the score decreases. It will still match. But the score will be smaller. Again that means that the hotel that perfectly matches our criteria will be at the top. But if we have a super good match outside, it will still show up. And then we also have the popularity. How have people been happy with the hotel? And let's take that into account. So we have a special thing called field value factor, which is essentially just telling us at Search. There is a numerical value in there that determines the quality. Put it into the score. And finally, we add some random numbers. And this is actually taken from a real life example, because people use this to mix things up a little bit, to give users a chance to discover something new, something they wouldn't otherwise see. So all of these things together will make sure that you find your perfect hotel. We're not limiting your choices. We are not just because you say that you want a balcony. We will still show you the hotel that is almost perfect for you, except for the balcony part. We are also not just sorting by popularity, so that's something that's really not that good a match, but is really popular would be at the top. We're just taking all these factors and combining them together. So this is one of the main ways what we can do with the score and how we can use it in a more advanced way. Just take all the factors that go into the perfect result and just combine them. You're not limited to just picking up one and sorting by it. You can combine them all together, and then it's just a matter of figuring out what these numbers are supposed to be to one and what will actually give your application the best results. Some people actually use machine learning techniques to figure out the best ones. They have a training set and everything. It's not that hard because you have only limited number of options, and typically those are fed numerical values. So if you know what a good match would be, you can actually train the perfect query for you. So this is if you're doing search when you already know what you're looking for. But sometimes it's the other way around. Sometimes it is you don't have the document, but you have the query and you want to find the document. So imagine that you want to do something like alerting or classification. For example, you're indexing documents, you're indexing stock prices, and you want to be alerted whenever a stock price rises above a certain value. Sure you could keep running a query in a continuous loop and see if there is something new. But what we can do instead with the percolator feature of Elastic Surge is to actually index that query into Elastic Surge. And then we just show it a document, and it will tell us all the queries that matched. And that is very powerful, especially because it can use all the features of Elastic Surge. So that's the alerting use case, sort of the stored search functionality. If you supply your users with a search functionality and you want them to be able to store the search and then be alerted whenever there is a new piece of content that actually matches their search. With percolator you get it essentially for free. You just index their query and whenever there is a new piece of content you just run it by the percolator and it will tell you, hey, you should probably send an email to that user that was here the other day. He was really interested in that. That's the sort of stored search. You can also use it to do a live search. So if you've ever been on a website, you did some searching, you were looking through the results and suddenly there was a pop-up that there are five new documents that match your query since you've been looking at it. Again, easy. Once you execute a query, you also store it as a percolator. And then whenever there is a new piece of content during that time, you can just push it to the browser to say, hey, there are new results, more recent. So again, something that's otherwise fairly hard to do or would require some busy loop or something and you can do it this way. But we'll go a little bit further than that. We'll look at the classification use case. That is essentially if you use the percolation to enrich the data in your document. So imagine that you're trying to index events and all you have as location goes is a search set of coordinates and you want to find the address. This is something that's easy to do the other way around. If you have the address and you want to find all the events in that location, you just do a geoshape filter that you're looking something that falls within this shape, within a shape of the city of Warsaw. And that's a super simple search. Well with percolator, we can make it into a super simple reverse search. Let's say we get our hands on a data set with all the cities in Europe or in the world. It's not so much. We index the cities into an index so we don't have to construct the polygon every single time. We store it in the index called shapes under the type city. And then we create a query for each city. We register it with a name and then when a document comes along and its coordinates, the field location fall within that shape, we will know that it is actually happening in Warsaw, Poland. So something that is super simple to do one way but difficult to do the other. We can do with a percolation just essentially using brute force but in a smart way and outsourcing the brute force to elastic search. We can do it very effectively and in a distributed fashion. So that's geoclassification. Another thing that's easy to search for but not that easy to do the other way around usually is language classification. Generally any language has a few words that are super specific to that language. They don't exist in any other. These are some of the examples. This is essentially just a test how many Polish people there in the audience. And the assumption here is that if we look for these specific words and we find at least four, because four is always a good number, because 42 would be too high, then the assumption is that this is actually a document that contains Polish language. And sure, it's a simplification, it's a heuristic but it actually works fairly well. It just depends on the quality of your words. And are super good for Polish that is. So again, and if you have a set of words for each language, you can just start a collection of queries like this. And then when a document comes along with a description of an event, with a geolocation and a description, you can immediately get back the classifiers. You can get back the location in actually a human readable format that it's actually Warsaw, note that it's 473 minus 74.1, which is by the way not Warsaw, but whatever. You also get the language back that it's in Polish. You can use a similar classifier to determine the topic. Like if you have within keywords something like programming and Python and Django, it's fairly accurate assessment to say that the conference probably is something about Python. So this is how we can use percolation to enrich our data and sort of to determine something that otherwise would be hard to do. Another use case for this is imagine that you have a blog, a CMS, and you have a category defined as a search. That's super easy to do one way, but then if you have a blog post and you want to see in which categories this blog post is, that's the harder part. Again with percolation with something like this, it's super easy to do when you can actually tag the blog post with the categories as they come in. Then you can do obviously a little bit more with a percolator. You can attach metadata to the percolators and you can filter on the metadata. You can aggregate them. So as the response, you will not only get the percolators that matched, but also let's say their distribution across categories. You can even use them to highlight something. So you can search for some words in your documents and then just highlight the fragments that actually contain those and store them separately in the document for easy presentation, et cetera, et cetera. You can get the top 10 hottest categories for this piece of content or something like that. But those are if we're working with individual documents. We can also look at more documents at the same time. So this is the traditional search interface. You're just looking something and you get back the top 10 links. What we also have here is something that's called faceted search. This part, the search part is really good when you know what you're looking for. This part shows you what is actually in your data. So you can immediately see the distribution. You can see that if you're looking for something related to Django, the most results are in Python and some in JavaScript. So it allows you to discover data. Some people have taken it even further and we have allowed that with aggregations with multi-dimensional aggregations that you can aggregate over multiple dimensions at the same time. But that is still boring. That is still just counting things. And that's not really interesting. Any database can do that. What we need is we need to use the data that we have, the statistics. So to do that, let's look how we would do recommendations using Elasticsearch. This is our data set. We have a document for users and then for each user, we have a list of artists, of musicians that they like. And we want to do recommendation. Something that I like these things, like what should I listen to next? So we have two, in this case, we have two users, they have artists being common and there are three other artists. So the naive way to do it is to just aggregate. Just ask for the most common thing that they have in common. So give me all the users that like the same things that I do and then give me the most popular artists in that group without the ones that I already know. That way I will get the most popular artists but not necessarily the relevant. It's like asking you, like, what is the most common website that you go to? Probably Google. Not interesting. Because everybody goes to Google. But if I ask the people in this room and I think about it, what is the more specific part for this group compared to if I asked somewhere on the street, it will be something like GitHub. You probably all go to GitHub. Nobody in the outside world goes there. Nobody even knows that it exists. That is relevant. That would be a good recommendation. And we can do that with Elasticsearch. We have all the information. We have the statistics about how rare a word that is and what is the distribution across the populace. So we can ask for the significant terms. It will use all the score, compare it to the background, and then the results will look something similar. This part is important. Because what I would expect is all those dots to be on the diagonal line, because that's what would happen if I had a random sample. It moves away from the central line. The more specific it is. And that is how we can do relevant recommendations. Because we see that this dot here, it is obviously much more common in this group than in the general populace that would be here. So it has moved greatly. And because we have all the information, because we have analyzed the data, because we are the search people, we understand the text, we understand the frequencies, and we can use it, we can actually produce something like that. There are obviously some caveats. For example, if I like a very popular band, like One Direction, then it will skew my results, because everybody likes One Direction, right? So I need a way to combat this, because otherwise I would just get completely irrelevant recommendations. And again, we are the search guys. We understand data. We understand documents. So we can find and sample just the users that are most similar to me. And we have all the tools already at our disposal. Remember TF, IDF, normalization and everything. TF, the people who like the more things that I like, the better they match me. IDF, the people who like the rarer things that I like, put them to the top. And then just take 500 of those best results and only drive the recommendations based on that group. It will make it both faster and more relevant. It will allow you to discard all the irrelevant connections that you might find and only focus on the meaningful connections, on the things that are relevant for your group, in this case the group of people who like the same things that you like. It will provide you with a recommendation. So just by applying the concepts that we have learned from search into other things like aggregations and everything, we can get much more out of it. Another example would be if you have Wikipedia articles when the labels and links are the words and you apply the same concept, you get a meaningful connection between different concepts. If you try to do it based on popularity, it would always be linked through something like yes, that person and that person, yeah, they're both people. Okay. Not exciting. But if you apply this principle, you get something more out of it. So if you combine aggregation and relevancy, all the statistics that we can do, that is actually how we as humans look at the world. If I ask you what is the most common website that you go to, you'll probably not say Google because you know that's not interesting. We as humans have been trained from the very beginning to recognize patterns and to spot anomalies at the very same time. And this concept can be used for other things as well. For example, if you use the same principle, the significant terms aggregation, and per time period, so you split your data into time period and you ask what is significant for that period, how do you call that feature? Well, it's a very common feature that we now see. It's what's trending. That's just it because it's more specific. It's not more popular than in any other area, not necessarily. But it is more specific for this one time period, for the current time period, let's say, compared to yesterday, compared to the general background. So again, once you're doing these aggregations, there's again one single caveat that can happen is that you can have too many options, too many buckets, too many things to calculate. And if that happens, so imagine that you're looking for a combination of actors that star together very often. So I'm looking for the top ten actors, and then for each of those, I'm looking for a set of top ten actors that act with them, that they appear together. If I just ran this, what will happen in the background is I will essentially get a matrix of all the actors and all the actors, and it would be huge. It wouldn't fit into memory. It would probably blow up my cluster. Actually, LSE search would probably refuse to run this query because it would say, hey, I would need too much memory. This is just not going to fly. So what you can do is you can just say, just do it breadth first. Just first get the list of the top ten actors and greatly limit the matrix that you will need to calculate. And then go ahead. So it will be a little slower. It will have to run through the data essentially twice, but it will actually finish, and it will still finish in quite a reasonable time. So that's just how to find the common caveat that people get into when they start exploring the aggregations, especially with the multidimensional. So just to wrap things up, because we are approaching the end in questions, the lesson here is that information is power. We have a lot of information about your data. We have all the statistics, all the distributions of the individual words. And if you understand this and if you can map your data to this problem, you can get a lot more out of Elasticsearch than just finding a good hotel in London or the conference events in Warsaw. So that's it for me, and if you have any questions, I'm here to answer them. There's a question in the back. Questions? That's a long question. Thanks. You show the example how to search people like the 500 more like you. Can you do that more like people that have 90% of being like me instead of a fixed number? Because fixed numbers you have to find and tune in. Of course, you can do that by a simple query. Because aggregations are always run on the results of a query. So we can very easily remember the example that I gave with the language classification when I was looking for at least four words. I could do the same. I could say give me only the users that have at least 70% or 90% or nine. I can use both relative and absolute numbers of the same artist that I like and use those as the basis for the aggregation. So yes, absolutely. And it would actually be much simpler. You wouldn't even need the sampler aggregation. Thanks. Any other questions? Is anyone still awake? Okay, I'll take that as is. So a question? A question going once? Going twice? Sold? Are there any performance implications of running, say, hundreds of percolators? Of course. But you can scale way beyond hundreds. I've seen people doing millions of percolations and it still works. It scales very well with the distributed nature of Elasticsearch. Essentially the only resource that the percolation consumes is CPU. So add more CPU, either to a box or add more boxes and it will scale fairly linearly. So and also just the more boxes and more CPU you will have, the faster it will get. You don't need anything else. You don't need much memory. You don't need faster disks. You only need the CPU. So it's very easy and fairly cheap to scale. To give you an idea, I think that if you want to run hundreds and thousands or millions of percolations, you will need like five reasonable boxes or something like that and you will get responses within milliseconds. So it actually does scale very well. Another question? Could you give us some examples of the customers you mentioned that you had like cases that were really impressive for you and you didn't expect those use cases? Could you give us some examples of the use cases from the customers that you mentioned that you didn't expect them? So some of what we didn't expect was the percolator example. There are some people running big clusters of Elasticsearch and they don't store any data in it. They have a cluster of 15, 20 machines without storing any data. That is a weird experience for essentially a data store. So that's definitely one of them. We also always run into these issues where we have a feature. We recommend people to use it and then people listen to our advice and we find out that we might have underestimated the people in the wild. For example, we introduced the idea of index aliases. That you can have an alias for index essentially like a simling or something. So you can sort of decouple the design of your indices from what the application sees. So you could have like an alias per user but all the users can live together in one big index and the alias will just point to that index and a filter. And that works very well unless until we encountered a user that had millions of users and suddenly we had millions of aliases and we didn't thought that that would ever happen. So as with anything else with computer engineering like assumptions, assumptions, assumptions. So we encountered something like that. We had to go back and fix it and rework the aliases. So these are the two most notable examples where we got really surprised by how our users used our product that we really didn't foresee. And it's good because we always learn something new and it allows us to sort of reorient ourselves better to what the users actually need. Okay? Any last questions? So hello. I have a question regarding reverse queries for language classification. So basically elastic search supports the Ngram indices. So could you use those actually for classification of languages? So Ngrams have the problem that they have a very wide spread. So they might give you some correlation with the language but they will definitely not be precise. So just to explain Ngrams essentially if I split a word into all the tuples of letters, for example with things I would have T-H-A, H-A-N, A-N-K and then I would essentially query for these triplets. And it will obviously have a correlation but it will by no way be decisive enough. Especially for something like language classification where you're really interested in the probability. Ngrams are very good for as an addition to something else because of their nature because they always match something. That's why you typically don't want to use them alone. But they're fine if you have some more optimistic methods like exact matching and then the regular like fuzzy matching and everything and then you just throw Ngrams into the mix to sort of boost the signal if it matches and sort of to catch some things if nothing else matches. So I definitely wouldn't use Ngrams for language classification and I typically only use them with a combination of other query types and other analysis process. Make sense? Okay. So I think that we're running out of time. So thank you very much. If you have more questions, I'll be outside.
 So, hello everyone and good morning.  So I'm here to talk about what's beyond the basics with Elasticsearch.  I work for Elastic, the company behind it, so we've seen a lot of use cases and some  of them actually surprised us and definitely surprised many people that are familiar with  Elasticsearch as sort of the full-text search solution.  But before we get beyond the basics, we first need to know what the basics are.  So super quickly, for us, that's where we come from.  We are a search product.  It's an open-source search product.  And search is not a new thing.  Like it's been around for a while, for a long while.  And the basic theory, the really down-to-earth basics haven't changed that much since those  times.  We still use the same data structures.  We still use the same data structure that you find in any book at the end, the index,  especially the inverted index, which looks something like this.  It looks the same in a book as it does in a computer.  It is a list of words that actually exist somewhere in our dataset.  Notice that they're sorted.  And for each of these words, we have, again, sorted, the list of documents or files or  pages when it's a book, where actually these words exist.  And we have some additional information stored there, too.  For example, how many files does actually contain the word Python?  Or how many times is it present in file one?  And on what positions?  And stuff like that.  Those information, those statistics will be very important for us as we go on through  the talks.  So this is the data structure that we use.  So how does search work, then?  Well, it's super simple.  If we're looking for Python and Django, it's the same search that you would do if you were  looking for those things in a book.  You locate the line mentioning Django and the line mentioning Python.  You can do that effectively, both as a computer and as a person, because, again, it's sorted.  And then you just walk the list, and if you find a file or a document that is actually  present in both lists, that's your result.  Naturally, if you want to do an OR search instead of AND, you just take everything from  both lists.  And that's not enough, because this will give you the information what matches, but it doesn't  give you the most important thing for us.  And that is the information how well does it match.  What is the difference between the Django book that talks specifically about Python  and Django?  And the biography of Django Reinhardt when it mentions in one passage that he had an  encounter with Python, the snake.  Obviously there is a big difference between those two books.  And the difference is in relevancy.  It is a numerical value, a score, essentially, saying you how well does a given document  match a given query?  And a lot of research has gone into how best to calculate the score.  And again, it hasn't changed that much since the beginning.  At the core of it, there is still the TFIDF formula.  Those are fancy words, fancy shortcuts.  It's a term frequency and inverse document frequency.  It essentially represents how rare a word we are looking for, and how many times have  we found it in the document.  So this essentially represents that if you found the word the in a document, that doesn't  really mean much.  Like every document in the world, if we are talking in English, will have the word the.  That's not a good information.  Because the IDF, the inverse document frequency, that's the part.  It will tell you that this is not a specific word.  It's almost in every document.  If you, however, find the word framework or something like that, that is fairly specific.  So that's the IDF part.  And the TF part is just how many times did you find it there?  If it's only mentioned once in a book, doesn't mean much.  But if it's there 100 times, that probably means more.  And we can keep building on top of that.  So for Lucine, for example, adds another factor to it, which is a normalization for the length  of the field.  That's essentially the equivalent of saying that, yeah, there is a fish somewhere in the  ocean.  Probably true.  Not really that relevant or surprising.  But if you have a bucket with water and you say there is a fish in it, that is much more  actionable information.  So that's the second part of it, the normalization for the field length.  If you find something in a super big field, okay, if you find it in a much shorter field,  for example, title compared to body, that probably means much more.  So already we have a formula that's baked into Lucine, it's baked into Elasticsearch  that does very well for text and for search.  But sometimes even that is not enough.  For example, you're not dealing with text, but with numerical information.  Or you have some additional information that Elasticsearch is not aware of.  For example, you have the quality of a document.  You have some user-contributed value, or even somebody paid you to promote this piece of  content or something.  Or you want to penalize or favor things based on a distance, let's say, from a geolocation  or distance from some numerical range.  So how do you do that?  We have a few ways of expressing that.  And the best way to show it is on an example.  So this is a standard query for Elasticsearch.  And it's using the function score query type.  The function score query type takes the regular query.  So normally we are looking for a hotel and we are looking for a hotel that's called the  Grand Hotel.  So far so good.  And then we want that hotel to have a balcony.  We want our balcony in our room.  But we don't want to just filter just the hotels that have balconies.  Because then we would be robbing ourselves of the opportunity to discover something else.  But if a hotel has a balcony, we want to favorize it.  We will just add two to the score.  So all the hotels with balconies will be towards the top.  Then we want the hotel to be in central London within one kilometer of the center.  If it's within one kilometer, it's a perfect match.  The further away from it that it gets, the score decreases.  It will still match.  But the score will be smaller.  Again that means that the hotel that perfectly matches our criteria will be at the top.  But if we have a super good match outside, it will still show up.  And then we also have the popularity.  How have people been happy with the hotel?  And let's take that into account.  So we have a special thing called field value factor, which is essentially just telling  us at Search.  There is a numerical value in there that determines the quality.  Put it into the score.  And finally, we add some random numbers.  And this is actually taken from a real life example, because people use this to mix things  up a little bit, to give users a chance to discover something new, something they wouldn't  otherwise see.  So all of these things together will make sure that you find your perfect hotel.  We're not limiting your choices.  We are not just because you say that you want a balcony.  We will still show you the hotel that is almost perfect for you, except for the balcony part.  We are also not just sorting by popularity, so that's something that's really not that  good a match, but is really popular would be at the top.  We're just taking all these factors and combining them together.  So this is one of the main ways what we can do with the score and how we can use it in  a more advanced way.  Just take all the factors that go into the perfect result and just combine them.  You're not limited to just picking up one and sorting by it.  You can combine them all together, and then it's just a matter of figuring out what these  numbers are supposed to be to one and what will actually give your application the best  results.  Some people actually use machine learning techniques to figure out the best ones.  They have a training set and everything.  It's not that hard because you have only limited number of options, and typically those are  fed numerical values.  So if you know what a good match would be, you can actually train the perfect query for  you.  So this is if you're doing search when you already know what you're looking for.  But sometimes it's the other way around.  Sometimes it is you don't have the document, but you have the query and you want to find  the document.  So imagine that you want to do something like alerting or classification.  For example, you're indexing documents, you're indexing stock prices, and you want to be  alerted whenever a stock price rises above a certain value.  Sure you could keep running a query in a continuous loop and see if there is something new.  But what we can do instead with the percolator feature of Elastic Surge is to actually index  that query into Elastic Surge.  And then we just show it a document, and it will tell us all the queries that matched.  And that is very powerful, especially because it can use all the features of Elastic Surge.  So that's the alerting use case, sort of the stored search functionality.  If you supply your users with a search functionality and you want them to be able to store the  search and then be alerted whenever there is a new piece of content that actually matches  their search.  With percolator you get it essentially for free.  You just index their query and whenever there is a new piece of content you just run it  by the percolator and it will tell you, hey, you should probably send an email to that user  that was here the other day.  He was really interested in that.  That's the sort of stored search.  You can also use it to do a live search.  So if you've ever been on a website, you did some searching, you were looking through the  results and suddenly there was a pop-up that there are five new documents that match your  query since you've been looking at it.  Again, easy.  Once you execute a query, you also store it as a percolator.  And then whenever there is a new piece of content during that time, you can just push  it to the browser to say, hey, there are new results, more recent.  So again, something that's otherwise fairly hard to do or would require some busy loop  or something and you can do it this way.  But we'll go a little bit further than that.  We'll look at the classification use case.  That is essentially if you use the percolation to enrich the data in your document.  So imagine that you're trying to index events and all you have as location goes is a search  set of coordinates and you want to find the address.  This is something that's easy to do the other way around.  If you have the address and you want to find all the events in that location, you just  do a geoshape filter that you're looking something that falls within this shape, within a shape  of the city of Warsaw.  And that's a super simple search.  Well with percolator, we can make it into a super simple reverse search.  Let's say we get our hands on a data set with all the cities in Europe or in the world.  It's not so much.  We index the cities into an index so we don't have to construct the polygon every single  time.  We store it in the index called shapes under the type city.  And then we create a query for each city.  We register it with a name and then when a document comes along and its coordinates, the  field location fall within that shape, we will know that it is actually happening in  Warsaw, Poland.  So something that is super simple to do one way but difficult to do the other.  We can do with a percolation just essentially using brute force but in a smart way and outsourcing  the brute force to elastic search.  We can do it very effectively and in a distributed fashion.  So that's geoclassification.  Another thing that's easy to search for but not that easy to do the other way around usually  is language classification.  Generally any language has a few words that are super specific to that language.  They don't exist in any other.  These are some of the examples.  This is essentially just a test how many Polish people there in the audience.  And the assumption here is that if we look for these specific words and we find at least  four, because four is always a good number, because 42 would be too high, then the assumption  is that this is actually a document that contains Polish language.  And sure, it's a simplification, it's a heuristic but it actually works fairly well.  It just depends on the quality of your words.  And are super good for Polish that is.  So again, and if you have a set of words for each language, you can just start a collection  of queries like this.  And then when a document comes along with a description of an event, with a geolocation  and a description, you can immediately get back the classifiers.  You can get back the location in actually a human readable format that it's actually  Warsaw, note that it's 473 minus 74.1, which is by the way not Warsaw, but whatever.  You also get the language back that it's in Polish.  You can use a similar classifier to determine the topic.  Like if you have within keywords something like programming and Python and Django, it's  fairly accurate assessment to say that the conference probably is something about Python.  So this is how we can use percolation to enrich our data and sort of to determine something  that otherwise would be hard to do.  Another use case for this is imagine that you have a blog, a CMS, and you have a category  defined as a search.  That's super easy to do one way, but then if you have a blog post and you want to see  in which categories this blog post is, that's the harder part.  Again with percolation with something like this, it's super easy to do when you can actually  tag the blog post with the categories as they come in.  Then you can do obviously a little bit more with a percolator.  You can attach metadata to the percolators and you can filter on the metadata.  You can aggregate them.  So as the response, you will not only get the percolators that matched, but also let's  say their distribution across categories.  You can even use them to highlight something.  So you can search for some words in your documents and then just highlight the fragments that  actually contain those and store them separately in the document for easy presentation, et  cetera, et cetera.  You can get the top 10 hottest categories for this piece of content or something like that.  But those are if we're working with individual documents.  We can also look at more documents at the same time.  So this is the traditional search interface.  You're just looking something and you get back the top 10 links.  What we also have here is something that's called faceted search.  This part, the search part is really good when you know what you're looking for.  This part shows you what is actually in your data.  So you can immediately see the distribution.  You can see that if you're looking for something related to Django, the most results are in  Python and some in JavaScript.  So it allows you to discover data.  Some people have taken it even further and we have allowed that with aggregations with  multi-dimensional aggregations that you can aggregate over multiple dimensions at the  same time.  But that is still boring.  That is still just counting things.  And that's not really interesting.  Any database can do that.  What we need is we need to use the data that we have, the statistics.  So to do that, let's look how we would do recommendations using Elasticsearch.  This is our data set.  We have a document for users and then for each user, we have a list of artists, of musicians  that they like.  And we want to do recommendation.  Something that I like these things, like what should I listen to next?  So we have two, in this case, we have two users, they have artists being common and  there are three other artists.  So the naive way to do it is to just aggregate.  Just ask for the most common thing that they have in common.  So give me all the users that like the same things that I do and then give me the most  popular artists in that group without the ones that I already know.  That way I will get the most popular artists but not necessarily the relevant.  It's like asking you, like, what is the most common website that you go to?  Probably Google.  Not interesting.  Because everybody goes to Google.  But if I ask the people in this room and I think about it, what is the more specific  part for this group compared to if I asked somewhere on the street, it will be something  like GitHub.  You probably all go to GitHub.  Nobody in the outside world goes there.  Nobody even knows that it exists.  That is relevant.  That would be a good recommendation.  And we can do that with Elasticsearch.  We have all the information.  We have the statistics about how rare a word that is and what is the distribution across  the populace.  So we can ask for the significant terms.  It will use all the score, compare it to the background, and then the results will look  something similar.  This part is important.  Because what I would expect is all those dots to be on the diagonal line, because that's  what would happen if I had a random sample.  It moves away from the central line.  The more specific it is.  And that is how we can do relevant recommendations.  Because we see that this dot here, it is obviously much more common in this group than in the  general populace that would be here.  So it has moved greatly.  And because we have all the information, because we have analyzed the data, because we are the  search people, we understand the text, we understand the frequencies, and we can use  it, we can actually produce something like that.  There are obviously some caveats.  For example, if I like a very popular band, like One Direction, then it will skew my results,  because everybody likes One Direction, right?  So I need a way to combat this, because otherwise I would just get completely irrelevant recommendations.  And again, we are the search guys.  We understand data.  We understand documents.  So we can find and sample just the users that are most similar to me.  And we have all the tools already at our disposal.  Remember TF, IDF, normalization and everything.  TF, the people who like the more things that I like, the better they match me.  IDF, the people who like the rarer things that I like, put them to the top.  And then just take 500 of those best results and only drive the recommendations based on  that group.  It will make it both faster and more relevant.  It will allow you to discard all the irrelevant connections that you might find and only focus  on the meaningful connections, on the things that are relevant for your group, in this  case the group of people who like the same things that you like.  It will provide you with a recommendation.  So just by applying the concepts that we have learned from search into other things like  aggregations and everything, we can get much more out of it.  Another example would be if you have Wikipedia articles when the labels and links are the  words and you apply the same concept, you get a meaningful connection between different  concepts.  If you try to do it based on popularity, it would always be linked through something  like yes, that person and that person, yeah, they're both people.  Okay.  Not exciting.  But if you apply this principle, you get something more out of it.  So if you combine aggregation and relevancy, all the statistics that we can do, that is  actually how we as humans look at the world.  If I ask you what is the most common website that you go to, you'll probably not say Google  because you know that's not interesting.  We as humans have been trained from the very beginning to recognize patterns and to spot  anomalies at the very same time.  And this concept can be used for other things as well.  For example, if you use the same principle, the significant terms aggregation, and per  time period, so you split your data into time period and you ask what is significant for  that period, how do you call that feature?  Well, it's a very common feature that we now see.  It's what's trending.  That's just it because it's more specific.  It's not more popular than in any other area, not necessarily.  But it is more specific for this one time period, for the current time period, let's  say, compared to yesterday, compared to the general background.  So again, once you're doing these aggregations, there's again one single caveat that can happen  is that you can have too many options, too many buckets, too many things to calculate.  And if that happens, so imagine that you're looking for a combination of actors that star  together very often.  So I'm looking for the top ten actors, and then for each of those, I'm looking for a  set of top ten actors that act with them, that they appear together.  If I just ran this, what will happen in the background is I will essentially get a matrix  of all the actors and all the actors, and it would be huge.  It wouldn't fit into memory.  It would probably blow up my cluster.  Actually, LSE search would probably refuse to run this query because it would say, hey,  I would need too much memory.  This is just not going to fly.  So what you can do is you can just say, just do it breadth first.  Just first get the list of the top ten actors and greatly limit the matrix that you will  need to calculate.  And then go ahead.  So it will be a little slower.  It will have to run through the data essentially twice, but it will actually finish, and it  will still finish in quite a reasonable time.  So that's just how to find the common caveat that people get into when they start exploring  the aggregations, especially with the multidimensional.  So just to wrap things up, because we are approaching the end in questions, the lesson  here is that information is power.  We have a lot of information about your data.  We have all the statistics, all the distributions of the individual words.  And if you understand this and if you can map your data to this problem, you can get  a lot more out of Elasticsearch than just finding a good hotel in London or the conference  events in Warsaw.  So that's it for me, and if you have any questions, I'm here to answer them.  There's a question in the back.  Questions?  That's a long question.  Thanks.  You show the example how to search people like the 500 more like you.  Can you do that more like people that have 90% of being like me instead of a fixed number?  Because fixed numbers you have to find and tune in.  Of course, you can do that by a simple query.  Because aggregations are always run on the results of a query.  So we can very easily remember the example that I gave with the language classification  when I was looking for at least four words.  I could do the same.  I could say give me only the users that have at least 70% or 90% or nine.  I can use both relative and absolute numbers of the same artist that I like and use those  as the basis for the aggregation.  So yes, absolutely.  And it would actually be much simpler.  You wouldn't even need the sampler aggregation.  Thanks.  Any other questions?  Is anyone still awake?  Okay, I'll take that as is.  So a question?  A question going once?  Going twice?  Sold?  Are there any performance implications of running, say, hundreds of percolators?  Of course.  But you can scale way beyond hundreds.  I've seen people doing millions of percolations and it still works.  It scales very well with the distributed nature of Elasticsearch.  Essentially the only resource that the percolation consumes is CPU.  So add more CPU, either to a box or add more boxes and it will scale fairly linearly.  So and also just the more boxes and more CPU you will have, the faster it will get.  You don't need anything else.  You don't need much memory.  You don't need faster disks.  You only need the CPU.  So it's very easy and fairly cheap to scale.  To give you an idea, I think that if you want to run hundreds and thousands or millions  of percolations, you will need like five reasonable boxes or something like that and you will  get responses within milliseconds.  So it actually does scale very well.  Another question?  Could you give us some examples of the customers you mentioned that you had like cases that  were really impressive for you and you didn't expect those use cases?  Could you give us some examples of the use cases from the customers that you mentioned  that you didn't expect them?  So some of what we didn't expect was the percolator example.  There are some people running big clusters of Elasticsearch and they don't store any  data in it.  They have a cluster of 15, 20 machines without storing any data.  That is a weird experience for essentially a data store.  So that's definitely one of them.  We also always run into these issues where we have a feature.  We recommend people to use it and then people listen to our advice and we find out that  we might have underestimated the people in the wild.  For example, we introduced the idea of index aliases.  That you can have an alias for index essentially like a simling or something.  So you can sort of decouple the design of your indices from what the application sees.  So you could have like an alias per user but all the users can live together in one big  index and the alias will just point to that index and a filter.  And that works very well unless until we encountered a user that had millions of users and suddenly  we had millions of aliases and we didn't thought that that would ever happen.  So as with anything else with computer engineering like assumptions, assumptions, assumptions.  So we encountered something like that.  We had to go back and fix it and rework the aliases.  So these are the two most notable examples where we got really surprised by how our users  used our product that we really didn't foresee.  And it's good because we always learn something new and it allows us to sort of reorient ourselves  better to what the users actually need.  Okay?  Any last questions?  So hello.  I have a question regarding reverse queries for language classification.  So basically elastic search supports the Ngram indices.  So could you use those actually for classification of languages?  So Ngrams have the problem that they have a very wide spread.  So they might give you some correlation with the language but they will definitely not  be precise.  So just to explain Ngrams essentially if I split a word into all the tuples of letters,  for example with things I would have T-H-A, H-A-N, A-N-K and then I would essentially  query for these triplets.  And it will obviously have a correlation but it will by no way be decisive enough.  Especially for something like language classification where you're really interested in the probability.  Ngrams are very good for as an addition to something else because of their nature because  they always match something.  That's why you typically don't want to use them alone.  But they're fine if you have some more optimistic methods like exact matching and then the regular  like fuzzy matching and everything and then you just throw Ngrams into the mix to sort  of boost the signal if it matches and sort of to catch some things if nothing else matches.  So I definitely wouldn't use Ngrams for language classification and I typically only use them  with a combination of other query types and other analysis process.  Make sense?  Okay.  So I think that we're running out of time.  So thank you very much.  If you have more questions, I'll be outside.
Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_0_7067.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_7068_8207.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_8208_8291.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_8292_9887.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_9888_13895.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_13896_22547.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_22548_26519.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_26520_29447.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_29448_38531.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_38532_39419.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_39420_45095.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_45096_47831.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_47832_47903.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_47904_48215.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_48216_48407.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_48408_48983.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_48984_49175.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_49176_53099.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_53100_53387.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_53388_53891.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_53892_54371.png Honza_Kral_-_Beyond_the_basics_with_Elasticsearch_54372_56009.png