{"action":"opened","issue":{"url":"https://api.github.com/repos/GenweiWu/Blog/issues/30","repository_url":"https://api.github.com/repos/GenweiWu/Blog","labels_url":"https://api.github.com/repos/GenweiWu/Blog/issues/30/labels{/name}","comments_url":"https://api.github.com/repos/GenweiWu/Blog/issues/30/comments","events_url":"https://api.github.com/repos/GenweiWu/Blog/issues/30/events","html_url":"https://github.com/GenweiWu/Blog/issues/30","id":310681952,"number":30,"title":"elastic search智能搜索","user":{"login":"GenweiWu","id":16630659,"avatar_url":"https://avatars2.githubusercontent.com/u/16630659?v=4","gravatar_id":"","url":"https://api.github.com/users/GenweiWu","html_url":"https://github.com/GenweiWu","followers_url":"https://api.github.com/users/GenweiWu/followers","following_url":"https://api.github.com/users/GenweiWu/following{/other_user}","gists_url":"https://api.github.com/users/GenweiWu/gists{/gist_id}","starred_url":"https://api.github.com/users/GenweiWu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GenweiWu/subscriptions","organizations_url":"https://api.github.com/users/GenweiWu/orgs","repos_url":"https://api.github.com/users/GenweiWu/repos","events_url":"https://api.github.com/users/GenweiWu/events{/privacy}","received_events_url":"https://api.github.com/users/GenweiWu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-03T03:30:13Z","updated_at":"2018-04-03T03:30:13Z","closed_at":null,"author_association":"OWNER","body":"[参照的教程](https://github.com/o19s/elasticsearch-learning-to-rank/blob/master/demo/README.md)\r\n==\r\n\r\n# Learning to Rank Demo\r\n\r\nThis demo uses data from [TheMovieDB](http://themoviedb.org) (TMDB) to demonstrate using [Ranklib](https://sourceforge.net/p/lemur/wiki/RankLib/) learning to rank models with Elasticsearch.\r\n\r\n# Install Dependencies and prep data...\r\n\r\nThis demo requires\r\n\r\n- Python 3+\r\n- Python `elasticsearch` and `requests` libraries\r\n- Python `elasticsearch_xpack` libraries if xpack support is necessary\r\n\r\n## An aside: X Pack\r\n\r\nUsing the LTR plugin with xpack requires configuring appropriate roles. These\r\ncan be setup automatically by `prepare_xpack.py` which takes a username and\r\nwill prompt for a password.  After this is run `settings.cfg` must be edited to\r\nuncomment the ESUser and ESPassword properties.\r\n\r\n```\r\npython prepare_xpack.py <xpack admin username>\r\n```\r\n\r\n## Download the TMDB Data & Ranklib Jar\r\n\r\nThe first time you run this demo, fetch RankLib.jar (used to train model) and tmdb.json (the dataset used)\r\n\r\n```\r\npython prepare.py\r\n```\r\n\r\n## Start Elasticsearch/install plugin\r\n\r\nStart a supported version of Elasticsearch and follow the [instructions to install](https://github.com/o19s/elasticsearch-learning-to-rank#installing) the learning to rank plugin.\r\n\r\n## Index to Elasticsearch\r\n\r\nThis script will create a 'tmdb' index with default/simple mappings. You can edit this file to play with mappings.\r\n\r\n```\r\npython indexMlTmdb.py\r\n```\r\n\r\n# Onto the machine learning...\r\n\r\n## TLDR\r\n\r\nIf you're actually going to build a learning to rank system, read past this section. But to sum up, the full Movie demo can be run by\r\n\r\n```\r\npython train.py\r\n```\r\n\r\nThen you can search using\r\n\r\n```\r\npython search.py Rambo\r\n```\r\n\r\nand search results can be printed to the console.\r\n\r\nMore on how all this actually works below:\r\n\r\n## Create and upload features (loadFeatures.py)\r\n\r\nA \"feature\" in ES LTR corresponds to an Elasticsearch query. The score yielded by the query is used to train and evaluate the model. For example, if you feel that a TF\\*IDF title score corresponds to higher relevance, then that's a feature you'd want to train on! Other features might include how old a movie is, the number of keywords in a query, or whatever else you suspect might correlate to your user's sense of relevance.\r\n\r\nIf you examine [loadFeatures.py](loadFeatures.py) you'll see how we create features. We first initialize the default feature store (`PUT /_ltr`). We create a feature set (`POST /_ltr/_featureset/movie_features`). Now we have a place to create features for both logging & use by our models!\r\n\r\nIn the demo features 1...n json are mustache templates that correspond to the features. In this case, the features are identified by *ordinal* (feature 1 is in 1.json). They are uploaded to Elasticsearch Learning to Rank with these ordinals as the feature name. In `eachFeature`, you'll see a loop where we access each mustache template an the file system and return a JSON body for adding the feature to Elasticsearch.\r\n\r\nFor traditional Ranklib models, the ordinal is the only way features are identified. Other models use feature *names* which make developing, logging, and managing features more maintainable.\r\n\r\n## Gather Judgments (sample_judgments.txt)\r\n\r\nThe first part of the training data is the *judgment list*. We've provided one in [sample_judgments.txt](sample_judgments.txt). \r\n\r\nWhat's a judgment list? A judgment list tells us how relevant a document is for a search query. In other words, a three-tuple of \r\n\r\n```\r\n<grade>,<docId>,<keywords>\r\n```\r\n\r\nQuality comes in the form of *grades*. For example if movie \"First Blood\" is considered extremely relevant for the query Rambo, we give it a grade of 4 ('exactly relevant'). The movie Bambi would receive a '0'. Instead of the notional CSV format above, Ranklib and other learning to rank systems use a format from LibSVM, shown below:\r\n\r\n```\r\n# qid:1: rambo\r\n#\r\n#\r\n# grade (0-4)\tqueryid\t # docId\ttitle\r\n4\tqid:1 #\t7555\tRambo\r\n```\r\n\r\nYou'll notice we bastardize this syntax to add comments identifying the keywords associated with each query id, and append metadata to each line. Code provided in [judgments.py](judgments.py) handles this syntax.\r\n\r\n## Log features (collectFeatures.py)\r\n\r\nYou saw above how we created features, the next step is to log features for each judgment 3-tuple. This code is in [collectFeatures.py](collectFeatures.py). Logging features can be done in several different contexts. Of course, in a production system, you may wish to log features as users search. In other contexts, you may have a hand-created judgment list (as we do) and wish to simply ask Elasticsearch Learning to Rank for feature values for query/document pairs.\r\n\r\nIs [collectFeatures.py](collectFeatures.py), you'll see an `sltr` query is included. This query points to a featureSet, not a model. So it does not influence the score. We filter down to needed document ids for each keyword and allow this `sltr` query to run.\r\n\r\nYou'll also notice an `ext` component in the request. This search extension is part of the Elasticsearch Learning to Rank plugin and allows you to configure feature logging. You'll noticed it refers to the query name of `sltr`, allowing it to pluck out the `sltr` query and perform logging associated with the feature set.\r\n\r\nOnce features are gathered, the judgment list is fleshed out with feature value, the ordinals below corresponding to the features in our 1..n.json files.\r\n\r\n```\r\n4\tqid:1\t1:12.318446\t2:9.8376875 # 7555\trambo\r\n```\r\n\r\n## Train (train.py and RankLib.jar)\r\n\r\nWith training data in place, it's time to ask RankLib to train a model, and output to a test file. RankLib supports linear models, ListNet, and several tree-based models such as LambdaMART. In [train.py](train.py) you'll notice how RankLib is called with command line arguments. Models `test_N` are created in our feature store for each type of RankLib model. In the `saveModel` function, you can see how the model is uploaded to our \"movie_features\" feature set.\r\n\r\n## Search using the model (search.py)\r\n\r\nSee what sort of search results you get! In `search.py` you'll see we execute the `sltr` query referring to a `test_N` model in the rescore phase. By default `test_6` is used (corresponding to LambdaMART), but you can change the used model at the command line.\r\n\r\nSearch with default LambdaMART:\r\n\r\n```\r\npython search.py rambo\r\n```\r\n\r\nTry a different model:\r\n\r\n```\r\npython search.py rambo test_8\r\n```\r\n"}}
{"actor":{"display_login":"GenweiWu"}}