16 min read

In the article Prabhanjan Tattar, author of book Practical Data Science Cookbook – Second Edition, explainsPython is an interpreted language (sometimes referred to as a scripting language), much like R. It requires no special IDE or software compilation tools and is therefore as fast as R to develop with and prototype. Like R, it also makes use of C shared objects to improve computational performance. Additionally, Python is a default system tool on Linux, Unix, and Mac OS X machines and is available on Windows. Python comes with batteries included, which means that the standard library is widely inclusive of many modules, from multiprocessing to compression toolsets. Python is a flexible computing powerhouse that can tackle any problem domain. If you find yourself in need of libraries that are outside of the standard library, Python also comes with a package manager (like R) that allows the download and installation of other code bases.


Python’s computational flexibility means that some analytical tasks take more lines of code than their counterpart in R. However, Python does have the tools that allow it to perform the same statistical computing. This leads to an obvious question: When do we use R over Python and vice versa? This article attempts to answer this question by taking an application-oriented approach to statistical analyses.



Understanding the data


Getting ready

To prepare for this recipe, and the rest of the article, download the MovieLens data from the GroupLens website of the University of Minnesota. You can find the data at http://grouplens.org/datasets/movielens/.


How to do it…

Perform the following steps to better understand the data that we will be working with throughout:

  1. Download the data from http://grouplens.org/datasets/movielens/.The 100K dataset is the one that you want (ml-100k.zip):

  3. The two files that we are mainly concerned with are u.data, which contains the user movie ratings, and u.item, which contains movie information and details. To get a sense of each file, use the head command at the command prompt for Mac and Linux or the more command for Windows:
    head -n 5 u.item


    moreu.item 2 n
  4. The preceding command gives you the following output:
    4|Get Shorty (1995)|01-Jan-1995||http://us.imdb.com/M/title-exact?Get%20Shorty%20(1995)|0|1|0|0|0|1|0|0|1|0|0|0|0|0|0|0|0|0|0
    5|Copycat (1995)|01-Jan-1995||http://us.imdb.com/M/title-exact?Copycat%20(1995)|0|0|0|0|0|0|1|0|1|0|0|0|0|0|0|0|1|0|0
  5. The following command will produce the given output:
    head -n 5 u.data
  6. For Windows, you can use the following command:
    moreu.item 2 n
    196  242  3  881250949
    186  302  3  891717742
    22  377  1  878887116
    244  51  2  880606923
    166  346  1  886397596

How it works…


  • u.data: This contains the user moving ratings
  • u.item: This contains the movie information and other details

Both are character-delimited files; u.data, which is the main file, is tab delimited, and u.item is pipe delimited.


There’s more…

Free, web-scale datasets that are appropriate for building recommendation engines are few and far between. As a result, the movie lens dataset is a very popular choice for such a task but there are others as well. The well-known Netflix Prize dataset has been pulled down by Netflix. However, there is a dump of all user-contributed content from the Stack Exchange network (including Stack Overflow) available via the Internet Archive (https://archive.org/details/stackexchange). Additionally, there is a book-crossing dataset that contains over a million ratings of about a quarter million different books (http://www2.informatik.uni-freiburg.de/~cziegler/BX/).

Ingesting the movie review data


Getting ready

You will need to have the appropriate movie lens dataset downloaded, as specified in the preceding recipe. If you skipped the setup in you will need to go back and ensure that you have NumPy correctly installed.

How to do it…


  1. Open your favorite Python editor or IDE. There is a lot of code, so it should be far simpler to enter directly into a text file than Read-Eval-Print Loop (REPL).
  2. We create a function to import the movie reviews:
    In [1]: import csv
       ...: import datetime
    In [2]: defload_reviews(path, **kwargs):
       ...: “““
       ...: Loads MovieLens reviews
       ...: “““
       ...: options = {
       ...: ‘fieldnames’: (‘userid’, ‘movieid’, ‘rating’, ‘timestamp’),
       ...: ‘delimiter’: ‘t’,
       ...: }
       ...: options.update(kwargs)
       ...: parse_date = lambda r,k: datetime.fromtimestamp(float(r[k]))
       ...: parse_int = lambda r,k: int(r[k])
       ...: with open(path, ‘rb’) as reviews:
       ...: reader = csv.DictReader(reviews, **options)
       ...: for row in reader:
       ...: row[‘movieid’] = parse_int(row, ‘movieid’)
       ...: row[‘userid’] = parse_int(row, ‘userid’)
       ...: row[‘rating’] = parse_int(row, ‘rating’)
       ...: row[‘timestamp’] = parse_date(row, ‘timestamp’)
       ...: yield row
  3. We create a helper function to help import the data:
    In  [3]: import os
        ...: defrelative_path(path):
        ...: “““
        ...: Returns a path relative from this code file
        ...: “““
        ...: dirname = os.path.dirname(os.path.realpath(‘__file__’))
        ...: path = os.path.join(dirname, path)
        ...: return os.path.normpath(path)
  4.   We create another function to load the movie information:
    In  [4]: defload_movies(path, **kwargs):
        ...: options = {
        ...: ‘fieldnames’: (‘movieid’, ‘title’, ‘release’, ‘video’, ‘url’),
        ...: ‘delimiter’: ‘|’,
        ...: ‘restkey’: ‘genre’,
        ...: }
        ...: options.update(kwargs)
        ...: parse_date = lambda r,k: datetime.strptime(r[k], ‘%d-%b-%Y’) if r[k] else None
        ...: with open(path, ‘rb’) as movies:
        ...: reader = csv.DictReader(movies, **options)
        ...: for row in reader:
        ...: row[‘release’] = parse_date(row, ‘release’)
        ...: row[‘video’] = parse_date(row, ‘video’)
        ...: yield row
  5. Finally, we start creating a MovieLens class that will be augmented later :

    In  [5]: from collections import defaultdict
    In  [6]: class MovieLens(object):
        ...: “““
        ...: Data structure to build our recommender model on.
        ...: “““
        ...: def __init__(self, udata, uitem):
        ...: “““
        ...: Instantiate with a path to u.data and u.item
        ...: “““
        ...: self.udata = udata
        ...: self.uitem = uitem
        ...: self.movies = {}
        ...: self.reviews = defaultdict(dict)
        ...: self.load_dataset()
        ...: defload_dataset(self):
        ...: “““
        ...: Loads the two datasets into memory, indexed on the ID.
        ...: “““
        ...: for movie in load_movies(self.uitem):
        ...: self.movies[movie[‘movieid’]] = movie
        ...: for review in load_reviews(self.udata):
        ...: self.reviews[review[‘userid’]][review[‘movieid’]] = review

    In  [7]: data = relative_path(‘../data/ml-100k/u.data’)
        ...: item = relative_path(‘../data/ml-100k/u.item’)
        ...: model = MovieLens(data, item)

How it works…




We’ll use each of these methodologies to load the datasets at various times through our computation that uses this dataset. We’ll need to know where these files are at all times, which can be a pain, especially in larger code bases; in the There’s more… section, we’ll discuss a Python pro-tip to alleviate this concern.

Finally, we created a data structure, which is the MovieLens class, with which we can hold our reviews’ data. This structure takes the udata and uitem paths, and then, it loads the movies and reviews into two Python dictionaries that are indexed by movieid and userid, respectively. To instantiate this object, you will execute something as follows:

    ...: model = MovieLens(data, item)



     Returns a path relative from this code file
dirname = os.path.dirname(os.path.realpath(‘__file__’))
path = os.path.join(dirname, path)

Keep in mind that this holds the entire data structure in memory; in the case of the 100k dataset, this will require 54.1 MB, which isn’t too bad for modern machines. However, we should also keep in mind that we’ll generally build recommenders using far more than just 100,000 reviews. This is why we have configured the data structure the way we have—very similar to a database. To grow the system, you will replace the reviews and movies properties with database access functions or properties, which will yield data types expected by our methods.

Finding the highest-scoring movies

If you’re looking for a good movie, you’ll often want to see the most popular or best rated movies overall. Initially, we’ll take a naïve approach to compute a movie’s aggregate rating by averaging the user reviews for each movie. This technique will also demonstrate how to access the data in our MovieLens class.

Getting ready


How to do it…


    In  [8]: class MovieLens(object):
        ...: defreviews_for_movie(self, movieid):
        ...: “““
        ...: Yields the reviews for a given movie
        ...: “““
        ...: for review in self.reviews.values():
        ...: if movieid in review:
        ...: yield review[movieid]
    In [9]: import heapq
        ...: from operator import itemgetter
        ...: class MovieLens(object):
        ...: defaverage_reviews(self):
        ...: “““
        ...: Averages the star rating for all movies. Yields a tuple of movieid,
        ...: the average rating, and the number of reviews.
        ...: “““
        ...: for movieid in self.movies:
        ...: reviews = list(r[‘rating’] for r in self.reviews_for_movie(movieid))
        ...: average = sum(reviews) / float(len(reviews))
        ...: yield (movieid, average, len(reviews)) 
        ...: deftop_rated(self, n=10):
        ...: “““
        ...: Yields the n top rated movies
        ...: “““
        ...: return heapq.nlargest(n, self.bayesian_average(), key=itemgetter(1))


  3. Now, let’s print the top-rated results:
    In [10]: for mid, avg, num in model.top_rated(10):
        ...: title = model.movies[mid][‘title’]
        ...: print “[%0.3f average rating (%i reviews)] %s” % (avg, num,title)
    Out [10]: [5.000 average rating (1 reviews)] Entertaining Angels: The 
    Dorothy Day Story (1996)
     [5.000 average rating (1 reviews)] Aiqingwansui (1994)
     [5.000 average rating (1 reviews)] Someone Else’s America (1995)
    The (1993)
    and Light (1996)

How it works…




There’s more…


See also




In the recipes we learned various ways for understanding data and finding highest scoring reviews using IPython. 

Resources for Article:

Further resources on this subject:


* indicates required


Please enter your comment!
Please enter your name here