Recommender systems: When you have no idea

Yesterday, I went to dinner with a group of my friends. I was an invitee, so I had no choice but accepting their invitation and go to the restaurant they had already reserved. I was completely new to that restaurant and had no idea about what were the best options for me. I went through the menu and found few familiar dishes I have had before from other places. We had 10 people sitting around the table and 7 out of them had already decided what to get for the dinner as they were members of the restaurant. 3 of us were completely lost :). I was switching between dishes as I had no idea. What did I do?

 

I decided to ask for help from my friends to take the decision on dishes. I selected 3 friends who have similar taste to me. We know each other from the childhood. I asked the best options from those 3 folks. 2 of them recommend me fried noodles with chicken and other one suggested me pasta instead of noodles. Thanks to my friends, I could reduce the list to 2 dishes. Then, I decided to take the final decision considering my past experience. I like noodles more than pasta and I took fried noodles with chicken dish. It was a great choice and their recommendation worked for me. I would like to recommend you that dish if you go to that restaurant one day. Just let me know :)

Sometimes we face these kinds of situations where we need suggestions, advice or recommendations from experts on specific subjects. It may be a movie to watch, book to read, dish from a restaurant, whom to vote, doctor to visit, song to listen, hotel to stay and much more. By nature, we would like to ask it from our friend or a similar person who has expertise in the topic or with previous experience. Also, we consider our own experience or preferences in such situations. In real life, sometimes we have to rely on other's recommendations or our previous experiences/choices or take the decision considering both. This is the foundation of Recommender Systems.

Recommender systems are tools and technologies which recommend items/concepts/services/actions or solutions to users. Today, Recommender Systems are heavily and widely used in social networks, e-commerce websites, recommendation software which recommends movies, books, songs. Let's look at the technical concepts and methods under the hood.

We can use several different methods to generate recommendations. We can recommend the most popular items to users. If we look at our restaurant example, the restaurant can recommend most popular dishes to new(or existing) customers. It can collect data from purchasing history and identify a list of popular items which can be recommended later. This method is called Popularity Based Recommendations. Do you see any problem in this method? While this is very easy to implement, we are missing personalization in generated recommendations. Personalization is the process of tailoring items/solutions to individual users' characteristics or preferences. Systems based on popularity based recommendations provide a fixed set of items for any user in a given time interval. Can't we think the problem as a classification problem? Yes, we can. We can consider an item and predict whether the user is going to like it or not. Here we look at the attributes of the item and take decisions based on those attributes. We can consider user attributes too. Identifying recommendation problem as a classification problem is called Classification Based Recommendations method. Normally, recommendation problems deal with a very large number of users and items. So, we are facing a problem here. If we are going forward with classification approach, we have to work with a large number of features(from user's perspective, items are features and vice versa). Can't we find a solution for the problem of generating recommendations based on our real life experiences? Can't we use our neighbor's experiences or our own experiences to solve this? Yes, we can. Neighborhood Based Recommendations method addresses the problem using our neighborhood. Content Based Recommendations method uses user's past experiences to generate recommendations. Let's discuss these two methods it in detail.

There are two commonly used methods in RS.

1. Collaborative filtering
2. Content-based filtering

 

Collaborative Filtering(CF)

Collaborative Filtering(CF) is the most popular and widely adapted method among two. CF relies on past ratings of the active user and other users. It generates recommendations based on the concept of 'similar users prefer similar items'. So, based on similarity of users, CF generates recommendations for the active user.

If we peep into our restaurant experience through Collaborative Filtering window, First, I selected people who have similar taste to me. Then, they suggested me few dishes according to their ratings and I accepted their choices considering the similarity between us.

Content-based filtering

CBF generates recommendations based on user's past behavior or preferences. This method retrieves the list of items the user has used/rated previously and try to find new items which have similar features/attributes with past items. Here, we are walking through user profile considering attributes of the user to generate recommendations.

This can be related to my past experience of taking dinner from a new restaurant. At the moment of getting the decision of dishes, I considered my preferences and past experience. I was looking for similar dishes to my favorite choices.

*Hybrid method

Another blooming research area is using a hybrid method combining Collaborative filtering and Content-based filtering methods. The goal is to mitigate weaknesses of individual methods while taking advantages of strengths of both.

When I get my final decision on the dish, I used concepts of the hybrid method. I got suggestions from my friends and then used my preferences/past experience to select one.

 

According to what we have discussed so far, it is obvious that the science of Recommender Systems is developed imitating human rational decision-making process.

Let me wind up our recommendation story with a true story of Touching the Void and Into the Air books.

In 1988, a British mountain climber Joe Simpson wrote a book called Touching the Void, a harrowing account of near-death in the Peruvian Andes. It got good reviews, only a modest success, it was soon forgotten. Then, a decade later, a strange thing happened. Jon Krakauer wrote Into Thin Air, another book about a mountain-climbing tragedy, which become a publishing sensation. Suddenly, Touching the Void started to sell again. Amazon’s recommendation system noticed a few people who bought both books, and started recommending Touching the Void to people who bought, or were considering, Into Thin Air. Had there been no on-line bookseller, Touching the Void might never have been seen by potential buyers, but in the on-line world, Touching the Void eventually became very popular in its own right, in fact, more so than Into Thin Air.

Let's meet again.

K-Means clustering with Scikit-learn

K-Means clustering is a popular unsupervised classification algorithm. In simple terms we have unlabeled dataset with us. Unlabeled dataset means we have a dataset but we don't have any clue about how we are going to categorized each row in the dataset. Following is an example few rows from unlabeled dataset about crime data in USA. Here we have one row for each state and set of features related to crime information. We have this dataset with us but we don't know what to do this with this data. One thing we can do is finding similarities of the states. In other way we can try to prepare few buckets and put states into those buckets based on the similarities in crime information.

State		Murder	Assault	UrbanPop	Rape
Alabama		13.2	236	58	21.2
Alaska		10	263	48	44.5
Arizona		8.1	294	80	31
Arkansas		8.8	190	50	19.5
California		9	276	91	40.6

Now let's discuss how we can implement K-Means cluster for our dataset with Scikit-learn. You can download USA crime dataset from my github location.

Import KMeans from Scikit-learn.

from sklearn.cluster import KMeans

Load your datafile into Pandas dataframe

df = Utils.get_dataframe("crime_data.csv")

Create KMean model providing required number of clusters. Here I have defined required number of clusters to 5

KMeans_model = KMeans(n_clusters=5, random_state=1)

Refine your data removing non-numeric data, unimportant features..etc.

df.drop(['crime$cluster'], inplace=True, axis=1) df.rename(columns={df.columns[0]: 'State'}, inplace=True)

Select only numeric data in your dataset.

numeric_columns = df._get_numeric_data()

Train KMeans-clustering model

KMeans_model.fit(numeric_columns)

Now you can see the label of each row in your training dataset.

labels = KMeans_model.labels_ print(labels)

Predic new state’s crime cluster as follows

print(KMeans_model.predict([[15, 236, 58, 21.2]]))

Model Evaluation With Coress Validation

We can use cross validation to evaluate the prediction accuracy of the model. We can keep subset of our dataset without using it for training purposes. So those are new or unknown data for the model once we train that with the rest of data. Then we can use that subset of unused data to evaluate the accuracy of the trained model. Here, first we partition data into test dataset and training dataset and then train the model with the training dataset. Finally we evaluate the model with the test dataset. This process is called "Cross Validation".

In this blog post I would like to demonstrate how we can cross validate a decision tree classification model which is build using scikit-learn + Panda. Please visit decision-tree-classification-using scikit-learn post if you haven't create your classification model yet. As a recap at this point we have a decision tree model which predicts whether a given person in Titanic ship is going to survive from the tragedy or die in the cold, dark sea :(.

In previous blog post we have used entire Titanic dataset for training the model. Let's see how we can use only 80% of data for training and the rest 20% for evaluation purpose.

# separating 80% data for training train = df.sample(frac=0.8, random_state=1) # rest 20% data for evaluation purpose test = df.loc[~df.index.isin(train.index)]

Then we train the model normally but we use training dataset

dt = DecisionTreeClassifier(min_samples_split=20, random_state=9) dt.fit(train[features], train["Survived"])

Then we predict the result for rest 20% data.

predictions = dt.predict(test[features])

Then we can calculate Mean Squared Error of the predictions vs. actual values as a measurement of the prediction accuracy of the trained model.

We can use scikit-learn built in mean squared error function for this. First import it to current module.

from sklearn.metrics import mean_squared_error

Then we can do the calculation as follows

mse = mean_squared_error(predictions, test["Survived"]) print(mse)

You can play with the data partition ratio and the features and observe the variation of the Mean Squared Error with those parameters.

Decision Tree Classification using scikit-learn

Please visit Preparing-machine-learning-developing blog post if you haven't prepared your development environment yet.

First we have to load data from a dataset. We can use a dataset in our hand at this point or online dataset for this. Use following python method to load titanic.csv data file into Pandas[1] dataframe.

Here I have used my downloaded csv file. You can download that file from https://github.com/caesar0301/awesome-public-datasets/tree/master/Datasets or Goolge it

def load_data():   df = pand.read_csv("/home/malintha/projects/ML/datasets/titanic.csv");   return df

before you use Pandas functions you have to import that module.

import pandas as pand

Now we can print the first few rows of the dataframe using

print(df.head(), end = "\n\n")

And it will output

PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
1	0	3	Braund, Mr. Owen Harris	male	22	1	0	A/5 21171	7.25		S
2	1	1	Cumings, Mrs. John Bradley (Florence Briggs Thayer)	female	38	1	0	PC 17599	71.2833	C85	C
3	1	3	Heikkinen, Miss. Laina	female	26	0	0	STON/O2. 3101282	7.925		S
4	1	1	Futrelle, Mrs. Jacques Heath (Lily May Peel)	female	35	1	0	113803	53.1	C123	S

We can remove “Name”,  “Ticket” and “PassengerId”  features for the dataset as they are not much important features over other features. We can use Pandas ‘drop’ facility to remove column from a dataframe.

df.drop(['Name','Ticket',’PassengerId’], inplace=True, axis=1)

Next task is mapping nominal data into integers in order to create the model in scikit-learn.

Here we have 3 nominal feature in our dataset.

1. Sex
2. Cabin
3. Embarked

We can replace the original values with integers with following code segment.

def map_nominal_to_integers(df):   df_refined = df.copy()   sex_types = df_refined['Sex'].unique()   cabin_types = df_refined['Cabin'].unique()   embarked_types = df_refined["Embarked"].unique()   sex_types_to_int = {name: n for n, name in enumerate(sex_types)}   cabin_types_to_int = {name: n for n, name in enumerate(cabin_types)}   embarked_types_to_int = {name: n for n, name in enumerate(embarked_types)}   df_refined["Sex"] = df_refined["Sex"].replace(sex_types_to_int)   df_refined["Cabin"] = df_refined["Cabin"].replace(cabin_types_to_int)   df_refined["Embarked"] = df_refined["Embarked"].replace(embarked_types_to_int)   return df_refined

We have one more step to shape-up our dataset. If you look at the refined dataset carefully ,you may able to see there are “NaN” value for some of age values. We should replace this NaN with appropriate integer value. Pandas provide built-in function for this. I will use 0 as the replacement for NaN.

df["Age"].fillna(0, inplace=True)

Now we all set to build the decision tree from our refined dataset. We have to choose the features and the target value for the decision tree.

features = ['Pclass','Sex','Age','SibSp','Parch','Fare','Cabin','Embarked'] X = df[features] Y = df["Survived"]

Here,X is the feature set and Y is the target set. Now we build the decision tree. For this you should import the scikit-learn decision tree into your python module.

from sklearn.tree import DecisionTreeClassifier

And build the decision tree with our feature set and the target set

dt = DecisionTreeClassifier(min_samples_split=20, random_state=9) dt.fit(X,Y)

Now it is time to do a prediction with our trained decision tree. We can use sample feature data set and predict the target for that feature value set.

Z = [1,1,22.0,1,0,7.25,0,0] print(dt.predict(Z))

Please find source code from my github location https://github.com/malinthaadikari/machine-learning/tree/master/DecisionTreeClassification

[1]. http://pandas.pydata.org/