\chapter{Experiments}

In order to evaluate the performance of a recommender system, there are several scenarios to be considered depending on the structure of the dataset and the prediction accuracy. It is therefore necessary to determine a suitable experiment for the evaluation of our proposed hybrid music recommender that employs a rating matrix and vector representation of songs as inputs to produce \textit{top-N} song recommendations.

In addition, the performance of our hybrid approaches is compared with a pure content-based recommender algorithm.

%\section{Experiment aims}
%deviation between the actual and predicted ratings is measured 
%the prediction ratings are compared with a model-based collaborative filtering.

\section{Evaluation for recommender systems}

\subsection{Types of experiments}
The scenarios for experiments requires to define an hypothesis, controlling variables and generalization of the results. Three types of experiments \parencite{export:115396} can be used to compare and evaluate recommender algorithms:
\begin{itemize}
\item \textbf{Offline experiments:} where recorded historic data of users' ratings are used to simulate online users behaviour. The aim of this type of experiment is to refine approaches before testing with real users. On the other hand, results may have biases due to distribution of users.
\item \textbf{User studies:} where test subjects interact with the recommendation system and its behaviour is recorded giving a large sets of quantitative measurements. One disadvantage of this type of experiment is to recruit subjects that represent the population of the users of the real recommendation system.
\item \textbf{Online evaluation:} where the designer of the recommender application expect to influence the users' behaviour. Usually, this type of evaluation are run after extensive offline studies.
\end{itemize}

\subsection{Evaluation strategies}
On the other hand, evaluation of recommender systems can be classified \parencite{1242} in:
\begin{itemize}
\item \textbf{System-centric} process has been extensively exploited in CF systems. The accuracy of recommendations is based exclusively on users' dataset and is evaluated through predictive accuracy, decision based and rank based metrics.
\item \textbf{Network-centric} process examines other components of the recommendation system, such as diversity of recommendations, and they are measured as a complement of the metrics of system-centric evaluation.
\item \textbf{User-centric:} The perceived quality and usefulness of recommendations for the users are measured via provided feedback.
\end{itemize}

\subsection{Decision based metrics}
Our hybrid recommender produces a list of songs for each user, hence, it is necessary to evaluate the recommendation with a metrics derived from \textit{confusion matrix} that reflects the categorisation of test items as true positives (TP), false positives (FP), true negatives (TN) and false negatives (FN). In this project we consider the following metrics \parencite{1242}:
\subsubsection{Precision}
\begin{equation}
Precision = \frac{TP} {TP+FP}\label{eq:1}
\end{equation}
%Text \eqref{eq:1}
\subsubsection{Recall}
\begin{equation}
Recall = \frac{TP} {TP+FN}\label{eq:2}
\end{equation}
%Text \eqref{eq:2}
\subsubsection{F1}
\begin{equation}
Recall = \frac{2 \times Precision \times Recall} {Precision+Recall}\label{eq:3}
\end{equation}
\subsubsection{Accuracy}
\begin{equation}
Recall = \frac{TP+TN} {TP+FP+TN+FN}\label{eq:4}
\end{equation}

\section{Evaluation method}
The hybrid music recommender system proposed in this project is evaluated through an offline experiment and the results are presented with decision based metrics described in the previous section.

\subsection{Training set and test set}
For the purpose of evaluate and compare the performance, we use a random sample of 20 \% of the total size of the cleaned Taste Profile subset for testing and the rest 80 \% is used to train both the hybrid recommender approach and content-based recommender baseline. The sampling process is repeated for ten times.

%\subsection{Evaluation measures}
%Because the dataset does not include explicit ratings, hence, the number of plays of tracks are considered as users' behaviours,



