Concepts and Indicators for Recommendation System Evaluation (2)

Novelty usually refers to the appearance of a new item in a recommendation, which has been defined differently in different literature. Here, we divide the definition and indicators of novelty into three levels, as shown in the table below. The novelty index is called 𝑛𝑜𝑣 (𝑅𝑢) in this paper.

1. Everyday life level novelty

Creating indicators that measure novelty at the level of everyday life is not easy. Novelty level 1 measures must take into account information in the context of the system in order to measure what is known and unknown to users.

2. System-level novelty

There are many definitions of system-level novelty. Simply put, for users, a new project is one about which the user knows little or nothing.

Some people believe that novelty refers to the recommendation system predicting items that users do not know about and would not find through other channels. Novelty is also defined as the difference between recommended items and items already consumed by users. Novelty is also defined as the proportion of unknown items in the user’s predicted list.

In practice, the above definition only considers new items when looking at previously consumed items in the user’s consumption history, not items consumed outside the system. In summary, system-level novelty refers to items unknown to users in system information.

Most of the novelties proposed in the literature are system-level novelties. An evaluation method has been proposed to calculate the novelty in the recommendation list as the similarity between the items in the recommendation list and the items in the user’s historical consumption (𝐻𝑢), which is the metric in Formula 7.

Others propose novelty by calculating the sum of the popularity of items in a user’s recommended list, which is shown in Formula 8. For example, the popularity of a project can be calculated by the number of users consuming the project (𝑝𝑜𝑝). In addition, they also provides a measure of variation, for example – 𝑙 𝑜 𝑔 2 𝑝 𝑜 𝑝 (𝑖) | 𝑈 |.

3. Novelty at the recommended list level

Level 3 relates to novelty at the recommended list level, i.e., not repeating recommended items. In this sense, novelty is defined as a non-repeating item on the recommended list that does not contain user information. Says novelty is associated with non-redundant items in a recommended list unknown to the user. In short, level 3 is the extreme of Level 2, which does not even allow redundant items or duplicate recommendation results in the recommendation list.

To measure novelty level 3 simply survey the items on the recommended list. Novelty level 3 indicators do not require user information. In this sense, formula 10 can calculate the similarity of items in the recommended list, where 𝑑 (𝑖, 𝑗) represents the distance between items 𝑖 and 𝑖. However, this metric is more like a measure of in-list similarity and may not measure novelty.

In addition, a measure of novelty in the recommended list has been proposed, as shown in Formula 11. This measure takes into account an item’s position in the sorted recommendation list and is used to calculate a discount function for browsing the list (𝑑𝑖𝑠𝑐 (𝑖𝑘). In addition, the measure also calculate users when browsing see project (𝑝 (𝑠 𝑒 𝑒 𝑛 | 𝑖 𝑘) probability. Since this probability refers to the uncertainty of information consumed by users, the measure is best classified between levels 2 and 3 of novelty.

diversity

Diversity focuses on the richness of the items on the recommended list. For diversity metrics, the notation used in this article is 𝑑𝑖𝑣 (𝑅𝑢).

Some argue that the diversity of recommendation systems has the opposite similarity effect. The authors point out that recommendations with small changes may not be of interest to users. Others argue that recommendation systems often predict similar items compared to a user’s spending history. Therefore, diversity means balancing the list of recommendations to cover the user’s entire set of interests.

Unlike novelty, the definition of diversity is largely consistent in the literature. It is generally accepted that diversity represents the variety of items on the recommended list.

In terms of measuring diversity, there is a tendency to count diversity as dissimilarity between items on a recommended list. Some have proposed a measure for similarity within a list, as shown in Formula 12. The function 𝑑 (𝑖, 𝑗) calculates the distance between the item 𝑖 and the item in the recommendation list 𝑅𝑢. This metric actually captures the similarity of lists; Thus, low values for this metric represent more similar lists, where items are similar to each other.

The internal list similarity measure is also used in other diversity efforts. Others use cosine similarity as a function of distance, a metric that can be seen in Equation 13.

Other indicators have been proposed, as described in Formula 14. The formula in Figure 14 is a more specific calculation of the similarity within a list. This measure takes into account a discount function for the location of each pair of items being analyzed (𝑑𝑖𝑠𝑐 (𝑘). In addition, the metric uses distances between items (𝑑 (𝑖𝑘, 𝑖𝑙), such as cosine similarity distances.