A classical recommendation system is generally divided into recall stage and sorting stage. Recall stage refers to the process of selecting candidate sets (generally ranging from thousands to hundreds) from a large number of items (possibly up to billions), while sorting stage refers to sorting items in candidate sets according to user preferences. In essence, the goal of all recommendation algorithms is to identify the user’s preference for a certain item, that is, for a user-item-preference matrix as follows, we need to predict the value of any point in it:

For the most part, the list is fairly sparse and has been updated over time, making it impossible to calculate all of it. So basically, the model will calculate the preference value of a user only when we query, and different recommendation systems mean:

  1. Different definitions of user preference. Sometimes, users’ ratings are explicit. For example, we rate items on Douban and Taobao. But more often than not, we don’t have an explicit rank. In this case, the data we have is implicit feedback from users, such as browsing, clicking, playing, etc. We need to use the user’s behavior to determine the value of preference, and in many cases we use the weighted sum of different behaviors as preference. The weight of user behavior will change according to our attention. For example, in a video website, we may pay attention to the playback behavior. On Taobao, the focus may be on buying behavior. We define user preference differently according to different concerns, which determines the input of the recommendation system and often also determines the target of the recommendation system.

  2. Different definitions of prediction targets. A recommendation system is very complex, and we will use many different recommendation algorithms in different scenarios to compose the whole recommendation algorithm. The user’s preference is often complex and cannot be observed directly in many scenarios. In this case, we usually use other indicators as its measurement. In many cases, preference is not a suitable target value for the recommendation algorithm, so we still set different related target values for the recommendation algorithm in different scenarios. Typical examples are CTR (Click rate), playback_duration, etc.

  3. Different definitions of metrics. Metrics are an overall goal, such as user satisfaction, user purchase rate, etc., and we use these metrics when we finally evaluate the quality of a recommendation system. For example, a video site, the real focus must be on user retention, profitability and other data, which are long-term evaluation indicators; The total user watch minutes is a short-term evaluation index. Preference is a value determined by a user’s play behavior, click behavior, favorite behavior, etc. In the recommendation algorithm of different scenarios, the prediction target may be the user’s CTR, playback_duration and so on.

  4. Different screening strategies for recall phase. They are basically general rules, such as Popular content + Saved Content + Historic Related content and so on.

  5. Prediction models for different sorting stages. There are so many kinds of recommender systems that they are now the main focus of the field.

  6. Different model update times, either immediate or offline. This is related to the underlying architecture of the recommendation system, which is usually decided at the time of the whole system design. However, there are still some exceptions. For example, some algorithms are updated immediately without pre-training, such as MAB, Kalman filter and so on. However, our daily work basically focuses on the fifth point, which is the prediction model of different sorting stages in different scenarios. The algorithms we’re going to talk about are all about this point.

Collaborative Filtering: Collaborative Filtering

Collaborative filtering is one of the most classical neighborhood recommendation algorithms. The neighborhood algorithm means that the new recommendation item is generated from its neighbor’s data. In different collaborative filtering algorithms, the definition of neighbor is different.

User-based CF

Let’s consider the user-item matrix above. Let’s say we now have some user preference for item, and our next goal is to fill in the blanks, what do we choose to fill in? In user-based CF, we use two steps:

(1) Find a set of users with similar interests to the target users.

(2) Find the items favored by the users in this set but not consumed by the target users, update their user-item preference and sort them.

We define user similarity as cosine similarity:

  • Recall stage, the establishment of goods to the user of inverted index table, for each of the items are stored on the item produced behavior of users, scanning the table, find all | N (u) studying N (v) | for users who do not is 0, computing the cosine similarity. For user V, the k users most similar to user V are his neighbors. The set of items where the user did something is the recall set.

  • Ordering stage: User U’s interest in item I is as follows, where S(u, K) is the k users closest to user U’s interest, N(I) is the set of users who have behaviors on item I, WUV is the interest similarity between users U and V, and RVI is user V’s interest in item I.

The improvement of user-based CF is basically the change of these two formulas, that is, the calculation method of user similarity or the calculation method of item interest degree.

Item-based CF

So let’s look at this user-item matrix. In the item-based CF, we recommend items to users that are similar to items they previously liked. In contrast to user-based CF, it mainly calculates the similarity between items through users’ behaviors.

Item similarity:

Where N(I) represents the users who like item I

Similar to the User-CF algorithm, item-CF can first establish the user-item inverted index table to calculate the cosine similarity between items. After obtaining the similarity between items, the following formula can be used to calculate the preference of User U to item J:

Where N(u) is the item that users have interacted with, S(j,K) is the most similar K item, Wji is the similarity between item J and item I, rui is user U’s preference for item I.

The biggest advantage of item-CF is that it can provide recommended explanations from historical data.

We can constantly improve item-CF by normalizing preferences, punishing popular items, and changing the similarity formula.