This is a review of the contents of Deep learning-based Recommender System in recent years. Specifically, most of the contents come from Deep Learning Based Recommender System: A Survey and New Perspectives, I translated and summarized some of the contents. At the same time, I sometimes read the original text of the jobs mentioned in the survey and supplement some of the more specific ones.

The text start

Deep learning has been widely applied in various fields in recent years, and recommendation system is no exception. Then, how does deep learning algorithm relate to recommendation system? The purpose of this paper is to review the recent research results of recommendation system based on deep learning and forecast the future trend.

Recommendation system is the defense line of information explosion and plays a vital role in users’ choice. Typically, we generate recommendation lists based on user preferences, project characteristics, user history, and some other additional information from temporal and spatial data. We divide the recommendation system into collaborative filtering, Content-based recommendation system and hybrid recommendation system. Recently, recommendation systems based on deep learning have made significant progress by overcoming some problems of traditional models and achieving high recommendation quality. Deep learning can effectively capture nonlinear and nontrivial user-project relationships, and can represent data as more complex and abstract codes. In addition, it captures the intricacies of the data itself from a wealth of accessible data sources (above and below, text and visual information).

In recent years, the number of research publications on recommendation methods based on deep learning has increased exponentially, and deep learning has become an inevitable part of recommendation system research. RecSys has been conducting regular deep learning recommendation systems workshops since 2016. The workshop aims to promote research and encourage the application of recommendation systems based on deep learning. We should understand the strengths, weaknesses and application scenarios of deep learning models in academia and industry. This survey survey is mainly from NIPS, ICML, ICLR, KDD, WWW, SIGIR, WSDM, RecSys and other top conferences, with a considerable degree of representativeness. The purpose of the survey is to provide a quick directory for practitioners to query. The survey seems to be constantly updated so far, and my series only includes the content from the survey as of October 08, 2018.

First, before we can discuss the problem, we need to determine the scope of our discussion and do some pre-definition.

Overview of Recommendation Systems and Deep Learning

The recommendation system estimates the user’s preferences for items and recommends items that the user might like. We divide the recommendation system into collaborative filtering, Content-based recommendation system and hybrid recommendation system. Collaborative filtering learns and makes recommendations from the history of User-item interactions, where interactions include either explicit feedback (for example, a user’s previous rating of an item) or implicit feedback (for example, a user’s browsing history). Content-based recommendation is mainly based on some auxiliary information of item and user, and some recommendation results can be obtained by comparing these auxiliary information. Consider various auxiliary information, such as text, images, and video. A hybrid model is a recommender system that integrates two or more recommender strategies

Symbol definition

We set up the following discussion scenario: We haveA user andThe item,Is the interaction matrix,Is the predicted interaction matrix.On behalf of the userFor the itemThe preference,Are the users predicted by the modelFor the itemPreferences.A line of represents the preferred behavior of a user and is expressed as.A column of represents the user preference behavior of an item, and is expressed as.andThey represent observed and unobserved interactions. We also useandTo represent user and item latent factor.Is the dimension of the hidden feature.

Deep learning technology

Deep learning is often thought of as a sub-field of machine learning. Deep learning is typically defined as learning deep representations, that is, multi-level representations and abstractions from data. For practical reasons, we consider any architecture that can be decomposed into a neural network to be “deep learning” as long as it optimizes differentiable objective functions using a variant of stochastic gradient descent (SGD). Neural network architecture has achieved great success in both supervised and unsupervised learning tasks. In this section, we illustrate the various architectural paradigms that are closely related to this survey.

• Multilayer Perceptron (MLP) : MLP is a feedforward neural network with multiple (one or more) hidden layers between the input and output layers. Here, the perceptron can use any activation function, not necessarily a strict binary classifier. MLP can be represented as a stack layer of nonlinear transformation, learning hierarchical feature representation. MLPS are also known as universal approximators.
• Autoencoder(AE) : An unsupervised model used to reconstruct its input data in the output layer. Typically, the bottleneck layer (the most intermediate layer) is used as a characteristic representation of input data. There are many variations of autoencoders, such as denoising autoencoder, marginalised denoising autoencoder, Sparse autoencoder, Contractive autoencoder and variational autoencoder, VAE.
• Convolutional Neural Network (CNN) is a special feedforward Neural Network, which has Convolutional layer and pooling operation. It can capture global and local features and significantly improve efficiency and accuracy. It does a good job of processing data with a grid-like topology.
• Recurrent Neural Network (RNN) : Recurrent Neural Network (RNN) is suitable for modeling sequential data. Different from feedforward neural networks, RNN has cyclic and memory structures to remember the previous calculation results. Variations such as short and long memory (LSTM) and gated loop unit (GRU) networks are often deployed in practice to overcome the gradient extinction problem.
• Restricted Boltzmann Machine (RBM) is a double layer neural network composed of visible layer and hidden layer. It can be easily stacked into the deep net. Restricted here means that there is no intra-layer communication in either the visible or hidden layers.
• Neural Autoregressive Distribution Estimation (NADE) is an unsupervised Neural network based on Autoregressive model and feedforward Neural network. It is a tractable and efficient estimator for modeling data distribution and density.
• Adversarial Networks (AN) : Adversarial Networks (AN) is a generative neural network consisting of discriminators and generators. Train two neural networks simultaneously by competing against each other in a minimax game framework.
• Attentional Models (AM) is a skill. Attention mechanisms are generally ubiquitous and are accepted in the fields of computer vision and natural language processing. However, it is also an emerging trend in in-depth recommendation system research.
• Reinforcement Learning (DRL) : Deep Reinforcement Learning. Combination of deep neural networks and reinforcement learning.

Advantages of deep learning techniques

We summarize the advantages of the recommendation model based on deep learning:

• Nonlinear transformation. In contrast to linear models, deep neural networks can model nonlinearity in data using nonlinear activation (e.g. Relu, SigmoID, TANH, etc.). This property makes it possible to capture complex and complex patterns of user item interaction. Conventional methods such as MF, FM and sparse linear models are basically linear models.
• Representational learning. Deep neural networks effectively learn potential explanatory factors and useful representations from input data. Often, a lot of descriptive information about projects and users is available in real-world applications. Using this information can improve our understanding of the project and our users, leading to better referees. Therefore, it is a natural choice to apply deep neural networks to representation learning in recommendation models. The advantages of using deep neural network to assist representation learning are twofold :(1) it reduces the workload of manual feature design. Feature engineering is a labor-intensive task. Deep neural networks can automatically learn features from raw data in an unsupervised or supervised way. (2) It enables the recommendation model to include heterogeneous content information, such as text, image, audio and even video. The deep Web has made breakthroughs in multimedia data processing and demonstrates the potential of learning from a variety of sources as a form of representation.
• Sequence modeling. Deep neural networks have shown promise for many sequential modeling tasks, such as machine translation, natural language understanding, speech recognition, chatbots, and more. RNN and CNN play key roles in these missions. RNN does this by storing state internally, while CNN does this through filters that slide over time. They have wide applicability and flexibility in data mining sequence structure.
• Flexibility. Deep learning techniques are highly flexible, especially with the emergence of many popular deep learning frameworks such as Tensorflow, Keras, Caffe, MXnet, DeepLearning4j, PyTorch, Theano, etc. Most of these tools are developed in a modular manner and have active communities and professional support. Good modularity makes development more efficient. For example, it is easy to combine different neural structures to form powerful hybrid models, or to replace one module with another. Therefore, we can easily build hybrid and composite recommendation models to capture different characteristics and factors at the same time.

Potential disadvantages of deep learning technology

• Explanatory. Despite its success, deep learning works like a black box, and providing interpretable predictions seems like a very challenging task. A common argument against deep neural networks is that hidden weights and activations are usually unexplainable, limiting explainability. However, some of this concern has been alleviated with the advent of neuronal models, where the most advanced models available have been able to explain meaning to a degree that allows interpretable recommendations to be made.
• Data requirements. The second possible limitation is that deep learning is considered data-hungry because it requires sufficient data to fully support its rich parameterization. However, it is relatively easy to obtain large amounts of data in the context of recommendation systems research compared to other areas where marker data is scarce, such as language or vision. Mega/billion datasets are not only common in industry, but are also published as academic datasets.
• Extensive hyperparameter tuning. A third mature argument against deep learning is the need for extensive hyperparametric tuning. However, we note that hyperparametric tuning is not unique to deep learning, but occurs in general machine learning (for example, normalization factors and learning rates must also be adjusted for traditional matrix factorization, etc.). However, in some cases, deep learning may introduce additional hyperparameters.

Classification of recommended methods for deep learning

To provide a bird’s eye view of the field, we categorize existing models according to the type of deep learning techniques employed. We further divide deep learning-based recommendation models into the following two categories. The following figure summarizes the classification scheme.

• Recommendation with Neural Building Blocks: Recommendations using a single deep learning approach. We discuss the different recommended methods according to the previous eight technology categories.
• Recommendation with Deep Hybrid Models.

Table 1 lists all the models that have been investigated and we organize them according to the above classification scheme. In addition, we also summarized some publications from the task perspective in Table 2. The publications we investigated dealt with a variety of referral tasks. Due to the use of deep neural networks (e.g., session-based recommendations, image recommendations, video recommendations), some previously uncommon tasks have begun to gain attention. Some tasks may not be new to the recommended research area, but DL offers more possibilities for finding better solutions. For example, processing images and videos would be a daunting task without the help of deep learning techniques. The sequence modeling function of deep neural network can easily capture the sequential patterns of user behavior.

We will discuss the details of these methods in a future blog post. The current list of references for this survey is as follows, and interested readers can index articles themselves: