Some people are addicted to Brushing Douyin, while others are addicted to brushing Zhihu. The recommendation system has influenced and even controlled people’s lives. This article will start with the simplest algorithms and processes, using Flask and Gorse to quickly build a Steam game recommendation system.

Recommended System Architecture

Before starting development, we need to design the architecture of my recommendation system, as shown below:

It can be divided into three parts:

  • Gorse: Gorse is an offline recommendation system that submits user-game purchase records to it, and it can automatically train the model to generate a list of game recommendations;
  • Flask: A Web service written in Flask is responsible for user login, requesting user inventory information from Steam, pushing inventory information to Gorse, and pulling push results;
  • Steam: Provide inventory information via the API, and provide game cover art.

The Steam game recommendation system has been deployed in steamlens.gorse. IO, and if you have a Steam account and access to the Steam community (you get the idea), try out its personalized recommendations. The code is also open source on GitHub, so if you have a VPS with access to Steam’s community server, you can try to deploy it yourself.

Create the recommendation system server

The installation

First of all, we need to install the recommendation system back-end gorse, if you have installed the language environment, will join the environment variable $$GOBIN PATH, you can directly use the following command to install:

$ go get github.com/zhenghaoz/gorse/...
Copy the code

Data preparation

Everything is based on data, but fortunately there is already a Steam dataset that someone else is sharing online. The raw data is huge, and for the sake of the demo, it was sampled to Games.csv. We create a folder and download the data:

$ mkdir SteamLens
$ cdSteamLens $ wget http://cdn.sine-x.com/backups/games.csv ... $head games.csv 76561197960272226,10,505 76561197960272226,20,0 76561197960272226,30,0 76561197960272226,40,0 76561197960272226,50,0 76561197960272226,60,0 76561197960272226,70,0 76561197960272226,130,0 76561197960272226,80,0 76561197960272226100, 0Copy the code

You can see that the data has three columns: users, games, and duration.

The test model

Before creating the recommendation service, you need to choose the most suitable recommendation algorithm. Gorse provides the evaluation of various models. You can run gorse test-H or check the online documentation to learn how to use it. Our dataset is weighted (game duration) implicit feedback, with four models available based on the input supported by each model: Item-POP, KNN_Implicit, BPR, and WRMF.

First test out the unpersonalized recommendations as a benchmark:

$ gorse test item-pop --load-csv games.csv --csv-sep ', '--eval-precision --eval-recall --eval-ndcg --eval-map --eval-mrr ... +--------------+----------+----------+----------+----------+----------+----------------------+ | | FOLD 1 | FOLD 2 | FOLD 3 | FOLD 4 | FOLD 5 | MEAN | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | | Precision @ 10, 0.080942 | | | | | 0.078248 0.078880 0.080253 0.080655 0.079796 (0.001548 mm) | | Recall @ 10 | | | | 0.312299 0.310532 0.308894 0.305665 0.308428 0.309163 (0.003498 mm) | | | | NDCG @ 10 | | | | | | 0.210466 0.209945 0.209004 0.209796 0.211919 0.210226 (0.001693 mm) | | MAP @ 10 | | | | 0.130520 0.132018 0.133684 0.133500 0.135297 0.133004 (0.002484 mm) | | | | | MRR @ 10 | | | | 0.244244 0.240176 0.242664 0.247601 0.241920 0.243321 (0.004280 mm) | | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + 2019/11/07 09:56:51 Complete cross validation (22.037387763 s)Copy the code

Test the implicit KNN:

$ gorse test knn_implicit --load-csv games.csv --csv-sep ', '--eval-precision --eval-recall --eval-ndcg --eval-map --eval-mrr ... +--------------+----------+----------+----------+----------+----------+----------------------+ | | FOLD 1 | FOLD 2 | FOLD 3 | FOLD 4 | FOLD 5 | MEAN | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | | Precision @ 10, 0.150892 | | | | | 0.150013 0.152162 0.147429 0.153211 0.150742 (0.003312 mm) | | Recall @ 10 | | | | 0.533619 0.546523 0.529160 0.543382 0.533702 0.537277 (0.009245 mm) | | | | NDCG @ 10 | | | | | | 0.530433 0.545167 0.529590 0.546386 0.528442 0.536004 (0.010383 mm) | | MAP @ 10 | | | | 0.453748 0.469989 0.451220 0.468641 0.453865 0.459493 (0.010497 mm) | | | | | MRR @ 10 | | | | 0.658769 0.636238 0.656008 0.635610 0.636045 0.644534 (0.014235 mm) | | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + 2019/11/07 09:59:14 Complete cross validation (169339752 s) 1 m4.Copy the code

Test BPR again:

$ gorse test bpr --load-csv games.csv --csv-sep ', '--eval-precision --eval-recall --eval-ndcg --eval-map --eval-mrr ... +--------------+----------+----------+----------+----------+----------+----------------------+ | | FOLD 1 | FOLD 2 | FOLD 3 | FOLD 4 | FOLD 5 | MEAN | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | | Precision @ 10, 0.127123 | | | | | 0.126719 0.124914 0.129396 0.128440 0.127318 (0.002405 mm) | | Recall @ 10 | | | | 0.515385 0.511863 0.502971 0.503914 0.505500 0.507926 (0.007458 mm) | | | | NDCG @ 10 | | | | | | 0.424385 0.405582 0.427279 0.421336 0.434958 0.422708 (0.017126 mm) | | MAP @ 10 | | | | 0.336659 0.332219 0.350960 0.313238 0.337824 0.334180 (0.020942 mm) | | | | | MRR @ 10 | | | | 0.447885 0.477137 0.466407 0.495087 0.475176 0.472338 (0.024453 mm) | | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + 2019/11/07 10:01:51 Complete cross validation (56.85278659 s)Copy the code

Finally, test the WRMF, since the game duration is very large, we need to set a small weight factor:

$ gorse test wrmf --load-csv games.csv --csv-sep ', 'Eval-precision --eval-recall -- eval-NDCG --eval-map -- EVAL-MRR --set-alpha 0.001... +--------------+----------+----------+----------+----------+----------+----------------------+ | | FOLD 1 | FOLD 2 | FOLD 3 | FOLD 4 | FOLD 5 | MEAN | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + | | Precision @ 10, 0.145834 | | | | | 0.143163 0.146564 0.147034 0.148021 0.146123 (0.002960 mm) | | Recall @ 10 | | | | 0.533113 0.533390 0.524673 0.535772 0.525784 0.530546 (0.005873 mm) | | | | NDCG @ 10 | | | | | | 0.501728 0.513855 0.506967 0.504544 0.499655 0.505350 (0.008505 mm) | | MAP @ 10 | | | | 0.423166 0.419840 0.415299 0.431339 0.421243 0.422177 (0.009161 mm) | | | | | MRR @ 10 | | | | 0.610589 0.596109 0.592858 0.592257 0.590023 0.596367 (0.014222 mm) | | + -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- + 2019/11/07 10:06:52 Complete cross validation (912709237 s) 3 m52.Copy the code

So far it seems (we didn’t really tune it well) that THE KNN algorithm performs best on our data set and has satisfactory speed, so we chose KNN as the recommended algorithm for this case. No recommended algorithm must be due to other algorithms, and the best algorithm depends on the characteristics of the data set. For example, the best model on MovieLens 100K is WRMF rather than KNN.

Import data

After selecting the model, we import the data into gorse’s built-in database, create a folder data for existing data, and import the data into data/gorse.db:

$ mkdir data
$ gorse import-feedback data/gorse.db games.csv --sep ', '
Copy the code

Start the server

Next, the config/gorse.toml configuration file of the recommended service needs to set the server listening address, port, database file location and some trivial recommended configurations. The implicit KNN does not need to be overparameter, so [params] is left blank.

# This section declares settings for the server.
[server]
host = "0.0.0.0"        # server host
port = 8080             # server port

# This section declares setting for the database.
[database]
file = "data/gorse.db"  # database file

# This section declares settings for recommendation.
[recommend]
model = "knn_implicit"  # recommendation model
cache_size = 100        # the number of cached recommendations
update_threshold = 10   # update model when more than 10 ratings are added
check_period = 1        # check for update every one minute
similarity = "implicit" # similarity metric for neighbors

# This section declares hyperparameters for the recommendation model.
[params]
Copy the code

After saving the configuration file, run the recommendation server:

$ gorse serve -c config/gorse.toml
...
2019/11/07 16:45:05 update recommends
2019/11/07 16:47:02 update neighbors by implicit
Copy the code

If the last two lines appear, the recommendation result has been generated.

Test recommended interface

We can use the RESTful API provided by Gorse to get the recommendation results:

$ curl http://127.0.0.1:8080/recommends/76561197960272226?number=10
[
 {
  "ItemId": 4540,
  "Score": 23.479386364078838},... {"ItemId": 57300,
  "Score": 22.156954153653245}]Copy the code

We got 10 recommendations, including game ids and recommended ratings.

Create a front-end presentation server

To apply for the key

We need to connect the user’s Steam account to get the inventory game, so the user needs to log in, visit the page of “Register Steam Webpage API key” and apply for the API key from Steam to call the API.

Flask development environment

Flask is now ready to install the Pythn package for Flask development.

$ pip install Flask
$ pip install Flask-OpenID
$ pip install Flask-SQLAlchemy
$ pip install uWSGI
Copy the code

We can create a folder named SteamLens to hold the Flask code:

$ mkdir steamlens
Copy the code

The front page

Front-end design is not the focus of this article. For HTML templates, see Steamlens /templates, and for static resources, see steamlens/static.

The template role data
page_gallery.jinja2 Show a list of games current_time: time,title: the title,items: List of games,nickname: Embrace nicknames
page_app.jinja2 Show a game and a list of similar games current_time: time,item_id: the game ID,title: the title,items: Similarity list,nickname: User name

Filling in the Configuration File

Before writing the back-end code, fill in the configuration information:

# Configuration for gorse
GORSE_API_URI = 'http://127.0.0.1:8080'
GORSE_NUM_ITEMS = 30

# Configuration for SQL
SQLALCHEMY_DATABASE_URI = 'sqlite:///.. /data/steamlens.db'
SQLALCHEMY_TRACK_MODIFICATIONS = False

# Configuration for OpenID
OPENID_STIRE = '.. /data/openid_store'
SECRET_KEY = 'STEAM_API_KEY'
Copy the code

Remember to replace STEAM_API_KEY with Steam’s key.

The user login

We’ll start by writing the basic framework and the Steam functionality to connect to. The file is located in steamlens/app.py and it will do as follows:

  1. Create a Flask App object from the environment variablesSTEAMLENS_SETTINGSRead configuration;
  2. Create an OpenID object to connect to Steam authentication.
  3. Create an SQLAlchemy object to connect to the database;
  4. When the user logs in, the user name and ID are retrieved and saved to the database, and the inventory game list is pushed to the Gorse server.
import json
import os.path
import re
from datetime import datetime
from urllib.parse import urlencode
from urllib.request import urlopen

import requests
from flask import Flask, render_template, redirect, session, g
from flask_openid import OpenID
from flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config.from_envvar('STEAMLENS_SETTINGS')

oid = OpenID(app, os.path.join(os.path.dirname(__file__), app.config['OPENID_STIRE']))
db = SQLAlchemy(app)

# # # # # # # # # # # # # # # # #
# Steam Service #
# # # # # # # # # # # # # # # # #

class User(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    steam_id = db.Column(db.String(40))
    nickname = db.Column(db.String(80))

    @staticmethod
    def get_or_create(steam_id):
        rv = User.query.filter_by(steam_id=steam_id).first()
        if rv is None:
            rv = User()
            rv.steam_id = steam_id
            db.session.add(rv)
        return rv


@app.route("/login")
@oid.loginhandler
def login(a):
    if g.user is not None:
        return redirect(oid.get_next_url())
    else:
        return oid.try_login("http://steamcommunity.com/openid")


@app.route('/logout')
def logout(a):
    session.pop('user_id'.None)
    return redirect('/pop')


@app.before_request
def before_request(a):
    g.user = None
    if 'user_id' in session:
        g.user = User.query.filter_by(id=session['user_id']).first()


@oid.after_login
def new_user(resp):
    _steam_id_re = re.compile('steamcommunity.com/openid/id/(.*?)$')
    match = _steam_id_re.search(resp.identity_url)
    g.user = User.get_or_create(match.group(1))
    steamdata = get_user_info(g.user.steam_id)
    g.user.nickname = steamdata['personaname']
    db.session.commit()
    session['user_id'] = g.user.id
    # Add games to gorse
    games = get_owned_games(g.user.steam_id)
    data = [{'UserId': int(g.user.steam_id), 'ItemId': int(v['appid']), 'Feedback': float(v['playtime_forever'])} for v in games]
    headers = {"Content-Type": "application/json"}
    requests.put('http://127.0.0.1:8080/feedback', data=json.dumps(data), headers=headers)
    return redirect(oid.get_next_url())


def get_user_info(steam_id):
    options = {
        'key': app.secret_key,
        'steamids': steam_id
    }
    url = 'http://api.steampowered.com/ISteamUser/' \
          'GetPlayerSummaries/v0001/? %s' % urlencode(options)
    rv = json.load(urlopen(url))
    return rv['response'] ['players'] ['player'] [0] or {}


def get_owned_games(steam_id):
    options = {
        'key': app.secret_key,
        'steamid': steam_id,
        'format': 'json'
    }
    url = 'http://api.steampowered.com/IPlayerService/GetOwnedGames/v0001/?%s' % urlencode(options)
    rv = json.load(urlopen(url))
    return rv['response'] ['games']


# Create tables if not exists.
db.create_all()
Copy the code

Recommend to show

Then add recommendations to steamlens/app.py and use the RESTful API provided by Gorse to get popular games, random games, personalized recommendations, and similar games for a particular game.

# # # # # # # # # # # # # # # # # # # # # # #
# Recommender Service #
# # # # # # # # # # # # # # # # # # # # # # #

@app.context_processor
def inject_current_time(a):
    return {'current_time': datetime.utcnow()}


@app.route('/')
def index(a):
    return redirect('/pop')


@app.route('/pop')
def pop(a):
    # Get nickname
    nickname = None
    if g.user:
        nickname = g.user.nickname
    # Get items
    r = requests.get('%s/popular? number=%d' % (app.config['GORSE_API_URI'], app.config['GORSE_NUM_ITEMS']))
    items = [v['ItemId'] for v in r.json()]
    # Render page
    return render_template('page_gallery.jinja2', title='Popular Games', items=items, nickname=nickname)


@app.route('/random')
def random(a):
    # Get nickname
    nickname = None
    if g.user:
        nickname = g.user.nickname
    # Get items
    r = requests.get('%s/random? number=%d' % (app.config['GORSE_API_URI'], app.config['GORSE_NUM_ITEMS']))
    items = [v['ItemId'] for v in r.json()]
    # Render page
    return render_template('page_gallery.jinja2', title='Random Games', items=items, nickname=nickname)


@app.route('/recommend')
def recommend(a):
    # Check login
    if g.user is None:
        return render_template('page_gallery.jinja2', title='Please login first', items=[])
    # Get items
    r = requests.get('%s/recommends/%s? number=%s' %
                     (app.config['GORSE_API_URI'], g.user.steam_id, app.config['GORSE_NUM_ITEMS']))
    # Render page
    if r.status_code == 200:
        items = [v['ItemId'] for v in r.json()]
        return render_template('page_gallery.jinja2', title='Recommended Games', items=items, nickname=g.user.nickname)
    return render_template('page_gallery.jinja2', title='Generating Recommended Games... ', items=[], nickname=g.user.nickname)


@app.route('/item/<int:app_id>')
def item(app_id: int):
    # Get nickname
    nickname = None
    if g.user:
        nickname = g.user.nickname
    # Get items
    r = requests.get('%s/neighbors/%d? number=%d' %
                     (app.config['GORSE_API_URI'], app_id, app.config['GORSE_NUM_ITEMS']))
    items = [v['ItemId'] for v in r.json()]
    # Render page
    return render_template('page_app.jinja2', item_id=app_id, title='Similar Games', items=items, nickname=nickname)


@app.route('/user')
def user(a):
    # Check login
    if g.user is None:
        return render_template('page_gallery.jinja2', title='Please login first', items=[])
    # Get items
    r = requests.get('%s/user/%s' % (app.config['GORSE_API_URI'], g.user.steam_id))
    # Render page
    if r.status_code == 200:
        items = [v['ItemId'] for v in r.json()]
        return render_template('page_gallery.jinja2', title='Owned Games', items=items, nickname=g.user.nickname)
    return render_template('page_gallery.jinja2', title='Synchronizing Owned Games ... ', items=[], nickname=g.user.nickname)
Copy the code

Running server

We use uWSGI to start the Flask server, so we need to create a uwsgi.ini in the outermost folder SteamLens:

[uwsgi]

# Bind to the specified UNIX/TCP socket using default protocol
socket=0.0.0.0:5000

# Point to the main directory of the Web Site
chdir=/path/to/SteamLens/steamlens/

# Python startup file
wsgi-file=app.py

# The application variable of Python Flask Core Oject 
callable=app

# The maximum numbers of Processes
processes=1

# The maximum numbers of Threads
threads=2

# Set internal buffer size 
buffer-size=8192
Copy the code

Remember to change chdir to the SteamLens/ SteamLens path. Finally run the Flask application by executing the following command:

$ STEAMLENS_SETTINGS .. /config/steamlens.cfg uwsgi --ini uwsgi.iniCopy the code

Check out the online demo at steamlens.gorse. IO /, log in and wait for a while to generate personalized recommendations. Recommended results for the author are as follows:

I love the FPS genre, and it recommended a lot of FPS games to me. However, we can see that the recommended games are all older because the project uses data sets from around 2013, and with Steam’s updated privacy policy, it’s currently impossible to access user inventory without user authorization.