Write an NPM package from scratch

preface

This article mainly records my experience of developing a NPM package: PIxiv-login, which interweaves with some daily development processes and skills, hoping to inspire the novice, big guys have a look.

pixiv-login

The function of pixiv-login is to simulate a user to login to the website pixiv and obtain cookies

The source code

npm

Installation:

npm install --save pixiv-login
Copy the code

Use:

const pixivLogin = require('pixiv-login'); PixivLogin ({username: 'your username ', password:' your password '}). Then ((cookie) => {console.log(cookie); }).catch((error) => { console.log(error); });Copy the code

The development tools

In daily development, I often use VScode + WebStorm +sublime, among which VScode is favored by most developers because of its fast startup, multiple functions and convenient debugging. In the rest of the tutorial, I’ll use vscode to demonstrate this. As for the terminal, since it is a Windows platform, I chose CMder instead of the native CMD, since cmder supports most Linux commands.

Initialize the project

mkdir pixiv-login
cd pixiv-login
npm init
Copy the code

Just drive back to the car

Install dependencies

To simulate login, we’re going to need an HTTP library, so I’m going to choose Axios, and we’re going to parse the HTML string that we get, and Cheerio is the first choice

npm i axios cheerio --save
Copy the code

debug

It has been several months since I started working after graduation. One of the most important skills I have learned is debug. In college, debug is console.log. Breakpoints are usually not used for tracing. I still remember when I saw my colleagues’ fancy DEBUG, I couldn’t help feeling in my heart: why are you so skilled?

Node debugging with vscode is very convenient

First create a new index.js file with the following project structure (my local NPM version is 5.x, so there will be an extra package-lock.json file, NPM 3.x does not have this file) :

Then click the fourth icon on the left to add the configuration

The configuration file is as follows:

{ // Use IntelliSense to learn about possible Node.js debug attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "Configurations ": [{"type": "node", "request": "launch", "name": "launch Program"," Program" : "${workspaceRoot}\\index.js" }] }Copy the code

The most important of these is “program”: “${workspaceRoot}\\index.js”, which indicates that when debugging, the project’s startup file is index.js

At this point, the debug configuration is complete.

Now write the index.js file

const axios = require('axios');
const cheerio = require('cheerio');

axios.get('https://www.pixiv.net')
    .then(function(response) {
        const $ = cheerio.load(response.data);
        const title = $('title').text();
        debugger;
        console.log(title);
    })
    .catch(function(error) {
        console.log(error);
    });
Copy the code

Press F5 to start debug mode, and if everything works, it looks like this:

As you can see, the program is stuck on line 8

If you hover your mouse over the response variable, you can see that vscode automatically displays the value of the variable, which is much cleaner than going directly to console.log(response)

To continue, press F5 or the green arrow in the upper right corner

When the program is finished, the console prints the title value of the pixiv home page

In addition to using the debugger statement break point, you can also directly click the line count break point of the code

For example, in the image above, I hit a breakpoint at line 8, which has the same effect

Another trick is that in Debug mode, you can change the value of a variable as much as you want. For example, if the program is stuck on line 8, you can change the value of title on the console

Press Enter and continue executing the code, at which point the console outputs the title value as ‘deepred’ rather than the actual title value

This technique is useful when you need to bypass some validation during normal development

The official start of the

Although we will eventually write an NPM package, we will first implement the function of getting cookies, and then think about how to package it into an NPM package for others to use.

Enter the login page login page, let’s log in once to see what data the front end sends to the back end

In particular, we need to check ‘Preserve log’ so that the HTTP request is still logged even if the page refresh jumps

As you can see, post_key is the key point for logging in, and p station uses this value to prevent CSRF

How do I get post_key?

After page analysis, it was found that there was a hidden form field in the login page (later it was found that it had been written on the home page) :

We can clearly see that post_key is already written, and we just need to use the Cheerio to parse the input value

const post_key = $('input[name="post_key"]').val();
Copy the code

Get post_key

const axios = require('axios');
const cheerio = require('cheerio');

const LOGIN_URL = 'https://accounts.pixiv.net/login?lang=zh&source=pc&view_type=page&ref=wwwtop_accounts_index';
const USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36';
const LOGIN_API = 'https://accounts.pixiv.net/api/login?lang=zh';


const getKey = axios({
    method: 'get',
    url: LOGIN_URL,
    headers: {
        'User-Agent': USER_AGENT
    }
}).then((response) => {
    const $ = cheerio.load(response.data);
    const post_key = $('input[name="post_key"]').val();
    const cookie = response.headers['set-cookie'].join('; ');
    if (post_key && cookie) {
        return { post_key, cookie };
    }
    return Promise.reject("no post_key");
}).catch((error) => {
    console.log(error);
});


getKey.then(({ post_key, cookie }) => {
    debugger;
})
Copy the code

F5 runs the code

Note: when opening the registration page, the registration page will return some cookies. These cookies also need to be sent along with the password and user name when logging in

Once we get the post_key, cookie, we can happily send the login data to the background interface

const querystring = require('querystring'); getKey.then(({ post_key, cookie }) => { axios({ method: 'post', url: LOGIN_API, headers: { 'User-Agent': USER_AGENT, 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Origin': 'https://accounts.pixiv.net', 'Referer': 'https://accounts.pixiv.net/login?lang=zh&source=pc&view_type=page&ref=wwwtop_accounts_index', 'X-Requested-With': 'XMLHttpRequest', 'Cookie': Cookie}, data: queryString.stringify ({pixiv_id: 'your username ', password:' Your password ', captcha: '', g_recaptcha_response: '', post_key: post_key, source: 'pc', ref: 'wwwtop_accounts_index', return_to: 'http://www.pixiv.net/' }) }).then((response) => { if (response.headers['set-cookie']) { const cookie = response.headers['set-cookie'].join(' ; '); debugger; } else { return Promise.reject(new Error("no cookie")) } }).catch((error) => { console.log(error); }); });Copy the code

Notice this code:

Data: queryString.stringify ({pixiv_id: 'your username ', password:' your password ', captcha: ', g_RECAPtcha_response: ', post_key: post_key, source: 'pc', ref: 'wwwtop_accounts_index', return_to: 'http://www.pixiv.net/' })Copy the code

There is a huge pit here, axios converts data to JSON by default, and if you want to send Application/X-www-form-urlencoded data, you need to use the QueryString module

For details, see: using- Applicationx-www-form-urlencoded -format

If all is well, the effect is as follows:

PHPSESSID and Device_token are the login ids returned by the server, indicating that the login is successful

While the program is running, you are likely to receive a login email from the P site

Okay, so far, we’ve managed to get cookies, which is pretty basic.

Pay special attention to

Do not run the program too many times, because each time you run, you log in to P station, if P station detects frequent login, it will turn on the captcha mode, in this case, you need to send user name and password, also need to send the background captcha value

Data: queryString. stringify({pixiv_id: 'your username ', password:' your password ', captcha: 'You also need to fill in a captcode ', g_RECAPtcha_response: '', post_key: post_key, source: 'pc', ref: 'wwwtop_accounts_index', return_to: 'http://www.pixiv.net/' })Copy the code

The CAPTCHA field is no longer null!

Complete code for basic functions

const axios = require('axios'); const cheerio = require('cheerio'); const querystring = require('querystring'); const LOGIN_URL = 'https://accounts.pixiv.net/login?lang=zh&source=pc&view_type=page&ref=wwwtop_accounts_index'; Const USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'; const LOGIN_API = 'https://accounts.pixiv.net/api/login?lang=zh'; const getKey = axios({ method: 'get', url: LOGIN_URL, headers: { 'User-Agent': USER_AGENT } }).then((response) => { const $ = cheerio.load(response.data); const post_key = $('input[name="post_key"]').val(); const cookie = response.headers['set-cookie'].join('; '); if (post_key && cookie) { return { post_key, cookie }; } return Promise.reject("no post_key"); }).catch((error) => { console.log(error); }); getKey.then(({ post_key, cookie }) => { axios({ method: 'post', url: LOGIN_API, headers: { 'User-Agent': USER_AGENT, 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Origin': 'https://accounts.pixiv.net', 'Referer': 'https://accounts.pixiv.net/login?lang=zh&source=pc&view_type=page&ref=wwwtop_accounts_index', 'X-Requested-With': 'XMLHttpRequest', 'Cookie': Cookie}, data: queryString.stringify ({pixiv_id: 'your username ', password:' Your password ', captcha: '', g_recaptcha_response: '', post_key: post_key, source: 'pc', ref: 'wwwtop_accounts_index', return_to: 'http://www.pixiv.net/' }) }).then((response) => { if (response.headers['set-cookie']) { const cookie = response.headers['set-cookie'].join(' ; '); console.log(cookie); } else { return Promise.reject(new Error("no cookie")); } }).catch((error) => { console.log(error); }); });Copy the code

Packaged as an NPM package

If we want to make it easy for other developers to call the function of logging in to P site to get cookies, we can consider packaging it as a NPM package and release it, which is also a contribution to the open source community.

First let’s recall what we did when we called other NPM packages.

const cheerio = require('cheerio');
const $ = cheerio.load(response.data);
Copy the code

Similarly, we now specify the use of piXiv-login:

const pixivLogin = require('pixiv-login'); PixivLogin ({username: 'your username ', password:' your password '}). Then ((cookie) => {console.log(cookie); }).catch((error) => { console.log(error); })Copy the code

Pixiv-login exposes a function that takes a configuration object that records the username and password

Now, let’s transform index.js

const pixivLogin = ({ username, password }) => {

};


module.exports = pixivLogin;
Copy the code

The basic skeleton is to define a function and then export that function

Since we need to support the Promise writing, the exported pixivLogin itself returns a Promise

const pixivLogin = ({ username, password }) => {
    return new Promise((resolve, reject) => {

    })
};
Copy the code

After that, you just plug in the original code

Complete code:

const axios = require('axios'); const cheerio = require('cheerio'); const querystring = require('querystring'); const LOGIN_URL = 'https://accounts.pixiv.net/login?lang=zh&source=pc&view_type=page&ref=wwwtop_accounts_index'; Const USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'; const LOGIN_API = 'https://accounts.pixiv.net/api/login?lang=zh'; const pixivLogin = ({ username, password }) => { return new Promise((resolve, reject) => { const getKey = axios({ method: 'get', url: LOGIN_URL, headers: { 'User-Agent': USER_AGENT } }).then((response) => { const $ = cheerio.load(response.data); const post_key = $('input[name="post_key"]').val(); const cookie = response.headers['set-cookie'].join('; '); if (post_key && cookie) { return { post_key, cookie }; } reject(new Error('no post_key')); }).catch((error) => { reject(error); }); getKey.then(({ post_key, cookie }) => { axios({ method: 'post', url: LOGIN_API, headers: { 'User-Agent': USER_AGENT, 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8', 'Origin': 'https://accounts.pixiv.net', 'Referer': 'https://accounts.pixiv.net/login?lang=zh&source=pc&view_type=page&ref=wwwtop_accounts_index', 'X-Requested-With': 'XMLHttpRequest', 'Cookie': cookie }, data: querystring.stringify({ pixiv_id: username, password: password, captcha: '', g_recaptcha_response: '', post_key: post_key, source: 'pc', ref: 'wwwtop_accounts_index', return_to: 'http://www.pixiv.net/' }) }).then((response) => { if (response.headers['set-cookie']) { const cookie = response.headers['set-cookie'].join(' ; '); resolve(cookie); } else { reject(new Error('no cookie')); } }).catch((error) => { reject(error); }); }); }) } module.exports = pixivLogin;Copy the code

Release NPM package

README

Each NPM package usually comes with an introductory paragraph telling the user how to install it. For example, the lodash home page

Create a readme.md and fill in the relevant information

Sometimes, we see some NPM packages with nice version number ICONS:

These ICONS can actually be made at https://shields.io/

Log on to the site and scroll down to the bottom

Enter the text you want, version number, color, and click the button

You can get the access address of the image

Modify the readme. md and add our version number.

gitignore

Our current folder directory should look like this:

Node_modules and.vscode don’t need to be uploaded at all, so we’ll create a.gitignore to avoid Posting with these folders

.vscode/
node_modules/
Copy the code

registered

Then enter it at the terminal

npm adduser
Copy the code

Enter the user name, password, email can login successfully

Here’s another pit!

If your NPM is using taobao mirror, it is unable to land successfully

The simplest solution:

npm i nrm -g
nrm use npm
Copy the code

NRM is an NPM image management tool that can easily switch image sources

After a successful login, enter

npm whoami
Copy the code

If your username appears, you have logged in successfully

release

Special attention:

Since the name piXiv-login is already occupied by me, you need to change it to another name

Modify the name field in pacakge.json

npm publish
Copy the code

You can publish successfully!

download

After a successful launch, we were able to download our own packages

npm i pixiv-login
Copy the code

Use pixiv – login package

We can do some interesting things with PiXiv-Login

Such as:

Download images of the R-18 weekly leaderboards

Users who are not logged in cannot access R18, so we need to simulate logging in

const fs = require('fs'); const axios = require('axios'); const pixivLogin = require('pixiv-login'); const cheerio = require('cheerio'); Const USER_AGENT = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36'; PixivLogin ({username: 'your username ', password: Fs.writefilesync ('cookie.txt', cookie); fs.writefilesync ('cookie.txt', cookie); }).then((response) => { const cookie = fs.readFileSync('cookie.txt', 'utf8'); axios({ method: 'get', url: 'https://www.pixiv.net/ranking.php?mode=weekly_r18', headers: { 'User-Agent': USER_AGENT, 'Referer': 'https://www.pixiv.net', 'Cookie': cookie }, }) .then(function(response) { const $ = cheerio.load(response.data); const src = $('#1 img').data('src'); return src; }).then(function(response) { axios({ method: 'get', url: response, responseType: 'stream' }) .then(function(response) { const url = response.config.url; const fileName = url.substring(url.lastIndexOf('/') + 1); Response.data.pipe (fs.createWritestream (fileName)).on('close', function() {console.log(' ${fileName} download completed '); });; }); })})Copy the code

Also, our piXiv-Login supports async await!

Const pixivStart = async() => {try {const cookie = await pixivLogin({username: 'your username ', password:' your password '}); fs.writeFileSync('cookie.txt', cookie); const data = fs.readFileSync('cookie.txt', 'utf8'); const response = await axios({ method: 'get', url: 'https://www.pixiv.net/ranking.php?mode=weekly_r18', headers: { 'User-Agent': USER_AGENT, 'Referer': 'https://www.pixiv.net', 'Cookie': cookie }, }); const $ = cheerio.load(response.data); const src = $('#1 img').data('src'); const pic = await axios({ method: 'get', url: src, responseType: 'stream' }); const fileName = pic.config.url.substring(pic.config.url.lastIndexOf('/') + 1); Pic.data.pipe (fs.createWritestream (fileName)).on('close', function() {console.log(' ${fileName} download completed '); });; } catch (err) { console.log(err) } }; pixivStart();Copy the code

reference

1. Log in to pixiv.net

2. Getting started with a Python crawler: Crawl pixiv

Blog Address: