preface

Due to business requirements and some restrictions, we had to build a short link system of wheels. There is currently an open source solution for this type of system, YOURLS, but this system is a PHP version, considering the cost of maintenance, we decided to develop our own.

Current business requirements include:

  1. Long links are converted to short links
  2. Custom short links
  3. Short link traffic statistics

technology

Back end: egg.js + Oracle + redis

Front-end: None (the system directly outputs API, and links are created in other systems)

implementation

Long links are converted to short links

This is the basic requirement of short link system. Considering specific business requirements, long link and short link are not one-to-one corresponding relationship, that is, the same long link will generate different short link each time conversion.

Each link in the database has five key fields:

  1. Short link prefix (we currently have more than one domain name)

  2. Short link ID (increment)

  3. Custom link (user fill in)

  4. Hash (converted by ID)

  5. PV(Link volume)

  6. STATUS (link STATUS, enabled/disabled)

Create an incremented ID in the Oracle database, assign an ID to each long link, and hash the ID from base 10 to base 62. In this case, you need to check whether the hash value conflicts with the existing user-defined link in the database because the user-defined link exists. If a conflict occurs, the ID continues to increment and the hash is recalculated until it no longer conflicts.

Note:

  • The order 0-9A-ZA-Z can be scrambled if secrecy is considered.

  • If you consider that 0, O, O, 1, and I are not distinguishable, you can leave these letters out.

  • If it is not a commercial service, it is possible to use 6 – or 8-bit alphanumeric random encoding.

Custom short links

The problem of user-defined short links is that the user-defined links conflict with the existing hash.

Solution steps:

  1. First, obtain the numeric ID of the current link, convert it to base 62, compare the hash with the user-defined link, and check whether the hash conflicts with the existing user-defined link in the database.

  2. Search the database to see if the user-defined link conflicts with an existing hash. If it does, the creation failure is returned.

Use Redis to cache links

Although custom links do not conflict with hash and can be used as unique ids, hash is still used as the identification of links in consideration of the uniqueness and accuracy of later data statistics.

Create hash table URL_lIST in Redis, store all data of visited links in Redis through hset, create hash table PERSONALIZED_URL_LIST, store custom links and corresponding hash, use REDis INCR to create a counter for each link. Used for PV statistics.

When the user accesses the link http://xxx.xxx/abc, the user first queries the URL_LIST for the hash value ABC (case 1), and if not, the PERSONALIZED_URL_LIST for the personalized link with the hash value ABC (case 2), If not, query the data directly to the database.

For case 2, if the data is found, then query the corresponding data in URL_LIST.

Due to the complexity of the real world, there may be cases where the data is present in PERSONALIZED_URL_LIST but not in URL_LIST, and you need to query it in the database. However, these are synchronous operation sequence, in principle, do not need to be handled separately.

Short link data statistics

At the beginning, considering the need for data visualization in the future, I hoped to go to ELK directly to solve the problem of data statistics as well as data visualization. However, due to ELK’s limited knowledge, logs are seriously lost after being transmitted to ELK, and this scheme is temporarily shelved.

Because the short link system has multi-node and multi-thread, it is necessary to build a short link statistical system independent of the short link system, the main work is to complete the scheduled task of data statistics.

As mentioned earlier, redis incR creates a counter for each link that maintains the incremental PV of the current link.

The task is to periodically obtain the links in URL_LSIT and search for the PV increment corresponding to the links. If the PV increment is greater than zero, add the PV value of the links in URL_LIST to the PV increment, set the PV increment to zero, and update the URL_LSIT and the data in the database.

TODO

  • Data visualization based on ELK
  • Links to disable and enable controls
  • .

Short link system based on Node.js – Listen to sea Pavilion