This article was first published on the wechat public account “Shopee Technical Team”.

Abstract

The development of terminal logistics in Southeast Asia is still in the initial stage of relying on manual labor due to various reasons, such as the diversity of languages, lack of corpus, weakness of GIS geographic information, etc., resulting in low efficiency, limited accuracy and limited speed of expansion. This sharing will introduce how Shopee realizes automation and intelligence of end logistics sorting in southeast Asian markets based on big data, artificial intelligence and other technologies, so as to support the best practices of Shopee’s rapid development.

In ArchSummit 2021 Shenzhen last week, Zewu, head of Shopee intelligent sorting team, shared Shopee’s experience in improving the efficiency of intelligent end logistics in Southeast Asia. This article is based on the content of the speech.

1. The pain of end logistics in Southeast Asia

Shopee is an e-commerce platform that operates in Singapore, Malaysia, Brazil and other markets. Shopee sold more than 2 billion items in this year’s 11.11 promotion. What are the specific problems of large orders delivered in Southeast Asia? How will Shopee solve and improve its efficiency?

1.1 background

Take a look at the business model of Shopee Express. Due to different markets and business types, the whole business flow is complicated. After the business flow is abstracted, the overall process can be divided into three parts: seller, Shopee express and buyer. Among them, sellers include cross-border sellers, local big sellers (usually with their own warehouses) and local small sellers (with smaller warehouses).

Shopee express is generally divided into three processes: first, middle and last.

The First Mile (FM) phase involves collecting and sorting goods from sellers by vehicle and routing them to a mid-range SOC. For small sellers, self-delivery service is also available in the first stage. After the seller delivers the goods to the self-delivery point, the staff collects them and sorts them at the Hub.

The intermediate stage is mainly to sort and pack the first delivery goods at the sorting center, so as to move to the next sorting center, or to enter the final stage.

After starting from the middle journey, the goods enter the Last Mile (LM) and arrive at the Hub (Last Mile Hub, LH). After unpacking, the goods are sorted according to the routing information and finally delivered to the driver for delivery. Of course, there are a variety of delivery modes, including home delivery, to pick up points, and some mobile sorting stations. In general, the scenario and business are quite complex. Next, we will mainly share the end logistics, which is the “last mile” of receiving and delivering goods.

Just after the Double 11, you can imagine that after placing an order in the mall, how the goods arrived at your door. What processes or operations does this process go through? It can be roughly divided into the following points:

• After the buyer places an order in the mall, the merchant receives the order, packages the goods, prints the sheet according to the address filled by the buyer, and waits for the driver to pick up the goods.
• The driver receives instructions to pick up the goods and send them to the First Mile Hub (FH for short); FH receives a lot of goods picked up by drivers, packages them according to the address, and then enters the intermediate route;
• After the package was delivered to LH in the middle distance, the package was unpacked and the driver was arranged to deliver the package.

There may be multiple FH/LH for a parcel, and there may be multiple drivers for a Hub for a parcel. For the overall package routing, how to choose LM is determined by multiple factors such as package timeliness and economic cost. For FH/LH, how to choose which driver to pick up/deliver goods is also determined by multiple factors such as parcel timeliness and economic cost.

An important question for the first and last trips is which driver on which Hub is assigned to pick up or deliver the goods? All we have is the address entered by the user, so the problem turns into how to match the most suitable Hub and the most suitable driver under the Hub according to the address text.

1.2 Status Analysis

Before optimization, we did some research and analysis on the service scope and activity track of some sites and drivers in a certain market.

The first chart delineates the delivery range of drivers. It can be seen that the delivery range of different drivers overlaps greatly.

The second picture shows the delivery points of drivers. In addition to the fact that many drivers’ delivery points are very similar, some drivers’ delivery points actually span a range of 8km.

After identifying this problem through technical means, we analyzed the business process. From the point of view of Hub, its service scope is divided according to administrative divisions. However, in some areas of Southeast Asia, the scope of administrative divisions is relatively extensive, and some complete standard administrative regions are only divided into State, City and District. As a result, the service scope of each site is relatively large, and the driver management in the Hub is also relatively difficult. The sorting operation is often conducted by experienced operators who check the detailed text of the address, sort piles according to their own experience, and then arrange the driver delivery according to the piles.

There are obvious problems with this:

• It takes about 10 seconds for each package to read the address text manually, which is inefficient in sorting;
• Relying on artificial and experienced knowledge, it is hard to avoid human errors and cognitive biases;
• Relying on experienced sorters, personnel training costs are high, especially to promote the temporary increase of sorters is relatively difficult;
• Because the driver delivery scope overlaps and the individual driver delivery scope spans greatly, the driver delivery efficiency is affected.

Adam Smith’s The Wealth of Nations mentions an important concept — division of labor. Any work which can be divided into two parts, if it is used, can increase correspondingly the productive forces. If the industrial and labor productivity of a country is very high, then the division of labor in various industries is usually also very high. This theory is undoubtedly also suitable for the field of logistics.

2. End logistics solutions in Southeast Asia

According to the theory of division of labor, to solve this problem, the general approach of the industry is to plan the service scope of Hub smaller, so as to improve the sorting efficiency of the site; At the same time, within the site, the service scope of the site is divided according to the dimensions of drivers, so as to improve the delivery efficiency of drivers.

2.1 Service Challenges

As mentioned above, the administrative divisions in many regions of Southeast Asia are extensive and even change frequently. It is difficult for us to set up stations or dispatch drivers according to the administrative divisions.

With only the user’s address text and delivery record, the idea is to start from the address. But what are the problems in Southeast Asia from the address?

Very direct thinking, is to take the address text to the map to search for the corresponding latitude and longitude, and then according to the Hub and the service scope of the driver to deliver.

The most prominent difficulty is that the address filling is not standard, especially the specific text written in dialect, due to the lack of special thesaurus, resulting in poor text matching effect. To take two examples, user address one: Jalan Petinggi Umar, Rt. 23, Depan Kantr Desa Loa Duri Ilir, Loa Janan. Jalan Petinggi Umar, RT.23, in front of the Loa Duri Ilir village office in Loa Janan. This address fails to match the valid address information (Jalan Petinggi Umar, Rt.23) using Google Map. On the contrary, auxiliary information (Depan Kantr Desa Loa Duri Ilir, Loa Janan) is used as the matching result, which has a large deviation, which will greatly affect order sorting and personnel scheduling for logistics delivery.

User Address 2: Jl Marsma R Iswahyudi RT 15 (Masuk 75 M Dari Jembatan Sungai Sepinggan Rumah Didepan Sungai Tingkat Warna Pink Biru). Translated into Chinese is Jl Marsma R Iswahyudi RT 15 (75M from the entrance of River Bridge Sepinggan House on the river, powder blue). The result of Google Map search for this address shows that it is in a chaotic state of information matching. The main information and auxiliary information are interlaced and difficult to distinguish, which makes it difficult to know what to do for parcel delivery.

In addition, there are other challenges:

• Multilingualism in the absence of a corpus;
• Extensive administrative divisions or changes, resulting in confusion of users’ cognition of the administrative region to which they belong and filling in the wrong address;
• In the absence of thesaurus, how to identify and correct the wrong address typed by the user?
• Since the implementation degree of SOP varies, how to judge the reliability of an address with a large number of historical addresses?
• When the service scope of the station or the service scope of the driver is gradually refined, how to solve the address locating failure caused by latitude and longitude deviation?
• How to screen and confirm the address with multiple longitude and latitude?

2.2 Service Architecture

Back to the beginning, let’s think about ways and means of solving the problem. It is generally divided into the following steps:

• Read address surface list;
• According to the existing knowledge, search the geographic location corresponding to the address text in the brain;
• According to the existing knowledge, according to the geographical location search and clear the corresponding service scope;
• Complete the sorting and arrange for delivery by drivers within this service area.

According to the above steps, our naive solution is shown below:

It mainly includes two parts:

(1) Off-line training

• According to the historical address order cleaning, forming the trusted address library;
• According to the verification data, a suitable matching model is obtained by training the trusted address base.

(2) Online reasoning

• When placing an order, the online address is obtained and preprocessed to form a standard address text.
• According to the offline training address base and matching model, online inference is carried out and inference results are obtained.
• Sorting and delivery are carried out according to the reasoning results.

It can be seen that the trusted address base and matching model obtained by off-line training are the basis.

In addition, in the face of these challenges, we record two facts and two advantages.

Two facts:

• Although there are multiple factors such as inconsistent SOP execution and address offset, the longitude and latitude collected at the time of receipt are relatively accurate in terms of probability.
• The address text with similar geographical position also has higher similarity.

Two advantages:

• With massive historical order data, naturally also has massive address data;
• Have a strong offline team, can verify and correct the address.

Therefore, based on these conditions, our business architecture is as follows:

There are two main sources of data: the historical address of mall orders and the address library of offline team contributions.

First of all, the address text is formatted and preprocessed to obtain the standard and uniform address text.

Secondly, as far as possible, the administrative division and other information is used to segment the formatted standard text to form the formatted address text. AOI is aggregated into the formatted address to form the AOI set. Meanwhile, address cleaning is carried out according to the policy to form the trusted address library.

Thirdly, based on AOI set and trusted address library, the training address library is used for supervised learning training, and the matching model is obtained.

2.3 Technical Architecture

According to the above business architecture, the system is naturally divided into two systems: online address inference service and offline training service.

Online reasoning service mainly consists of three parts:

• Address service: it mainly provides the address library and matching model that can be used. It uses various strategies to provide address service, including administrative division service, rule-based matching, similarity matching, keyword matching and local annotated data matching. Meanwhile, it also needs to score, compare and recommend the matching results. Obtaining this part is the core of online reasoning service;
• Sorting service: Provides OpenAPI services, including address sorting service and text segmentation service. Its customers are not only Shopee self-established Logistics, but also external 3PL (3rd Party Logistics, third-party Logistics);
• Operation platform: External Zone generation tool, local annotation data upload, system monitoring, policy configuration and other functions.

Offline training Service:

• It mainly conducts address training for various data sources (Shopee’s own data, map data, third-party data, etc.) in accordance with various rules, including address cleaning and AOI aggregation service, mainly including rule-based training, similarity-based training, keyword-based training;
• After training, the trusted address library and AOI set are formed and stored in the storage layer. In order to make the online address service optimize the address base and matching model over time, and to decouple the online inference service and the offline training service, the online inference service and the offline training service use message queue for data transmission.

After the technical architecture is determined, our technology selection is also determined according to the problems we face. As shown below:

2.4 Best Practices

2.4.1 Building a Trusted Address Library

The construction of trusted address database faces great challenges.

One is how to determine the credibility of the address?

• For the same text address, its latitude and longitude appear discrete phenomenon, how to pick out the point with low reliability? How to choose the degree of dispersion?
• We believe that the same address delivered many times, its credibility will be relatively high. How do you define multiple deliveries?
• For the same address, how to choose the reliability comparison between the latest delivery and multiple delivery?
• For different discrete points of the same address, how to choose the reliability between central node and random node?

A typical distribution is as follows:

Second, if the user enters the wrong address:

• In the case of lack of administrative divisions or confusion of administrative divisions, how to complete incomplete addresses entered by users?
• In the absence of administrative divisions or confusion of administrative divisions, how to identify and correct the wrong addresses entered by users?
• How to tell whether a user enters an incorrect address or a new one?

The third is the use of data marked by local personnel:

• To ensure its accuracy and coverage, how to select the address density threshold for different regions?
• How can local annotated data be effectively identified?
• How to use local annotated data efficiently?

Also, how to train multilingualism in the absence of a corpus?

To this end, we designed the cleaning process as shown in the figure below. The raw data goes through several steps, including pre-processing, cleaning engine, validating engine and output, before entering the trusted address library.

(1) Policy center: pre-processing in data cleaning. There are multiple policies for cleaning the engine. You can configure the cleaning engine through the configuration center, and then adjust the cleaning engine based on the feedback obtained by the configuration center

(2) Pre-processing: it mainly includes standardized format, text segmentation, batch processing, etc. Its algorithm and strategy can be completed through the configuration center. Output the format data required by the cleaning engine;

(3) Cleaning engine: there are mainly a variety of technical schools, including rules-based, deep learning-based, machine learning-based, etc. It is up to the configuration center to decide whether to use one or more;

(4) Validation engine: it mainly verifies the accuracy and coverage of cleaning results, outputs bad cases for analysis, and gives feedback to the configuration center according to the performance of accuracy and coverage;

(5) Trusted address library: Version management is performed for the address library whose output meets the requirements, including time dimension and region dimension, so as to update and roll back.

In this paper, for some area addresses, the distance-based cleaning policy is taken as an example, and the relevant thresholds are mainly tuned according to training, including distance thresholds and duplicate address thresholds.

AOI vs POI 2.4.2

Because the administrative division is large, the service scope of the station and driver needs to be much smaller than the minimum administrative division. So how do you form it? One way is to map zoning, the other way is to automate generation in some way. There are two grains, POI or AOI. That is to say, when we match the address, should we match the POI text or AOI text? POI is a point on the map, AOI is a region on the map.

POI and AOI generation have their own advantages and disadvantages. The following table analyzes the difficulty of collection, address matching accuracy, maintenance cost and guidance value respectively:

	AOI	POI
Collection Difficulty	Middle	High
Matching Performance	High	Middle
Maintenance Costs	Middle	High
Guidance Value	Middle	High

For our logistics scenario, it is not necessary to accurately match a point, but only to match the service range of the site and the service range of the driver, so AOI becomes a priority option.

According to the input address information extraction, can form some keywords, thus forming AOI. Different levels of AOI can be formed according to different extraction latitude. When extracting keywords, we use tFIDF, BM25, textRank and other methods.

At the same time, there is a problem of multiple languages and multiple models. Due to different administrative divisions and languages in each region, the application algorithm is different, which will lead to the problem of multi-model management and maintenance. At present we are managed separately, how to efficiently integrate is our ongoing attempt.

After obtaining AOI at different latitudes, aggregation of AOI levels can be carried out according to business needs:

• According to business needs, stations and drivers of different granularity can be generated. For example, in some places where ADO is relatively high, AOI of a lower level can be adopted, while in some places where ADO is relatively small, AOI of a higher level can be adopted.
• Changes to the service range of the station or driver can be easily adjusted through AOI or AOI aggregation latitude, which is much more efficient than POI latitude.

AOI itself also needs to know where it is on the map. How do you define it? We adopt the longitude and latitude aggregation of POI contained in AOI. Since the location of AOI does not need to be particularly precise, this approach can basically meet the requirements.

For manually delimited stations or driver service scope, how to form a match with AOI requires business intervention. For example, in the figure above, optimization means can be added to special zones. In addition to aggregation, the minimum distance point given by local data can also be added to verify the data with the aggregation center point, and the data will be matched successfully if it falls into the Zone at the same time.

Once confirmed, the Zone basically does not change, and there will be almost no further errors.

According to the above scheme, the existing service scope can not only be differentiated according to different drivers, but also can be divided into more details according to the single volume, so as to increase the division of labor and improve efficiency.

In addition, with an accurate division of the scope, you can go on some high-speed automation equipment to further improve efficiency.

3. Future outlook of end logistics in Southeast Asia

Shopee’s outlook for terminal logistics is mainly divided into four dimensions of “people, goods, goods and field”.

The first is the effectiveness and safety of personnel; The second is goods, such as the route of goods and the safety of goods; The third “thing” includes transportation safety, and the transportation efficiency of the vehicle itself; The fourth “field” is the energy efficiency of the site, including the utilization rate of crenel, the use of the site and the safety situation.

Around the above, Shopee will make more attempts with artificial intelligence technology.

In this paper, the author

Zewu, head of Shopee Intelligent Sorting team, from Shopee Supply Chain Express Service (SPX) team.

Join us

Shopee Express is our self-operated Express product, committed to providing high-quality Express services suitable for the local market in Southeast Asia. At present, it has been rolled out in many markets in Southeast Asia and Latin America, supporting the distribution of tens of millions of orders per day, which is one of the core capabilities of constructing Shopee’s efficient and low-cost supply chain services.

At present, the team continues to recruit a large number of positions, a large number of HC covers back-end, front-end, products, tests, algorithms, etc. Interested students can send their resumes to: [email protected] (also for consultation, indicated from Shopee technology blog).