2. Overall structure of Idle fish search

Before the formal introduction of search stability guarantee scheme, we need to have a simple and general understanding of idle fish search technology; We have compared many external open source search engines, none of which perfectly support the requirements listed in the background; Idle fish use alibaba is the latest research and development of search engine platform Ha3 [[1]] (yq.aliyun.com/articles/58… , Ha3 is a very efficient, intelligent and powerful search engine, it fully meets the requirements of idle fish search; Elasticsearch[[2]](www.elastic.co/) is a quasi-real-time search engine based on Lucene, which is also a relatively common open source search engine. However, its ability of algorithm extension support/absolute real-time is far from that of Ha3. Ha3 is 4 times higher in QPS and 4 times lower in query latency than ElasticSearch [[3]](help.aliyun.com/document_de… ; The performance and stability of Elasticsearch in large-scale data scenarios are far from that of HA3.

2.1.1 Index Building process

Index building is what we call an offline process whose executors BuildService① are responsible for building plain text commodity data of different storage types into index files in search engine format. The original commodity data can be divided into two types. One is the full commodity data stored in storage, which is produced by DUMP② on a regular basis (usually every day), and the other is the real-time changed data. After the commodity information is changed, the business system synchronizes it to Swift③. Searcher④ updates the index that is eventually distributed to the online service.

2.2 Idle fish search system team composition

Idle fish search operation and maintenance system, is a very complex composition; It involves a lot of teams working together; First, there must be a Ha3 search engine team at the bottom to provide core search engine capability support; Mainly responsible for the construction and maintenance of the core capability of Ha3 search engine; Provide and maintain engine operation and maintenance platform and real-time engine search service. Then the algorithm team customized the Ha3 search engine to optimize the user’s search experience. The unstructured commodity of idle fish is understood by algorithm model, and the structured information is predicted and extracted for use in search engine commodity index. Monitor and maintain QP cluster services; Developed and used Ha3 engine sequencing plug-in to conduct bucket experiment of recall data and verify tuning. Finally, our business engineering team connected the whole search process to monitor and maintain the availability of the whole search link. Mainly maintains the data of search docking, manages the access of Ha3 search engine, arranges SP search services and makes reasonable query plans; And idle fish search unified online query service research and development and maintenance work. This paper also expounds how to guarantee the stability of complex search business system from the perspective of business engineering team.

3.1 Deployment Architecture Optimization

3.1.2 Multi-equipment Room Dr Deployment

At the beginning, the Ha3 search engine docking of xianyu commodity search service was only deployed in a machine room; When a serious problem occurs in the equipment room, the impact on upstream services is severe, or even a fault may occur. In view of this, the online and offline cluster of idle fish commodity search engine is deployed for disaster recovery. Before expanding in detail, we first have a general understanding of the principle of Ha3 engine DUMP process;



As shown in the figure above, the DUMP process of Ha3 engine can be divided into the following steps:

Prepare source data: assess business requirements and prepare data that needs to be connected to the engine; Most of the general service data are DB data tables, and there are a few ODPS⑨ offline data tables. Most of the data provided by the algorithm team are ODPS offline data tables; * DUMP data: By Ha3 engine team provides operations platform, can be some of the data field access these tables to create a good search engine, subsequent DUMP execution, Ha3 offline engine will pull the access table field data, the formation of a table of engines for image data, in this step, we can use the UDF tool for engine team, Data cleaning/filtering processing; * Data Merge: The engine joins all mirror table data with our specified primary key; Finally form a data wide table; Used by the engine to create indexes; After data Join in this step, the final data can be further cleaned/filtered through UDF, and the verified data will be entered into the large-width table. * Create updated index: the Ha3 offline engine will rebuild the index by buildService, using the large and wide table data, and align it with the index Schame that we specified in the Ha3 engine operation platform in advance;Copy the code

The above processes can be manually triggered by the Ha3 engine o&M platform. After the above processes are executed, a new index will be generated. After the new index cluster service becomes available, the online real-time module will switch the query service to the new index cluster to complete an index update. This whole process is what we call “full”; After the full volume is completed, when the system has a new commodity information change and the corresponding data table is enabled with real-time update (we call it incremental function,DB table is implemented through binlog/ODPS table and Swift message notification), the offline DUMP engine will sense the change and then update the commodity data in the corresponding mirror data table. The change information will be delivered to the upper layer of the engine according to the above steps in the offline DUMP process until the corresponding data in the engine index is successfully updated or discarded by the system rules during the process. This process of updating data in real time is called “incremental”; There is another channel for incremental update: Algorithm students can use a special way to directly update the data to be updated to the Ha3 engine index without passing the DUMP process through Swift incremental message. The commercial quantity of idle fish has increased rapidly, reaching billions at present. Hundreds of index fields are connected. Due to the unstructured nature of idle fish commodities, only a small part of the index fields are available for business use. In addition, most of them are indexes accessed by algorithms, such as a large number of extracted label data and vectorized data, which are very large. Finally, the DUMP processing logic of idle fish commodity search engine is complicated, and the total amount of index data is extremely large, and the amount of incremental messages is also at a very high level, coupled with the status of idle fish commodity inventory. Therefore, the real-time requirement of data update is very high, which brings great constraints to the stability. Index data is the core content of search engine. If the index data entering the engine has problems, or the newly changed data is not updated in the engine index, the quality of search service will be directly affected. During the single-server deployment of search engines, full DUMP fails, or incremental delay, or even stops due to some unstable factors. Once the engine DUMP problem occurs, it is very difficult to restore the engine. In many cases, it even needs to run the whole engine again to solve the problem. However, due to the large volume of idle fish commodity index data, it usually takes more than half a day to do a full volume, so there is no way to quickly stop bleeding, which has a great impact on business. Therefore, the SEARCH engine is deployed in two rooms (M/N rooms) for mutual backup. The two offline DUMP rooms use the same engine configuration and the same data source, and produce the same index data for the two online rooms. The online traffic ratio of the two rooms can also be adjusted in real time as required. When there is an irreversible problem in M machine room, the flow will be switched to N machine room automatically or manually to achieve on-line rapid hemostasis, and then the problem in M machine room will be solved step by step.

The following figure shows the final deployment of the search engine room.

     

Although the deployment of two engine rooms increases the cost of machine resources, it has the following benefits in addition to the above service Dr Advantages.

* Lack of effective grayscale process before the release of engine requirements; When there is a major change/upgrade of the search engine and a high-risk release occurs, beta test can be carried out in the small flow of the single room first and release to another room after data comparison and verification is passed. * Ordinary single room can support the flow of all search and query business, when there is a big promotion or large-scale activities, the two rooms will be mounted at the same time to provide services, so that the search service capacity and capacity can be doubled directly; It avoids the trouble of frequent expansion and shrinkage of single room; * During performance evaluation, pressure test can be carried out on the machine room without carrying traffic separately. Even if the machine room goes down due to pressure test, online services will not be affected;Copy the code

3.3 Hierarchical monitoring system

For high stability systems, it is particularly important to be able to sense problems when they occur or are about to occur. Convenient real-time tracking processing to prevent further expansion; At present, the main means is to establish and improve the multidimensional monitoring and alarm system;

3.3.2 Online service monitoring simulating user behavior

As mentioned above, the index volume of Xianyu search engine is relatively large, which requires the cooperation of many teams and the complexity of the search process is relatively high. Moreover, with the help of algorithm students, a lot of AI recognition has been done for unstructured products of idle fish. Moreover, idle fish are all single inventory products, which have high requirements for real-time performance of the engine. Some disaster protection schemes have been done in front; However, the perception of real-time need to be further, in order to timely know the accuracy of data, whether there is update delay, and how long the delay time and a series of health information; The solution is to monitor alarms in real time at the business level. Extract the idle fish are sold more update is more frequent category K, in the background of idle fish business system, through jkeins interval of time (time step can be real-time adjustment), use category K as the keyword and category, according to the commodity index update time descending recall, simulated users polling way send search queries, Recall the first page of products that meet the requirements; Then, compare the difference between the product update time of engine recall data and the current system time. If the time is greater than the threshold (which can be adjusted in real time), it indicates that there is a serious data update delay, and alarms are sent.

3.4.1 Full-link pressure test

The search service and the upstream business system of the whole link pressure measurement transformation; In addition, large quantities of pressure measurement data are constructed by using the real online user requests. Under the premise of not affecting the normal online services, the system capacity and resource allocation of the link under the super-large traffic model are verified, the performance bottleneck of the link is found, and the network device and cluster capacity are verified.

3.5 Grayscale Release

The unstructured characteristics of idle fish commodities are inseparable from algorithmic empowerment. In our research and development cycle, we maintained deep cooperation with two algorithm teams and quite a number of algorithm students. To idle fish search has brought a leapfrog development, but in the team collaboration and research and development efficiency also brought us great challenges.

Algorithm team, engine team, plus business engineering team, very large search project development team, every week there are a lot of new algorithm model, new engine transformation, new business modules need to be online.

A large number of new logic changes are put online directly, causing a lot of problems; First, at the code level, although the pre-release environment has been fully tested, it is difficult to ensure that there is no test omission of edge logic. Even if the pre-delivery test is completely covered, the online and pre-delivery environment is different, and the online heavy traffic environment may expose some hidden code problems; Second, if the code does not have any quality problem, but all the functions of binding, all logic are mixed together, how to assess the effect after a module on-line become great, especially the optimization algorithm model, and try on new business patterns, all need according to the effect of the detailed feedback data indicators to guide the optimization direction of the next step;

Therefore, there is an urgent need for a grayscale experimental guarantee system, which can not only be used to coordinate and isolate each module in the whole search business, but also to evaluate the effect of each module independently. And can also improve the efficiency of collaboration, so that each module can carry out quick trial and error, fast iteration;

              

In order to solve the above very important problems, the business engineering team developed a set of experimental management system for gray scale scheduling management of search experiments. The system functions are shown in the figure above. It has the following characteristics.

* The experiment is flexible and convenient, one experiment can contain multiple experimental components, one experimental component can be used for multiple experiments; An experimental component can contain multiple experimental buckets; * Experiments of each page module can be real-time regulated in the system, including experiment on/off; And the relationship between experiments; * Search the whole link of the buried point of the experiment, statistics of various experimental data reports; * The statistical data is connected to the Idle fish portal and the Babel tower, and the experimental curves of different buckets of each index can be viewed; * Improve experimental iteration speed, improve algorithm/business efficiency, fast trial and error, accelerate the growth of search transaction conversion;Copy the code

4. Summarize the outlook

Idle fish were introduced in this paper how to guarantee the stability of the search engine service under complex scene, mainly from the architecture deployment, isolation, capacity assessment, risk perception & control, etc, introduces how to stable support 20 + online search business scenarios, do a recovery quickly found problem come from line, efficient forecast risk aversion case 50 +, Greatly improve the user experience of the search service, to ensure that the idle fish search year without failure; After the above management scheme, the stability of idle fish search system has been greatly guaranteed. At the same time, we will continue to deepen our efforts to make the search ability more available and easier to use, so as to make the upstream business more smooth. I hope to bring you some thinking and inspiration.

6. Note