In order to find a job, Touch fish sauce here to share a set of their own from 0 to 0.1 development of the auxiliary stock trading system

Touch fish sauce article statement: content guarantee original, pure technology dry goods to share exchanges, do not play advertising do not brag force.

Preface: digging friends good, nearly a year did not update digging gold article, I do not know whether someone remember me touch fish sauce. Between a short year, touch fish sauce from join ali to work hard to contact fry leave ali full-time fry again, from laughing to doubt life, everything is so fast so suddenly, as if a dream, a dream ~.

Without further ado, let’s get to the point. For the writing logic of this auxiliary stock trading system, I will not pay too much attention to the discussion of system technology stack and technical details, but will focus more on the needs to be solved by the system itself and the problems faced, as shown in the following brain diagram:

For the rest of this article, I will focus on this brain diagram. If you are interested in the following content or the source code implementation, I hope you will spend a lot of time with this diagram, because it will be very helpful to your understanding. Hopefully, if you can make some comments or suggestions about the system based on your programming or stock trading experience, so much the better.

Well, following the logic of the brain map above, I will explore this article in the following sections.

• Pulling and Modeling of transaction data (TUShare)
• How the program helps the transaction (service-core)
• Fast mass data service
• Service – Notify

Ok, now that we have cleared up the idea of the article, let’s move on to the first part, transaction data extraction and Modeling (Tushare).

I. Pull and Modeling of Transaction Data (TUShare)

If you have ever wanted to do their own quantification or as I write some of the auxiliary stock trading tools, then you must encounter the first question is certainly trading data from where. I believe that many people may have the same idea as me at first, which is to write crawler script to crawl the data of financial websites (such as Sina, Oriental Wealth, Tiandi Fund, etc.) and store it locally. But there are many problems, such as:

• Trouble in finding data: some data are difficult or even impossible to find crawl channels, such as multiple weighting factors, flush concepts and so on
• Face to face anti-creep mechanism: for example, limit the number of access times or directly block IP addresses
• Parsing modeling is cumbersome: for example, with the odd interface response format, parsing modeling the data in the DOM is even more troublesome
• Data reliability: for example, is the data accurate, when is the update time

I experienced all of these problems while writing the system’s predecessor, a script to pick ETFs, and it was annoying Yuppie. After that, I surveyed several communities and platforms that support quantification (various Quants) and found four main forms:

• The first is a graphical operation completion strategy
• The second is to provide a platform for code to be written and run (mostly in Python).
• The third is an SDK that provides a data API (mostly Python only)
• The last option is to provide data directly through HTTP (such as tushare)

The quantitative platform is guokernel network, which belongs to the first kind. It is characterized by no need to write code, tick boxes can complete a strategy, which is very convenient for those who like technical analysis of stocks to test back. Ps: A lot of them have to pay, I just to try out the function, he meow 300 membership (can you give me the money back ~)

Obviously for us front-end dogs to write their own programs, there is only one of the last. There are probably a few other platforms that provide transaction data like Tushare, but I’ve never seen a platform as well documented and maintained as Tushare.

Ps: The access to some advanced data and high-frequency data in Tushare also needs to be charged. Personally, ABOUT 1500 yuan is charged.

Ok, let me share how I used the Tushare platform to pull data and model data in the Share-Help system.

1. Program pull data problem

For data pulling, with the help of tushare platform, we only need to find the corresponding data interface description in tushare official documents, and then send HTTP request. For specific implementation, we take the daily data of A-share stock as an example:

• Tushare pull in a-share stock line data interface documentation: www.tushare.pro/document/2?…
• Tushare a-share stock date line in the data interface online test: www.tushare.pro/document/2?… , you can also send it directly via Postman.
• Share-help (ts + node) pull data:

The following code may seem long, but there are only two aspects of logic involved: 1 is the function that encapsulates the request to fetch data (getDaily), 2 is the test function logic (test).

import axios from 'axios';
import { tushare_token } from '.. /.. /tushare.config';
// Interface: daily, you can use data tools to debug and view data.
// Data description: Trading day between 15:00 ~ 16:00. This interface is not reply to the market, do not provide data during suspension.
// Description: get stock ticker data, or through the general ticker interface to get data, including before and after weighting data.
const api_name = 'daily';

export typeDailyParams = { ts_code? :string; // Stock code (support multiple stock extraction at the same time, comma separated)trade_date? :string; // Transaction date (YYYYMMDD)start_date? :string; // Starting date (YYYYMMDD)end_date? :string; // End date (YYYYMMDD)
};

export typeDailyResult = { ts_code? :string; // Ticker symboltrade_date? :string; // Transaction dateopen? :number; / / the openinghigh? :number; / / the highestlow? :number; / / the lowest priceclose? :number; / / closepre_close? :number; / / a closing price yesterdaychange? :number; / / or foreheadpct_chg? :number; // up or down (not right, if it is right, please use general market interface)vol? :number; // Volume (hand)amount? :number; // Transaction amount (thousand yuan)
};

export type HsConstField =
  | 'ts_code'
  | 'trade_date'
  | 'open	'
  | 'high'
  | 'low'
  | 'close'
  | 'pre_close'
  | 'change'
  | 'pct_chg'
  | 'vol'
  | 'amount';

export const getDaily = async (
  token: string, params? : DailyParams, fields? : HsConstField[] |string,
): Promise<DailyResult[]> | null= > {
  let result = null;
  if (Array.isArray(fields)) {
    fields = fields.join(', ');
  }
  const reqBody = {
    api_name,
    token,
    params: params ? params : {},
    fields: fields,
  };
  await axios
    .post('http://api.waditu.com', reqBody)
    .then((res) = > {
      const { code, msg, data } = res.data;
      if(code ! = =0) {
        console.log('reqError', msg, reqBody);
        result = null;
      }
      const { fields, items } = data;
      const mapResult = items.map((itemArr) = > {
        const obj = {};
        for (let i = 0; i < itemArr.length; i++) {
          obj[fields[i]] = itemArr[i];
        }
        return obj;
      });
      result = mapResult;
    })
    .catch((error) = > {
      // console.log('reqBody', reqBody);
      // console.log(error);
      result = null;
    });
  return result;
};

async function test() {
  const data = await getDaily(tushare_token, {
    ts_code: '600132.SH'.trade_date: '20211027'});console.log(data);
}

// test();

Copy the code

The above code is a sample of the share-Help pull transaction data. In addition to exposing the pull data request method and its parameters and the TS type of the returned data for external use, the code also provides a test method to test the function logic (using TS-Node to run the specified file when testing separately). In Share-Help, each interface request file maintains the request test logic for only one interface and is one-to-one with the interface documentation in Tushare.

If there are interface updates or additions to Tushare, you can use the build script in tushare/help-script to quickly generate these files.

So much for the sharing of pull transaction data in share-help, let’s discuss data modeling based on Tushare.

2. Transaction data modeling

You may not understand what I mean by transaction data modeling. For example, every stock will generate a lot of data every day, such as price, rise and fall, trading volume, re-option factor, market value, turnover rate and many other factors. So how should we reasonably define, summarize and combine these factors to make the program run more logically and efficiently? In the previous stage when I used crawler to crawl trading data, the modeling was not reasonable due to my experience in stock trading and programming, which led to several database adjustments and modification of strategy code, which was a waste of time.

As we all know, the design of the underlying data structure is critical to a program. Once the underlying data structure changes, it is easy to bring about database reconstruction and significant changes in the program.

The solution of this headache problem is my biggest achievement after introducing Tushare. For example, the factors of the data generated by each stock every day mentioned above are reasonably distributed in three tables, daily, dailyBasic and Adj_factor, in Tushare.

Just like the dailyResult object exposed in the daily interface request file above, it establishes the transmission structure for daily quotation data in the entire Shares-Help. That is, the Dtos for the transaction data in the Shares-Help program are defined in each interface request file.

The sharing of Shares-Help for trading data pulling and modeling problems is now over. Let’s move to the core part. In what ways can Shares-Help help trading?

2. How the program helps the transaction (service-core)

From my understanding, the ideal programmed trading link should be like this: automatic analysis of the market => automatic selection of stocks => automatic buying and selling of stocks => automatic backtest returns => notification of trading results, commonly known as automatic trading, just like a ruthless printing money machine ~

Unfortunately, the current Share-Help cannot realize intra-day trading data analysis and automatic ordering functions, which is due to the fact that it cannot obtain minute-level data and has not introduced the trading interface open to securities firms. After all, we are only 0.1 now, maybe a later version will be implemented.

Given the trading mechanism and concept hype style of A shares T+1, minute-level data and automatic ordering functions are not as important.

Since intra-day data cannot be obtained and automatic orders cannot be placed, we think about what capabilities Share-Help should have at present according to the above trading link:

• After the market close, some data statistics and analysis can be conducted on the current day and the previous market to understand the market overview and think about the operation strategy of the next trading day
• After the close, get the day trading data to execute stock selection strategy, for the next day
• Simulate trading, using daily line data to calculate returns
• Close back test day, or back test the historical trading day to determine whether the stock selection strategy and operation strategy is reasonable

These program logic are divided into four modules in the micro-service core-service, which correspond to the above four requirements as follows:

• Shares-analysis: market analysis
• Shares-select: indicates the selection of an object
• Share-operate: simulates trading
• Shares-backtest: indicates the policy backTest

Uml class diagram is used to describe the implementation and dependence of these modules in the system as shown below:

I’ll continue with examples of how to do all four of these things in Share-Help, but it should be noted in advance that the examples and strategies I’ll use will be fairly shallow. This is limited by my stock trading experience, I do not know what is effective to analyze the market, and there is no good stock selection and operation strategy. So try to focus on the share-Help structure rather than the strategy

The most confused time is not do not know how to do, but do not know what to do ~

1. Share-analysis

The market analysis module is designed to be used at will. If you have ideas, you can analyze the market, foreign capital, individual stocks and anything else here. For example, the examples in Share-Analysis come from the following ideas:

Assume that our operating idea is to buy a stock at the end of the day, the next day in the position of the order is 1, so as long as this stock in a day of shock, as long as the price of more than 1 will be clinked and profit, the amplitude between no more than 1 will be lost. In order to improve the profit probability, we choose, to choose the next day to 1 point on the amplitude of individual stocks, use the program to test the selected stocks rose more than 1% of the stocks in the actual amplitude of selected stock proportion, the greater the ratio means that the profit the greater the probability (big doesn’t mean that profit, profit probability because odds small).

We are not going to argue with the logic of this example. As I mentioned earlier, these examples are both crude and arbitrary, so let’s introduce the basic idea of implementation with uml class diagrams (see share-help source code for more details) :

• Step1: create a new AmplitudeAnalysis class in shares-analysis/stock
• Step2: Write the logic to pull the stock pool in getStockPool
• Step3: Write the stock selection logic in poolFilter to screen the selected stocks
• Step4: Write stock filters with amplitude increases of more than 1% in stockFilter
• Step5: Call these methods in sequence in the program entry start, and use the quantity filtered by stockFilter/the quantity filtered by poolFilter to get the probability we require.

I will briefly introduce the member attributes in the uml class diagram above. The member attribute tushare is used to quickly retrieve the required data, notifyClient is used to quickly send notifications such as emails, and sharesAnalusisUtil is used to call some tool methods.

The member property Tushare can quickly pull data thanks to the services provided by the service-data module, and notifyClient can quickly send notifications thanks to the services provided by the service-notify module. These two modules will be shared separately later.

Ok, the example is so simple to share the target analysis to this, interested in the details of the implementation of friends can look at the share-filter source code. If you have some crazy idea that you want to test back, you can also write code based on Shares-Filter to quickly verify it.

Now let’s enter the target selection module ~

2. Share-select

The module of target selection is full of target selection strategies, such as selecting stocks, funds, convertible bonds, digital currencies and so on. Share-select has temporarily compiled an example of A share leading plate chasing strategy, which comes from the following considerations:

Assuming that we are A short-term a-share players, as far as my cognition of stock trading is concerned, the short-term A-share is mainly to follow the short-term capital dominated by hot money to hype the concept, which is A game of licking blood on the tip of A knife, high risk and high return. This game is usually initiated by the big money, and when we, the belated small investors, discover it, it is often the end of the game. Obviously, it’s a long shot, long shot gamble, like a piece of cheese in a mousetrap, designed to kill us smart or lucky short-term mice.

For this unfair game, it must be a very good choice to hold your hands and not be a rat. But his meow short-term cheese is delicious, and there is no time cost, I really want to eat, ha ha, the so-called once entering the stock market is like the sea, from then on, friends and relatives are passers-by is also not unreasonable, most of the addicted to the stock market are incurable.

I have been thinking and trying to be a smart mouse for nearly half a year since I left Ali to speculate in the stock market. Is there any success? Ha ha, when you see this article looking for a job, it is the verdict that I am a loser. I, did not find the answer. Of course, this half year is not nothing, I learned some investment books, tried to practice various types of stock speculation strategy, such as technical analysis of stock speculation, fundamental speculation, news speculation, sentiment speculation and so on.

By the way, here I mention a mouth quantitative fry, as far as MY cognition of AI is concerned, quantitative fry essentially belongs to technical analysis fry, and MACD, KDJ these traditional methods are different is that AI quantitative factors are more, the model is more comprehensive. All technical analysis stock options have a fatal problem, that is to look in the rearview mirror. Only if the road is straight can the technical analysis not tip over. In general, as far as I understand, quantification, as a technical analysis scheme of the new era, has advantages compared with traditional technical analysis scheme in the case of medium and long term or stir-band. In short-term scenarios, it is feasible to quantify capital and emotion in combination with large capital advantages.

All right, let’s get back to the short term stock picking example. It goes like this:

• Look for the sector effect: the leading sector of the day before the flush is found (with the biggest increase of more than 5 points)
• Look for strong stocks in the plate: find the previous day under the plate all stocks trading limit (PS: the reason is not only looking for the leading, that is because the leading probability can not buy)

Also, I will share the implementation ideas with the following UML class diagram:

• Step1: create a LimitThsSelect class in shares-select/ THS
• Step2: Implement the logic of pulling all flush plates in getSharesPool of LimitThsSelect class
• Step3: In the LimitThsSelect class thsFilter method to find the first plate of the previous day’s gain logic
• Step4: Implement the LimitThsSelect class start method for controller to use directly, implement the start_core method for the following stock selection logic use.
• Step5: Create a LimitStockSelect class in shares-Select /stock and inherit the BaseStockSelect abstract class
• Step6: in the LimitStockSelect class getStockPool method to achieve the need to pull the stock pool, such as as long as the main board or as long as the stock below 50 billion
• Step7: implement the logic to find LimitStockSelect stocks in stockFilter1 of LimitStockSelect class

After the above steps, we can realize our idea of picking stocks, if you feel this stock selection method for trading in its real help, you can also in shares – select/schedule. Select the ts implementation after the close of regular run strategy for yourself every day in the overload of the next day, the code is very simple and can be as follows:

@Injectable(a)export default class ScheduleSelect {
  constructor(
    public readonly tushareData: TushareData,
    public readonly limitStockSelect: LimitStockSelect,
    @Inject('SERVICE_NOTIFY') public readonly notifyClient: ClientProxy,
  ) {}

  @Cron('0 30 20 * * 1-5') // Monday to Friday at 8:30 PM push stock selection of the day
  async dailyNotify() {
    const tradeCal = await this.tushareData.getTradeCal();
    const nowDate = getNowDate();
    const afterDay = addDay(nowDate);
    const isTradeDay = tradeCal[nowDate].is_open === 1;
    if (isTradeDay) {
      const adviseCodes = await this.limitStockSelect.start_core(afterDay);
      await this.notifyClient.emit('stock_select', adviseCodes); }}}Copy the code

Ok, this is the end of sharing how to complete the stock selection strategy in the share-Help. Next, we enter the third module, which simulates buying and selling operations in the share-help.

3. Share-operate

Trade in this module are some of the business strategy in the reality in the abstract, here you can write some short-term trading strategies such as do, calculate a choose a strategy first day offer to buy until the next day in the morning and evening how many proceeds from the sales of plate in the morning and evening, or when doing band set under the condition of a certain stock check of a stop loss: how long will it take a look at the average position. That is to say, this module can be used directly as a profit or position length calculator, and can serve for the back test module.

Share-operate currently has an example of A short term tag operation strategy that can be used to calculate the returns of the a-share leading sector tag selection strategy described above. Its specific buying and selling logic is as follows:

• Buy: equal position distribution, buy at the opening price of the day
• Sell: buy the day break open to sell, constantly board in the intraday to the day’s lowest price and the highest price of the compromise to sell

Also, I will share the implementation ideas with the following UML class diagram:

• Step1: create a LimitStockOperate class under shares-operate/stock to inherit BaseStockOperate
• Step2: Implement the buy logic in getBuyInfo of the LimitStockOperate class
• Step3: Implement the sell logic in getBuyInfo of the LimitStockOperate class
• Step4: Implement holding logic in getHoldInfo of the LimitStockOperate class

After the above steps, we can achieve their own calculation of short-term chasing board operation stock selection strategy. As for the development of other buying and selling strategies here is not much to do, I will quickly enter the strategy back test module to share.

4. Share-backtest

Policy back to the sensor module can be used to measure the income or earnings history, back and forth in the execution procedures stock selection and sales strategy, to measure the income can be judged whether their actual operation and application expectations have deviation, if there is deviation, consider whether their emotions not put control in place, isn’t it too greedy or too unintelligent. Backtesting historical returns is often used to verify whether one’s stock selection and strategy has stood the test of history.

There is currently an example of short-term stock selection and supporting operation strategies in share-help. Their stock selection and buying and selling strategies will not be introduced repeatedly here. After the code of stock selection and operation strategy has been written, the back test is very simple.

• Call the stock selection strategy to screen out stocks
• Get the selected stock into the operation strategy, calculate the operation results
• Get the results of the operation and tally the profits

The single-day back-test logic is completed, and the multi-day back-test logic is simply loops and statistics. Of course, you can make it more complicated, but the example is that simple

What is worth mentioning here is the design and implementation of the dependencies between stock selection, operation and backtest. Let’s take a look at the UML diagram:

In the previous version of the design, the above three classes were combined into a strategy class, a strategy contains all the logic for stock selection, operation and backtesting, the disadvantage is that the logic is coupled together, poor scalability and very bloated code. Therefore, in the current version of the design, after the re-sorting of the trading link, I used the idea of adapter mode to reconstruct, stock selection strategy object and operation strategy object as the member attributes of the back test object.

If you are interested in the source code, you can directly visit github.com/iamjwe/shar… . In share-help, the code of service-core can be written very briefly, largely because the logic of pulling data and notifying are encapsulated in service-data and service-notify microservices respectively. Let me start with the sharing of service-data services and explain the problems encountered in service-data and the solutions.

If you see this, please give it a thumbs up, Aligardo

Three: Fast and large amount of service-data

What shares-Help does well is decouple the data processing logic from the policy logic as a microservice. The biggest benefit of this is to separate the two concerns and greatly reduce the complexity of the application. The above has shared how to write strategies in share-Help. Now, before sharing the data service module of Share-Help, let’s briefly think about what needs to be done by a data service that can make users feel comfortable and worry free for a system assisting stock trading. I think it should include the following four points:

• Accuracy: the transaction data returned under the request condition must be accurate. Secondly, whether the data is empty and whether the request fails must be handled reasonably.
• Large amount: In historical backtest scenarios, a large amount of data needs to be provided at once. Therefore, high-frequency data retrieval from third-party platforms is bound to be limited. This problem must be solved.
• Efficiency: Speed is constantly optimized, frequently accessed data should be cached, and frequently used data should be pre-cached at program startup.
• Ready-to-eat: Some data that need to be processed need to be processed in this link, such as the re-weighting of stock prices. Secondly, the external response data should be encapsulated into a data structure that supports reasonable and efficient access.

Of these four goals, accuracy and immediacy can be better addressed by adding response logic to the program, which will not be discussed here. Below, I will share how to solve the two problems of large amount of pull data and fast response data in share-Help. The following figure is the code file structure of solving these two problems in service-data module:

1. Massive data pulling problem

Tushare platform will be limited by the high frequency pull data, most of the interfaces are limited to 500 800 times per minute, and the number of A-shares is more than 4000. It is obvious that there is no way around this limitation. It is quite natural for you to think of self-built database, whether you use a local computer or buy a cloud server. Personally, I choose to buy a cloud server because there is no storage space on my computer and it is convenient for several friends to share it with me. I suggest building a local database to get faster response time. The 10 yuan/month student database is really slow (sorry about fleece, I’m poor).

For the synchronization and integrity check logic of database data, the logic is distributed in each.sync.ts file under each service-data module. For specific logic, I will take the daily synchronization and integrity check of stock data as an example (stock.sync.ts) :

export const CACHE_KEY_STOCK_DATA_SYNC = {
  STOCK_BASIC_READY: 'stock_basic_ready'.ADJ_FACTOR_READY: 'adj_factor_ready'.DAILY_BASIC_READY: 'daily_basic_ready'.DAILY_READY: 'daily_ready'.WEEKLY_READY: 'weekly_ready'.MONTHLY_READY: 'monthly_ready'};// Synchronize Tuhsare data to the database
@Injectable(a)export class StockDataSync implements OnModuleInit {
  useMyDb: boolean;
  token: string;
  tradeCalBegin: string;
  constructor(
    @InjectRepository(StockBasic)
    private stockBasicRepository: Repository<StockBasic>,
    @InjectRepository(AdjFactor)
    private adjFactorRepository: Repository<AdjFactor>,
    @InjectRepository(DailyBasic)
    private dailyBasicRepository: Repository<DailyBasic>,
    @InjectRepository(Daily)
    private dailyRepository: Repository<Daily>,
    @InjectRepository(Weekly)
    private weeklyRepository: Repository<Weekly>,
    @InjectRepository(Monthly)
    private monthlyRepository: Repository<Monthly>,
    @Inject(CACHE_MANAGER) public readonly cacheManager: Cache,
    @Inject(ConfigService) private readonly configServices: ConfigService,
    private readonly baseData: BaseData,
    private readonly baseDataUtil: BaseDataUtil,
  ) {
    const { useMyDb, token, tradeCalBegin } =
      this.configServices.get('tushare');
    this.token = token;
    this.tradeCalBegin = tradeCalBegin; // Get the start date of quantization calendar and quantization data
    this.useMyDb = useMyDb;
  }

  @Cron('0 0 17 * * 1-5') // Last update time of all data items to be checked
  async onModuleInit() {
    if (this.useMyDb) {
      console.log('Start stock data integrity check');
      // Check whether the data is reliable in real time at how many points per day at startup. If not, the console outputs a warning
      let stock_basic_ready;
      let adj_factor_ready;
      let daily_basic_ready;
      let daily_ready;
      let weekly_ready;
      let monthly_ready;
      const pArr = [
        this.checkStockBasicReady(),
        this.checkAdjFactorReady(),
        this.checkDailyBasicReady(),
        this.checkDailyReady(),
        this.checkWeeklyReady(),
        this.checkMonthlyReady(),
      ];
      await Promise.all(pArr).then((values) = > {
        stock_basic_ready = values[0];
        adj_factor_ready = values[1];
        daily_basic_ready = values[2];
        daily_ready = values[3];
        weekly_ready = values[4];
        monthly_ready = values[5];
        console.log('End stock data integrity check'); }); }}/ /... Other transaction data integrity check and synchronization functions

  async fillDailyDataByTradesDates(tradeDates: string[]) :Promise<void> {
    const dailys = [];
    const tradeCal = await this.baseData.getTradeCal();
    const pArr = [];
    for (let j = 0; j < tradeDates.length; j++) {
      const trade_date = tradeDates[j];
      if (tradeCal[trade_date].is_open === 1) {
        pArr.push(
          getDaily(this.token, {
            trade_date,
          }).catch((e) = > {
            console.log('daily data fetching error, please check the date${trade_date}`); })); }}await Promise.all(pArr).then((values) = > {
      values.forEach((val) = > {
        if(! val) {return; } dailys.push(... val); }); });for (let i = 0; i < dailys.length; i += 5000) {
      const tempArr = dailys.slice(i, i + 5000);
      await this.dailyRepository.save(tempArr); }}async fillDailyData(startDate: string.endDate: string) :Promise<void> {
    const datesSpliceByYear = spliceByYear(startDate, endDate);
    const years = Object.keys(datesSpliceByYear);
    for (let i = 0; i < years.length; i++) {
      const tradeDates = datesSpliceByYear[years[i]];
      await this.fillDailyDataByTradesDates(tradeDates); }}async checkDailyReady(): Promise<boolean> {
    let daily_ready = false;
    const needStartDate = (
      await this.baseDataUtil.utilGetAfterTradeDates(this.tradeCalBegin, 1))0];
    const needEndDate = (
      await this.baseDataUtil.utilGetBeforeTradeDates(
        isTradeDayAfterHour(new Date(), 17)? addDay(getNowDate()) : getNowDate(),1,))0];
    // Determine the time range of data in the database: [minimum value, maximum value]
    const { startDate, endDate } = (
      await this.dailyRepository.query(
        `select min(trade_date) as startDate, max(trade_date) as endDate from daily; `,))0];
    // case1: No data in database
    if(! (startDate && endDate)) {console.log(
        'daily data filling starts, start date:${needStartDate}, end date:${needEndDate}`,);this.fillDailyData(needStartDate, needEndDate).then(() = > {
        console.log(
          'Daily data filling end, start date:${needStartDate}, end date:${needEndDate}`,); }); }else if (startDate > needStartDate || endDate < needEndDate) {
      //case2: data in the database is incomplete
      if (startDate > needStartDate) {
        console.log(
          'Daily data supplement start, start date:${needStartDate}, end date:${subDay( startDate, )}`,);this.fillDailyData(needStartDate, subDay(startDate)).then(() = > {
          console.log(
            'Daily data supplement end, start date:${needStartDate}, end date:${subDay( startDate, )}`,); }); }if (endDate < needEndDate) {
        console.log(
          'Daily data supplement start, start date:${addDay( endDate, )}, end date:${needEndDate}`,);this.fillDailyData(addDay(endDate), needEndDate).then(() = > {
          console.log(
            'Daily data supplement end, start date:${addDay( endDate, )}, end date:${needEndDate}`,); }); }}else {
      const need_trade_dates =
        await this.baseDataUtil.utilGetTradeDatesByRangeDate(
          needStartDate,
          needEndDate,
        );
      const db_trade_dates = (
        await this.dailyRepository.query(
          `SELECT DISTINCT(trade_date) FROM daily WHERE trade_date BETWEEN '${needStartDate}' AND '${needEndDate}'; `,
        )
      ).map((obj) = > {
        return obj.trade_date;
      });
      const diff_trade_dates = need_trade_dates.filter((trade_date) = > {
        return! db_trade_dates.includes(trade_date); });if(diff_trade_dates.length ! = =0) {
        console.log(
          'Daily data supplement begins, date enumeration:${diff_trade_dates.join(', ')}`,);this.fillDailyDataByTradesDates(diff_trade_dates).then(() = > {
          console.log(
            'Daily data supplement end, date enumeration:${diff_trade_dates.join(', ')}`,); }); }else {
        daily_ready = true;
        console.log(
          'Daily data verification is complete, total number of trading days:${db_trade_dates.length}, start date:${needStartDate}, end date:${needEndDate}`,); }}await this.cacheManager.set(
      CACHE_KEY_STOCK_DATA_SYNC.DAILY_READY,
      daily_ready,
      {
        ttl: 24 * 60 * 60,});returndaily_ready; }}Copy the code

The code above seems a bit long, so I’ll go over the main logic here:

• Tushare update data: For daily stock quotes, Tushare will update data by 5:00 p.m. each day.
• Self-built DB integrity check: for daily stock quotation, service-data will check the integrity of daily stock quotation in daily unit when the program starts and before 5:00 PM every day.
• Verify missing data: When data is missing, the program asynchronously fills data

The article is already quite long, but more detailed verification logic is not covered here. For those interested, check out the code above or github source code.

Investment is risky, entering the market should be cautious ~

2. Respond quickly to data problems

In the process of exploring a strategy, we often modify the policy parameters and then re-test them. If we do not cache the data we query from the database, it is inevitable that each backtest will take a long time. To solve this problem, service-data introduces in-memory caching, where high-frequency data is added to the cache after the first query for the next quick response. Service-data also precaches frequently used data, such as daily data, so that stock strategies can be run quickly after continuous startup.

For the implementation of this cache-specific logic, service-data is maintained in xx.data.ts file types. The following also takes the caching and pre-caching of daily stock data as an example, and the relevant codes are as follows (stock.data.ts) :

export const CACHE_KEY = {
  STOCK_BASICS: 'stock_basic'.STOCK_CODES: 'stock_codes'.STOCK_ADJS: 'stock_adjs'.STOCK_ADJ_LASTS: 'stock_adj_lasts'.STOCK_DAILY_BASICS: 'stock_daily_basics'.STOCK_DAILY_BASIC_LASTS: 'stock_daily_basic_lasts'.STOCK_DAILY_BASIC_All: 'stock_daily_basics_all'.STOCK_DAILYS: 'stock_dailys'.STOCK_DAILY_LASTS: 'stock_daily_lasts'.STOCK_WEEKLYS: 'stock_weeklys'.STOCK_WEEKLY_LASTS: 'stock_weekly_lasts'.STOCK_MONTHLYS: 'stock_monthlys'.STOCK_MONTHLY_LASTS: 'stock_monthly_lasts'};// Private fetch series fetching data, get series providing external calls (using caching)
@Injectable(a)export default class StockData implements OnApplicationBootstrap {
  token: string;
  tradeCalBegin: string;
  presetCache: string;
  constructor(
    @InjectRepository(StockBasic)
    private stockBasicRepository: Repository<StockBasic>,
    @InjectRepository(AdjFactor)
    private adjFactorRepository: Repository<AdjFactor>,
    @InjectRepository(DailyBasic)
    private dailyBasicRepository: Repository<DailyBasic>,
    @InjectRepository(Daily)
    private dailyRepository: Repository<Daily>,
    @InjectRepository(Weekly)
    private weeklyRepository: Repository<Weekly>,
    @InjectRepository(Monthly)
    private monthlyRepository: Repository<Monthly>,
    @Inject(ConfigService) public readonly configService: ConfigService,
    @Inject(CACHE_MANAGER) public readonly cacheManager: Cache,
  ) {
    const { token, tradeCalBegin, presetCache } =
      this.configService.get('tushare');
    this.token = token;
    this.tradeCalBegin = tradeCalBegin; // Get the start date of quantization calendar and quantization data
    this.presetCache = presetCache;
  }

  async onApplicationBootstrap() {
    // Improve the first cache speed: proactively pull the database data batch cache, avoid the query waiting when too many connections
    await Promise.all([
      this.presetCacheAdjs(),
      this.presetCacheDaily(),
      / / this. PresetCacheDailyBasic (), / / error: the Last what GCs, separate the cache is not an error, 3 pre cache is an error at the same time
    ]);
  }

  @Cron('0 15 17 * * 1-5')
  async clearCache() {
    await this.cacheManager.del(CACHE_KEY.STOCK_DAILY_BASICS);
    await this.cacheManager.del(CACHE_KEY.STOCK_DAILY_BASIC_All);
    await this.cacheManager.del(CACHE_KEY.STOCK_DAILY_BASIC_LASTS);
    await this.cacheManager.del(CACHE_KEY.STOCK_ADJS);
    await this.cacheManager.del(CACHE_KEY.STOCK_ADJ_LASTS);
    await this.cacheManager.del(CACHE_KEY.STOCK_DAILYS);
    await this.cacheManager.del(CACHE_KEY.STOCK_DAILY_LASTS);
  }
  
  / /... The logic of other stock data

  async presetCacheDaily() {
    const beginTime = Number(new Date());
    const haveStoreForDaily = await this.cacheManager.get(
      CACHE_KEY_STOCK_DATA_SYNC.DAILY_READY,
    );
    console.log('Start precaching daily');
    if (haveStoreForDaily) {
      const cacheStockDailys = {};
      const cacheStockDailyLast = {};
      const tsCodeDailysMap = {};
      const storeDaily = (await this.dailyRepository.query(`
      SELECT * FROM daily WHERE trade_date >= The ${this.tradeCalBegin}ORDER BY trade_date DESC; `)) as DailyResult[];
      storeDaily.forEach((dailys) = > {
        const ts_code = dailys.ts_code;
        if(! tsCodeDailysMap[ts_code]) { tsCodeDailysMap[ts_code] = []; } tsCodeDailysMap[ts_code].push(dailys); });Object.keys(tsCodeDailysMap).forEach((ts_code) = > {
        const curTsCodeDailys = tsCodeDailysMap[ts_code];
        const lastDaily = curTsCodeDailys[0];
        cacheStockDailyLast[ts_code] = lastDaily;
        const dailysMap = {};
        curTsCodeDailys.forEach((adj) = > {
          dailysMap[adj.trade_date] = adj;
        });
        cacheStockDailys[ts_code] = dailysMap;
      });
      await this.cacheManager.set(CACHE_KEY.STOCK_DAILYS, cacheStockDailys, {
        ttl: 24 * 60 * 60});await this.cacheManager.set(
        CACHE_KEY.STOCK_DAILY_LASTS,
        cacheStockDailyLast,
        {
          ttl: 24 * 60 * 60,});console.log(
        'Precache daily ends, cache start date:The ${this.tradeCalBegin}And time consuming:The ${Number(new Date()) - beginTime
        }`,); }}// pull the complex weights daily line data
  private async fetchStockDailyByTsCodeAfterAdj(ts_code: string) :PromiseThe < {stockDaily: { [trade_date: string]: DailyResult };
    lastStockDaily: DailyResult;
  } | null> {
    let dailys;
    const haveStoreForDaily = await this.cacheManager.get(
      CACHE_KEY_STOCK_DATA_SYNC.DAILY_READY,
    );
    const pArr = [];
    if (haveStoreForDaily) {
      pArr.push(
        this.dailyRepository.find({
          where: { ts_code, trade_date: MoreThanOrEqual(this.tradeCalBegin) },
        }),
      );
    } else {
      // dailys = await getDaily(this.token, {
      // ts_code,
      // start_date: this.tradeCalBegin,
      // });
      pArr.push(
        getDaily(this.token, {
          ts_code,
          start_date: this.tradeCalBegin,
        }),
      );
    }
    // const { adjFactor, lastAdj } =
    // (await this.getAdjFactorByTsCode(ts_code)) || {};
    let adjFactor, lastAdj;
    pArr.push(this.getAdjFactorByTsCode(ts_code));
    await Promise.all(pArr).then((datas) = > {
      dailys = datas[0];
      const curAdjs = datas[1) | | {}; adjFactor = curAdjs.adjFactor; lastAdj = curAdjs.lastAdj; });if(! (dailys && adjFactor)) {return null;
    }
    const lastStockDaily = dailys[0];
    const dailyMap = {};
    dailys.forEach((daily) = > {
      dailyMap[daily.trade_date] = daily;
    });
    // Preweights each transaction record (call adjStockRow to modify the original object directly without reassigning)
    Object.keys(dailyMap).forEach((trade_date) = > {
      const curDaily = dailyMap[trade_date];
      const curAdjFactor = adjFactor[trade_date];
      this.adjStockRow(curDaily, curAdjFactor, lastAdj);
    });
    this.adjStockRow(
      lastStockDaily,
      adjFactor[lastStockDaily.trade_date],
      lastAdj,
    );
    return {
      stockDaily: dailyMap,
      lastStockDaily: lastStockDaily,
    };
  }

  public async getStockDailyByTsCodeAfterAdj(ts_code: string) :PromiseThe < {stockDaily: { [trade_date: string]: DailyResult };
    lastStockDaily: DailyResult;
  } | null> {
    // Get it from the cache first
    const cacheStockDaily =
      (await this.cacheManager.get(CACHE_KEY.STOCK_DAILYS)) || {};
    const cacheStockDailyLast =
      (await this.cacheManager.get(CACHE_KEY.STOCK_DAILY_LASTS)) || {};
    if (cacheStockDaily[ts_code] && cacheStockDailyLast[ts_code])
      return {
        stockDaily: cacheStockDaily[ts_code],
        lastStockDaily: cacheStockDailyLast[ts_code],
      };
    const { stockDaily, lastStockDaily } =
      (await this.fetchStockDailyByTsCodeAfterAdj(ts_code)) || {};
    if(! stockDaily) {return null;
    }
    / / cache
    cacheStockDaily[ts_code] = stockDaily;
    await this.cacheManager.set(CACHE_KEY.STOCK_DAILYS, cacheStockDaily, {
      ttl: 24 * 60 * 60}); cacheStockDailyLast[ts_code] = lastStockDaily;await this.cacheManager.set(
      CACHE_KEY.STOCK_DAILY_LASTS,
      cacheStockDailyLast,
      {
        ttl: 24 * 60 * 60,});return{ stockDaily, lastStockDaily, }; }}Copy the code

The above code is quite long, so I will introduce the main logical process here:

• The program starts or the caller pulls data
• The program determines whether there is any relevant data in the cache, returns if there is, and goes through the next process if there is no data
• The program determines whether daily data is complete in the self-built DB (complete verification when the program starts)
• Completely pull data from self-built DB, incomplete pull data from Tushare platform
• Data is pulled and stored in the cache
• Return after recovery

Ok, after the sharing of service-data solutions for quickly returning large amounts of response data, we move on to the last one, the sharing of notification services.

4. Service-notify

This module is simple. The current realization of the mail notification function, write a back test notification mail and stock notification mail method. Service – Notify: service-notify: service-notify: service-notify: service-notify: service-notify: service-notify

The article is so long, if you can see it from top to bottom, I must thank you, Aligardo

That’s the end of sharing share-help. Perhaps you have forgotten that this is a job search article, haha ~. Let’s get to the subject:

Hello, everyone. I am Mu Yu Jiang, male, graduated from university in 1998. My last job was a front-end development engineer of Alibaba International Business Division (ICBU). Currently unemployed for more than four months, in this sincere job hunting, I hope you can give younger brother a front-end interview opportunity, so that I can start a comeback, turn over a new leaf and stand in the sun to do a happy code son. My job hunting tendency is as follows:

• City: Shenzhen or southern city, not Hangzhou.
• Salary: Accept a pay cut for the right job (but that’s not an excuse to undercut my salary)
• Overtime: can accept a little, but not too much ~
• Atmosphere: hope more young people, relaxed atmosphere ~
• Company: not too small, the best front-end department more than 10 people (not dislike small companies, on the contrary, I like the freedom of small companies, but personal is still far from being alone)
• Tech stack: I’ve been writing react, TS and Node since I started working. I prefer JS to CSS

Contact information: My wechat QR code is in the github link above

Writing is not easy, welcome to praise ~

Historical articles:

# Understand this article, handwritten ten Promise!

Maybe that’s how # Typescript gets started

# Familiar with ES6, these contents at least understand a 70 or 80 bar

Help you build a systematic understanding of front-end scaffolding

# Take the mystery out of automated builds

Virtual DOM and DOM Diff algorithms

# Take you through the four poses of Eslint code checking