Photon is a joiner for Google’s AD system. Let’s take a look today at how Photon can accurately join various times. The original static.googleusercontent.com/media/resea here…

Let’s first explain what joiner is. Let’s say I’m a melon eater and I log on to Taobao, and this login is the first event. Then I searched for “Suitcases of the same style as Xue Zhiqian”, which was the second event. I clicked on popularity, and this is the third event. Then I clicked on “Blablabla,” which was the fourth event. Then I clicked Buy Now, which was the fifth event. In this process, some manufacturers in the “search Xue Zhiqian with the same suitcase” that keyword advertising below, so this time if I click on the keyword following the bidding entry, Taobao is to follow the manufacturer to collect advertising fees. In the whole process, Taobao needs to know

  • Who clicked on the AD and when
  • That’s why I paid for it
  • Who is this user and what are his characteristics

All this information needs to be stored so that advertisers can see what kind of people click on their ads, how effective their ads are, and how Taobao can optimize recommendations to users based on their preferences. In the whole process, each page can be separate, AS long as I ensure that my access to the beginning of the end has a unique identity, each page separately in the server log their own data and ID, and then I rely on a Joiner system to combine the time and features I am interested in

So this kind of Joiner is very important in any company that takes recommendations as a skill. It has to have high availability, so it can’t be stuck collecting money; It can be slightly slower but not too slow, preferably within a few hours 100% of events that need to be joined can be joined; Its throughput must be large, because often features need to be logged along with events; It needs to be able to handle unordered traffic, because my requests may be processed by several different servers, and the order in which these events are read by Joiner may not be the order in which the user accesses them.

At the heart of Photon is a Paxos-based ID Registry. One reason for paxOS is that the system requires high availability, and the joined time may be handled by several machines in different data centers. The last thing you want to store in this database is which event ids have been joined in the last N days. The N can be large or small, depending on the business needs (N is large for offline conversion, but can be small for clicking).

Before sending a Joined event to a Joiner, the Joiner needs to confirm that the event has already been joined. If it has already been joined, skip this event. If not, try telling the PAxOS database that the event is about to be joined and output the result of the join.

The problem is that Google has to make sure that different data centers can handle these events. Data center events can have a latency of hundreds of milliseconds. How to scale a PaxOS-based database with such a large latency?

  • Writes on the server side are processed in batches, ensuring that each communication is more efficient
  • Different shards based on event IDS can increase throughput by increasing the number of machines
    • In order to add shards dynamically, the sharding mechanism of the database is based on the timestamp; Time point T1 may have only 10 shards, and T1 +100 May have 20 shards. The server determines which shard the event should be in based on the access event
    • Since this is timestamp based, Google’s TrueTime system is also used here. This is a system that spanner talked about in detail, but I’m going to skip over here
  • To ensure that the ID is different from time to time, the ID has three parts: server IP, process ID, and timestamp. The timestamp for each thread on each server is monotonically increasing (I don’t know why multithreading is not considered here, but thread ids are also used), so you can ensure that each event ID is unique
  • Periodically, this system will back up all events over N days and delete them

The process below is where the business logic is dealt with

Here you can see that the click log and the access log are separate, because the click log is small, so joiner is the same as the click log. Joiner’s output must be backed up on multiple servers. There are two parts that are not mentioned here, Dispatcher and EventStore. What are they used for?

  • EventStore is a service that returns this Query information based on the Query ID. The EventStore has two parts: the live part (because most of the time happens in a short period of time), and the non-live part. The real-time part is in memory, and the non-real-time part is based on BigTable
    • Every time Joiner processes the business logic, it goes to the EventStore to find the corresponding access to the click. If no corresponding access is found, Joiner returns a failure when handling this event.
    • If successful, joiner tells ID Registry and prints the result of the join
    • In order to prevent the registry’s RPC from failing due to other accidents when the Joiner confirms with Registry, every time the Joiner tries to write, it will bring a token. If the server receives the same token under the same ID, it will return success. If the JOINer receives an RPC failure, it uses the same token to retry
  • The Dispatcher is responsible for ensuring that at least once. The log of Google server is stored in GFS at last. Dispatcher will go to GFS regularly to find unprocessed files and send them to Joiner in batches. Dispatcher will store the progress of unprocessed files. The Dispatcher will confirm with the ID Registry before sending a time that has not been joined
    • If joiner fails, the Dispatcher will log the click again and try join again later

With various GFS and memory +BigTable, Photon’s performance is still good

The P90 delay is about 6-7 seconds

The throughput of the ID Registry server is greatly increased by server-side batch processing.

To be honest, the article was a bit watery and finished in a flash, but I read the original text quite long at first. Next week I still read a not so water of article, everybody have what recommendation can leave a message. The text of the Photon static.googleusercontent.com/media/resea here…