introduce

Bullet screen was born in The video platform of Japan, and was later introduced to China by the short video platform of B station, and developed and grew in China. Later, it was gradually accepted by long video platforms, and now video related applications basically have bullet screen.

However, the long video bullet screen is not quite the same as the short video bullet screen of website B. The short video platform has its own unique bullet screen culture, so the bullet screen pays more attention to the interaction with users. The long video platform still focuses on watching dramas. The bullet screen is similar to the function of comments, so it cannot affect users to watch dramas. The bullet screen should not be too dense, and it is better not to cover each other, otherwise the video content will be significantly affected.

This article mainly discusses the realization principle of bullet screen from the perspective of long video platform, but in fact, bullet screen of short video platform is also the same principle, the difference is that there may be more kinds of bullet screen in short video.

The technical implementation

The canvas

Take the application of our company as an example. There are two platforms, iPhone and iPad. On the iPhone platform, there is the concept of horizontal and vertical screens, which requires the display of bullet screen. On the iPad, there is the concept of both large and small screens, and also the need to display bullets. The technical solution of bullet screen is definitely one set for two platforms, but it needs to consider the situation of different devices and screens.

Therefore, for this problem, I use the concept of canvas to solve the problem of universality. Canvas does not distinguish between the concept of screen size and scale, it is simply used to display bullets, does not handle other business logic, through a Render class to control the canvas rendering. For the differences on different devices, for example, the iPad font is larger and the iPhone font is smaller, the config class is used to control the differences, and no judgment is made inside the canvas.

The canvas will be shown proportionally less on a smaller screen and more on a larger one. When the font becomes larger, the canvas will also be controlled according to the proportion and left-right spacing to ensure that the display proportion is correct. In addition, after the screen width and height change, it will automatically adapt to the new size, avoiding the problem of bullet screen connection disconnection, such as the screen size switch on iPad. For external use, you only need to pass in a frame and do not need to worry about the internal adjustment of the canvas.

Barrage orbit

Viewed from the screen, it can be seen that the barrage is generally line by line. In order to facilitate the management of bullet screen view and the subsequent expansion work, I designed the concept of “track” for bullet screen. Each line is a track that horizontally manages the barrage. This line includes parameters such as speed, end barrage, height, etc. These parameters are applicable to all the barrage in this line. A track is a virtual concept with no corresponding view.

The track has a class to implement it, and that class will contain an array of all the bullets in that row. This idea is a bit like playing a game – rhythm Master, which also has the concept of music tracks, each track corresponds to different speed and color notes, the number of notes is also not fixed, according to the rhythm to decide.

Orbit also has the advantage of being easier to control at different speeds. For example, the bullet screen of Tencent video actually has different speeds, but if you observe carefully, you can find that their bullet screen has “odd and even lines with different speeds”, that is, the odd lines have one speed, and the even lines have one speed, which makes people feel that all bullet screens have different speeds. If the way through the track is very good to achieve, different tracks according to the current number of lines, set different speeds for the projectile barrage.

Sometimes when watching a video, a live bullet screen will appear from the right side, which may be a meme in the video or an interaction similar to an advertisement. However, when the live barrage appears, the screen is usually single line and clear, that is, it is mutually exclusive with the ordinary barrage. There is no ordinary barrage before and after the live barrage. Each bullet barrage corresponds to a time period. According to the time and speed of the active bullet barrage, the time before and after the active bullet barrage is displayed, and the orbit during this time period is temporarily closed and only the active bullet barrage is retained.

polling

Each barrage corresponds to a display time, so it is necessary to look for any barrage that needs to be displayed every once in a while. My design is to drive the barrage display through polling.

Using CADisplayLink for rotation, set the frameInterval to 60, polling once per second. In the callback of polling, check whether there is a barrage to be displayed. If there is, check each track from top to bottom. If a certain track has a position to display, it will be handed to this track for display. Whether there is a position is determined according to whether the last bullet screen has been fully displayed on the right side of the screen and there is a vacant position behind it.

For fetching the data, the logic of the data and view is separate and not coupled to each other. When getting data, it just takes out the data of the barrage used from a very small dictionary according to the time and converts it into Model. The dictionary has very little data, ten seconds of data at most, and there’s no reading of the database, there’s no network request logic, which is all separate logic, which WE’ll talk about later.

Barrage view

Those who often watch videos will know that there are many ways to display bullets, such as pictures of stars, likes, and rectangular background colors. In order to better organize the view, I used a very ordinary UIView presentation and did not do complicated rendering operations for performance.

The main benefit of UIView is layout and subview management, but when you animate on screen, you render the CALayer. In other words UIView is used to organize views, not to render directly, which is in line with Apple’s design philosophy.

Reuse pool

Bullets are a frequently used control, so creating them all the time, as well as adding and removing views, can affect performance. So, like many of your modules, I also introduced the concept of a cache pool, which I’ll call a reuse pool.

The marquee reuse pool is similar to the UITableView reuse pool, the marquee that leaves the screen will be placed in the reuse pool to wait for reuse, next time directly from the reuse pool without recreating. The job of the barrage view is to receive new Model objects and make different view layouts based on the barrage type.

It is not removed from the parent view when it leaves the screen, so it does not need to be added to the addSubview when it is removed from the reuse pool. When the animation is finished, the bullet screen is left at the end of the animation and will automatically return to the fromValue position the next time the animation is done. The view structure actually looks like the one above, and the gray area is the viewable area.

System barrage

At the beginning of the video, there will be guidance information, such as guiding the user to send a bullet screen, or indicating how many bullets there are, which is called system bullet screen. System barrage is generally displayed in the middle of the screen before the subsequent barrage is displayed. However, in order to accurately calculate the barrage until it reaches the middle of the screen, and then display the subsequent barrage, it is not accurate to use the barrage before and after clearing the specific time period, so we adopt another set of implementation scheme.

The system is implemented with a higher accuracy of CADisplayLink polling detection, which means that the frameInterval is set to be smaller. I set the frameInterval to 10, which is six times per second. However, CALayer cannot be used to determine the detection directly. It is necessary to use the presentationLayer, that is, the layer being displayed on the screen, for detection. The frame obtained by this layer is consistent with that displayed on the screen.

Here is a brief introduction to the structure of CALayer. We all know that UIView is the encapsulation of CALayer. In fact, the display on the screen is realized through layer, and layer itself is divided into the following three layers and has different functions.

  • The presentationLayer, which itself is a copy of the current frame, gets a new object each time, the same position shown on the screen during the animation.
  • ModelLayer, saidlayerThe real value after the animation, if I print itmodelLayerandlayerIf so, it turns out that the two are actually the same object.
  • The renderLayer, the rendering frames, the application is going to be made up of, depending on the view hierarchylayerRender tree,renderLayerIt representslayerObjects in the render tree.

Dazzle colour barrage

In the process of playing a bullet screen, we can see a bullet screen with gradient color, which is called “dazzling color bullet screen”. This kind of bullet screen has an obvious feature, that is, its color is gradual. At this time, performance should be considered, because the performance consumption of hd video itself is very high. When the amount of bullet screen is relatively large, more performance consumption will be caused, so it is important to reduce the performance consumption, and the gradual change of bullet screen may aggravate the performance consumption.

For gradient text, it is generally realized by mask. A CAGradientLayer is placed below for gradient, and a layer of text is placed on top. However, this will trigger off-screen rendering, which will lead to performance degradation, so it is not possible to use this scheme. After trying, we decided to use the gradient text color to solve the problem.

CGFloat scale = [UIScreen mainScreen].scale;
UIGraphicsBeginImageContextWithOptions(imageSize, NO, scale);
CGContextRef context = UIGraphicsGetCurrentContext(a);CGContextSaveGState(context);
CGColorSpaceRef colorSpace = CGColorGetColorSpace([[colors lastObject] CGColor]);
CGGradientRef gradient = CGGradientCreateWithColors(colorSpace, (CFArrayRef)ar, NULL);
CGPoint start = CGPointMake(0.0.0.0);
CGPoint end = CGPointMake(imageSize.width, 0.0);
CGContextDrawLinearGradient(context, gradient, start, end, kCGGradientDrawsBeforeStartLocation | kCGGradientDrawsAfterEndLocation);
UIImage *image = UIGraphicsGetImageFromCurrentImageContext(a);CGGradientRelease(gradient);
CGContextRestoreGState(context);
UIGraphicsEndImageContext(a);Copy the code

The way to do this is to create a context for drawing the image, then draw the context with a gradient, and finally get a UIImage and assign the image to the textColor of UILabel.

From the off-screen detection, there was no off-screen rendering and the FPS remained at a very high level.

Pause and start

Barrage is played and suspended with the video, so it is necessary to provide support for the suspension and continuation of barrage. For this part, I use the CAMediaTiming protocol to deal with, through which the process of animation can be controlled.

Add 0.05 to the code to avoid bouncing back when the barrage is paused, so add a time difference. The specific reason is that there is still a slight time difference between the time calculated by convertTime:fromLayer: method and the position of the barrage on the screen, which leads to the rebound of the view position during rendering. The 0.05 value is an empirical value obtained from practice.

- (void)pauseAnimation {
    // Add judgment criteria to avoid repeated calls
    if (self.layer.speed == 0.f) {
        return;
    }
    CFTimeInterval pausedTime = [self.layer convertTime:CACurrentMediaTime() fromLayer:nil];
    self.layer.speed = 0.f;
    self.layer.timeOffset = pausedTime + 0.05f;
}

- (void)resumeAnimation {
    // Add judgment criteria to avoid repeated calls
    if (self.layer.speed == 1.f) {
        return;
    }
    CFTimeInterval pausedTime = self.layer.timeOffset;
    self.layer.speed = 1.0;
    self.layer.timeOffset = 0.0;
    self.layer.beginTime = 0.0;
    CFTimeInterval timeSincePause = [self.layer convertTime:CACurrentMediaTime() fromLayer:nil] - pausedTime;
    self.layer.beginTime = timeSincePause;
}
Copy the code

The CAMediaTiming protocol is a protocol used to control the animation process, such as animations created through CoreAnimation, and CALayer follows this protocol. In this way, if you need to control the animation, you don’t need to reference a CABasicAnimation object and then modify the animation properties to control the animation process. You just need to modify the properties of layer directly.

The following are some of the key attributes of the CAMediaTiming protocol, some of which have also been used in this article.

  • BeginTime, animation start time, can control animation delay display. Generally an absolute time, in order to ensure accuracy, the best first for the presentlayerPerform a conversion, and the delay display is followed by the corresponding time.
  • Duration: indicates the end time of animation.
  • Speed: animation execution speed. The default value is 1. Animation final execution time =duration/speed, that is,durationIs for 3 seconds,speedIt is 2 and the final animation execution time is 1.5 seconds.
  • TimeOffset, controls the animation process, mainly used for combinationspeedTo pause and start the animation.
  • RepeatCount, number of repetitions, andrepeatDurationThe mutex.
  • RepeatDuration, time to repeat execution, if the time is notdurationMultiples of, the last animation will perform incomplete.
  • Autoreverses. Whether an animation is reversely performed once it has been executed. This property sets the value ofdurationThere is a overlay effect ifdurationIs 1 s,autoreversesSet toYESThe time after that is 2s.
  • FillMode, if you want the animation to start, stays atfromValueCan be set tokCAFillModeBackwards. If you want to stop at the end of the animationtoValueIs set tokCAFillModeForwardsIf both are required, set it tokCAFillModeBothThe default iskCAFillModeRemoved, which is removed after the animation ends.

Send a barrage

Insert the barrage

Now, danmaku usually combines the main characters in the drama and various text colors for you to choose, which can also bring some paying users. When sending a barrage, track will be searched from top to bottom, and the frame is judged by the presentationLayer. If the most right side of the layer is not outside the screen and there is a certain gap between the right side of the screen, 10pt is written in the project, it means there is space to insert the next barrage. The barrage will be placed on this track.

If there is no vacant space in the current orbit, track by track is searched from top to bottom until one is found. If there are too many bullets on the current screen and there are no empty tracks, this one will be discarded.

If the bullet screen is posted by the user, this must be displayed, because the user’s bullet screen should give the user a feedback on the interface. – there will be a queue jumping action for your own barrage, which has a higher priority than any other barrage. Their bullets are not in the local database, just a network request to the server, and display on the interface.

Select role

As you can see in the image above, the text is preceded by the role and the role name, which are independent of the input text. If the user deletes the input text, then click Delete to delete the role. The input field page is a UITextField, and the character avatar and character name on the left is a custom View that is displayed as the leftView of the textField. If you delete it, you just set leftView to nil.

The problem is that if you use UIControlEventEditingChanged events, can only access to the contents of a text is changed, if the text input box has been finish delete, and the role is a leftView, but since the text is empty, is unable to get to delete events, also can’t delete the role.

For this problem, we found the following protocol to implement. UITextField complies with the UITextInput protocol, but UITextInput inherits from UIKeyInput, so it has the following two methods. The following two methods, called when text is inserted and when the delete button is clicked, receive a callback from deleteBackward even if the text is empty. In this callback, you can determine whether the text is empty and delete the role if it is.

@protocol UIKeyInput <UITextInputTraits>
@property(nonatomic.readonly) BOOL hasText;
- (void)insertText:(NSString *)text;
- (void)deleteBackward;
@end
Copy the code

Barrage set

Parameter adjustment

Generally, bullet screen is not in the same form, and many parameters can be adjusted. For iPhone and iPad, the parameters are different, and the adjustment range is also different. These parameters certainly cannot be judged in the business code, so various judgment conditions are scattered in the project, resulting in serious code coupling.

Our approach to this problem is to use BarrageConfig to differentiate between platforms, placing the numerical differences between the two platforms in this class. The business part can read the attributes directly without making any judgments, and the persistence of the exit process is also done internally, which allows the business part to use no awareness and ensures the consistency of values in each class.

BarrageConfig can be modified by calling Render’s layoutBarrageSubviews when any parameters are changed. After adjusting parameters, the bullet screen displayed on the screen also needs to change, and the process of changing is still in the animation execution process, animation execution can not be broken, so the processing of animation is very important. This part is a little bit more complicated to deal with, so I won’t go into it.

give a like

There will also be a “like” and “long press” function. “Like” usually means clicking on the screen and then a selection view will appear. After clicking on “like”, there will be an animation effect. Long press is to select a bullet screen, recognize the gesture after long press, a report page will appear on the right.

I used tap and longPress gestures to process these two gestures, and set a recognition time of 0.2s for longPress. The recognition of these two gestures was handed over to the system, which was relatively easy.

Both gestures are added to Render instead of one for each barrage view, which makes it easier to manage. In this way, in gesture recognition, it is necessary to find the gesture touch point first, and then search the corresponding bullet screen view according to the touch point. During the search, the region is still searched through the presentationLayer, instead of using the view to search.

- (void)singleTapHandle:(UITapGestureRecognizer *)tapGestureRecognizer {
    CGPoint touchLocation = [tapGestureRecognizer locationInView:self.barrageRender];
    __block BOOL barrageLiked = NO;
    weakifyself;
    [self enumerateObjectsUsingBlock:^(SVBarrageItemLabelView *itemLabel, NSUInteger index, BOOL *stop) {
        strongifyself;
        if ([itemLabel.layer.presentationLayer hitTest:touchLocation] && barrageLiked == NO) {
            barrageLiked = YES;
            [self likeAction:itemLabel withTouchLocation:touchLocation];
            *stop = YES; }}]; }Copy the code

Barrage of advertising

advertising

For such a good display position, the advertising department will not let go. During the video playback, an advertisement barrage will be displayed at a specified time according to the requirements of the sponsor’s father, and the form of this barrage is still not fixed. This means that the size and animation form can not be determined, and the projectile has to be displayed on the top layer.

For this problem, the solution we adopted is to leave a special view for the advertisement, the view level is higher than Render, and pass it to the SDK when initializing the advertisement SDK, so that the control of the advertisement barrage is handed over to the SDK, and we do not deal with it.

Layer management

There are many layers on the player, such as playback control, Render, ads and so on. There are many visible and invisible layers. For this problem, the player creates a view manager that inherits from NSObject and can manage views hierarchically.

All views on the player need to call the specified method to add themselves to the corresponding layer, and remove them need to call the corresponding method. When you need to adjust the order, you can change the enumeration of the definition.

Data separation

I’ve been talking about the view part, not the data part, because the UI and the data are decoupled, they’re not strongly coupled, so I can talk about them separately. The data part of the design, similar to the player’s Local Server solution, is split between requesting data locally and reading data locally.

The request data

The amount of bullet screen data is large, so it cannot be requested all at one time, which is easy to cause the request failure. Therefore, this piece takes a fragment data of five minutes. Ten seconds before the end of the current five-minute barrage, it begins to request the barrage of the next period. If you drag the progress bar, it will start to request the new location of the barrage after dragging. Before each request, the library is checked to see if the data already exists.

The requested data is driven by the business part. The requested data is not directly used, but stored in the local database. This part is similar to the operation of the server writing TS fragments to the local. For database storage, IT is recommended to use WCDB. The bullet screen is mainly for batch data processing, and WCDB has higher performance than FMDB for batch data processing.

Take the data

Data fetching is also driven by the business layer. In order to reduce frequent database reads and writes, the database is read in batches every ten seconds and converted to model and returned to the upper layer. The bullet screen module maintains a dictionary in memory, with the time as the key and the array as value, because there may be more than one bullet screen at the same time.

The data obtained in batches from the database is stored in the dictionary. The upper-layer business layer uses the dictionary to obtain data, which decouples the data layer from the business layer. The upper business layer reads the data from the dictionary every second, finds the appropriate track through the data, and passes the data to the appropriate track for processing.

Barrage defense exploration

At present, many video websites have launched the bullet screen shielding scheme. For the characters in the video, the bullet screen will be displayed under them instead of blocking them. Other apps have made a new exploration for bullet screen occlusion, that is, after becoming a paid member, you can choose only your favorite Aidou not to be blocked, while others are still blocked.

Semantic segmentation

According to the analysis of the business scenario, the following operations can be carried out only after the portrait part is segmented and the position of the portrait is obtained. Therefore, the part of portrait segmentation is realized by semantic segmentation, and the video key frames are marked in advance. This workload is very large, so a special annotation team is needed to complete it. According to the labeled model, through machine learning, the computer can accurately identify the location of people and derive the polygon path.

There is also a problem involved, that is, close-range recognition and distant recognition. The machine only needs to recognize close-range characters, but not distant characters, otherwise the display effect of bullet screen will be greatly affected. Semantic segmentation can be achieved through Google Mask_RCNN.

Client implementation scheme

The solution of the client is to extract the portrait from the original video and export a new video through the polygon path of the portrait. When playing, it is actually played by two players before and after, and the bullet screen is sandwiched between the two players. In addition, the front portrait layer needs to be blurred to make the transition of bullets appear more natural, otherwise it will be too abrupt.

The transition would be better. Because when cutting each frame of video, each frame cannot guarantee that the edges of adjacent frames are not much different, that is, the edges of adjacent near frames cannot guarantee good connection, so it is easy to appear the problem of video continuity. In the superimposed scheme of two players, the video content of the two layers is actually very closely connected. If you remove the bullet screen layer, you can’t tell that it is two layers of players, so the problem of continuity is not obvious.

Front-end implementation scheme

Front-end implementation scheme is the server polygon path in an SVG file, and the file is sent to the front end, the front end through the CSS mask image mask. When the mask is used to cut out the part of the portrait, there is still a black area outside the portrait. Black is the displayable area, which is similar to the mask attribute of iOS.

Station B is the first station to do barrage defense, now station B is not limited to live-action barrage defense, now many cartoon characters also support barrage defense. You can watch the video below to get a feel for it.

B station play barrage defense