Those of you who have developed the AVVideoComposition framework have heard of it, and it’s a very powerful class that allows you to do things like stickers, video effects, transitions, basically all the things you see in short video editing programs.
This article will focus on the video of how stickers and special effects are achieved. All functions are implemented based on the AVFoundation framework.
If you want to implement the sticker feature, there are two solutions:
1, based on AVVideoComposition AVVideoCompositionCoreAnimationTool interface, principle is in the video broadcast add stickers on the layer, AVSynchronizedLayer is used to manage the display state of stickers based on playback time.
2, follow AVVideoCompositing protocol, custom video synthesizer.
Method 1 above cannot be used to implement video effects because they are in different levels and you can only manage the sticker layer. Method 2 allows stickers and special effects. This article is also based on the second way to explain the implementation of this part of the function.
A, principle
If we want to add a texture or special effect to the video, do we need to get the frame first so that we can add our stickers to it, or do we need to render the frame after the special effect. Now placed in front of us is how to get this frame picture 🤔?
AVVideoComposition inside a customVideoCompositorClass attributes, it requires us to follow AVVideoCompositing protocol type in (note: Not the instance), inside a method func startRequest (_ asyncVideoCompositionRequest: AVAsynchronousVideoCompositionRequest), said to start a synthesis request, we are in this function and stickers to complete the original video picture under the synthetic, or will be treated as one frame into the special effects, After the processing is complete, call finish to submit the composited frame. The startRequest function is called several times, synchronizing with the current screen until it pauses.
Add AVVideoComposition to AVPlayerItem
An instance of AVVideoComposition can be passed in as an AVPlayerItem property so that when AVPlayerItem plays it will use the AVVideoComposition we passed in for each frame. It’s going to look like this.
let videoCompostion = AVMutableVideoComposition(propertiesOf: asset)
videoComposition.renderSize = CGSize(width: 1920, height: 1080)
playerItem.videoComposition = videoCompostion
player = AVPlayer.init(playerItem: playerItem)
Copy the codeCustomize AVVideoCompositing
Now, we have created a AVMutableVideoComposition instance, so how do we take over every frame image? That’s customizing AVVideoCompositing. AVVideoComposition has a property called customVideoCompositorClass, Type of AVVideoCompositing. Type? . Through it, we can obtain the original picture information of every frame displayed when the video is playing. Here is some information about the protocol AVVideoCompositing.
public class VideoCustomComposition: NSObject.AVVideoCompositing {
public var sourcePixelBufferAttributes: [String : Any]? = [String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
String(kCVPixelBufferOpenGLESCompatibilityKey): true]
public var requiredPixelBufferAttributesForRenderContext: [String : Any] = [String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
String(kCVPixelBufferOpenGLESCompatibilityKey): true]
public func renderContextChanged(_ newRenderContext: AVVideoCompositionRenderContext) {
renderQueue.sync { [weak self] in
self?.renderContext = newRenderContext
}
}
public func startRequest(_ asyncVideoCompositionRequest: AVAsynchronousVideoCompositionRequest) {
// compositing logic}}Copy the codeThese are required, and we focus on the implementation of the startRequest method. This method is called when the player has moved to the next frame but has not yet shown it to tell the developer that a new frame needs to be shown and what to do with it. And all information included in the asyncVideoCompositionRequest. In AVAsynchronousVideoCompositionRequest class view, mainly has the following properties and methods:
// Render context
open var renderContext: AVVideoCompositionRenderContext { get }
// The compositing time, which is related to the frame rate, can be understood as the display time of each frame
open var compositionTime: CMTime { get }
// Video channel ID
open var sourceTrackIDs: [NSNumber] { get }
/ / synthesis command, you can customize, default is the system to provide AVVideoCompositionInstruction classes
open var videoCompositionInstruction: AVVideoCompositionInstructionProtocol { get }
// Returns the pixel-level information of the video frame under the specified trackID, synchronized with the current playback time
open func sourceFrame(byTrackID trackID: CMPersistentTrackID) -> CVPixelBuffer?
// Submit the processed pixel information
open func finish(withComposedVideoFrame composedVideoFrame: CVPixelBuffer)
// Submit the composition operation, and set the error, it means that the composition failed
open func finish(with error: Error)
// The request was cancelled
open func finishCancelledRequest(a)
Copy the codeLet’s look at the implementation of the startRequest method
public func startRequest(_ asyncVideoCompositionRequest: AVAsynchronousVideoCompositionRequest) {
renderQueue.async { [weak self] in
guard let strongSelf = self else {
return
}
if strongSelf.shouldCancelAllPendingRequests {
asyncVideoCompositionRequest.finishCancelledRequest()
} else {
autoreleasepool {
// Process the compositing request and return the processed data
if let pixelBuffer = strongSelf.handleNewPixelBuffer(from: asyncVideoCompositionRequest) {
asyncVideoCompositionRequest.finish(withComposedVideoFrame: pixelBuffer)
} else {
// Failed to compose, return error
asyncVideoCompositionRequest.finish(with: VideoCustomCompositionError.newPixelBufferRequestFailed)
}
}
}
}
}
Copy the codeProcessing merge request specific logic, each section of code I added a look, or more clear.
func handleNewPixelBuffer(from request: AVAsynchronousVideoCompositionRequest) -> CVPixelBuffer? {
// Create a blank canvas
guard let pixelBuffer = request.renderContext.newPixelBuffer() else {
return nil
}
// The canvas size is the renderSize of VideoComposition
let width = CVPixelBufferGetWidth(pixelBuffer)
let height = CVPixelBufferGetHeight(pixelBuffer)
var image: CIImage?
// Set the default background color
var backgroundColor: CGColor = UIColor.black.cgColor
if let instruction = request.videoCompositionInstruction as? AVVideoCompositionInstruction {
backgroundColor = instruction.backgroundColor ?? UIColor.black.cgColor
}
// Set a custom background color
if let coordinator = coordinator {
if let timeLineBackgroundColor = coordinator.timeLine.backgroundColor {
backgroundColor = timeLineBackgroundColor.cgColor
}
}
// Fill the background color
let backgroundImage = CIImage(color: CIColor(cgColor: backgroundColor)).cropped(to: CGRect(x: 0, y: 0, width: width, height: height))
// Real video frames
for trackID in request.sourceTrackIDs {
if let sourcePixelBuffer = request.sourceFrame(byTrackID: trackID.int32Value) {
let sourceImage = CIImage(cvPixelBuffer: sourcePixelBuffer)
image = sourceImage
}
}
// Return the default background color when there is no video screen
guard var videoImage = image else {
VideoCustomComposition.ciContext.render(backgroundImage, to: pixelBuffer)
return pixelBuffer
}
// Externally process this frame, make stickers or special effects
if let coordinator = coordinator {
videoImage = coordinator.apply(source: videoImage, at: request.compositionTime)
}
// Merge the processed video frames onto the background image
videoImage = videoImage.composited(over: backgroundImage)
// Outputs the final image to the pixel buffer
VideoCustomComposition.ciContext.render(videoImage, to: pixelBuffer)
return pixelBuffer
}
Copy the codeHere we see a property coordinator, which is a property of the VideoCustomComposition private var coordinator: CompositionCoordinator? = CompositionCoordinatorPool. Shared. The pop coordinator (). There’s another class coming up here called TimeLine. Coordinator is used to coordinate the data interaction between VideoCustomComposition and TimeLine. The Apply method simply does a passthrough.
struct CompositionCoordinator {
let timeLine: TimeLine
func apply(source: CIImage.at time: CMTime) -> CIImage {
return timeLine.apply(source: source, at: time)
}
}
Copy the codeAdd stickers and special effects
In step 3, we have successfully retrieved each frame of the video and passed it to the TimeLine class for processing. TimeLine is a class that represents the complete TimeLine of video playback. You can add stickers to it and set the display effects of videos in a certain period of time.
A, stickers,
@discardableResult func insert(element: OverlayProvider) -> VisualElementIdentifer {
let id = eidBuilder.get()
element.visualElementId = id
overlayElementDic[id] = element
return id
}
Copy the codeThe stickers in TimeLine are a class that complies with the OverlayProvider protocol, which requires that the stickers be implemented as follows: frame position of the stickers, original size extent of the stickers, and picture to be displayed at a certain time func applyEffect(at time: CMTime) -> CIImage? . The OverlayProvider protocol inherits from the VisualProvider protocol, which requires that an id of the type VisualElementIdentifer be provided. The id is automatically set within the TimeLine. Therefore, set it to the default invalid. The VisualProvider protocol inherits the TimingProvider protocol, which requires a timeRange of CMTimeRange type, indicating that it is valid in that period of time of video.
Let’s look at an example of implementing a good static sticker:
/// static image stickers
public class StaticImageOverlay: OverlayProvider {
public var frame: CGRect = .zero
public var timeRange: CMTimeRange = .zero
public var extent: CGRect = .zero
public var visualElementId: VisualElementIdentifer = .invalid
public var image: CIImage!
public func applyEffect(at time: CMTime) -> CIImage? {
image
}
public init(image: CIImage) {
self.image = image
frame = CGRect(origin: .zero, size: image.extent.size)
extent = image.extent
}
private init(a){}}Copy the codeIt is also very simple to use, as shown below, inserting a static sticker between 0s and 2s of the timeline of the current video.
let uiimage = UIImage(named: "biaozhun")!
let ciimage = CIImage(cgImage: uiimage.cgImage!)
let overlay = StaticImageOverlay.init(image: ciimage)
overlay.timeRange = CMTimeRange.init(start: CMTime.init(value: 0, timescale: 1), end: CMTime.init(value: 2, timescale: 1))
overlay.frame = CGRect(x: 20, y: 20, width: 160, height: 60)
timeLine.insert(element: overlay)
let videoCompostion = builder.buildVideoCompositon()
playerItem.videoComposition = videoCompostion
player.replaceCurrentItem(with: playerItem)
Copy the code
The principle of dynamic stickers is the same as that of static stickers, but with the added process of parsing giFs. We need to read each frame of the GIF, get the duration of each frame, the total duration, the total number of frames. This allows you to decide which frame to play at any given moment.
public func applyEffect(at time: CMTime) -> CIImage? {
let curTime = CMTimeSubtract(time, timeRange.start)
var curTimeSeconds = CMTimeGetSeconds(curTime)
if curTimeSeconds > totalDuration {
// We need to repeat this
curTimeSeconds - = totalDuration
}
var nearTime: TimeInterval = 0
for i in 0..<frameCount {
if nearTime > = curTimeSeconds {
// get i
if let imageSource = imageSource {
if let image = CGImageSourceCreateImageAtIndex(imageSource, i, nil) {
return CIImage(cgImage: image)
}
}
break
}
nearTime + = frameDuration
}
// The last frame is displayed to prevent blank space
if frameCount > 0 {
if let imageSource = imageSource {
if let image = CGImageSourceCreateImageAtIndex(imageSource, frameCount - 1.nil) {
return CIImage(cgImage: image)
}
}
}
return nil
}
Copy the code
In addition to static and dynamic stickers, I also provide an animated sticker that implements four basic animation types: opacity, rotate, scale, and Translate shift. Except for transparency changes, the other animations are implemented based on CAAffineTransform. The principle is to calculate which stage of the animation the current state is in, and thus calculate the intermediate state. For example, make rotation changes:
func handleAnimation(basic an: BasicAnimation.progress ratio: CGFloat.image: CIImage) -> CIImage {
guard an.from ! = nil && an.to ! = nil else {
return image
}
// omit other code
let by = an.anyFloatValue(an.from) + (an.anyFloatValue(an.to) - an.anyFloatValue(an.from)) * ratio
return image.apply(rotate: by, extent: image.extent)
}
Copy the code/// rotate the image
/// -parameter rotate: radian
/// -parameter extent: the true size of the image: 'ciimage.extent'
/// - Returns the rotated image
func apply(rotate: CGFloat.extent: CGRect) -> CIImage {
var t = CGAffineTransform.identity
t = t.concatenating(CGAffineTransform(translationX: -(extent.origin.x + extent.width/2), y: -(extent.origin.y + extent.height/2)))
t = t.concatenating(CGAffineTransform.init(rotationAngle: rotate))
t = t.concatenating(CGAffineTransform(translationX: (extent.origin.x + extent.width/2), y: (extent.origin.y + extent.height/2)))
return transformed(by: t)
}
Copy the codeSee my implementation of animated stickers for details.
Second, the special effects
TimeLine also provides an interface for adding effects, which requires an object that complies with the SpecialEffectsProvider protocol.
@discardableResult func insert(element: SpecialEffectsProvider) -> VisualElementIdentifer {
let id = eidBuilder.get()
element.visualElementId = id
specialEffectsElementDic[id] = element
renderCurrentFrameAgain()
return id
}
Copy the codepublic protocol SpecialEffectsProvider: VisualProvider {
func applyEffect(image: CIImage.at time: CMTime) -> CIImage?
}
Copy the codeThis protocol is very simple. It gives you a callback that takes two arguments, image for the original view and time for the frame to play, and asks you to return the processed image. This protocol also inherits from the VisualProvider protocol, which is explained in the sticker section and will not be detailed here.
In fact, at this point, you should have some ideas for adding special effects to the video. Here, I directly use the CoreImage framework to simply add several special effects to the video, and realize the video distortion effect and point-like effect. Let’s take a look at the distortion.
You can see that the video is distorted between 1s and 5s. I used the CoreImage’s CIVortexDistortion filter to achieve this effect.
The code is as follows:
/// Distortion based on 'CIVortexDistortion'
public class DistortionEffects: SpecialEffectsProvider {
public var visualElementId: VisualElementIdentifer = .invalid
public var timeRange: CMTimeRange = .zero
public var maxAngle: CGFloat = 360.0
public var radius: CGFloat = 1800
private let filter: CIFilter!
public init(a) {
filter = CIFilter(name: "CIVortexDistortion")}public func applyEffect(image: CIImage.at time: CMTime) -> CIImage? {
filter.setValue(image, forKey: kCIInputImageKey)
filter.setValue(CIVector(x: image.extent.center.x, y: image.extent.center.y), forKey: kCIInputCenterKey)
filter.setValue(radius, forKey: kCIInputRadiusKey)
let relateTime = CMTimeSubtract(time, timeRange.start)
let ratio = CMTimeGetSeconds(relateTime) / CMTimeGetSeconds(timeRange.duration)
filter.setValue(ratio * maxAngle, forKey: kCIInputAngleKey)
return filter.outputImage
}
}
Copy the codeAgain, it’s very simple to use. Just initialize a special effect object, set the time range, and insert it into TimeLine immediately.
let spe = DistortionEffects()
spe.timeRange = CMTimeRange.init(start: CMTime.init(value: 1, timescale: 1), duration: CMTime.init(value: 4, timescale: 1))
spe.maxAngle = 3600
timeLine.insert(element: spe)
let videoCompostion = builder.buildVideoCompositon()
playerItem.videoComposition = videoCompostion
player.replaceCurrentItem(with: playerItem)
Copy the codeAt this point, the principle and implementation of adding stickers and special effects to videos have been finished. If you want to export the video, just use our VideoComposition generator to generate one, or just use the synthesizer in AVPlayerItem. The exported video is automatically added with special effects and stickers. Is it very convenient?
let export = AVAssetExportSession.init(asset: playerItem.asset, presetName: AVAssetExportPresetHighestQuality)
export?.outputURL = URL(fileURLWithPath: outputURL)
export?.outputFileType = .mp4
export?.shouldOptimizeForNetworkUse = true
export?.videoComposition = builder.buildVideoCompositon()
export?.exportAsynchronously {
// ...
}
Copy the codeSpecific implementation details, we can download the framework I wrote: iVisual
Write at the end
Record of problems encountered in the development of iVisual framework:
1, CGAffinetransform
When we set the CIImage rotation or scaling, the default origin is at the lower left corner of the image. Therefore, we need to translate the image first by placing the center point of the image at the original lower left corner and then rotate or translate it.
func apply(rotate: CGFloat.extent: CGRect) -> CIImage {
var t = CGAffineTransform.identity
t = t.concatenating(CGAffineTransform(translationX: -(extent.origin.x + extent.width/2), y: -(extent.origin.y + extent.height/2)))
t = t.concatenating(CGAffineTransform.init(rotationAngle: rotate))
t = t.concatenating(CGAffineTransform(translationX: (extent.origin.x + extent.width/2), y: (extent.origin.y + extent.height/2)))
return transformed(by: t)
}
func apply(scale: CGFloat.extent: CGRect) -> CIImage {
var t = CGAffineTransform.identity
t = t.concatenating(CGAffineTransform(translationX: -(extent.origin.x + extent.width/2), y: -(extent.origin.y + extent.height/2)))
t = t.concatenating(CGAffineTransform.init(scaleX: scale, y: scale))
t = t.concatenating(CGAffineTransform(translationX: (extent.origin.x + extent.width/2), y: (extent.origin.y + extent.height/2)))
return transformed(by: t)
}
Copy the code2. How to add stickers in real time during playback
When we play, iVisual will render that frame in real time according to the TimeLine context, and when we pause, AVFoundation has submitted the current frame, Who is asyncVideoCompositionRequest of VideoComposition. Finish (withComposedVideoFrame: pixelBuffer). So, if a new sticker is added at this moment, it needs to be displayed immediately after clicking Add. So this frame is going to be recomposed and displayed. However, AVFoundation does not directly provide such a redrawing method, so we need to find another method.
1. Cover a fake picture on the screen
2. VideoComposition rerendered this frame and tried setting isFinished to false, but it didn’t work
On the second question, if any students have a good way, welcome to discuss 👏.