preface

We completed the import of short video materials, this article formally began to use AVFoundation for short video Editing, from the actual Editing function introduced the framework of the Editing module has various capabilities.

I. Video stitching + BGM

Let’s start with the simplest of features: splicing two (or more) videos together and adding a piece of background music.

1.1 AVMutableComposition

As we mentioned in the previous, AVAssetWriter when writing does not support preview (although by AVSampleBufferDisplayLayer can display CMSambuffer, it undoubtedly has increased a lot of effort against us from the macroscopic view video editor of beginner’s mind). The AVPlayerItem required for video playback needs to be initialized with an AVAsset instance. We want a class that inherits from AVAsset and can modify the AVAssetTrack arbitrarily, either to handle editing or to preview the AVAssetTrack before a time-consuming export. AVFoundation provides a class called AVComposition whose mutable subclass, AvMutable LecomPosition, satisfies these requirements.

Many classes in the AVFoundation Editing module have an immutable parent class that has many read-only attributes, and its mutable subclass inherits the attributes of the parent class, making some of the attributes readable and writable. Note that not all of the attributes are readable and writable. Then we’ll go straight to “Mutable” classes and edit videos instead of covering all the immutable classes.

In the Assets module, we said that AVAsset contains one or more AvassetTracks, and that AVComposition as a subclass also contains one or more AvCompositionTracks, And we handle object is mutable subclasses AVMutableComposition and AVMutableCompositionTrack.

AVMutableComposition provides two class methods to create an empty resource.

+ (instancetype)composition;
+ (instancetype)compositionWithURLAssetInitializationOptions:(nullable NSDictionary<NSString *, id> *)URLAssetInitializationOptions NS_AVAILABLE(10_11, 9_0);
Copy the code

The method of add and remove from the composition AVMutableCompositionTrack:

/ / to add a composition specified media resources types of empty AVMutableCompositionTrack - (AVMutableCompositionTrack *)addMutableTrackWithMediaType:(NSString *)mediaType preferredTrackID:(CMPersistentTrackID)preferredTrackID; // remove a specified track from composition - (void)removeTrack:(AVCompositionTrack *)track;Copy the code

Modify AVCompositionTrack method:

// add all tracks from asset for the specified time period to composition at startTime - (BOOL)insertTimeRange:(CMTimeRange)timeRange ofAsset:(AVAsset *)asset atTime:(CMTime)startTime error:(NSError * _Nullable * _Nullable)outError; // add an empty timeRange to all tracks in composition - (void)insertEmptyTimeRange:(CMTimeRange)timeRange; // Delete tracks from Composition for a period of time. Track segment - (void)removeTimeRange:(CMTimeRange)timeRange // change the length of all tracks in composition - (void)scaleTimeRange:(CMTimeRange)timeRange toDuration:(CMTime)duration;Copy the code

Provides methods can be seen from above, we can add empty to AVMutableComposition AVMutableCompositionTrack orbit, Then prepared AVAsset AVAssetTrack inserted into AVMutableCompositionTrack orbit, video stitching is such a simple operation.

Using AVMutableComposition, we can finish the stitching of the video, and then add an AVCompositionAudioTrack with the same duration as the total time to have the background music. If you want to adjust the volume of each track when multiple audio tracks are mixed and played, We also need another class, AVAudioMix, AVPlayerItem, which also has this property and applies the mixed audio parameters when playing.

1.2 AVMutableAudioMix

AVAudioMixInputParameters AVMutableAudioMix consists of a group, each AVAudioMixInputParameters corresponding to a audio AVCompositionTrack it control. AVAudioMixInputParameters contains a MTAudioProcessingTap for real-time processing of audio, a AVAudioTimePitchAlgorithm, it can be used to set the tone, the two relative to a little bit complicated, We’re not paying attention. Now we want to set the original video and background music track playback volume size, can use AVAudioMixInputParameters provides such as the following methods:

// change the volume to volume - (void)setVolume:(float)volume atTime:(CMTime)time // change the volume linearly within timeRange, From startVolume gradually becoming endVolume - (void) setVolumeRampFromStartVolume: (float) startVolume toEndVolume endVolume: (float) timeRange:(CMTimeRange)timeRange;Copy the code

By creating each audio track AVAudioMixInputParameters, configure the volume, adding multiple AVAudioMixInputParameters arrays, as AVAudioMix inputParameters properties.

AVAudioMix doesn’t directly change how audio is played, it just stores how audio is played, the same goes for AVVideoComposition.

Here is an example of code to splice video and add background music:

/ / 1. Create an array of AVMutableComposition, AVMutableudioMix, and AVAudioMixInputParameters AVMutableComposition * composition = [AVMutableComposition composition]; AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix]; NSMutableArray *audioMixInputParameters = [NSMutableArray array]; / / 2. Insert null audio and video tracks AVMutableCompositionTrack * videoCompositionTrack = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid]; AVMutableCompositionTrack* audioCompositionTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid]; CMTime startTime = kCMTimeZero; CMTime duration = kCMTimeZero; For (int I = 0; i < assetArray.count; i++) { AVAsset* asset = assetArray[i]; AVAssetTrack* videoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] firstObject]; AVAssetTrack* audioTrack = [[asset tracksWithMediaType:AVMediaTypeAudio] firstObject]; / / 3. Insert the corresponding audio and video in orbit [videoCompositionTrack insertTimeRange: CMTimeRangeMake (kCMTimeZero, asset.duration) ofTrack:videoTrack atTime:startTime error:nil]; [audioCompositionTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, asset.duration) ofTrack:audioTrack atTime:startTime error:nil]; / / 4. The configuration of the original video AVMutableAudioMixInputParameters AVMutableAudioMixInputParameters * audioTrackParameters = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:audioTrack]; AudioTrackParameters setVolume:0.2 atTime:startTime]; // Set the original video sound volume [audioTrackParameters setVolume:0.2 atTime:startTime]; [audioMixInputParameters addObject:audioTrackParameters]; AudioTrackParameters setVolume:0.2 atTime:startTime]; // Set the original video sound volume [audioTrackParameters setVolume:0.2 atTime:startTime]; [audioMixInputParameters addObject:audioTrackParameters]; StartTime = CMTimeAdd(startTime, asset-.duration); }; // 5. Add BGM audio track AVAsset *bgmAsset =... ; AVMutableCompositionTrack *bgmAudioTrack = [composition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid]; AVAssetTrack *bgmAssetAudioTrack = [[bgmAsset tracksWithMediaType:AVMediaTypeAudio] objectAtIndex:0]; [bgmAudioTrack insertTimeRange:CMTimeRangeMake(kCMTimeZero, duration) ofTrack:bgmAssetAudioTrack atTime:kCMTimeZero error:nil]; AVMutableAudioMixInputParameters *bgAudioTrackParameters = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:bgmAudioTrack]; // 6. Set the background music volume [bgAudioTrackParameters setVolume:0.8 atTime:kCMTimeZero]; [audioMixArray addObject:bgAudioTrackParameters]; Set inputParameters audioMixArray. InputParameters = audioMixArray; AVPlayerViewController *playerViewController = [[AVPlayerViewController alloc]init]; / / use the AVMutableComposition create AVPlayerItem AVPlayerItem * playerItem = [[AVPlayerItem alloc] initWithAsset: composition]; // 8. Pass the audio blending parameter to AVPlayerItem playeritem. audioMix = audioMix; playerViewController.player = [[AVPlayer alloc] initWithPlayerItem:playerItem]; playerViewController.view.frame = self.view.frame; [playerViewController.player play]; [self presentViewController:playerViewController animated:YES completion:nil];Copy the code

The simplest functions of video stitching and adding background music are finished, but it seems that the video transition is rather rigid. We want to control the composition of each video track as well as the volume of the audio track, and even add the video transition effect. At this time, we need AVVideoComposition. And by the way, let’s do a “superimposed” video transition effect.

Second, video transfer

2.1 AVMutableVideoComposition

AVVideoComposition is available in iOS4.0. It looks like AVVideoComposition is related to AVVideoComposition, but it’s not. AVVideoComposition inherits from NSObject, and we can think of it as a parallel to AVAudioMix, which also inherits from NSObject, one for compositing control of audio tracks and one for compositing control of video tracks.

Create AVMutableVideoComposition method is as follows:

/ / returns the attribute is null instance + (AVMutableVideoComposition *) videoComposition; / / returns the instance contains the appropriate instructions + (AVMutableVideoComposition *) videoCompositionWithPropertiesOfAsset asset: (AVAsset *) API_AVAILABLE (macos (10.9), the ios (6.0), tvos (9.0)) API_UNAVAILABLE (watchos); // iOS13.0 added, In order to create directly set the background color + (AVMutableVideoComposition *) videoCompositionWithPropertiesOfAsset asset: (AVAsset *) PrototypeInstruction: (AVVideoCompositionInstruction *) prototypeInstruction API_AVAILABLE (macos (10.15), the ios (13.0), Tvos (13.0)) API_UNAVAILABLE (watchos);Copy the code

Part AVMutableVideoComposition property list:

// Update time @property (nonatomic) CMTime frameDuration; @property (nonatomic) CGSize renderSize; // Video display scope size scale @property (nonatomic) float renderScale; // Describes a collection of information about a specific video playback mode in a video collection. @property (nonatomic, copy) NSArray<id <AVVideoCompositionInstruction>> *instructions; // These three properties set the video primaries, matrix, transfer function @Property (Nonatomic, nullable) NSString *colorPrimaries; @property (nonatomic, nullable) NSString *colorYCbCrMatrix; @property (nonatomic, nullable) NSString *colorTransferFunction; @property (nonatomic, copy) NSArray<NSNumber *> *sourceSampleDataTrackIDs;Copy the code

Property list can be seen from a AVMutableVideoComposition output video, we can also specify the output size renderSize (cutting), scaling renderScale frameDuration, frame rate, etc. At the same time, AVPlayerItem also contains the videoComposition property, which shows the video content in accordance with the composition instructions when playing.

Neither AVVideoComposition nor AVAudioMix is strongly associated with AVComposition, which gives us more flexibility in previewing, exporting, and capturing video thumbnails.

2.2 AVMutableVideoCompositionInstruction

VideoComposition is one of the most important attribute instructions, the array contains one or more AVMutableVideoCompositionInstruction, it has a backgroundColor attribute is used to modify the background color of video, One of the most critical attributes is timeRange, which describes the timeRange in which the composition occurs.

AVMutableVideoCompositionInstruction properties and methods list:

// The time range in which the directive applies. @property (nonatomic) CMTimeRange timeRange; // Video compositing background color. @property (nonatomic, retain, nullable) __attribute__((NSObject)) CGColorRef backgroundColor; // Specifies how to layer and write video frames from the source track. @property (nonatomic, copy) NSArray<AVVideoCompositionLayerInstruction *> *layerInstructions; // Indicates whether the instruction needs post-processing. @property (nonatomic) BOOL enablePostProcessing; // The track ID required by the instruction followed by the synthesizer to synthesize the video frame. @property (nonatomic) NSArray<NSValue *> *requiredSourceTrackIDs; // passthrough track id @property (nonatomic, readonly) CMPersistentTrackID passthroughTrackID; // New in iOS15 for video metadata synthesis. @property (nonatomic) NSArray<NSNumber *> *requiredSourceSampleDataTrackIDs;Copy the code

Here pay attention to the AVMutableVideoCompositionLayerInstruction have a passthroughTrackID attributes, in spite of the variable in the class is a read-only property.

2.3 AVMutableVideoCompositionLayerInstruction

AVMutableVideoCompositionInstruction have a layerInstructions attribute in the array is AVMutableVideoCompositionLayerInstruction types of instances, Create ` through AVMutableCompositionTrack, or associated with the trackID, described for the synthesis of the rail way, offers to modify specific point in time or a time within the scope of the linear transform method, crop, opacity, as follows, There is not much to do.

/ / the gradient affine transformation - (void) setTransformRampFromStartTransform: (startTransform CGAffineTransform) toEndTransform:(CGAffineTransform)endTransform timeRange:(CMTimeRange)timeRange; - (void)setTransform (CGAffineTransform)transform atTime (CMTime)time; / / transparency gradient - (void) setOpacityRampFromStartOpacity: (float) startOpacity toEndOpacity endOpacity: (float) timeRange:(CMTimeRange)timeRange; // setOpacity - (void)setOpacity:(float)opacity atTime:(CMTime)time; / / cutting area gradient - (void) setCropRectangleRampFromStartCropRectangle startCropRectangle: (CGRect) toEndCropRectangle:(CGRect)endCropRectangle timeRange:(CMTimeRange)timeRange; // set the cliparea - (void)setCropRectangle:(CGRect)cropRectangle atTime:(CMTime)time;Copy the code

In conclusion, AVMutableVideoComposition for video synthesis, make an overall description of video combination, AVMutableVideoCompositionInstruction is used to specify its contains a collection of AVMutableVideoCompositionLayerInstruction controlled by the time range, A mixed track and AVMutableVideoCompositionLayerInstruction is described the present form. So here’s the question: why are the three classes designed this way? So let’s go down.

We start with creating AVMutableVideoComposition first method + (AVMutableVideoComposition *) videoComposition, he will return to a basic empty instance attributes, we from zero to create, That makes sense.

To mix the two videos, first of all, the two videos need to contain overlapping areas in the timeline, and then create the instructions for appearing or disappearing in the mixed area respectively. According to apple’s official documentation, each video track will be configured with A separate decoder, and it is not recommended to add too many tracks. We usually use A/B track method — create two video tracks and insert avassetTrack alternately into A/B track. As shown below, we need to add corresponding instruction to the video. The one that does not contain the overlapping area is called pass through and only needs to specify the time range. The overlapping area is called transition, which requires a layerInstruction describing the hiding mode of the previous video and a layerInstruction describing the appearance mode of the latter video.

The time range of control should be set for each instruction. Once the time range of instruction is incomplete or crossed, errors will occur, such as crash or normal playback. Before we do this we can call AVVideoComposition – (BOOL)isValidForAsset: timeRange: ValidationDelegate: check the instruction which is used to describe the time range is available, the delegate for the method of AVVideoCompositionValidationHandling agreement provides gave us more error description, You can also pass nil if you don’t need it.

Part of the transition AVMutableVideoCompositionInstruction create a sample (stack effect) :

CMTimeRange atTime_end = kCMTimeRangeZero; __block CMTimeRange atTime_begin = kCMTimeRangeZero; NSMutableArray* layerInstructions = [NSMutableArray array]; / / video synthesis command AVMutableVideoCompositionInstruction * videoCompositionInstruction = [AVMutableVideoCompositionInstruction videoCompositionInstruction]; videoCompositionInstruction.timeRange = CMTimeRangeMake(kCMTimeZero, totalDuration); for (int i = 0; i < compositionVideoTracks.count; i++) { AVMutableCompositionTrack *compositionTrack = compositionVideoTracks[i]; AVAsset *asset = assets[i]; // layerInstruction AVMutableVideoCompositionLayerInstruction *layerInstruction = [AVMutableVideoCompositionLayerInstruction videoCompositionLayerInstructionWithAssetTrack:compositionTrack]; If (compositionVideoTracks. Count > 1) {/ / fold change faced atTime_begin = atTime_end; atTime_end = CMTimeRangeMake(CMTimeAdd(CMTimeSubtract(atTime_end.start, transTime), asset.duration), transTime); CMTimeRangeShow(atTime_begin); CMTimeRangeShow(atTime_end); If (I = = 0) {[layerInstruction setOpacityRampFromStartOpacity: toEndOpacity 1.0:0.0 timeRange: atTime_end]; } else if (I = = compositionVideoTracks. Count - 1) {[layerInstruction setOpacityRampFromStartOpacity: 0.0 ToEndOpacity: 1.0 timeRange: atTime_begin]; ToEndOpacity} else {[layerInstruction setOpacityRampFromStartOpacity: 1.0:0.0 timeRange: atTime_end]; ToEndOpacity [layerInstruction setOpacityRampFromStartOpacity: 0.0:1.0 timeRange: atTime_begin]; } } [layerInstructions addObject:layerInstruction]; } videoCompositionInstruction.layerInstructions = layerInstructions;Copy the code

Transition effect:

2.4 Other ways to build a video synthesizer

Of course, creating “pass through Instruction “and “transition Instruction” one by one as described above is definitely not what we want, Described above way is to use the + (AVMutableVideoComposition *) videoComposition; Method to create AVMutableVideoComposition, returns an object of each attribute is null, so we need to add instructions one by one.

Apple also provides a + (AVMutableVideoComposition *) videoCompositionWithPropertiesOfAsset (AVAsset *) method of the asset, To add good video track AVMutableComposition, method returns AVMutableVideoComposition instance contains the set good attribute values, and is suitable for the time and according to its geometric properties and the properties of the present course specified resource video track’s instructions, Instructons and their layerInstructions have been created for us. We can directly extract fromLayerInstruction and toLayerInstruction in transition. These two ways of creating videoComposition are called bultin-in Compositor. Although they have the problem of insufficient space, However, the advantage is that Apple can automatically adapt the packaged interface to new technologies or devices, such as the ADAPTATION of HDR video files mentioned in WWDC2021. The built-in synthesizer will synthesize and output an HDR video containing HDR video.

Apple and the start of the iOS9.0 provided can be used for video CIFilter add filter effects such as fuzzy, color way to create AVMutableVideoComposition: + (AVMutableVideoComposition *)videoCompositionWithAsset:(AVAsset *)asset applyingCIFiltersWithHandler:(void (^) (AVAsynchronousCIImageFilteringRequest * request)) applier, but this way can only use system CIFilter support synthesis export of HDR, otherwise you need to modify the parameters.

CIFilter *filter = [CIFilter filterWithName:@"CIGaussianBlur"]; AVMutableVideoComposition *videocomposition = [AVMutableVideoComposition videoCompositionWithAsset:asset ApplyingCIFiltersWithHandler: ^ (AVAsynchronousCIImageFilteringRequest * _Nonnull request) {/ / to get source ciimage ciimage * source = request. SourceImage. ImageByClampingToExtent; / / to add filter [filter setValue: source forKey: kCIInputImageKey];  Float64 seconds = CMTimeGetSeconds(request.compositionTime);  CIImage *output = [filter.outputImage imageByCroppingToRect:request.sourceImage.extent]; Filter setValue:seconds * 10.0 forKey:kCIInputRadiusKey]; // Submit output [request finishWithImage:output context:nil]; }];Copy the code

Note: Use this method to create instances of AVMutableVideoComposition, the instructions of the array data type is a private class AVCoreImageFilterVideoCompositionInstruction, official document without any information, We can’t create or modify it and its layerInstructions, but we can use CIilter of CIAffineTransform to directly reorient the sourceImage and fix the orientation problem.

In iOS13.0 and provides + (AVMutableVideoComposition *) videoCompositionWithPropertiesOfAsset asset: (AVAsset *) PrototypeInstruction: (prototypeInstruction AVVideoCompositionInstruction *) method, we can create a prototype order ahead of time, set the background color, After calling this method to create AVMutableVideoComposition get a paragraphs instruction background color set good AVMutableVideoComposition instance.

But in order to realize completely custom transitions or custom synthesis, can do processing for each frame, these methods are not enough, apple in iOS7.0 AVMutableVideoComposition class customVideoCompositorClass attribute has been added, It requires a class that complies with the AVVideoCompositing protocol, and notice that it is a class that needs to be passed.

@property (nonatomic, retain, nullable) Class<AVVideoCompositing> customVideoCompositorClass;
Copy the code

The AVVideoCompositing protocol is defined as follows:

@protocol AVVideoCompositing<NSObject> @property (nonatomic, nullable) NSDictionary<NSString * id> *sourcePixelBufferAttributes; @property (nonatomic) NSDictionary<NSString *, id> *requiredPixelBufferAttributesForRenderContext; / / notify rendering context switch - (void) renderContextChanged (newRenderContext AVVideoCompositionRenderContext *); / / synthetic requests in the - (void) startVideoCompositionRequest (asyncVideoCompositionRequest AVAsynchronousVideoCompositionRequest *); / / cancel synthesis request - (void) cancelAllPendingVideoCompositionRequests;Copy the code

Which startVideoCompositionRequest method of AVAsynchronousVideoCompositionRequest object, With – (CVPixelBufferRef) sourceFrameByTrackID (trackID CMPersistentTrackID); CVPixelBufferRef (CVPixelBufferRef) method can be used to obtain the CVPixelBufferRef (CVPixelBufferRef) that needs to be synthesized for a certain track at the moment. Then we can customize the synthesis method by combining, core Image, Opengles or Metal to achieve rich effects.

Example of creating a custom synthesizer:

/ / returns the source PixelBuffer properties - (NSDictionary *) sourcePixelBufferAttributes {return @ {(nsstrings *)kCVPixelBufferPixelFormatTypeKey : [NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange], (NSString*)kCVPixelBufferOpenGLESCompatibilityKey : [NSNumber numberWithBool:YES]}; } / / return VideoComposition created PixelBuffer properties - (NSDictionary *) requiredPixelBufferAttributesForRenderContext {return @ { (NSString *)kCVPixelBufferPixelFormatTypeKey : [NSNumber numberWithUnsignedInt:kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange], (NSString*)kCVPixelBufferOpenGLESCompatibilityKey : [NSNumber numberWithBool:YES]}; } / / notify rendering context switch - (void) renderContextChanged: nonnull AVVideoCompositionRenderContext *) newRenderContext {} / / request - synthesis (void)startVideoCompositionRequest:(nonnull AVAsynchronousVideoCompositionRequest *)request { @autoreleasepool { Dispatch_async (_renderingQueue, ^ {the if (self shouldCancelAllRequests) {/ / used to cancel the synthetic [request finishCancelledRequest]; } else { NSError *err = nil; CVPixelBufferRef resultPixels = nil; / / get the current synthetic instruction AVVideoCompositionInstruction * currentInstruction = request. VideoCompositionInstruction; CVPixelBufferRef currentPixelBuffer = [request sourceFrameByTrackID:currentInstruction.trackID]; / / here can custom processing CVPixelBuffer resultPixels = [self handleByYourSelf: currentPixelBuffer]; if (resultPixels) { CFRetain(resultPixels); / / processed submitted after processing CVPixelBufferRef [request finishWithComposedVideoFrame: resultPixels]; CFRelease(resultPixels); } else { [request finishWithError:err]; }}}); }} / / cancel synthesis request - (void) cancelAllPendingVideoCompositionRequests {_shouldCancelAllRequests = YES; dispatch_barrier_async(_renderingQueue, ^() { self.shouldCancelAllRequests = NO; }); }Copy the code

The process can be decomposed into:

1. AVAsynchronousVideoCompositionRequest binding the current time series of the original frame, and the current time’s Instruction.
2. Received startVideoCompositionRequest: callback, and receives the Request.
3. According to the original frame and Instruction related mixing parameters, render the composite frame.
4. Call finishWithComposedVideoFrame, deliver the rendered frames.

In creating and using custom synthesis, we usually don’t use AVVideoCompositionInstruction, but observe the AVVideoCompositionInstruction AVVideoCompositionInstruction agreement, Create custom compositing instructions. Why is that?

Because our custom synthesizer class is passed to AVVideoComposition as a class, not as a property, the actual compositing is done inside the custom synthesizer, which means we don’t actually get the object of the custom synthesizer, So how do we tell the synthesizer about all the different combinations that we design? Instances can get AVVideoCompositionInstruction AVAsynchronousVideoCompositionRequest, So as long as we follow AVVideoCompositionInstruction agreement to create their own synthesis class instruction, we can add parameters passed to the synthesizer, complete the data flow.

AVVideoCompositionInstruction agreement content as follows:

@ protocol AVVideoCompositionInstruction < NSObject > @ the time range of the required / / directive @ property (nonatomic, readonly) CMTimeRange timeRange; @property (nonAtomic, readOnly) BOOL enablePostProcessing; // YES indicates that rendering frames from the same source buffer under the same combined instructions between two different and cities may produce different output frames. The NO value indicates that two compositions produce the same frame. @property (nonatomic, readonly) BOOL containsTweening; @property (nonatomic, ReadOnly, Nullable) NSArray<NSValue *> *requiredSourceTrackIDs; @property (nonatomic, readonly) CMPersistentTrackID passthroughTrackID; @ optional / / iOS15 new, used for synthesis of video metadata @ property (nonatomic, readonly) NSArray < > NSNumber * * requiredSourceSampleDataTrackIDs; @endCopy the code

Can be found by comparing AVVideoCompositionInstruction, the content of the class and AVVideoCompositionInstruction agreement, just more backgroundCoclor and layerInstructions attributes. Also can say apple just followed AVVideoCompositionInstruction agreement, created a we see AVVideoCompositionInstruction “class system”, And AVVideoCompositionInstruction class is just a new changes the background color of “design” and one for the synthesis of detail layerInstructions properties, do a simple example for us.

The synthesizer startVideoCompositionRequest: method shall be carried out every time? We’re talking about the beginning of the transition, and we’ve divided the instructions into “pass through” and “transition”, and in theory, if we specify that a certain instruction is “pass through”, we should pass it without asking for composition, Here requires a read-only passthroughTrackID in front, but as we follow the agreement to create their own “CustomMutableVideoCompositionInstruction”, then we can modify, After setting the passthroughTrackID, will not call when needed to “pass through” the segment track startVideoCompositionRequest: method.

After the test, using the + (AVMutableVideoComposition *) videoCompositionWithPropertiesOfAsset (AVAsset *) asset method of synthesis, The automatic as we create good AVMutableVideoCompositionLayerInstruction passthroughTrackID value is nil.

At this point, the previous question is easy to explain: why do we have these three classes?

To synthetic video requires a synthesizer (built-in or follow the custom synthesis agreement “), a class that is under the direction of synthetic agreement to specify the time range of segment, but we don’t need AVVideoCompositionLayerInstruction class to control the synthesis of details, so that the three are designed separately, AVVideoCompositionLayerInstruction just system examples in order to pass parameters and packaging container.

To sum up, the process of a complete custom video synthesizer should look like this:

1. throughAVAsset(s)buildAVMutableCompositionThe instancecomposition;
2. throughcompositioncreateAVMutableVideoCompositionThe instancevideoComposition;
3. Create custom synthesizer and custom synthesis instructions;
4. Set up thevideoCompositionthecustomVideoCompositorClassSet to custom synthesizer;
5. willvideoCompositioninstructionsIs replaced by a custom synthesis instruction, and the custom synthesis parameters of each section are configured respectively.
6. In a custom synthesizerstartVideoCompositionRequest:The method takes out the custom synthesis instruction and processes the synthesis of each frame according to the synthesis parameters of the instruction.

Finally, let’s explain the role of two new iOS15 properties — time base metadata composition: IOS15 supports custom time metadata composition. WWDC is an example where we have a set of GPS data that is time-stamped and synchronized with video, and if we want to use this GPS data to influence how frames are combined, The first step is to write the GPS data to a timed metadata track in the source movie (we can use AVAssetWriter), Then we can use new sourceSampleDataTrackIDs AVMutableVideoComposition attribute tells the synthesizer for synthesis of primitive data track ID, Set AVMutableVideoCompositionInstruction requiredSourceSampleDataTrackIDs attribute to tell it to track ID associated with the current instruction, the last in the custom synthesis for metadata to perform custom synthesis.

Examples of WWDC2021:

func startRequest(_ request: AVAsynchronousVideoCompositionRequest){ for trackID in request.sourceSampleDataTrackIDs { // You can also use sourceSampleBuffer(byTrackID: obtain CMSampleBuffer let metadata: AVTimedMetadataGroup? = request.sourceTimedMetadata(byTrackID: Using metadata, here.} Request. finish(withComposedVideoFrame: composedFrame)}Copy the code

Add text and stickers

Although we can handle CVPixelBuffer, it is not difficult to implement text stickers, but we have a simpler way to use the Core Animation framework we are familiar with, AVFoundation provides us with a way to connect the Core Animation from playing and exporting videos.

3.1 play – AVSynchronizedLayer

AVFoundation provides a specialized CALayer subclass, AVSynchronizedLayer, for synchronizing time with a given AVPlaverltem instance. This layer does not display anything by itself, but is used to coordinate time with the graph layer subtree. AVSynchronizedLayer is typically incorporated into the layer inheritance of the player view. AVSynchronizedLayer appears directly on top of the video layer, allowing you to add titles, stickers, or watermarks to the playing video and to synchronize with the player’s behavior.

In daily use of Core Animation, the time model depends on the system host, and the host’s time will never stop. However, the video Animation has its own time line and also supports stop, pause, rollback or fast forward effects. Therefore, it is not possible to directly add time-based Animation to a video using the time model of the system host. So beginTime can’t be set directly to 0.0 because it will be CACurrentMediaTime() representing the current host time. Apple also says that any CoreAnimation layer that has animation properties, If added as a child of AVSynchronizedLayer, the beginTime property of the animation should be set to a non-zero positive value so that the animation can be interpreted on the playerItem timeline. In addition, we must set removedOnCompletion = NO, otherwise the animation is a one-off.

Using GIF stickers as an example, you can get the code for each frame and the duration of each frame using the GIF mentioned above.

CAKeyframeAnimation *animation = [CAKeyframeAnimation animationWithKeyPath:@"contents"]; animation.beginTime = AVCoreAnimationBeginTimeAtZero; animation.removedOnCompletion = NO; Animation. KeyTimes = times; animation. KeyTimes = times; animation.values = images; animation.timingFunction = [CAMediaTimingFunction functionWithName:kCAMediaTimingFunctionLinear]; animation.duration = totalTime; animation.repeatCount = HUGE_VALF; // Create GIF layer _gifLayer = [CALayer layer]; _gifLayer.frame = CGRectMake(0, 0, 150, 150); [_gifLayer addAnimation:animation forKey:@"gif"]; AVPlayerViewController *playerVC = [[AVPlayerViewController alloc] init]; AVPlayerItem *playerItem = [[AVPlayerItem alloc] initWithAsset:_asset]; AVPlayer *player = [[AVPlayer alloc] initWithPlayerItem:playerItem]; playerVC.player = player; AVSynchronizedLayer AVSynchronizedLayer *asyLayer = [AVSynchronizedLayer synchronizedLayerWithPlayerItem:playerItem]; // Add GIF layer to asyLayer [asyLayer addSublayer:_gifLayer]; // Add asyLayer layer to player layer [playervc.view.layer addSublayer:asyLayer]; [player play];Copy the code

The effect is shown below:

At this point we can solve the problem of previewing AVAssetWriter before exporting it, and of course we want to use AVComposition, we want to look at the images we’re converting to video as individual textures, we want to use Core Animation to add CALayer to a video, After the preset residence time, replace the contents property. For this video, we can directly use the pure black video without any content as the video track from picture to video. (Honeycomb video editing framework design and iOS business practice 3.2.2 mentioned such design)

Add stickers character animation broadcast process effect, such as export video we need AVVideoCompositionCoreAnimationTool.

3.2 – AVVideoCompositionCoreAnimationTool derived

AVMutableVideoComposition has a AVVideoCompositionCoreAnimationTool type attribute animationTool, Build AVVideoCompositionCoreAnimationTool commonly used is + (instancetyp*)videoCompositionCoreAnimationToolWithPostProcessingAsVideoLayer:(CALayer *)videoLayer inLayer:(CALayer *)animationLayer; In which, we are required to pass two CALayer objects, a VideoLayer and an animationLayer. The official Apple document explains that the composite frame of the video is rendered together with the animationLayer to form the final video frame. The videoLayer should be in a sublayer of the animationLayer. The animationLayer should not come from or be added to any other layer tree.

// Create a merged layer CALayer *animationLayer = [CALayer layer]; CALayer *videoLayer = [CALayer layer]; [animationLayer addSublayer:videoLayer]; [animationLayer addSublayer:gifLayer]; / / create AVVideoCompositionCoreAnimationTool animationTool AVVideoCompositionCoreAnimationTool associated with videoComposition *animationTool = [AVVideoCompositionCoreAnimationTool videoCompositionCoreAnimationToolWithPostProcessingAsVideoLayer:videoLayer inLayer:animationLayer]; self.videoComposition.animationTool = animationTool;Copy the code

Note: After configuring the animationTool for videoComposition, you can no longer use the playItem for playback, AVVideoCompositionCoreAnimationTool can only be used for AVAssetExportSession and AVAssetReader this offline rendering, cannot be used for real-time rendering.

Four, export

4.1 Selecting a Video Cover

There is usually a function to select the video cover before exporting the video. The class AVAssetImageGenerator used to obtain the video cover has been described in the basic article.

The cover of the video is the first frame by default. After selecting the cover, the user can use AVAssetWriter to insert the corresponding video frame into the first frame of the video for a few frames, or merge the cover with the original video as a sticker. That’s all for AVFoundation. We should have a lot of ideas to solve the problem, in the actual attempt to choose the best or the most appropriate solution, after the selection of cover is completed, it is the category of video export.

4.2 AVAssetExportSession

The export part is relatively simple. You just need to pass the composition parameters created earlier to the instance used for export. The core class of the export section is AVAssetExportSession. Creating an AVAssetExportSession requires passing an asset and a default parameter presetName. The default parameter supports H.264, HEVC, Apple ProRes encoding, Different video resolutions are supported, and different video quality levels are supported, but not all presetNames are compatible with all asset and file types, so we should export the previous call below to check compatibility for a particular combination, as follows:

+ (void)determineCompatibilityOfExportPreset:(NSString *)presetName
withAsset:(AVAsset *)asset
outputFileType:(AVFileType)outputFileType
completionHandler:(void (^)(BOOL compatible))handler;
Copy the code

The important attributes of AVAssetExportSession are listed below.

@property (nonatomic, Copy, Nullable) AVFileType outputFileType; @property (nonatomic, copy, nullable) NSURL *outputURL; / / whether to use network to optimize @ property (nonatomic) BOOL shouldOptimizeForNetworkUse; / / export state @ property (nonatomic, readonly) AVAssetExportSessionStatus status; @property (nonatomic, copy, nullable) AVAudioMix *audioMix; // @property (nonatomic, copy, nullable) AVVideoComposition *videoComposition; @property (nonatomic) CMTimeRange timeRange; @property (nonatomic) long Long fileLengthLimit; @property (nonatomic, readOnly) CMTime maxDuration; @property (nonatomic, copy, nullable) NSArray<AVMetadataItem *> *metadata; @property (nonatomic, retain, Nullable) AVMetadataItemFilter *metadataItemFilter; @property (nonatomic, readonly) float progress;Copy the code

As you can see from the property list, AVAssetExportSession can not only set the file type, but also set the file size, length, range and other properties, and has several key properties for video composition described earlier. Audio blending AVAudioMix and video composition AVVideoComposition can be applied when exporting.

Exports are a relatively time-consuming operation, provides asynchronous AVAssetExportSession export interface – (void) exportAsynchronouslyWithCompletionHandler: (void (^) (void)) handler, Derived in the block, we can always get the progress schedule, according to the value of AVAssetExportSessionStatus at the same time, to observe whether export the result is in line with expectations.

Create AVAssetExportSession self.exportSession = [[AVAssetExportSession alloc] initWithAsset:composition presetName:AVAssetExportPresetHEVCHighestQuality]; Synthetic video / / configuration parameters of the self. ExportSession. VideoComposition = videoComposition; / / configure audio mixing parameters self. ExportSession. AudioMix = audioMix; / / configuration output url self. ExportSession. OutputURL = [NSURL fileURLWithPath: path]; / / output configuration file format the self exportSession. OutputFileType = AVFileTypeQuickTimeMovie; / / derived asynchronous [self exportSession exportAsynchronouslyWithCompletionHandler: ^ (void) {/ / monitor export state switch (self.exportSession.status) { case AVAssetExportSessionStatusCompleted: if (complete) { complete(); } break; case AVAssetExportSessionStatusFailed: NSLog(@"%@",self.exportSession.error); break; case AVAssetExportSessionStatusCancelled: NSLog(@"AVAssetExportSessionStatusCancelled"); break; default: break; } }];Copy the code

We also learned to use in front of the AVAssetReader and AVAssetWriter to code written to the file, including AVAssetReaderAudioMixOutput audioMix property, AVAssetReaderVideoCompositionOutput videoCompositionOutput property, so the whole composition of synthetic configuration are AVAssetReaderOutput as a parameter. Now we’ve learned two ways to export files: AVAssetExportSession and AVAssetWriter. If you simply export to a file format without demanding too much detail, using AVAssetExportSession is sufficient, The advantage of using AVAssetWriter is that we have complete control over the export process by specifying bit rate, frame rate, video frame size, color space, keyframe interval, video bit rate, H.264 profile, pixel aspect ratio and even the video encoder for export.

conclusion

To sum up this article, here’s a picture:

Avfoundationdemo features include:

• Audio:
  1. AVAudioPlayer Plays audio
  2. AVAudioEngineRecorder Audio recording
• Video:
  1. Video Mosaic synthesis, add background music, overlapping transition effect
  2. Video add stickers, text, GIF emojis
• Image:
  1. Ordinary pictures to video
  2. Live photo to video, GIF to video
• Other:
  1. Get file data sample format information

Reference documentation

AVFoundation WWDC2021-What’s New in AVFoundation WWDC2020-Edit and Play back HDR Video with AVFoundation Wwdc2020-what’s New in Camera Capture VideoLab – High performance and flexible iOS video editing framework design and business practice in iOS