The most simple iOS push stream code, video capture, soft coding (FAAC, X264), hard coding (AAC, H264), beauty, FLV coding, RTMP protocol, updated code parsing, you want to learn the knowledge here, willing to understand live technology students come to see!!

Source: https://github.com/hardman/AWLive

Soft coding consists of three parts:

  1. Encode PCM/YUV data into AAC/H264 format
  2. Encapsulate AAC/H264 data into FLV format
  3. In addition, regardless of soft coding or hard coding, the data obtained in FLV format needs to be sent to the server through RTMP protocol.

This article covers part 1. The other two parts will be covered in subsequent articles.

According to the above introduction, the corresponding audio/video coding is AWSWFaacEncoder and AWSWX264Encoder.

These two classes are just shells wrapped in OC and are actually handled using libfaac and libx264.

Audio soft coding

The aw_faac.h and aw_faac.c files are a simple wrapper around the libfaac library. The two files are supposed to encapsulate a function to convert PCM data into AAC data.

How to use FAAC:

  1. Use faacEncOpen to enable the encoding environment to configure encoding properties.
  2. Encode using the faacEncEncode function.
  3. When finished, call faacEncClose to close the coding environment.

Following this step, look at the aw_faac.c file.

The first step of FAAC encapsulation: open the coding environment

Aw_faac_context /* aw_faac_context is a self-created structure used to aid AAC encoding. It stores the necessary data of the FAAC library, as well as some procedure variables. See the code in the demo to create and close it. It's simple and doesn't need to be explained here. */ static void aw_open_faac_enc_handler(aw_faac_context *faac_ctx){ // Input sample rate (44100) Number of channels (2) // Get the maximum input sample number (1024) maximum output bytes (2048) faac_ctx->faac_handler = faacEncOpen(faac_ctx->config.sample_rate, faac_ctx->config.channel_count, &faac_ctx->max_input_sample_count, &faac_ctx->max_output_byte_count); Faac_ctx ->max_input_byte_count = faac_ctx->max_input_sample_count * faac_ctx->config.sample_size / 8;if(! faac_ctx->faac_handler){ aw_log("[E] aac handler open failed");
        return; } faac_ctx->aac_buffer = aw_alloc(faac_ctx->max_output_byte_count); / / get configuration faacEncConfigurationPtr faac_config = faacEncGetCurrentConfiguration (faac_ctx - > faac_handler);if (faac_ctx->config.sample_size == 16) {
        faac_config->inputFormat = FAAC_INPUT_16BIT;
    }else if (faac_ctx->config.sample_size == 24) {
        faac_config->inputFormat = FAAC_INPUT_24BIT;
    }else if (faac_ctx->config.sample_size == 32) {
        faac_config->inputFormat = FAAC_INPUT_32BIT;
    }else{ faac_config->inputFormat = FAAC_INPUT_FLOAT; } // set faac_config->aacObjectType = LOW; // object type: LOW Main LTP faac_config->mpegVersion = MPEG4; Mpeg version: MPEG2 MPEG4 faac_config->useTns = 1; // faac_config->allowMidside = 0; // Whether to use mid/side encodingif(faac_ctx->config.bitrate){// The bitrate of each channel per second faac_config-> bitrate = faac_ctx->config.bitrate / faac_ctx->config.channel_count; } faacEncSetConfiguration(faac_ctx->faac_handler, faac_config); // Get the audio Specific config, which stores some of the key data in the AAC format. This data must be sent uint8_t * Audio_specific_data = NULL before all audio frames; unsigned long audio_specific_data_len = 0; faacEncGetDecoderSpecificInfo(faac_ctx->faac_handler, &audio_specific_data, &audio_specific_data_len); // Store the acquired audio Specific config data in faac_ctxif(audio_specific_data_len > 0) { faac_ctx->audio_specific_config_data = alloc_aw_data(0); memcpy_aw_data(&faac_ctx->audio_specific_config_data, audio_specific_data, (uint32_t)audio_specific_data_len); }} // For specific parameter configurations in the function, see:  //http://wenku.baidu.com/link?url=0E9GnSo7hZ-3WmB_eXz8EfnG8NqJJJtvjrVNW7hW-VEYWW-gYBMVM-CnFSicDE-veDl2tzfL-nu2FQ8msGcCO ALuT8VW1l_NjQL9Gvw5V6_Copy the code

Faac Encapsulation Step 2: Start coding

Extern void aw_encode_PCM_frame_2_aAC (aw_faAC_context * CTX, int8_t * pcM_data, extern void aw_encode_PCM_frame_2_aac, int8_t *pcm_data, Long len){// Determine the input parameterif(! pcm_data || len <= 0) { aw_log("[E] aw_encode_pcm_frame_2_aac params error");
        return; } // Empty encoded_aac_data, the encoded data will eventually be stored in this field each time, so empty first. reset_aw_data(&ctx->encoded_aac_data); /* The following code follows the first step"Turn on the coding environment"The maximum number of input subsections calculated in the function splits PCM_data into the appropriate size and encodes PCM data into AAC data using the faacEncEncode function. After the following code is executed, the encoded AAC data will be stored in the enCODED_AAC_data field. */ long max_input_count = ctx->max_input_byte_count; long curr_read_count = 0;do{
        long remain_count = len - curr_read_count;
        if (remain_count <= 0) {
            break;
        }
        long read_count = 0;
        if (remain_count > max_input_count) {
            read_count = max_input_count;
        }else{
            read_count = remain_count;
        }
        
        long input_samples = read_count * 8 / ctx->config.sample_size;
        int write_count = faacEncEncode(ctx->faac_handler, (int32_t * )(pcm_data + curr_read_count), (uint32_t)input_samples, (uint8_t *)ctx->aac_buffer, (uint32_t)ctx->max_output_byte_count);
        
        if (write_count > 0) {
            data_writer.write_bytes(&ctx->encoded_aac_data, (const uint8_t *)ctx->aac_buffer, write_count);
        }
        
        curr_read_count += read_count;
    } while (curr_read_count + max_input_count < len);
}
Copy the code

Faac package Step 3: Close the encoder:

extern void free_aw_faac_context(aw_faac_context **context_p){ ... // Close the faAC encoder faacEncClose(context->faac_handler); . }Copy the code

The above code is only used as the package of the FAAC encoder to open the encoder.

Aw_sw_faac_encoder. H/aw_SW_FAac_encoder. C

The function of this file is to convert the incoming PCM data into AAC data format through the function provided by AW_faac. c, and then convert the AAC data format into FLV format. How to convert the DATA into FLV format will be described in the subsequent article.

Take a look at the implementation of the aw_SW_faac_encoder. C file. The logic of this file is also very clear, and it implements the following functions:

  1. Start the encoder and create some process variables.
  2. Convert audio Specific Config data to FLV frame data.
  3. The received PCM data is converted into AAC data, and then the AAC data is converted into FLV audio data.
  4. Close the encoder.

As you can see, this kind of similar functional code is generally a trilogy: open – use – close.

Let’s look at the code. Audio soft encoder Step 1: Enable the encoder

/ * faac_config: */ extern void aw_sw_encoder_open_FAac_encoder (aw_faac_config *faac_config){// Check whether this function is enabledif (aw_sw_faac_encoder_is_valid()) {
        aw_log("[E] aw_sw_encoder_open_faac_encoder when encoder is already inited");
        return; } int32_t faac_cfg_len = sizeof(aw_faac_config);if(! s_faac_config) { s_faac_config = aw_alloc(faac_cfg_len); } memcpy(s_faac_config, faac_config, faac_cfg_len); S_faac_ctx = alloc_aw_FAac_context (*faac_config); }Copy the code

Audio soft coding Step 2: Convert audio Specific Config data to FLV frame data.

extern aw_flv_audio_tag *aw_sw_encoder_create_faac_specific_config_tag(){// Whether the encoder is enabledif(! aw_sw_faac_encoder_is_valid()){ aw_log("[E] aw_sw_encoder_create_faac_specific_config_tag when audio encoder is not inited");
        returnNULL; } // create audio specfic config record aw_flv_audio_tag *aac_tag = aw_sw_encoder_create_flv_audio_tag(&s_faac_ctx->config); // According to FLV: The aAC_packet_type corresponding to audio specific data is fixed to aw_FLv_A_aAC_package_type_aAC_sequence_header value is 0. The value is 1. Aac_tag -> aAC_packet_type = AW_FLv_A_aAC_package_type_AAC_sequence_header; aac_tag->config_record_data = copy_aw_data(s_faac_ctx->audio_specific_config_data); aac_tag->common_tag.timestamp = 0; aac_tag->common_tag.data_size = s_faac_ctx->audio_specific_config_data->size + 11 + aac_tag->common_tag.header_size;return aac_tag;
}
Copy the code

Audio soft encoder step 3: Convert received PCM data to AAC data, and then convert AAC data to FLV audio data

Len: PCM data length timestamp: FLV timestamp, RTMP protocol requires that the timestamp of the sent FLV audio and video frame should be uniformly increased, it is not allowed to send the last data timestamp is smaller than the first data timestamp. Aw_flv_audio_tag: Return type, generated FLV audio data (in FLV, each frame of data is called a tag). */ extern aw_flv_audio_tag *aw_sw_encoder_encode_faac_data(int8_t *pcm_data, long len, uint32_t timestamp){if(! aw_sw_faac_encoder_is_valid()) { aw_log("[E] aw_sw_encoder_encode_faac_data when encoder is not inited");
        returnNULL; Aw_encode_pcm_frame_2_aac (s_faAC_CTX, pcM_DATA, len); // Data encoded using FAAC will have a 7-byte ADTS header. RTMP does not accept this value and removes the first 7 bytes here. int adts_header_size = 7; // Remove 7 bytes from the ADTS headerif (s_faac_ctx->encoded_aac_data->size <= adts_header_size) {
        returnNULL; } // Encapsulate AAC data into FLV audio frames. FLV frames simply add some fixed information to the AAC data. Aac data is not encoded. aw_flv_audio_tag *audio_tag = aw_encoder_create_audio_tag((int8_t *)s_faac_ctx->encoded_aac_data->data + adts_header_size, s_faac_ctx->encoded_aac_data->size - adts_header_size, timestamp, &s_faac_ctx->config); audio_count++; // Return the resultreturn audio_tag;
}
Copy the code

Audio soft encoder Step 4: Turn off the encoder

extern void aw_sw_encoder_close_faac_encoder(){// Avoid repeated closingif(! aw_sw_faac_encoder_is_valid()) { aw_log("[E] aw_sw_encoder_close_faac_encoder when encoder is not inited");
        return; } // if aw_faac_context is used, the FAAC encoding environment is closed. free_aw_faac_context(&s_faac_ctx); // Release configuration dataif(s_faac_config) { aw_free(s_faac_config); s_faac_config = NULL; }}Copy the code

So far, audio soft encoder is over. Converting PCM data into FLV audio frames has been successfully implemented.

Video soft coding is introduced below. The routine is consistent with the audio coding, and the corresponding video soft coding is the encapsulation of the X264 library. The file is in aw_x264.h/aw_x264.c.

It implements the following functions:

  1. Initialize x264 parameters and open the encoding environment
  2. coding
  3. Close the coding environment.

X264 encapsulation step 1: initialize x264 parameters, open the coding environment

Aw_x264_context is a custom structure used to store x264 encoding important attributes and procedure variables. */ extern aw_x264_context *alloc_aw_x264_context(aw_x264_config config){ aw_x264_context *ctx = aw_alloc(sizeof(aw_x264_context)); memset(ctx, 0, sizeof(aw_x264_context)); // Data Data is I420 by defaultif(! config.input_data_format) { config.input_data_format = X264_CSP_I420; } // create handler memcpy(&ctx->config, &config, sizeof(aw_x264_config)); x264_param_t *x264_param = NULL; / / x264 parameters, specific please refer to: http://blog.csdn.net/table/article/details/8085115 aw_create_x264_param (CTX, & x264_param); // Enable aw_open_x264_handler(CTX, x264_param); aw_free(x264_param); // Create pic_in, a space inside X264 used to store input image data. x264_picture_t *pic_in = aw_alloc(sizeof(x264_picture_t)); x264_picture_init(pic_in); //aw_stride is a macro that converts the width of the video to a multiple of 16. If it is not a multiple of 16, sometimes coding fails (missing colors, etc.). int alloc_width = aw_stride(config.width); x264_picture_alloc(pic_in, config.input_data_format, alloc_width, config.height); pic_in->img.i_csp = config.input_data_format; //i_stride indicates the length of the newline step, which is related to the number of planes and the format. It is used internally in X264 to determine how much data needs to be newline. // The yuV data format was introduced in Chapter 2.if(config.input_data_format == X264_CSP_NV12) {// Nv12 data contains 2 planes, the first plane stores y data size is width * height, // the second plane stores UV data, I_stride [0] = alloc_width; width * (height / 2) pic_in->img.i_stride[0] = alloc_width; pic_in->img.i_stride[1] = alloc_width; pic_in->img.i_plane = 2; }else if(config. Input_data_format = = X264_CSP_BGR | | config. Input_data_format = = X264_CSP_RGB) {/ / RGB data contains a plane, The data length is width * 3 * height. pic_in->img.i_stride[0] = alloc_width * 3; pic_in->img.i_plane = 1; }else ifInput_data_format == X264_CSP_BGRA){pic_in->img.i_stride[0] = alloc_width * 4; pic_in->img.i_plane = 1; }else{//YUV420 //YUV420 is the I420 format. // The first plane stores y data size is width * height // the second plane stores U data size is width * height // the third plane stores V data size, Width * height / 4 pic_in->img.i_stride[0] = width; pic_in->img.i_stride[1] = alloc_width / 2; pic_in->img.i_stride[2] = alloc_width / 2; pic_in->img.i_plane = 3; CTX -> pic_IN = pic_IN; CTX ->pic_in = pic_in; ctx->pic_out = aw_alloc(sizeof(x264_picture_t)); x264_picture_init(ctx->pic_out); CTX -> encoDED_h264_data = alloc_aw_data(0); ctx->sps_pps_data = alloc_aw_data(0); The RTMP protocol requires that all FLV video frames must be sent before the SPS PPS data. It stores some key attributes of h264 video. // See demo for more details. aw_encode_x264_header(ctx);return ctx;
}
Copy the code

X264 encapsulation Step 2: Start coding

Extern void AW_encode_YUv_frame_2_x264 (aw_x264_context * AW_ctx, int8_t *yuv_frame, int len){if(len > 0 &&yuv_frame) {// Fill the video data into pic_in, as described above. X264 needs to do this. int actual_width = aw_stride(aw_ctx->config.width); // Data is saved to pic_inif (aw_ctx->config.input_data_format == X264_CSP_NV12) {
            aw_ctx->pic_in->img.plane[0] = (uint8_t *)yuv_frame;
            aw_ctx->pic_in->img.plane[1] = (uint8_t *)yuv_frame + actual_width * aw_ctx->config.height;
        }else if(aw_ctx->config.input_data_format == X264_CSP_BGR || aw_ctx->config.input_data_format == X264_CSP_RGB){
            aw_ctx->pic_in->img.plane[0] = (uint8_t *)yuv_frame;
        }else if(aw_ctx->config.input_data_format == X264_CSP_BGRA){
            aw_ctx->pic_in->img.plane[0] = (uint8_t *)yuv_frame;
        }else{//YUV420 aw_ctx->pic_in->img.plane[0] = (uint8_t *)yuv_frame; aw_ctx->pic_in->img.plane[1] = (uint8_t *)yuv_frame + actual_width * aw_ctx->config.height; aw_ctx->pic_in->img.plane[2] = (uint8_t *)yuv_frame + actual_width * aw_ctx->config.height * 5 / 4; } //x264 encoding, The encoded data is stored in aw_CTx -> NAL x264_encoder_encode(AW_CTx -> X264_handler, & AW_CTX -> NAL, & AW_CTx ->nal_count, AW_CTx ->pic_in, aw_ctx->pic_out); aw_ctx->pic_in->i_pts++; } // Save the encoded data to encoded_h264_data, which stores the encoded H264 video frames. reset_aw_data(&aw_ctx->encoded_h264_data);if (ctx->nal_count > 0) {
        int i = 0;
        for(; i < ctx->nal_count; i++) { data_writer.write_bytes(&ctx->encoded_h264_data, ctx->nal[i].p_payload, ctx->nal[i].i_payload); }}}Copy the code

X264 encapsulation step 3: close the coding environment.

/* Release pic_in, pic_out, Extern void free_aw_x264_context(aw_x264_context **ctx_p){aw_x264_context * CTX = *ctx_p;if(CTX) {// Release pic_INif(ctx->pic_in) { x264_picture_clean(ctx->pic_in); aw_free(ctx->pic_in); ctx->pic_in = NULL; } // Release pic_outif(ctx->pic_out) { aw_free(ctx->pic_out); ctx->pic_out = NULL; }... / / close the handlerif(ctx->x264_handler) { x264_encoder_close(ctx->x264_handler); ctx->x264_handler = NULL; }... }}Copy the code

The above code simply encapsulates the X264 coding flow. Aw_sw_x264_encoder. H/aw_SW_x264_encoder. C

It implements the following functions:

  1. Encode the received YUV data into H264 format.
  2. Generate FLV video frames containing SPS/PPS data.
  3. Convert h264 data into FLV video data.
  4. Close the encoder.

Video soft encoder step 1: receive YUV data and encode it into H264 format.

// Open the encoder, which is based on aw_x264, seal a layer. extern void aw_sw_encoder_open_x264_encoder(aw_x264_config *x264_config){if (aw_sw_x264_encoder_is_valid()) {
        aw_log("[E] aw_sw_encoder_open_video_encoder when video encoder is not inited");
        return;
    }
    
    int32_t x264_cfg_len = sizeof(aw_x264_config);
    if(! s_x264_config) { s_x264_config = aw_alloc(x264_cfg_len); } memcpy(s_x264_config, x264_config, x264_cfg_len); s_x264_ctx = alloc_aw_x264_context(*x264_config); }Copy the code

Video soft encoder Step 2: Generate FLV video frames containing SPS/PPS data

// Extern aw_FLv_video_tag * in aw_encode_flv.h/ aw_encode_flV. caw_sw_encoder_create_x264_sps_pps_tag() {if(! aw_sw_x264_encoder_is_valid()){ aw_log("[E] aw_sw_encoder_create_video_sps_pps_tag when video encoder is not inited");
        returnNULL; Tag aw_flv_video_tag *sps_pps_tag = aw_sw_encoder_create_FLv_video_tag (); // sps_ppS_tag ->frame_type = aw_flv_v_frame_type_key; // packagetypeSps_pps_tag -> H264_package_type = aw_FLv_V_H264_packet_type_seq_header; Sps_pps_tag -> h264_comPOSItion_time = 0; // Copy SPS/PPS data generated in AW_x264 to tag spS_ppS_TAG -> config_recorD_data = copy_AW_data (s_x264_CTx ->sps_pps_data); Sps_pps_tag ->common_tag.timestamp = 0; // FLV tag length is: Header size + Data header(11 bytes) + Data length (later) SPS_PPS_TAG ->common_tag. datA_size = S_x264_CTX -> SPS_pps_data ->size + 11 + sps_pps_tag->common_tag.header_size;return sps_pps_tag;
}
Copy the code

Video soft encoder step 3: Convert h264 data into FLV video data.

// The collected video YUV data, Extern aw_FLv_video_tag * AW_SW_encoder_encode_x264_data (INT8_t *yuv_data, long len, Uint32_t timeStamp){// Whether encoding is enabledif(! aw_sw_x264_encoder_is_valid()) { aw_log("[E] aw_sw_encoder_encode_video_data when video encoder is not inited");
        returnNULL; } aw_encode_YUV_frame_2_x264 (s_x264_ctx, yuv_data, (int32_t)len); // Whether the data can be retrieved after encodingif (s_x264_ctx->encoded_h264_data->size <= 0) {
        returnNULL; } // convert h264 data to FLV tag x264_picture_t *pic_out = s_x264_ctx->pic_out; aw_flv_video_tag *video_tag = aw_encoder_create_video_tag((int8_t *)s_x264_ctx->encoded_h264_data->data, s_x264_ctx->encoded_h264_data->size, timeStamp, (uint32_t) ((pic_out - > i_pts pic_out - > i_dts) * 1000.0 / s_x264_ctx - > config. The FPS), pic_out - > b_keyframe); .return video_tag;
}

Copy the code

Video soft encoder Step 4: Close the encoder

// Close the encoder extern voidaw_sw_encoder_close_x264_encoder(){// Avoid repeated closingif(! aw_sw_x264_encoder_is_valid()) { aw_log("[E] aw_sw_encoder_close_video_encoder s_faac_ctx is NULL");
        return; } // Release configurationif(s_x264_config) { aw_free(s_x264_config); s_x264_config = NULL; } // Free context free_AW_x264_context (&s_x264_ctx); }Copy the code

At this point, the soft coding code is introduced. Can be invoked by AWSWFaacEncoder/AWSWX264Encoder soft encoder of the above, to the upper provides a consistent interface.

To sum up, the contents of the soft encoder are as follows:

  1. Third-party encoder: libfaac/libx264
  2. Third-party encoder package: AW_faac. h/ AW_FAac. c, AW_X264.h/AW_X264.c
  3. C, aw_SW_x264_encoder. H/aw_SW_x264_encoder. C
  4. The top abstract: AWSWFaacEncoder/AWSWX264Encoder

Points to note during coding:

  1. Note the audio Specific config and SPS/PPS data fetch, without which the server will not be able to identify audio/video frames.
  2. Note to remove the ADTS header after faAC coding.
  3. If the width of the input resolution is not a multiple of 16, the x264 encoder needs to expand it to a multiple of 16, otherwise the coding may have problems (color loss, UV confusion).

The article lists

  1. 1 hour to learn: the simplest iOS live push stream (A) project introduction
  2. 1 hour learning: The simplest iOS Live Streams (II) Code Architecture Overview
  3. 1 hour to learn: the simplest iOS live push stream (3) Using the system interface capture audio and video
  4. 1 hour learning: the simplest iOS live push stream (4) how to use GPUImage, how to beauty
  5. 1 hour learning: the simplest iOS live stream push (5) yuV, PCM data introduction and acquisition
  6. 1 hour to learn: the simplest iOS live push stream (6) H264, AAC, FLV introduction
  7. Learn in 1 hour: The simplest iOS Live Stream push (7) H264 / AAC hard coding
  8. 1 hour to learn: The simplest iOS live Push stream (8) H264 / AAC soft coding
  9. Learn in 1 hour: The simplest iOS live stream push (nine) FLV encoding with audio and video timestamp synchronization
  10. 1 hour learning: The simplest iOS live push stream (10) Librtmp usage introduction
  11. Introduction to SPS&PPS and AudioSpecificConfig (End)