The problems solved in this paper

This paper mainly uses MediaCodec hard encoder to encode audio and video collected by Android devices

  1. Package audio and video basic encoder
  2. Encapsulated audio encoder
  3. Encapsulated video encoder
  4. Retrofit example 2 using a newly packaged video encoder
  5. Use Camera to record video (YUV420SP) and save as video stream (H.264)
  6. Record audio (PCM) with AudioRecord and save as audio stream (AAC)
  7. Merge video and audio into one stream using MediaMuxer Mixer (H.264) (to be done)

The sample link

1. Packaged audio and video basic encoder

  1. Define the encoding interface classICodec:
interface ICodec {
    / / team
    fun putBuf(data: ByteArray, offset: Int, size: Int)
    // Process data
    fun dealWith(data: ByteArray)
    // Stop the thread
    fun stopWorld(a)
}
Copy the code
  1. Define the coding task threadBaseCodec:

Encoding audio and video frame by frame is a time-consuming task, so it is necessary to define a queue to store the data frames to be encoded.

BaseCodec is an abstract class and implements the ICodec interface, which continuously fetches data from the queue when a task is started, and dealWith is used for processing

abstract class BaseCodec : Thread(), ICodec {

    val inBlockingQueue = ArrayBlockingQueue<ByteArray>(30)


    override fun putBuf(data: ByteArray, offset: Int, size: Int) {
        val byteArray = ByteArray(size)
        System.arraycopy(data, offset, byteArray, 0, size)
        inBlockingQueue.put(byteArray)
    }

    var threadRunning = true;

    override fun run(a) {

        try {
            BaseCodecLoop1@ while (threadRunning) {
                val item = inBlockingQueue.take()
                dealWith(item)
            }
        } catch (e: Exception) {
            e.printStackTrace()
        }
    }

    override fun dealWith(data: ByteArray) {}

    override fun stopWorld(a) {
        inBlockingQueue.clear()
        threadRunning = false;
        interrupt()
        join(1000)}}const val SAMPLE_RATE_IN_HZ = 44100 // The sampling rate is 44.1khz
const val CHANNEL = AudioFormat.CHANNEL_IN_MONO // Mono, stereo: CHANNEL_IN_STEREO
const val AUDIO_FORMAT = AudioFormat.ENCODING_PCM_16BIT // Each sampling point is 16 bits
const val DEST_BIT_RATE = 128000 // Code rate
Copy the code
  1. encapsulationMediaCodecAudio and video coding basic classesBaseMediaCodec:

This class contains

  • Initialize MediaCodec for the mime type
/** * Mime type format * video/ AVC: H.264 * video/hevc: H.265 * audio/ MP4A-latm: aAC */Copy the code
  • Data entry
  • Send end mark
  • Data decoded out of the queue thread
  • CodecListenerCode ends the interface for sending back results
  • StopWorld Release interface

Two, packaging audio encoder

  • Initialization: The system is initialized by defaultaudio/mp4a-latmThe encoder
createCodec("audio/mp4a-latm")
val format = MediaFormat.createAudioFormat(mime, SAMPLE_RATE_IN_HZ, 1)
format.setInteger(MediaFormat.KEY_BIT_RATE, DEST_BIT_RATE)
/ / buffer maximum
val bufferSize = AudioRecord.getMinBufferSize(SAMPLE_RATE_IN_HZ, CHANNEL, AUDIO_FORMAT)
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, bufferSize)
format.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC)
configEncoderBitrateMode(format)
codec.start()
Copy the code
  • Receive encoded data
override fun bufferUpdate(buffer: ByteBuffer, bufferInfo: MediaCodec.BufferInfo) {
    buffer.position(bufferInfo.offset)
    buffer.limit(bufferInfo.offset + bufferInfo.size)
    val data = ByteArray(bufferInfo.size + 7)
    addADTStoPacket(data.data.size)
    buffer.get(data.7, bufferInfo.size) buffer.position(bufferInfo.offset) listener? .bufferUpdate(data)}/** * Add 7 bytes to the ADTS header */
private fun addADTStoPacket(packet: ByteArray, packetLen: Int) {
    val profile = 2 // AAC LC
    val freqIdx: Int = 4/ / 44.1 kHz
    val chanCfg = 2 // CPE
    packet[0] = 0xFF.toByte()
    packet[1] = 0xF9.toByte()
    packet[2] = ((profile - 1 shl 6) + (freqIdx shl 2) + (chanCfg shr 2)).toByte()
    packet[3] = ((chanCfg and 3 shl 6) + (packetLen shr 11)).toByte()
    packet[4] = ((packetLen and 0x7FF) shr 3).toByte()
    packet[5] = ((packetLen and 7 shl 5) + 0x1F).toByte()
    packet[6] = 0xFC.toByte()
}
Copy the code

Three, encapsulation video encoder

  • Initialize the

init {
    createCodec("video/avc")}fun setUpVideoCodec(width: Int, height: Int) {
    val format = MediaFormat.createVideoFormat(mime, width, height)
    format.setInteger(
        MediaFormat.KEY_COLOR_FORMAT,
        MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible
    )
    //width*height*frameRate*[0.1-0.2
    format.setInteger(MediaFormat.KEY_BIT_RATE, width * height * 3)
    format.setInteger(MediaFormat.KEY_FRAME_RATE, 30)
    // Output one keyframe per second. Set 0 to each keyframe
    format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)
// format.setInteger(
// MediaFormat.KEY_BITRATE_MODE,
/ / MediaCodecInfo. EncoderCapabilities. BITRATE_MODE_CBR / / abide by the code rate of user Settings
/ /)
    configEncoderBitrateMode(format)

// format.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface)
// codec.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE)
    codec.start()
}
Copy the code
  • Receiving code product
override fun bufferUpdate(buffer: ByteBuffer, bufferInfo: MediaCodec.BufferInfo){ listener? .bufferUpdate(buffer, bufferInfo) }Copy the code

Four, using the new package of video encoder transformation example 2

Next, modify the yuV code of the second article.

The steps are as follows: Sample code link

  • Defining mixer
  • Define encoder
  • Read the YUV file and start coding
  • inCodecListenerThe encoded data is received in the callback and the video file is saved using the MediaMuxer mixer
  • End of reading file, end of encoding
fun convertYuv2Mp4_2(context: Context) {
    val yuvPath = "${context.filesDir}/test.yuv"
    val saveMp4Path = "${context.filesDir}/test.mp4"
    File(saveMp4Path).deleteOnExit()

    // Define mixer: Output and save THE H.264 bit stream as MP4
    val mediaMuxer =
        MediaMuxer(
            saveMp4Path,
            MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4
        );
    var muxerTrackIndex = -1

    val videoEncoder = VideoEncoder()
    videoEncoder.setUpVideoCodec(1920.1080)
    videoEncoder.start()
    videoEncoder.setCodecListener(object : CodecListener {
        override fun formatUpdate(format: MediaFormat) {
            //step3.1 marks the arrival of new decoded data, where the video track is added to the mixer
            muxerTrackIndex = mediaMuxer.addTrack(format)
            mediaMuxer.start()
        }

        override fun bufferUpdate(buffer: ByteBuffer, bufferInfo: MediaCodec.BufferInfo) {
            mediaMuxer.writeSampleData(muxerTrackIndex, buffer, bufferInfo)
        }

        override fun bufferOutputEnd(a) {
            mediaMuxer.release()
            videoEncoder.stopWorld()
        }
    })


    val byteArray = ByteArray(1920 * 1080 * 3 / 2)
    var read = 0
    FileInputStream(yuvPath).use { fis ->
        while (true) {
            read = fis.read(byteArray)
            if (read == byteArray.size) {
                Thread.sleep(30)
                videoEncoder.putBuf(byteArray, 0, byteArray.size)
            } else {
                videoEncoder.putBufEnd()
                break}}}}Copy the code

Use Camera to record video and save it as video stream

The process is basically the same as the previous example, except to get the YUV data from the file modification to the camera real-time stream

  • Define encoderVideoEncoderMixer,MediaMuxer
  • Start the Camera and get the YUV data stream input encoder in the callback
  • The encoded data is retrieved in the encoder callback and saved locally using a mixer
  • Call when exiting the pagevideoEncoder.putBufEnd()Method to tell the encoder to terminate

Example code is as follows: link

var capture = false;

// Video encoding is MP4
val videoEncoder = VideoEncoder()
override fun initView(a) {

    videoEncoder.setUpVideoCodec(640.480)
    videoEncoder.start()
    binding.cameraview0.apply {
        cameraParams.facing = 1
        cameraParams.isScaleWidth = false
        cameraParams.oritationDisplay = 90
        cameraParams.previewSize.previewWidth = 640
        cameraParams.previewSize.previewHeight = 480
        cameraParams.isFilp = false

        addPreviewFrameCallback(object : CameraView.PreviewFrameCallback {
            override fun analyseData(data: ByteArray?).: Any {
                if (capture) {
                    videoEncoder.putBuf(data!!!!! .0.data.size)
                }
                return 0
            }
            override fun analyseDataEnd(p0: Any?). {}
        })
    }
    addLifecycleObserver(binding.cameraview0)


    val saveMp4Path = "${filesDir}/test.mp4"
    File(saveMp4Path).deleteOnExit()
    // Define mixer: Output and save THE H.264 bit stream as MP4
    val mediaMuxer =
        MediaMuxer(
            saveMp4Path,
            MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4
        )
    var muxerTrackIndex = -1

    videoEncoder.setCodecListener(object : CodecListener {
        override fun formatUpdate(format: MediaFormat) {
            muxerTrackIndex = mediaMuxer.addTrack(format)
            mediaMuxer.start()
        }

        override fun bufferUpdate(buffer: ByteBuffer, bufferInfo: MediaCodec.BufferInfo) {
            mediaMuxer.writeSampleData(muxerTrackIndex, buffer, bufferInfo)
        }

        override fun bufferOutputEnd(a) {
            mediaMuxer.release()
            videoEncoder.stopWorld()
        }
    })

    binding.cameraview0.setOnClickListener {
        DLog.d("Record:$capture") capture = ! capture } }override fun onDestroy(a) {
    videoEncoder.putBufEnd()
    super.onDestroy()
}
Copy the code

Use AudioRecord to record audio and save it as an audio stream

PCM raw data is captured using an AudioRecord and sent to a MediaCodec encoder for encoding as AAC

The sample code

= = = = = = = = = = =

== Deep attention points: When initializing AudioRecord recording audio must be initialized with the use of the number of channels when the encoder MediaFormat consistent with the input of the number of channels, can appear otherwise naked PCM data broadcast is normal, good aac encoding data broadcast is unusual, loud-mouthed, ji, and seems to accelerate, android devices is 1 in writing,,, Stereo is basically not supported

CHANNEL_IN_MONO: number of channels corresponding to mono 1 Audioformat. CHANNEL_IN_STEREO: number of channels corresponding to stereo 2Copy the code

= = = = = = = = = = =

The process is basically the same as video recording

    1. Create an AudioRecord to collect audio information
    1. Create an AudioEncoder to encode the PCM raw streams that are captured
    1. Create MediaMuxer to store encoded AAC data
// Store PCM raw data
val file_pcm = File("$filesDir/test.pcm")
// Direct FileOutputStream writes out the encoded AAC data
val file_mp3 = File("$filesDir/test.mp3")
// Use the mixer mediaMuxer to store encoded AAC data
val file_mp32 = File("$filesDir/test2.mp3")

binding.startRecord.setOnClickListener {
    file_pcm.deleteOnExit()
    file_mp3.deleteOnExit()
    file_mp32.deleteOnExit()
    thread {
        DLog.d("Start recording.")
        isCapture = true
        val bufferSize = AudioRecord.getMinBufferSize(
            SAMPLE_RATE_IN_HZ, CHANNEL, AUDIO_FORMAT
        )
        val audioRecord = AudioRecord(
            MediaRecorder.AudioSource.MIC,
            SAMPLE_RATE_IN_HZ,
            CHANNEL,
            AUDIO_FORMAT,
            bufferSize
        )

        audioRecord.startRecording()
        val data = ByteArray(bufferSize)
        val fos = FileOutputStream(file_pcm)
        val fos_mp3 = FileOutputStream(file_mp3)

        // Define mixer: output and save the encoded AAC as mp3
        val mediaMuxer =
            MediaMuxer(
                file_mp32.absolutePath,
                MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4
            )
        var muxerTrackIndex = -1

        val audioEncoder = AudioEncoder()
        audioEncoder.setCodecListener(object : CodecListener {
            override fun formatUpdate(format: MediaFormat) {
                super.formatUpdate(format)

            }

            override fun bufferUpdate(data: ByteArray) {
                super.bufferUpdate(data)
                DLog.d("bufferUpdate ${data.size}")
                fos_mp3.write(data.0.data.size)
            }

            override fun bufferOutputEnd(a) {
                super.bufferOutputEnd()
                audioEncoder.stopWorld()
                fos_mp3.flush()
                fos_mp3.close()
            }
        })
        audioEncoder.start()

        while (isCapture) {
            val len = audioRecord.read(data.0.data.size)
            audioEncoder.putBuf(data.0, len)
            fos.write(data.0, len);
        }

        fos.flush()
        fos.close()

        audioEncoder.putBufEnd()
        audioRecord.stop()
        DLog.d("End of recording.")}}Copy the code

Use the MediaMuxer mixer to merge video and audio into one stream