[sound Ming]
First of all, this series of articles are based on their own understanding and practice, there may be wrong places, welcome to correct.
Secondly, this is an introductory series, covering only enough knowledge, and there are many blog posts on the Internet for in-depth knowledge. Finally, in the process of writing the article, I will refer to the articles shared by others and list them at the end of the article, thanking these authors for their sharing.
Code word is not easy, reproduced please indicate the source!
| Tutorial code: [Making portal】 |
|---|
directory
First, Android audio and video hard decoding:
- 1. Basic knowledge of audio and video
- 2. Audio and video hard decoding process: packaging basic decoding framework
- 3. Audio and video playback: audio and video synchronization
- 4, audio and video unencapsulation and packaging: generate an MP4
2. Use OpenGL to render video frames
- 1. Preliminary understanding of OpenGL ES
- 2. Use OpenGL to render video images
- 3, OpenGL rendering multi-video, picture-in-picture
- 4. Learn more about EGL of OpenGL
- 5, OpenGL FBO data buffer
- 6, Android audio and video hardcoding: generate an MP4
Android FFmpeg audio and video decoding
- 1, FFmpeg SO library compilation
- 2. Android introduces FFmpeg
- 3, Android FFmpeg video decoding playback
- 4, Android FFmpeg+OpenSL ES audio decoding playback
- 5, Android FFmpeg+OpenGL ES play video
- Android FFmpeg Simple Synthesis MP4: Video unencapsulation and Reencapsulation
- 7, Android FFmpeg video encoding
You can read about it in this article
This article will combine the knowledge of MediaCodec, OpenGL, EGL, FBO, MediaMuxer introduced in the previous series to realize the process of decoding, editing, encoding, and finally saving a video as a new video.
Finally an article by the end of this chapter, in front of a series of articles, surrounding the OpenGL, introduces how to use OpenGL to realize video image rendering and display, and how to video image editing, with the above foundation, we certainly want to have a good video editor preserved, realize the whole process of editing the closed loop, This article will fill in the last link.
MediaCodec encoder package
In this article, we introduce how to use MediaCodec, a hard codec tool provided by Android native, to decode video. MediaCodec also hardcodes audio and video.
Let’s take a look at the official codec data flow chart
- The decoding process
During decoding, a free input buffer is queried through the dequeueInputBuffer, the undecoded data is pressed into the decoder through the queueInputBuffer, and the decoded data is obtained through the dequeueOutputBuffer.
- Encoding process
In fact, the encoding process is basically the same as the decoding process. The difference is that the data pushed into the dequeueInputBuffer input buffer is unencoded data, whereas the data that is passed through the dequeueOutputBuffer is encoded data.
According to the gourd, imitate the process of packaging decoder, to package a basic encoder BaseEncoder.
1. Define encoder variables
See BaseEncoder for the complete code
abstract class BaseEncoder(muxer: MMuxer, width: Int = -1, height: Int = -1) : Runnable {
private val TAG = "BaseEncoder"
// The target video width is valid only when the video is encoded
protected val mWidth: Int = width
// The target video is high, which is valid only when the video is encoded
protected val mHeight: Int = height
// Mp4 synthesizer
private var mMuxer: MMuxer = muxer
// The thread runs
private var mRunning = true
// Encode the frame sequence
private var mFrames = mutableListOf<Frame>()
/ / encoder
private lateinit var mCodec: MediaCodec
// Current encoding frame information
private val mBufferInfo = MediaCodec.BufferInfo()
// Encode the output buffer
private var mOutputBuffers: Array<ByteBuffer>? = null
// Encode input buffer
private var mInputBuffers: Array<ByteBuffer>? = null
private var mLock = Object()
// Whether the encoding ends
private var mIsEOS = false
// Encode status listener
private var mStateListener: IEncodeStateListener? = null
/ /...
}
Copy the codeFirst, this is an abstract class and inherits Runnable, which defines the internal variables that need to be used. Basic and decoding similar.
Note that the width and height here are only valid for video, MMuxer is the Mp4 packaging tool defined previously in Mp4 Repackaging. There is also a cache queue, mFrames, for caching frame data that needs to be encoded.
For details on how to write data to MP4, see MP4 Repackage.
One of the frames is defined as follows:
class Frame {
// Unencoded data
var buffer: ByteBuffer? = null
// Unencoded data information
var bufferInfo = MediaCodec.BufferInfo()
private set
fun setBufferInfo(info: MediaCodec.BufferInfo) {
bufferInfo.set(info.offset, info.size, info.presentationTimeUs, info.flags)
}
}
Copy the codeThe encoding process is relatively simple compared with the decoding process, which is divided into three steps:
- Initializing the encoder
- Press data into the encoder
- Take the data from the encoder and press it into mp4
2. Initialize the encoder
abstract class BaseEncoder(muxer: MMuxer, width: Int = -1, height: Int = -1) : Runnable {
// Omit other code......
init {
initCodec()
}
/** * Initializes the encoder */
private fun initCodec(a) {
mCodec = MediaCodec.createEncoderByType(encodeType())
configEncoder(mCodec)
mCodec.start()
mOutputBuffers = mCodec.outputBuffers
mInputBuffers = mCodec.inputBuffers
}
/** * Encoding type */
abstract fun encodeType(a): String
/** * subclass encoder */
abstract fun configEncoder(codec: MediaCodec)
/ /...
}
Copy the codeThere are two virtual functions defined that subclasses must implement. One is used to configure the encoding type of audio and video. For example, if the encoding type of video is H264, the encoding type is “Video/AVC”. The audio encoding type of AAC is “Audio/MP4A-LATm”.
Depending on the encoding type obtained, an encoder can be initialized.
Next, we call configEncoder to configure the encoding parameters in the subclass. We won’t go into details here until we define the audio and video encoding subclass.
2. Start the coding cycle
abstract class BaseEncoder(muxer: MMuxer, width: Int = -1, height: Int = -1) : Runnable {
// Omit other code......
override fun run(a) {
loopEncode()
done()
}
/** * loop encoding */
private fun loopEncode(a) {
while(mRunning && ! mIsEOS) {val empty = synchronized(mFrames) {
mFrames.isEmpty()
}
if (empty) {
justWait()
}
if (mFrames.isNotEmpty()) {
val frame = synchronized(mFrames) {
mFrames.removeAt(0)}if (encodeManually()) {
// [1. Data compression encoding]
encode(frame)
} else if (frame.buffer == null) { // If it is automatic encoding (such as video), when it encounters the end frame, it will end directly
// This may only be used with encoders receiving input from a Surface
mCodec.signalEndOfInputStream()
mIsEOS = true}}// [2. Select the encoded data]
drain()
}
}
/ /...
}
Copy the codeThe loop code is placed in the run method of Runnable.
In loopEncode, you combine the 2 (pressing data) and 3 (fetching data) mentioned earlier. The logic is simple.
Determine whether the unencoded cache queue is empty, if so, the thread suspends and enters the wait. Otherwise encode the data, and retrieve the data.
There are two things to note:
- The encoding process for audio and video is slightly different
Audio coding requires us to press the data into the encoder to realize the data coding.
During video coding, the Surface can be bound to OpenGL, and the system will automatically fetch data from the Surface to achieve automatic coding. That is, instead of manually pressing in the data themselves, the user can simply fetch the data from the output buffer.
Therefore, we define a virtual function that lets subclasses control whether or not data needs to be manually pressed. The default is true: manual pressing.
These two forms are referred to as manual encoding and automatic encoding in the following sections
abstract class BaseEncoder(muxer: MMuxer, width: Int = -1, height: Int = -1) : Runnable {
// Omit other code......
/** * Whether to manually encode * Video: false audio: true ** Note: Video encoding is done automatically by Surface, MediaCodec; Audio data needs to be pressed into the encoding buffer by the user to complete the encoding */
open fun encodeManually(a) = true
/ /...
}
Copy the code- The end of the coding
During the encoding process, if the buffer in the Frame is null, the encoding is considered complete and no data needs to be pressed in. At this point, there are two ways to tell the encoder to stop coding.
First, an empty data is forced through queueInputBuffer and the data type flag is set to mediacodec.buffer_FLAG_end_OF_stream. Details are as follows:
mCodec.queueInputBuffer(index, 0.0,
frame.bufferInfo.presentationTimeUs,
MediaCodec.BUFFER_FLAG_END_OF_STREAM)
Copy the codeThe second is to send the end signal through signalEndOfInputStream.
We already know that the video is automatically encoded, so you can’t end the encoding in the first way, you can only end the encoding in the second way.
Audio is manually encoded and can be terminated in the first way.
A pit
The test found that after the signalEndOfInputStream at the end of video coding, the data of the end coding mark was not obtained when the encoding data output was obtained. Therefore, in the above code, if the automatic encoding is performed, when the Frame buffer is judged to be empty, MIsEOF is set to true, leaving the coding process.
3. Manual coding
abstract class BaseEncoder(muxer: MMuxer, width: Int = -1, height: Int = -1) : Runnable {
// Omit other code......
/** ** encoding */
private fun encode(frame: Frame) {
val index = mCodec.dequeueInputBuffer(-1)
/* Input data to the encoder */
if (index >= 0) {
valinputBuffer = mInputBuffers!! [index] inputBuffer.clear()if(frame.buffer ! =null) {
inputBuffer.put(frame.buffer)
}
if (frame.buffer == null || frame.bufferInfo.size <= 0) { // If the value is less than or equal to 0, it is the end of the audio signal
mCodec.queueInputBuffer(index, 0.0,
frame.bufferInfo.presentationTimeUs, MediaCodec.BUFFER_FLAG_END_OF_STREAM)
} else {
mCodec.queueInputBuffer(index, 0, frame.bufferInfo.size,
frame.bufferInfo.presentationTimeUs, 0) } frame.buffer? .clear() } }/ /...
}
Copy the codeAs with decoding, an available input buffer index is queried and data is pushed into the input buffer.
In this case, it determines whether to end the encoding or not, and presses the encoding end flag into the input buffer
4. Pull data
After pressing a frame into the encoder, the drain method, as the name implies, drains all of the data in the encoder’s output buffer. So here is a while loop until the output buffer has no data mediacodec.info_try_AGAIN_later, or the encoding ends mediacodec.buffer_FLAG_end_OF_stream.
abstract class BaseEncoder(muxer: MMuxer, width: Int = -1, height: Int = -1) : Runnable {
// Omit other code......
/** ** squeeze the encoded output data */
private fun drain(a) {
loop@ while(! mIsEOS) {val index = mCodec.dequeueOutputBuffer(mBufferInfo, 0)
when (index) {
MediaCodec.INFO_TRY_AGAIN_LATER -> break@loop
MediaCodec.INFO_OUTPUT_FORMAT_CHANGED -> {
addTrack(mMuxer, mCodec.outputFormat)
}
MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED -> {
mOutputBuffers = mCodec.outputBuffers
}
else- > {if (mBufferInfo.flags == MediaCodec.BUFFER_FLAG_END_OF_STREAM) {
mIsEOS = true
mBufferInfo.set(0.0.0, mBufferInfo.flags)
}
if (mBufferInfo.flags == MediaCodec.BUFFER_FLAG_CODEC_CONFIG) {
// SPS or PPS, which should be passed by MediaFormat.
mCodec.releaseOutputBuffer(index, false)
continue@loop
}
if(! mIsEOS) { writeData(mMuxer, mOutputBuffers!! [index], mBufferInfo) } mCodec.releaseOutputBuffer(index,false)}}}}/** * Configuremp4 audio and video track */
abstract fun addTrack(muxer: MMuxer, mediaFormat: MediaFormat)
/** * Writes audio and video data to mp4 */
abstract fun writeData(muxer: MMuxer, byteBuffer: ByteBuffer, bufferInfo: MediaCodec.BufferInfo)
/ /...
}
Copy the codeA very important point
When mCodec. DequeueOutputBuffer returned is MediaCodec INFO_OUTPUT_FORMAT_CHANGED, explain encoding parameters format has been generated (such as video bit rate, frame rate, SPS/PPS frame information, etc.), This information needs to be written to the media track corresponding to mp4 (here addTrack is used to configure the audio and video encoding format in the subclass) before the encoded data can be written to the corresponding media channel via MediaMuxer.
5. Exit encoding to release resources
abstract class BaseEncoder(muxer: MMuxer, width: Int = -1, height: Int = -1) : Runnable {
// Omit other code......
/** * End of encoding, whether resource */
private fun done(a) {
try {
release(mMuxer)
mCodec.stop()
mCodec.release()
mRunning = falsemStateListener? .encoderFinish(this)}catch (e: Exception) {
e.printStackTrace()
}
}
/** * Releases subclass resources */
abstract fun release(muxer: MMuxer)
/ /...
}
Copy the codeCall the virtual function release in the subclass. The subclass needs to release the corresponding media channel in MP4 according to its media type.
6. Some externally called methods
abstract class BaseEncoder(muxer: MMuxer, width: Int = -1, height: Int = -1) : Runnable {
// Omit other code......
/** * Queues a frame of data to wait for encoding */
fun encodeOneFrame(frame: Frame) {
synchronized(mFrames) {
mFrames.add(frame)
notifyGo()
}
// Delay a bit to avoid dropping frames
Thread.sleep(frameWaitTimeMs())
}
/** * Notification end encoding */
fun endOfStream(a) {
Log.e("ccccc"."endOfStream")
synchronized(mFrames) {
val frame = Frame()
frame.buffer = null
mFrames.add(frame)
notifyGo()
}
}
/** * sets the status listener */
fun setStateListener(l: IEncodeStateListener) {
this.mStateListener = l
}
/** * queue time per frame */
open fun frameWaitTimeMs(a) = 20L
/ /...
}
Copy the codeIt is important to note that a default delay of 20ms is made after the data is pushed into the queue, and the subclasses can change the time by overwriting the frameWaitTimeMs method.
One is to avoid audio decoding too fast, resulting in too much data accumulation, audio in the subclass reset wait to 5ms, see subclass AudioEncoder code.
Another reason is that the system automatically obtains Surface data for the video. If the decoding data is refreshed too fast, it may cause frame miss. The default 20ms is used here.
So there’s a crude delay here, but it’s not the best solution.
Second, video encoder
With basic encapsulation, it is not so easy to write a video encoder?
Backhand pasted a video encoder:
const val DEFAULT_ENCODE_FRAME_RATE = 30
class VideoEncoder(muxer: MMuxer, width: Int, height: Int): BaseEncoder(muxer, width, height) {
private val TAG = "VideoEncoder"
private var mSurface: Surface? = null
override fun encodeType(a): String {
return "video/avc"
}
override fun configEncoder(codec: MediaCodec) {
if (mWidth <= 0 || mHeight <= 0) {
throw IllegalArgumentException("Encode width or height is invalid, width: $mWidth, height: $mHeight")}val bitrate = 3 * mWidth * mHeight
val outputFormat = MediaFormat.createVideoFormat(encodeType(), mWidth, mHeight)
outputFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitrate)
outputFormat.setInteger(MediaFormat.KEY_FRAME_RATE, DEFAULT_ENCODE_FRAME_RATE)
outputFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1)
outputFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface)
try {
configEncoderWithCQ(codec, outputFormat)
} catch (e: Exception) {
e.printStackTrace()
// Catch the exception and set it to the default BITRATE_MODE_VBR
try {
configEncoderWithVBR(codec, outputFormat)
} catch (e: Exception) {
e.printStackTrace()
Log.e(TAG, "Failed to configure video encoder")
}
}
mSurface = codec.createInputSurface()
}
private fun configEncoderWithCQ(codec: MediaCodec, outputFormat: MediaFormat) {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
// This part of the phone does not support BITRATE_MODE_CQ mode, which may be abnormal
outputFormat.setInteger(
MediaFormat.KEY_BITRATE_MODE,
MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_CQ
)
}
codec.configure(outputFormat, null.null, MediaCodec.CONFIGURE_FLAG_ENCODE)
}
private fun configEncoderWithVBR(codec: MediaCodec, outputFormat: MediaFormat) {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
outputFormat.setInteger(
MediaFormat.KEY_BITRATE_MODE,
MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_VBR
)
}
codec.configure(outputFormat, null.null, MediaCodec.CONFIGURE_FLAG_ENCODE)
}
override fun addTrack(muxer: MMuxer, mediaFormat: MediaFormat) {
muxer.addVideoTrack(mediaFormat)
}
override fun writeData(
muxer: MMuxer,
byteBuffer: ByteBuffer,
bufferInfo: MediaCodec.BufferInfo
) {
muxer.writeVideoData(byteBuffer, bufferInfo)
}
override fun encodeManually(a): Boolean {
return false
}
override fun release(muxer: MMuxer) {
muxer.releaseVideoTrack()
}
fun getEncodeSurface(a): Surface? {
return mSurface
}
}
Copy the codeInherit BaseEncoder and implement all the virtual functions.
Focus on the configEncoder method.
I. The bit rate KEY_BIT_RATE is configured.
Calculation formula derived from [MediaCodec Encoding OpenGL Speed and clarity balancing]
Biterate = Width * Height * FrameRate * Factor Factor: 0.1 to 0.2Copy the codeIi. set the frame rate KEY_FRAME_RATE, here 30 frames/SEC iii. Set the frequency of keyframes to KEY_I_FRAME_INTERVAL (1 frame/second iv). Configuration data source KEY_COLOR_FORMAT, COLOR_FormatSurface, both from Surface. V. Configure the bit rate mode KEY_BITRATE_MODE
- BITRATE_MODE_CQ Ignores the bit rate set by the user, controls the bit rate by the encoder itself, and ensures the clarity and bit rate balance as much as possible. - BITRATE_MODE_CBR -bitrate_MODE_vBR follows the user-set bit rate as much as possible, but dynamically adjusts the bit rate according to the motion vector between frames (commonly understood as the degree of picture change between frames). If the motion vector is large, the bit rate will be increased during this period. If the picture change is small, Then the bit rate decreases.Copy the codeBITRATE_MODE_CQ is preferred. If the encoder does not support BITRATE_MODE_VBR, switch to the default BITRATE_MODE_VBR
Vi. Finally, create a Surface using the encoder codec.createInputSurface() for EGL window binding. The images from the decoded video will be rendered into the Surface, and MediaCodec will automatically pull the data from it and encode it.
Audio encoder
Audio encoders are even simpler.
// Code sampling rate
val DEST_SAMPLE_RATE = 44100
// Code rate
private val DEST_BIT_RATE = 128000
class AudioEncoder(muxer: MMuxer): BaseEncoder(muxer) {
private val TAG = "AudioEncoder"
override fun encodeType(a): String {
return "audio/mp4a-latm"
}
override fun configEncoder(codec: MediaCodec) {
val audioFormat = MediaFormat.createAudioFormat(encodeType(), DEST_SAMPLE_RATE, 2)
audioFormat.setInteger(MediaFormat.KEY_BIT_RATE, DEST_BIT_RATE)
audioFormat.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 100*1024)
try {
configEncoderWithCQ(codec, audioFormat)
} catch (e: Exception) {
e.printStackTrace()
try {
configEncoderWithVBR(codec, audioFormat)
} catch (e: Exception) {
e.printStackTrace()
Log.e(TAG, "Failed to configure audio encoder")}}}private fun configEncoderWithCQ(codec: MediaCodec, outputFormat: MediaFormat) {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
// This part of the phone does not support BITRATE_MODE_CQ mode, which may be abnormal
outputFormat.setInteger(
MediaFormat.KEY_BITRATE_MODE,
MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_CQ
)
}
codec.configure(outputFormat, null.null, MediaCodec.CONFIGURE_FLAG_ENCODE)
}
private fun configEncoderWithVBR(codec: MediaCodec, outputFormat: MediaFormat) {
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
outputFormat.setInteger(
MediaFormat.KEY_BITRATE_MODE,
MediaCodecInfo.EncoderCapabilities.BITRATE_MODE_VBR
)
}
codec.configure(outputFormat, null.null, MediaCodec.CONFIGURE_FLAG_ENCODE)
}
override fun addTrack(muxer: MMuxer, mediaFormat: MediaFormat) {
muxer.addAudioTrack(mediaFormat)
}
override fun writeData(
muxer: MMuxer,
byteBuffer: ByteBuffer,
bufferInfo: MediaCodec.BufferInfo
) {
muxer.writeAudioData(byteBuffer, bufferInfo)
}
override fun release(muxer: MMuxer) {
muxer.releaseAudioTrack()
}
}
Copy the codeAs you can see, the configEncoder implementation is also relatively simple:
I. Set audio bit rate mediaformat. KEY_BIT_RATE to 128000 ii. Set the input buffer size KEY_MAX_INPUT_SIZE to 100 x 1024
Four, integrating
Now that the audio and video coding tools have been completed, let’s look at how to connect the decoder, OpenGL, EGL and encoder to achieve the video editing function.
- Modify the EGL renderer
Before we begin, we need to modify the EGL renderer defined in the “Inside Out EGL for OpenGL” article.
I. In the renderer defined previously, only one SurfaceView is supported and bound to the EGL display window. Here we need it to support setting up a Surface and receiving the Surface created in The VideoEncoder as the render window.
Ii. Since it is to encode the picture of the window, there is no need to constantly refresh the picture in the renderer, just refresh the picture when the video decoder decodes a frame. It also passes the timestamp of the current frame to OpenGL.
The complete code is shown below, with the additions marked:
class CustomerGLRenderer : SurfaceHolder.Callback {
private val mThread = RenderThread()
private var mSurfaceView: WeakReference<SurfaceView>? = null
private var mSurface: Surface? = null
private val mDrawers = mutableListOf<IDrawer>()
init {
mThread.start()
}
fun setSurface(surface: SurfaceView) {
mSurfaceView = WeakReference(surface)
surface.holder.addCallback(this)
surface.addOnAttachStateChangeListener(object : View.OnAttachStateChangeListener{
override fun onViewDetachedFromWindow(v: View?). {
stop()
}
override fun onViewAttachedToWindow(v: View?).{}})}//------------------- added part -----------------
// Add interface for setting Surface
fun setSurface(surface: Surface, width: Int, height: Int) {
mSurface = surface
mThread.onSurfaceCreate()
mThread.onSurfaceChange(width, height)
}
// Add RenderMode as shown below
fun setRenderMode(mode: RenderMode) {
mThread.setRenderMode(mode)
}
// Add notification update screen method
fun notifySwap(timeUs: Long) {
mThread.notifySwap(timeUs)
}
/----------------------------------------------
fun addDrawer(drawer: IDrawer) {
mDrawers.add(drawer)
}
fun stop(a) {
mThread.onSurfaceStop()
mSurface = null
}
override fun surfaceCreated(holder: SurfaceHolder) {
mSurface = holder.surface
mThread.onSurfaceCreate()
}
override fun surfaceChanged(holder: SurfaceHolder, format: Int, width: Int, height: Int) {
mThread.onSurfaceChange(width, height)
}
override fun surfaceDestroyed(holder: SurfaceHolder) {
mThread.onSurfaceDestroy()
}
inner class RenderThread: Thread() {
// Render state
private var mState = RenderState.NO_SURFACE
private var mEGLSurface: EGLSurfaceHolder? = null
// Whether EGLSurface is bound
private var mHaveBindEGLContext = false
// Whether the EGL context has been created to determine whether a new texture ID needs to be produced
private var mNeverCreateEglContext = true
private var mWidth = 0
private var mHeight = 0
private val mWaitLock = Object()
private var mCurTimestamp = 0L
private var mLastTimestamp = 0L
private var mRenderMode = RenderMode.RENDER_WHEN_DIRTY
private fun holdOn(a) {
synchronized(mWaitLock) {
mWaitLock.wait()
}
}
private fun notifyGo(a) {
synchronized(mWaitLock) {
mWaitLock.notify()
}
}
fun setRenderMode(mode: RenderMode) {
mRenderMode = mode
}
fun onSurfaceCreate(a) {
mState = RenderState.FRESH_SURFACE
notifyGo()
}
fun onSurfaceChange(width: Int, height: Int) {
mWidth = width
mHeight = height
mState = RenderState.SURFACE_CHANGE
notifyGo()
}
fun onSurfaceDestroy(a) {
mState = RenderState.SURFACE_DESTROY
notifyGo()
}
fun onSurfaceStop(a) {
mState = RenderState.STOP
notifyGo()
}
fun notifySwap(timeUs: Long) {
synchronized(mCurTimestamp) {
mCurTimestamp = timeUs
}
notifyGo()
}
override fun run(a) {
initEGL()
while (true) {
when (mState) {
RenderState.FRESH_SURFACE -> {
createEGLSurfaceFirst()
holdOn()
}
RenderState.SURFACE_CHANGE -> {
createEGLSurfaceFirst()
GLES20.glViewport(0.0, mWidth, mHeight)
configWordSize()
mState = RenderState.RENDERING
}
RenderState.RENDERING -> {
render()
// New judgment: if 'RENDER_WHEN_DIRTY' mode is used, suspend the thread and wait for the next frame after rendering
if(mRenderMode == RenderMode.RENDER_WHEN_DIRTY) { holdOn() } } RenderState.SURFACE_DESTROY -> { destroyEGLSurface() mState = RenderState.NO_SURFACE } RenderState.STOP -> { releaseEGL()return
}
else -> {
holdOn()
}
}
if (mRenderMode == RenderMode.RENDER_CONTINUOUSLY) {
sleep(16)}}}private fun initEGL(a){ mEGLSurface = EGLSurfaceHolder() mEGLSurface? .init(null, EGL_RECORDABLE_ANDROID)
}
private fun createEGLSurfaceFirst(a) {
if(! mHaveBindEGLContext) { mHaveBindEGLContext =true
createEGLSurface()
if (mNeverCreateEglContext) {
mNeverCreateEglContext = false
GLES20.glClearColor(0f.0f.0f.0f)
// Turn on the blending, i.e. Translucent
GLES20.glEnable(GLES20.GL_BLEND)
GLES20.glBlendFunc(GLES20.GL_SRC_ALPHA, GLES20.GL_ONE_MINUS_SRC_ALPHA)
generateTextureID()
}
}
}
private fun createEGLSurface(a){ mEGLSurface? .createEGLSurface(mSurface) mEGLSurface? .makeCurrent() }private fun generateTextureID(a) {
val textureIds = OpenGLTools.createTextureIds(mDrawers.size)
for ((idx, drawer) in mDrawers.withIndex()) {
drawer.setTextureID(textureIds[idx])
}
}
private fun configWordSize(a) {
mDrawers.forEach { it.setWorldSize(mWidth, mHeight) }
}
/ / -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- modified part of the code -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
// Determine whether to refresh the screen based on the render mode and the timestamp of the current frame
private fun render(a) {
val render = if (mRenderMode == RenderMode.RENDER_CONTINUOUSLY) {
true
} else {
synchronized(mCurTimestamp) {
if (mCurTimestamp > mLastTimestamp) {
mLastTimestamp = mCurTimestamp
true
} else {
false}}}if(render) { GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT or GLES20.GL_DEPTH_BUFFER_BIT) mDrawers.forEach { it.draw() } mEGLSurface? .setTimestamp(mCurTimestamp) mEGLSurface? .swapBuffers() } }//------------------------------------------------------
private fun destroyEGLSurface(a){ mEGLSurface? .destroyEGLSurface() mHaveBindEGLContext =false
}
private fun releaseEGL(a){ mEGLSurface? .release() } }/** * Render state */
enum class RenderState {
NO_SURFACE, // There is no valid surface
FRESH_SURFACE, // Hold a new, uninitialized surface
SURFACE_CHANGE, // Change the surface size
RENDERING, // After initialization, you can start rendering
SURFACE_DESTROY, / / surface destruction
STOP // Stop drawing
}
//--------- added render mode definition ------------
enum class RenderMode {
// Automatic loop rendering
RENDER_CONTINUOUSLY,
// Render by notifySwap externally
RENDER_WHEN_DIRTY
}
//-------------------------------------
}
Copy the codeThe new part has been marked out, and it is not complicated. The main thing is to add the Settings Surface and distinguish the two rendering modes. Please look at the code.
- Modified decoder
Remember from the previous article that you need to synchronize audio and video to play properly?
And because of the coding, there is no need to play out the video picture and audio, so you can get rid of audio and video synchronization, speed up the coding.
It is also easy to modify. Add a new variable mSyncRender to BaseDecoder. If mSyncRender == false, the audio and video synchronization will be removed.
Only the modifications are listed here. See BaseDecoder for the complete code
abstract class BaseDecoder(private val mFilePath: String): IDecoder {
// omit extraneous code......
// Whether audio and video render synchronization is required
private var mSyncRender = true
final override fun run(a) {
// omit irrelevant code...
while (mIsRunning) {
/ /...
// --------- -------------
if (mSyncRender && mState == DecodeState.DECODING) {
sleepRender()
}
if (mSyncRender) {// If it is only used to encode new video, no rendering is requiredrender(mOutputBuffers!! [index], mBufferInfo) }/ /...
}
//
}
override fun withoutSync(a): IDecoder {
mSyncRender = false
return this
}
/ /...
}
Copy the code- integration
class SynthesizerActivity: AppCompatActivity(), MMuxer.IMuxerStateListener {
private val path = Environment.getExternalStorageDirectory().absolutePath + "/mvtest_2.mp4"
private val path2 = Environment.getExternalStorageDirectory().absolutePath + "/mvtest.mp4"
private val threadPool = Executors.newFixedThreadPool(10)
private var renderer = CustomerGLRenderer()
private var audioDecoder: IDecoder? = null
private var videoDecoder: IDecoder? = null
private lateinit var videoEncoder: VideoEncoder
private lateinit var audioEncoder: AudioEncoder
private var muxer = MMuxer()
override fun onCreate(savedInstanceState: Bundle?). {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_synthesizer)
muxer.setStateListener(this)}fun onStartClick(view: View) {
btn.text = "Coding"
btn.isEnabled = false
initVideo()
initAudio()
initAudioEncoder()
initVideoEncoder()
}
private fun initVideoEncoder(a) {
// Video encoder
videoEncoder = VideoEncoder(muxer, 1920.1080) renderer.setRenderMode(CustomerGLRenderer.RenderMode.RENDER_WHEN_DIRTY) renderer.setSurface(videoEncoder.getEncodeSurface()!! .1920.1080)
videoEncoder.setStateListener(object : DefEncodeStateListener {
override fun encoderFinish(encoder: BaseEncoder) {
renderer.stop()
}
})
threadPool.execute(videoEncoder)
}
private fun initAudioEncoder(a) {
// Audio encoder
audioEncoder = AudioEncoder(muxer)
// Start the encoding thread
threadPool.execute(audioEncoder)
}
private fun initVideo(a) {
val drawer = VideoDrawer()
drawer.setVideoSize(1920.1080)
drawer.getSurfaceTexture {
initVideoDecoder(path, Surface(it))
}
renderer.addDrawer(drawer)
}
private fun initVideoDecoder(path: String, sf: Surface){ videoDecoder? .stop() videoDecoder = VideoDecoder(path,null, sf).withoutSync() videoDecoder!! .setStateListener(object : DefDecodeStateListener {
override fun decodeOneFrame(decodeJob: BaseDecoder? , frame:Frame) {
renderer.notifySwap(frame.bufferInfo.presentationTimeUs)
videoEncoder.encodeOneFrame(frame)
}
override fun decoderFinish(decodeJob: BaseDecoder?).{ videoEncoder.endOfStream() } }) videoDecoder!! .goOn()// Start the decoding thread
threadPool.execute(videoDecoder!!)
}
private fun initAudio(a){ audioDecoder? .stop() audioDecoder = AudioDecoder(path).withoutSync() audioDecoder!! .setStateListener(object : DefDecodeStateListener {
override fun decodeOneFrame(decodeJob: BaseDecoder? , frame:Frame) {
audioEncoder.encodeOneFrame(frame)
}
override fun decoderFinish(decodeJob: BaseDecoder?).{ audioEncoder.endOfStream() } }) audioDecoder!! .goOn()// Start the decoding thread
threadPool.execute(audioDecoder!!)
}
override fun onMuxerFinish(a) {
runOnUiThread {
btn.isEnabled = true
btn.text = "Code completed"} audioDecoder? .stop() audioDecoder =nullvideoDecoder? .stop() videoDecoder =null}}Copy the codeAs you can see, the process is simple: initialize the decoder, initialize EGL Render, initialize the encoder, and then throw the decoded data into the encoder queue, listen for the decoding state and encoding state, and do the corresponding operations.
The decoding process is basically the same as playing video with EGL, except that the rendering mode is different.
In this code, the original video is simply decoded, rendered to OpenGL, and re-encoded into a new MP4, which means that the output video is exactly the same as the original video.
- What can be achieved?
Although the above is just a common decoding and coding process, but it can be derived from infinite imagination.
Such as:
-
Realize video clipping: set a start and end time for the decoder.
-
Achieve cool video screen editing: for example, if you change the video renderer VideoDrawer to the previously written SoulVideoDrawer, you will get an out-of-body effect video; Combined with the previous picture-in-picture, video can be superimposed.
-
Video splicing: combine multiple video decoders to connect multiple videos and encode new videos.
-
Add watermark: combine OpenGL render picture, add a watermark super simple.
.
As long as there is imagination, that is not a matter!
V. Concluding remarks
If you’ve read every article so far and started coding, I’m sure you’ve stepped into the door of Android audio and video development and can implement some previously mysterious video effects. Then save it to a real playable video.
Each of these articles is a long one, and I think we should all thank ourselves for reading this. It’s really hard to keep up.
Finally, I would like to express my gratitude to everyone who has praised, left comments, asked questions and encouraged the article. It is you who make the cold words full of warmth and the motivation for me to persist.
Let’s, next chapter, be there or be square!
