Android Audio and video (a) Camera2 API data collection
Android audio and video (2) audio AudioRecord and AudioTrack
Android audio and video (three)FFmpeg Camera2 push stream live
The MediaCodec class has access to the underlying MediaCodec framework (StageFright or OpenMAX), the codec component that is part of Android’s basic multimedia support infrastructure, Commonly used with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack. It is not Codec itself; it obtains the capabilities of Codec by calling the underlying Codec component.
The way MediaCodec works
MediaCodec processes input data to produce output data. When processing data asynchronously, a set of input and output Buffer queues is used. Normally, logically, the client requests (or receives) data and fills a predetermined empty input Buffer, which is then passed to MediaCodec for codec. The MediaCodec codec data is then filled into an output Buffer. Finally, the client requests (or receives) the output Buffer, consumes the output Buffer, releases it when it’s done, and gives MediaCodec back to refill the output data.
The input and output queues must be non-empty at the same time, that is, at least one input Buffer and one output Buffer.
MediaCodec Status cycle diagram
There are three states in the life cycle of MediaCodec: Stopped, Executing, and Released.
The Stopped state can actually be in three other states: Uninitialized, Configured, and Error.
Uploading state is also known as uploading state, Running state, and end-of-stream state.
As can be seen from the above picture:
- Is uninitialized when the codec is created. First you need to call configure(…). The start() method makes it Configured and then calls the start() method to make it a Executing state. In Executing, you can use the above buffer to process data.
- Uploading state is also known as uploading state, Running state, and end-of-stream state. After the Flushed () call, the uploading codec is as the uploading state, where it saves all the buffers. Once the first input buffer is present, the codec automatically runs to the Running state. When a buffer with the end-of-stream flag is entered, the codec enters the end-of-stream state. In this state, the codec no longer accepts the input buffer but still generates the output buffer. At this point you can call the flush() method to reset the uploading codec to the Flushed state.
- A call to stop() returns the codec to an uninitialized state, which can then be reconfigured. Once you have finished using the codec, you must release it by calling release().
- In rare cases, the codec may encounter an error and go to an error state. This is conveyed using invalid return values from queued operations or sometimes via exceptions. Call reset() to make the codec available again. You can call it from any state to move the codec back to the uninitialized state. Otherwise, a call to release() moves to the terminal release state.
Advantages and disadvantages of MediaCodec
Advantages: Low power consumption, fast speed
Disadvantages: poor scalability, different chip manufacturers provide different support schemes, resulting in poor program portability
Application scenario: Suitable for fixed hardware projects, such as smart home; It takes a long time.
MediaCodec codec implementation
Made a Demo, using AudioRecord recording, using MediaCodec to encode AAC and save the file, and then can decode from AAC to PCM data, and then play with AudioTrack.
1. Encode PCM data and save it as AAC file
Initialize the AudioRecord and encoder
private void initAudioRecord(a) {
int audioSource = MediaRecorder.AudioSource.MIC;
int sampleRate = 44100;
int channelConfig = AudioFormat.CHANNEL_IN_MONO;
int audioFormat = AudioFormat.ENCODING_PCM_16BIT;
int minBufferSize = AudioRecord.getMinBufferSize(sampleRate, channelConfig, audioFormat);
mAudioRecorder = new AudioRecord(audioSource, sampleRate, channelConfig, audioFormat, Math.max(minBufferSize, 2048));
}
Copy the code/** * Initializes the encoder */
private void initAudioEncoder(a) {
try {
mAudioEncoder = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_AUDIO_AAC);
MediaFormat format = MediaFormat.createAudioFormat(MediaFormat.MIMETYPE_AUDIO_AAC, 44100.1);
format.setInteger(MediaFormat.KEY_BIT_RATE, 96000);/ / bitrate
format.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, MAX_BUFFER_SIZE);
mAudioEncoder.configure(format, null.null, MediaCodec.CONFIGURE_FLAG_ENCODE);
} catch (IOException e) {
e.printStackTrace();
}
if (mAudioEncoder == null) {
Log.e(TAG, "create mediaEncode failed");
return;
}
mAudioEncoder.start(); // Start MediaCodec and wait for incoming data
encodeInputBuffers = mAudioEncoder.getInputBuffers(); // Input and output Buffer queues described above
encodeOutputBuffers = mAudioEncoder.getOutputBuffers();
mAudioEncodeBufferInfo = new MediaCodec.BufferInfo();
}
Copy the codeStart recording and coding
Use a thread pool with two threads, one for recording and the other for coding. The recording thread stores the PCM data into a queue, and the coding thread retrieves the data encoding from the queue.
// Start the recording thread
mExecutorService.submit(new Runnable() {
@Override
public void run(a) { startRecorder(); }});// Start the encoding thread
mExecutorService.submit(new Runnable() {
@Override
public void run(a) { encodePCM(); }});/**
* 将PCM数据存入队列
*/
private void putPCMData(byte[] pcmChunk) {
Log.e(TAG, "putPCMData");
try {
queue.put(pcmChunk);
} catch(InterruptedException e) { e.printStackTrace(); }}/** * Fetch PCM data from queue */
private byte[] getPCMData() {
try {
if (queue.isEmpty()) {
return null;
}
return queue.take();
} catch (InterruptedException e) {
e.printStackTrace();
}
return null;
}
/** * Add the ADTS header. If you want to merge with the video stream, you don't need to add the ADTS header
public static void addADTStoPacket(int sampleRateType, byte[] packet, int packetLen) {
int profile = 2; // AAC LC
int chanCfg = 2; // CPE
packet[0] = (byte) 0xFF;
packet[1] = (byte) 0xF9;
packet[2] = (byte) (((profile - 1) < <6) + (sampleRateType << 2) + (chanCfg >> 2));
packet[3] = (byte) (((chanCfg & 3) < <6) + (packetLen >> 11));
packet[4] = (byte) ((packetLen & 0x7FF) > >3);
packet[5] = (byte) (((packetLen & 7) < <5) + 0x1F);
packet[6] = (byte) 0xFC;
}
Copy the codeAudio data
/** * get audio data */
private void startRecorder(a) {
try {
mFilePath = Environment.getExternalStorageDirectory().getAbsolutePath() + "/RecorderTest/" + System.currentTimeMillis() + ".aac";
mAudioFile = new File(mFilePath);
if(! mAudioFile.getParentFile().exists()) { mAudioFile.getParentFile().mkdirs(); } mAudioFile.createNewFile(); mFileOutputStream =new FileOutputStream(mAudioFile);
mAudioBos = new BufferedOutputStream(mFileOutputStream, 200 * 1024);
mAudioRecorder.startRecording();
start = System.currentTimeMillis();
while (mIsRecording) {
int read = mAudioRecorder.read(mBuffer, 0.2048);
if (read > 0) {
byte[] audio = new byte[read];
System.arraycopy(mBuffer, 0, audio, 0, read);
putPCMData(audio); // PCM data is queued for encoding}}}catch (IOException | RuntimeException e) {
e.printStackTrace();
} finally {
if(mAudioRecorder ! =null) {
mAudioRecorder.release();
mAudioRecorder = null; }}}Copy the codecoding
The data is looped out of the queue, encoded by MediaCodec, and written to a file.
/** * PCM */
private void encodePCM(a) {
int inputIndex;
ByteBuffer inputBuffer;
int outputIndex;
ByteBuffer outputBuffer;
byte[] chunkAudio;
int outBitSize;
int outPacketSize;
byte[] chunkPCM;
while(mIsRecording || ! queue.isEmpty()) { chunkPCM = getPCMData();// Get the data output of the thread where the decoder is located
if (chunkPCM == null) {
continue;
}
inputIndex = mAudioEncoder.dequeueInputBuffer(-1);// Same as decoder
if (inputIndex >= 0) {
inputBuffer = encodeInputBuffers[inputIndex];// Same as decoder
inputBuffer.clear();// Same as decoder
inputBuffer.limit(chunkPCM.length);
inputBuffer.put(chunkPCM);//PCM data is populated to inputBuffer
mAudioEncoder.queueInputBuffer(inputIndex, 0, chunkPCM.length, 0.0);// Notification encoder encoding
}
outputIndex = mAudioEncoder.dequeueOutputBuffer(mAudioEncodeBufferInfo, 10000);
while (outputIndex >= 0) {
outBitSize = mAudioEncodeBufferInfo.size;
outPacketSize = outBitSize + 7;//7 is the size of the ADTS header
outputBuffer = encodeOutputBuffers[outputIndex];// Get the output Buffer
outputBuffer.position(mAudioEncodeBufferInfo.offset);
outputBuffer.limit(mAudioEncodeBufferInfo.offset + outBitSize);
chunkAudio = new byte[outPacketSize];
addADTStoPacket(44100, chunkAudio, outPacketSize);/ / add ADTS
outputBuffer.get(chunkAudio, 7, outBitSize);// Fetch the encoded AAC data into byte[] at offset=7
outputBuffer.position(mAudioEncodeBufferInfo.offset);
try {
mAudioBos.write(chunkAudio, 0, chunkAudio.length);//BufferOutputStream saves files to memory card *.aac
} catch (IOException e) {
e.printStackTrace();
}
mAudioEncoder.releaseOutputBuffer(outputIndex, false);
outputIndex = mAudioEncoder.dequeueOutputBuffer(mAudioEncodeBufferInfo, 10000);
}
}
stopRecorder();
}
Copy the code2. Decode AAC AudioTrack to play
Initialize the AudioTrack and decoder
/** * Initialize AudioTrack and wait for the data to play */
private void initAudioTrack(a) {
int streamType = AudioManager.STREAM_MUSIC;
int sampleRate = 44100;
int channelConfig = AudioFormat.CHANNEL_OUT_MONO;
int audioFormat = AudioFormat.ENCODING_PCM_16BIT;
int mode = AudioTrack.MODE_STREAM;
int minBufferSize = AudioTrack.getMinBufferSize(sampleRate, channelConfig, audioFormat);
audioTrack = new AudioTrack(streamType, sampleRate, channelConfig, audioFormat,
Math.max(minBufferSize, 2048), mode);
audioTrack.play();
}
Copy the code/** * Initializes the decoder */
private void initAudioDecoder(a) {
try {
mMediaExtractor = new MediaExtractor();
mMediaExtractor.setDataSource(mFilePath);
MediaFormat format = mMediaExtractor.getTrackFormat(0);
String mime = format.getString(MediaFormat.KEY_MIME);
if (mime.startsWith("audio")) {// Get the audio track
mMediaExtractor.selectTrack(0);// Select this audio track
format.setString(MediaFormat.KEY_MIME, "audio/mp4a-latm");
format.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
format.setInteger(MediaFormat.KEY_SAMPLE_RATE, 0);
format.setInteger(MediaFormat.KEY_BIT_RATE, 96000);
format.setInteger(MediaFormat.KEY_IS_ADTS, 1);
format.setInteger(MediaFormat.KEY_AAC_PROFILE, 0);
mAudioDecoder = MediaCodec.createDecoderByType(mime);// Create Decode decoder
mAudioDecoder.configure(format, null.null.0);
} else {
return; }}catch (IOException e) {
e.printStackTrace();
}
if (mAudioDecoder == null) {
Log.e(TAG, "mAudioDecoder is null");
return;
}
mAudioDecoder.start();// Start MediaCodec and wait for incoming data
}
Copy the codeDecode and play
private void decodeAndPlay(a) {
boolean isFinish = false;
MediaCodec.BufferInfo decodeBufferInfo = new MediaCodec.BufferInfo();
while(! isFinish && mIsPalying) {int inputIdex = mAudioDecoder.dequeueInputBuffer(10000);InputBuffer -1 indicates that the buffer is always waiting. 0 indicates that the buffer is not waiting. 10000 indicates that the buffer times out in 10 seconds
if (inputIdex < 0) {
isFinish = true;
}
ByteBuffer inputBuffer = mAudioDecoder.getInputBuffer(inputIdex);
inputBuffer.clear();// Empties the data passed into the inputBuffer
int samplesize = mMediaExtractor.readSampleData(inputBuffer, 0);
if (samplesize > 0) {
mAudioDecoder.queueInputBuffer(inputIdex, 0, samplesize, 0.0); // Inform the decoder to decode
mMediaExtractor.advance(); //MediaExtractor moves to the next sample
} else {
isFinish = true;
}
int outputIndex = mAudioDecoder.dequeueOutputBuffer(decodeBufferInfo, 10000);// Retrieve the decoded byte[] data
ByteBuffer outputBuffer;
byte[] chunkPCM;
// A while loop is used to ensure that the decoder spits out all data
while (outputIndex >= 0) {
outputBuffer = mAudioDecoder.getOutputBuffer(outputIndex);
chunkPCM = new byte[decodeBufferInfo.size];
outputBuffer.get(chunkPCM);
outputBuffer.clear();// Remember to empty the Buffer after the data is retrieved. MediaCodec uses these buffers in a loop. If you do not empty them, you will get the same number next time
// Play decoded PCM data
audioTrack.write(chunkPCM, 0, decodeBufferInfo.size);
mAudioDecoder.releaseOutputBuffer(outputIndex, false);
outputIndex = mAudioDecoder.dequeueOutputBuffer(decodeBufferInfo, 10000);// Get the data again
}
}
stopPlay();
}
Copy the codeDemo completed, mobile phone test effect is good. The use of MediaCodec is more complicated than I expected. It took me a long time to finish this Demo after checking online. I hope it can help people in need.
If you have any questions, please leave a message, Github source code – MediaCodecActivity