JavaCV camera combat seven: Push stream (with sound)

Make writing a habit together! This is my first day to participate in the “Gold Digging Day New Plan · April More text challenge”, click to see the details of the activity

Welcome to my GitHub

Here classification and summary of xinchen all original (including supporting source code) : github.com/zq2599/blog…

This paper gives an overview of

• In this article, we push the content of the camera to the media server, and then play it successfully with VLC. I believe you must have noticed a flaw: there is no sound
• Although the topic of the JavaCV Camera In Action series is camera processing, audio and video sound is obviously the most common situation, so this article will fill in the previous deficiencies: coding to push the camera and microphone stream, and verify that the remote playback of audio and video can be successful

About the collection and recording of audio

• This code is in the “JavaCV camera actual combat five: push stream” source code based on the increase of audio processing part
• Before coding, let’s analyze how the specific code logic changes with the addition of audio processing
• The operation of saving only video is different from that of saving audio as shown below. The dark block is the new operation:

• In contrast, at the end of the application, there are more audio and video operations when all resources are released than when only video is released, as shown in the figure below. Dark colors are the operation that releases audio related resources:

• To keep the code simple, I’ve put all the audio-related processing in a class called AudioService, which means that the dark code in the two images above is in AudioService.java. The main program uses this class to do the audio processing
• So let’s start coding

Develop the audio processing class AudioService

• The first is audioService.java, which focuses on the functionality of the dark blocks in the previous figure, with a few caveats that will be mentioned later:

@Slf4j
public class AudioService {

    / / sampling rate
    private final static int SAMPLE_RATE = 44100;

   
    private final static int CHANNEL_NUM = 2;

    // Frame recorder
    private FFmpegFrameRecorder recorder;

    / / timer
    private ScheduledThreadPoolExecutor sampleTask;

    private TargetDataLine line;

    byte[] audioBytes;

    private volatile boolean isFinish = false;

    public void setRecorderParams(FrameRecorder recorder) throws Exception {
        this.recorder = (FFmpegFrameRecorder)recorder;

        // The bit rate is constant
        recorder.setAudioOption("crf"."0");
        // Maximum sound quality
        recorder.setAudioQuality(0);
        // 192 Kbps
        recorder.setAudioBitrate(192000);

        / / sampling rate
        recorder.setSampleRate(SAMPLE_RATE);

        / / stereo
        recorder.setAudioChannels(2);
        / / encoder
        recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
    }


    public void initSampleService(a) throws Exception {
       
        AudioFormat audioFormat = new AudioFormat(SAMPLE_RATE, 16, CHANNEL_NUM, true.false);

        DataLine.Info dataLineInfo = new DataLine.Info(TargetDataLine.class, audioFormat);

        line = (TargetDataLine)AudioSystem.getLine(dataLineInfo);

        line.open(audioFormat);

        line.start();

        final int audioBufferSize = SAMPLE_RATE * CHANNEL_NUM;

        audioBytes = new byte[audioBufferSize];

        sampleTask = new ScheduledThreadPoolExecutor(1);
    }

    public void releaseOutputResource(a) {
        // The end flag prevents the sampled code from exiting in the whlie loop
        isFinish = true;
        // End the scheduled task
        sampleTask.shutdown();
        // Stop the data line
        line.stop();
        // Close the data line
        line.close();
    }

    public void startSample(double frameRate) {

        // Start a scheduled task that executes once per second to collect audio data for the frame recorder
        sampleTask.scheduleAtFixedRate((Runnable) new Runnable() {
            @Override
            public void run(a) {
                try
                {
                    int nBytesRead = 0;

                    while (nBytesRead == 0 && !isFinish) {
                        
                        nBytesRead = line.read(audioBytes, 0, line.available());
                    }

                   
                    if (nBytesRead<1) {
                        return;
                    }

                    int nSamplesRead = nBytesRead / 2;
                    short[] samples = new short[nSamplesRead];

                    
                    ByteBuffer.wrap(audioBytes).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(samples);
                    
                    ShortBuffer sBuff = ShortBuffer.wrap(samples, 0, nSamplesRead);

                    
                    recorder.recordSamples(SAMPLE_RATE, CHANNEL_NUM, sBuff);
                }
                catch(FrameRecorder.Exception e) { e.printStackTrace(); }}},0.1000 / (long)frameRate, TimeUnit.MILLISECONDS); }}Copy the code

• In the above code, there are two things to note:

1. Focus on recorder. RecordSamples, which stores audio into mp4 files
2. The timed task is executed on a new thread, so when the main thread finishes recording, the while loop in the timed task needs to be interrupted. Therefore, the volatile isFinish variable is added to help the code in the timed task determine whether to terminate the while loop immediately

Transform the original push stream only push video code

• Next is “JavaCV camera combat five: Push flow, the article RecordCamera. Java transformation, in order not to affect the chapters on making the code before, here I added a class RecordCameraWithAudio. Java, contents and RecordCamera. Java as like as two peas, So let’s change this RecordCameraWithAudio class
• Add a member variable of type AudioService:

    // Audio service class
    private AudioService audioService = new AudioService();
Copy the code

• The initOutput method is responsible for initializing the frame recorder, and now it needs to initialize the audio and start a scheduled task to collect and process the audio, as shown in the following.

    @Override
    protected void initOutput(a) throws Exception {
        // instantiate FFmpegFrameRecorder to pass in the SRS push address
        recorder = FrameRecorder.createDefault(RECORD_ADDRESS, getCameraImageWidth(), getCameraImageHeight());

        recorder.setVideoOption("tune"."zerolatency");
        
        recorder.setVideoOption("preset"."ultrafast");
        
        recorder.setVideoOption("crf"."28");
       
        recorder.setVideoBitrate(2000000);

        // Set the encoding format
        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);

        // Set the encapsulation format
        recorder.setFormat("flv");

        // FPS (frames per second)
        // The number of frames per second
        recorder.setFrameRate(getFrameRate());
        // Key frame interval, in our case every 2 seconds -> 30 (fps) * 2 = 60
        // Keyframe interval
        recorder.setGopSize((int)getFrameRate()*2);

        // Set the audio parameters of the frame recorder
        audioService.setRecorderParams(recorder);

        // Audio sampling-related initialization operations
        audioService.initSampleService();

        // The frame recorder starts initialization
        recorder.start();

        // Start the scheduled task to capture audio frames to the frame recorder
        audioService.startSample(getFrameRate());
    }
Copy the code

• The output method is saved as is and only handles video frames (audio is handled in a timed task)

    @Override
    protected void output(Frame frame) throws Exception {
        if (0L==startRecordTime) {
            startRecordTime = System.currentTimeMillis();
        }

        / / timestamp
        recorder.setTimestamp(1000 * (System.currentTimeMillis()-startRecordTime));

        / / save
        recorder.record(frame);
    }
Copy the code

• In the method of releasing resources, added the operation of releasing audio resources:

    @Override
    protected void releaseOutputResource(a) throws Exception {
        // Perform a resource release operation for the audio service
        audioService.releaseOutputResource();

        // Close the frame recorder
        recorder.close();
    }
Copy the code

• At this point, the function of pushing camera video and microphone audio to the media server has been developed, and then write the main method to indicate the push stream for ten minutes:

    public static void main(String[] args) {
        new RecordCameraWithAudio().action(600);
    }
Copy the code

• Run the main method and wait until the console outputs the content in the red box below, indicating that it is pushing:

• On another computer with VLC software open just push flow address RTMP: / / 192.168.50.43:21935 / HLS/camera, wait a few seconds later began to normal play, image noise is normal (note can’t use the computer, otherwise the microphone acquisition is VLC player voice) :

• Check the media stream information with the tools of VLC, as shown in the picture below, both video stream and audio stream can be recognized normally:

• Open the monitoring page of the media server itself, as shown below, and you can see all real-time data:

• So far, we have completed the audio and video push stream function, (a little like the appearance of live), thanks to the powerful JavaCV, the whole process is so easy and happy, next please continue to pay attention to Xinchen original, “JavaCV camera actual battle” series will present more rich applications;

Download the source code

• The full source code for JavaCV Camera In Action is available on GitHub at github.com/zq2599/blog…

• The Git project has multiple folders. The source code for this project is in the Javacv-tutorials folder, as shown in the red box below:

• Javacv-tutorials have many subprojects. The Code for the JavacV Camera Hands-on series was based on the Simply-grab-push project:

Welcome to the Nuggets: programmer Chen Chen

Learning on the road, you are not alone, Xinchen original all the way accompanied by…