Android 音视频采集基本流程

围巾🧣 2024年06月26日 633次浏览

在Android平台上进行音视频采集，通常涉及到使用设备的摄像头和麦克风来捕获视频和音频数据。这个过程可以分为以下几个步骤：

1. 初始化和权限管理

在进行音视频采集之前，需要确保应用具有适当的权限，包括摄像头权限和录音权限。这些权限需要在AndroidManifest.xml文件中声明，并在运行时请求用户授权。

在 AndroidManifest.xml 中声明权限：

  1. <uses-permission android:name="android.permission.CAMERA"/>
  2. <uses-permission android:name="android.permission.RECORD_AUDIO"/>
  3. <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

在代码中请求权限（以Android 6.0及以上为例）：

  1. if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED ||
  2.     ContextCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED) {
  3.     ActivityCompat.requestPermissions(this, new String[]{
  4.         Manifest.permission.CAMERA,
  5.         Manifest.permission.RECORD_AUDIO
  6.     }, REQUEST_CODE_PERMISSIONS);
  7. }

2. 设置摄像头（Camera）和麦克风（AudioRecord）

Android提供了两种主要的API来处理视频采集：Camera API（旧的API）和 Camera2 API（新的API）。下面是使用 Camera2 API 进行视频采集的基本流程。

初始化 Camera2 API：

  1. private void initializeCamera() {
  2.     CameraManager cameraManager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);
  3.     try {
  4.         String cameraId = cameraManager.getCameraIdList()[0]; // 获取前置/后置摄像头ID
  5.         CameraCharacteristics characteristics = cameraManager.getCameraCharacteristics(cameraId);
  6.         StreamConfigurationMap map = characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
  7.         Size previewSize = map.getOutputSizes(SurfaceTexture.class)[0];

  9.         // 配置捕获会话
  10.         cameraManager.openCamera(cameraId, new CameraDevice.StateCallback() {
  11.             @Override
  12.             public void onOpened(@NonNull CameraDevice camera) {
  13.                 cameraDevice = camera;
  14.                 createCameraPreviewSession();
  15.             }

  17.             @Override
  18.             public void onDisconnected(@NonNull CameraDevice camera) {
  19.                 camera.close();
  20.                 cameraDevice = null;
  21.             }

  23.             @Override
  24.             public void onError(@NonNull CameraDevice camera, int error) {
  25.                 camera.close();
  26.                 cameraDevice = null;
  27.             }
  28.         }, null);
  29.     } catch (CameraAccessException e) {
  30.         e.printStackTrace();
  31.     }
  32. }

  34. private void createCameraPreviewSession() {
  35.     SurfaceTexture texture = textureView.getSurfaceTexture();
  36.     texture.setDefaultBufferSize(previewSize.getWidth(), previewSize.getHeight());
  37.     Surface surface = new Surface(texture);

  39.     try {
  40.         captureRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
  41.         captureRequestBuilder.addTarget(surface);

  43.         cameraDevice.createCaptureSession(Arrays.asList(surface), new CameraCaptureSession.StateCallback() {
  44.             @Override
  45.             public void onConfigured(@NonNull CameraCaptureSession session) {
  46.                 cameraCaptureSession = session;
  47.                 try {
  48.                     cameraCaptureSession.setRepeatingRequest(captureRequestBuilder.build(), null, null);
  49.                 } catch (CameraAccessException e) {
  50.                     e.printStackTrace();
  51.                 }
  52.             }

  54.             @Override
  55.             public void onConfigureFailed(@NonNull CameraCaptureSession session) {
  56.                 // Handle failure
  57.             }
  58.         }, null);
  59.     } catch (CameraAccessException e) {
  60.         e.printStackTrace();
  61.     }
  62. }

初始化音频录制：

  1. private void initializeAudioRecord() {
  2.     int bufferSize = AudioRecord.getMinBufferSize(SAMPLE_RATE_IN_HZ, CHANNEL_CONFIG, AUDIO_FORMAT);
  3.     audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC, SAMPLE_RATE_IN_HZ, CHANNEL_CONFIG, AUDIO_FORMAT, bufferSize);
  4. }

  6. private void startAudioRecording() {
  7.     audioRecord.startRecording();
  8.     isRecording = true;
  9.     new Thread(new AudioRecordRunnable()).start();
  10. }

  12. private void stopAudioRecording() {
  13.     if (audioRecord != null) {
  14.         isRecording = false;
  15.         audioRecord.stop();
  16.         audioRecord.release();
  17.         audioRecord = null;
  18.     }
  19. }

  21. private class AudioRecordRunnable implements Runnable {
  22.     @Override
  23.     public void run() {
  24.         byte[] buffer = new byte[BUFFER_SIZE];
  25.         while (isRecording) {
  26.             int read = audioRecord.read(buffer, 0, buffer.length);
  27.             if (read > 0) {
  28.                 // 处理音频数据
  29.             }
  30.         }
  31.     }
  32. }

3. 音视频数据处理

在捕获到音频和视频数据之后，可以对这些数据进行处理，比如编码、存储或实时传输。常见的处理方式包括使用 MediaCodec 进行编码，使用 MediaMuxer 进行封装，以及将数据发送到服务器进行直播。

4. 释放资源

在完成音视频采集后，需要释放资源以避免内存泄漏。

  1. private void closeCamera() {
  2.     if (cameraCaptureSession != null) {
  3.         cameraCaptureSession.close();
  4.         cameraCaptureSession = null;
  5.     }
  6.     if (cameraDevice != null) {
  7.         cameraDevice.close();
  8.         cameraDevice = null;
  9.     }
  10. }

  12. @Override
  13. protected void onPause() {
  14.     super.onPause();
  15.     closeCamera();
  16.     stopAudioRecording();
  17. }

5. 示例代码整合

下面是一个完整的示例代码，展示如何使用 ImageReader 捕捉每一帧视频数据，并使用 MediaCodec 将原始的 YUV420 数据编码为 H.264 格式。

1. 使用 ImageReader 捕捉视频帧

我们需要在 createCameraPreviewSession 方法中创建 ImageReader 并设置 OnImageAvailableListener 来捕捉每一帧的数据。

2. 使用 MediaCodec 编码 YUV420 到 H.264

我们需要初始化一个 MediaCodec 编码器，并将从 ImageReader 获取的 YUV420 数据传递给它进行编码。

以下是完整的示例代码：

AVActivity.java

  1. public class AVActivity extends AppCompatActivity {
  2.     private static final String TAG = "AVActivity";

  4.     private TextureView textureView;
  5.     private CameraDevice cameraDevice;
  6.     private CameraCaptureSession cameraCaptureSession;
  7.     private CaptureRequest.Builder captureRequestBuilder;
  8.     private ImageReader imageReader;

  10.     private AudioRecord audioRecord;
  11.     private boolean isAudioRecording;
  12.     private boolean isVideoRecording;

  14.     private MediaCodec mediaCodec;
  15.     private MediaCodec.BufferInfo bufferInfo;
  16.     private FileOutputStream fileOutputStream;

  18.     HandlerThread h264EncodeThread = new HandlerThread("H264EncodeThread");
  19.     Handler h264Handler;
  20.     HandlerThread aacEncodeThread = new HandlerThread("AACEncodeThread");
  21.     Handler aacHandler;

  23.     private static final int SAMPLE_RATE_IN_HZ = 44100;
  24.     private static final int CHANNEL_CONFIG = AudioFormat.CHANNEL_IN_MONO;
  25.     private static final int AUDIO_FORMAT = AudioFormat.ENCODING_PCM_16BIT;
  26.     private static final int BUFFER_SIZE = AudioRecord.getMinBufferSize(SAMPLE_RATE_IN_HZ, CHANNEL_CONFIG, AUDIO_FORMAT);
  27.     private static final int VIDEO_WIDTH = 1280;
  28.     private static final int VIDEO_HEIGHT = 720;
  29.     private Size previewSize;
  30.     private H264ToMp4Converter converter;

  32.     // 音频
  33.     private MediaCodec audioCodec;
  34.     private MediaCodec.BufferInfo audioBufferInfo;
  35.     private MediaFormat audioFormat;


  38.     @Override
  39.     protected void onCreate(Bundle savedInstanceState) {
  40.         super.onCreate(savedInstanceState);
  41.         setContentView(R.layout.activity_av);
  42.         textureView = findViewById(R.id.texture_view);

  44.         h264EncodeThread.start();
  45.         h264Handler = new Handler(h264EncodeThread.getLooper());
  46.         aacEncodeThread.start();
  47.         aacHandler = new Handler(aacEncodeThread.getLooper());

  49.         converter = new H264ToMp4Converter();

  51.         // 检查和请求权限
  52.         if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED ||
  53.                 ContextCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED) {
  54.             ActivityCompat.requestPermissions(this, new String[]{
  55.                     Manifest.permission.CAMERA,
  56.                     Manifest.permission.RECORD_AUDIO
  57.             }, 1);
  58.         } else {
  59.             initializeAudioRecord();
  60.         }
  61.     }

  63.     @Override
  64.     protected void onResume() {
  65.         super.onResume();
  66.         if (textureView.isAvailable()) {
  67.             initializeCamera();
  68.         } else {
  69.             textureView.setSurfaceTextureListener(surfaceTextureListener);
  70.         }
  71.     }

  73.     @Override
  74.     protected void onPause() {
  75.         super.onPause();
  76.         closeCamera();
  77.         stopVideoRecording();
  78.         stopAudioRecording();
  79.         releaseMediaCodec();

  81.         if (converter != null) {
  82.             converter.stopMuxer();
  83.         }
  84.     }

  86.     private TextureView.SurfaceTextureListener surfaceTextureListener = new TextureView.SurfaceTextureListener() {
  87.         @Override
  88.         public void onSurfaceTextureAvailable(SurfaceTexture surface, int width, int height) {
  89.             initializeCamera();
  90.             initializeMediaCodec();
  91.             initAudioEncoder();
  92.         }

  94.         @Override
  95.         public void onSurfaceTextureSizeChanged(SurfaceTexture surface, int width, int height) {
  96.         }

  98.         @Override
  99.         public boolean onSurfaceTextureDestroyed(SurfaceTexture surface) {
  100.             return false;
  101.         }

  103.         @Override
  104.         public void onSurfaceTextureUpdated(SurfaceTexture surface) {
  105.         }
  106.     };

  108.     @SuppressLint("MissingPermission")
  109.     private void initializeCamera() {
  110.         CameraManager cameraManager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);
  111.         try {
  112.             String cameraId = cameraManager.getCameraIdList()[0];
  113.             CameraCharacteristics characteristics = cameraManager.getCameraCharacteristics(cameraId);
  114.             StreamConfigurationMap map = characteristics.get(CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP);
  115.             // 获取镜头支持的尺寸
  116.             previewSize = map.getOutputSizes(SurfaceTexture.class)[0];

  118.             // 初始化 ImageReader 用于捕捉视频帧
  119.             imageReader = ImageReader.newInstance(VIDEO_WIDTH, VIDEO_HEIGHT, ImageFormat.YUV_420_888, 2);
  120.             // 放到子线程编码
  121.             imageReader.setOnImageAvailableListener(onImageAvailableListener, h264Handler);

  123.             cameraManager.openCamera(cameraId, new CameraDevice.StateCallback() {
  124.                 @Override
  125.                 public void onOpened(@NonNull CameraDevice camera) {
  126.                     cameraDevice = camera;
  127.                     createCameraPreviewSession();
  128.                 }

  130.                 @Override
  131.                 public void onDisconnected(@NonNull CameraDevice camera) {
  132.                     camera.close();
  133.                     cameraDevice = null;
  134.                 }

  136.                 @Override
  137.                 public void onError(@NonNull CameraDevice camera, int error) {
  138.                     camera.close();
  139.                     cameraDevice = null;
  140.                 }
  141.             }, null);
  142.         } catch (CameraAccessException e) {
  143.             e.printStackTrace();
  144.         }
  145.     }

  147.     private void createCameraPreviewSession() {
  148.         SurfaceTexture texture = textureView.getSurfaceTexture();
  149.         texture.setDefaultBufferSize(VIDEO_WIDTH, VIDEO_HEIGHT);
  150.         Surface surface = new Surface(texture);

  152.         Surface imageReaderSurface = imageReader.getSurface();

  154.         try {
  155.             captureRequestBuilder = cameraDevice.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW);
  156.             captureRequestBuilder.addTarget(surface);
  157.             captureRequestBuilder.addTarget(imageReaderSurface);

  159.             cameraDevice.createCaptureSession(Arrays.asList(surface, imageReaderSurface), new CameraCaptureSession.StateCallback() {
  160.                 @Override
  161.                 public void onConfigured(@NonNull CameraCaptureSession session) {
  162.                     cameraCaptureSession = session;
  163.                     try {
  164.                         cameraCaptureSession.setRepeatingRequest(captureRequestBuilder.build(), null, null);
  165.                     } catch (CameraAccessException e) {
  166.                         e.printStackTrace();
  167.                     }
  168.                 }

  170.                 @Override
  171.                 public void onConfigureFailed(@NonNull CameraCaptureSession session) {
  172.                 }
  173.             }, null);
  174.         } catch (CameraAccessException e) {
  175.             e.printStackTrace();
  176.         }
  177.     }

  179.     private ImageReader.OnImageAvailableListener onImageAvailableListener = reader -> {
  180.         Image image = null;
  181.         try {
  182.             image = reader.acquireLatestImage();
  183.             if (image != null) {
  184.                 // 将 YUV 数据编码为 H.264
  185.                 if (isVideoRecording) {
  186.                     encodeImageToH264(image);
  187.                 }
  188.             }
  189.         } finally {
  190.             if (image != null) {
  191.                 image.close();
  192.             }
  193.         }
  194.     };

  196.     @SuppressLint("MissingPermission")
  197.     private void initializeAudioRecord() {
  198.         int bufferSize = AudioRecord.getMinBufferSize(SAMPLE_RATE_IN_HZ, CHANNEL_CONFIG, AUDIO_FORMAT);
  199.         audioRecord = new AudioRecord(MediaRecorder.AudioSource.MIC, SAMPLE_RATE_IN_HZ, CHANNEL_CONFIG, AUDIO_FORMAT, bufferSize);
  200.     }

  202.     private void starVideoRecording() {
  203.         isVideoRecording = true;
  204.     }

  206.     private void stopVideoRecording() {
  207.         isVideoRecording = false;
  208.     }

  210.     private void startAudioRecording() {
  211.         audioRecord.startRecording();
  212.         isAudioRecording = true;
  213.         aacHandler.post(new AudioRecordRunnable());
  214.     }

  216.     private void stopAudioRecording() {
  217.         if (audioRecord != null) {
  218.             isAudioRecording = false;
  219.             audioRecord.stop();
  220.             audioRecord.release();
  221.             audioRecord = null;
  222.         }
  223.     }

  225.     private class AudioRecordRunnable implements Runnable {
  226.         @Override
  227.         public void run() {
  228.             byte[] buffer = new byte[BUFFER_SIZE];
  229.             while (isAudioRecording) {
  230.                 int read = audioRecord.read(buffer, 0, buffer.length);
  231.                 if (read > 0) {
  232.                     encodeAudio(buffer, read);
  233.                 }
  234.             }
  235.         }
  236.     }


  239.     private void closeCamera() {
  240.         if (cameraCaptureSession != null) {
  241.             cameraCaptureSession.close();
  242.             cameraCaptureSession = null;
  243.         }
  244.         if (cameraDevice != null) {
  245.             cameraDevice.close();
  246.             cameraDevice = null;
  247.         }
  248.     }

  250.     private void initializeMediaCodec() {
  251.         try {
  252.             mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC);
  253.             MediaFormat mediaFormat = MediaFormat.createVideoFormat(MediaFormat.MIMETYPE_VIDEO_AVC, VIDEO_WIDTH, VIDEO_HEIGHT);
  254.             mediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, 18000000);
  255.             mediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, 60);
  256.             mediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, MediaCodecInfo.CodecCapabilities.COLOR_FormatYUV420Flexible);
  257.             mediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5);
  258.             mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
  259.             mediaCodec.start();

  261.             bufferInfo = new MediaCodec.BufferInfo();
  262.             File directory = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS);
  263. //            fileOutputStream = new FileOutputStream(new File(directory, "output.h264"));
  264.         } catch (IOException e) {
  265.             e.printStackTrace();
  266.         }

  268.         starVideoRecording();
  269.     }

  271.     private void initAudioEncoder() {
  272.         try {
  273.             audioBufferInfo = new MediaCodec.BufferInfo();
  274.             audioFormat = new MediaFormat();
  275.             audioFormat.setString(MediaFormat.KEY_MIME, MediaFormat.MIMETYPE_AUDIO_AAC);
  276.             audioFormat.setInteger(MediaFormat.KEY_SAMPLE_RATE, SAMPLE_RATE_IN_HZ);
  277.             audioFormat.setInteger(MediaFormat.KEY_CHANNEL_COUNT, 1);
  278.             audioFormat.setInteger(MediaFormat.KEY_BIT_RATE, 96000);
  279.             audioFormat.setInteger(MediaFormat.KEY_AAC_PROFILE, MediaCodecInfo.CodecProfileLevel.AACObjectLC);
  280.             audioFormat.setInteger(MediaFormat.KEY_MAX_INPUT_SIZE, 1024 * 100);//作用于inputBuffer的大小

  282.             audioCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_AUDIO_AAC);
  283.             audioCodec.configure(audioFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
  284.             audioCodec.start();
  285.         } catch (IOException e) {
  286.             e.printStackTrace();
  287.         }

  289.         startAudioRecording();
  290.     }


  293.     private void releaseMediaCodec() {
  294.         if (mediaCodec != null) {
  295.             mediaCodec.stop();
  296.             mediaCodec.release();
  297.             mediaCodec = null;
  298.         }
  299.         if (audioCodec != null) {
  300.             audioCodec.stop();
  301.             audioCodec.release();
  302.             audioCodec = null;
  303.         }
  304.         if (fileOutputStream != null) {
  305.             try {
  306.                 fileOutputStream.close();
  307.             } catch (IOException e) {
  308.                 e.printStackTrace();
  309.             }
  310.             fileOutputStream = null;
  311.         }
  312.     }

  314.     private void encodeImageToH264(Image image) {
  315.         Log.i(TAG, "encodeImageToH264: thread: " + Thread.currentThread().getName());

  317.         try {
  318.             ByteBuffer[] inputBuffers = mediaCodec.getInputBuffers();
  319.             ByteBuffer[] outputBuffers = mediaCodec.getOutputBuffers();

  321.             int inputBufferIndex = mediaCodec.dequeueInputBuffer(10000); // 增加超时时间
  322.             if (inputBufferIndex >= 0) {
  323.                 Log.i(TAG, "获取到视频输入缓冲区索引: " + inputBufferIndex);
  324.                 ByteBuffer inputBuffer = inputBuffers[inputBufferIndex];
  325.                 inputBuffer.clear();

  327.                 // 将 YUV 数据填充到 inputBuffer 中
  328.                 fillInputBufferWithImageData(inputBuffer, image);

  330.                 long presentationTimeUs = System.nanoTime() / 1000;
  331. //                Date date = new Date(presentationTimeUs / 1000);
  332. //                String time = new SimpleDateFormat("HH:mm:ss SSSS", Locale.CHINA).format(date);
  333. //                Log.i(TAG, "encodeImageToH264: " + time);
  334.                 mediaCodec.queueInputBuffer(inputBufferIndex, 0, inputBuffer.limit(), presentationTimeUs, 0);
  335.             } else {
  336.                 Log.w(TAG, "未获取到有效的输入缓冲区索引");
  337.             }

  339.             int outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, 10000); // 增加超时时间
  340.             while (outputBufferIndex >= 0 && isVideoRecording) {
  341.                 Log.i(TAG, "获取到视频输出缓冲区索引: " + outputBufferIndex);
  342.                 ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
  343.                 byte[] outData = new byte[bufferInfo.size];
  344.                 outputBuffer.get(outData);
  345.                 outputBuffer.clear();

  347.                 // 直接写出 H.264 数据
  348. //                try {
  349. //                    fileOutputStream.write(outData);
  350. //                } catch (IOException e) {
  351. //                    e.printStackTrace();
  352. //                }
  353.                 // 将 H.264 数据传递给转换器
  354.                 // 初始化视频频编码轨道
  355.                 if (!converter.isAddVideoTrack()) {
  356.                     MediaFormat videoFormat = mediaCodec.getOutputFormat();
  357.                     converter.addTrack(videoFormat, true);
  358.                     converter.start();
  359.                 }
  360.                 converter.encodeToMp4(outData, bufferInfo, true);
  361.                 mediaCodec.releaseOutputBuffer(outputBufferIndex, false);
  362.                 outputBufferIndex = mediaCodec.dequeueOutputBuffer(bufferInfo, 0);
  363.             }
  364.         } catch (Exception e) {
  365.             Log.e(TAG, "编码视频时发生异常", e);
  366.         }
  367.     }

  369.     private void encodeAudio(byte[] buffer, int length) {
  370.         // 打印当前线程的名称
  371.         Log.i(TAG, "encodeAudio: thread: " + Thread.currentThread().getName());

  373.         try {
  374.             // 获取音频编解码器的输入和输出缓冲区
  375.             ByteBuffer[] inputBuffers = audioCodec.getInputBuffers();
  376.             ByteBuffer[] outputBuffers = audioCodec.getOutputBuffers();

  378.             // 尝试获取一个可用的输入缓冲区索引，等待时间为10秒（10000微秒）
  379.             int inputBufferIndex = audioCodec.dequeueInputBuffer(10000);
  380.             if (inputBufferIndex >= 0) {
  381.                 // 成功获取到输入缓冲区索引
  382.                 Log.i(TAG, "获取到音频输入缓冲区索引: " + inputBufferIndex);
  383.                 ByteBuffer inputBuffer = inputBuffers[inputBufferIndex];
  384.                 inputBuffer.clear(); // 清空缓冲区，准备写入新数据

  386.                 // 如果输入数据长度超过缓冲区容量，限制写入长度为缓冲区容量
  387.                 if (length > inputBuffer.capacity()) {
  388.                     length = inputBuffer.capacity();
  389.                 }

  391.                 // 将输入数据写入缓冲区
  392.                 inputBuffer.put(buffer, 0, length);
  393.                 // 将缓冲区排队到编解码器进行编码
  394.                 long presentationTimeUs = System.nanoTime() / 1000;
  395. //                Date date = new Date(presentationTimeUs / 1000);
  396. //                String time = new SimpleDateFormat("HH:mm:ss SSSS", Locale.CHINA).format(date);
  397. //                Log.i(TAG, "encodeAudio: " + time);
  398.                 audioCodec.queueInputBuffer(inputBufferIndex, 0, length, presentationTimeUs, 0);
  399.             } else {
  400.                 // 未获取到有效的输入缓冲区索引
  401.                 Log.w(TAG, "未获取到有效的输入缓冲区索引");
  402.             }

  404.             // 尝试获取一个可用的输出缓冲区索引，等待时间为10秒（10000微秒）
  405.             int outputBufferIndex = audioCodec.dequeueOutputBuffer(audioBufferInfo, 10000);
  406.             // 循环处理输出缓冲区，直到没有更多可用的输出缓冲区或者音频录制停止
  407.             while (outputBufferIndex >= 0 && isAudioRecording) {
  408.                 Log.i(TAG, "获取到音频输出缓冲区索引: " + outputBufferIndex);
  409.                 ByteBuffer outputBuffer = outputBuffers[outputBufferIndex];
  410.                 if (audioBufferInfo.size > 0) {
  411.                     // 初始化音频编码轨道（如果尚未初始化）
  412.                     if (!converter.isAddAudioTrack()) {
  413.                         MediaFormat audioFormat = audioCodec.getOutputFormat();
  414.                         converter.addTrack(audioFormat, false);
  415.                         converter.start();
  416.                     }
  417.                     // 设置输出缓冲区的位置和限制
  418.                     outputBuffer.position(audioBufferInfo.offset);
  419.                     outputBuffer.limit(audioBufferInfo.offset + audioBufferInfo.size);
  420.                     // 将输出数据读取到字节数组中
  421.                     byte[] outData = new byte[audioBufferInfo.size];
  422.                     outputBuffer.get(outData);
  423.                     // 将编码后的数据写入MP4文件
  424.                     converter.encodeToMp4(outData, audioBufferInfo, false);
  425.                 }
  426.                 // 释放输出缓冲区
  427.                 audioCodec.releaseOutputBuffer(outputBufferIndex, false);
  428.                 // 尝试获取下一个可用的输出缓冲区索引
  429.                 outputBufferIndex = audioCodec.dequeueOutputBuffer(audioBufferInfo, 0);
  430.             }
  431.         } catch (Exception e) {
  432.             // 捕捉并记录编码过程中发生的异常
  433.             Log.e(TAG, "编码音频时发生异常", e);
  434.         }
  435.     }


  438.     private void fillInputBufferWithImageData(ByteBuffer inputBuffer, Image image) {
  439.         Image.Plane[] planes = image.getPlanes();
  440.         for (Image.Plane plane : planes) {
  441.             ByteBuffer buffer = plane.getBuffer();
  442.             byte[] bytes = new byte[buffer.remaining()];
  443.             buffer.get(bytes);
  444.             inputBuffer.put(bytes);
  445.         }
  446.     }
  447. }

H264ToMp4Converter.java

  1. public class H264ToMp4Converter {

  3.     private static final String TAG = "H264ToMp4Converter";
  4.     static File directory = Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS);
  5.     private static final String OUTPUT_FILE_PATH = new File(directory, "output.mp4").toString();

  7.     private MediaMuxer mediaMuxer;
  8.     private int videoTrackIndex = -1;
  9.     private int audioTrackIndex = -1;
  10.     private boolean isMuxerStarted = false;

  12.     public H264ToMp4Converter() {
  13.         try {
  14.             mediaMuxer = new MediaMuxer(OUTPUT_FILE_PATH, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
  15.         } catch (IOException e) {
  16.             Log.e(TAG, "Failed to initialize MediaMuxer", e);
  17.         }
  18.     }

  20.     public void stopMuxer() {
  21.         if (isMuxerStarted) {
  22.             mediaMuxer.stop();
  23.             mediaMuxer.release();
  24.             isMuxerStarted = false;
  25.         }
  26.     }

  28.     public void addTrack(MediaFormat format, boolean isVideo) {
  29.         if (isVideo) {
  30.             videoTrackIndex = mediaMuxer.addTrack(format);
  31.             Log.i(TAG, "addTrack: videoTrackIndex:" + videoTrackIndex);
  32.         } else {
  33.             audioTrackIndex = mediaMuxer.addTrack(format);
  34.             Log.i(TAG, "addTrack: audioTrackIndex:" + audioTrackIndex);
  35.         }
  36.     }

  38.     public void start() {
  39.         if (isAddAudioTrack() && isAddVideoTrack()) {
  40.             mediaMuxer.start();
  41.             isMuxerStarted = true;
  42.         }
  43.     }

  45.     public void encodeToMp4(byte[] data, MediaCodec.BufferInfo info, boolean isVideo) {
  46.         if (!isMuxerStarted) {
  47.             return;
  48.         }

  50.         // 确保 presentationTimeUs 递增且正确
  51.         Date date = new Date(info.presentationTimeUs / 1000);
  52.         String time = new SimpleDateFormat("HH:mm:ss SSSS", Locale.CHINA).format(date);
  53.         Log.w(TAG, "presentation time: isVideo: "+isVideo + " time: " + time);

  55.         ByteBuffer byteBuffer = ByteBuffer.wrap(data);
  56.         int trackIndex = isVideo ? videoTrackIndex : audioTrackIndex;
  57.         mediaMuxer.writeSampleData(trackIndex, byteBuffer, info);
  58.     }

  60.     public boolean isMuxerStarted() {
  61.         return isMuxerStarted;
  62.     }

  64.     public boolean isAddVideoTrack() {
  65.         return videoTrackIndex != -1;
  66.     }

  68.     public boolean isAddAudioTrack() {
  69.         return audioTrackIndex != -1;
  70.     }
  71. }

解释

  1. 初始化 ImageReader 并设置监听器：
    在 createCameraPreviewSession 方法中，初始化 ImageReader 并将其添加为摄像头捕获的目标。

  2. 捕捉每一帧视频数据：
    在 onImageAvailableListener 中获取每一帧的 Image 对象，并调用 encodeImageToH264 方法对其进行编码。

  3. 初始化 MediaCodec：
    初始化 MediaCodec 编码器，设置编码参数并开始编码。

  4. 编码 YUV 数据为 H.264：
    在 encodeImageToH264 方法中，将 Image 对象的 YUV 数据填充到 MediaCodec 的输入缓冲区，并从输出缓冲区获取编码后的 H.264 数据，写入文件。

通过以上步骤，我们实现了在 Android 上使用 ImageReader 捕捉视频帧，并使用 MediaCodec 将原始 YUV420 数据编码为 H.264 格式。这样我们就可以实时处理和存储摄像头捕获的视频数据。

上一篇： java 和 jni 相互调用