Frame processors

README.md

@@ -39,11 +39,12 @@ See below for a [list of what was done](#roadmap) and [licensing info](#contribu
   - Take high-resolution pictures with `capturePicture`
   - Take quick snapshots as a freeze frame of the preview with `captureSnapshot` (similar to Snapchat and Instagram)
 - Control HDR, flash, zoom, white balance, exposure correction and more
+- **Frame processing** support
 - **Metadata** support for pictures and videos
   - Automatically detected orientation tags
   - Plug in location tags with `setLocation()` API
 - `CameraUtils` to help with Bitmaps and orientations
-- Lightweight, no dependencies, just support `ExifInterface`
+- **Lightweight**, no dependencies, just support `ExifInterface`
 - Works down to API level 15
 
 # Docs
@@ -57,12 +58,12 @@ See below for a [list of what was done](#roadmap) and [licensing info](#contribu
   - [Center Inside](#center-inside)
   - [Center Crop](#center-crop)
 - [Camera Controls](#camera-controls)
+- [Frame Processing](#frame-processing)
 - [Other APIs](#other-apis)  
 - [Permissions Behavior](#permissions-behavior)
-- [Manifest file](#manifest-file)
 - [Logging](#logging)
-- [Roadmap](#roadmap)
 - [Device-specific issues](#device-specific-issues)
+- [Roadmap](#roadmap)
 
 ## Usage
 
@@ -437,6 +438,43 @@ cameraView.setPlaySounds(true);
 cameraView.setPlaySounds(false);
 ```
 
+## Frame Processing
+
+We support frame processors that will receive data from the camera preview stream:
+
+```java
+cameraView.addFrameProcessor(new FrameProcessor() {
+    @Override
+    @WorkerThread
+    public void process(Frame frame) {
+        byte[] data = frame.getData();
+        int rotation = frame.getRotation();
+        long time = frame.getTime();
+        Size size = frame.getSize();
+        int format = frame.getFormat();
+        // Process...
+    }
+}
+```
+
+For your convenience, the `FrameProcessor` method is run in a background thread so you can do your job
+in a synchronous fashion. Once the process method returns, internally we will re-use the `Frame` instance and
+apply new data to it. So:
+
+- you can do your job synchronously in the `process()` method
+- if you must hold the `Frame` instance longer, use `frame = frame.freeze()` to get a frozen instance
+  that will not be affected
+
+|Frame API|Type|Description|
+|---------|----|-----------|
+|`frame.getData()`|`byte[]`|The current preview frame, in its original orientation.|
+|`frame.getTime()`|`long`|The preview timestamp, in `System.currentTimeMillis()` reference.|
+|`frame.getRotation()`|`int`|The rotation that should be applied to the byte array in order to see what the user sees.|
+|`frame.getSize()`|`Size`|The frame size, before any rotation is applied, to access data.|
+|`frame.getFormat()`|`int`|The frame `ImageFormat`. This will always be `ImageFormat.NV21` for now.|
+|`frame.freeze()`|`Frame`|Clones this frame and makes it immutable. Can be expensive because requires copying the byte array.|
+|`frame.release()`|`-`|Disposes the content of this frame. Should be used on frozen frames to release memory.|
+
 ## Other APIs
 
 Other APIs not mentioned above are provided, and are well documented and commented in code.
@@ -461,7 +499,7 @@ Other APIs not mentioned above are provided, and are well documented and comment
 |`getSnapshotSize()`|Returns `getPreviewSize()`, since a snapshot is a preview frame.|
 |`getPictureSize()`|Returns the size of the output picture. The aspect ratio is consistent with `getPreviewSize()`.|
 
-Take also a look at public methods in `CameraUtils`, `CameraOptions`, `ExtraProperties`, `CameraLogger`.
+Take also a look at public methods in `CameraUtils`, `CameraOptions`, `ExtraProperties`.
 
 ## Permissions behavior
 
@@ -470,13 +508,13 @@ Take also a look at public methods in `CameraUtils`, `CameraOptions`, `ExtraProp
 - `android.permission.CAMERA` : required for capturing pictures and videos
 - `android.permission.RECORD_AUDIO` : required for capturing videos with `Audio.ON` (the default)
 
-You can handle permissions yourself and then call `CameraView.start()` once they are acquired. If they are not, `CameraView` will request permissions to the user based on whether they are needed. In that case, you can restart the camera if you have a successful response from `onRequestPermissionResults()`.
+### Declaration
 
-## Manifest file
+The library manifest file declares the `android.permission.CAMERA` permission, but not the audio one.
+This means that:
 
-The library manifest file is not strict and only asks for camera permissions. This means that:
-
-- If you wish to record videos with `Audio.ON` (the default), you should also add `android.permission.RECORD_AUDIO` to required permissions
+- If you wish to record videos with `Audio.ON` (the default), you should also add
+  `android.permission.RECORD_AUDIO` to required permissions
 
 ```xml
 <uses-permission android:name="android.permission.RECORD_AUDIO"/>
@@ -490,7 +528,18 @@ The library manifest file is not strict and only asks for camera permissions. Th
     android:required="true"/>
 ```
 
-If you don't request this feature, you can use `CameraUtils.hasCameras()` to detect if current device has cameras, and then start the camera view.
+If you don't request this feature, you can use `CameraUtils.hasCameras()` to detect if current
+device has cameras, and then start the camera view.
+
+### Handling
+
+On Marshmallow+, the user must explicitly approve our permissions. You can
+
+- handle permissions yourself and then call `cameraView.start()` once they are acquired
+- or call `cameraView.start()` anyway: `CameraView` will present a permission request to the user based on
+  whether they are needed or not with the current configuration.
+
+In the second case, you should restart the camera if you have a successful response from `onRequestPermissionResults()`.
 
 ## Logging
 
@@ -512,9 +561,21 @@ CameraLogger.registerLogger(new Logger() {
 Make sure you enable the logger using `CameraLogger.setLogLevel(@LogLevel int)`. The default will only
 log error events.
 
+## Device-specific issues
+
+There are a couple of known issues if you are working with certain devices. The emulator is one of
+the most tricky in this sense.
+
+- Devices, or activities, with hardware acceleration turned off: this can be the case with emulators.
+  In this case we will use SurfaceView as our surface provider. That is intrinsically flawed and can't
+  deal with all we want to do here (runtime layout changes, scaling, etc.). So, nothing to do in this case.
+- Devices with no support for MediaRecorder: the emulator does not support it, officially. This means
+  that video/audio recording is flawed. Again, not our fault.
+
 ## Roadmap
 
-This is what was done since the library was forked. I have kept the original structure, but practically all the code was changed.
+This is what was done since the library was forked. I have kept the original structure, but practically
+all the code was changed.
 
 - *a huge number of serious bugs fixed*
 - *decent orientation support for both pictures and videos*
@@ -542,28 +603,18 @@ This is what was done since the library was forked. I have kept the original str
 - *Tests!*
 - *`CameraLogger` APIs for logging and bug reports*
 - *Better threading, start() in worker thread and callbacks in UI*
+- *Frame processor support*
+- *inject external loggers*
 
 These are still things that need to be done, off the top of my head:
 
 - [ ] `Camera2` integration
-- [ ] check onPause / onStop / onSaveInstanceState consistency
 - [ ] add a `setPreferredAspectRatio` API to choose the capture size. Preview size will adapt, and then, if let free, the CameraView will adapt as well
 - [ ] animate grid lines similar to stock camera app
 - [ ] add onRequestPermissionResults for easy permission callback
 - [ ] better error handling, maybe with a onError(e) method in the public listener, or have each public method return a boolean
 - [ ] decent code coverage
 
-## Device-specific issues
-
-There are a couple of known issues if you are working with certain devices. The emulator is one of
-the most tricky in this sense.
-
-- Devices, or activities, with hardware acceleration turned off: this can be the case with emulators.
-  In this case we will use SurfaceView as our surface provider. That is intrinsically flawed and can't
-  deal with all we want to do here (runtime layout changes, scaling, etc.). So, nothing to do in this case.
-- Devices with no support for MediaRecorder: the emulator does not support it, officially. This means
-  that video/audio recording is flawed. Again, not our fault.
-
 # Contributing and licenses
 
 The original project which served as a starting point for this library,

cameraview/src/androidTest/java/com/otaliastudios/cameraview/CameraCallbacksTest.java

@@ -38,6 +38,7 @@ public class CameraCallbacksTest extends BaseTest {
 
     private CameraView camera;
     private CameraListener listener;
+    private FrameProcessor processor;
     private MockCameraController mockController;
     private MockCameraPreview mockPreview;
     private Task<Boolean> task;
@@ -50,6 +51,7 @@ public void setUp() {
             public void run() {
                 Context context = context();
                 listener = mock(CameraListener.class);
+                processor = mock(FrameProcessor.class);
                 camera = new CameraView(context) {
                     @Override
                     protected CameraController instantiateCameraController(CameraCallbacks callbacks) {
@@ -70,6 +72,7 @@ protected boolean checkPermissions(SessionType sessionType, Audio audio) {
                 };
                 camera.instantiatePreview();
                 camera.addCameraListener(listener);
+                camera.addFrameProcessor(processor);
                 task = new Task<>();
                 task.listen();
             }
@@ -294,4 +297,13 @@ public Object answer(InvocationOnMock invocation) throws Throwable {
         Bitmap bitmap = BitmapFactory.decodeByteArray(result, 0, result.length);
         return new int[]{ bitmap.getWidth(), bitmap.getHeight() };
     }
+
+    @Test
+    public void testProcessFrame() {
+        completeTask().when(processor).process(any(Frame.class));
+        camera.mCameraCallbacks.dispatchFrame(new byte[]{0, 1, 2, 3}, 1000, 90, new Size(1, 1), 0);
+
+        assertNotNull(task.await(200));
+        verify(processor, times(1)).process(any(Frame.class));
+    }
 }

cameraview/src/androidTest/java/com/otaliastudios/cameraview/CameraViewTest.java

@@ -3,6 +3,7 @@
 
 import android.content.Context;
 import android.location.Location;
+import android.support.annotation.NonNull;
 import android.support.test.filters.MediumTest;
 import android.support.test.runner.AndroidJUnit4;
 import android.view.MotionEvent;
@@ -551,5 +552,49 @@ public void testVideoQuality() {
 
     //endregion
 
+    //region Lists of listeners and processors
+
+    @Test
+    public void testCameraListenerList() {
+        assertTrue(cameraView.mListeners.isEmpty());
+
+        CameraListener listener = new CameraListener() {};
+        cameraView.addCameraListener(listener);
+        assertEquals(cameraView.mListeners.size(), 1);
+
+        cameraView.removeCameraListener(listener);
+        assertEquals(cameraView.mListeners.size(), 0);
+
+        cameraView.addCameraListener(listener);
+        cameraView.addCameraListener(listener);
+        assertEquals(cameraView.mListeners.size(), 2);
+
+        cameraView.clearCameraListeners();
+        assertTrue(cameraView.mListeners.isEmpty());
+    }
+
+    @Test
+    public void testFrameProcessorsList() {
+        assertTrue(cameraView.mFrameProcessors.isEmpty());
+
+        FrameProcessor processor = new FrameProcessor() {
+            public void process(@NonNull Frame frame) {}
+        };
+        cameraView.addFrameProcessor(processor);
+        assertEquals(cameraView.mFrameProcessors.size(), 1);
+
+        cameraView.removeFrameProcessor(processor);
+        assertEquals(cameraView.mFrameProcessors.size(), 0);
+
+        cameraView.addFrameProcessor(processor);
+        cameraView.addFrameProcessor(processor);
+        assertEquals(cameraView.mFrameProcessors.size(), 2);
+
+        cameraView.clearFrameProcessors();
+        assertTrue(cameraView.mFrameProcessors.isEmpty());
+    }
+
+    //endregion
+
     // TODO: test permissions
 }

cameraview/src/androidTest/java/com/otaliastudios/cameraview/IntegrationTest.java

@@ -504,4 +504,24 @@ public void testCaptureSnapshot_size() throws Exception {
         assertTrue(b.getHeight() == size.getHeight() || b.getHeight() == size.getWidth());
     }
 
+    //endregion
+
+    //region Frame Processing
+
+    @Test
+    public void testFrameProcessing() throws Exception {
+        FrameProcessor processor = mock(FrameProcessor.class);
+        camera.addFrameProcessor(processor);
+
+        camera.start();
+        waitForOpen(true);
+
+        // Expect 30 frames
+        CountDownLatch latch = new CountDownLatch(30);
+        doCountDown(latch).when(processor).process(any(Frame.class));
+        boolean did = latch.await(4, TimeUnit.SECONDS);
+        assertTrue(did);
+    }
+
+    //endregion
 }

cameraview/src/main/java/com/otaliastudios/cameraview/Camera1.java

@@ -21,7 +21,7 @@
 
 
 @SuppressWarnings("deprecation")
-class Camera1 extends CameraController {
+class Camera1 extends CameraController implements Camera.PreviewCallback {
 
     private static final String TAG = Camera1.class.getSimpleName();
     private static final CameraLogger LOG = CameraLogger.create(TAG);
@@ -143,10 +143,14 @@ private void setup() throws Exception {
         );
         synchronized (mLock) {
             Camera.Parameters params = mCamera.getParameters();
+            mPreviewFormat = params.getPreviewFormat();
             params.setPreviewSize(mPreviewSize.getWidth(), mPreviewSize.getHeight()); // <- not allowed during preview
             params.setPictureSize(mCaptureSize.getWidth(), mCaptureSize.getHeight()); // <- allowed
             mCamera.setParameters(params);
         }
+
+        mCamera.setPreviewCallback(this); // Frame processing
+
         LOG.i("setup:", "Starting preview with startPreview().");
         mCamera.startPreview();
         LOG.i("setup:", "Started preview with startPreview().");
@@ -475,7 +479,6 @@ public void onPreviewFrame(final byte[] data, Camera camera) {
                 // Got to rotate the preview frame, since byte[] data here does not include
                 // EXIF tags automatically set by camera. So either we add EXIF, or we rotate.
                 // Adding EXIF to a byte array, unfortunately, is hard.
-                Camera.Parameters params = mCamera.getParameters();
                 final int sensorToDevice = computeExifRotation();
                 final int sensorToDisplay = computeSensorToDisplayOffset();
                 final boolean exifFlip = computeExifFlip();
@@ -484,7 +487,7 @@ public void onPreviewFrame(final byte[] data, Camera camera) {
                 final int preHeight = mPreviewSize.getHeight();
                 final int postWidth = flip ? preHeight : preWidth;
                 final int postHeight = flip ? preWidth : preHeight;
-                final int format = params.getPreviewFormat();
+                final int format = mPreviewFormat;
                 WorkerHandler.run(new Runnable() {
                     @Override
                     public void run() {
@@ -496,11 +499,21 @@ public void run() {
                         mIsCapturingImage = false;
                     }
                 });
+                mCamera.setPreviewCallback(Camera1.this);
             }
         });
         return true;
     }
 
+    @Override
+    public void onPreviewFrame(byte[] data, Camera camera) {
+        mCameraCallbacks.dispatchFrame(data,
+                System.currentTimeMillis(),
+                computeExifRotation(),
+                mPreviewSize,
+                mPreviewFormat);
+    }
+
     @Override
     boolean shouldFlipSizes() {
         int offset = computeSensorToDisplayOffset();

cameraview/src/main/java/com/otaliastudios/cameraview/CameraController.java

@@ -32,6 +32,7 @@ abstract class CameraController implements CameraPreview.SurfaceCallback {
 
     protected Size mCaptureSize;
     protected Size mPreviewSize;
+    protected int mPreviewFormat;
 
     protected ExtraProperties mExtraProperties;
     protected CameraOptions mOptions;