NOTE: SDK v9.0+ is required to use this
NOTE: an OpenRC-based SDK is NOT required to use this
Finally, a straightforward and easy way to use OpenCV on an FTC robot! With this library, you can go from a stock SDK to running a sample OpenCV OpMode, with either an internal or external camera, in just a few minutes!
Features at a glance:
- Supports concurrent streaming from:
- An internal camera and a webcam
- Two webcams
- Two internal cameras (select devices; internal cameras must not share the same bus)
- Supports Driver Station camera preview feature introduced in SDK v5.1
- Supports tapping on the viewport to cycle through the various stages of a pipeline (see PipelineStageSwitchingExample)
- Supports using webcams directly with OpenCV instead of going through a Vuforia instance
- Supports changing pipelines on-the-fly (while a streaming session is in flight)
- Supports dynamically pausing/resuming live viewport to save battery and CPU time
- Support for rotating stream based on physical camera orientation (e.g. use a webcam in portrait without having to mess with rotation yourself)
Unfortunately, due to a known bug with OpenCV 4.x, EasyOpenCV is only compatible with devices that run Android 5.0 or higher. For FTC, this means that it is incompatible with the ZTE Speed. EasyOpenCV will work fine on all other FTC-legal devices (including the new Control Hub).
IMPORTANT NOTE: EasyOpenCV delivers RGBA frames, but desktop OpenCV (what you may be used to) delivers BGR frames. Beware when porting code between the two!
As of FTC SDK v8.2, EasyOpenCV is now packaged with the SDK as part of the Vision module. No manual installation is neccesary whether you are using Android Studio or Blocks.
PLEASE NOTE THAT THERE IS CURRENTLY A KNOWN BUG IN v9.0 and v9.0.1 OF THE FTC SDK WHICH PREVENTS IMPORTING THE OpenCvCameraFactory
CLASS IN OBJ
- This means you cannot use the "raw" EOCV APIs from OBJ
- As a workaround, you can port your pipeline to a
VisionProcessor
and use the Vision Portal API. There is a chapter in the LearnJava4FTC book that covers this - Alternatively, you can switch to using Android Studio.
- THIS WILL BE FIXED IN A FUTURE RELEASE OF THE FTC SDK
- Updates
libjpeg-turbo
to 3.0.3 - Updates OpenCV to v4.10.0
- Show more detail about camera supported resolutions and formats when user requests unsupported configuration
- Show warning if performance could be improved by using MJPEG format
- Fix race conditions when handling exception during viewport insertion
- IMPORTANT NOTE: SDK v9.0 or higher is now required!
- Build against FTC SDK v9.x; remove support for Vuforia integration
- Fixes inability to use CameraFactory in OnBotJava due to reference of Vuforia classes no longer in SDK
- Fixes entire RC app crash due to ABI breakage when user pipeline throws an exception
- Handle exceptions in user-provided canvas annotators the same way exceptions in pipelines are handled to prevent entire RC app crash
- Use the modern stacktrace display when handling user exceptions instead of the legacy ESTOP telemetry message
- Adds new
NATIVE_VIEW
viewport renderer option- Attempts to provide a balance between the stability of the
SOFTWARE
renderer and the speedup seen with theGPU_ACCELERATED
renderer - Drawing is done on the UI thread using the GPU accelerated main canvas, instead of making another canvas as the
GPU_ACCELERATED
renderer does
- Attempts to provide a balance between the stability of the
- Uses anti-aliasing when drawing the statistics overlay to make text more readable on low resolution screens
- Adds ability for user pipelines to hook into the Canvas rendering of frames to the live view
- This is an alternate means besides OpenCV calls to draw annotations from a pipeline
- Allows drawing annotations at the full screen resolution even if performing image processing at a much lower resolution (e.g. 320x240). This allows e.g. drawing actually legible text on a low res image feed.
- Because live view rendering happens on a different thread than the image processing, in order to make use of this feature, pipelines must call
requestViewportDrawHook(object)
duringprocessFrame()
, providing any type of context object they may wish which encapsulates the data needed to draw the annotations. Pipelines then also need to overrideonDrawFrame(...)
which can receive that same object back, and perform the annotation drawing there. - The image sent to the DriverStation is now rendered using an offsceen canvas to ensure that annotations rendered using the canvas will be visible on the DS as well.
- Adds support for MJPEG streaming for webcams
- Requires FTC SDK v8.2
- Uses libjpeg-turbo for JPEG decompression routine
- Allows for streaming at full frame rate at higher resolutions (e.g. 1280x720) which were previously limited to 10FPS due to bandwidth constraints
- Improves ability to use multiple cameras simultaneously by reducing bandwidth usage
- CPU load will be increased due to additional overhead from JPEG decompression
- Uncompressed streaming is still the default; in order to request MJPEG, use the overloaded
startStreaming()
method inOpenCvWebcam
which takes aStreamFormat
argument
- Fixes a deadlock when trying to switch cameras when using
OpenCvSwitchableWebcam
- Fixes cases where mutex might not be released in internal camera v2 implementation
- Updates OpenCV-Repackaged transitive dependency to
4.7.0-A
- Add generic
getControl()
method to OpenCvWebcam - Fix corrupted camera frame delivery when using Camera2 API on some devices
- Prevent deadlock if pipeline tried to perform a synchronized UI thread operation and the device orientation was changed
- Fixes bug where if using a webcam, frames were not delivered to user pipeline when calling
startStreaming()
after a previous call tostopStreaming()
even though the stream was in fact restarted successfully (#65) - Scales viewport statistics overlay based on pixel density so that it's not overly large on some devices
- Add support for getting WhiteBalanceControl for webcams
- Handle pipeline returning empty Mat for viewport display with an error message instead of an unclear exception
- Add SENSOR_NATIVE to camera rotation enum
- Desynchronize setPipeline() from active pipeline frame processing (fixes #58)
- Synchronize getting webcam controls with opening/closing camera
- Add support for getting the CameraCalibrationIdentity for an OpenCvWebcam
- Improve memory leak detection warning
- Dependency on OpenCV-Repackaged has been changed to a version which bundles the OpenCV native library with the artifact instead of requiring it to be copied to external storage manually. This change was made because:
- The original reason for not bundling the native library was to reduce APK size for wireless deploy time, but that was something that it seems I cared about more than anyone else, and people often seem to have difficulty with setting it up properly for some reason
- There have been multiple requests for 64-bit support, which would have made the dynamic loading from external storage even more complicated
- 64-bit support added
- Increases default webcam permission timeout to 5 seconds
- Removes app name resource strings which shouldn't have ever been there in the first place
- Fixes compatibility with SDK v8.0. You MUST use v1.5.2 (or later) for SDK 8.0. Previous versions Will not work!! Backwards compatibility is NOT maintained for this release unfortunately!
- Fixes possible leak of framebuffer when viewport render thread was restarted
- Fixes crash with SDK v7.0 when memory leak warning was generated
- Fixes compatibility with SDK v7.0
- You MUST use 1.5.0 (or later) for SDK 7.0. Previous versions Will not work!!
- Backwards compatibility with SDK v6.1 is maintained.
- First release supporting OnBotJava! (See setup instructions)
- API CHANGE: OpenCV core upgraded to OpenCV v4.5.3 (transitive dependency on
opencv-repackaged
updated to4.5.3-B
)- This change also requires an updated native library to be copied to the device (see installation instructions above)
- Failure to open the camera device is now properly handled (previously, the
onOpened()
callback would be called even in the case of failure)- API CHANGE: User-defined
AsyncCameraOpenListener
instances must now also implement thevoid onError(int errorCode)
function
- API CHANGE: User-defined
- Change webcam opening timeout to be user-configurable (new function
void setMillisecondsPermissionTimeout(int ms)
added) - Fix race condition when closing camera which could cause the camera worker thread to crash with a null pointer when trying to send a frame to the viewport
- Fix issue with viewport where user-drawn parts of the image (e.g. rect boxes) would not appear in the correct color unless an alpha parameter for the color was specified
- Fix bug where Camera2 backend was broken on some devices due to reading the image timestamp after closing the Image object
- Samples moved to
org.firstinspires.ftc.teamcode
package
- Add support for Vuforia passthrough mode, which allows running Vuforia and OpenCV simultaneously on the same camera. Please see OpenCvAndVuforiaOnSameCameraExample.
- IMPORTANT NOTE: SDK v6.1 or higher is now required!
- Add support for additional webcam controls introduced in SDK v6.1
- Add
saveMatToDiskFullPath()
method to pipeline class - Add
TimestampedOpenCvPipeline
class which extendsOpenCvPipeline
and delivers capture time timestamps along with frames. See newTimestampedPipelineExample
file.
- Add ability to set FocusMode to Internal camera v1 API
- General improvements to Internal camera v2 API
- Make startStreaming() check to make sure camera is opened
- Make startStreaming() check to make sure requested resolution is supported
- Fix potential failure to release mutex
- Cleaned up the conversion of raw camera data into the Mat by using some native C++ code
- Viewport improvements
- Image is now centered inside the SurfaceView
- Unused space of the surface view is set to the same color as the activity background color
- Added optional GPU-accelerated rendering mode for viewport
- Warn instead of throwing when setting viewport rendering policy for webcams
- Optimized webcam frame delivery by using some native C++ code to avoid unnecessary hidden
memcpy
operations - Add API for recording pipelines to a video file
- This is a BETA API and may be unstable, and subject to change! Caveat emptor.
- Fix memory leak detector to trip only after settle delay, instead of only before settle delay.
- Transitive dependency on OpenCV-Repackged updated to 4.1.0-C, which specifically handles error case of failure to load 32-bit library when FTC Robot Controller app has already loaded another native library as 64-bit
- Fixes issue which prevented webcams from initializing in v1.4.0 which was found in prerelease testing, fixed, and yet somehow didn't make it into git...
- Adds support for Android Camera2 API
- New
OpenCvInternalCamera2
interface. Camera2 instances can be obtained fromOpenCvCameraFactory
, just like other types - Supports manual control over sensor parameters:
- ISO (gain)
- Exposure
- Focus
- White balance
- Frame interval (FPS)
- New
- Make
OpenCvCamera
interface extendCameraStreamSource
so that casting to implementation objects isn't required to use a camera as a stream source for something other than the DS - Adds
setViewportRenderingPolicy()
API toOpenCvCamera interface
, provides option to:MAXIMIZE_EFFICIENCY
Keep viewport behavior as it always has been, OROPTIMIZE_VIEW
At the expense of CPU time (and viewport smoothness), automatically orient preview image such that it's not constantly 90 degrees out from expected with an internal camera when the physical device orientation does not match the streaming orientation
- Add memory leak detector for pipelines
- Not 100% accurate but, seems to be fairly effective
- Has a crude garbage collector run detector
- Can be enabled/disabled or have parameters tweaked by modifying superclass variables from your pipeline constructor
- Add
init(Mat m)
method to pipeline class, which will be called with the first frame from the camera, allowing you to initialize submats and the like for your pipeline - Adds pipeline utility function for saving Mats to disk.
- Save function clones input mat and writes to disk asynchronously to prevent stalling pipeline
- Up to 5 save operations can be running simultaneously; once this limit is reached, the pipeline will be stalled until one has completed
- Adds APIs for closing and opening the camera asynchronously. This is now the recommended way to open and close, as it can help to prevent
stuckInXYZ()
issues and the like. Please consult theOpenCvCamera
interface javadoc for details - Adds support for switchable webcams
- New
OpenCvSwitchableWebcam
interface. Instances can be obtained fromOpenCvCameraFactory
, just like other types
- New
- Fix deadlock when closing webcams
- Increase webcam open timeout to 2 seconds. This increases compatibility with random nobrand cameras.
- Adds new
OpenCvWebcam
interface which exposes some additional functionality for webcams- Instances can be obtained from
OpenCvCameraFactory
, just like other types - Support for exposure control & focus control, using the SDK UVC driver's built-in interfaces
- Instances can be obtained from
- Adds new samples demonstrating some of the new functionality
- Library version now printed to logcat when creating camera instance
- Misc. bug fixes
- Resolutions >480p are now possible with webcams (at reduced framerates)
- Add exposure compensation and autoexposure lock APIs for internal camera
- Fix blank display when user pipeline returned cropped mat of type CV_8UC1 (e.g. masks)
- Print supported resolutions when user selects illegal resolution for camera
- Transitive dependency on OpenCV-Repackged updated to 4.1.0-B, which drastically improves error handling when loading native library
- Add official support for multiple concurrent camera streams (was possible before but required manual activity UI modifications)
- Also allows for running Vuforia alongside EasyOpenCV
- Add "TrackerAPI" classes (ability to run multiple OpenCV algorithms in the same pipeline, and switch between which output is rendered to the screen in realtime by tapping the viewport)
- Add support for rendering cropped returns from user pipeline
- A little internal code cleanup
- Optimise viewport to re-use existing framebuffer memory
- Fix issue where if a user pipeline created a submat from the input Mat, the submat would be de-linked from the input buffer on the next frame
- Added ability to use some advanced features for internal cameras:
- Added ability to set "recording hint"
- Added ability to set "hardware frame timing range"
- Added ability to control zoom
- Added ability to control flashlight
- Added support for using double buffering (default; can improve FPS)
- API change: camera instances are now created by invoking
OpenCvCameraFactory.getInstance().create...
- Add examples:
- InternalCameraAdvancedFeaturesExample
- MultipleCameraExample
- MultipleCameraExampleOpenCvAlongsideVuforia
- TrackerApiExample
- HOTFIX: implement workaround for SDK bug of RenderScript failing to initialize on some devices which prevented the webcam frames from being forwarded through the JNI to the Java side (See issue #1)
- SDK v5.1 or higher now required
- Add support for stream preview on Driver Station
- Fix bug where internal camera was not correctly released
- Fix bug where a null pipeline caused a crash
- API change: user pipelines now need to
extends OpenCvPipeline
instead ofimplements OpenCvPipeline
- Add ability for user pipeline to override
onViewportTapped()
to be notified if the user taps the viewport - Add
PipelineStageSwitchingExample
to show how to useonViewportTapped()
to change which stage of your pipeline is drawn to the viewport for debugging purposes. It also shows how to get data from your pipeline to your OpMode.
- Initial release