Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Berry: add cam module, img class #21743

Merged
merged 3 commits into from
Jul 7, 2024
Merged

Berry: add cam module, img class #21743

merged 3 commits into from
Jul 7, 2024

Conversation

Staars
Copy link
Contributor

@Staars Staars commented Jul 7, 2024

Description:

The main intention of this PR is to free up as much heap memory as possible for machine learning applications while using a connected camera module. I measured about 20 kB heap use reduction. This is not meant as replacement for any of the other webcam drivers. Uses JPEGDEC library with SIMD support on the ESP32-S3.

Compile with:

-DUSE_BERRY_CAM ; needs -DUSE_BERRY_IMAGE too
-DUSE_BERRY_IMAGE ; could be used without -DUSE_BERRY_CAM
  • this will strip down the main webcam driver to the setup functions only, but will not influence the normal webcam firmware in any way
  • add webcam module to Berry with simple picture acquisition
  • add img class to Berry as a thin container for image data in order to read images from the camera or from a file, while providing conversions and extraction of regions of interest

Tested with a self trained model to read 7 segment digits of a display (which could be an "unsmart" device of any kind) with still about 80 kB free heap on a cheap ESP32 cam board with 4 MB PS-RAM.

More examples:

  • Show camera data on a connected display as a video (tested on ESP32-S3-Eye)
  • Show images from an external camera (another ESP32 in LAN with the normal webcam driver) on a connected display
  • Use image data in LVGL as image source

I would update Tasmotas docs with the content of my preliminary documentation: https://staars.github.io/docs/Berry_import/#img-class

Checklist:

  • The pull request is done against the latest development branch
  • Only relevant files were touched
  • Only one feature/fix was added per PR and the code change compiles without warnings
  • The code change is tested and works with Tasmota core ESP8266 V.2.7.7
  • The code change is tested and works with Tasmota core ESP32 V.3.0.2
  • I accept the CLA.

NOTE: The code change must pass CI tests. Your PR cannot be merged unless tests pass

@Staars
Copy link
Contributor Author

Staars commented Jul 7, 2024

C family builds fail because of missing dsp library in our framework.

@Jason2866 Can we add this back for these chips? It will not increase firmware size if not used.

Both additions are 99,9% pointless for PSRAM-less boards anyway.

@Jason2866
Copy link
Collaborator

@Staars Good timing. Planning a Platform release for tomorrow. The CI should be green with the updated Platform build.

@Staars
Copy link
Contributor Author

Staars commented Jul 7, 2024

A new planned framework update is planned anyway, which should solve the compile issue for the C family of chips.
I would suggest to keep this PR open and we'll see, if the build failures will disappear.

(Parallel comments - see above)

@Jason2866 Jason2866 merged commit 7cb8a3f into arendst:development Jul 7, 2024
59 checks passed
@Staars Staars deleted the cam branch July 11, 2024 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants