Android R migration #336

courville · 2020-04-26T12:02:45Z

In order to target SDK 30 you need:

diff --git a/build.gradle b/build.gradle
index ea5d6de..bd9c633 100644
--- a/build.gradle
+++ b/build.gradle
@@ -115,7 +115,7 @@ android {
         sourceCompatibility JavaVersion.VERSION_1_8
         targetCompatibility JavaVersion.VERSION_1_8
     }
-    compileSdkVersion 29
+    compileSdkVersion 'android-R'
     buildToolsVersion '29.0.3'
     sourceSets {
         main {
@@ -130,7 +130,7 @@ android {
     defaultConfig {
         generatedDensities = []
         minSdkVersion 21
-        targetSdkVersion 29
+        targetSdkVersion 'R'
         multiDexEnabled true
         versionCode getDate()
         version = '4.48.1'

When launching the app you get this obvious introspection not allowed anymore to solve:

2020-04-26 13:56:05.111 7491-7550/org.courville.nova W/.courville.nov: Accessing hidden method Landroid/os/storage/StorageVolume;->getPath()Ljava/lang/String; (greylist-max-q,test-api, reflection, denied)
2020-04-26 13:56:05.111 7491-7550/org.courville.nova W/System.err: java.lang.NoSuchMethodException: android.os.storage.StorageVolume.getPath []
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at java.lang.Class.getMethod(Class.java:2072)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at java.lang.Class.getMethod(Class.java:1693)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at com.archos.filecorelibrary.ExtStorageManager.updateAllVolumes(ExtStorageManager.java:127)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at com.archos.filecorelibrary.ExtStorageManager.getExtStorageManager(ExtStorageManager.java:51)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at com.archos.mediaprovider.video.Blacklist.isBlacklisted(Blacklist.java:84)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at com.archos.mediaprovider.video.VideoStoreImportImpl$Job.<init>(VideoStoreImportImpl.java:246)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at com.archos.mediaprovider.video.VideoStoreImportImpl.handleScanCursor(VideoStoreImportImpl.java:171)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at com.archos.mediaprovider.video.VideoStoreImportImpl.doScan(VideoStoreImportImpl.java:331)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at com.archos.mediaprovider.video.VideoStoreImportImpl.doFullImport(VideoStoreImportImpl.java:105)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at com.archos.mediaprovider.video.VideoStoreImportService.doImport(VideoStoreImportService.java:417)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at com.archos.mediaprovider.video.VideoStoreImportService.handleMessage(VideoStoreImportService.java:364)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at android.os.Handler.dispatchMessage(Handler.java:102)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at android.os.Looper.loop(Looper.java:216)
2020-04-26 13:56:05.112 7491-7550/org.courville.nova W/System.err:     at android.os.HandlerThread.run(HandlerThread.java:67)

The text was updated successfully, but these errors were encountered:

courville · 2020-05-09T12:05:05Z

androidR branch created for Video/MediaLib/FileCoreLibrary

See nova-video-player/aos-AVP#336

courville · 2021-04-08T12:36:15Z

getPath is greylisted reflection denied on API30:
See nova-video-player/aos-FileCoreLibrary@e81467b
See nova-video-player/aos-FileCoreLibrary@04839ca

courville · 2021-04-08T19:11:06Z

Pending google play to be ready to manage properly android.permission.MANAGE_EXTERNAL_STORAGE and request exemption.

See nova-video-player/aos-Video@fa42e91

courville · 2021-04-09T07:25:51Z

Check https://support.google.com/googleplay/android-developer/answer/9956427 to request exemption.

courville · 2021-07-13T11:33:01Z

Exemption requested and denied.
For the record here are the elements sent.
App functionality requiring the permission (limit 500 chars):

Nova is a video player/manager that catalogs (though MediaStore.Files import)  and presents all videos as a structured medialibrary (Movie/TVshow/Anime) after media information scraping (themoviedb/thetvdb cloud services).
It needs to:
- access all videos already present on the device wherever located (internal, SDcard, USB)
- act as a file manager (import/delete/download) for all media files even those not recognized as such by Android: e.g. NFO metadata, SUB/SSA subtitles, XML resume points

Why can't your app follow good confidentiality practices (i.e. Storage Access Framework or Media Store API or Scoped Storage) (limit 500 chars):

Without loss of functionality, scoped storage cannot be used since Nova needs to index all preexisting videos & related files at any location (internal/SDcard/USB)
MediaStoreAPI does not allow managing (read/download/delete) all media files processed today by Nova: e.g. NFO metadata, SUB/SSA subtitle, XML resume points
Storage Access Framework is not compatible with Nova full device media persistent library design capturing overall device content
Nova is open-source and can be audited on github

Less than 30s video explaining the claim: https://home.courville.org/nova_video_player-faq/nova-MANAGE_EXTERNAL_STORAGE-permission.mp4

Generic reject email received from Google without any specific information.

Overall this is a very frustrating experience.

courville · 2021-07-13T11:34:25Z

Note that Google failed to implement on Android TV (at least API30 emulator) the required dialog to manage properly android.permission.MANAGE_EXTERNAL_STORAGE:

2021-07-10 15:25:25.798 4151-4151/org.courville.nova E/AndroidRuntime: FATAL EXCEPTION: main
    Process: org.courville.nova, PID: 4151
    android.content.ActivityNotFoundException: No Activity found to handle Intent { act=android.settings.MANAGE_ALL_FILES_ACCESS_PERMISSION (has extras) }
        at android.app.Instrumentation.checkStartActivityResult(Instrumentation.java:2067)
        at android.app.Instrumentation.execStartActivity(Instrumentation.java:1727)
        at android.app.Activity.startActivityForResult(Activity.java:5320)
        at androidx.activity.ComponentActivity.startActivityForResult(ComponentActivity.java:574)
        at android.app.Activity.startActivityForResult(Activity.java:5278)
        at androidx.activity.ComponentActivity.startActivityForResult(ComponentActivity.java:560)
        at android.app.Activity.startActivity(Activity.java:5664)
        at android.app.Activity.startActivity(Activity.java:5617)
        at com.archos.mediacenter.video.browser.PermissionChecker$4.onClick(PermissionChecker.java:161)
        at androidx.appcompat.app.AlertController$ButtonHandler.handleMessage(AlertController.java:167)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loop(Looper.java:223)
        at android.app.ActivityThread.main(ActivityThread.java:7656)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:592)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:947)

This is a mess.

courville · 2021-07-13T18:16:23Z

For SSA/ASS cf. https://android-review.googlesource.com/c/platform/packages/providers/MediaProvider/+/1764553

courville · 2021-07-13T20:48:20Z

Found old issue on issuetracker somehow related https://issuetracker.google.com/issues/136774034
And precious command:

adb shell content query --uri "content://media/external/file" --projection name:_data.

courville · 2021-07-14T13:58:50Z

API30debacle plan of record:

rely on Media API to import storage content (valid for video&srt, no ssa/nfo --> request to google?)
use storage access framework for video delete/subs download? (groupped actions possible?), check if there is no annoying directory picker
possible use scoped storage to replicate external/local storage tree for NFO/cover/(no XML on internal storage) with /sdcard/Android/org.courville.nova prefix (lifecycle question for external drive)

courville · 2021-07-14T14:01:38Z

In terms of deadline 2021/11/1 for nova cf. https://support.google.com/googleplay/android-developer/answer/9859152#zippy=%2Ctarget-api-level-requirements-for-play-console

API level requirement	Starting date
Android 8.0 (API level 26)	August 1, 2018: Required for new appsNovember 1, 2018: Required for app updates
Android 9 (API level 28)	August 1, 2019: Required for new appsNovember 1, 2019: Required for app updates
Android 10 (API level 29)*	August 3, 2020: Required for new appsNovember 2, 2020: Required for app updates
Android 11 (API level 30)*	August 2, 2021: Required for new appsNovember 1, 2021: Required for app updates

moneytoo · 2021-10-10T19:11:17Z

As mentioned here, it is still possible to use the classic File API, though it is limited to media files only (just like the MediaStore API). The greylisted methods could be possibly replaced by MediaStore.getExternalVolumeNames() (?).

courville · 2021-10-23T15:02:36Z

Reading https://stackoverflow.com/questions/56468539/getexternalstoragepublicdirectory-deprecated-in-android-q
leads to the conclusion that nova needs to use MediaStore API.

Code sample to implement create/delete/etc... classes compatible with Android Q/R (FileUtilsQ.java) are available here https://gist.github.com/fiftyonemoon/433b563f652039e32c07d1d629f913fb and mentioned here https://stackoverflow.com/questions/64582269/how-can-i-delete-file-on-android-11-api-30-without-system-confirmation-dialog

This requires however to have proper content:// uri and https://stackoverflow.com/questions/7305504/convert-file-uri-to-content-uri provides some more or less working methods.

courville · 2021-10-23T15:11:55Z

Experimenting a bit:

when srt files are created by nova, nova can delete them without a securityException handling using directly content://media/external_primary/file/## uri
video files can be delete only using content://media/external_primary/video/media/### uri
nova is unable to create NFO (no XML resume point files on local storage) anywhere (SRT ok) --> use scoped storage for this (will eat local storage space with external USB drives) and treat covers the same way (or deprecate external nfo/cover)

Some useful query methods via adb:

for videos adb shell content query --uri "content://media/external_primary/video/media" --projection _data
for files adb shell content query --uri "content://media/external/file" --projection _data

To get the content uri, this asynchronous way works:

MediaScannerConnection.scanFile(mContext, new String[]{ uri.getPath() },
                null, // mimetypes
                new MediaScannerConnection.OnScanCompletedListener() {
                    @Override
                    public void onScanCompleted(String path, Uri uri) {
                        contentUri = uri;
                        log.debug("scanFile: contentUri " + contentUri + " for path " + path);
                    }
                });

Note that the hack in ExternalSDFileWriter make Android believe that the file is an image via insertion in the MediaStore.Images.Media does not work anymore and will need to be removed from the code.

Note that this is a PoC. it works with video files. SRT files created by nova can be deleted. Every other format is ko. Should add jpg file handling the same way. Launcher pending intent should be wired back to UI logic and not global. See nova-video-player/aos-AVP#336

See nova-video-player/aos-AVP#336

courville · 2021-10-24T08:46:08Z

TODO:

check why srt can be created and not nfo reading Video/src/main/java/com/archos/mediacenter/video/utils/SubtitlesDownloaderActivity.java:953 and MediaLib/src/com/archos/mediascraper/NfoWriter.java:326. For this use nova setting "Export all descriptions" to see the FileNotFoundException errors (open failed: EPERM). Perhaps use new FileUtilsQ create method?
verify that video delete works with hdd attached to shield (i.e. using correct MediaStore.VOLUME_EXTERNAL or MediaStore.Video.Media.EXTERNAL_CONTENT_URI in getContentUri from FileCoreLibrary/src/com/archos/filecorelibrary/FileUtilsQ.java. Could be done first doing adb content query on box after indexing.

courville · 2021-10-28T08:13:54Z

NFO files (text/plain mime type only https://cs.android.com/android/platform/superproject/+/master:packages/providers/MediaProvider/src/com/android/providers/media/util/MimeUtils.java?q=MimeUtils.java) can only be created in Documents/Download folders. It does not make sense to prefix them in app private directory getFilesDir() since nobody would access them. See https://developer.android.com/training/data-storage
Alternative is to use getExternalFilesDir(null) /storage/emulated/0/Android/data/org.courville.nova/files visible from users and where nova can write.
Links to refresh crash courses on directories: https://gist.github.com/granoeste/5574148 https://medium.com/microsoft-mobile-engineering/scoped-storage-in-android-10-android-11-28d58d989f3c

courville · 2021-10-30T16:15:38Z

Some TODO (for me):

check nfo/jpg relocate in AutoScrapeService (should be abstracted from nfoParser)
wire back the UI notif of the delete for browser refresh
bug (not a regression): when removing folder it does the associatedFiles on folder name (not a video)

courville · 2021-12-19T11:48:55Z

API31 targeted for v6.0.30 onward.

moneytoo · 2021-12-22T19:35:04Z

VLC 3.4.3 beta 6 available from the public testing channel on Google Play uses the MANAGE_EXTERNAL_STORAGE permission.

courville · 2021-12-22T20:07:44Z

@moneytoo thanks for sharing this news.
It hurts since I have been denied this privilege by Google.
I am glad for vlc to have managed to lobby for it.
At least nova complies with the security requirements of Android...
/me cries into a pillow

courville · 2021-12-23T12:23:58Z

/me having fun looking at https://www.reddit.com/r/androiddev/comments/rmgdis/how_come_vlc_is_allowed_to_use_manage_external/
Thanks @moneytoo

courville · 2021-12-23T13:53:19Z

@moneytoo, I just made the connection with just video player (I am slow sorry).
FYI nova relies on the old Archos Video Player AVOS C based multimedia engine that used to be the software basis for all Archos multimedia PMPs.

courville · 2021-12-23T14:41:38Z

From the VLC folks and for track record https://issuetracker.google.com/issues/143549606

courville · 2022-09-21T21:10:55Z

OK just noticed on Google Play console that I have been awarded the android.permission.MANAGE_EXTERNAL_STORAGE sensitive permission (after it has been denied). It appeared out of the blue somehow.

@moneytoo you should check too (finger crossed).

EDIT: I also checked that I can deploy an internal release with the perm.

moneytoo · 2022-09-21T21:29:30Z

Knowing Google, something like that usually sounds too good to be true. But if it sticks, it's great. Could it be because you left a version of an app with such permission in some release channel untouched and they eventually approved it? At least for me, I only see my declaration for this permission in App content section and nothing more.

And disable it by default since it afterall foogle does not allow it despite the google play console cryptic status message. See nova-video-player/aos-AVP#336

courville · 2022-09-22T17:02:00Z

OK this was indeed too good to be true. After granted deployment it has been cancelled.
FTR this is what was indicated on the google play console:

moneytoo · 2023-02-18T10:16:13Z

BTW, it's not only VLC possessing such permission, but a "few more" other players at least:

MX Player (com.mxtech.videoplayer.ad)
MX Player Pro (com.mxtech.videoplayer.pro)
VLC (org.videolan.vlc)
OPlayer Lite (com.olimsoft.android.oplayer)
OPlayer (com.olimsoft.android.oplayer.pro)
KMPlayer (com.kmplayer)
XPlayer (video.player.videoplayer)
PLAYit (com.playit.videoplayer)
HD Video Player by iJoysoft (free.online.hd.video.player)
Video Player by Leopard V7 (com.mine.videoplayer)

courville · 2023-02-18T10:27:49Z

@moneytoo indeed list is expanding... Not quite sure what is the magic for requesting permission since it has been denied twice.
I hate to resort to this, but I might ping some google folks to understand better the process.
Please let me know if you find more.

moneytoo · 2023-02-18T10:41:40Z

My opinion is that the magic is either support of some not normally indexed file format (.iso, .wmv, .rm, possibly some other subtitle or audio formats) or some advanced file functionality (local p2p file sharing, folder encryption or lockings etc.).

moneytoo · 2023-06-08T09:08:23Z

https://www.reddit.com/r/androiddev/comments/rmgdis/comment/jnddkp8/?context=3

courville · 2023-06-10T11:57:00Z

@moneytoo thanks for sharing, have you requested on play console the right?

courville · 2023-07-18T17:22:14Z

FTR after downloading a couple (not being exhaustive there) of video players on apkpure (I know...), a simple for i in *apk; do aapt d permissions "$i" | grep '(package|MANAGE)'; done provides:

package: com.kmplayer
uses-permission: name='android.permission.MANAGE_EXTERNAL_STORAGE'
package: org.xbmc.kodi
uses-permission: name='android.permission.MANAGE_EXTERNAL_STORAGE'
package: com.mxtech.videoplayer.ad
uses-permission: name='android.permission.MANAGE_EXTERNAL_STORAGE'
package: video.player.videoplayer
uses-permission: name='android.permission.MANAGE_EXTERNAL_STORAGE'
package: com.kmp.video
uses-permission: name='android.permission.MANAGE_EXTERNAL_STORAGE'
package: com.mine.videoplayer
uses-permission: name='android.permission.MANAGE_EXTERNAL_STORAGE'
package: org.videolan.vlc
uses-permission: name='android.permission.MANAGE_EXTERNAL_STORAGE'

courville · 2023-08-26T10:40:05Z

OK after discussing with Google Developper Support, I was not able to get someone that could help on MANAGE_EXTERNAL_STORAGE permission.
However, declaration on Play Console "application content section", the "access to all files" is marked treated and seems enabled. I gave another shot at publishing the nova on internal track and it has not been removed by Google for a week.
I reworked the code to grant the permission to be "universally compatible" with Android ecosystem.
What I learned is:

MANAGE_EXTERNAL_STORAGE is supported for API>=30, needs to be granted by user on API>=31 via android.settings.MANAGE_APP_ALL_FILES_ACCESS_PERMISSION that does not exist on Android TV until API>=34
WRITE_EXTERNAL_STORAGE is maxSdkVersion 29, auto-granted API<=22, requires explicit user permission 23<=API<=32, i.e. not required to be in manifest 30<=API<=32 but programmatically requested
READ_EXTERNAL_STORAGE is maxSdkVersion 32, auto-granted API<=22, requires explicit user permission 23<=API<=32
for API>=33 instead of using WRITE_EXTERNAL_STORAGE and READ_EXTERNAL_STORAGE, more granularity is required and READ_MEDIA_(VIDEO|AUDIO|IMAGE) need to be requested when MANAGE_EXTERNAL_STORAGE is not used
for API>=33, POST_NOTIFICATIONS permission is also required

What is "fun" is this special case for Android TV.

No wonder that boxes supporting USB storage does not want to migrate to Android 12. Staying on Android 11 waiting for Android 14 is the only way to get apps such as file managers working without fiddling with Android framework introducing some "customizations" forking from a vanilla base.

note: android tv 12&14 cannot grant this permission... See nova-video-player/aos-AVP#336

courville · 2023-10-21T10:33:34Z

Couple of references for the record on Android not detecting correctly media files:

courville added the task TODO label Apr 26, 2020

courville self-assigned this Apr 26, 2020

courville added a commit to nova-video-player/aos-FileCoreLibrary that referenced this issue May 9, 2020

Get external storage back on Android N

cf8c503

See nova-video-player/aos-AVP#336

courville added the blocker Bug severity label Jul 13, 2021

moneytoo mentioned this issue Aug 28, 2021

[Feature Request] Show only folders with videos and videos moneytoo/Player#13

Closed

courville added a commit to nova-video-player/aos-FileCoreLibrary that referenced this issue Oct 23, 2021

dump mediaDb into /sdcard/Android/data/org.courville.nova/files

e696278

See nova-video-player/aos-AVP#336

courville closed this as completed Dec 19, 2021

courville added this to the nova v6 milestone Dec 20, 2021

moneytoo mentioned this issue Jan 19, 2022

No access to NAS / USB since Android TV 11 moneytoo/Player#248

Closed

courville mentioned this issue Jan 19, 2022

[Bug]: After Update to "Nvidia Shield Experience 9.0" (Android 11 Update) NOVA no longer recognizes/scans media files #539

Closed

courville mentioned this issue Feb 11, 2022

[FEAT]: delete confirmation change in v6 API31 #559

Open

mo123 mentioned this issue Apr 9, 2022

[Android] SDK 30 and All files access permission xbmc/xbmc#21229

Closed

13 tasks

courville mentioned this issue Jan 19, 2023

Disque dur externe/Carte SD Manage_external_storage #806

Closed

courville mentioned this issue Jun 11, 2023

How to use an already existing nfo file？ #901

Open

courville mentioned this issue Jul 18, 2023

[Bug]: Problem downloading subtitles #830

Closed

courville added a commit to nova-video-player/aos-Video that referenced this issue Aug 26, 2023

enable MANAGE_EXTERNAL_STORAGE

24490e2

note: android tv 12&14 cannot grant this permission... See nova-video-player/aos-AVP#336

courville mentioned this issue Jan 21, 2024

Why is usesCleartextTraffic set? #1042

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Android R migration #336

Android R migration #336

courville commented Apr 26, 2020

courville commented May 9, 2020

courville commented Apr 8, 2021

courville commented Apr 8, 2021

courville commented Apr 9, 2021

courville commented Jul 13, 2021

courville commented Jul 13, 2021

courville commented Jul 13, 2021

courville commented Jul 13, 2021

courville commented Jul 14, 2021 •

edited

Loading

courville commented Jul 14, 2021

moneytoo commented Oct 10, 2021

courville commented Oct 23, 2021

courville commented Oct 23, 2021 •

edited

Loading

courville commented Oct 24, 2021 •

edited

Loading

courville commented Oct 28, 2021

courville commented Oct 30, 2021

courville commented Dec 19, 2021

moneytoo commented Dec 22, 2021

courville commented Dec 22, 2021

courville commented Dec 23, 2021

courville commented Dec 23, 2021

courville commented Dec 23, 2021

courville commented Sep 21, 2022 •

edited

Loading

moneytoo commented Sep 21, 2022

courville commented Sep 22, 2022 •

edited

Loading

moneytoo commented Feb 18, 2023

courville commented Feb 18, 2023

moneytoo commented Feb 18, 2023

moneytoo commented Jun 8, 2023

courville commented Jun 10, 2023

courville commented Jul 18, 2023 •

edited

Loading

courville commented Aug 26, 2023 •

edited

Loading

courville commented Oct 21, 2023

Android R migration #336

Android R migration #336

Comments

courville commented Apr 26, 2020

courville commented May 9, 2020

courville commented Apr 8, 2021

courville commented Apr 8, 2021

courville commented Apr 9, 2021

courville commented Jul 13, 2021

courville commented Jul 13, 2021

courville commented Jul 13, 2021

courville commented Jul 13, 2021

courville commented Jul 14, 2021 • edited Loading

courville commented Jul 14, 2021

moneytoo commented Oct 10, 2021

courville commented Oct 23, 2021

courville commented Oct 23, 2021 • edited Loading

courville commented Oct 24, 2021 • edited Loading

courville commented Oct 28, 2021

courville commented Oct 30, 2021

courville commented Dec 19, 2021

moneytoo commented Dec 22, 2021

courville commented Dec 22, 2021

courville commented Dec 23, 2021

courville commented Dec 23, 2021

courville commented Dec 23, 2021

courville commented Sep 21, 2022 • edited Loading

moneytoo commented Sep 21, 2022

courville commented Sep 22, 2022 • edited Loading

moneytoo commented Feb 18, 2023

courville commented Feb 18, 2023

moneytoo commented Feb 18, 2023

moneytoo commented Jun 8, 2023

courville commented Jun 10, 2023

courville commented Jul 18, 2023 • edited Loading

courville commented Aug 26, 2023 • edited Loading

courville commented Oct 21, 2023

courville commented Jul 14, 2021 •

edited

Loading

courville commented Oct 23, 2021 •

edited

Loading

courville commented Oct 24, 2021 •

edited

Loading

courville commented Sep 21, 2022 •

edited

Loading

courville commented Sep 22, 2022 •

edited

Loading

courville commented Jul 18, 2023 •

edited

Loading

courville commented Aug 26, 2023 •

edited

Loading