Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Scoped Storage Migration Service, Notification & Deck Picker Implementation #13152

Merged
merged 5 commits into from
Feb 7, 2023

Conversation

david-allison
Copy link
Member

Pull Request template

Purpose / Description

This Pull Request implements the foreground service which performs the Scoped Storage Migration, and hooks it up to the Deck Picker

Review Goals

  • Ensure alpha users will not be placed in a bad position by this. This PR moves us to 'developers can test the migration', I'd rather keep this as too unstable for alpha, despite the amount of testing, given the impact.
  • Question the onStartCommand isStarted logic
  • Determine the TODOs which need to be moved to the Plan

This also contains a couple refactorings which could be split out if requested in the review.

Fixes

Related #5304

Approach

  • Define a Service
  • Define code to run a repeated task
  • Define a Service Binder to abstract out the intricacies from DeckPicker
  • Define an API which determines the state of the migration

How Has This Been Tested?

⚠️ This needs more thorough testing. I'm submitting this PR to allow for this to occur. Once this testing is complete, we can allow 'GitHub alpha' users to use it.

  • Tested on my test collection. All media transferred.

Known Issues

  • Reviewer is not tied in to the migration process yet
  • Media is not blocked

Checklist

  • You have a descriptive commit message with a short title (first line, max 50 chars).
  • You have commented your code, particularly in hard-to-understand areas
  • You have performed a self-review of your own code
  • UI changes: include screenshots of all affected screens (in particular showing any new or changed strings)
  • UI Changes: You have tested your change using the Google Accessibility Scanner

TaskDelegate was used only as a generic 'start/progress/complete + result' interface, which isn't necessary and only increases coupling and complexity
@david-allison david-allison changed the title feat: Scoped Storage Migration Service, Notification & Implementation feat: Scoped Storage Migration Service, Notification & Deck Picker Implementation Jan 25, 2023
@github-actions
Copy link
Contributor

Message to maintainers, this PR contains strings changes.

  1. Before merging this PR, it is best to run the "Sync Translations" GitHub action, then make and merge a PR from the i18n_sync branch to get translations cleaned out.
  2. Then merge this PR, and immediately do another translation PR so the huge change made by this PR's key changes are all by themselves.

Read more about updating strings on the wiki,

@david-allison david-allison added Review High Priority Request for high priority review Scoped storage and removed Strings labels Jan 25, 2023
Copy link
Member

@BrayanDSO BrayanDSO left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About the overall process:

  • My AnkiDroid directory still exists after the migration with the "essential files". Shouldn't they be deleted as well?
  • I can't see the progress bar here, and the progress report string shows 0.00MB for the total. I think that the progress bar is enough, so we may not need the Migrated %s of %s MB string, as there won't be much time to translate it on most languages

AnkiDroid/src/main/java/com/ichi2/utils/Repeater.kt Outdated Show resolved Hide resolved
}

/**
* Show a dialog that explains no sync can occur during migration.
*/
private fun warnNoSyncDuringMigration() {
// TODO: Fetch and display real numbers
// TODO: handle value updates
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest handling it by using a progress dialog (not for this PR, just an idea for the TODO task)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes I was waiting for it, I have that part ready at my end

AnkiDroid/src/main/java/com/ichi2/anki/DeckPicker.kt Outdated Show resolved Hide resolved
@BrayanDSO BrayanDSO added the Needs Author Reply Waiting for a reply from the original author label Jan 25, 2023
@BrayanDSO
Copy link
Member

Oh, and why have you removed the Strings label?

@criticalAY
Copy link
Contributor

@BrayanDSO I am the one working with the progress bar that is to be shown during migration and the UI element, Basically STOR-04 and STOR-05 so will pull the updates and then push the PR after this PR gets merged

@BrayanDSO
Copy link
Member

Forgot to reply about your points:

Ensure alpha users will not be placed in a bad position by this. This PR moves us to 'developers can test the migration', I'd rather keep this as too unstable for alpha, despite the amount of testing, given the impact.

I think it is stable enough to alpha testers, and they can always refure to migrate or keep using an older version

Question the onStartCommand isStarted logic

Seems correct to me.

Determine the TODOs which need to be moved to the Plan

  • Dialog offering to migrate on startup
    • After first install
    • After postpone
  • Better migration complete message
    • My preference is snackbar if on-app + notification out of the app, but toast instead of snackbar is acceptable inside the app

@david-allison david-allison removed the Needs Author Reply Waiting for a reply from the original author label Jan 28, 2023
@david-allison
Copy link
Member Author

Fixed the issue with remaining being 0: getFileSize was the culprit (conflict didn't exist, AND it threw if we tried to get the directory size of an empty directory)

Screenshot 2023-01-31 at 00 12 12

@david-allison
Copy link
Member Author

My AnkiDroid directory still exists after the migration with the "essential files". Shouldn't they be deleted as well?

  • We want to keep the collection as a backup
  • We want to add a text file explaining the migration for advanced users

Copy link
Member

@BrayanDSO BrayanDSO left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Left an implementer's choice comment

AnkiDroid/src/main/res/values/01-core.xml Outdated Show resolved Hide resolved
@BrayanDSO BrayanDSO added the Needs Second Approval Has one approval, one more approval to merge label Jan 31, 2023
A service which migrates the AnkiDroid collection from a legacy directory
to an app-specific directory.

Contains a persistent notification and is designed to be run as a foreground service

Issue 5304
* Tapping the Migration icon on the Deck Picker offers to migrate storage
* Implements binding to MigrationService to allow:
  * Refresh when migration is completed
  * Update of progress, for an in-app dialog
This hasn't worked for directories on any of my devices:

```
Recovering failed rename /storage/emulated/0/AnkiDroid/collection.media to /storage/emulated/0/Android/data/com.ichi2.anki/files/AnkiDroid20/collection.media

Rename recovery failed
        android.system.ErrnoException: rename failed: EXDEV (Cross-device link)
            at libcore.io.Linux.rename(Native Method)
            ...
```

We previously assumed this would fail for a directory first.
So did not set `attemptRename` to false for a file copy.

Now we no longer handle directories, so set this value on a file copy.
Added in 0b869c7 - doesn't seem intentional
@mikehardy
Copy link
Member

Can someone please explain the expected testing flow? I did this:

prep1) git co v2.15.6
prep2) (make sure I am on JDK that is compatible with the old version, for me that is running jdk11, normally I'm testing higher versions...)
prep3) ./gradlew clean installPlayDebug --rerun-tasks (the clean and --rerun-tasks are superstitious when I'm testing, if caching is working correctly in gradle they are unnecessary extra compile/build work)
prep4) sync my ankiweb test account so I have a functioning pre-historic AnkiDroid running with data in /sdcard/AnkiDroid

Now I try to test:

test0) make sure you have the github command line utility installed, because it's useful, it is called gh
test1) gh pr checkout 13152
test2) ./gradlew clean installPlayDebug --rerun-tasks (again, I'm superstitious about clean builds)
test3) I have two AnkiDroid apps installed because one has package name suffix .debug now, okay, click that one
test4) ...nothing? I allowed it access to storage and I have an empty deck list. It doesn't see my old collection

If I sync etc, the collection is already scoped storage so I'm not offered any migration.

🤔

@david-allison
Copy link
Member Author

david-allison commented Feb 6, 2023

TL;DR:

we can manually set the collection location to a legacy location (/storage/emulated/0/AnkiDroid). A collection will be created, and the migration can be tested.


com.ichi2.anki.debug has been introduced to ensure that development-based activities will not uninstall com.ichi2.anki (equivalent to a data wipe).


A new collection is now initialised at /storage/emulated/0/Android/data/com.ichi2.anki.debug/files. This does not require a storage update.

An existing user would have a collection at /storage/emulated/0/AnkiDroid. This requires a storage upgrade, and the UI to do so appears

note Since changes to main affect alpha testers, and we want this functionality to be opt-in while additional testing rakes place, this UI is currently optional (a button to migrate is provided).

We have not yet moved to API 30, so we can manually set the collection location to a legacy location (/storage/emulated/0/AnkiDroid). A collection will be created, and the migration can be tested.

@mikehardy
Copy link
Member

Hmm - what I thought was strange was that I had a collection there in /sdcard/AnkiDroid - why didn't the app (.debug suffix or not...) see it and use it? Even if the migration was optional this seems like the prime testing flow doesn't it? (start with 2.15.6 synced with ankiweb, install this, check it? It was the most "happy path" thing I could think of...)

I'll try with a manual storage location setting as new step test4 and current step test4 will become test5

Thanks!

@mikehardy
Copy link
Member

Okay that worked, but here's another testing note, that is not fixable really: Don't do this on a Play Services emulator or you can't use su to go into inaccessible areas of the filesystem (that is, the migrated scoped storage location...) which is obvious unless you forget 😆

Anyway, it seemed to work fine. A couple UX notes:

1- The migrate button on deck picker stayed there post-migration until deck picker was made to UI refresh (app background + foreground did it for me)
2- When the migration is really really fast (I did it on a small collection first) I get the toast that says I can keep using AnkiDroid but there is no notification about the migration (too fast!) and no indication that it finished (see item 1) so I was a little confused if it was still running or not

But it worked it seems! More testing in a bit on a non-play services emulator and with a large collection, and I'll read through the code etc as well.

@BrayanDSO
Copy link
Member

why didn't the app (.debug suffix or not...) see it and use it?

New installs default to the in-app directory (e.g. storage/0/emulated/com.ichi2.anki.debug/files). If you want to make new installs default to the legacy storage, add legacy_storage=true to your local.properties file

@BrayanDSO
Copy link
Member

BrayanDSO commented Feb 6, 2023

When the migration is really really fast (I did it on a small collection first) I get the toast that says I can keep using AnkiDroid

The snackbar shouldn't be shown if the essential files migration is too fast (800ms). Apparently this PR breaks that. Good catch

and no indication that it finished (see item 1) so I was a little confused if it was still running or not

#13131

@criticalAY
Copy link
Contributor

When the migration is really really fast (I did it on a small collection first) I get the toast that says I can keep using AnkiDroid

The snackbar shouldn't be shown if the essential files migration is too fast (800ms). Apparently this PR breaks that. Good catch

and no indication that it finished (see item 1) so I was a little confused if it was still running or not

#13131

Yes waiting for this to be merged then will implement and push

@mikehardy
Copy link
Member

Great - thanks and sorry if I'm duplicating already known stuff. I'm attempting to help with review + testing on this one but I'm catching up on progress and have obviously missed huge amounts of work + tracking, so I'm sadly duplicating some comments. Hopefully it's still a net positive :-)

Copy link
Member

@mikehardy mikehardy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks good on a first and second scan, and it didn't crash or anything when I ran it locally - I'd just be slowing it down further by making it wait more. Let's go! Also, of course: this is awesome as a chunk of progress, really excited

@mikehardy mikehardy merged commit 179ce6f into ankidroid:main Feb 7, 2023
@github-actions github-actions bot added this to the 2.16 release milestone Feb 7, 2023
@github-actions github-actions bot removed Review High Priority Request for high priority review Needs Second Approval Has one approval, one more approval to merge labels Feb 7, 2023
@mikehardy
Copy link
Member

@criticalAY it seemed like you had some work queued up behind this? I'm going to do a string sync now but hopefully you're unblocked 🚂

@criticalAY
Copy link
Contributor

criticalAY commented Feb 7, 2023

@criticalAY it seemed like you had some work queued up behind this? I'm going to do a string sync now but hopefully you're unblocked 🚂

Yep! Thankyou I'll go ahead as soon as it gets merged
Edit : it's already merged so I'll pull the updates:)

@david-allison david-allison deleted the scoped-storage-2023-2 branch February 7, 2023 22:04
* See: #5304
* @return true: Interrupt startup. `false`: continue as normal
*/
open fun startingStorageMigrationInterruptsStartup(): Boolean {
Copy link
Contributor

@oakkitten oakkitten Feb 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name of this function is super hard to read and does not sufficiently reflect what it does. The documentation is not very helpful. I suggest to put into the documentation a high-level overview of the issue it solves.

You could perhaps slightly improve it by thinking of more simple names, e.g.

enum class ShouldContinueActivityStartup { Yes, No }

fun migrateStorageIfNeeded(): ActivityStartup { ... }

fun handleStartup() { // bad name, but w/e
    if (migrateStorageIfNeeded() == ShouldContinueActivityStartup.No) return
    ...
}

But really, the main problem is that this function, being itself very high-level, does very much high-level actions. Consider removing it altogether:

fun handleStartup() {
    val migrationStatus = ScopedStorageService.migrationStatus(this)

    when (migrationStatus) {
        Status.NEEDS_MIGRATION -> TODO("Propose migration")
        Status.IN_PROGRESS -> startMigrateUserDataService()
        Status.REQUIRES_PERMISSION -> requestStoragePermission()
    }

    val shouldHaltStartup = migrationStatus == Status.REQUIRES_PERMISSION
    if (shouldHaltStartup) return

    ...
}

@@ -771,7 +831,7 @@ open class DeckPicker :
}
R.id.action_scoped_storage_migrate -> {
Timber.i("DeckPicker:: migrate button pressed")
showDialogThatOffersToMigrateStorage()
showDialogThatOffersToMigrateStorage(onPostpone = null)
Copy link
Contributor

@oakkitten oakkitten Feb 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not very important, but I'm not sure if onPostpone should have been a part of this PR

@@ -936,6 +996,18 @@ open class DeckPicker :
mFloatingActionMenu.isFABOpen = savedInstanceState.getBoolean("mIsFABOpen")
}

fun onStorageMigrationCompleted() {
migrationService.unbind(this)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The service should stop itself after finishing the migration. I am assuming that we don't have to unbind from it here, or at all; if we do, it should probably happen not in this method. If this needs to be here, I'd like a comment regarding the necessity of this line.

@@ -936,6 +996,18 @@ open class DeckPicker :
mFloatingActionMenu.isFABOpen = savedInstanceState.getBoolean("mIsFABOpen")
}

fun onStorageMigrationCompleted() {
migrationService.unbind(this)
invalidateOptionsMenu() // reapply the sync icon
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“reapply the sync icon” is hard to understand


override fun onStart() {
super.onStart()
if (userMigrationIsInProgress(this)) {
Copy link
Contributor

@oakkitten oakkitten Feb 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel that we should use either this, or ScopedStorageService.migrationStatus(), throughout.

*
* @param T The service to encapsulate. Note: service's [Service.onBind] must return a [SimpleBinder]
*/
abstract class ServiceConnection<T : Service> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I'm not entirely sure about this.

When I had to use a 1 activity + 1 service combination, I finally settled on a simplified approach of putting the service into a static variable (Java times). No binders, no boilerplate. Coupled with an event bus (actually EventBus) approach (now obsolete but yeah), it was, I think, an approach that should be still valid today with some reasonable modifications.

To be fair, on of my gripes was the async nature of onServiceConnected IIRC. I wanted that to be sync. So I chose other options that allowed me to do more seamless animations and stuff. This might be irrelevant for AnkiDroid and my memories might be obsolete, so this is no more than food for thought

The bottom line here is, I think this entire class might be an overcomplication. I see that it's generic and is supposed to simplify things, but can't we get away with just not using a connection, or using one that merely forwards to the service object itself?

return file.length()
} else if (!file.exists()) {
return 0L
}
Copy link
Contributor

@oakkitten oakkitten Feb 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be doing the same thing that CoroutineScope.calculateSize(file: File) does. Can we use that?

@@ -84,3 +84,11 @@ object HandlerUtils {
)
}
}

fun runOnUiThread(runnable: () -> Unit) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest consolidating this with stuff like executeFunctionUsingHandler, executeFunctionWithDelay, etc

* @param delayMs the interval to wait after the execution of [runnable] before the next call is made
* @param runnable Called once every [delayMs] until [terminate] is called
*/
class Repeater private constructor(val delayMs: Long, val runnable: () -> Unit) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd suggest replacing this class with a coroutine, or removing the logic entirely in favor of StateFlow and friends

@oakkitten oakkitten mentioned this pull request Feb 19, 2023
5 tasks
@github-actions
Copy link
Contributor

Hi there @david-allison! This is the OpenCollective Notice for PRs merged from 2023-02-01 through 2023-02-28

If you are interested in compensation for this work, the process with details is here:

https://github.com/ankidroid/Anki-Android/wiki/OpenCollective-Payment-Process#how-to-get-paid

We only post one comment per person per month to avoid spamming you, regardless of the number of PRs merged, but this note applies to all PRs merged for this month

Please note that GSoC contributions are okay for this process. Our philosophy is that our users have donated to AnkiDroid for all contributions. The only PRs that will not go through the OpenCollective process are ones directly related to am accepted GSoC project from a selected participant, since those receive a stipend from GSoC itself.

Please understand that our monthly budget is never guaranteed to cover all claims - the cap on payments-per-person may be lower, but we try to make our process as fair and transparent as possible, we just need your understanding.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants