Compatibility with older devices

[ljbade]

• compile both armeabi and armeabi-v7
• Use Android support libraries
  • for 3.x
  • for 2.x (android support lib v4)
• Test Android versions in emulator
  • 5.0 (API 21)
  • 4.4 (API 19)
  • 4.3 (API 18)
  • 4.2 (API 17)
  • 4.1 (API 16)
  • 4.0.3 (API 15)
  • 4.0 (API 14)
  • 3.x (API 13-11)
  • 2.3.3 (API 10)
  • 2.3.0 (API 9)
• Test for Android versions on real devices
  • 5.0 (API 21)
  • 4.4 (API 19)
  • 4.3 (API 18)
  • 4.2 (API 17)
  • 4.1 (API 16)
  • 4.0.3 (API 15)
  • 4.0 (API 14)
  • 3.x (API 13-11)
  • 2.3.3 (API 10)
  • 2.3.0 (API 9)
• Test different screen sizes and densities
• Test different GPUs
• List any problems found
• Fix any problems
• Test again

[ljbade]

OK I have tested the app in the emulator all the way back to API 15 (4.0.3).

The 4.0 emulator seems broken, seeing:

FB::flushWindowSurfaceColorBuffer: window handle 0x4 not found
FB: closeColorBuffer cb handle 0x3 not found
FB::flushWindowSurfaceColorBuffer: window handle 0x8 not found
FB: closeColorBuffer cb handle 0x7 not found
FB::flushWindowSurfaceColorBuffer: window handle 0xc not found
FB: closeColorBuffer cb handle 0xb not found
...

During Android boot.

But I can't see why it won't work with API 14 anyway, so marking it as done since it compiles with API 14.

This means we have compat with Android devices from Ice Cream Sandwich, Jelly Bean, Kit Kat, and Lollipop

[ljbade]

Found the answer to emulator 4.0 problems here

Turns out Host GPU emulator support did not come till API 15, so we will need a real API 14 device to test on 👎

[ljbade]

OK after removing the fragment stuff (for now - #606) and switching the C++ compile to slower armeabi profile I have got it working on my old 2.3.3 phone.

However I discovered problems using functions that don't exist back in API 10 even if you check the version first. Need to do something like this: http://stackoverflow.com/questions/6495007/verifyerror-deploying-on-api-1-6

I think there might be some functions in the Android support library to help with this too.

[ljbade]

Here is a screenshot of it running on Android 2.3.3:

It is a little bit slow, but usable.

[incanus]

Since this matrix is so vast, does it make sense to possibly outsource this testing to a service? I assume we are talking about actual use testing of the library and not just building?

[ljbade]

What do you mean by 'service'?

At least for the moment we should be able to get by with getting feedback from other team members with older devices.

But when we go live we will probably want a more serious testing scheme.

[incanus]

What do you mean by 'service'?

I mean like someone we pay to test things on many devices, a QA firm or similar. But agree on short-term team testing.

[incanus]

Here's an example of a service; came across my radar today. Bit of an ad for the company, but interesting look at the landscape:

http://blog.testmunk.com/quizup-mastering-android-device-fragmentation-automated-testing/

[ljbade]

Cool, testmunk sounds like an Android version of BrowserStack (hopefully minus the security breaches).

[ljbade]

OK I signed up for a free testmunk account, will see what it takes to get our app running on it.

[ljbade]

@bleege @incanus I have set up a very rough automated travis push to testmunk using my free account for the moment. It is using curl and a bash script.

See this testrun for an example of you get currently. As a free account I only have access to a Nexus 5.

They said they could add older devices (e.g API 9 - 14) if we need this. They already have a reasonable range of devices with more to come.

If we decide to go ahead with testmunk we will want to write some sort of node.js tool that travis can use to upload the APK, wait for test results, then pass/fail as well as printing the result details to the travis console.

[bleege]

Testmunk looks pretty awesome, as well as pretty affordable compared to some of their competitors like Xamarin, AppThwack , and Perfecto. It was cool to see Testmunk in action. Thanks for JFDI @ljbade. There definitely is no substitute for actual device testing when it comes to Android.

My only questions are around how Android GL would best make use of service like this. Going full TDD could get expensive in a hurry, while also being a bit of a cultural shift for Mapbox. Neither are bad, but something to be aware of.

Where this could be super useful (and cost effective) though is in device / API breadth testing at defined checkpoints (i.e. new feature was implemented / after a sprint week / every 2 weeks, etc). Trying to achieve 100% Device testing in Android can be a bit like chasing Moby Dick. This is where trying to choose the most widely used device (Samsung S4 / S5 in the USA) for daily development and then relying on beta testers / testing service comes from. See #604 for more details.

[ljbade]

@bleege, good thoughts. Interesting that Testmunk is cheaper. One of their developers spent his Saturday working with me on figuring out a bug to do with the zipinfo command they were using. Very easy to work with them.

Yeah perhaps we can have a tag or something that triggers Travis to send it off to Testmunk. @incanus was thinking along the same lines for uploading new APKs to hey-apps.
Also what does JFDI and TDD mean?

[bleege]

@ljbade That's awesome that Testmunk has that type of customer focused customer service. Big plus in my book.

Yeah perhaps we can have a tag or something that triggers Travis to send it off to Testmunk.

That sounds like it'd be ideal. Make it something that a human has to trigger. This is something that will require human interpretation of the results so that make sense. It's also a nice way to consciously manage the costs. When it's setup it should be ported over for the Raster SDK too.

TDD stands for Test Driven Development. Essentially it means to write tests first for every piece of functionality before writing the actual code and then repeat that process. For more info check out this Introduction to Test Driven Development (TDD). For a good critique of it check out "TDD is dead. Long live testing.".

JFDI is a play on Nike's slogan and an ethos. More info here.

[ljbade]

Just discovered emulator with API 19 is crashing somewhere in EGL, but also sometimes in a libuvasset (yet to catch it in GDB).

Works in API 21 emulator though.

[bleege]

Capturing offline discussion with @ljbade regarding Testmunk / testing:

• We'll want a minimum of 1 device (for each) 2.2 and 2.3.x (API 8, 9, and 10), but anything more will be a bonus due to very limited marketshare (and falling) of those devices.
• Jelly Bean seems to be the current sweet spot of marketshare so device breadth there will be critical.
• We'll want Nexus devices (preferably the latest / greatest) so that Android devs can see the full power of Mapbox GL on Android and hopefully get stoked.

[ljbade]

Starting to see unusual errors on devices I haven't tested such as #719 #738 and #608

I think we need to start expanding our range of test devices and start writing unit tests.

This will likely need to wait until after merge though.

[ljbade]

I should make a note that now we build arm-v5 I need to get the Java side working on Android 2.x and make it an officially supported platform.

[bleege]

get the Java side working on Android 2.x and make it an officially supported platform.

This Week In Android API Marketshare is reporting that API 10 is the only 2.x level with any significant usage (it's at 7%). Just something to keep in mind so that we don't spend cycles on something that won't be used.

[ljbade]

@bleege last time I looked the work required to get API 10 supported also supported API 9 (and 8? can't remember) for free

[bleege]

@ljbade Cool! Just wanted to make sure that we were careful and didn't accidentally go down a rabbit hole that we didn't need to.

[erf]

Rendering of some tiles fails on my Samsung S3 (4.4.4), the most used Android device.

[ljbade]

@erf can you provide more details/screenshots in a new ticket?

[erf]

added a new issue #1776

[ljbade]

@bleege I think we can close this now after #934