It's impossible to debug applications on Android O public preview 3 with either gdbserver or ndk-gdb from NDK r15b

[nvidiamfilimonov]

Description

It's impossible to debug applications on Android O public preview 3 with either gdbserver or ndk-gdb
Here's are the results what I get for “native-activity” Android NDK sample built from the source publised at https://github.com/googlesamples/android-ndk.git (I'm using android-mk branch):

c:\android-ndk-r15b\prebuilt\windows-x86_64\bin\ndk-gdb.cmd --attach --verbose
ADB command used: 'adb -s CVH7N15B09000307'
ADB version: Android Debug Bridge version 1.0.36 Revision 0e9850346394-android
Using project directory: c:\ android-ndk-samples\native-activity
Found package name: com.example.native_activity
Attaching to existing application process.
Application ABIs: arm64-v8a, armeabi, armeabi-v7a, mips, mips64, x86, x86_64
Device ABIs: arm64-v8a, armeabi-v7a, armeabi
Selecting ABI: arm64-v8a
Detected pretty printer: none
ERROR: Failed to make application data directory world executable

angler:/ $ run-as com.example.native_activity /data/data/com.example.native_activity/gdbserver --multi :7856
Can't open socket: Permission denied.
Exiting

1|angler:/ $ run-as com.example.native_activity a+x /data/data/com.example.native_activity/
run-as: exec failed for a+x: Permission denied

I've spotted an AVC denial entries in ADB logcat, here's the one of them:
07-07 19:35:04.333 25626-25626/? W/gdbserver: type=1400 audit(0.0:160573): avc: denied { read } for name="stat" dev="proc" ino=4026548024 scontext=u:r:untrusted_app_25:s0:c512,c768 tcontext=u:object_r:proc_stat:s0 tclass=file permissive=0

Environment Details

[ro.bootimage.build.date]: [Wed May 31 20:07:56 UTC 2017]
[ro.bootimage.build.date.utc]: [1496261276]
[ro.bootimage.build.fingerprint]: [google/angler/angler:8.0.0/OPP3.170518.006/4055970:user/release-keys]
[ro.build.characteristics]: [nosdcard]
[ro.build.date]: [Wed May 31 20:07:56 UTC 2017]
[ro.build.date.utc]: [1496261276]
[ro.build.description]: [angler-user 8.0.0 OPP3.170518.006 4055970 release-keys]
[ro.build.display.id]: [OPP3.170518.006]
[ro.build.expect.baseband]: [angler-03.81]
[ro.build.expect.bootloader]: [angler-03.69]
[ro.build.fingerprint]: [google/angler/angler:8.0.0/OPP3.170518.006/4055970:user/release-keys]
[ro.build.flavor]: [angler-user]
[ro.build.host]: [wphr7.hot.corp.google.com]
[ro.build.id]: [OPP3.170518.006]
[ro.build.product]: [angler]
[ro.build.tags]: [release-keys]
[ro.build.type]: [user]
[ro.build.user]: [android-build]
[ro.build.version.all_codenames]: [REL]
[ro.build.version.base_os]: []
[ro.build.version.codename]: [REL]
[ro.build.version.incremental]: [4055970]
[ro.build.version.preview_sdk]: [0]
[ro.build.version.release]: [8.0.0]
[ro.build.version.sdk]: [26]
[ro.build.version.security_patch]: [2017-05-05]
[ro.vendor.build.date]: [Wed May 31 20:07:56 UTC 2017]
[ro.vendor.build.date.utc]: [1496261276]
[ro.vendor.build.fingerprint]: [google/angler/angler:8.0.0/OPP3.170518.006/4055970:user/release-keys]

• NDK Version: Value from Pkg.Revision in ndk-bundle/source.properties.
  Pkg.Desc = Android NDK
  Pkg.Revision = 15.1.4119039
• Build sytem: ndk-build
• Host OS: Windows 10 x64
• Compiler: If GCC, check Clang before filing. GCC is no longer supported.
• ABI:
• STL:
• NDK API level:
• Device API level:

[enh]

1|angler:/ $ run-as com.example.native_activity a+x /data/data/com.example.native_activity/
run-as: exec failed for a+x: Permission denied

there's a missing "chmod" there. where did this command come from?

[nvidiamfilimonov]

there's a missing "chmod" there. where did this command come from?

My bad - somehow the "chmod" command was lost during the copy&paste.
This line was executed by me to manually verify the error reported by ndk-gdb script.

Below is an extract from android-ndk-r15b/prebuilt/windows-x86_64/bin/ndk-gdb.py

# Applications with minSdkVersion >= 24 will have their data directories
# created with rwx------ permissions, preventing adbd from forwarding to
# the gdbserver socket. To be safe, if we're on a device >= 24, always
# chmod the directory.
if get_api_level(args.props) >= 24:
    chmod_cmd = ["/system/bin/chmod", "a+x", data_dir]
    chmod_cmd = gdbrunner.get_run_as_cmd(package_name, chmod_cmd)
    (rc, _, _) = args.device.shell_nocheck(chmod_cmd)
    if rc != 0:
        error("Failed to make application data directory world executable")

[enh]

what does ls -l say? does the chmod work when you try it manually?

[nvidiamfilimonov]

Yes, chmod correctly does its job then invoked manually, but it doesn't help in the end:

angler:/ $ run-as com.example.native_activity chmod a+x /data/data/com.example.native_activity/
angler:/ $ run-as com.example.native_activity ls -laR /data/data/com.example.native_activity
/data/data/com.example.native_activity:
total 612
drwxr-x--x 4 u0_a168 u0_a168 4096 2017-07-07 19:38 .
drwxrwx--x 183 system system 8192 2017-07-07 19:29 ..
drwxrws--x 2 u0_a168 u0_a168_cache 4096 2017-07-07 19:29 cache
drwxrws--x 2 u0_a168 u0_a168_cache 4096 2017-07-07 19:29 code_cache
-rwxrwxrwx 1 u0_a168 u0_a168 596448 2017-07-07 19:38 gdbserver

/data/data/com.example.native_activity/cache:
total 12
drwxrws--x 2 u0_a168 u0_a168_cache 4096 2017-07-07 19:29 .
drwxr-x--x 4 u0_a168 u0_a168 4096 2017-07-07 19:38 ..

/data/data/com.example.native_activity/code_cache:
total 12
drwxrws--x 2 u0_a168 u0_a168_cache 4096 2017-07-07 19:29 .
drwxr-x--x 4 u0_a168 u0_a168 4096 2017-07-07 19:38 ..
angler:/ $ run-as com.example.native_activity /data/data/com.example.native_activity/gdbserver --multi :7856
Can't open socket: Permission denied.
Exiting

[enh]

does that app have INTERNET permission? if not, it won't be able to open the TCP socket.

(and where did you get this command from? ndk-gdb uses unix domain sockets, not TCP.)

[nvidiamfilimonov]

Okay, then let's separate concerns:

1. ndk-gdb doesn't work because of the "ERROR: Failed to make application data directory world executable"
2. gdbserver can't open TCP server on Android O even if a package has INTERNET permission.
   Note that the same command works just fine on Android 7.1 and below

c:\wrk\android\android-ndk-samples\native-media>c:\wrk\android-sdk/build-tools/25.0.2/aapt.exe d permissions c:\wrk\android\android-ndk-samples\native-media\bin\NativeMedia-debug.apk
package: com.example.nativemedia
uses-permission: name='android.permission.INTERNET'
uses-permission: name='android.permission.READ_EXTERNAL_STORAGE'

c:\wrk\android\android-ndk-samples\native-media>adb install -r c:\wrk\android\android-ndk-samples\native-media\bin\NativeMedia-debug.apk
Success

c:\wrk\android\android-ndk-samples\native-media>adb shell
angler:/ $ pm list packages -3 | grep media
package:com.example.nativemedia
angler:/ $ ls /data/local/tmp/gdbserver
/data/local/tmp/gdbserver
angler:/ $ cat /data/local/tmp/gdbserver | run-as com.example.nativemedia sh -c 'cat > /data/data/com.example.nativemedia/gdbserver'
1|angler:/ $ run-as com.example.nativemedia chmod +x /data/data/com.example.nativemedia/gdbserver
angler:/ $ run-as com.example.nativemedia /data/data/com.example.nativemedia/gdbserver --multi :7856
Can't open socket: Permission denied.
Exiting
1|angler:/ $ run-as com.example.nativemedia chmod a+x /data/data/com.example.nativemedia/
angler:/ $ run-as com.example.nativemedia /data/data/com.example.nativemedia/gdbserver --multi :7856
Can't open socket: Permission denied.
Exiting
1|angler:/ $ dumpsys package com.example.nativemedia
Activity Resolver Table:
Non-Data Actions:
android.intent.action.MAIN:
1636d43 com.example.nativemedia/.NativeMedia filter 97bde11
Action: "android.intent.action.MAIN"
Category: "android.intent.category.LAUNCHER"

Key Set Manager:
[com.example.nativemedia]
Signing KeySets: 151

Packages:
Package [com.example.nativemedia] (7721fc0):
userId=10170
pkg=Package{fc434f9 com.example.nativemedia}
codePath=/data/app/com.example.nativemedia-3XviQRy83yavrxmKBDtM5g==
resourcePath=/data/app/com.example.nativemedia-3XviQRy83yavrxmKBDtM5g==
legacyNativeLibraryDir=/data/app/com.example.nativemedia-3XviQRy83yavrxmKBDtM5g==/lib
primaryCpuAbi=armeabi-v7a
secondaryCpuAbi=null
versionCode=0 minSdk=14 targetSdk=14
versionName=null
splits=[base]
apkSigningVersion=1
applicationInfo=ApplicationInfo{449c3e com.example.nativemedia}
flags=[ DEBUGGABLE HAS_CODE ALLOW_CLEAR_USER_DATA ALLOW_BACKUP ]
dataDir=/data/user/0/com.example.nativemedia
supportsScreens=[small, medium, large, xlarge, resizeable, anyDensity]
timeStamp=2017-07-07 21:52:40
firstInstallTime=2017-07-07 21:52:40
lastUpdateTime=2017-07-07 21:52:40
signatures=PackageSignatures{ac59a9f [4c1c83e8]}
installPermissionsFixed=true installStatus=1
pkgFlags=[ DEBUGGABLE HAS_CODE ALLOW_CLEAR_USER_DATA ALLOW_BACKUP ]
requested permissions:
android.permission.INTERNET
android.permission.READ_EXTERNAL_STORAGE
install permissions:
android.permission.INTERNET: granted=true
android.permission.READ_EXTERNAL_STORAGE: granted=true
User 0: ceDataInode=1769722 installed=true hidden=false suspended=false stopped=true notLaunched=true enabled=0 instant=false
gids=[3003]
runtime permissions:

Package Changes:
Sequence number=36
User 0:
seq=0, package=com.microsoft.launcher
seq=3, package=com.android.stk
seq=15, package=com.google.android.gms
seq=32, package=com.microsoft.skydrive
seq=34, package=com.example.native_activity
seq=35, package=com.example.nativemedia

Dexopt state:
[com.example.nativemedia]
Instruction Set: arm
path: /data/app/com.example.nativemedia-3XviQRy83yavrxmKBDtM5g==/base.apk
status: /data/app/com.example.nativemedia-3XviQRy83yavrxmKBDtM5g==/oat/arm/base.odex[status=kOatUpToDate, compilat
ion_filter=quicken]

Compiler stats:
[com.example.nativemedia]
base.apk - 328

Enabled overlay paths:

[sgopale]

This issue still exists with the stable release.

[DanAlbert]

@jmgao: have you had a chance to look at this?

[DelphiWorlds]

Same issue here, using gdbserver from NDK r15c, Android O release version

[snorp]

Does anyone have a workaround here?

[DanAlbert]

@rprichard: maybe something you can look at since @jmgao is unlikely to be able to any time soon?

[rprichard]

I think the SELinux error is happening because gdbserver is trying to read /proc/stat, which isn't allowed as of Android O (https://issuetracker.google.com/issues/37140047). It doesn't look like gdbserver reads /proc/stat directly. gdbserver is linked statically with the NDK's libc.a, and it seems to be getting old (e.g. pre-M) Bionic code, where functions like sysconf sometimes read /proc/stat. Bionic in newer platform versions doesn't use /proc/stat AFAICT. #272 looks related.

I would expect gdbserver's sysconf (_SC_NPROCESSORS_ONLN) call to return 1 instead of the correct value. I'm not sure what the consequence of that would be.

The Can't open socket: Permission denied. error seems unrelated. I'm still looking into that.

[dantipov]

If sysconf (_SC_NPROCESSORS_ONLN) is faked to 1 instead of an actual number of online cores,
'info os processes' (gdb's builtin analog of 'ps') will not work correctly. Worse, current
trunk doesn't check whether sysconf (_SC_NPROCESSORS_ONLN) returns -1; if so, gdbserver will
crash trying to allocate an enormously huge chunk of memory. For the details, see:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=gdb/nat/linux-osdata.c;hb=HEAD

IIUC neither gdb nor gdbserver never reads /proc/stat directly. But both heavily relies on per-process
data in /proc/[PID] directory, see:

https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=gdb/nat/linux-procfs.h;hb=HEAD

Currently gdb/gdbserver uses the following sysconf()'s:

• _SC_PAGE_SIZE. With bionic, it is extracted by getauxval(), which is a parser of auxiliary
  vector passed by the kernel's ELF loader. No /proc access needed.
• _SC_NPROCESSORS_ONLN. With bionic, the result is calculated after parsing sysfs data under
  /sys/devices/system/cpu. Again, no /proc access needed.

[DanAlbert]

_SC_NPROCESSORS_ONLN. With bionic, the result is calculated after parsing sysfs data under
/sys/devices/system/cpu. Again, no /proc access needed.

Not on a modern device, no, but gdbserver is built using an outdated libc.a from Lollipop which did use /proc/stat: https://android.googlesource.com/platform/bionic/+/lollipop-release/libc/bionic/sysconf.cpp#98

[dosvidos]

So, there are 2 separate permission issues on Android Oreo, both of them seems to be caused by SELinux policies.
Assuming that a process is launched with a package user's permissions via run-as utility:

1. The process can't access /proc/stat
2. The process can't open a TCP socket, even if a package has android.permission.INTERNET: granted=true:

Can't open socket: Permission denied.

[rprichard]

1. The process can't access /proc/stat

AFAICT, the consequence of this issue is that info os processes will only display core 0 in the right-hand column. (If the process is running on some other core, nothing is displayed.)

2. The process can't open a TCP socket, even if a package has android.permission.INTERNET: granted=true:

There's a problem with the /system/bin/run-as tool in Android O. I've filed an internal bug report for it. The run-as tool in O is clearing the supplementary gid list. Previously, adb shell run-as <id> groups would list the inet group, but now it doesn't:

$ adb shell groups
uid=2000(shell) gid=2000(shell) groups=2000(shell)input log adb sdcard_rw sdcard_r net_bt_admin net_bt inet net_bw_stats readproc
$ adb shell /system/bin/run-as com.example.native_activity groups
uid=10116(u0_a116) gid=10116(u0_a116) groups=10116(u0_a116)

I think this run-as issue also affects the output of info os processes. ndk-gdb in O currently shows only the app user's processes.

The ERROR: Failed to make application data directory world executable error from ndk-gdb is unrelated to the above 2 issues and AFAICT only affects O Preview 3. The problem there is that adb shell run-as <id> pwd isn't printing anything, so ndk-gdb runs something like adb shell run-as chmod a+x, which fails because the path argument is missing. See the beginning of get_app_data_dir:

def get_app_data_dir(args, package_name):
    cmd = ["/system/bin/sh", "-c", "pwd", "2>/dev/null"]
    cmd = get_run_as_cmd(package_name, cmd)
    (rc, stdout, _) = args.device.shell_nocheck(cmd)
    ...

The command isn't printing anything because a bad SELinux policy prevents run-as from accessing the stdin/stdout/stderr Unix domain sockets that adbd passes it. That bug was present in O Preview 3, and was fixed in this commit -- https://android-review.googlesource.com/#/c/platform/system/sepolicy/+/409621/. It looks fixed in O Preview 4. Because it's just a preview, I don't think there's a need for ndk-gdb to work around it, but if there were, passing -t to adb shell might work.

[DelphiWorlds]

Using the -t switch for adb shell didn't solve anything for me. I still have the "Can't open socket: Permission denied" error.

[DanAlbert]

Yes, that only fixes one of the three issues. There's a platform bug here with no workaround.

[rprichard]

Right, the adb shell -t switch is only a workaround for the "ERROR: Failed to make application data directory world executable" error that only happens with O Preview 3.

The "Can't open socket: Permission denied" error happens when gdbserver is run with a TCP socket. Using gdbserver with a Unix domain socket (as is done with ndk-gdb) works. adb forward can forward a TCP socket from a development machine to a domain socket on the device. Allowing gdbserver to listen to a TCP socket requires a fix in the platform (e.g. in the /system/bin/run-as tool).

[DelphiWorlds]

How does one use gdbserver with a Unix domain socket?

[rprichard]

To use gdbserver with a Unix domain socket rather than a TCP socket, you'll need commands that look something like this:

$ adb forward tcp:1234 localfilesystem:/data/data/com.example.native_activity/mysocket
$ adb shell run-as com.example.native_activity ./arm64-gdbserver --multi +/data/data/com.example.native_activity/mysocket
... waits for connection ...
Remote debugging from host 127.0.0.0

In another shell:

$ gdb
...
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
The target is not running (try extended-remote?)

These commands establish the gdb connection, but the debugger immediately exits. The ndk-gdb script does more setup work that I've omitted here.

[DelphiWorlds]

@DanAlbert You mentioned: "There's a platform bug here with no workaround". Is this going to be addressed, and if so, when? I'm wondering why there's apparently no outcry from many developers that are unable to debug their NDK apps on Android 8 because of this issue?

[rprichard]

I'm wondering why there's apparently no outcry from many developers that are unable to debug their NDK apps on Android 8 because of this issue?

ndk-gdb works on everything newer than O Public Preview 3. gdbserver works as long as it's listening to a domain socket. Android Studio uses LLDB/lldb-server, which also use domain sockets for I/O, so they should also work with (at least) the final O and O MR1 releases.

There are a few issues in this bug report:

• O Public Preview 3: The output from adb shell run-as is lost, breaking ndk-gdb. Fix: upgrade to a final O or O MR1 release. Workaround: pass -t to adb shell.

• O Release: adb shell run-as ... gdbserver is unable to listen on a TCP socket. Fix: upgrade to O MR1. Workarounds: use a domain socket (like ndk-gdb), maybe hack something into your app to spawn gdbserver somehow (a domain socket seems easier)

• O and O MR1 Releases: gdbserver can't read /proc/stat on O+, so the info os processes output is a little wrong. An SELinux error is logged.

I think I'll file a separate issue for the last item, so I can close this one.

[rprichard]

I think the inability of adb shell run-as ... gdbserver to listen to a TCP socket was the platform bug @DanAlbert was talking about, and it's fixed in O MR1.

[DelphiWorlds]

@rprichard Ah, thanks.. MR1 is in Developer Preview soon, yes?

[rprichard]

@DelphiWorlds

Ah, thanks.. MR1 is in Developer Preview soon, yes?

It looks like O MR1 is already in Developer Preview: https://developer.android.com/preview/download.html