Having problem with github actions

[kanelloc]

Description

Hello guys,
I'm having problem when running detox on CI (github actions).
Detox works perfectly locally but I keep getting errors on the CI after the test command.

I provide below the yaml file with the build/test configuration of detox on the github action.

name: Main pipeline

on:
  push:
    branches: [develop, master, 'feature/detox-setup']
  pull_request:
    branches: [develop, master]

jobs:
  detox-build:
    runs-on: macos-latest
    timeout-minutes: 60

    env:
      DEVELOPER_DIR: /Applications/Xcode_12.4.app

    steps:
      - uses: actions/checkout@v2

      - name: Use Node.js v12
        uses: actions/setup-node@v2
        with:
          node-version: 12.13

      - name: Set up Node
        run: |
          npm install -g react-native-cli
          npm install -g detox-cli
          npm install -g yarn

      - name: Cache node modules
        uses: actions/cache@v1
        id: npmcache
        with:
          path: node_modules
          key: node-modules-${{ hashFiles('**/yarn.lock') }}

      - name: Install node modules
        run: yarn install

      - name: Run eslint
        run: yarn lint

      - uses: actions/cache@v2
        with:
          path: 'ios/Pods'
          key: ${{ runner.os }}-pods-${{ hashFiles('**/Podfile.lock') }}
          restore-keys: |
            ${{ runner.os }}-pods-

      - name: Install Pod Dependencies
        run: |
          gem update cocoapods xcodeproj
          cd ios && pod install && cd ..

      - name: Install Detox Dependencies
        run: |
          brew tap wix/brew
          brew install applesimutils

      - name: Run Detox Build
        run: yarn e2e:ios:build:dev

      - name: Run Detox Test(s)
        run: yarn detox test -c ios.sim.dev --cleanup

Logs

yarn run v1.22.10

detox[11703] INFO:  [test.js] DETOX_CLEANUP=true DETOX_CONFIGURATION="ios.sim.dev" DETOX_REPORT_SPECS=true DETOX_START_TIMESTAMP=1625231623394 DETOX_USE_CUSTOM_LOGGER=true jest --config e2e/config.json --testNamePattern '^((?!:android:).)*$' --maxWorkers 1 e2e
detox[11708] INFO:  Login Screen Fail Tests is assigned to BD0E83AC-6DB2-46EE-AAA7-0ACE13B49E62 {"type":"iPhone 12"}
detox[11708] INFO:  [AppleSimUtils.js] com.*****.**** launched. To watch simulator logs, run:
        /usr/bin/xcrun simctl spawn BD0E83AC-6DB2-46EE-AAA7-0ACE13B49E62 log stream --level debug --style compact --predicate 'process == "****"'
detox[11708] ERROR: [WS_ERROR] The app has crashed, see the details below:

Signal 6 was raised
(
	0   Detox                               0x000000010bef5c45 +[NSThread(DetoxUtils) dtx_demangledCallStackSymbols] + 37
	1   Detox                               0x000000010bef8e50 __DTXHandleCrash + 464
	2   Detox                               0x000000010bef9595 __DTXHandleSignal + 59
	3   libsystem_platform.dylib            0x00007fff6115a5fd _sigtramp + 29
	4   ???                                 0x0000700002fa0de0 0x0 + 123145352252896
	5   libsystem_c.dylib                   0x00007fff200fab94 abort + 120
	6   VSale                               0x000000010b4a727c -[MSPLCrashReporter setCrashCallbacks:].cold.1 + 0
	7   VSale                               0x000000010b47c5e8 -[MSPLCrashReporter generateLiveReportWithThread:] + 0
	8   CoreFoundation                      0x00007fff20421e2c __handleUncaughtException + 716
	9   libobjc.A.dylib                     0x00007fff201780cb _objc_terminate() + 90
	10  VSale                               0x000000010b4709b0 MSCrashesUncaughtCXXTerminateHandler() + 776
	11  Detox                               0x000000010bef9a8f __dtx_terminate() + 157
	12  libc++abi.dylib                     0x00007fff20252c47 std::__terminate(void (*)()) + 8
	13  libc++abi.dylib                     0x00007fff20252be9 std::terminate() + 41
	14  libdispatch.dylib                   0x00007fff2010651c _dispatch_client_callout + 28
	15  libdispatch.dylib                   0x00007fff2010c3f9 _dispatch_lane_serial_drain + 715
	16  libdispatch.dylib                   0x00007fff2010cf74 _dispatch_lane_invoke + 403
	17  libdispatch.dylib                   0x00007fff20117577 _dispatch_workloop_worker_thread + 782
	18  libsystem_pthread.dylib             0x00007fff61163a3d _pthread_wqthread + 290
	19  libsystem_pthread.dylib             0x00007fff61162b77 start_wqthread + 15
)
detox[11708] INFO:  Login Screen Fail Tests: Should render validator email msg
detox[11708] INFO:  Login Screen Fail Tests: Should render validator email msg [FAIL]
detox[11708] INFO:  Login Screen Fail Tests: Should Type email information
detox[11708] INFO:  Login Screen Fail Tests: Should Type email information [FAIL]

detox[11708] INFO:  Login Screen Fail Tests: Should Type wrong password information
detox[11708] INFO:  Login Screen Fail Tests: Should Type wrong password information [FAIL]
detox[11708] INFO:  Login Screen Fail Tests: Should tap Login btn
detox[11708] INFO:  Login Screen Fail Tests: Should tap Login btn [FAIL]
detox[11708] INFO:  Login Screen Fail Tests: Should show Error Toast
detox[11708] INFO:  Login Screen Fail Tests: Should show Error Toast [FAIL]
detox[11708] INFO:  Login Screen Fail Tests: Should dismiss Error toast
detox[11708] INFO:  Login Screen Fail Tests: Should dismiss Error toast [FAIL]
FAIL e2e/tests/02-login-fail.e2e.js (185.604 s)
  Login Screen Fail Tests
    ✕ Should render validator email msg (1 ms)
    ✕ Should Type email information (1 ms)
    ✕ Should Type wrong password information (1 ms)
    ✕ Should tap Login btn (1 ms)
    ✕ Should show Error Toast (1 ms)
    ✕ Should dismiss Error toast

  ● Login Screen Fail Tests › Should render validator email msg

    thrown: "Exceeded timeout of 120000 ms for a hook.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      2 |
      3 | describe('Login Screen Fail Tests', () => {
    > 4 |   beforeAll(async () => {
        |   ^
      5 |     await device.launchApp({ permissions: { notifications: 'YES' } });
      6 |     await navigateToLogin();
      7 |   });

      at tests/02-login-fail.e2e.js:4:3
      at Object.<anonymous> (tests/02-login-fail.e2e.js:3:1)

  ● Login Screen Fail Tests › Should Type email information

    thrown: "Exceeded timeout of 120000 ms for a hook.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      2 |
      3 | describe('Login Screen Fail Tests', () => {
    > 4 |   beforeAll(async () => {
        |   ^
      5 |     await device.launchApp({ permissions: { notifications: 'YES' } });
      6 |     await navigateToLogin();
      7 |   });

      at tests/02-login-fail.e2e.js:4:3
      at Object.<anonymous> (tests/02-login-fail.e2e.js:3:1)

  ● Login Screen Fail Tests › Should Type wrong password information

    thrown: "Exceeded timeout of 120000 ms for a hook.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      2 |
      3 | describe('Login Screen Fail Tests', () => {
    > 4 |   beforeAll(async () => {
        |   ^
      5 |     await device.launchApp({ permissions: { notifications: 'YES' } });
      6 |     await navigateToLogin();
      7 |   });

      at tests/02-login-fail.e2e.js:4:3
      at Object.<anonymous> (tests/02-login-fail.e2e.js:3:1)

  ● Login Screen Fail Tests › Should tap Login btn

    thrown: "Exceeded timeout of 120000 ms for a hook.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      2 |
      3 | describe('Login Screen Fail Tests', () => {
    > 4 |   beforeAll(async () => {
        |   ^
      5 |     await device.launchApp({ permissions: { notifications: 'YES' } });
      6 |     await navigateToLogin();
      7 |   });

      at tests/02-login-fail.e2e.js:4:3
      at Object.<anonymous> (tests/02-login-fail.e2e.js:3:1)

  ● Login Screen Fail Tests › Should show Error Toast

    thrown: "Exceeded timeout of 120000 ms for a hook.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      2 |
      3 | describe('Login Screen Fail Tests', () => {
    > 4 |   beforeAll(async () => {
        |   ^
      5 |     await device.launchApp({ permissions: { notifications: 'YES' } });
      6 |     await navigateToLogin();
      7 |   });

      at tests/02-login-fail.e2e.js:4:3
      at Object.<anonymous> (tests/02-login-fail.e2e.js:3:1)

  ● Login Screen Fail Tests › Should dismiss Error toast

    thrown: "Exceeded timeout of 120000 ms for a hook.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      2 |
      3 | describe('Login Screen Fail Tests', () => {
    > 4 |   beforeAll(async () => {
        |   ^
      5 |     await device.launchApp({ permissions: { notifications: 'YES' } });
      6 |     await navigateToLogin();
      7 |   });

      at tests/02-login-fail.e2e.js:4:3
      at Object.<anonymous> (tests/02-login-fail.e2e.js:3:1)

[rawatnaresh]

@kanelloc Did it work after increasing the testTimeOut value in e2e/config/config.json file?

[kanelloc]

@kanelloc Did it work after increasing the testTimeOut value in e2e/config/config.json file?

No I got the same error

[rawatnaresh]

unfortunately I ran into the similar error. #2889

[d4vidi]

Updating that we cannot ATM account for the integrity of running Detox on Github actions. Would really wish for you guys to be able to provide feedback as to whether this played out well, after all.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
If you believe the issue is still relevant, please test on the latest Detox and report back.

Thank you for your contributions!

For more information on bots in this reporsitory, read this discussion.

[stale]

The issue has been closed for inactivity.

[mikehardy]

Just a note that we use Detox (v17 super stable, and v18 with some issues in DetoxSync that also show locally - #3000 ) - GitHub Actions using the macos runners are fine in the react-native-firebase test suite. You will need to pin Xcode to 12.5.1 via github action xcode-select to make sure you don't use iOS15 simulators that are attached to Xcode 13 (c.f. iOS15 support ticket #2895). Xcode 12.5.1 is currently the default in the macos-11 (same as macos-current) runner but it could switch without you paying attention

[shamilovtim]

@d4vidi Wanted to comment on this that it's not just Github Actions, CircleCI has these same exact issues which do not reproduce locally on bare metal. Detox seems like it has some sort of incompatibility and flakiness with virtualization in general. This is critical since everyone uses virtualization on CI, and CI providers aren't giving us the option to use bare metal.

[ShivamJoker]

@mikehardy what's wrong with Xcode 13 and iOS 15 ?
I am not able to launch the simulator in github action.

[mikehardy]

@ShivamJoker nothing that I know of other than flakiness in DetoxSync from #3000 - the simulator is fine though
Evidence https://github.com/invertase/react-native-firebase/actions/workflows/tests_e2e_ios.yml
And it's not false positive, it really runs and reports success and code coverage etc and you can read the simulator log if you like https://github.com/invertase/react-native-firebase/actions/runs/1469180698

[ShivamJoker]

I was looking on those only but the whole setup of firebase is bit complex for a simple app.

Also the building and testing is taking almost 30 minutes. I saw you have made some caching action as well, not sure if it will help with build and launch time.

I had to increase the timeout of test to 3 minutes which was only 25 sec.

[mikehardy]

None of this is detox-specific.
"A simple app"...that relies on two gigantic native toolchains for cross-platform native compilation, a gigantic toolchain for javascript/typescript and bundling etc, a programmable CI provider that runs virtual machines in the cloud, a simulator running on that thing, and a configurable testing harness that shims its way in there and lets you assert on UI views

Nothing about that is simple nor will it ever be I am afraid, this stuff is kind of hard

At core though: none of the things you are mentioning are detox specific, I do not believe it belongs on this issue tracker

[mikehardy]

Here's a script that goes from zero to detox running, if that's interesting - it's still incredibly complicated what is happening under the covers, but I expect that after running this script, if I were to launch the emulator on github ci it should work... https://github.com/invertase/react-native-google-ads/blob/main/refresh-example.sh