'element is not defined' when running detox in CI #670

Angry-Potato · 2018-04-12T13:48:32Z

Thank you for this awesome lib, it works fantastically well locally!

Description (Detox version 7.3.3)

I am building a RN app, and testing it using Detox in Jest. I followed the setup guide in the Detox documentation (also checked the linked Medium post to make sure I set things up correctly) for running detox under Jest.

So far I have created two Detox spec files; one for the initial screen that opens on app load, and one for the login screen that is navigated to via a 'Log In' button on the initial screen.

Running locally is no problem, Detox behaves exactly as expected and all tests pass.

Running on CI (CircleCI) is extremely slow (slowness is likely a machine resource issue, not Detox), and fails with:

ReferenceError: element is not defined
          at Object._callee2$ (/Users/distiller/project/e2e/LoginScreen.spec.js:10:28)
          at tryCatch (/Users/distiller/project/node_modules/regenerator-runtime/runtime.js:62:40)
          at Generator.invoke [as _invoke] (/Users/distiller/project/node_modules/regenerator-runtime/runtime.js:296:22)
          at Generator.prototype.(anonymous function) [as next] (/Users/distiller/project/node_modules/regenerator-runtime/runtime.js:114:21)
          at tryCatch (/Users/distiller/project/node_modules/regenerator-runtime/runtime.js:62:40)
          at invoke (/Users/distiller/project/node_modules/regenerator-runtime/runtime.js:152:20)

          at /Users/distiller/project/node_modules/regenerator-runtime/runtime.js:195:11
          at new Promise (<anonymous>)
          at callInvokeWithMethodAndArg (/Users/distiller/project/node_modules/regenerator-runtime/runtime.js:194:16)
          at AsyncIterator.enqueue (/Users/distiller/project/node_modules/regenerator-runtime/runtime.js:217:13)
          at AsyncIterator.prototype.(anonymous function) [as next] (/Users/distiller/project/node_modules/regenerator-runtime/runtime.js:114:21)
          at Object.<anonymous>.runtime.async (/Users/distiller/project/node_modules/regenerator-runtime/runtime.js:241:14)
          at Suite._callee2 (/Users/distiller/project/e2e/LoginScreen.spec.js:10:28)
          at addSpecsToSuite (/Users/distiller/project/node_modules/jest-jasmine2/build/jasmine/Env.js:302:25)
          at Env.describe (/Users/distiller/project/node_modules/jest-jasmine2/build/jasmine/Env.js:271:7)
          at describe (/Users/distiller/project/node_modules/jest-jasmine2/build/jasmine/jasmine_light.js:97:18)
          at Suite.<anonymous> (/Users/distiller/project/e2e/LoginScreen.spec.js:10:3)
          at addSpecsToSuite (/Users/distiller/project/node_modules/jest-jasmine2/build/jasmine/Env.js:302:25)
          at Env.describe (/Users/distiller/project/node_modules/jest-jasmine2/build/jasmine/Env.js:271:7)
          at describe (/Users/distiller/project/node_modules/jest-jasmine2/build/jasmine/jasmine_light.js:97:18)
          at Object.<anonymous> (/Users/distiller/project/e2e/LoginScreen.spec.js:3:1)
          at Runtime._execModule (/Users/distiller/project/node_modules/jest-runtime/build/index.js:448:13)
          at Runtime.requireModule (/Users/distiller/project/node_modules/jest-runtime/build/index.js:265:14)
          at /Users/distiller/project/node_modules/jest-jasmine2/build/index.js:114:13
          at Generator.next (<anonymous>)
          at step (/Users/distiller/project/node_modules/jest-jasmine2/build/index.js:144:191)
          at /Users/distiller/project/node_modules/jest-jasmine2/build/index.js:144:437
          at new Promise (<anonymous>)
          at /Users/distiller/project/node_modules/jest-jasmine2/build/index.js:144:99
          at jasmine2 (/Users/distiller/project/node_modules/jest-jasmine2/build/index.js:122:17)
          at /Users/distiller/project/node_modules/jest-runner/build/run_test.js:73:28
          at Generator.next (<anonymous>)
          at step (/Users/distiller/project/node_modules/jest-runner/build/run_test.js:147:191)
          at /Users/distiller/project/node_modules/jest-runner/build/run_test.js:147:361
          at <anonymous>

Steps to Reproduce

Create a RN app.
Follow Detox setup guide to integrate with Jest.
Automate build using CircleCI.

Detox, Node, Device, Xcode and macOS Versions

Detox: 7.3.3
React Native: 0.55.1
Node: 8.9.1
Device: iPhone 7 (11.2)
Xcode: 9.2.0
macOS: 10.12.6

There is a more comprehensive list of the environment in which Detox is run here https://circle-macos-docs.s3.amazonaws.com/image-manifest/build-298/index.html

Device and verbose Detox logs

detox_circleci_fail_logs.txt

The text was updated successfully, but these errors were encountered:

Angry-Potato · 2018-04-13T08:21:25Z

maybe some more context will help, here is the circleci job config:

  detox_tests:
    working_directory: ~/project/ios
    macos:
      xcode: "9.2.0"
    shell: /bin/bash --login -o pipefail
    environment:
      - LANG: "en_GB.UTF-8"
    steps:
      - checkout:
          path: ~/project
      - run:
          name: Set Ruby Version
          command:  echo "ruby-2.4" > ~/.ruby-version
      - restore_cache:
          key: 1-gems-{{ checksum "Gemfile.lock" }}
      - run: defaults write com.apple.iphonesimulator ConnectHardwareKeyboard -bool no
      - run: xcrun simctl boot "iPhone 7"
      - run: brew update
      - run: brew upgrade node
      - run: gem install bundler
      - run: bundle check || bundle install --path vendor/bundle
      - run: brew tap wix/brew && brew install applesimutils
      - save_cache:
          key: 1-gems-{{ checksum "Gemfile.lock" }}
          paths:
            - vendor/bundle 
      - run: npm i -g detox-cli
      - run: yarn
      - run: yarn run test:e2e:build
      - run: yarn run test:e2e:ci

here are the package scripts:

    "test:e2e:ci": "detox test -c ios.sim.debug -l verbose --cleanup",
    "test:e2e:build": "detox build"

here is the jest and detox config in the package.json:

  "jest": {
    "testMatch": [
      "<rootDir>/Tests/**/*.js",
      "**/?(*.)(spec|test).js?(x)"
    ],
    "testPathIgnorePatterns": [
      "/node_modules/",
      "<rootDir>/App/test-setup.js",
      "<rootDir>/e2e"
    ],
    "moduleNameMapper": {
      "^.+\\.(jpg|jpeg|png|gif|eot|otf|webp|svg|ttf|woff|woff2|mp4|webm|wav|mp3|m4a|aac|oga)$": "identity-obj-proxy"
    },
    "setupFiles": [
      "<rootDir>/App/test-setup"
    ],
    "preset": "react-native",
    "transformIgnorePatterns": [
      "node_modules/(?!(jest-)?react-native|react-clone-referenced-element|glamorous-native)"
    ]
  },
  "detox": {
    "test-runner": "jest",
    "runner-config": "e2e/config.json",
    "configurations": {
      "ios.sim.debug": {
        "binaryPath": "ios/build/Build/Products/Debug-iphonesimulator/AngryPotatoApp.app",
        "build": "xcodebuild -project ios/AngryPotatoApp.xcodeproj -scheme AngryPotatoApp -configuration Debug -sdk iphonesimulator -derivedDataPath ios/build",
        "type": "ios.simulator",
        "name": "iPhone 7"
      }
    }
  },

here is the e2e/config.json:

{
  "setupTestFrameworkScriptFile": "./init.js"
}

and finally the e2e/init.js:

const detox = require('detox');
const config = require('../package.json').detox;

jest.setTimeout(120000);

beforeAll(async () => {
  await detox.init(config);
});

afterAll(async () => {
  await detox.cleanup();
});

Angry-Potato · 2018-04-13T08:25:18Z

there is a case in the circleci discuss forum that is remarkably similar, no resolution or activity there though https://discuss.circleci.com/t/wix-detox-ui-tests-dont-run-on-circleci/20920

kristfal · 2018-04-14T06:56:54Z

Can you provide a non-verbose log with time stamps.

Our experience: We got tests running on AppCenter (also fairly low end build machines), and had to increase our default Jest timeout from 120s to some 520s (could probably reduce it a bit but we didn’t bother). In addition, the first test in any given suite should start with a waitFor of 20sec just in case of slowness.

Angry-Potato · 2018-04-16T10:57:10Z

@kristfal I have struggled to attach timestamps to each detox output, closest I got is turning jest verbose mode on (time taken for each unit), and piping the test cmd to moreutils ts.

angry-app-ts-logs.txt

Hopefully you can see from the logs a situation much like the one you described, only the first test of the first suite is failing. I have put in a waitFor as you suggested in the first test:

    await waitFor(element(by.id('RegistrationScreen')))
      .toExist()
      .withTimeout(20000);
    await expect(element(by.id('RegistrationScreen'))).toExist();

But that is either not helping or not long enough.

kristfal · 2018-04-16T11:12:57Z

@Angry-Potato I'd suggest an even higher timeout for testing, both the waitFor and the jest timout. Ours are:

// Set the default test timeout of 520s for it to pass on AppCenter
jest.setTimeout(520000);

// Used in by initial test in all suites and screen navigations
export const INITIAL_SCREEN_TIMEOUT = 20000;

Try doubling the initial waitFor timeout. If that one still doesn't pass, try a different selector (since the following tests pass)

Angry-Potato · 2018-04-16T12:58:48Z

@kristfal thanks for the suggestions but neither have helped (increased timeout to 60000, changed selector to one that was passing later on in the suite). It seems the first assertion / expectation fails regardless of what it is.

Angry-Potato · 2018-04-16T13:04:37Z

to clarify, I increased the initial test screen timeout to 60000, and I have also increased the jest timeout to 520000 as you suggested.

kristfal · 2018-04-16T13:16:15Z

Ok, that is strange, its probably not timing related then. Since all other selectors do pass, are you certain of the following:

The tests pass locally and you're running the same bundle on the CI
No code push services are interfering/updating your bundle on the CI (we got bitten by this one before)

One longshot fix: We were unable to run the debug build (that is using the js bundle served by metro) on the CI. So we're building a release build with an embedded JS bundle and running the CI tests on that release build instead.

Can you try that?

We're building release / debug based on schemes, so our detox config looks like this:

"ios.sim.release": {
  "binaryPath": "ios/build/Build/Products/Release-iphonesimulator/Embark.app",
  "build": "xcodebuild -workspace ios/Embark.xcworkspace -scheme 'Embark Release' -sdk iphonesimulator -derivedDataPath ios/build",
  "type": "ios.simulator",
  "name": "iPhone 7"
},

Here is how they are called:

"appcenter-build-test-ios": "RN_FLAVOR=E2E detox build -c ios.sim.release",
"appcenter-run-test-ios": "detox test -c ios.sim.release",

Angry-Potato · 2018-04-16T15:40:22Z

@kristfal I am positive the tests pass locally, and we have not integrated any code push services. I have tried with a release build as you suggest but the same issue occurs.

A hacky work-around might be to duplicate the first expectation in each suite, and wrap it in a try-catch. Far from what I would like but unable to see any other options 😢

Angry-Potato · 2018-04-17T07:16:03Z

@kristfal I have reproduced the issue in this repo https://github.com/Angry-Potato/detox-testing

Angry-Potato · 2018-04-17T07:29:07Z

though to repro, you need to run a macos build in circleci 😬

Angry-Potato · 2018-04-17T09:35:15Z

I updated the https://github.com/Angry-Potato/detox-testing repo's readme to consolidate the issue tracking around this

Angry-Potato · 2018-05-07T08:22:47Z

closing this as the original issue was my mistake, and now the title is a little misleading, opened a separate issue #717

Angry-Potato closed this as completed May 7, 2018

wix locked and limited conversation to collaborators Jul 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'element is not defined' when running detox in CI #670

'element is not defined' when running detox in CI #670

Angry-Potato commented Apr 12, 2018

Angry-Potato commented Apr 13, 2018

Angry-Potato commented Apr 13, 2018

kristfal commented Apr 14, 2018

Angry-Potato commented Apr 16, 2018

kristfal commented Apr 16, 2018

Angry-Potato commented Apr 16, 2018

Angry-Potato commented Apr 16, 2018

kristfal commented Apr 16, 2018

Angry-Potato commented Apr 16, 2018

Angry-Potato commented Apr 17, 2018

Angry-Potato commented Apr 17, 2018

Angry-Potato commented Apr 17, 2018

Angry-Potato commented May 7, 2018

'element is not defined' when running detox in CI #670

'element is not defined' when running detox in CI #670

Comments

Angry-Potato commented Apr 12, 2018

Description (Detox version 7.3.3)

Steps to Reproduce

Detox, Node, Device, Xcode and macOS Versions

Device and verbose Detox logs

Angry-Potato commented Apr 13, 2018

Angry-Potato commented Apr 13, 2018

kristfal commented Apr 14, 2018

Angry-Potato commented Apr 16, 2018

kristfal commented Apr 16, 2018

Angry-Potato commented Apr 16, 2018

Angry-Potato commented Apr 16, 2018

kristfal commented Apr 16, 2018

Angry-Potato commented Apr 16, 2018

Angry-Potato commented Apr 17, 2018

Angry-Potato commented Apr 17, 2018

Angry-Potato commented Apr 17, 2018

Angry-Potato commented May 7, 2018