Incorrect Jest coverage

[stephane-arista]

When using @swc/jest to transpile ts to commonjs and running jest --coverage certain branches are shown as not covered (console logging in these branches show that tests do run the code path). Using babel to transpile and run the tests shows the correct coverage.

[pspeter3]

I added this to my Jest configuration in package.json which seemed to help.

"jest": {
    "collectCoverage": true,
    "transform": {
      "^.+\\.(t|j)sx?$": [
        "@swc/jest",
        {
          "sourceMaps": true
        }
      ]
    }
  }

[krutoo]

@pspeter3 Thank you for your code snippet, it helps, but now there is another problem:

It looks like not covered code highlights are not displayed correctly.

There is an example of coverage report from project with:

• jest@27.3.1
• @swc/core@1.2.118
• @swc/jest@0.2.11

the yellow highlight here says that the comma is supposedly not covered

[sebald]

@krutoo ran into the same issues and guessed it is caused by the conversion to older JS syntax. We could fix these false-positives by targeting a newer ES version.

  transform: {
    '^.+\\.(t|j)sx?$': [
      '@swc/jest',
      {
        jsc: {
          target: 'es2021',
        },
        sourceMaps: true,
      },
    ],
  },

[krutoo]

@sebald Thank you, looks like it realy works with target: "es2021"

[JoshuaKGoldberg]

Another reproduction here: typescript-eslint/tslint-to-eslint-config#1367

I ended up only needing target: "es2021". Thanks!

[kdy1]

I think swc reusing span for various places may cause this.

Can you try https://sokra.github.io/source-map-visualization/#custom by manually invoking swc?

[bobaaaaa]

@kdy1 does this help? Maybe the Object.keys(_node).forEach produces the mismatch? Jest complains that this line is not covered: export * as node from './BodyNodesBuilder';

Edit: tested with

• @swc/jest 0.2.20
• jest 27.5.1

[kdy1]

@bobaaaaa Can you share some code? I think the test file would be enough, as it does not have good sourcemap.

[kdy1]

Oh... Maybe assumption about monotonic increment of source map position can be the cause.
I'll add it to my tasklist

[bobaaaaa]

@kdy1 You can find it here: https://gist.github.com/bobaaaaa/3649b3a7e6312793a257bf67c500128a
Let me know if something is missing.

(thx for investigating ❤️)

[kdy1]

You need sourceMaps: true or sourceMaps: "inline".
I verified that it's working

[bobaaaaa]

@kdy1 Hm, I tested both sourceMaps in .swcrc and as an option in the jest.config.json. Both did not work for me. Even with target: "es2021".

[kdy1]

Hmm... Source maps are valid, maybe emitted tokens without sourcemap entry can be the cause I guess?
Some sourcemap libraries have bugs related to it.

[bobaaaaa]

@kdy1 I updated the gist with the generated .js + .js.map files: https://gist.github.com/bobaaaaa/3649b3a7e6312793a257bf67c500128a

[rogisolorzano]

For my NestJS project, I also noticed that test coverage dropped quite a bit when switching from ts-jest to @swc/jest.

Digging into it, it seems to be related to how the Typescript metadata for constructors are transpiled.

Take this constructor:

constructor(
  @Inject(AppService)
  private readonly appService: AppService,
) {}

The output of swc for the design:paramtypes metadata for the constructor is:

_ts_metadata("design:paramtypes", [
    typeof _appservice.AppService === "undefined" ? Object : _appservice.AppService
])

Notice the typeof ternary. This gets flagged as an "uncovered branch" by istanbul, the
coverage calculator used by jest.

Compare this to how ts-jest outputs this:

__metadata("design:paramtypes", [app_service_1.AppService])

No ternary, so no impact to code coverage.

I created a workaround that tells istanbul to ignore these lines that incorrectly impact code coverage. It inserts an /* istanbul ignore next */ comment in the necessary places, resulting in transpiled code like this:

_ts_metadata("design:paramtypes", [
    /* istanbul ignore next */typeof _appservice.AppService === "undefined" ? Object : _appservice.AppService
])

I setup this repo with instructions to reproduce: https://github.com/rogisolorzano/nest-swc-coverage

The workaround I'm using is in create-swc-transformer.js

Curious if there are any thoughts on this and potential solutions that don't involve having to insert /* istanbul ignore next */ into the transpiled output by swc?

[ctsstc]

I saw mention of using @swc-node/jest vs @swc/jest what are the differences?

I'm using NestJS as well and this seems to be related to decorators, so removing the legacyDecorator and/or decoratorMetadata is not an approach I can test.

This seems to be very related to an issue ts-jest was/is having here, maybe there's some insight to be taken from there? If they did resolve it I think it was via a similar method as rogisolorzano provided which is adding istanbul ignore comments.

I think I've tried all configuration permutations listed here so far without any luck, as well as installing @swc-node/jest & @swc/helpers and trying it as the jest transformer. One thing to note to keep you sane is that you may need to run npx jest --clearCache between configuration changes it seems before running your next jest coverage run (I noticed removing the transform decorators in my swcrc config broke tests, and still showed they were broken after reverting everything ?!).

Potentially Related Issues on Other Projects

• why is a constructor parameter considered a branch and not covered? istanbuljs/istanbuljs#70
• fix for decorator coverage kulshekhar/ts-jest#488

Ultimately it sounds like source mapping can help some scenarios, changing the target in other scenarios, but specific decorators don't seem to care about either of those in some scenarios like Angular and NestJS commonly it seems.

[klutzer]

I've tested here with vitest+swc in a Nestjs project and the issue still happens. I can't apply a transformer as @rogisolorzano suggested (I guess transformers are just for Jest).

There's also an issue in istanbul related to the same problem: istanbuljs/istanbuljs#70

[ctsstc]

I took some time to checkout the related "fix for decorator coverage" that I linked above. It does seem they took the same approach as @rogisolorzano 👍

I'm not sure if this project has a way to implement a "pre-processor" but maybe that's something @rogisolorzano could open a PR for is this is how folks seem to be getting around this. Otherwise it sounds like there needs to be a major rework on the system to implement so that it doesn't require a ternary (or is this an underlying transpilation implementation issue of these frameworks with how the decorator is written?).

https://github.com/kulshekhar/ts-jest/pull/488/files#diff-0548fd42c45d69a916cd9689c388d2c60b74746b4b7626a008167273557b4e57

PS: Thanks @rogisolorzano for the minimal reproduction and digging deep into this.

@klutzer are you not using Jest? I believe that's the scope of this issue, but I believe the coverage issue is not directly related to jest but the coverage tool [istanbul].

[unicornware]

i'm using vitest instead of jest, but i'm dealing with the same coverage issues outlined above in my nestjs project. i ended up creating a custom plugin that adds /* c8 ignore next */ comments to constructor parameters in my source code. adding comments to the transformed code didn't work for me (this may or may not have something to do with using v8 as my coverage provider, but i'm not sure)

my vitest.config.ts:

/**
 * @file Vitest Configuration
 * @module config/vitest
 * @see https://vitest.dev/config/
 */

import { DECORATOR_REGEX } from '@flex-development/decorator-regex'
import pathe from '@flex-development/pathe'
import { cast, split, type Nullable } from '@flex-development/tutils'
import swc from '@swc/core'
import ci from 'is-ci'
import tsconfigpaths from 'vite-tsconfig-paths'
import GithubActionsReporter from 'vitest-github-actions-reporter'
import {
  defineConfig,
  type UserConfig,
  type UserConfigExport
} from 'vitest/config'
import { BaseSequencer, type WorkspaceSpec } from 'vitest/node'
import tsconfig from './tsconfig.json' assert { type: 'json' }

/**
 * Vitest configuration export.
 *
 * @const {UserConfigExport} config
 */
const config: UserConfigExport = defineConfig((): UserConfig => {
  /**
   * [`lint-staged`][1] check.
   *
   * [1]: https://github.com/okonet/lint-staged
   *
   * @const {boolean} LINT_STAGED
   */
  const LINT_STAGED: boolean = !!Number.parseInt(process.env.LINT_STAGED ?? '0')

  return {
    define: {},
    plugins: [
      {
        enforce: 'pre',
        name: 'decorators',

        /**
         * Transforms source `code` containing decorators.
         *
         * @see https://github.com/swc-project/swc/issues/3854
         *
         * @param {string} code - Source code
         * @param {string} id - Module id of source code
         * @return {Promise<Nullable<{ code: string }>>} Transform result
         */
        async transform(
          code: string,
          id: string
        ): Promise<Nullable<{ code: string }>> {
          // do nothing if source code does not contain decorators
          DECORATOR_REGEX.lastIndex = 0
          if (!DECORATOR_REGEX.test(code)) return null

          /**
           * Regular expression used to match constructor parameters.
           *
           * @see https://regex101.com/r/kTq0JK
           *
           * @const {RegExp} CONSTRUCTOR_PARAMS_REGEX
           */
          const CONSTRUCTOR_PARAMS_REGEX: RegExp =
            /(?<=constructor\(\s*)([^\n)].+?)(?=\n? *?\) ?{)/gs

          // add "/* c8 ignore next */" before constructor parameters
          for (const [match] of code.matchAll(CONSTRUCTOR_PARAMS_REGEX)) {
            code = code.replace(match, (params: string): string => {
              return split(params, '\n').reduce((acc, param) => {
                return acc.replace(
                  param,
                  param.replace(/(\S)/, '/* c8 ignore next */ $1')
                )
              }, params)
            })
          }

          return {
            code: (
              await swc.transform(code, {
                configFile: false,
                filename: id,
                inlineSourcesContent: true,
                jsc: {
                  keepClassNames: true,
                  parser: {
                    decorators: true,
                    dynamicImport: true,
                    syntax: 'typescript',
                    tsx: pathe.extname(id) === '.tsx'
                  },
                  target: cast<swc.JscTarget>(tsconfig.compilerOptions.target),
                  transform: {
                    decoratorMetadata:
                      tsconfig.compilerOptions.emitDecoratorMetadata,
                    legacyDecorator: true,
                    useDefineForClassFields:
                      tsconfig.compilerOptions.useDefineForClassFields
                  }
                },
                sourceMaps: 'inline',
                swcrc: false
              })
            ).code
          }
        }
      },
      tsconfigpaths({ projects: [pathe.resolve('tsconfig.json')] })
    ],
    test: {
      allowOnly: !ci,
      benchmark: {},
      chaiConfig: {
        includeStack: true,
        showDiff: true,
        truncateThreshold: 0
      },
      clearMocks: true,
      coverage: {
        all: !LINT_STAGED,
        clean: true,
        cleanOnRerun: true,
        exclude: [
          '**/__mocks__/**',
          '**/__tests__/**',
          '**/index.ts',
          'src/interfaces/',
          'src/metadata/',
          'src/types/'
        ],
        extension: ['.ts'],
        include: ['src'],
        provider: 'v8',
        reporter: [...(ci ? [] : (['html'] as const)), 'lcovonly', 'text'],
        reportsDirectory: './coverage',
        skipFull: false
      },
      environment: 'node',
      environmentOptions: {},
      globalSetup: [],
      globals: true,
      hookTimeout: 10 * 1000,
      include: [
        `**/__tests__/*.${LINT_STAGED ? '{spec,spec-d}' : 'spec'}.{ts,tsx}`
      ],
      isolate: true,
      mockReset: true,
      outputFile: { json: './__tests__/report.json' },
      passWithNoTests: true,
      reporters: [
        'json',
        'verbose',
        ci ? new GithubActionsReporter() : './__tests__/reporters/notifier.ts'
      ],
      /**
       * Stores snapshots next to `file`'s directory.
       *
       * @param {string} file - Path to test file
       * @param {string} extension - Snapshot extension
       * @return {string} Custom snapshot path
       */
      resolveSnapshotPath(file: string, extension: string): string {
        return pathe.resolve(
          pathe.resolve(pathe.dirname(pathe.dirname(file)), '__snapshots__'),
          pathe.basename(file).replace(/\.spec.tsx?/, '') + extension
        )
      },
      restoreMocks: true,
      root: process.cwd(),
      sequence: {
        sequencer: class Sequencer extends BaseSequencer {
          /**
           * Determines test file execution order.
           *
           * @public
           * @override
           * @async
           *
           * @param {WorkspaceSpec[]} specs - Workspace spec objects
           * @return {Promise<WorkspaceSpec[]>} `files` sorted
           */
          public override async sort(
            specs: WorkspaceSpec[]
          ): Promise<WorkspaceSpec[]> {
            return (await super.sort(specs)).sort(([, file1], [, file2]) => {
              return file1.localeCompare(file2)
            })
          }
        }
      },
      setupFiles: ['./__tests__/setup/index.ts'],
      silent: false,
      singleThread: true,
      slowTestThreshold: 3000,
      snapshotFormat: {
        callToJSON: true,
        min: false,
        printBasicPrototype: false,
        printFunctionName: true
      },
      testTimeout: 10 * 1000,
      typecheck: {
        allowJs: false,
        checker: 'tsc',
        ignoreSourceErrors: false,
        include: ['**/__tests__/*.spec-d.ts'],
        tsconfig: pathe.resolve('tsconfig.typecheck.json')
      },
      unstubEnvs: true,
      unstubGlobals: true
    }
  }
})

export default config

[kdy1]

I think #7900 may improve the situation.

[DavideDaniel]

Our issue was that components that were using emotion were being tested but not captured in the coverage reporting. Using sourceMaps: 'inline' was all it took to get coverage back up to where it was.

[mateus4k]

@kdy1, would @rogisolorzano's suggestion work for a definitive solution in @swc/jest? Personally, this bug prevents me from migrating from ts-jest to @swc/jest in some projects due to a false drop in code coverage, which affects code quality analyzers like SonarQube. I believe that other people may be blocked from migrating for the same reason.

[kdy1]

I'll give this another try tomorrow. Thanks!

[bdentino]

fwiw, i created a simple plugin to fix this in my projects, as @rogisolorzano did not work for me (probably because I am using v8 for coverage like @unicornware). it works for the specific problem @rogisolorzano described, but can't say it's a complete solution. also my first foray into Rust so constructive feedback is welcome.
https://github.com/bdentino/swc-plugin-fix-decorator-metadata-cov

[kdy1]

Okay, about the ternary case, tsc ignores isolatedModules: true in tsconfig.json and that's why it can generate app_service_1.AppService instead of a ternary. Under isolatedModules: true, the behavior of SWC is correct. I'll try adjusting source map to patch coverage.

[kdy1]

It seems like the default coverage provider is not good at reading source maps emitted by SWC.

+"coverageProvider": "v8"

-        Expr::Cond(CondExpr {
-            span: DUMMY_SP,
-            test: check_object_existed(Box::new(member_expr.clone())),
-            cons: Box::new(quote_ident!("Object").into()),
-            alt: Box::new(member_expr),
-        })
+		member_expr

[kdy1]

Oh. I need to generate less source map. I fixed it for "coverageProvider": "v8".

[kdy1]

#9074 should fix this issue for users using "coverageProvider": "v8".

[dalimian]

the above fixed the coverage issue, but unfortunately with "coverageProvider": "v8" my tests crash half way through with heap out of memory error. also "v8" seems lot (maybe 2x) slower than with default coverage provider

I was on "@swc/core": "1.6.13",

[mcmule]

The solution of @rogisolorzano solved it for us and was plug and play.

The nice part is that their implementation doesnt forget to forward config to underlying module (useful if you customize it). Ex:

transform: {
        '^.+\\.ts$': [
            '../configs/create-swc-transformer.js',
            { ...config }, // <-- this is the content of our swcrc file
        ],

Our tools and configs: @swc/jest 0.2.36, @swc/core 1.6.13, jest 29.7.0, targetting commonjs.
Note that our swc config doesn't use "sourceMaps: true" (false coverage errors not related to this discussion).

[felipfr]

The issue still persists. I had to go back to ts-jest as it was having a huge impact on my coverage. Here we use NestJs too.

"@swc/cli": "^0.5.2",
"@swc/core": "^1.9.3",
"@swc/jest": "^0.2.37",