cc-test-reporter still broken with SimpleCov

[stepchud]

I have a dependabot issue open on my repo to upgrade SimpleCov from 0.17.1 => 0.22.0, which is failing on CircleCI.
The error message I am experiencing is the same issue that was reported in [#413, #418], however none of the suggested solutions are working for me.

I have tried every combination of ways to configure environment in CirceCI, and I have confirmed with puts statements that the CC_TEST_REPORTER_ID is available and correct during the build.

1. it was originally already configured in the Project Settings on the CircleCI build
2. using the :environment key on the rspec job
3. using the :env key on the step when rspec is executed
4. using export CC_TEST_REPORTER_ID=... as the first line of the command array
5. applying it directly to the command CC_TEST_REPORTER_ID=... bundle exec rspec

I added the gem to my bundle and explicitly set the formatter:

require 'simplecov_json_formatter'
SimpleCov.formatter = SimpleCov::Formatter::JSONFormatter

I can also see the formatter for SimpleCov is correct via puts (SimpleCov::Formatter::JSONFormatter). I also tried using the simplecov-json gem to no avail because it is mentioned on some docs.

We have been using a gem called rspec_junit_formatter and I tried removing that in case it was causing issues, just trying to remove variables and simplify what might be wrong.

Here is the step that's failing in CircleCI:

      - run:
          name: Run Tests
          env:
            CC_TEST_REPORTER_ID: a561...0703
          command: |
            mkdir /tmp/test-results
            ./cc-test-reporter before-build

            # Run Rspec tests
            echo "Run Rspec tests"
            CC_TEST_REPORTER_ID=a561...0703 bundle exec rspec

            echo "cc-test-reporter format-coverage"
            CC_TEST_REPORTER_ID=a561...0703 ./cc-test-reporter format-coverage \
              --output coverage/codeclimate.rspec.$CIRCLE_NODE_INDEX.3.2.4.json

Just to be clear I am using the actual key, not the ellipses present in this example. CodeClimate states in their documentation that they don't consider CC_TEST_REPORTER_ID to be a secret that needs to be protected but I didn't feel comfortable posting it here anyways. The error message output after the bundle exec rspec command:

Error: json: cannot unmarshal object into Go struct field resultSet.coverage of type []formatters.NullInt

I was initially able to reproduce the error locally. I downloaded cc-test-reporter binary for arm64 and have it on my PATH. Exporting CC_TEST_REPORTER_ID env var worked to fix the output locally, but not on CircleCI. I have tried all of the command line options to cc-test-reporter both locally and on Circle but they're no help. I've spent 2 days trying to get this to work. My last resort will be to try to debug the CircleCI run via SSH since the puts are not helping.

CC: @fede-moya if you are still offering personal assistance I will try to email you directly.

[wli0503]

We encountered a similar error using simplecov(0.21.2) + simplecov_json_formatter(0.1.4) + cc-test-reporter. The culprit was this line in our case:

# Bad one 
./cc-test-reporter format-coverage -t simplecov -o codeclimate.backend.json coverage/.resultset.json
# Should be
./cc-test-reporter format-coverage -t simplecov -o codeclimate.backend.json coverage/coverage.json

It turned out that what we actually need is the coverage.json file, not the .resultset.json. Maybe it's worth trying to use the other file if you have it and see how it goes since format-coverage takes the first coverage file, which in your case might be .resultset.json.

[stepchud]

@wli0503 thanks for the recommendation, unfortunately it didn't work. this command is still producing the nullType exception

./cc-test-reporter format-coverage \
              --input-type simplecov \
              --output coverage/codeclimate.rspec.$CIRCLE_NODE_INDEX.3.2.4.json \
              coverage/coverage.json

Error: json: cannot unmarshal object into Go struct field resultSet.coverage of type []formatters.NullInt

[presidentbeef]

Strange. I just ran across this same issue with simplecov 0.22.0 and this did work for me:

-            ./tmp/cc-test-reporter format-coverage -t simplecov -o tmp/codeclimate.json coverage/.resultset.json
+            ./tmp/cc-test-reporter format-coverage -t simplecov -o tmp/codeclimate.json coverage/coverage.json

[stepchud]

@presidentbeef @wli0503 can you guys share your test setup and run commands so I can compare them with my setup too?

[stepchud]

after more investigation, I don't even have the coverage/ dir coverage/coverage.json on circleci. when I run the specs locally I do. I'm not sure how to generate that file/output.

[wli0503]

@stepchud Sure. For our setup use simplecov and added it to our spec_helper.rb to start it so it looks like(this is a recreation of our implementation):

# spec_helper.rb
require 'simplecov'
require 'simplecov_json_formatter'
SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
                                                                  SimpleCov::Formatter::HTMLFormatter,
                                                                  SimpleCov::Formatter::JSONFormatter,
                                                                ])

SimpleCov.start :rails

We basically follow the guide here: https://github.com/simplecov-ruby/simplecov?tab=readme-ov-file#getting-started

[stepchud]

after more investigation, I don't even have the coverage/ dir on circleci. when I run the specs locally I do. I'm not sure how to generate that file/output.

I misspoke, we do have coverage/.resultset.json but not coverage/coverage.json file after running specs.

[stepchud]

Just to follow up, this can be closed as resolved now. I believe it came down to three changes on my branch:

1. define CC_TEST_REPORTER_ID in the rspec job environment
2. add coverage/coverage.json filename argument to the cc-test-uploader format-coverage command
3. Add

SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new(
  [SimpleCov::Formatter::HTMLFormatter, SimpleCov::Formatter::JSONFormatter]
)

4. change SimpleCov.start 'rails' to SimpleCov.start :rails 🤯 think this was the missing link to create coverage.json