Format test reports

Results on the Tests tab are parsed from test reports specified in the Report Paths setting in Run and Run Tests steps. Test reports must be in JUnit XML format to appear on the Tests tab, because Harness parses test reports that are in JUnit XML format only.

For optimal rendering in the Harness UI, there is a limit of 8,000 characters per field. If a field in your XML file contains more than 8,000 characters, the output might render incorrectly on the Tests tab.

JUnit XML format resources

Use these resources to learn about JUnit XML formatting.

• Common JUnit XML format and examples from Testmo
• Examples of JUnit XML format mapping from IBM
• Apache Ant JUnit schema

Tools with built-in JUnit XML output

Here are some Harness YAML examples for test tools that produce JUnit XML output by default.

C, C++

You can use the --output-junit command with CTest.

              - step:
                  type: Run
                  identifier: test
                  name: Test
                  spec:
                    shell: Sh
                    command: |-
                      mkdir build
                      cmake -S . -B build
                      ctest --test-dir build --output-junit out.xml
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - /harness/build/out.xml

Java - Gradle

This example runs Gradle with Test Intelligence. You can also run Java tests in Run steps.

              - step:
                  type: RunTests
                  identifier: Run_Tests_with_Intelligence
                  name: Run Tests with Intelligence
                  spec:
                    args: test --tests
                    buildTool: Gradle
                    enableTestSplitting: true
                    language: Java
                    reports:
                      spec:
                        paths:
                          - "/harness/results.xml"
                      type: JUnit
                    runOnlySelectedTests: true

PHP

              - step:
                  type: Run
                  identifier: test
                  name: Test
                  spec:
                    shell: Sh
                    command: |-
                      mkdir -p /harness/phpunit
                      phpunit --log-junit /harness/phpunit/junit.xml tests
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - /harness/phpunit/junit.xml

Python

For Python, use pytest or unittest. You can also use pytest to run unittest.

Run step

This example runs pytest in a Run step.

              - step:
                  type: Run
                  identifier: test
                  name: Test
                  spec:
                    shell: Sh
                    command: |-
                      . venv/bin/activate
                      mkdir /harness/test-results
                      pytest --junitxml=harness/test-results/junit.xml
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - /harness/test-results/junit.xml

info

If you use test splitting with pytest in a Run step, you must set junit_family=xunit1 in your code repo's pytest.ini file or include -o junit_family="xunit1" in the step's command.

Run Tests step (Test Intelligence)

This example runs pytest with Test Intelligence.

              - step:
                  type: RunTests
                  name: Run Python Tests
                  identifier: Run_Python_Tests
                  spec:
                    language: Python
                    buildTool: Pytest
                    args: "--junitxml=out_report.xml"
                    runOnlySelectedTests: true
                    preCommand: |
                      python3 -m venv .venv
                      . .venv/bin/activate

                      python3 -m pip install -r requirements/test.txt
                      python3 -m pip install -e .
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - out_report.xml*

Ruby - Cucumber

              - step:
                  type: Run
                  identifier: test
                  name: Test
                  spec:
                    shell: Sh
                    command: |-
                      bundle check || bundle install
                      mkdir -p /harness/cucumber
                      bundle exec cucumber --format junit --out /harness/cucumber/junit.xml
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - /harness/cucumber/junit.xml

JUnit converters, formatters, and plugins

If your test tool doesn't automatically produce test results in JUnit XML format, there are JUnit converters, formatters, and plugins available for all major languages. Some examples of conversion tools and corresponding Harness YAML are provided below.

C# - .NET Core, NUnit

• NUnit to JUnit
• .NET trx2JUnit

              - step:
                  type: Run
                  identifier: test
                  name: Test
                  spec:
                    shell: Powershell
                    command: |-
                      cd dotnet-agent/TestProject1
                      wget -UseBasicParsing https://dot.net/v1/dotnet-install.ps1 -o dotnet-install.ps1
                      .\dotnet-install.ps1
                      dotnet build

                      wget https://raw.githubusercontent.com/nunit/nunit-transforms/master/nunit3-junit/nunit3-junit.xslt -o nunit3-junit.xslt

                      "C:/Program Files (x86)/NUnit.org/nunit-console/nunit3-console.exe" dotnet-agent/TestProject1/bin/Debug/net48/TestProject1.dll --result="UnitTestResults.xml;transform=nunit3-junit.xslt"
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - UnitTestResults.xml

Clojure

• Kaocha JUnit Plugin
• Clojure.test JUnit Plugin

Go

You can use the go-junit-report tool.

              - step:
                  type: Run
                  identifier: test
                  name: Test
                  spec:
                    shell: Sh
                    command: |-
                      go install github.com/jstemmer/go-junit-report/v2@latest
                      go test -v ./... | tee report.out
                      cat report.out | $HOME/go/bin/go-junit-report -set-exit-code > report.xml
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - report.xml

Java - Maven

This example uses the Maven Surefire Plugin and runs tests with Maven and Test Intelligence.

              - step:
                  type: RunTests
                  identifier: Run_Tests_with_Intelligence
                  name: Run Tests with Intelligence
                  spec:
                    args: test -Dmaven.test.failure.ignore=true -DfailIfNoTests=false
                    buildTool: Maven
                    enableTestSplitting: true
                    language: Java
                    reports:
                      spec:
                        paths:
                          - "target/surefire-reports/*.xml"
                      type: JUnit
                    runOnlySelectedTests: true

JavaScript

ESLint

ESLint JUnit Formatter

  - step:
      type: Run
      name: Run ESLint Tests
      identifier: run_eslint_tests
      spec:
        shell: Sh
        command: |
          mkdir -p /harness/reports
          eslint ./src/ --format junit --output-file /harness/reports/eslint.xml
      reports:
        type: JUnit
        spec:
          paths:
            - "/harness/reports/eslint.xml"

Jest

Jest JUnit Reporter

  - step:
      type: Run
      name: Run Jest Tests
      identifier: run_jest_tests
      spec:
        shell: Sh
        command: |
          yarn add --dev jest-junit
          jest --ci --runInBand --reporters=default --reporters=jest-junit
      envVariables:
         JEST_JUNIT_OUTPUT_DIR: "/harness/reports"
      reports:
        type: JUnit
        spec:
          paths:
            - "/harness/reports/*.xml"

Karma

Karma JUnit Reporter

  - step:
      type: Run
      name: Run Karma Tests
      identifier: run_karma_tests
      spec:
        shell: Sh
        command: |
          npm install
          mkdir /harness/junit
          karma start ./karma.conf.js
      envVariables:
        JUNIT_REPORT_PATH: /harness/junit/
        JUNIT_REPORT_NAME: test-results.xml
      reports:
        type: JUnit
        spec:
          paths:
            - "/harness/junit/test-results.xml"

Mocha

Mocha JUnit Reporter

  - step:
      type: Run
      name: Run Mocha Tests
      identifier: run_mocha_tests
      spec:
        shell: Sh
        command: |
          npm install
          mkdir /harness/junit
          mocha test --reporter mocha-junit-reporter --reporter-options mochaFile=./path_to_your/file.xml
      reports:
        type: JUnit
        spec:
          paths:
            - "/harness/junit/test-results.xml"

Ruby

Minitest

Add the Minitest Junit Formatter to your Gemfile.

  - step:
      type: Run
      name: Run Ruby Tests
      identifier: run_ruby_tests
      spec:
        shell: Sh
        command: |
          bundle check || bundle install
          bundle exec rake test --junit
      reports:
        type: JUnit
        spec:
          paths:
            - "/harness/report.xml"

RSpec

Add the RSpec JUnit formatter to your Gemfile.

  - step:
      type: Run
      name: Run RSpec Tests
      identifier: run_rspec_tests
      spec:
        shell: Sh
        command: |
          bundle check --path=vendor/bundle || bundle install --path=vendor/bundle --jobs=4 --retry=3
          mkdir /harness/rspec
          bundle exec rspec --format progress --format RspecJunitFormatter -o /harness/rspec/rspec.xml
      reports:
        type: JUnit
        spec:
          paths:
            - "/harness/rspec/rspec.xml"

Code coverage reports and test report artifacts

For information about code coverage reports and publishing report URLs to the Artifacts tab, go to Code Coverage.