test_runner: add coverage support to run function #53937

atlowChemi · 2024-07-19T08:00:17Z

nodejs-github-bot · 2024-07-19T08:00:21Z

Review requested:

@nodejs/test_runner

cjihrig

Left a couple comments regarding a possible longer term strategy, but this is looking good. Thank you for picking this up!

lib/internal/test_runner/harness.js

lib/internal/test_runner/runner.js

doc/api/test.md

cjihrig

I think these changes are moving in the right direction, but I think this should be split up into multiple PRs.

lib/internal/main/test_runner.js

lib/internal/test_runner/harness.js

lib/internal/test_runner/test.js

test/parallel/test-runner-run.mjs

lib/internal/test_runner/test.js

lib/internal/test_runner/coverage.js

cjihrig · 2024-08-22T19:05:03Z

@atlowChemi just a heads up if you come back to this PR - there will likely be new flags to account for from #54429. A lot of internal refactoring also happened, so the diff here should be a lot simpler.

atlowChemi · 2024-08-25T13:44:54Z

@atlowChemi just a heads up if you come back to this PR - there will likely be new flags to account for from #54429. A lot of internal refactoring also happened, so the diff here should be a lot simpler.

@cjihrig Thanks for the heads up!

I am trying yo get back to this, been pretty busy in work & personal life lately, hoping to get back to this soon 🙏🏽

cjihrig

Other than docs, this looks pretty much done!

test/parallel/test-runner-run-coverage.mjs

cjihrig · 2024-08-30T15:23:25Z

I have some issues with the tests

Do @redyetidev's comments make a difference? If they still fail with those changes, I can take a look.

atlowChemi · 2024-09-05T04:54:58Z

I have some issues with the tests

Do @redyetidev's comments make a difference? If they still fail with those changes, I can take a look.

They do! Thanks @redyetidev 🙂
I actually noticed it just after I wrote here seeking for help, but took some time to get to it..

atlowChemi · 2024-09-20T08:47:18Z

@atlowChemi

ci.nodejs.org/job/node-test-binary-windows-js-suites/30322/RUN_SUBSET=3,nodes=win2019-COMPILED_BY-vs2022/testReport/junit/(root)/parallel/test_runner_run_coverage seems related:

00:02:15.600 not ok 686 parallel/test-runner-run-coverage
00:02:15.600   ---
00:02:15.600   duration_ms: 4266.02000
00:02:15.600   severity: fail
00:02:15.600   exitcode: 1
00:02:15.600   stack: |-
00:02:15.600     â–¶ require('node:test').run coverage settings
00:02:15.601       â–¶ validation
00:02:15.601         âœ” should only allow boolean in options.coverage (63.9542ms)
00:02:15.601         âœ” should only allow string|string[] in options.coverageExcludeGlobs (60.1064ms)
00:02:15.601         âœ” should only allow string|string[] in options.coverageIncludeGlobs (54.7812ms)
00:02:15.601         âœ” should only allow an int within range in options.lineCoverage (51.6314ms)
00:02:15.601         âœ” should only allow an int within range in options.branchCoverage (47.7735ms)
00:02:15.601         âœ” should only allow an int within range in options.functionCoverage (45.1432ms)
00:02:15.601       â–¶ validation (66.871ms)
00:02:15.601       â–¶ run with coverage
00:02:15.601         âœ” should run with coverage (868.6549ms)
00:02:15.601         âœ” should run with coverage and exclude by glob (1350.5099ms)
00:02:15.601         âœ– should run with coverage and include by glob (600.2586ms)
00:02:15.602           AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
00:02:15.602           
00:02:15.602           false !== true
00:02:15.602           
00:02:15.602               at TestsStream.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:131:16)
00:02:15.602               at TestsStream.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:493:15)
00:02:15.602               at TestsStream.emit (node:events:519:28)
00:02:15.602               at [kEmitMessage] (node:internal/test_runner/tests_stream:140:10)
00:02:15.602               at TestsStream.coverage (node:internal/test_runner/tests_stream:127:23)
00:02:15.602               at Test.postRun (node:internal/test_runner/test:1075:18)
00:02:15.602               at postRun (node:internal/test_runner/runner:700:28)
00:02:15.602               at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
00:02:15.602             generatedMessage: true,
00:02:15.602             code: 'ERR_ASSERTION',
00:02:15.603             actual: false,
00:02:15.603             expected: true,
00:02:15.603             operator: 'strictEqual'
00:02:15.603           }
00:02:15.603     
00:02:15.603         âœ– should run while including and excluding globs (412.3843ms)
00:02:15.603           AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
00:02:15.603           
00:02:15.604           false !== true
00:02:15.604           
00:02:15.604               at TestsStream.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:149:16)
00:02:15.604               at TestsStream.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:493:15)
00:02:15.604               at TestsStream.emit (node:events:519:28)
00:02:15.604               at [kEmitMessage] (node:internal/test_runner/tests_stream:140:10)
00:02:15.604               at TestsStream.coverage (node:internal/test_runner/tests_stream:127:23)
00:02:15.605               at Test.postRun (node:internal/test_runner/test:1075:18)
00:02:15.605               at postRun (node:internal/test_runner/runner:700:28)
00:02:15.605               at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
00:02:15.605             generatedMessage: true,
00:02:15.605             code: 'ERR_ASSERTION',
00:02:15.605             actual: false,
00:02:15.605             expected: true,
00:02:15.605             operator: 'strictEqual'
00:02:15.606           }
00:02:15.606     
00:02:15.606         âœ– should run with coverage and fail when below line threshold (1.1888ms)
00:02:15.606           AssertionError [ERR_ASSERTION]: Expected "actual" to be strictly unequal to: 1
00:02:15.606               at TestContext.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:158:14)
00:02:15.606               at Test.runInAsyncScope (node:async_hooks:211:14)
00:02:15.606               at Test.run (node:internal/test_runner/test:930:25)
00:02:15.606               at Suite.processPendingSubtests (node:internal/test_runner/test:629:18)
00:02:15.606               at Test.postRun (node:internal/test_runner/test:1026:19)
00:02:15.606               at Test.run (node:internal/test_runner/test:969:12)
00:02:15.607               at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
00:02:15.607               at async Suite.processPendingSubtests (node:internal/test_runner/test:629:7) {
00:02:15.607             generatedMessage: true,
00:02:15.607             code: 'ERR_ASSERTION',
00:02:15.607             actual: 1,
00:02:15.607             expected: 1,
00:02:15.607             operator: 'notStrictEqual'
00:02:15.608           }
00:02:15.608     
00:02:15.608       â–¶ run with coverage (3266.002ms)
00:02:15.608     â–¶ require('node:test').run coverage settings (3268.4502ms)
00:02:15.608     â„¹ tests 11
00:02:15.608     â„¹ suites 3
00:02:15.608     â„¹ pass 8
00:02:15.608     â„¹ fail 3
00:02:15.608     â„¹ cancelled 0
00:02:15.608     â„¹ skipped 0
00:02:15.609     â„¹ todo 0
00:02:15.609     â„¹ duration_ms 3288.5163
00:02:15.609     
00:02:15.609     âœ– failing tests:
00:02:15.609     
00:02:15.609     test at test\parallel\test-runner-run-coverage.mjs:121:11
00:02:15.609     âœ– should run with coverage and include by glob (600.2586ms)
00:02:15.609       AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
00:02:15.609       
00:02:15.609       false !== true
00:02:15.609       
00:02:15.610           at TestsStream.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:131:16)
00:02:15.610           at TestsStream.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:493:15)
00:02:15.610           at TestsStream.emit (node:events:519:28)
00:02:15.610           at [kEmitMessage] (node:internal/test_runner/tests_stream:140:10)
00:02:15.610           at TestsStream.coverage (node:internal/test_runner/tests_stream:127:23)
00:02:15.610           at Test.postRun (node:internal/test_runner/test:1075:18)
00:02:15.610           at postRun (node:internal/test_runner/runner:700:28)
00:02:15.610           at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
00:02:15.610         generatedMessage: true,
00:02:15.610         code: 'ERR_ASSERTION',
00:02:15.610         actual: false,
00:02:15.610         expected: true,
00:02:15.611         operator: 'strictEqual'
00:02:15.611       }
00:02:15.611     
00:02:15.611     test at test\parallel\test-runner-run-coverage.mjs:137:11
00:02:15.611     âœ– should run while including and excluding globs (412.3843ms)
00:02:15.611       AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
00:02:15.611       
00:02:15.612       false !== true
00:02:15.612       
00:02:15.612           at TestsStream.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:149:16)
00:02:15.612           at TestsStream.<anonymous> (C:\workspace\node-test-binary-windows-js-suites\node\test\common\index.js:493:15)
00:02:15.612           at TestsStream.emit (node:events:519:28)
00:02:15.612           at [kEmitMessage] (node:internal/test_runner/tests_stream:140:10)
00:02:15.612           at TestsStream.coverage (node:internal/test_runner/tests_stream:127:23)
00:02:15.612           at Test.postRun (node:internal/test_runner/test:1075:18)
00:02:15.612           at postRun (node:internal/test_runner/runner:700:28)
00:02:15.612           at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
00:02:15.612         generatedMessage: true,
00:02:15.612         code: 'ERR_ASSERTION',
00:02:15.612         actual: false,
00:02:15.612         expected: true,
00:02:15.612         operator: 'strictEqual'
00:02:15.612       }
00:02:15.612     
00:02:15.613     test at test\parallel\test-runner-run-coverage.mjs:155:11
00:02:15.613     âœ– should run with coverage and fail when below line threshold (1.1888ms)
00:02:15.613       AssertionError [ERR_ASSERTION]: Expected "actual" to be strictly unequal to: 1
00:02:15.613           at TestContext.<anonymous> (file:///C:/workspace/node-test-binary-windows-js-suites/node/test/parallel/test-runner-run-coverage.mjs:158:14)
00:02:15.613           at Test.runInAsyncScope (node:async_hooks:211:14)
00:02:15.613           at Test.run (node:internal/test_runner/test:930:25)
00:02:15.613           at Suite.processPendingSubtests (node:internal/test_runner/test:629:18)
00:02:15.613           at Test.postRun (node:internal/test_runner/test:1026:19)
00:02:15.613           at Test.run (node:internal/test_runner/test:969:12)
00:02:15.613           at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
00:02:15.613           at async Suite.processPendingSubtests (node:internal/test_runner/test:629:7) {
00:02:15.613         generatedMessage: true,
00:02:15.613         code: 'ERR_ASSERTION',
00:02:15.614         actual: 1,
00:02:15.614         expected: 1,
00:02:15.614         operator: 'notStrictEqual'
00:02:15.614       }
00:02:15.614     (node:4360) ExperimentalWarning: glob is an experimental feature and might change at any time
00:02:15.615     (Use `node --trace-warnings ...` to show where the warning was created)

Was using posix separator 🙂
Fixed, can someone approve latest commi? @cjihrig @mcollina @MoLow

nodejs-github-bot · 2024-09-20T10:16:00Z

CI: https://ci.nodejs.org/job/node-test-pull-request/62591/

nodejs-github-bot · 2024-09-20T13:20:47Z

Landed in f79fd03

nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Jul 19, 2024

cjihrig reviewed Jul 19, 2024

View reviewed changes

lib/internal/test_runner/harness.js Outdated Show resolved Hide resolved

lib/internal/test_runner/runner.js Outdated Show resolved Hide resolved

lib/internal/test_runner/runner.js Outdated Show resolved Hide resolved

cjihrig mentioned this pull request Jul 20, 2024

test_runner: refactor coverage to pass in config options #53931

Merged

atlowChemi force-pushed the coverage-support-via-run branch 3 times, most recently from e9d7a22 to 05f4e39 Compare July 22, 2024 05:53

MoLow reviewed Jul 23, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

MoLow reviewed Jul 23, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

MoLow reviewed Jul 23, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

MoLow reviewed Jul 23, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

atlowChemi force-pushed the coverage-support-via-run branch 3 times, most recently from 2907841 to 7ef36f2 Compare July 29, 2024 18:35

cjihrig reviewed Jul 31, 2024

View reviewed changes

avivkeller mentioned this pull request Aug 12, 2024

enable --experimental-test-coverage using run method #54343

Closed

avivkeller reviewed Aug 13, 2024

View reviewed changes

lib/internal/test_runner/coverage.js Outdated Show resolved Hide resolved

avivkeller added wip Issues and PRs that are still a work in progress. coverage Issues and PRs related to native coverage support. labels Aug 22, 2024

atlowChemi force-pushed the coverage-support-via-run branch from 7ef36f2 to 9a9c295 Compare August 29, 2024 07:41

cjihrig reviewed Aug 29, 2024

View reviewed changes

This comment was marked as resolved.

Sign in to view

avivkeller reviewed Aug 29, 2024

View reviewed changes

test/parallel/test-runner-run-coverage.mjs Outdated Show resolved Hide resolved

test/parallel/test-runner-run-coverage.mjs Outdated Show resolved Hide resolved

atlowChemi force-pushed the coverage-support-via-run branch from 4ab6491 to ffe1e56 Compare September 5, 2024 05:03

atlowChemi marked this pull request as ready for review September 5, 2024 05:03

MoLow approved these changes Sep 20, 2024

View reviewed changes

atlowChemi added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Sep 20, 2024

cjihrig added commit-queue Add this label to land a pull request using GitHub Actions. and removed needs-ci PRs that need a full CI run. labels Sep 20, 2024

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Sep 20, 2024

nodejs-github-bot merged commit f79fd03 into nodejs:main Sep 20, 2024

atlowChemi deleted the coverage-support-via-run branch September 20, 2024 13:34

avivkeller mentioned this pull request Sep 20, 2024

test_runner: add 'test:summary' event #54851

Merged

aduh95 mentioned this pull request Oct 9, 2024

V22.10.0 proposal #55343

Merged

RafaelGSS mentioned this pull request Oct 11, 2024

2024-10-16, Version 23.0.0 (Current) #55338

Merged

Uh oh!

test_runner: add coverage support to run function #53937

test_runner: add coverage support to run function #53937

Uh oh!

Conversation

atlowChemi commented Jul 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodejs-github-bot commented Jul 19, 2024

Uh oh!

cjihrig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cjihrig left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cjihrig commented Aug 22, 2024

Uh oh!

atlowChemi commented Aug 25, 2024

Uh oh!

cjihrig left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

cjihrig commented Aug 30, 2024

Uh oh!

atlowChemi commented Sep 5, 2024

Uh oh!

atlowChemi commented Sep 20, 2024

Uh oh!

nodejs-github-bot commented Sep 20, 2024

Uh oh!

nodejs-github-bot commented Sep 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

atlowChemi commented Jul 19, 2024 •

edited

Loading