Skip to content

Update JaCoCo coverage badge#214

Open
github-actions[bot] wants to merge 2 commits into
mainfrom
auto/update-jacoco-badge
Open

Update JaCoCo coverage badge#214
github-actions[bot] wants to merge 2 commits into
mainfrom
auto/update-jacoco-badge

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

Automated JaCoCo coverage badge update from CI.

Copy link
Copy Markdown
Collaborator

@edburns edburns left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot look at the test counts on these three runs corresponding to these three PRs:

  1. 3 weeks ago. PR #127 https://github.com/github/copilot-sdk-java/actions/runs/25004480422
    • 817 tests.
  2. 4 days ago. PR #189 https://github.com/github/copilot-sdk-java/actions/runs/25884708743
    • 888 tests
  3. 49 minutes ago. PR #212 https://github.com/github/copilot-sdk-java/actions/runs/26063266618
    • 998 tests.

These test numbers are monotonically increasing. Yet the coverage percentage is dropping.

Why?

Is the coverage number accurate?

Are we missing some things in the way we track coverage?

Copy link
Copy Markdown
Contributor

Copilot AI commented May 18, 2026

@copilot look at the test counts on these three runs corresponding to these three PRs:

  1. 3 weeks ago. PR Fix jbang-example.java shebang line and dependency version #127 https://github.com/github/copilot-sdk-java/actions/runs/25004480422 ...

I checked the three workflow runs and the current config. The badge number is accurate for what we currently measure: JaCoCo instruction coverage from the Java SDK Tests job, generated from target/site/jacoco-coverage/jacoco.csv after mvn verify.

Why it dropped while tests went up: the denominator grew faster than the tests improved executed coverage. PR #212 in particular pulled in a large reference sync with many new/generated classes under com.github.copilot.sdk.generated and com.github.copilot.sdk.generated.rpc. The current report is 78.3% overall, but generated.rpc is only 64.6%, and several newly added generated types are at 0%.

So I don’t think the badge script is wrong. I do think our tracking has a couple of blind spots:

  • the badge is instruction coverage only (not line/branch/method/class)
  • it mixes generated code with handwritten SDK code
  • it only uses the Java SDK Tests JaCoCo exec; the separate smoke-test jobs are not merged into that number

If we want the badge to better reflect SDK logic quality, the follow-up change should be to exclude generated packages from the badge or publish separate coverage numbers for generated vs handwritten code.

Copilot AI requested a review from edburns May 18, 2026 23:18
@edburns
Copy link
Copy Markdown
Collaborator

edburns commented May 19, 2026

@copilot wrote:

If we want the badge to better reflect SDK logic quality, the follow-up change should be to exclude generated packages from the badge or publish separate coverage numbers for generated vs handwritten code.

Please work into this PR the change to do the latter. I want separate coverage numbers for generated vs. handwritten code, and badges for both.

Co-authored-by: edburns <75821+edburns@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI commented May 19, 2026

@copilot wrote:

If we want the badge to better reflect SDK logic quality, the follow-up change should be to exclude generated packages from the badge or publish separate coverage numbers for gen...

Addressed in e7da399. The PR now generates separate JaCoCo badges for handwritten and generated code, updates the README to show both, and adds the same handwritten/generated instruction-coverage split to the CI test summary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants