I've been on both sides of the GitHub issue wall.
As a contributor, I've spent time writing what I thought was a decent bug report, only to get a one-liner back: "Can you share a repro?" Three days later I reply. No response. Issue goes stale.
As someone who's poked around in open source repos, I've seen maintainers close issues with a "not enough info" label that they clearly copy-pasted from a template. You can't blame them — doing that manually for every bad issue is exhausting.
That friction is what I tried to fix.
The Context: A 72-Hour Hackathon About Slop Detection
Hackathon Raptors ran the Slop Scan hackathon with a clear brief: build something that detects low-quality content. Most participants went after social media posts or AI-generated text.
I looked at GitHub issues.
They're a form of content too, and they have a very clear quality signal: does this issue give a maintainer enough information to act on it, or not? That's binary and measurable. I could build a scoring system around it.
The result was Signal-OSS — a GitHub Action that scores every new issue from 0 to 10, always posts a structured comment with a checklist tailored to that score, and edits that comment if the contributor updates their issue.
I placed 5th overall and won the Best Detection Accuracy Award. Here's what I actually built.
The Problem With Current Issue Triage
Most repos handle bad issues in one of two ways:
- The maintainer manually replies asking for more info. This is time-consuming and inconsistent.
- A generic bot closes the issue after N days of inactivity. This is blunt and often alienates contributors who just forgot to follow up.
Neither approach tells the contributor what is actually missing. That's the gap Signal-OSS targets.
The goal isn't to reject bad issues — it's to convert them into good ones by being specific about what's missing.
What It Looks Like in Practice
When a new issue is opened, Signal-OSS runs and always posts a comment — within seconds. What changes based on the score is the content of that comment: a high-scoring issue gets a checklist confirming what's already present, while a low-scoring one gets a checklist of what's specifically missing from that issue.
a newly opened GitHub issue with the bot comment appearing automatically at the bottom
The comment isn't generic. It's generated based on the actual signals detected in that issue — not a boilerplate reply, but a specific breakdown the contributor can act on.
How the Scoring Engine Works
This is the part I find most interesting to explain.
The naive approach would be: send the issue to an LLM, ask it to score it, done. That works, but it couples every issue to an external API call — slow, potentially costly, and requires a key just to get started.
Signal-OSS takes a different approach: heuristics first, LLM only as an optional tiebreaker.
Step 1: 7 Heuristic Signals
Before any LLM is involved, the action checks 7 rule-based signals:
- Does it include a stack trace?
- Are reproduction steps present?
- Is a version number mentioned?
- Is there expected vs actual behavior described?
- Is there a code block with relevant code?
- Is there an environment description?
- Does the issue have a clear, descriptive title (not just "bug" or "error")?
Each signal contributes to the score. Issues that pass most of these signals will likely land above 7. Issues that fail most will land below 3. Both cases get handled without any LLM call.
Step 2: The Ambiguous Zone (4–6)
Issues that land in the 4–6 range are genuinely ambiguous. Maybe the repro steps are there but vague. Maybe there's a stack trace but no environment context. For these, the heuristics can't reliably decide.
This is the only zone where an LLM can get called — and only if you've provided an API key. Signal-OSS is BYOK: bring your own LLM key if you want the smarter scoring, leave it out and the action still works. For the ambiguous zone specifically, omitting the key falls back to the standard heuristic checklist, which is good enough for most repos.
The result: the action is fully functional with zero external API keys. The LLM layer is an opt-in upgrade, not a requirement.
The Comment and Labels
the bot's comment showing the checklist andneeds-infolabel option
the bot's comment showing thesignal-oss-ignorelabel option
The bot also:
- Applies a
needs-infolabel automatically on low scores - Adds a
signal-oss-ignorelabel option — if a maintainer adds this to any issue, the bot stays completely silent for that issue - Edits its own comment if the contributor updates the issue and the score improves — no spamming the thread with multiple bot replies
Setup Is One File
This was a deliberate design constraint I set early: the activation path has to be a single YAML file. No secrets to configure, no dashboard to sign up for.
# .github/workflows/triage.yml
name: Signal-OSS Issue Triage
on:
issues:
types: [opened, reopened]
permissions:
contents: read
issues: write
jobs:
triage:
runs-on: ubuntu-latest
if: github.actor != 'github-actions[bot]'
steps:
- name: Triage issue
uses: Divyesh-5981/signal-oss@v1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
model: 'none'
The GITHUB_TOKEN is automatically provided by GitHub Actions — no setup needed. The action works fully without the LLM key, using the standard heuristic checklist for all issues including the ambiguous 4–6 zone. If you add a key, ambiguous issues get LLM-refined scoring instead. Drop the file into .github/workflows/ and you're done either way.
the summary view in the Actions tab showing the run result and the score output
Benchmark Numbers (and Why Precision Is Lower Than Recall)
I benchmarked Signal-OSS on real issues from three large repositories: microsoft/vscode, facebook/react, and rust-lang/rust.
| Metric | Score |
|---|---|
| Precision | 0.66 |
| Recall | 0.93 |
| F1 | 0.77 |
The precision being lower than recall was intentional. Here's the tradeoff:
High recall means the system rarely misses a bad issue. If something is genuinely low-quality, Signal-OSS will catch it.
Lower precision means it sometimes flags issues that are actually fine. This results in false positives — good issues that get a comment asking for more detail.
For this use case, that's the right tradeoff. Missing a bad issue (false negative) wastes maintainer time. Flagging a good issue (false positive) is slightly annoying for the contributor but recoverable — they can see the score is borderline and move on, or the maintainer can apply signal-oss-ignore.
If I'd optimized for precision, I'd be missing bad issues constantly. That defeats the purpose.
What I'd Do Differently
A few honest notes from the hackathon:
The 4–6 threshold was arbitrary. I chose it based on intuition and tested it during the benchmark, but a configurable threshold would make Signal-OSS more useful across different repo cultures. A high-volume OSS project might want stricter rules than a small internal tool.
The heuristics are English-biased. All 7 signal checks were designed for English-language issues. Issues written in other languages will score lower by default, which isn't fair. This is worth fixing before any serious adoption.
I benchmarked on large repos. The issue quality distribution in vscode or react is probably different from smaller, less-structured projects. The benchmark numbers may not generalize.
Try It
The repo is linked below. If you maintain an open-source project and issue triage is eating your time, this is worth a look.
Two things I'd find genuinely useful from you:
- If you test it: tell me in the comments whether the false positive rate felt acceptable on your repo.
- If you've solved this differently: I'm curious what other approaches people have tried. Probot? Custom webhooks? Just closing everything immediately?
👇 Drop your thoughts below. And if this was useful, the ❤️ helps other maintainers find it.
Built during the Slop Scan hackathon organized by Hackathon Raptors. Repo in the comments.
Note: The code, benchmarks, and project idea in this post are 100% my own work from the Slop Scan hackathon, but I used AI assistance to help structure and write this article.