mirror of
https://github.com/rojo-rbx/rojo.git
synced 2026-10-23 05:51:39 +00:00
Refine testing philosophy and success criteria in comprehensive_plugin_tests plan
- Updated the testing philosophy to emphasize investigation of failing tests, clarifying that failures may indicate either code bugs or invalid tests. - Revised success criteria to focus on analyzing failing tests rather than skipping them, ensuring a thorough understanding of test outcomes. - Enhanced documentation on failure triage and anti-patterns to avoid, promoting a more robust approach to quality assurance in testing.
This commit is contained in:
parent
5298664bad
commit
bb423312aa
2 changed files with 27 additions and 19 deletions
|
|
@ -73,10 +73,14 @@ isProject: false
|
||||||
|
|
||||||
**Testing Philosophy**:
|
**Testing Philosophy**:
|
||||||
|
|
||||||
- Tests exist to EXPOSE limitations, not just pass
|
- Tests exist to EXPOSE potential issues, not just pass
|
||||||
- Tests that break the system are GOOD - they're the forcing function for quality
|
- A failing test requires INVESTIGATION - it could mean:
|
||||||
- Do NOT back down when tests fail - fix the underlying issue
|
- The code has a bug (fix the code)
|
||||||
- A failing stress test reveals a real bug that will hit users in production
|
- The test is invalid or has wrong assumptions (fix the test)
|
||||||
|
- The test is testing something that's not actually a requirement (remove/adjust the test)
|
||||||
|
- Don't automatically assume failing test = broken code, but also don't dismiss failures
|
||||||
|
- When a test fails, determine the root cause before deciding on the fix
|
||||||
|
- Tests should be grounded in real requirements and real-world scenarios
|
||||||
|
|
||||||
**Code Quality Awareness**:
|
**Code Quality Awareness**:
|
||||||
|
|
||||||
|
|
@ -483,35 +487,39 @@ These are the types of bugs that stress tests should expose:
|
||||||
|
|
||||||
## Success Criteria
|
## Success Criteria
|
||||||
|
|
||||||
**Primary Goal: Expose Limitations**
|
**Primary Goal: Build Confidence Through Stress Testing**
|
||||||
|
|
||||||
1. Tests should initially FAIL if the underlying code has bugs - this is expected and good
|
1. Failing tests require investigation - could be code bug OR invalid test
|
||||||
2. Each failing test should be tracked as a bug to fix, not a test to skip
|
2. Each failing test should be analyzed: is this a real bug, or is the test wrong?
|
||||||
3. Tests that pass immediately may not be stressing the system enough
|
3. Don't skip tests without understanding why they fail
|
||||||
|
4. Tests that pass immediately are fine - they confirm expected behavior
|
||||||
|
|
||||||
**Quality Gates**:
|
**Quality Gates**:
|
||||||
|
|
||||||
1. All tests eventually pass in CI (after bugs are fixed)
|
1. All valid tests pass in CI
|
||||||
2. Tests catch regressions in the critical paths identified
|
2. Tests catch regressions in the critical paths identified
|
||||||
3. Edge cases like duplicate names, ambiguous paths are thoroughly covered
|
3. Edge cases like duplicate names, ambiguous paths are thoroughly covered
|
||||||
4. Both scripts-only and everything modes have dedicated test coverage
|
4. Both scripts-only and everything modes have dedicated test coverage
|
||||||
5. Large-scale scenarios (1000+ instances) run without timeouts
|
5. Large-scale scenarios (1000+ instances) run without timeouts
|
||||||
6. Memory usage is reasonable during stress tests
|
6. Memory usage is reasonable during stress tests
|
||||||
7. No "fix on fix" pattern - if a test keeps failing after patches, redesign the subsystem
|
7. No "fix on fix" pattern - if code keeps needing patches, consider redesign
|
||||||
|
|
||||||
**Bug Tracking**:
|
**Failure Triage**:
|
||||||
|
|
||||||
When a stress test fails, document:
|
When a stress test fails, investigate:
|
||||||
|
|
||||||
- The symptom (what failed)
|
- The symptom (what failed)
|
||||||
- The root cause (after investigation)
|
- Is this a real bug, or is the test invalid?
|
||||||
- Whether it needs a patch or a redesign
|
- If bug: does it need a patch or a redesign?
|
||||||
- Link to the fix
|
- If invalid test: fix the test or remove it
|
||||||
|
- Document the decision and reasoning
|
||||||
|
|
||||||
## Anti-Patterns to Avoid
|
## Anti-Patterns to Avoid
|
||||||
|
|
||||||
- Skipping tests because they're "too hard to fix"
|
- Skipping tests without understanding why they fail
|
||||||
- Adding workarounds instead of fixing root causes
|
- Adding workarounds instead of fixing root causes
|
||||||
- Reducing test scope to make tests pass
|
- Reducing test scope arbitrarily to make tests pass
|
||||||
- Ignoring intermittent failures (they reveal real race conditions)
|
- Ignoring intermittent failures without investigation
|
||||||
|
- Assuming all failing tests mean broken code (tests can be wrong too)
|
||||||
|
- Writing tests that don't reflect real requirements or real-world usage
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -74,7 +74,7 @@ local function VersionIndicator(props)
|
||||||
end
|
end
|
||||||
|
|
||||||
local function Header(props)
|
local function Header(props)
|
||||||
return Theme.with(function(theme)
|
return Theme.with(function(_theme)
|
||||||
return e("Frame", {
|
return e("Frame", {
|
||||||
Size = UDim2.new(1, 0, 0, 32),
|
Size = UDim2.new(1, 0, 0, 32),
|
||||||
LayoutOrder = props.layoutOrder,
|
LayoutOrder = props.layoutOrder,
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue