mirror of
https://github.com/rojo-rbx/rojo.git
synced 2026-10-23 05:51:39 +00:00
Enhance playtesting and error handling in atlas skills
- Added a new `atlas-playtest` skill to automate quick playtesting in Roblox Studio, designed to catch runtime errors after code changes. - Updated the `atlas-push` skill to automatically trigger the playtest after a successful push, ensuring immediate feedback on introduced errors. - Enhanced the TODO list with detailed plans for error resolution and playtesting workflows, improving project management and user guidance. These changes aim to streamline the development process by integrating automated testing and error handling into the workflow.
This commit is contained in:
parent
cb3036a5d0
commit
9a25109e1a
4 changed files with 172 additions and 0 deletions
120
.cursor/skills/atlas-playtest/SKILL.md
Normal file
120
.cursor/skills/atlas-playtest/SKILL.md
Normal file
|
|
@ -0,0 +1,120 @@
|
||||||
|
---
|
||||||
|
name: atlas-playtest
|
||||||
|
description: Quick playtest to catch runtime errors after code changes. Starts a Roblox Studio play session via Atlas MCP, waits for scripts to initialize, collects errors, then stops. Use when the user says "atlas playtest", "playtest", "test in studio", "check for errors", "run the game", or after pushing code to Studio.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Atlas Playtest
|
||||||
|
|
||||||
|
Starts a play session in Roblox Studio, waits for scripts to boot, collects runtime errors, then stops. Designed to be fast -- catch obvious breakage, not exhaustive QA.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- `atlas serve` must be running with the Atlas MCP server reachable
|
||||||
|
- Roblox Studio must be open with the Atlas plugin connected
|
||||||
|
- MCP server identifier: `project-0-rojo-atlas`
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
### Step 1: Run Play Session
|
||||||
|
|
||||||
|
```
|
||||||
|
CallMcpTool:
|
||||||
|
server: "project-0-rojo-atlas"
|
||||||
|
toolName: "run_script_in_play_mode"
|
||||||
|
arguments: {
|
||||||
|
"code": "task.wait(7)\nprint(\"boot complete\")",
|
||||||
|
"mode": "start_play",
|
||||||
|
"timeout": 15
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This starts a play session, waits 7 seconds for all scripts to initialize and error if they're going to, then stops automatically.
|
||||||
|
|
||||||
|
**If it returns "Previous call to start play session has not been completed":** call `start_stop_play` with `mode: "stop"` first, wait a moment, then retry.
|
||||||
|
|
||||||
|
### Step 2: Parse Response
|
||||||
|
|
||||||
|
The response contains structured JSON:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"value": "boot complete",
|
||||||
|
"error": null,
|
||||||
|
"logs": [{ "level": "output", "message": "...", "ts": 0.0 }],
|
||||||
|
"errors": [{ "level": "error", "message": "...", "ts": 0.0 }],
|
||||||
|
"duration": 7.1,
|
||||||
|
"isTimeout": false
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- `errors` array: contains all `"error"` and `"warning"` level messages from the session
|
||||||
|
- `logs` array: all messages (including output/info)
|
||||||
|
|
||||||
|
### Step 3: Filter Errors
|
||||||
|
|
||||||
|
From the `errors` array, keep:
|
||||||
|
|
||||||
|
- **All errors** (level `"error"`): runtime errors, require failures, nil indexing, type mismatches, missing modules, etc. These always matter.
|
||||||
|
- **Severely problematic warnings** (level `"warning"`): only keep warnings that indicate real breakage:
|
||||||
|
- "Infinite yield possible" (waiting on something that will never resolve)
|
||||||
|
- Security/permission errors
|
||||||
|
- Warnings that indicate imminent failure
|
||||||
|
|
||||||
|
**Ignore:**
|
||||||
|
- Routine deprecation notices
|
||||||
|
- Asset loading warnings
|
||||||
|
- Cosmetic/informational warnings
|
||||||
|
- Any message starting with `[Atlas]` (internal plugin messages, already filtered by the tool)
|
||||||
|
|
||||||
|
### Step 4: Diagnose and Fix
|
||||||
|
|
||||||
|
**If no relevant errors:** Report success. Done.
|
||||||
|
|
||||||
|
**If errors found:**
|
||||||
|
|
||||||
|
1. **Parse each error message** for the script path. Roblox errors follow the pattern:
|
||||||
|
`ServiceName.Path.To.Script:LineNumber: error message`
|
||||||
|
e.g. `ServerScriptService.GameManager.PlayerHandler:42: attempt to index nil with 'Character'`
|
||||||
|
|
||||||
|
2. **Map the script path to a filesystem path.** The service path segments correspond to the Atlas project tree. Use `get_script` with the instance path if you need the `fsPath`, or infer it from the project structure.
|
||||||
|
|
||||||
|
3. **Read the erroring script** from the filesystem.
|
||||||
|
|
||||||
|
4. **Diagnose the error** using the error message and line number. Common patterns:
|
||||||
|
- `attempt to index nil` -- variable is nil when it shouldn't be
|
||||||
|
- `attempt to call a nil value` -- function doesn't exist or module didn't return it
|
||||||
|
- `Module code did not return exactly one value` -- bad module return
|
||||||
|
- `Requested module experienced an error while loading` -- cascading require failure
|
||||||
|
- `X is not a valid member of Y` -- wrong property/child name
|
||||||
|
|
||||||
|
5. **Fix the code** and write the corrected file.
|
||||||
|
|
||||||
|
6. **If called standalone** (not from atlas-push): tell the user what you fixed. They'll need to sync the fix to Studio themselves.
|
||||||
|
|
||||||
|
7. **If the error is ambiguous** or you can't determine the fix: ask the user. Show the error, the relevant code, and ask what they want to do via plain text.
|
||||||
|
|
||||||
|
### Plan Mode Behavior
|
||||||
|
|
||||||
|
If you're in plan mode when this skill triggers, do NOT run the playtest. Instead, create a plan that:
|
||||||
|
1. Lists the MCP call to make
|
||||||
|
2. Notes that errors will need to be diagnosed and fixed
|
||||||
|
3. Outlines the fix loop
|
||||||
|
|
||||||
|
## Quick Reference
|
||||||
|
|
||||||
|
**Run playtest:**
|
||||||
|
```
|
||||||
|
server: "project-0-rojo-atlas", toolName: "run_script_in_play_mode"
|
||||||
|
arguments: { "code": "task.wait(7)\nprint(\"boot complete\")", "mode": "start_play", "timeout": 15 }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Stop play (if stuck):**
|
||||||
|
```
|
||||||
|
server: "project-0-rojo-atlas", toolName: "start_stop_play", arguments: { "mode": "stop" }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Get script by path (to find fsPath for an erroring script):**
|
||||||
|
```
|
||||||
|
server: "project-0-rojo-atlas", toolName: "get_script", arguments: { "fsPath": "src/server/MyScript.server.luau" }
|
||||||
|
```
|
||||||
|
|
@ -176,6 +176,42 @@ Parse the response status:
|
||||||
|
|
||||||
**Keep retrying** until sync succeeds or user explicitly says to stop. Each retry is safe because `studioHash` prevents clobbering.
|
**Keep retrying** until sync succeeds or user explicitly says to stop. Each retry is safe because `studioHash` prevents clobbering.
|
||||||
|
|
||||||
|
### Step 8: Post-Push Playtest
|
||||||
|
|
||||||
|
**Skip this step** if the user's message contains "no test", "no playtest", "skip test", "no check", or "without testing".
|
||||||
|
|
||||||
|
After a successful sync, automatically run a quick playtest to catch runtime errors introduced by the push.
|
||||||
|
|
||||||
|
1. **Record pushed files.** Collect all `fsPath` values from changes where you pushed (direction `"push"` in the final sync response, or any script you merged and pushed). Also note the corresponding instance path segments (e.g. `src/server/GameManager.server.luau` corresponds to `ServerScriptService.GameManager`). You need these for error attribution.
|
||||||
|
|
||||||
|
2. **Run the playtest:**
|
||||||
|
|
||||||
|
```
|
||||||
|
CallMcpTool:
|
||||||
|
server: "project-0-rojo-atlas"
|
||||||
|
toolName: "run_script_in_play_mode"
|
||||||
|
arguments: {
|
||||||
|
"code": "task.wait(7)\nprint(\"boot complete\")",
|
||||||
|
"mode": "start_play",
|
||||||
|
"timeout": 15
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Parse and filter errors.** From the response `errors` array, keep:
|
||||||
|
- All entries with level `"error"`
|
||||||
|
- Warnings that indicate real breakage ("Infinite yield possible", security errors)
|
||||||
|
- Ignore routine/cosmetic warnings, asset loading warnings, deprecation notices
|
||||||
|
|
||||||
|
4. **Attribute errors to pushed files.** Roblox error messages include the script path, e.g.:
|
||||||
|
`ServerScriptService.GameManager.PlayerHandler:42: attempt to index nil`
|
||||||
|
|
||||||
|
Match the script name/path segments against your pushed `fsPath` list. Attribution is at the **whole-file level** -- if an error comes from a script we pushed, it's ours even if the erroring line wasn't part of our specific diff. Our change may have broken something else in that file.
|
||||||
|
|
||||||
|
5. **Handle errors:**
|
||||||
|
- **Our errors** (originate from a pushed file): read the script, diagnose the error using the message and line number, fix the code, write the corrected file, then **loop back to Step 1** (dryrun the fix, sync it, re-test). Keep looping until the playtest is clean.
|
||||||
|
- **Others' errors** (no match to any pushed file): ask the user. Show the error, explain it doesn't originate from files we pushed, and recommend they investigate in a planning phase. Let the user decide via text input what to do. Do not auto-fix code you didn't push.
|
||||||
|
- **No errors**: report success. Push and playtest complete.
|
||||||
|
|
||||||
## Safety Invariants
|
## Safety Invariants
|
||||||
|
|
||||||
1. **Never sync without a dryrun first** -- always review what will change.
|
1. **Never sync without a dryrun first** -- always review what will change.
|
||||||
|
|
|
||||||
9
TODO.txt
9
TODO.txt
|
|
@ -89,3 +89,12 @@ in all 3 matchers, simply randomize the order of the children - this will break
|
||||||
[syncback via extension]
|
[syncback via extension]
|
||||||
can we allow the terminal to stay so i can like run some git commands myself without having to make a whole new terminal?
|
can we allow the terminal to stay so i can like run some git commands myself without having to make a whole new terminal?
|
||||||
i also want to just cleanup terminals / use one terminal for syncback and serve...
|
i also want to just cleanup terminals / use one terminal for syncback and serve...
|
||||||
|
|
||||||
|
[skill]
|
||||||
|
'/playtest-quick'
|
||||||
|
simply runs the game and checks for literal errors, typically created just after completing a code change
|
||||||
|
resolve these errors in the project, or if in plan mode create a plan to resolve the errors
|
||||||
|
this should utilize the mcp to run the game and fetch the errors, make sure the server is fully booted up (INTELLIGENT 5-10s wait) and then fix stop the game
|
||||||
|
|
||||||
|
modify /atlas-push to always trigger /playtest-quick after it finishes pushing, looping back around to fix issues if needed. make sure the errors are related to what we actually pushed, so it doesnt try to fix others work. if it's others work, ask the user if they want to fix it - thought recommend they should do a planning phase.
|
||||||
|
"/atlas-push no test" or similar it just skips the automated test
|
||||||
|
|
|
||||||
7
assets/project-templates/place/.cursor/mcp.json
Normal file
7
assets/project-templates/place/.cursor/mcp.json
Normal file
|
|
@ -0,0 +1,7 @@
|
||||||
|
{
|
||||||
|
"mcpServers": {
|
||||||
|
"atlas": {
|
||||||
|
"url": "http://localhost:34873/mcp"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Loading…
Add table
Add a link
Reference in a new issue