hakowork/rojo
3
0
Fork 0
mirror of https://github.com/rojo-rbx/rojo.git synced 2026-10-23 05:51:39 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Enhance connected mode test suite and introduce VFS staleness fixtures

Browse source 
- Updated the connected mode test expansion plan, marking all todos as completed, including multi-step sequence tests and various phase implementations.
- Added a new plan for VFS staleness test fixtures, outlining the creation of a validation API, test helpers, and multiple staleness tests to improve detection of VFS event issues.
- Introduced new rules for the Atlas API and project file format specifications, enhancing the overall structure and documentation for better clarity and maintainability.

This update significantly improves testing coverage and addresses potential VFS synchronization issues.
This commit is contained in:
Joe 2026-02-11 17:56:37 -06:00
parent 316d7ea70e
commit f798f3f5e8
107 changed files with 4484 additions and 41 deletions
Download patch file Download diff file Expand all files Collapse all files

655
.cursor/rules/rojo-api.mdc
View file

@ -1,655 +0,0 @@
---
description: "Web API and plugin communication: WebSocket protocol, live sync architecture, two-way sync, patch format, ServeSession state management, and ChangeBatcher."
globs:
- "src/web/**/*.rs"
- "src/serve_session.rs"
- "plugin/src/ServeSession.lua"
- "plugin/src/ChangeBatcher/**/*.lua"
- "plugin/src/ApiContext.lua"
---
# Rojo API & Communication
**Protocol Version:** 5 | Server/plugin MAJOR.MINOR must match
## Architecture
```
Filesystem → Server (serve_session.rs) → WebSocket (web/api.rs) → Plugin (ServeSession.lua) → Studio
Two-way (experimental):
Studio → ChangeBatcher → Plugin → WebSocket → Server → Filesystem
```
## HTTP Endpoints (`src/web/api.rs`)
| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/rojo` | GET | Server info (version, sessionId, projectName, rootInstanceId, servePlaceIds) |
| `/api/read/:id` | GET | Read instance tree from ID |
| `/api/serialize/:id` | GET | Instance tree as msgpack |
| `/api/write` | POST | Two-way sync: plugin sends changes |
| `/api/open/:id` | POST | Open script in editor (`{"cursor": [line, col]}`) |
| `/api/ref-patch/:id` | GET | Fix Ref properties after syncback |
### Server Info Response (`GET /api/rojo`)
```json
{
"serverVersion": "7.7.0-rc.1",
"protocolVersion": 5,
"sessionId": "uuid",
"projectName": "MyProject",
"expectedPlaceIds": [123456],
"rootInstanceId": "00000000-0000-0000-0000-000000000000",
"syncSourceOnly": true,
"visibleServices": ["Workspace", "ServerScriptService"]
}
```
### Write Request (`POST /api/write`)
```json
{
"removed": ["id-1"],
"added": { "id-2": { "id": "id-2", "className": "Folder", "name": "New", "properties": {}, "children": [] } },
"updated": [{ "id": "id-3", "changedName": "Renamed", "changedProperties": { "Source": "print('hi')" } }]
}
```
**Format detection:** `Name/init.luau` → directory | `Name.luau` → file | neither → use `has_children` (`.lua` also supported)
## WebSocket Protocol (v5)
**Connection:** `ws://host:port/api/rojo`
1. Connect → Receive `ServerInfo` handshake → Send `{type = "Subscribe"}` → Receive patches
### Patch Message (`ApplyPatch`)
```lua
{
type = "ApplyPatch",
sessionId = "uuid",
patch = {
removed = {"id-1", "id-2"},
added = {
["id-3"] = {
id = "id-3", className = "Folder", name = "New",
properties = {}, metadata = { ignoreUnknownInstances = false },
children = {"child-1"},
},
},
updated = {
{ id = "id-4", changedName = "Renamed", changedProperties = { Source = "code" } },
},
}
}
```
**Format:** MessagePack (~30% smaller than JSON, ~2x faster)
## Server ServeSession (`src/serve_session.rs`)
```rust
pub struct ServeSession {
session_id: Uuid,
tree_handle: Arc<Mutex<RojoTree>>,
message_queue: MessageQueue,
project: Project,
vfs: Vfs,
}
```
**Lifecycle:** Create → Watch filesystem → Generate patches on changes → Broadcast to plugins → Shutdown
**Message Queue:** Batches changes (~100ms window), cursor system for reconnect recovery
**WebSocket loop:**
```rust
loop {
select! {
msg = receiver.next() => handle_plugin_message(msg),
patch = message_queue.recv() => sender.send(patch).await,
}
}
```
## Plugin ServeSession (`plugin/src/ServeSession.lua`)
```lua
ServeSession = {
__status = "NotStarted", -- NotStarted → Connecting → Connected → Disconnected
__apiContext = ApiContext,
__twoWaySync = false,
__reconciler = Reconciler,
__instanceMap = InstanceMap,
__changeBatcher = ChangeBatcher,
__confirmingPatch = nil,
__isConfirming = false, -- Prevents stacking dialogs, changes during review
}
```
### Patch Confirmation
**Setting `confirmationBehavior`:** `"Always"` (default) | `"Initial"` | `"Large Changes"` | `"Unlisted PlaceId"` | `"Never"`
**Flow:** Receive patch → Check setting → Show PatchVisualizer (pause ChangeBatcher) → User confirms → Apply via Reconciler → Resume ChangeBatcher
### Patch Application
```lua
-- Remove
for _, id in patch.removed do instanceMap:fromId(id):Destroy(); instanceMap:removeId(id) end
-- Add
for id, data in patch.added do instanceMap:insert(id, Reconciler.reify(data)) end
-- Update
for _, update in patch.updated do Reconciler.hydrate(instanceMap:fromId(update.id), update) end
```
## Two-Way Sync (Experimental)
**Setting:** `twoWaySync` (default: true) | Syncs Studio changes → filesystem in real-time
### Complete Data Flow
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ PLUGIN SIDE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ Studio Instance │
│ │ PropertyChanged / DescendantAdded / etc. │
│ ▼ │
│ InstanceMap.__connectSignals() │
│ │ Fires onInstanceChanged(instance, propertyName) │
│ │ (skipped if RunService:IsRunning() or instance is paused) │
│ ▼ │
│ ServeSession.onInstanceChanged() │
│ │ Guards: if not self.__twoWaySync then return end │
│ ▼ │
│ ChangeBatcher:add(instance, propertyName) │
│ │ Stores in __pendingPropertyChanges[instance][propertyName] = true │
│ ▼ │
│ ChangeBatcher:__cycle(dt) [RenderStepped, every frame] │
│ │ Accumulates dt until >= BATCH_INTERVAL (0.2s) │
│ │ Guards: if __paused then return end │
│ ▼ │
│ ChangeBatcher:__flush() │
│ │ Calls createPatchSet(instanceMap, pendingChanges, syncSourceOnly) │
│ │ Clears __pendingPropertyChanges │
│ ▼ │
│ onChangesFlushed(patch) │
│ │ Guards: if Settings:get("oneShotSync") then return end │
│ ▼ │
│ ApiContext:write(patch) → HTTP POST /api/write (MessagePack) │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ SERVER SIDE │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ handle_api_write() [src/web/api.rs] │
│ │ Validates session_id │
│ │ Deserializes MessagePack → WriteRequest │
│ ▼ │
│ For each removed: syncback_removed_instance() │
│ │ Looks up InstigatingSource::Path → fs::remove_file() │
│ ▼ │
│ For each added: syncback_added_instance() │
│ │ Determines file format, writes to filesystem │
│ ▼ │
│ Sends PatchSet to tree_mutation_sender channel │
│ ▼ │
│ ChangeProcessor::handle_tree_event() [src/change_processor.rs] │
│ │ For removed: fs::remove_file(instigating_source.path) │
│ │ For updated: if "Source" property → fs::write(path, value) │
│ ▼ │
│ apply_patch_set() → Updates RojoTree │
│ ▼ │
│ message_queue.push_messages() → Broadcasts to all plugins │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
### InstanceMap: Change Detection Foundation (`plugin/src/InstanceMap.lua`)
The InstanceMap is a bidirectional map (ID ↔ Instance) that also hooks property change signals:
```lua
InstanceMap = {
fromIds = {}, -- Server ID → Studio Instance
fromInstances = {}, -- Studio Instance → Server ID
pausedUpdateInstances = {}, -- Set of instances to ignore changes from
instancesToSignal = {}, -- Instance → RBXScriptConnection(s)
onInstanceChanged = fn, -- Callback: (instance, propertyName) → void
}
```
**Signal Connection:**
```lua
function InstanceMap:__connectSignals(instance)
if instance:IsA("ValueBase") then
-- ValueBase.Changed only fires for Value, need explicit signals
signals = {
instance:GetPropertyChangedSignal("Name"):Connect(...),
instance:GetPropertyChangedSignal("Value"):Connect(...),
instance:GetPropertyChangedSignal("Parent"):Connect(...),
}
else
-- Regular instances use Changed event (fires for any property)
instance.Changed:Connect(function(propertyName)
self:__maybeFireInstanceChanged(instance, propertyName)
end)
end
end
```
**Change Guards:**
```lua
function InstanceMap:__maybeFireInstanceChanged(instance, propertyName)
if self.pausedUpdateInstances[instance] then return end -- Paused during reconciliation
if self.onInstanceChanged == nil then return end -- No callback registered
if RunService:IsRunning() then return end -- Don't sync during Play mode
self.onInstanceChanged(instance, propertyName)
end
```
**Pause Mechanism:** When applying incoming patches, instances are paused to prevent their changes from being sent back to the server (feedback loop prevention).
### ChangeBatcher (`plugin/src/ChangeBatcher/`)
| File | Purpose |
|------|---------|
| `init.lua` | Main batcher: accumulator, RenderStepped loop, pause/resume |
| `createPatchSet.lua` | Converts pending changes → PatchSet |
| `encodePatchUpdate.lua` | Encodes property updates for existing instances |
| `encodeInstance.lua` | Full instance encoding (for additions/syncback) |
| `encodeProperty.lua` | Single property value encoding via RbxDom |
**Core State:**
```lua
ChangeBatcher = {
__accumulator = 0, -- Time since last flush
__pendingPropertyChanges = {}, -- Instance → { PropertyName → true }
__syncSourceOnly = false, -- Only sync Source property
__paused = false, -- Pause during confirmation
__renderSteppedConnection = conn, -- Frame loop connection
__instanceMap = instanceMap,
__onChangesFlushed = callback,
}
```
**Batch Cycle (every frame):**
```lua
local BATCH_INTERVAL = 0.2 -- 200ms
function ChangeBatcher:__cycle(dt)
if self.__paused then return end
self.__accumulator += dt
if self.__accumulator >= BATCH_INTERVAL then
self.__accumulator -= BATCH_INTERVAL
local patch = self:__flush()
if patch then
self.__onChangesFlushed(patch)
end
end
self.__instanceMap:unpauseAllInstances() -- Resume paused instances
end
```
**Why 200ms batch interval?**
- Studio fires many rapid property changes (e.g., dragging a part fires Position.X, Y, Z separately)
- Batching combines related changes into single patch
- Reduces server load and WebSocket traffic
- Still feels responsive to users
### createPatchSet.lua: Patch Generation
```lua
function createPatchSet(instanceMap, propertyChanges, syncSourceOnly)
local patch = PatchSet.newEmpty()
for instance, properties in propertyChanges do
local instanceId = instanceMap.fromInstances[instance]
if instanceId == nil then
Log.warn("Ignoring change for unknown instance")
continue
end
if properties.Parent then
if instance.Parent == nil then
-- Instance was deleted in Studio
table.insert(patch.removed, instanceId)
else
Log.warn("Cannot sync non-nil Parent changes yet") -- Reparenting not supported
end
else
if syncSourceOnly then
-- Server in source-only mode: only sync Source property
if properties.Source then
local update = encodePatchUpdate(instance, instanceId, { Source = true })
if update then table.insert(patch.updated, update) end
end
else
local update = encodePatchUpdate(instance, instanceId, properties)
if update then table.insert(patch.updated, update) end
end
end
end
return patch
end
```
### encodePatchUpdate.lua: Update Encoding
```lua
function encodePatchUpdate(instance, instanceId, properties)
local update = {
id = instanceId,
changedProperties = {},
}
for propertyName in properties do
if propertyName == "Name" then
update.changedName = instance.Name
else
local descriptor = RbxDom.findCanonicalPropertyDescriptor(instance.ClassName, propertyName)
if not descriptor then continue end
local success, encoded = encodeProperty(instance, propertyName, descriptor)
if success then
update.changedProperties[propertyName] = encoded
end
end
end
if next(update.changedProperties) == nil and update.changedName == nil then
return nil -- No actual changes
end
return update
end
```
### encodeInstance.lua: Full Instance Encoding (Syncback/Additions)
Used when pulling new instances from Studio to filesystem. **Critical: handles duplicate detection.**
```lua
-- Duplicate handling (matches server's ref_properties.rs::is_path_unique)
local function isPathUnique(instance)
local current = instance
while current do
if hasDuplicateSiblings(current) then
return false -- Any duplicate in ancestor chain = ambiguous path
end
current = current.Parent
if current == game then break end
end
return true
end
function encodeInstance(instance, parentId, _skipPathCheck)
-- Skip instances with ambiguous paths (duplicate-named siblings)
if not _skipPathCheck and not isPathUnique(instance) then
Log.warn("Skipped '{}' - path contains duplicate-named siblings", instance:GetFullName())
return nil
end
local properties = {}
-- Encode Attributes and Tags (work on all instance types)
encodeAttributes(instance, properties)
encodeTags(instance, properties)
-- For scripts, Source is required
if SCRIPT_CLASS_NAMES[instance.ClassName] then
local success, encoded = encodeProperty(instance, "Source", descriptor)
if not success then return nil end
properties.Source = encoded
end
-- Encode all serializable properties
local classDescriptor = RbxDom.findClassDescriptor(instance.ClassName)
for propertyName, meta in classDescriptor.properties do
if SKIP_PROPERTIES[propertyName] then continue end -- Parent, Name, Archivable
if SKIP_PROPERTIES_BY_CLASS[instance.ClassName]?[propertyName] then continue end -- Source, ScriptGuid, etc.
local isReadable = meta.scriptability == "ReadWrite" or meta.scriptability == "Read"
local doesSerialize = meta.serialization ~= "DoesNotSerialize"
if isReadable and doesSerialize then
local success, encoded = encodeProperty(instance, propertyName, descriptor)
if success and encoded ~= nil then
properties[propertyName] = encoded
end
end
end
-- Recursively encode children (skipping duplicates at each level)
local children = {}
local childDuplicates = findDuplicateNames(instance:GetChildren())
for _, child in instance:GetChildren() do
if childDuplicates[child.Name] then continue end
local encoded = encodeInstance(child, nil, true) -- Skip path check, already validated
if encoded then table.insert(children, encoded) end
end
return {
parent = parentId,
name = instance.Name,
className = instance.ClassName,
properties = properties,
children = children,
}
end
```
**Properties skipped per class:**
| Class | Skipped Properties | Reason |
|-------|-------------------|--------|
| Script/LocalScript/ModuleScript | Source, ScriptGuid, RunContext | Encoded in file contents/suffix |
| LocalizationTable | Contents | Encoded in .csv file |
| StringValue | Value | Encoded in .txt file |
### encodeProperty.lua: Value Encoding
```lua
function encodeProperty(instance, propertyName, descriptor)
local readSuccess, value = descriptor:read(instance)
if not readSuccess then return false, nil end
local dataType = descriptor.dataType
local encodeSuccess, encoded = RbxDom.EncodedValue.encode(value, dataType)
if not encodeSuccess then return false, nil end
return true, encoded
end
```
**Encoded value format (via RbxDom):**
```lua
-- Primitives
true, false, 42, "string"
-- Roblox types
{ Vector3 = {1, 2, 3} }
{ CFrame = { position = {0,0,0}, orientation = {...} } }
{ UDim2 = {{0, 100}, {0, 50}} }
{ Color3 = {1, 0.5, 0} }
{ BrickColor = 194 }
{ Ref = "server-instance-id" } -- Object references
{ Attributes = { key = value, ... } }
{ Tags = {"tag1", "tag2"} }
```
### Server-Side: ChangeProcessor (`src/change_processor.rs`)
Receives patches from plugin via `tree_mutation_receiver` channel:
```rust
fn handle_tree_event(&self, patch_set: PatchSet) {
let mut tree = self.tree.lock().unwrap();
// Process removals
for &id in &patch_set.removed_instances {
if let Some(instance) = tree.get_instance(id) {
match &instance.metadata().instigating_source {
Some(InstigatingSource::Path(path)) => {
log::info!("Two-way sync: Removing file {}", path.display());
fs::remove_file(path)?;
}
Some(InstigatingSource::ProjectNode { .. }) => {
log::warn!("Cannot remove instance from project file");
}
None => {
log::warn!("Cannot remove instance without instigating source");
}
}
}
}
// Process updates
for update in &patch_set.updated_instances {
if let Some(instance) = tree.get_instance(update.id) {
// Currently only Source property is supported
if let Some(Variant::String(source)) = update.changed_properties.get("Source") {
match &instance.metadata().instigating_source {
Some(InstigatingSource::Path(path)) => {
log::info!("Two-way sync: Writing Source to {}", path.display());
fs::write(path, source)?;
}
_ => log::warn!("Cannot update instance without path source"),
}
}
// Other property changes logged but not written
if update.changed_name.is_some() { log::warn!("Cannot rename instances yet"); }
if update.changed_class_name.is_some() { log::warn!("Cannot change ClassName yet"); }
}
}
apply_patch_set(&mut tree, patch_set); // Update DOM
self.message_queue.push_messages(&[applied_patch]); // Broadcast to plugins
}
```
**InstigatingSource:** Every instance in RojoTree tracks where it came from:
- `InstigatingSource::Path(PathBuf)` - Created from a file, can be written back
- `InstigatingSource::ProjectNode { path, name, node, parent_class }` - Defined in project file, **cannot be modified via two-way sync**
### Pause Mechanism: Critical for Consistency
**Why pause is essential:**
1. **During incoming patch confirmation:** User is reviewing changes. If ChangeBatcher sends Studio changes mid-review, it creates race conditions.
2. **During patch application:** Reconciler modifies instances, triggering property change events. Without pause, these would be sent back as "changes" (feedback loop).
3. **Defense-in-depth:** `__isConfirming` flag in ServeSession prevents multiple dialogs and ensures atomicity.
```lua
-- In ServeSession, when showing confirmation dialog:
self.__changeBatcher:pause()
-- ... show PatchVisualizer, wait for user ...
self.__changeBatcher:resume()
```
```lua
-- In Reconciler, when modifying instances:
instanceMap:pauseInstance(instance)
instance.SomeProperty = newValue -- Won't trigger ChangeBatcher
-- Later, in ChangeBatcher:__cycle():
instanceMap:unpauseAllInstances()
```
### One-Shot Mode Integration
**Setting:** `oneShotSync` (default: true)
One-shot mode adds an extra guard in `onChangesFlushed`:
```lua
local function onChangesFlushed(patch)
if Settings:get("oneShotSync") then
Log.info("One-shot mode: blocking automatic outgoing write from ChangeBatcher")
return -- Silently discard patch
end
self.__apiContext:write(patch)
end
```
**Behavior:**
- ChangeBatcher still collects changes (needed for diff visualization)
- Automatic writes are blocked
- Only explicit "pull" selections in confirmation dialog can write
- Disconnects after initial sync completes
- No sync lock acquired (multiple users can view same place)
**Use cases:**
- Quick preview of changes without risk of accidental writes
- Safe for production environments
- Team viewing same project simultaneously
### Known Limitations (Detailed)
| Issue | Cause | Impact |
|-------|-------|--------|
| **Undo/redo triggers spurious changes** | Studio's ChangeHistoryService fires property changes during undo | Can cause unexpected writes or conflicts |
| **Rapid changes interleave** | 200ms batch window may split logically related changes | Partial updates can leave inconsistent state |
| **Reparenting not supported** | `Parent` property changes to non-nil values are ignored | Moving instances in Studio doesn't sync |
| **Renaming not supported** | `changedName` logged but not written | Renaming in Studio doesn't sync |
| **ClassName changes ignored** | Would require file type change | Changing script type doesn't sync |
| **Only Source property writes** | Other properties logged but not written to filesystem | Only script content syncs reliably |
| **Project node instances immutable** | InstigatingSource::ProjectNode cannot be modified | Instances from project file can't be changed |
| **Duplicate-named siblings skipped** | Path ambiguity makes reliable sync impossible | Instances with same-name siblings ignored |
| **Ref properties complex** | Instance references may become invalid when instances move | ObjectValue.Value may not sync correctly |
**Recommended workflow:** Use one-way sync (server → Studio) as primary. Use two-way sync only for script Source changes with careful attention to undo/redo behavior.
## Protocol Enforcement
```rust
// Server
if client_protocol != PROTOCOL_VERSION { return error("Protocol mismatch") }
```
```lua
-- Plugin
if serverInfo.protocolVersion ~= Config.protocolVersion then showError("Protocol mismatch") end
```
**Breaking changes:** Bump `protocolVersion` in `src/web/interface.rs` and `plugin/src/Config.lua`
## Session Management
1. Server starts → Generate UUID
2. Plugin connects → Receives session ID in handshake
3. All requests include session ID
4. Server restart → New session ID → Plugin auto-reconnects on mismatch
## Debugging
```bash
# Server
RUST_LOG=rojo::web=debug,rojo::serve_session=debug rojo serve
```
```lua
-- Plugin
Settings:set("logLevel", "Trace")
```
Browser: `http://localhost:34872` → Network tab → WebSocket messages
## Key Files
| Server | Plugin |
|--------|--------|
| `src/web/api.rs` - Endpoints | `plugin/src/ServeSession.lua` - Connection |
| `src/web/interface.rs` - Packets | `plugin/src/ApiContext.lua` - HTTP/WS client |
| `src/serve_session.rs` - Session | `plugin/src/ChangeBatcher/` - Two-way sync |
| | `plugin/src/Reconciler/` - Patch application |