--- description: "Web API and plugin communication: WebSocket protocol, live sync architecture, two-way sync, patch format, ServeSession state management, and ChangeBatcher." globs: - "src/web/**/*.rs" - "src/serve_session.rs" - "plugin/src/ServeSession.lua" - "plugin/src/ChangeBatcher/**/*.lua" - "plugin/src/ApiContext.lua" --- # Atlas API & Communication > Atlas is a fork of Rojo. Many internal identifiers, API endpoints, and struct names still reference the original "Rojo" naming. ## Quality Standard This is a production-grade synchronization system. The filesystem must represent a 1:1 copy of the Studio place. The core invariant: **Round-trip identity:** Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a bit-identical instance tree -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug. Every audit item must be evaluated against this standard. A "works most of the time" finding is not acceptable. If there is ANY code path where an instance name, property, or hierarchy relationship can be lost, mangled, or silently altered through a syncback→rebuild cycle, flag it as critical. ## Code Quality Standard We are building a maintainable codebase. During audits, watch for duplicated logic -- the same slugify/dedup/meta-update pattern copy-pasted across `change_processor.rs`, `api.rs`, `dir.rs`, `project.rs`, etc. If you find duplicated code that could be consolidated into a shared helper: - **Small refactor** (extracting a helper function, consolidating 2-3 call sites): flag it AND fix it as part of the audit. - **Major system rewrite** (restructuring data flow, changing function signatures across 10+ call sites, redesigning the snapshot pipeline): flag it clearly in a "Deferred Refactors" section at the bottom of the report. We will evaluate whether it's worth doing after the audit is complete. The goal is clean, DRY code that is easy to reason about and hard to get wrong. If the same "compute meta path, check if name needs slugify, upsert/remove name field" sequence appears in 3+ places, that's a refactor candidate. **Protocol Version:** 6 | Server/plugin MAJOR.MINOR must match ## Architecture ``` Filesystem → Server (serve_session.rs) → WebSocket (web/api.rs) → Plugin (ServeSession.lua) → Studio Two-way (experimental): Studio → ChangeBatcher → Plugin → WebSocket → Server → Filesystem ``` ## HTTP Endpoints (`src/web/api.rs`) | Endpoint | Method | Purpose | |----------|--------|---------| | `/api/rojo` | GET | Server info (version, sessionId, projectName, rootInstanceId, servePlaceIds) | | `/api/read/:id` | GET | Read instance tree from ID | | `/api/serialize/:id` | GET | Instance tree as msgpack | | `/api/write` | POST | Two-way sync: plugin sends changes | | `/api/open/:id` | POST | Open script in editor (`{"cursor": [line, col]}`) | | `/api/ref-patch/:id` | GET | Fix Ref properties after syncback | | `/api/socket/:cursor` | GET | WebSocket upgrade for live patch streaming | ### Server Info Response (`GET /api/rojo`) ```json { "serverVersion": "7.7.0-rc.1", "serverFork": "atlas", "protocolVersion": 6, "sessionId": "uuid", "projectName": "MyProject", "expectedPlaceIds": [123456], "unexpectedPlaceIds": [111111], "rootInstanceId": "00000000-0000-0000-0000-000000000000", "syncSourceOnly": true, "visibleServices": ["Workspace", "ServerScriptService"], "placeId": 123456, "gameId": 789012, "ignoreHiddenServices": true } ``` ### Write Request (`POST /api/write`) ```json { "sessionId": "uuid", "removed": ["id-1"], "added": { "id-2": { "parent": "parent-id", "className": "Folder", "name": "New", "properties": {}, "children": [] } }, "updated": [{ "id": "id-3", "changedName": "Renamed", "changedClassName": "Script", "changedProperties": { "Source": "print('hi')" }, "changedMetadata": {} }] } ``` **Format detection:** `Name/init.luau` → directory | `Name.luau` → file | neither → use `has_children` (`.lua` also supported) ## WebSocket Protocol (v6) **Connection:** `ws://host:port/api/socket/:cursor` 1. Connect → Receive `ServerInfo` handshake → Send `{type = "Subscribe"}` → Receive patches ### Patch Message (`ApplyPatch`) ```lua { type = "ApplyPatch", sessionId = "uuid", patch = { removed = {"id-1", "id-2"}, added = { ["id-3"] = { id = "id-3", className = "Folder", name = "New", properties = {}, metadata = { ignoreUnknownInstances = false }, children = {"child-1"}, }, }, updated = { { id = "id-4", changedName = "Renamed", changedProperties = { Source = "code" } }, }, } } ``` **Format:** MessagePack (~30% smaller than JSON, ~2x faster) ## Server ServeSession (`src/serve_session.rs`) ```rust pub struct ServeSession { session_id: SessionId, tree: Arc>, message_queue: Arc>, root_project: Project, vfs: Arc, change_processor: ChangeProcessor, start_time: Instant, tree_mutation_sender: Sender, suppressed_paths: Arc>>, } ``` **Lifecycle:** Create → Watch filesystem → Generate patches on changes → Broadcast to plugins → Shutdown **Message Queue:** Batches changes (~100ms window), cursor system for reconnect recovery **WebSocket loop:** ```rust loop { select! { msg = receiver.next() => handle_plugin_message(msg), patch = message_queue.recv() => sender.send(patch).await, } } ``` ## Plugin ServeSession (`plugin/src/ServeSession.lua`) ```lua ServeSession = { __status = "NotStarted", -- NotStarted → Connecting → Connected → Disconnected __apiContext = ApiContext, __twoWaySync = false, __reconciler = Reconciler, __instanceMap = InstanceMap, __changeBatcher = ChangeBatcher, __confirmingPatch = nil, __isConfirming = false, -- Prevents stacking dialogs, changes during review } ``` ### Patch Confirmation **Setting `confirmationBehavior`:** `"Always"` (default) | `"Initial"` | `"Large Changes"` | `"Unlisted PlaceId"` | `"Never"` **Flow:** Receive patch → Check setting → Show PatchVisualizer (pause ChangeBatcher) → User confirms → Apply via Reconciler → Resume ChangeBatcher ### Patch Application ```lua -- Remove for _, id in patch.removed do instanceMap:fromId(id):Destroy(); instanceMap:removeId(id) end -- Add for id, data in patch.added do instanceMap:insert(id, Reconciler.reify(data)) end -- Update for _, update in patch.updated do Reconciler.hydrate(instanceMap:fromId(update.id), update) end ``` ## Two-Way Sync (Experimental) **Setting:** `twoWaySync` (default: true) | Syncs Studio changes → filesystem in real-time ### Complete Data Flow ``` ┌─────────────────────────────────────────────────────────────────────────────┐ │ PLUGIN SIDE │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ Studio Instance │ │ │ PropertyChanged / DescendantAdded / etc. │ │ ▼ │ │ InstanceMap.__connectSignals() │ │ │ Fires onInstanceChanged(instance, propertyName) │ │ │ (skipped if RunService:IsRunning() or instance is paused) │ │ ▼ │ │ ServeSession.onInstanceChanged() │ │ │ Guards: if not self.__twoWaySync then return end │ │ ▼ │ │ ChangeBatcher:add(instance, propertyName) │ │ │ Stores in __pendingPropertyChanges[instance][propertyName] = true │ │ ▼ │ │ ChangeBatcher:__cycle(dt) [RenderStepped, every frame] │ │ │ Accumulates dt until >= BATCH_INTERVAL (0.2s) │ │ │ Guards: if __paused then return end │ │ ▼ │ │ ChangeBatcher:__flush() │ │ │ Calls createPatchSet(instanceMap, pendingChanges, syncSourceOnly) │ │ │ Clears __pendingPropertyChanges │ │ ▼ │ │ onChangesFlushed(patch) │ │ │ Guards: if Settings:get("oneShotSync") then return end │ │ ▼ │ │ ApiContext:write(patch) → HTTP POST /api/write (MessagePack) │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────────────────┐ │ SERVER SIDE │ ├─────────────────────────────────────────────────────────────────────────────┤ │ │ │ handle_api_write() [src/web/api.rs] │ │ │ Validates session_id │ │ │ Deserializes MessagePack → WriteRequest │ │ ▼ │ │ For each removed: syncback_removed_instance() │ │ │ Looks up InstigatingSource::Path → fs::remove_file() │ │ ▼ │ │ For each added: syncback_added_instance() │ │ │ Determines file format, writes to filesystem │ │ ▼ │ │ Sends PatchSet to tree_mutation_sender channel │ │ ▼ │ │ ChangeProcessor::handle_tree_event() [src/change_processor.rs] │ │ │ For removed: fs::remove_file(path) + remove meta │ │ │ For updated: Source writes, renames, ClassName changes, properties │ │ ▼ │ │ apply_patch_set() → Updates RojoTree │ │ ▼ │ │ message_queue.push_messages() → Broadcasts to all plugins │ │ │ └─────────────────────────────────────────────────────────────────────────────┘ ``` ### InstanceMap: Change Detection Foundation (`plugin/src/InstanceMap.lua`) The InstanceMap is a bidirectional map (ID ↔ Instance) that also hooks property change signals: ```lua InstanceMap = { fromIds = {}, -- Server ID → Studio Instance fromInstances = {}, -- Studio Instance → Server ID pausedUpdateInstances = {}, -- Set of instances to ignore changes from instancesToSignal = {}, -- Instance → RBXScriptConnection(s) onInstanceChanged = fn, -- Callback: (instance, propertyName) → void } ``` **Signal Connection:** ```lua function InstanceMap:__connectSignals(instance) if instance:IsA("ValueBase") then -- ValueBase.Changed only fires for Value, need explicit signals signals = { instance:GetPropertyChangedSignal("Name"):Connect(...), instance:GetPropertyChangedSignal("Value"):Connect(...), instance:GetPropertyChangedSignal("Parent"):Connect(...), } else -- Regular instances use Changed event (fires for any property) instance.Changed:Connect(function(propertyName) self:__maybeFireInstanceChanged(instance, propertyName) end) end end ``` **Change Guards:** ```lua function InstanceMap:__maybeFireInstanceChanged(instance, propertyName) if self.pausedUpdateInstances[instance] then return end -- Paused during reconciliation if self.onInstanceChanged == nil then return end -- No callback registered if RunService:IsRunning() then return end -- Don't sync during Play mode self.onInstanceChanged(instance, propertyName) end ``` **Pause Mechanism:** When applying incoming patches, instances are paused to prevent their changes from being sent back to the server (feedback loop prevention). ### ChangeBatcher (`plugin/src/ChangeBatcher/`) | File | Purpose | |------|---------| | `init.lua` | Main batcher: accumulator, RenderStepped loop, pause/resume | | `createPatchSet.lua` | Converts pending changes → PatchSet | | `encodePatchUpdate.lua` | Encodes property updates for existing instances | | `encodeInstance.lua` | Full instance encoding (for additions/syncback) | | `encodeProperty.lua` | Single property value encoding via RbxDom | **Core State:** ```lua ChangeBatcher = { __accumulator = 0, -- Time since last flush __pendingPropertyChanges = {}, -- Instance → { PropertyName → true } __syncSourceOnly = false, -- Only sync Source property __paused = false, -- Pause during confirmation __renderSteppedConnection = conn, -- Frame loop connection __instanceMap = instanceMap, __onChangesFlushed = callback, } ``` **Batch Cycle (every frame):** ```lua local BATCH_INTERVAL = 0.2 -- 200ms function ChangeBatcher:__cycle(dt) if self.__paused then return end self.__accumulator += dt if self.__accumulator >= BATCH_INTERVAL then self.__accumulator -= BATCH_INTERVAL local patch = self:__flush() if patch then self.__onChangesFlushed(patch) end end self.__instanceMap:unpauseAllInstances() -- Resume paused instances end ``` **Why 200ms batch interval?** - Studio fires many rapid property changes (e.g., dragging a part fires Position.X, Y, Z separately) - Batching combines related changes into single patch - Reduces server load and WebSocket traffic - Still feels responsive to users ### createPatchSet.lua: Patch Generation ```lua function createPatchSet(instanceMap, propertyChanges, syncSourceOnly) local patch = PatchSet.newEmpty() for instance, properties in propertyChanges do local instanceId = instanceMap.fromInstances[instance] if instanceId == nil then Log.warn("Ignoring change for unknown instance") continue end if properties.Parent then if instance.Parent == nil then -- Instance was deleted in Studio table.insert(patch.removed, instanceId) else Log.warn("Cannot sync non-nil Parent changes yet") -- Reparenting not supported end else if syncSourceOnly then -- Server in source-only mode: only sync Source property if properties.Source then local update = encodePatchUpdate(instance, instanceId, { Source = true }) if update then table.insert(patch.updated, update) end end else local update = encodePatchUpdate(instance, instanceId, properties) if update then table.insert(patch.updated, update) end end end end return patch end ``` ### encodePatchUpdate.lua: Update Encoding ```lua function encodePatchUpdate(instance, instanceId, properties) local update = { id = instanceId, changedProperties = {}, } for propertyName in properties do if propertyName == "Name" then update.changedName = instance.Name else local descriptor = RbxDom.findCanonicalPropertyDescriptor(instance.ClassName, propertyName) if not descriptor then continue end local success, encoded = encodeProperty(instance, propertyName, descriptor) if success then update.changedProperties[propertyName] = encoded end end end if next(update.changedProperties) == nil and update.changedName == nil then return nil -- No actual changes end return update end ``` ### encodeInstance.lua: Full Instance Encoding (Syncback/Additions) Used when pulling new instances from Studio to filesystem. **Critical: handles duplicate detection.** ```lua -- Duplicate handling (matches server's ref_properties.rs::is_path_unique) local function isPathUnique(instance) local current = instance while current do if hasDuplicateSiblings(current) then return false -- Any duplicate in ancestor chain = ambiguous path end current = current.Parent if current == game then break end end return true end function encodeInstance(instance, parentId, _skipPathCheck) -- Skip instances with ambiguous paths (duplicate-named siblings) if not _skipPathCheck and not isPathUnique(instance) then Log.warn("Skipped '{}' - path contains duplicate-named siblings", instance:GetFullName()) return nil end local properties = {} -- Encode Attributes and Tags (work on all instance types) encodeAttributes(instance, properties) encodeTags(instance, properties) -- For scripts, Source is required if SCRIPT_CLASS_NAMES[instance.ClassName] then local success, encoded = encodeProperty(instance, "Source", descriptor) if not success then return nil end properties.Source = encoded end -- Encode all serializable properties local classDescriptor = RbxDom.findClassDescriptor(instance.ClassName) for propertyName, meta in classDescriptor.properties do if SKIP_PROPERTIES[propertyName] then continue end -- Parent, Name, Archivable if SKIP_PROPERTIES_BY_CLASS[instance.ClassName]?[propertyName] then continue end -- Source, ScriptGuid, etc. local isReadable = meta.scriptability == "ReadWrite" or meta.scriptability == "Read" local doesSerialize = meta.serialization ~= "DoesNotSerialize" if isReadable and doesSerialize then local success, encoded = encodeProperty(instance, propertyName, descriptor) if success and encoded ~= nil then properties[propertyName] = encoded end end end -- Recursively encode children (skipping duplicates at each level) local children = {} local childDuplicates = findDuplicateNames(instance:GetChildren()) for _, child in instance:GetChildren() do if childDuplicates[child.Name] then continue end local encoded = encodeInstance(child, nil, true) -- Skip path check, already validated if encoded then table.insert(children, encoded) end end return { parent = parentId, name = instance.Name, className = instance.ClassName, properties = properties, children = children, } end ``` **Properties skipped per class:** | Class | Skipped Properties | Reason | |-------|-------------------|--------| | Script/LocalScript/ModuleScript | Source, ScriptGuid, RunContext | Encoded in file contents/suffix | | LocalizationTable | Contents | Encoded in .csv file | | StringValue | Value | Encoded in .txt file | ### encodeProperty.lua: Value Encoding ```lua function encodeProperty(instance, propertyName, descriptor) local readSuccess, value = descriptor:read(instance) if not readSuccess then return false, nil end local dataType = descriptor.dataType local encodeSuccess, encoded = RbxDom.EncodedValue.encode(value, dataType) if not encodeSuccess then return false, nil end return true, encoded end ``` **Encoded value format (via RbxDom):** ```lua -- Primitives true, false, 42, "string" -- Roblox types { Vector3 = {1, 2, 3} } { CFrame = { position = {0,0,0}, orientation = {...} } } { UDim2 = {{0, 100}, {0, 50}} } { Color3 = {1, 0.5, 0} } { BrickColor = 194 } { Ref = "server-instance-id" } -- Object references { Attributes = { key = value, ... } } { Tags = {"tag1", "tag2"} } ``` ### Server-Side: ChangeProcessor (`src/change_processor.rs`) Receives patches from plugin via `tree_mutation_receiver` channel. Handles removals, Source writes, renames (with slugify/dedup/meta lifecycle), ClassName changes (script type transitions via file renames), and property writes to meta/model files. **Capabilities:** - **Removals:** Deletes file + associated meta file via `InstigatingSource::Path` - **Source writes:** Updates script file contents - **Renames:** Slugifies new name, deduplicates against siblings, renames file on disk, creates/updates/removes meta `name` field as needed. Handles both standalone files and init-file directories. - **ClassName changes:** Script-to-script transitions (e.g. ModuleScript → Script) via file extension renames - **Property writes:** Attributes and other properties written to meta/model files via `syncback_updated_properties` in `api.rs` **InstigatingSource:** Every instance in RojoTree tracks where it came from: - `InstigatingSource::Path(PathBuf)` - Created from a file, can be written back - `InstigatingSource::ProjectNode { path, name, node, parent_class }` - Defined in project file, **cannot be modified via two-way sync** ### Pause Mechanism: Critical for Consistency **Why pause is essential:** 1. **During incoming patch confirmation:** User is reviewing changes. If ChangeBatcher sends Studio changes mid-review, it creates race conditions. 2. **During patch application:** Reconciler modifies instances, triggering property change events. Without pause, these would be sent back as "changes" (feedback loop). 3. **Defense-in-depth:** `__isConfirming` flag in ServeSession prevents multiple dialogs and ensures atomicity. ```lua -- In ServeSession, when showing confirmation dialog: self.__changeBatcher:pause() -- ... show PatchVisualizer, wait for user ... self.__changeBatcher:resume() ``` ```lua -- In Reconciler, when modifying instances: instanceMap:pauseInstance(instance) instance.SomeProperty = newValue -- Won't trigger ChangeBatcher -- Later, in ChangeBatcher:__cycle(): instanceMap:unpauseAllInstances() ``` ### One-Shot Mode Integration **Setting:** `oneShotSync` (default: true) One-shot mode adds an extra guard in `onChangesFlushed`: ```lua local function onChangesFlushed(patch) if Settings:get("oneShotSync") then Log.info("One-shot mode: blocking automatic outgoing write from ChangeBatcher") return -- Silently discard patch end self.__apiContext:write(patch) end ``` **Behavior:** - ChangeBatcher still collects changes (needed for diff visualization) - Automatic writes are blocked - Only explicit "pull" selections in confirmation dialog can write - Disconnects after initial sync completes - No sync lock acquired (multiple users can view same place) **Use cases:** - Quick preview of changes without risk of accidental writes - Safe for production environments - Team viewing same project simultaneously ### Known Limitations (Detailed) | Issue | Cause | Impact | |-------|-------|--------| | **Undo/redo triggers spurious changes** | Studio's ChangeHistoryService fires property changes during undo | Can cause unexpected writes or conflicts | | **Rapid changes interleave** | 200ms batch window may split logically related changes | Partial updates can leave inconsistent state | | **Reparenting not supported** | `Parent` property changes to non-nil values are ignored | Moving instances in Studio doesn't sync | | **Project node instances immutable** | InstigatingSource::ProjectNode cannot be modified | Instances from project file can't be changed | | **Duplicate-named siblings skipped** | Path ambiguity makes reliable sync impossible | Instances with same-name siblings ignored | | **Ref properties complex** | Instance references may become invalid when instances move | ObjectValue.Value may not sync correctly | ## Protocol Enforcement ```rust // Server if client_protocol != PROTOCOL_VERSION { return error("Protocol mismatch") } ``` ```lua -- Plugin if serverInfo.protocolVersion ~= Config.protocolVersion then showError("Protocol mismatch") end ``` **Breaking changes:** Bump `protocolVersion` in `src/web/interface.rs` and `plugin/src/Config.lua` ## Session Management 1. Server starts → Generate UUID 2. Plugin connects → Receives session ID in handshake 3. All requests include session ID 4. Server restart → New session ID → Plugin auto-reconnects on mismatch ## Debugging ```bash # Server RUST_LOG=rojo::web=debug,rojo::serve_session=debug atlas serve ``` ```lua -- Plugin Settings:set("logLevel", "Trace") ``` Browser: `http://localhost:34872` → Network tab → WebSocket messages ## Key Files | Server | Plugin | |--------|--------| | `src/web/api.rs` - Endpoints | `plugin/src/ServeSession.lua` - Connection | | `src/web/interface.rs` - Packets | `plugin/src/ApiContext.lua` - HTTP/WS client | | `src/serve_session.rs` - Session | `plugin/src/ChangeBatcher/` - Two-way sync | | | `plugin/src/Reconciler/` - Patch application |