mirror of
https://github.com/rojo-rbx/rojo.git
synced 2026-10-23 22:11:38 +00:00
- Added a new section in `audit.md` for the Live Syncback Pipeline, detailing the plugin encoding, transport, and server handling processes. - Included entries for newly introduced files `encodeHelpers.lua` and `encodeService.lua` in the audit documentation. - Updated the audit checklist to cover live syncback parity and related concerns. - Revised rule MDC files (`atlas-api.mdc`, `atlas-plugin.mdc`, `atlas-dedup.mdc`) to reflect recent changes, including the addition of `encodeHelpers.lua` and clarifications on `SyncbackConfirm`. - Ensured comprehensive coverage of new test suites in the audit documentation. These changes aim to enhance the clarity and accuracy of the audit documentation and rules, aligning them with the latest codebase updates.
797 lines
38 KiB
Text
797 lines
38 KiB
Text
---
|
|
globs: src/web/**,src/serve_session.rs,src/git.rs,,plugin/src/**
|
|
alwaysApply: false
|
|
---
|
|
|
|
# Atlas API & Communication
|
|
|
|
> Atlas is a fork of Rojo. Many internal identifiers, API endpoints, and struct names still reference the original "Rojo" naming.
|
|
|
|
## Quality Standard
|
|
|
|
This is a production-grade synchronization system. The filesystem must represent a 1:1 copy of the Studio place. The core invariant:
|
|
|
|
**Round-trip identity:** Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a bit-identical instance tree -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.
|
|
|
|
Every audit item must be evaluated against this standard. A "works most of the time" finding is not acceptable. If there is ANY code path where an instance name, property, or hierarchy relationship can be lost, mangled, or silently altered through a syncback→rebuild cycle, flag it as critical.
|
|
|
|
## Code Quality Standard
|
|
|
|
We are building a maintainable codebase. During audits, watch for duplicated logic -- the same slugify/dedup/meta-update pattern copy-pasted across `change_processor.rs`, `api.rs`, `dir.rs`, `project.rs`, etc. If you find duplicated code that could be consolidated into a shared helper:
|
|
|
|
- **Small refactor** (extracting a helper function, consolidating 2-3 call sites): flag it AND fix it as part of the audit.
|
|
- **Major system rewrite** (restructuring data flow, changing function signatures across 10+ call sites, redesigning the snapshot pipeline): flag it clearly in a "Deferred Refactors" section at the bottom of the report. We will evaluate whether it's worth doing after the audit is complete.
|
|
|
|
The goal is clean, DRY code that is easy to reason about and hard to get wrong. If the same "compute meta path, check if name needs slugify, upsert/remove name field" sequence appears in 3+ places, that's a refactor candidate.
|
|
|
|
**Protocol Version:** 6 | Server/plugin MAJOR.MINOR must match
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Filesystem → Server (serve_session.rs) → WebSocket (web/api.rs) → Plugin (ServeSession.lua) → Studio
|
|
|
|
Two-way (experimental):
|
|
Studio → ChangeBatcher → Plugin → WebSocket → Server → Filesystem
|
|
```
|
|
|
|
## HTTP Endpoints (`src/web/api.rs`)
|
|
|
|
| Endpoint | Method | Purpose |
|
|
|----------|--------|---------|
|
|
| `/api/rojo` | GET | Server info (version, sessionId, projectName, rootInstanceId, servePlaceIds) |
|
|
| `/api/read/:id` | GET | Read instance tree from ID |
|
|
| `/api/serialize/:id` | GET | Instance tree as msgpack |
|
|
| `/api/write` | POST | Two-way sync: plugin sends changes |
|
|
| `/api/open/:id` | POST | Open script in editor (`{"cursor": [line, col]}`) |
|
|
| `/api/ref-patch/:id` | GET | Fix Ref properties after syncback |
|
|
| `/api/syncback` | POST | Live syncback: full Roblox → filesystem sync |
|
|
| `/api/socket/:cursor` | GET | WebSocket upgrade for live patch streaming |
|
|
|
|
### Server Info Response (`GET /api/rojo`)
|
|
|
|
```json
|
|
{
|
|
"serverVersion": "7.7.0-rc.1",
|
|
"serverFork": "atlas",
|
|
"protocolVersion": 6,
|
|
"sessionId": "uuid",
|
|
"projectName": "MyProject",
|
|
"expectedPlaceIds": [123456],
|
|
"unexpectedPlaceIds": [111111],
|
|
"rootInstanceId": "00000000-0000-0000-0000-000000000000",
|
|
"syncSourceOnly": true,
|
|
"visibleServices": ["Workspace", "ServerScriptService"],
|
|
"placeId": 123456,
|
|
"gameId": 789012,
|
|
"ignoreHiddenServices": true,
|
|
"gitMetadata": {
|
|
"changedIds": ["ref-1", "ref-2"],
|
|
"scriptCommittedHashes": {
|
|
"ref-1": ["a1b2c3...40-char-sha1-hex", "d4e5f6...staged-sha1"]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**`gitMetadata`** is `null` when the project is not in a git repo. Contains:
|
|
- `changedIds`: Instance Refs whose backing files have changes relative to HEAD (staged + unstaged + untracked)
|
|
- `scriptCommittedHashes`: For changed script instances, 1-2 SHA1 hashes (HEAD version and staged version if different) computed using the git blob format (`"blob <size>\0<content>"`). Plugin checks if Studio `script.Source` matches ANY hash to default "Atlas".
|
|
|
|
Recomputed fresh on each `GET /api/rojo` connect. Uses a two-phase tree lock: snapshot path mappings briefly, then release before running git subprocesses.
|
|
|
|
### Write Request (`POST /api/write`)
|
|
|
|
```json
|
|
{
|
|
"sessionId": "uuid",
|
|
"removed": ["id-1"],
|
|
"added": { "id-2": { "parent": "parent-id", "className": "Folder", "name": "New", "properties": {}, "children": [] } },
|
|
"updated": [{ "id": "id-3", "changedName": "Renamed", "changedClassName": "Script", "changedProperties": { "Source": "print('hi')" }, "changedMetadata": {} }],
|
|
"stageIds": ["ref-1", "ref-2"]
|
|
}
|
|
```
|
|
|
|
**`stageIds`**: Instance IDs whose backing files should be staged via `git add` after writing. Includes all push-accepted IDs + auto-selected pull-accepted IDs. The server resolves these to filesystem paths and runs `git add`. Staging is split: `api.rs` stages additions/removals/push files directly; `change_processor` stages Source writes after they complete (via `stage_ids` on PatchSet).
|
|
|
|
**Format detection:** `Name/init.luau` → directory | `Name.luau` → file | neither → use `has_children` (`.lua` also supported)
|
|
|
|
### Live Syncback Endpoint (`POST /api/syncback`)
|
|
|
|
One-click full sync from Studio to filesystem. Does **not** require an active WebSocket connection -- the plugin POSTs directly from the NotConnected page using the host/port fields.
|
|
|
|
**Request (`SyncbackRequest`):**
|
|
|
|
```rust
|
|
pub struct SyncbackRequest {
|
|
pub protocol_version: f64, // f64 because Luau has no integer type
|
|
pub server_version: String,
|
|
pub place_id: Option<f64>,
|
|
pub data: Vec<u8>, // Single rbxm blob containing ALL children from ALL services
|
|
pub services: Vec<ServiceChunk>,
|
|
}
|
|
```
|
|
|
|
**`ServiceChunk`:**
|
|
|
|
```rust
|
|
pub struct ServiceChunk {
|
|
pub class_name: String,
|
|
pub child_count: u32,
|
|
pub ref_target_count: u32,
|
|
pub properties: HashMap<String, Variant>, // Service-level properties (omitted if empty)
|
|
pub refs: HashMap<String, u32>, // Ref property name → 1-based carrier index (omitted if empty)
|
|
}
|
|
```
|
|
|
|
**Data format:** The `data` field is a single rbxm binary blob containing all children from all services concatenated. Each service's range in the blob is `child_count + ref_target_count` consecutive entries. First `child_count` entries are real children (parented to service). Next `ref_target_count` entries are temporary ObjectValue carriers encoding Ref properties (destroyed after resolving). The `refs` map connects property names to carrier indices.
|
|
|
|
**Validation:**
|
|
- Protocol version must match `PROTOCOL_VERSION` (6) → 400 Bad Request
|
|
- Server version MAJOR.MINOR must match → 400 Bad Request
|
|
- Place ID must pass `servePlaceIds`/`blockedPlaceIds` checks if configured → 403 Forbidden
|
|
|
|
**Response:** `{"status": "syncback_initiated"}`
|
|
|
|
**Server-side types:**
|
|
|
|
```rust
|
|
pub struct SyncbackPayload {
|
|
pub data: Vec<u8>,
|
|
pub services: Vec<ServiceChunk>,
|
|
}
|
|
|
|
pub enum ServerExitReason {
|
|
SyncbackRequested(SyncbackPayload),
|
|
}
|
|
```
|
|
|
|
**`SyncbackSignal` (`src/web/mod.rs`):** Thread-safe signal for API handler → serve loop communication. Uses `Mutex<Option<SyncbackPayload>>` + `Notify`. `fire()` deposits payload and notifies; `take_payload()` retrieves it. `LiveServer::start()` uses `tokio::select!` to wait for either new connections OR the syncback signal, exiting the accept loop when fired.
|
|
|
|
### Live Syncback Data Flow
|
|
|
|
```
|
|
Plugin (NotConnected page)
|
|
│ User clicks "Syncback" button → SyncbackConfirm dialog
|
|
│ User confirms → performSyncback()
|
|
│ Encodes services via encodeService() + serializes children via SerializationService
|
|
▼
|
|
HTTP POST /api/syncback (MessagePack body)
|
|
│
|
|
▼
|
|
handle_api_syncback() [src/web/api.rs]
|
|
│ Validates protocol, version, place ID
|
|
│ Fires SyncbackSignal
|
|
▼
|
|
LiveServer::start() receives notification [src/web/mod.rs]
|
|
│ Exits accept loop
|
|
│ Returns ServerExitReason::SyncbackRequested(payload)
|
|
▼
|
|
run_live_syncback() [src/cli/serve.rs]
|
|
│ build_dom_from_chunks(payload) → reconstructs DOM from rbxm + service chunks
|
|
│ syncback_loop(incremental = false) → clean mode, full filesystem rewrite
|
|
│ write_to_vfs_parallel() → writes files
|
|
│ refresh_git_index() → updates git index
|
|
▼
|
|
Serve loop restarts (new session, ready for connections)
|
|
```
|
|
|
|
### Live Syncback vs Two-Way Sync
|
|
|
|
| Aspect | Live Syncback (`/api/syncback`) | Two-Way Sync (`/api/write`) |
|
|
|--------|--------------------------------|---------------------------|
|
|
| **Purpose** | Full Roblox → filesystem sync | Incremental property/instance changes |
|
|
| **Scope** | Entire instance tree (all services) | Individual instance changes |
|
|
| **Mode** | Clean mode (replaces everything) | Incremental (add/update/remove) |
|
|
| **Server restart** | Yes (restarts serve loop) | No (continues serving) |
|
|
| **Data format** | Single rbxm blob + service chunks | Patch-based (added/updated/removed) |
|
|
| **Connection** | No WebSocket needed (HTTP POST) | Requires active WebSocket session |
|
|
| **When used** | User-initiated one-click syncback | Real-time bidirectional sync |
|
|
| **Session** | Creates new session after restart | Uses existing live session |
|
|
|
|
## WebSocket Protocol (v6)
|
|
|
|
**Connection:** `ws://host:port/api/socket/:cursor`
|
|
|
|
1. Connect → Receive `ServerInfo` handshake → Send `{type = "Subscribe"}` → Receive patches
|
|
|
|
### Patch Message (`ApplyPatch`)
|
|
|
|
```lua
|
|
{
|
|
type = "ApplyPatch",
|
|
sessionId = "uuid",
|
|
patch = {
|
|
removed = {"id-1", "id-2"},
|
|
added = {
|
|
["id-3"] = {
|
|
id = "id-3", className = "Folder", name = "New",
|
|
properties = {}, metadata = { ignoreUnknownInstances = false },
|
|
children = {"child-1"},
|
|
},
|
|
},
|
|
updated = {
|
|
{ id = "id-4", changedName = "Renamed", changedProperties = { Source = "code" } },
|
|
},
|
|
}
|
|
}
|
|
```
|
|
|
|
**Format:** MessagePack (~30% smaller than JSON, ~2x faster)
|
|
|
|
## Server ServeSession (`src/serve_session.rs`)
|
|
|
|
```rust
|
|
pub struct ServeSession {
|
|
session_id: SessionId,
|
|
tree: Arc<Mutex<RojoTree>>,
|
|
message_queue: Arc<MessageQueue<AppliedPatchSet>>,
|
|
root_project: Project,
|
|
vfs: Arc<Vfs>,
|
|
change_processor: ChangeProcessor,
|
|
start_time: Instant,
|
|
tree_mutation_sender: Sender<PatchSet>,
|
|
suppressed_paths: Arc<Mutex<HashMap<PathBuf, (usize, usize)>>>,
|
|
}
|
|
```
|
|
|
|
**Lifecycle:** Create → Watch filesystem → Generate patches on changes → Broadcast to plugins → Shutdown
|
|
|
|
**Message Queue:** Batches changes (~100ms window), cursor system for reconnect recovery
|
|
|
|
**WebSocket loop:**
|
|
```rust
|
|
loop {
|
|
select! {
|
|
msg = receiver.next() => handle_plugin_message(msg),
|
|
patch = message_queue.recv() => sender.send(patch).await,
|
|
}
|
|
}
|
|
```
|
|
|
|
## Plugin ServeSession (`plugin/src/ServeSession.lua`)
|
|
|
|
```lua
|
|
ServeSession = {
|
|
__status = "NotStarted", -- NotStarted → Connecting → Connected → Disconnected
|
|
__apiContext = ApiContext,
|
|
__twoWaySync = false,
|
|
__reconciler = Reconciler,
|
|
__instanceMap = InstanceMap,
|
|
__changeBatcher = ChangeBatcher,
|
|
__confirmingPatch = nil,
|
|
__isConfirming = false, -- Prevents stacking dialogs, changes during review
|
|
}
|
|
```
|
|
|
|
### Patch Confirmation
|
|
|
|
**Setting `confirmationBehavior`:** `"Always"` (default) | `"Initial"` | `"Large Changes"` | `"Unlisted PlaceId"` | `"Never"`
|
|
|
|
**Flow:** Receive patch → Check setting → Show PatchVisualizer (pause ChangeBatcher) → User confirms → Apply via Reconciler → Resume ChangeBatcher
|
|
|
|
### Patch Application
|
|
|
|
```lua
|
|
-- Remove
|
|
for _, id in patch.removed do instanceMap:fromId(id):Destroy(); instanceMap:removeId(id) end
|
|
-- Add
|
|
for id, data in patch.added do instanceMap:insert(id, Reconciler.reify(data)) end
|
|
-- Update
|
|
for _, update in patch.updated do Reconciler.hydrate(instanceMap:fromId(update.id), update) end
|
|
```
|
|
|
|
## Two-Way Sync
|
|
|
|
**Setting:** `twoWaySync` (default: true) | Syncs Studio changes → filesystem in real-time
|
|
|
|
### Complete Data Flow
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ PLUGIN SIDE │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ Studio Instance │
|
|
│ │ PropertyChanged / DescendantAdded / etc. │
|
|
│ ▼ │
|
|
│ InstanceMap.__connectSignals() │
|
|
│ │ Fires onInstanceChanged(instance, propertyName) │
|
|
│ │ (skipped if RunService:IsRunning() or instance is paused) │
|
|
│ ▼ │
|
|
│ ServeSession.onInstanceChanged() │
|
|
│ │ Guards: if not self.__twoWaySync then return end │
|
|
│ ▼ │
|
|
│ ChangeBatcher:add(instance, propertyName) │
|
|
│ │ Stores in __pendingPropertyChanges[instance][propertyName] = true │
|
|
│ ▼ │
|
|
│ ChangeBatcher:__cycle(dt) [RenderStepped, every frame] │
|
|
│ │ Accumulates dt until >= BATCH_INTERVAL (0.2s) │
|
|
│ │ Guards: if __paused then return end │
|
|
│ ▼ │
|
|
│ ChangeBatcher:__flush() │
|
|
│ │ Calls createPatchSet(instanceMap, pendingChanges, syncSourceOnly) │
|
|
│ │ Clears __pendingPropertyChanges │
|
|
│ ▼ │
|
|
│ onChangesFlushed(patch) │
|
|
│ │ Guards: if Settings:get("oneShotSync") then return end │
|
|
│ ▼ │
|
|
│ ApiContext:write(patch) → HTTP POST /api/write (MessagePack) │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
│
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────────────┐
|
|
│ SERVER SIDE │
|
|
├─────────────────────────────────────────────────────────────────────────────┤
|
|
│ │
|
|
│ handle_api_write() [src/web/api.rs] │
|
|
│ │ Validates session_id │
|
|
│ │ Deserializes MessagePack → WriteRequest │
|
|
│ ▼ │
|
|
│ For each removed: syncback_removed_instance() │
|
|
│ │ Looks up InstigatingSource::Path → fs::remove_file() │
|
|
│ ▼ │
|
|
│ For each added: syncback_added_instance() │
|
|
│ │ Groups by parent, pre-seeds sibling_slugs, dedup on collision │
|
|
│ │ Determines file format, writes to filesystem │
|
|
│ ▼ │
|
|
│ Build added_paths map (for same-batch Ref resolution) │
|
|
│ │ Maps Ref → filesystem path via added_instance_fs_segment() │
|
|
│ ▼ │
|
|
│ For each updated: syncback_updated_properties() │
|
|
│ │ Extracts Ref props → Rojo_Ref_* attrs (path from tree/added_paths) │
|
|
│ │ merge_or_build_meta() writes props/attrs (supports attr removal) │
|
|
│ ▼ │
|
|
│ Sends PatchSet to tree_mutation_sender channel │
|
|
│ ▼ │
|
|
│ ChangeProcessor::handle_tree_event() [src/change_processor.rs] │
|
|
│ │ For removed: dedup cleanup (gap-tolerant, base-name promotion) │
|
|
│ │ For updated: Source writes, renames, ClassName changes, ref paths │
|
|
│ ▼ │
|
|
│ apply_patch_set() → Updates RojoTree │
|
|
│ ▼ │
|
|
│ message_queue.push_messages() → Broadcasts to all plugins │
|
|
│ │
|
|
└─────────────────────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
### InstanceMap: Change Detection Foundation (`plugin/src/InstanceMap.lua`)
|
|
|
|
The InstanceMap is a bidirectional map (ID ↔ Instance) that also hooks property change signals:
|
|
|
|
```lua
|
|
InstanceMap = {
|
|
fromIds = {}, -- Server ID → Studio Instance
|
|
fromInstances = {}, -- Studio Instance → Server ID
|
|
pausedUpdateInstances = {}, -- Set of instances to ignore changes from
|
|
instancesToSignal = {}, -- Instance → RBXScriptConnection(s)
|
|
onInstanceChanged = fn, -- Callback: (instance, propertyName) → void
|
|
}
|
|
```
|
|
|
|
**Signal Connection:**
|
|
```lua
|
|
function InstanceMap:__connectSignals(instance)
|
|
if instance:IsA("ValueBase") then
|
|
-- ValueBase.Changed only fires for Value, need explicit signals
|
|
signals = {
|
|
instance:GetPropertyChangedSignal("Name"):Connect(...),
|
|
instance:GetPropertyChangedSignal("Value"):Connect(...),
|
|
instance:GetPropertyChangedSignal("Parent"):Connect(...),
|
|
}
|
|
else
|
|
-- Regular instances use Changed event (fires for any property)
|
|
instance.Changed:Connect(function(propertyName)
|
|
self:__maybeFireInstanceChanged(instance, propertyName)
|
|
end)
|
|
end
|
|
end
|
|
```
|
|
|
|
**Change Guards:**
|
|
```lua
|
|
function InstanceMap:__maybeFireInstanceChanged(instance, propertyName)
|
|
if self.pausedUpdateInstances[instance] then return end -- Paused during reconciliation
|
|
if self.onInstanceChanged == nil then return end -- No callback registered
|
|
if RunService:IsRunning() then return end -- Don't sync during Play mode
|
|
self.onInstanceChanged(instance, propertyName)
|
|
end
|
|
```
|
|
|
|
**Pause Mechanism:** When applying incoming patches, instances are paused to prevent their changes from being sent back to the server (feedback loop prevention).
|
|
|
|
### ChangeBatcher (`plugin/src/ChangeBatcher/`)
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `init.lua` | Main batcher: accumulator, RenderStepped loop, pause/resume, deferred refs |
|
|
| `createPatchSet.lua` | Converts pending changes → PatchSet (passes instanceMap for Ref resolution) |
|
|
| `encodePatchUpdate.lua` | Encodes property updates including Ref properties via InstanceMap |
|
|
| `encodeInstance.lua` | Full instance encoding (for additions/syncback); uses `encodeHelpers` for property iteration; includes duplicate detection helpers (`findDuplicateNames`, `hasDuplicateSiblings`, `isPathUnique`) |
|
|
| `encodeHelpers.lua` | Shared encoding helpers: `encodeAttributes`, `encodeTags`, `forEachEncodableProperty`. Used by `encodeInstance.lua` and `encodeService.lua`. Defines `SKIP_PROPERTIES` (Parent, Name, Archivable). |
|
|
| `encodeService.lua` | Encodes service-level properties/attributes/refs for live syncback; uses ObjectValue carriers for Ref properties |
|
|
| `encodeProperty.lua` | Single property value encoding via RbxDom |
|
|
| `propertyFilter.lua` | Property type filtering (only `UniqueId` blocked; `Ref` is allowed) |
|
|
|
|
**Core State:**
|
|
```lua
|
|
ChangeBatcher = {
|
|
__accumulator = 0, -- Time since last flush
|
|
__pendingPropertyChanges = {}, -- Instance → { PropertyName → true }
|
|
__pendingUnresolvedRefs = {}, -- targetInstance → [{sourceInstance, propertyName}]
|
|
__syncSourceOnly = false, -- Only sync Source property
|
|
__paused = false, -- Pause during confirmation
|
|
__renderSteppedConnection = conn, -- Frame loop connection
|
|
__instanceMap = instanceMap,
|
|
__onChangesFlushed = callback,
|
|
}
|
|
```
|
|
|
|
**Batch Cycle (every frame):**
|
|
```lua
|
|
local BATCH_INTERVAL = 0.2 -- 200ms
|
|
|
|
function ChangeBatcher:__cycle(dt)
|
|
if self.__paused then return end
|
|
|
|
self.__accumulator += dt
|
|
if self.__accumulator >= BATCH_INTERVAL then
|
|
self.__accumulator -= BATCH_INTERVAL
|
|
local patch = self:__flush()
|
|
if patch then
|
|
self.__onChangesFlushed(patch)
|
|
end
|
|
end
|
|
|
|
self.__instanceMap:unpauseAllInstances() -- Resume paused instances
|
|
end
|
|
```
|
|
|
|
**Why 200ms batch interval?**
|
|
- Studio fires many rapid property changes (e.g., dragging a part fires Position.X, Y, Z separately)
|
|
- Batching combines related changes into single patch
|
|
- Reduces server load and WebSocket traffic
|
|
- Still feels responsive to users
|
|
|
|
**Deferred Ref Mechanism:**
|
|
|
|
When a Ref property targets an instance not yet in the InstanceMap (e.g., newly pulled instance whose server ID hasn't arrived), the property is deferred:
|
|
|
|
```lua
|
|
-- In encodePatchUpdate: onUnresolvedRef callback defers the property
|
|
-- In ChangeBatcher: __pendingUnresolvedRefs[targetInstance] = {sourceInstance, propertyName}
|
|
-- When target appears (onInstanceInserted), re-queues the property change for next batch
|
|
```
|
|
|
|
### createPatchSet.lua: Patch Generation
|
|
|
|
```lua
|
|
function createPatchSet(instanceMap, propertyChanges, syncSourceOnly, onUnresolvedRef)
|
|
local patch = PatchSet.newEmpty()
|
|
|
|
for instance, properties in propertyChanges do
|
|
local instanceId = instanceMap.fromInstances[instance]
|
|
if instanceId == nil then
|
|
Log.warn("Ignoring change for unknown instance")
|
|
continue
|
|
end
|
|
|
|
if properties.Parent then
|
|
if instance.Parent == nil then
|
|
table.insert(patch.removed, instanceId)
|
|
else
|
|
Log.warn("Cannot sync non-nil Parent changes yet")
|
|
end
|
|
else
|
|
if syncSourceOnly then
|
|
if properties.Source then
|
|
local update = encodePatchUpdate(instance, instanceId, { Source = true }, instanceMap, onUnresolvedRef)
|
|
if update then table.insert(patch.updated, update) end
|
|
end
|
|
else
|
|
local update = encodePatchUpdate(instance, instanceId, properties, instanceMap, onUnresolvedRef)
|
|
if update then table.insert(patch.updated, update) end
|
|
end
|
|
end
|
|
end
|
|
|
|
return patch
|
|
end
|
|
```
|
|
|
|
### encodePatchUpdate.lua: Update Encoding
|
|
|
|
Encodes property updates for existing instances. Now handles Ref properties via InstanceMap lookup.
|
|
|
|
```lua
|
|
function encodePatchUpdate(instance, instanceId, properties, instanceMap, onUnresolvedRef)
|
|
local update = {
|
|
id = instanceId,
|
|
changedProperties = {},
|
|
}
|
|
|
|
for propertyName in properties do
|
|
if propertyName == "Name" then
|
|
update.changedName = instance.Name
|
|
else
|
|
local descriptor = RbxDom.findCanonicalPropertyDescriptor(instance.ClassName, propertyName)
|
|
if not descriptor then continue end
|
|
|
|
if descriptor.dataType == "Ref" then
|
|
-- Ref property: resolve target via InstanceMap
|
|
local target = instance[propertyName]
|
|
if target == nil then
|
|
update.changedProperties[propertyName] = { Ref = NULL_REF }
|
|
else
|
|
local targetId = instanceMap.fromInstances[target]
|
|
if targetId then
|
|
update.changedProperties[propertyName] = { Ref = targetId }
|
|
elseif onUnresolvedRef then
|
|
onUnresolvedRef(target, instance, propertyName)
|
|
end
|
|
end
|
|
else
|
|
local success, encoded = encodeProperty(instance, propertyName, descriptor)
|
|
if success then
|
|
update.changedProperties[propertyName] = encoded
|
|
end
|
|
end
|
|
end
|
|
end
|
|
|
|
if next(update.changedProperties) == nil and update.changedName == nil then
|
|
return nil
|
|
end
|
|
|
|
return update
|
|
end
|
|
```
|
|
|
|
### encodeInstance.lua: Full Instance Encoding (Syncback/Additions)
|
|
|
|
Used when pulling new instances from Studio to filesystem. Uses `encodeHelpers.forEachEncodableProperty()` for property iteration (shared with `encodeService.lua`). Duplicate-named instances are encoded and sent to the server; the server handles filesystem collisions via the dedup suffix system (`~N`). Includes duplicate detection helpers (`findDuplicateNames`, `hasDuplicateSiblings`, `isPathUnique`).
|
|
|
|
```lua
|
|
function encodeInstance(instance, parentId, _skipPathCheck)
|
|
local properties = {}
|
|
|
|
Helpers.encodeAttributes(instance, properties)
|
|
Helpers.encodeTags(instance, properties)
|
|
|
|
if SCRIPT_CLASS_NAMES[instance.ClassName] then
|
|
local success, encoded = encodeProperty(instance, "Source", descriptor)
|
|
if not success then return nil end
|
|
properties.Source = encoded
|
|
end
|
|
|
|
Helpers.forEachEncodableProperty(instance.ClassName, SKIP_PROPERTIES_BY_CLASS, function(propertyName, descriptor)
|
|
local success, encoded = encodeProperty(instance, propertyName, descriptor)
|
|
if success and encoded ~= nil then
|
|
properties[propertyName] = encoded
|
|
end
|
|
end)
|
|
|
|
local children = {}
|
|
for _, child in instance:GetChildren() do
|
|
local encoded = encodeInstance(child, nil, true)
|
|
if encoded then table.insert(children, encoded) end
|
|
end
|
|
|
|
return {
|
|
parent = parentId,
|
|
name = instance.Name,
|
|
className = instance.ClassName,
|
|
properties = properties,
|
|
children = children,
|
|
}
|
|
end
|
|
```
|
|
|
|
**Properties skipped per class:**
|
|
| Class | Skipped Properties | Reason |
|
|
|-------|-------------------|--------|
|
|
| Script/LocalScript/ModuleScript | Source, ScriptGuid, RunContext | Encoded in file contents/suffix |
|
|
| LocalizationTable | Contents | Encoded in .csv file |
|
|
| StringValue | Value | Encoded in .txt file |
|
|
|
|
### encodeProperty.lua: Value Encoding
|
|
|
|
```lua
|
|
function encodeProperty(instance, propertyName, descriptor)
|
|
local readSuccess, value = descriptor:read(instance)
|
|
if not readSuccess then return false, nil end
|
|
|
|
local dataType = descriptor.dataType
|
|
local encodeSuccess, encoded = RbxDom.EncodedValue.encode(value, dataType)
|
|
if not encodeSuccess then return false, nil end
|
|
|
|
return true, encoded
|
|
end
|
|
```
|
|
|
|
**Encoded value format (via RbxDom):**
|
|
```lua
|
|
-- Primitives
|
|
true, false, 42, "string"
|
|
|
|
-- Roblox types
|
|
{ Vector3 = {1, 2, 3} }
|
|
{ CFrame = { position = {0,0,0}, orientation = {...} } }
|
|
{ UDim2 = {{0, 100}, {0, 50}} }
|
|
{ Color3 = {1, 0.5, 0} }
|
|
{ BrickColor = 194 }
|
|
{ Ref = "server-instance-id" } -- Object references
|
|
{ Attributes = { key = value, ... } }
|
|
{ Tags = {"tag1", "tag2"} }
|
|
```
|
|
|
|
### Server-Side: ChangeProcessor (`src/change_processor.rs`)
|
|
|
|
Receives patches from plugin via `tree_mutation_receiver` channel. Handles removals (with dedup cleanup), Source writes, renames (with slugify/dedup/meta lifecycle), ClassName changes (script type transitions via file renames), property writes to meta/model files, and ref path updates.
|
|
|
|
**Capabilities:**
|
|
- **Removals:** Deletes file + associated meta file via `InstigatingSource::Path`
|
|
- **Dedup cleanup after removals:** When a deletion leaves a dedup group needing suffix cleanup, performs renames (gap-tolerant, base-name promotion, group-to-1). Uses `removed_set` (HashSet) to exclude co-removed siblings from sibling enumeration. Calls `compute_cleanup_action()` from `dedup_suffix.rs`.
|
|
- **Source writes:** Updates script file contents
|
|
- **Renames:** Slugifies new name, deduplicates against siblings, renames file on disk, creates/updates/removes meta `name` field as needed. Handles both standalone files and init-file directories.
|
|
- **ClassName changes:** Script-to-script transitions (e.g. ModuleScript → Script) via file extension renames
|
|
- **Property writes:** Attributes and other properties written to meta/model files via `syncback_updated_properties` in `api.rs`
|
|
- **Ref path updates:** `RefPathIndex` tracks which meta/model files contain `Rojo_Ref_*` paths. `update_ref_paths_after_rename()` efficiently updates all affected files when an instance is renamed or dedup-cleaned.
|
|
- **VFS suppression for dedup renames:** `suppress_path_any(&from)` + `suppress_path(&to)` prevents the VFS watcher from re-triggering on files renamed during dedup cleanup.
|
|
- **InstigatingSource updates:** After dedup cleanup renames, the renamed sibling's `InstigatingSource::Path` and metadata path indexes are updated in-memory.
|
|
|
|
**InstigatingSource:** Every instance in RojoTree tracks where it came from:
|
|
- `InstigatingSource::Path(PathBuf)` - Created from a file, can be written back
|
|
- `InstigatingSource::ProjectNode { path, name, node, parent_class }` - Defined in project file, **cannot be modified via two-way sync**
|
|
|
|
**`handle_tree_event()` flow:**
|
|
1. Process removals (log but don't delete -- API already deleted files)
|
|
2. Dedup cleanup (runs before `apply_patch_set` to use parent/sibling relationships)
|
|
3. Process updates: renames, ClassName changes, Source writes, ref path updates
|
|
4. `apply_patch_set()` to update RojoTree
|
|
5. Apply metadata updates (InstigatingSource paths for both regular and dedup renames)
|
|
6. Send applied patch to message queue for plugin broadcast
|
|
|
|
### Pause Mechanism: Critical for Consistency
|
|
|
|
**Why pause is essential:**
|
|
1. **During incoming patch confirmation:** User is reviewing changes. If ChangeBatcher sends Studio changes mid-review, it creates race conditions.
|
|
2. **During patch application:** Reconciler modifies instances, triggering property change events. Without pause, these would be sent back as "changes" (feedback loop).
|
|
3. **Defense-in-depth:** `__isConfirming` flag in ServeSession prevents multiple dialogs and ensures atomicity.
|
|
|
|
```lua
|
|
-- In ServeSession, when showing confirmation dialog:
|
|
self.__changeBatcher:pause()
|
|
-- ... show PatchVisualizer, wait for user ...
|
|
self.__changeBatcher:resume()
|
|
```
|
|
|
|
```lua
|
|
-- In Reconciler, when modifying instances:
|
|
instanceMap:pauseInstance(instance)
|
|
instance.SomeProperty = newValue -- Won't trigger ChangeBatcher
|
|
-- Later, in ChangeBatcher:__cycle():
|
|
instanceMap:unpauseAllInstances()
|
|
```
|
|
|
|
### One-Shot Mode Integration
|
|
|
|
**Setting:** `oneShotSync` (default: true)
|
|
|
|
One-shot mode adds an extra guard in `onChangesFlushed`:
|
|
|
|
```lua
|
|
local function onChangesFlushed(patch)
|
|
if Settings:get("oneShotSync") then
|
|
Log.info("One-shot mode: blocking automatic outgoing write from ChangeBatcher")
|
|
return -- Silently discard patch
|
|
end
|
|
self.__apiContext:write(patch)
|
|
end
|
|
```
|
|
|
|
**Behavior:**
|
|
- ChangeBatcher still collects changes (needed for diff visualization)
|
|
- Automatic writes are blocked
|
|
- Only explicit "pull" selections in confirmation dialog can write
|
|
- Disconnects after initial sync completes
|
|
- No sync lock acquired (multiple users can view same place)
|
|
|
|
**Use cases:**
|
|
- Quick preview of changes without risk of accidental writes
|
|
- Safe for production environments
|
|
- Team viewing same project simultaneously
|
|
|
|
### Git-Based Sync Direction Defaults
|
|
|
|
When the plugin shows the sync confirmation dialog, items get smart default selections based on git status:
|
|
|
|
- **File has no git changes**: default to "Studio" (pull) -- any diff is because Studio was changed
|
|
- **File has git changes AND is a script AND Studio Source matches committed/staged hash**: default to "Atlas" (push) -- developer changed the file, Studio hasn't diverged
|
|
- **Otherwise**: `nil` -- user must decide (both sides changed, or non-script with changes)
|
|
- **No git repo**: all defaults remain `nil` (feature is no-op)
|
|
|
|
Only **Updated** items get defaults. Added/Removed items remain `nil`.
|
|
|
|
**Hash computation**: Both sides use `SHA1("blob <byte_len>\0<content>")` (git blob format). Server sends up to 2 hashes per changed script (HEAD + staged if different). Plugin computes hash of `script.Source` (non-draft version) and checks if ANY hash matches.
|
|
|
|
**Auto-staging**: After confirmation, accepted files are staged via `git add`:
|
|
- Push-accepted items: always staged (all types, including non-scripts)
|
|
- Pull-accepted items: only staged if auto-selected (`defaultSelection ~= nil`); manually-chosen pulls are left unstaged for manual diff
|
|
|
|
**Implementation files**: `src/git.rs` (server git helpers), `plugin/src/PatchTree.lua` (default selection logic), `plugin/src/SHA1.luau` (hash computation), `src/web/interface.rs` (`GitMetadata` struct, `stage_ids` on WriteRequest/PatchSet)
|
|
|
|
### Known Limitations (Detailed)
|
|
|
|
| Issue | Cause | Impact |
|
|
|-------|-------|--------|
|
|
| **Undo/redo triggers spurious changes** | Studio's ChangeHistoryService fires property changes during undo | Can cause unexpected writes or conflicts |
|
|
| **Rapid changes interleave** | 200ms batch window may split logically related changes | Partial updates can leave inconsistent state |
|
|
| **Reparenting not supported** | `Parent` property changes to non-nil values are ignored | Moving instances in Studio doesn't sync |
|
|
| **Project node instances immutable** | InstigatingSource::ProjectNode cannot be modified | Instances from project file can't be changed |
|
|
| **Greedy matching not globally optimal** | Ambiguous groups (same Name+ClassName) use greedy assignment | Rare edge cases may produce suboptimal pairing and rename churn |
|
|
| **Ref paths with ancestor-level ambiguity** | `Rojo_Ref_*` path-based refs written to disk may resolve to wrong sibling if duplicates exist at any ancestor level | Ref may point to wrong instance after rebuild; in-memory tree is correct during session |
|
|
|
|
### Reconciler Matching Algorithm (`plugin/src/Reconciler/`)
|
|
|
|
The plugin uses a recursive change-count scoring algorithm (implemented in `matching.lua`) to pair incoming virtual instances with existing Studio instances during hydration.
|
|
|
|
**Entry point:** `Matching.matchChildren(virtualChildren, studioChildren, virtualInstances)`
|
|
|
|
**Algorithm:**
|
|
1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring.
|
|
2. **Ambiguous groups:** For groups with multiple candidates, compute pairwise `computePairCost()` (recursive, scores entire subtree). Greedy assignment sorted by cost ascending, child-order tiebreaker.
|
|
3. **Leftovers:** Unmatched virtual = new instances, unmatched Studio = deleted.
|
|
|
|
**Performance:** File-level `--!native` and `--!optimize 2` directives for native codegen. `MatchingSession` caches scoring results across recursive calls (`matchCache` for match results, `costCache` for pair costs), avoiding redundant computation when hydrate re-calls `matchChildren` on the same children. `cacheVirtual()` filters children to only those existing in `virtualInstances`, preventing crashes on incomplete virtual instance trees.
|
|
|
|
**Value equality:** Shared `trueEquals.lua` module used by both `matching.lua` and `diff.lua` (fuzzy equality for floats with epsilon 0.0001, Color3, CFrame, Vector3; NaN handling; nil/null-ref equivalence).
|
|
|
|
## Protocol Enforcement
|
|
|
|
```rust
|
|
// Server
|
|
if client_protocol != PROTOCOL_VERSION { return error("Protocol mismatch") }
|
|
```
|
|
|
|
```lua
|
|
-- Plugin
|
|
if serverInfo.protocolVersion ~= Config.protocolVersion then showError("Protocol mismatch") end
|
|
```
|
|
|
|
**Breaking changes:** Bump `protocolVersion` in `src/web/interface.rs` and `plugin/src/Config.lua`
|
|
|
|
## Session Management
|
|
|
|
1. Server starts → Generate UUID
|
|
2. Plugin connects → Receives session ID in handshake
|
|
3. All requests include session ID
|
|
4. Server restart → New session ID → Plugin auto-reconnects on mismatch
|
|
|
|
## Debugging
|
|
|
|
```bash
|
|
# Server
|
|
RUST_LOG=rojo::web=debug,rojo::serve_session=debug atlas serve
|
|
```
|
|
|
|
```lua
|
|
-- Plugin
|
|
Settings:set("logLevel", "Trace")
|
|
```
|
|
|
|
Browser: `http://localhost:34872` → Network tab → WebSocket messages
|
|
|
|
## Key Files
|
|
|
|
| Server | Plugin |
|
|
|--------|--------|
|
|
| `src/web/api.rs` - Endpoints | `plugin/src/ServeSession.lua` - Connection |
|
|
| `src/web/interface.rs` - Packets, syncback types | `plugin/src/ApiContext.lua` - HTTP/WS client |
|
|
| `src/web/mod.rs` - SyncbackSignal, LiveServer | `plugin/src/ChangeBatcher/` - Two-way sync |
|
|
| `src/serve_session.rs` - Session | `plugin/src/ChangeBatcher/encodeService.lua` - Live syncback encoding |
|
|
| `src/cli/serve.rs` - Live syncback handler | `plugin/src/Reconciler/` - Patch application |
|
|
| `src/change_processor.rs` - FS writer | `plugin/src/Reconciler/matching.lua` - Matching |
|
|
| `src/rojo_ref.rs` - Ref paths | `plugin/src/PatchTree.lua` - Default selections |
|
|
| `src/git.rs` - Git integration | `plugin/src/App/init.lua` - Live syncback UI |
|