hakowork/rojo
3
0
Fork 0
mirror of https://github.com/rojo-rbx/rojo.git synced 2026-10-23 05:51:39 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add matching session caching and native codegen improvements

Browse source 
- Introduced a new matching session caching mechanism in the Lua matching module to enhance performance with native code generation.
- Rewrote the Lua matching algorithm to utilize session-based caching, reducing redundant computations and improving efficiency.
- Updated Rust implementations to include session caching, ensuring consistency across both Lua and Rust.
- Enhanced the `hydrate` function to support session management during instance hydration.
- Added comprehensive tests to validate the new caching mechanism and its integration with existing functionality.

These changes significantly improve the performance and maintainability of the matching process in the application.
This commit is contained in:
Joe 2026-02-18 04:55:31 -06:00
parent 307ac81aec
commit b39c01e5e9
14 changed files with 1674 additions and 461 deletions
Download patch file Download diff file Expand all files Collapse all files

609
.cursor/plans/matching_session_caching_cbfa7bda.plan.md Normal file
View file

@ -0,0 +1,609 @@
---
name: Matching session caching
overview: Rewrite the Lua matching module for native codegen performance (--!strict, --!native, --!optimize 2) with session-based caching, clean 2-function mutual recursion, concrete types, and local functions for inlining. Add session caching to both Rust implementations for consistency.
todos:
- id: lua-trueEquals
content: "Rewrite trueEquals.lua: --!strict/--!native/--!optimize 2, single typeof dispatch, dedicated helper functions per type (no temp arrays), adopt DeepEqualsPureUnsafe pattern for table comparison (rawlen/rawget/next, no checkedKeys set), keep fuzzy equality + null-ref + EnumItem cross-type"
status: completed
- id: lua-rewrite
content: "Rewrite matching.lua: --!strict/--!native/--!optimize 2, all functions as local function in dependency order (one forward declare for mutual recursion), 2-function mutual recursion (matchChildren + computePairCost), session caching, totalCost return, O(n) removeMatched, remove pcall in grouping"
status: completed
- id: lua-hydrate-threading
content: Thread session through hydrate.lua, Reconciler/init.lua, and ServeSession.lua (create session before hydrate call at line 628)
status: completed
- id: lua-verify-codegen
content: Run luau-compile --remarks on trueEquals.lua and matching.lua to verify inlining and native codegen quality, iterate until clean
status: completed
- id: lua-test
content: Build plugin, test Lua changes in Studio (hydrate + matching works correctly with session caching) before proceeding to Rust
status: completed
- id: rust-forward-session
content: "AFTER Lua passes: Add MatchingSession to src/snapshot/matching.rs with cost_cache, add total_cost to ForwardMatchResult, thread session through all functions and patch_compute.rs"
status: completed
- id: rust-syncback-session
content: "AFTER Lua passes: Add MatchingSession to src/syncback/matching.rs with cost_cache, add total_cost to MatchResult, thread session through all functions and dir.rs"
status: completed
- id: rust-test-updates
content: Update tests/tests/matching_fixtures.rs, run cargo test to verify Rust changes
status: completed
isProject: false
---
# Matching Session Caching + Native Codegen Rewrite
## Problem
1. **Redundant computation**: Scoring recurses into children, then hydrate re-calls matchChildren on the same children. The intermediate `_computeChildrenCost` calls matchChildren, iterates matched pairs calling `_computeChangeCount`, which recurses again -- all discarded after scoring.
2. **Poor native codegen structure**: All functions are on the `Matching` table (table lookup per call, uninlineable). No type annotations. No `--!strict`/`--!native`/`--!optimize 2`. A `pcall` with closure allocation in the hot grouping loop.
3. **4-function recursion chain** (`matchChildren -> _computeChildrenCost -> _computeChangeCount -> _computeChildrenCost -> matchChildren`) is complex and hard for both the optimizer and the developer to reason about.
## Design
### Conceptual Model
`matchChildren` is fundamentally `computeSubtreeCost` -- it finds the cost-minimizing pairing of children and returns the total cost. The actual match assignments are a bonus output. The 2-function mutual recursion reflects this:
- `**matchChildren(session, vChildren, sChildren, virtualInstances, parentVId, parentSInst)`**: Given two child sets, find optimal pairing. Returns `MatchResult` with `totalCost`.
- `**computePairCost(session, virtualId, studioInstance, virtualInstances, bestSoFar)`**: Given one virtual and one studio instance, compute total cost including subtree. Calls `matchChildren` for children cost.
`_computeChildrenCost` and `_computeChangeCount` are eliminated. Their logic is absorbed into these two functions.
### MatchingSession
```lua
type MatchingSession = {
matchCache: { [string]: { [Instance]: MatchResult } },
costCache: { [string]: { [Instance]: number } },
}
```
- **matchCache**: keyed by `(parentVirtualId, parentStudioInstance)`. Parent uniquely determines children set.
- **costCache**: keyed by `(virtualId, studioInstance)`. Only stored when `cost < bestSoFar` (exact, not early-exited).
Caching replaces the need for a depth limit: each unique pair/children-set is computed at most once.
### bestSoFar Interaction
- **costCache**: only store when computation was complete (`cost < bestSoFar`). If cached, return immediately regardless of caller's `bestSoFar` (the cached value IS the exact cost).
- **matchCache**: `matchChildren` always completes (no `bestSoFar` on it), so results are always cacheable.
---
## Lua Plugin Rewrite: [matching.lua](plugin/src/Reconciler/matching.lua)
### Module Header
```lua
--!strict
--!native
--!optimize 2
```
### Type Definitions (all concrete, minimal `any`)
```lua
type MatchPair = {
virtualId: string,
studioInstance: Instance,
}
type MatchResult = {
matched: { MatchPair },
unmatchedVirtual: { string },
unmatchedStudio: { Instance },
totalCost: number,
}
type MatchingSession = {
matchCache: { [string]: { [Instance]: MatchResult } },
costCache: { [string]: { [Instance]: number } },
}
type VCache = {
props: { [string]: any },
extraProps: { string }?,
tags: { [string]: boolean }?,
attrs: { [string]: any }?,
childCount: number,
}
type SCache = {
instance: Instance,
props: { [string]: any },
tags: { [string]: boolean }?,
attrs: { [string]: any }?,
children: { Instance },
childCount: number,
}
```
### All Functions as `local function` in Dependency Order
Every function is a `local function` defined in dependency order. No `Matching.foo()` table lookups. One forward declaration for the mutual recursion pair.
```lua
local UNMATCHED_PENALTY: number = 10000
local function newSession(): MatchingSession
return { matchCache = {}, costCache = {} }
end
local function cacheVirtual(vInst: any, classKeys: any): VCache
-- same logic as current _cacheVirtual, with return type annotation
end
local function cacheStudio(studioInstance: Instance, classKeys: any, extraProps: { string }?): SCache
-- same logic as current _cacheStudio
-- ADDITION: store sCache.instance = studioInstance
end
local function countOwnDiffs(vCache: VCache, sCache: SCache, classKeys: any): number
-- same logic as current _countOwnDiffsCached
end
local function removeMatched(arr: { any }, matchedIndices: { [number]: boolean }): ()
-- O(n) in-place compaction instead of O(n^2) table.remove loop
local write = 1
for read = 1, #arr do
if not matchedIndices[read] then
arr[write] = arr[read]
write += 1
end
end
for i = write, #arr do
arr[i] = nil
end
end
```
### Single Forward Declaration for Mutual Recursion
`computePairCost` calls `matchChildren` and vice versa. Define `matchChildren` first (forward-declared), then `computePairCost` as a `local function`, then assign `matchChildren`:
```lua
local matchChildren -- forward declare (matchChildren is defined after computePairCost)
local function computePairCost(...)
-- can reference matchChildren (captured as upvalue, assigned before first call)
end
matchChildren = function(...)
-- can reference computePairCost (already defined above)
end
```
### computePairCost (replaces `_computeChangeCount` + `_computeChildrenCost`)
```lua
computePairCost = function(
session: MatchingSession,
virtualId: string,
studioInstance: Instance,
virtualInstances: { [string]: any },
bestSoFar: number
): number
-- 1. Cache lookup
local vc = session.costCache[virtualId]
if vc then
local cached = vc[studioInstance]
if cached ~= nil then return cached end
end
local vInst = virtualInstances[virtualId]
if not vInst then return UNMATCHED_PENALTY end
-- 2. Own diffs
local classKeys = RbxDom.getClassComparisonKeys(vInst.ClassName)
local vCache = cacheVirtual(vInst, classKeys)
local sCache = cacheStudio(studioInstance, classKeys, vCache.extraProps)
local cost: number = countOwnDiffs(vCache, sCache, classKeys)
if cost >= bestSoFar then return cost end
-- 3. Children cost (direct matchChildren call, no intermediate function)
local vChildren = vInst.Children
local studioKids = sCache.children
if (not vChildren or #vChildren == 0) and #studioKids == 0 then
-- leaf: no children on either side
elseif not vChildren or #vChildren == 0 then
cost += #studioKids * UNMATCHED_PENALTY
elseif #studioKids == 0 then
for _, childId in vChildren do
if virtualInstances[childId] then cost += UNMATCHED_PENALTY end
end
else
local validVChildren: { string } = {}
for _, childId in vChildren do
if virtualInstances[childId] then
table.insert(validVChildren, childId)
end
end
local childResult = matchChildren(
session, validVChildren, studioKids, virtualInstances,
virtualId, studioInstance
)
cost += childResult.totalCost
end
-- 4. Cache if exact (no early exit)
if cost < bestSoFar then
if not session.costCache[virtualId] then
session.costCache[virtualId] = {}
end
session.costCache[virtualId][studioInstance] = cost
end
return cost
end
```
### matchChildren (restructured, returns totalCost)
```lua
matchChildren = function(
session: MatchingSession,
virtualChildren: { string },
studioChildren: { Instance },
virtualInstances: { [string]: any },
parentVirtualId: string?,
parentStudioInstance: Instance?
): MatchResult
-- 1. Cache lookup
if parentVirtualId and parentStudioInstance then
local pc = session.matchCache[parentVirtualId]
if pc then
local cached = pc[parentStudioInstance]
if cached then return cached end
end
end
local matched: { MatchPair } = {}
local remainingVirtual: { string } = table.clone(virtualChildren)
local remainingStudio: { Instance } = table.clone(studioChildren)
-- 2. Group by (Name, ClassName) -- NO pcall, direct property access
local vByKey: { [string]: { number } } = {}
for i, id in remainingVirtual do
local vInst = virtualInstances[id]
if vInst then
local key = vInst.Name .. "\0" .. vInst.ClassName
local group = vByKey[key]
if not group then
group = {}
vByKey[key] = group
end
table.insert(group, i)
end
end
local sByKey: { [string]: { number } } = {}
for i, inst in remainingStudio do
local key = inst.Name .. "\0" .. inst.ClassName
local group = sByKey[key]
if not group then
group = {}
sByKey[key] = group
end
table.insert(group, i)
end
-- 3. 1:1 instant match
local matchedV: { [number]: boolean } = {}
local matchedS: { [number]: boolean } = {}
for key, vIndices in vByKey do
local sIndices = sByKey[key]
if sIndices and #vIndices == 1 and #sIndices == 1 then
local vi, si = vIndices[1], sIndices[1]
if not matchedV[vi] and not matchedS[si] then
table.insert(matched, {
virtualId = remainingVirtual[vi],
studioInstance = remainingStudio[si],
})
matchedV[vi] = true
matchedS[si] = true
end
end
end
removeMatched(remainingVirtual, matchedV)
removeMatched(remainingStudio, matchedS)
-- 4. Ambiguous groups (same structure as before but using computePairCost)
if #remainingVirtual > 0 and #remainingStudio > 0 then
-- Rebuild groups, pre-compute caches, N*M scoring...
-- Score via: ownDiffs (from pre-computed caches) + childrenCost
-- For childrenCost: call computePairCost (which calls matchChildren recursively)
-- BUT: the ambiguous loop uses pre-computed caches for ownDiffs directly
-- (cacheVirtual/cacheStudio/countOwnDiffs), then adds children cost via
-- a direct matchChildren call (same as computePairCost but with caches already built)
--
-- The inner scoring for an ambiguous (vi, si) pair:
-- cost = countOwnDiffs(vCaches[vi], sCaches[si], classKeys)
-- if cost < bestSoFar then
-- local vInst = virtualInstances[remainingVirtual[vi]]
-- local studioKids = sCaches[si].children
-- -- filter valid virtual children, call matchChildren, add totalCost
-- end
--
-- Greedy assign sorted by cost ascending (tie-break by insertion index)
end
-- 5. Compute totalCost for ALL matched pairs (uses session cache)
local totalCost: number = 0
for _, pair in matched do
totalCost += computePairCost(
session, pair.virtualId, pair.studioInstance, virtualInstances, math.huge
)
end
totalCost += (#remainingVirtual + #remainingStudio) * UNMATCHED_PENALTY
local result: MatchResult = {
matched = matched,
unmatchedVirtual = remainingVirtual,
unmatchedStudio = remainingStudio,
totalCost = totalCost,
}
-- 6. Cache result
if parentVirtualId and parentStudioInstance then
if not session.matchCache[parentVirtualId] then
session.matchCache[parentVirtualId] = {}
end
session.matchCache[parentVirtualId][parentStudioInstance] = result
end
return result
end
```
### pcall Removal in Studio Grouping
The current code wraps `inst.Name, inst.ClassName` in a `pcall(function() ... end)`. This allocates a closure per studio child and prevents inlining. During matching, studio children come from `:GetChildren()` on a live instance within a single-threaded sync frame -- they cannot be destroyed mid-operation. Replace with direct property access.
### O(n) removeMatched
Current `_removeMatched` uses `table.remove` in a reverse loop -- O(n^2) due to element shifting. Replace with single-pass in-place compaction (shown above).
### Module Export
```lua
return {
newSession = newSession,
matchChildren = matchChildren,
}
```
Only two symbols exported. All helpers are local (invisible outside module, inlineable by compiler).
---
## Lua Callers
### [hydrate.lua](plugin/src/Reconciler/hydrate.lua)
```lua
local Matching = require(script.Parent.matching)
local function hydrate(instanceMap, virtualInstances, rootId, rootInstance, session)
-- ...
local result = Matching.matchChildren(
session, validVirtualIds, existingChildren, virtualInstances,
rootId, rootInstance
)
for _, pair in result.matched do
hydrate(instanceMap, virtualInstances, pair.virtualId, pair.studioInstance, session)
end
-- ...
end
```
### [Reconciler/init.lua](plugin/src/Reconciler/init.lua)
```lua
function Reconciler:hydrate(virtualInstances, rootId, rootInstance, session)
Timer.start("Reconciler:hydrate")
local result = hydrate(self.__instanceMap, virtualInstances, rootId, rootInstance, session)
Timer.stop()
return result
end
```
### [ServeSession.lua](plugin/src/ServeSession.lua)
Create session before hydrate at line 628:
```lua
local Matching = require(script.Parent.Reconciler.matching)
-- ...
local session = Matching.newSession()
self.__reconciler:hydrate(readResponseBody.instances, serverInfo.rootInstanceId, game, session)
```
---
## trueEquals Rewrite: [trueEquals.lua](plugin/src/Reconciler/trueEquals.lua)
### Module Header
```lua
--!strict
--!native
--!optimize 2
```
### Problems in Current Implementation
1. **Redundant type dispatch**: Every branch checks both `typeA` and `typeB` (`typeA == "CFrame" and typeB == "CFrame"`). Check type once, early-out if they differ.
2. **Heap allocations in hot path**: `{ a:GetComponents() }`, `{ a.X, a.Y, a.Z }`, `{ a.X, a.Y }` build temporary arrays for every CFrame/Vector3/Vector2 comparison.
3. `**checkedKeys` set**: Allocates a table for every table-vs-table comparison. Unnecessary -- second loop only needs to check for extra keys in `b`.
4. **No type annotations**, no `--!strict`, no `--!native`, no `--!optimize 2`.
### Structure: Single typeof Dispatch + Dedicated Helpers
Adopt the [DeepEqualsPureUnsafe](escape-tsunami reference) pattern: `rawequal` identity check, `rawlen` for quick array length, `rawget`/`next` for table traversal, no temporary allocations. Bake in fuzzy equality for the matching use case.
```lua
local EPSILON: number = 0.0001
local function fuzzyEq(a: number, b: number): boolean
local diff = math.abs(a - b)
local maxVal = math.max(math.abs(a), math.abs(b), 1)
return diff < EPSILON or diff < maxVal * EPSILON
end
local function color3Eq(a: Color3, b: Color3): boolean
return math.floor(a.R * 255) == math.floor(b.R * 255)
and math.floor(a.G * 255) == math.floor(b.G * 255)
and math.floor(a.B * 255) == math.floor(b.B * 255)
end
local function vector2Eq(a: Vector2, b: Vector2): boolean
return fuzzyEq(a.X, b.X) and fuzzyEq(a.Y, b.Y)
end
local function vector3Eq(a: Vector3, b: Vector3): boolean
return fuzzyEq(a.X, b.X) and fuzzyEq(a.Y, b.Y) and fuzzyEq(a.Z, b.Z)
end
local function cframeEq(a: CFrame, b: CFrame): boolean
local ax, ay, az, aR00, aR01, aR02, aR10, aR11, aR12, aR20, aR21, aR22 = a:GetComponents()
local bx, by, bz, bR00, bR01, bR02, bR10, bR11, bR12, bR20, bR21, bR22 = b:GetComponents()
return fuzzyEq(ax, bx) and fuzzyEq(ay, by) and fuzzyEq(az, bz)
and fuzzyEq(aR00, bR00) and fuzzyEq(aR01, bR01) and fuzzyEq(aR02, bR02)
and fuzzyEq(aR10, bR10) and fuzzyEq(aR11, bR11) and fuzzyEq(aR12, bR12)
and fuzzyEq(aR20, bR20) and fuzzyEq(aR21, bR21) and fuzzyEq(aR22, bR22)
end
```
All helpers are small `local function`s -- the compiler can inline them into `trueEquals`.
### Main Function: Single typeof Dispatch
```lua
local function trueEquals(a: any, b: any): boolean
if rawequal(a, b) then return true end
-- Null-ref equivalence: nil == {Ref = "000...0"}
if a == nil then
return type(b) == "table" and rawget(b, "Ref") == "00000000000000000000000000000000"
end
if b == nil then
return type(a) == "table" and rawget(a, "Ref") == "00000000000000000000000000000000"
end
local t = typeof(a)
if t ~= typeof(b) then
-- EnumItem/number cross-type
if t == "number" and typeof(b) == "EnumItem" then return a == (b :: EnumItem).Value end
if t == "EnumItem" and typeof(b) == "number" then return (a :: EnumItem).Value == b end
return false
end
-- Same-type dispatch (no redundant second type check)
if t == "table" then
-- DeepEqualsPureUnsafe pattern with fuzzy recursive equality
if rawlen(a) ~= rawlen(b) then return false end
for k, v in next, a do
local ov = rawget(b, k)
if ov == nil or not trueEquals(v, ov) then return false end
end
for k in next, b do
if rawget(a, k) == nil then return false end
end
return true
end
if t == "number" then return fuzzyEq(a, b) end
if t == "Color3" then return color3Eq(a, b) end
if t == "Vector3" then return vector3Eq(a, b) end
if t == "Vector2" then return vector2Eq(a, b) end
if t == "CFrame" then return cframeEq(a, b) end
-- NaN check (number case already handled above, this catches any remaining NaN-like values)
if a ~= a and b ~= b then return true end
return false
end
```
**Key improvements:**
- **Zero heap allocations**: No temporary arrays for CFrame/Vector3/Vector2. CFrame uses 12 direct `fuzzyEq` calls. Vector3/Vector2 use direct component access.
- **No `checkedKeys` set**: Second table loop only checks existence via `rawget`, doesn't compare values (already done in first loop).
- **Single `typeof` call path**: Check type of `a` once, verify `b` matches, then dispatch. No redundant `typeA == X and typeB == X` at every branch.
- `**rawequal` identity check first**: Catches same-reference case before any type checking.
- `**rawlen` fast-path for tables**: Different array lengths = not equal, no iteration needed.
---
## Codegen Verification
After each rewrite, run:
```bash
luau-compile --remarks plugin/src/Reconciler/trueEquals.lua
luau-compile --remarks plugin/src/Reconciler/matching.lua
```
**trueEquals.lua** -- check that:
- `fuzzyEq`, `color3Eq`, `vector2Eq`, `vector3Eq`, `cframeEq` are inlined into `trueEquals`
- No "can't inline" remarks on any helper
**matching.lua** -- check that:
- `countOwnDiffs`, `cacheVirtual`, `cacheStudio`, `removeMatched` are inlined at call sites
- `computePairCost` inlines or at minimum shows no optimization barriers
- No generic dispatch warnings for typed parameters
Iterate on any functions that fail to inline (split if too large, simplify control flow).
---
## Rust Forward Sync: [src/snapshot/matching.rs](src/snapshot/matching.rs)
Add session struct and thread through. Rust already has the clean 2-function structure (`match_forward` + `compute_change_count` / `match_children_for_scoring`).
```rust
pub struct MatchingSession {
cost_cache: RefCell<HashMap<(Ref, Ref), u32>>,
}
```
- Add `total_cost: u32` to `ForwardMatchResult`
- Add `session: &MatchingSession` param to `match_forward`, `compute_change_count`, `match_children_for_scoring`
- Cache `compute_change_count` results by `(snap.snapshot_id, tree_ref)`, only when `cost < best_so_far`
- Create session in [patch_compute.rs](src/snapshot/patch_compute.rs) `compute_patch_set`, thread through `compute_patch_set_internal` and `compute_children_patches`
## Rust Syncback: [src/syncback/matching.rs](src/syncback/matching.rs)
Same pattern. Session struct, `total_cost` on `MatchResult`, session param on all functions.
- Create session in [dir.rs](src/snapshot_middleware/dir.rs) before `match_children` call
## Test Updates: [matching_fixtures.rs](tests/tests/matching_fixtures.rs)
Pass `&MatchingSession::new()` to `match_children` and `match_forward` calls.
---
## Summary of Optimizations
| Change | Impact |
| ------------------------------------------------------- | ----------------------------------------------------------------- |
| **trueEquals**: single typeof dispatch | Eliminates redundant type checks at every branch |
| **trueEquals**: dedicated type helpers | Zero heap allocations (no temp arrays for CFrame/Vector3/Vector2) |
| **trueEquals**: DeepEqualsPureUnsafe table pattern | No checkedKeys set; rawlen/rawget/next |
| **trueEquals**: `--!strict`/`--!native`/`--!optimize 2` | Native codegen for the most-called function in matching |
| **matching**: local functions (not table methods) | Enables compiler inlining |
| **matching**: `--!strict`/`--!native`/`--!optimize 2` | Native codegen, type-guided optimization |
| **matching**: concrete types on all params/returns | Specialized native code paths |
| **matching**: remove pcall in grouping | Eliminates closure + pcall overhead per child |
| **matching**: 2-function recursion (from 4) | Simpler for optimizer, easier to reason about |
| **matching**: session caching (matchCache + costCache) | Eliminates redundant subtree scoring across recursion levels |
| **matching**: totalCost on matchChildren | Eliminates _computeChildrenCost re-iteration of matched pairs |
| **matching**: O(n) removeMatched | Linear vs quadratic element removal |

394
.cursor/plans/syncback_via_rbxm_ad0231a7.plan.md Normal file
View file

@ -0,0 +1,394 @@
---
name: Syncback via RBXM
overview: Add a "Syncback" button to the plugin that serializes the DataModel per-service via SerializationService, sends the rbxm chunks to a totally segregated server endpoint, which tears down the entire serve system, runs CLI-equivalent clean-mode syncback, and restarts serve -- all within the same process.
todos:
- id: server-exit-reason
content: Modify LiveServer::start() to return ServerExitReason via tokio select + shutdown channel (src/web/mod.rs)
status: completed
- id: api-types
content: Add SyncbackPayload/ServiceChunk types to src/web/interface.rs
status: completed
- id: api-endpoint
content: Add POST /api/syncback endpoint handler in src/web/api.rs (fully segregated, no session validation)
status: completed
- id: serve-loop
content: Wrap ServeCommand::run() in a loop that handles SyncbackRequested (src/cli/serve.rs)
status: completed
- id: reconstruct-dom
content: "Implement run_live_syncback: reconstruct WeakDom from service chunks, run syncback_loop (clean mode, all CLI defaults), write files"
status: completed
- id: plugin-button
content: Add Syncback button (Danger style, layoutOrder 2) to NotConnected.lua
status: completed
- id: plugin-confirm
content: Add confirmation dialog (StudioPluginGui, danger-themed) in App/init.lua
status: completed
- id: plugin-serialize
content: Implement DataModel serialization per SYNCBACK_SERVICES list using SerializationService
status: completed
- id: plugin-api
content: Add standalone syncback HTTP post function (no ApiContext/session needed)
status: completed
- id: plugin-flow
content: "Wire up full flow in App/init.lua: confirm -> serialize -> POST /api/syncback -> done (fire and forget)"
status: completed
- id: test-helpers
content: Add post_api_syncback, wait_to_come_back_online, make_service_chunk, make_rbxl_from_chunks, run_cli_syncback_on_chunks to serve_util.rs
status: completed
- id: test-fixtures
content: Create live_syncback, live_syncback_place_ids, and live_syncback_complex serve-test fixtures
status: completed
- id: test-parity
content: "CLI parity tests (byte-for-byte): scripts, models+properties, refs, special names, duplicates, deep hierarchy, mixed types, comprehensive properties, multi-service"
status: completed
- id: test-validation
content: "Rejection tests: bad protocol, bad version, blocked place, unlisted place, whitelisted place, unrestricted"
status: completed
- id: test-lifecycle
content: "Lifecycle tests: server comes back functional (WebSocket + file watch), syncback twice with different data, replaces all old files"
status: completed
- id: test-roundtrip
content: "Round-trip identity test: build -> decompose -> live syncback -> rebuild -> compare instance trees"
status: completed
isProject: false
---
# Syncback via RBXM
## Architecture
```mermaid
sequenceDiagram
participant User
participant Plugin
participant Server
participant Filesystem
User->>Plugin: Click "Syncback" button
Plugin->>Plugin: Show danger confirmation dialog
User->>Plugin: Confirm
Plugin->>Plugin: SerializeInstancesAsync per service
Plugin->>Server: POST /api/syncback (msgpack, segregated endpoint)
Server->>Plugin: 200 OK
Note over Server: Store payload, signal shutdown
Note over Server: Full teardown: drop ServeSession, VFS watcher, ChangeProcessor, WebSocket connections
Note over Server: Reconstruct WeakDom from service chunks
Note over Server: Run syncback_loop (clean mode, all CLI defaults)
Server->>Filesystem: Write files
Note over Server: Create fresh ServeSession + LiveServer
Note over Server: Bind to same address:port
Plugin->>Plugin: Poll GET /api/rojo until server is back
Plugin->>Server: Normal connect flow
```
## Server Changes
### 1. Make `LiveServer::start()` return instead of blocking forever
`[src/web/mod.rs](src/web/mod.rs)`: Change `start()` to return a `ServerExitReason` enum. Add a shutdown mechanism via `Arc<SyncbackSignal>` (wrapping a `Mutex<Option<SyncbackPayload>>` + `tokio::sync::Notify`). The accept loop uses `tokio::select!` to watch for the notify signal alongside `listener.accept()`. When signaled, it breaks and returns.
```rust
pub enum ServerExitReason {
SyncbackRequested(SyncbackPayload),
}
pub struct SyncbackSignal {
payload: Mutex<Option<SyncbackPayload>>,
notify: tokio::sync::Notify,
}
```
The `SyncbackSignal` is shared between `LiveServer` (which watches it) and the API handler (which deposits data into it). When the accept loop exits, the Tokio runtime drops, all spawned tasks (including active WebSocket connections) are cancelled, and the `Arc<ServeSession>` drops -- tearing down the ChangeProcessor thread, VFS watcher, and all in-memory state.
### 2. New segregated API endpoint: `POST /api/syncback`
`[src/web/api.rs](src/web/api.rs)`: Totally separate from the connect/sync flow. No session ID needed -- the plugin doesn't need to be "connected" to use this. But it does validate version and place ID.
Request body is msgpack-encoded:
```rust
pub struct SyncbackRequest {
pub protocol_version: u64,
pub server_version: String,
pub place_id: Option<u64>,
pub services: Vec<ServiceChunk>,
}
pub struct ServiceChunk {
pub class_name: String,
pub data: Vec<u8>, // raw rbxm bytes
}
// Payload passed to the serve loop (version info stripped after validation)
pub struct SyncbackPayload {
pub services: Vec<ServiceChunk>,
}
```
Types live in `[src/web/interface.rs](src/web/interface.rs)`.
Handler flow:
1. Deserialize msgpack body into `SyncbackRequest`
2. **Validate protocol version** -- reject with 400 if `request.protocol_version != PROTOCOL_VERSION`
3. **Validate server version** -- reject with 400 if major.minor doesn't match `SERVER_VERSION`
4. **Validate place ID** against the project's `servePlaceIds` (whitelist) and `blockedPlaceIds` (blocklist). Reject with 403 if not allowed.
5. Extract `SyncbackPayload` (just the services), store in `SyncbackSignal` -> notify -> return 200 `{"status": "syncback_initiated"}`
The response is sent before any teardown begins (the API handler is a separate tokio task from the accept loop).
### 3. Wrap `ServeCommand::run()` in a loop
`[src/cli/serve.rs](src/cli/serve.rs)`: The `run()` method becomes a loop. Each iteration creates a fresh VFS, ServeSession, and LiveServer. When `start()` returns `SyncbackRequested`, the old session is fully dropped (VFS watcher stops, ChangeProcessor thread joins, tree is freed), syncback runs, and the loop continues to create everything fresh:
```rust
loop {
let (vfs, errors) = Vfs::new_default_with_errors();
let session = Arc::new(ServeSession::new(vfs, project_path.clone(), Some(errors))?);
let server = LiveServer::new(session);
match server.start(addr) {
ServerExitReason::SyncbackRequested(payload) => {
log::info!("Live syncback requested, running...");
run_live_syncback(&project_path, payload)?;
log::info!("Syncback complete, restarting serve...");
continue;
}
}
}
```
IP and port are resolved once before the loop (they don't change).
### 4. Reconstruct WeakDom + run syncback (CLI-equivalent)
New function `run_live_syncback(project_path: &Path, payload: SyncbackPayload) -> anyhow::Result<()>` in `src/cli/serve.rs`:
1. **Build WeakDom** -- Create `DataModel` root, for each `ServiceChunk` parse rbxm via `rbx_binary::from_reader`, create a service instance of the correct class, deep-clone all children from the parsed dom into the target dom. Maintain a `HashMap<Ref, Ref>` ref-map to fix up Ref properties across chunks.
2. **Run syncback** -- Identical to what CLI `atlas syncback` does:
- Create oneshot VFS: `Vfs::new_oneshot()`
- Load old tree: `ServeSession::new_oneshot(vfs, project_path)`
- Call `syncback_loop(vfs, &mut old_tree, new_dom, project, incremental: false)` -- clean mode
- Write: `fs_snapshot.write_to_vfs_parallel(base_path, vfs)`
- `refresh_git_index(base_path)`
Same code path as CLI syncback, same defaults, same clean mode. The only difference is the WeakDom source.
## Plugin Changes
### 5. Syncback button on NotConnected page
`[plugin/src/App/StatusPages/NotConnected.lua](plugin/src/App/StatusPages/NotConnected.lua)`: Add "Syncback" button with `style = "Danger"` and `layoutOrder = 2`, shift "Sync" to `layoutOrder = 3`:
```
[Settings] [Syncback] [Sync]
```
Fires `self.props.onSyncback`.
### 6. Confirmation dialog
`[plugin/src/App/init.lua](plugin/src/App/init.lua)`: Floating `StudioPluginGui` dialog (same pattern as `AcceptConfirm` in Confirming.lua):
- Title: "Full Syncback"
- Message: "This will overwrite your project files with the current Studio state. This cannot be undone."
- Buttons: "Cancel" (Bordered) | "Syncback" (Danger)
- State: `self.state.showingSyncbackConfirm`
### 7. DataModel serialization
After confirmation, serialize per-service. Hardcode the same `VISIBLE_SERVICES` list from the Rust side:
```lua
local SYNCBACK_SERVICES = {
"Lighting", "MaterialService", "ReplicatedFirst", "ReplicatedStorage",
"ServerScriptService", "ServerStorage", "SoundService", "StarterGui",
"StarterPack", "StarterPlayer", "Teams", "TextChatService",
"VoiceChatService", "Workspace",
}
local services = {}
for _, className in SYNCBACK_SERVICES do
local service = game:FindService(className)
if service and #service:GetChildren() > 0 then
local buf = SerializationService:SerializeInstancesAsync(service:GetChildren())
table.insert(services, {
className = className,
data = buf, -- msgpack-luau natively encodes buffer as bin format (zero-copy)
})
end
end
```
The msgpack-luau library handles `buffer` types natively -- when `type(data) == "buffer"`, it encodes as msgpack bin 8/16/32 using `buffer.copy` directly. No string conversion needed. On the Rust side, `rmp_serde` deserializes msgpack bin into `Vec<u8>` automatically.
### 8. Direct HTTP post (no ApiContext needed)
Since this endpoint is fully segregated and doesn't need a session, the plugin just needs host:port from the address entry fields. A standalone function posts directly:
```lua
local function postSyncback(host, port, services)
local url = ("http://%s:%s/api/syncback"):format(host, port)
local body = Http.msgpackEncode({
protocolVersion = Config.protocolVersion,
serverVersion = Config.expectedServerVersionString,
placeId = game.PlaceId,
services = services,
})
return Http.post(url, body):andThen(rejectFailedRequests)
end
```
No `ApiContext` instance needed. This keeps it completely independent from the connect flow.
### 9. Full flow in App
`[plugin/src/App/init.lua](plugin/src/App/init.lua)`: The `onSyncback` handler:
1. Show confirmation dialog
2. On confirm: serialize DataModel per-service
3. POST to `/api/syncback` with the chunks
4. On 200: show a notification ("Syncback data sent. Server is processing.")
5. Done. Stay on NotConnected page. User clicks "Sync" manually whenever they're ready.
Fire and forget -- the plugin's job ends after the POST succeeds. No polling, no auto-reconnect, no special state.
## Key Design Decisions
- **Fully segregated endpoint**: `/api/syncback` has zero coupling to the session/connect system. No session ID, no auth. Plugin fires and forgets.
- **Complete teardown**: When the signal fires, the entire serve system is destroyed -- Tokio runtime drops, ServeSession drops (joining the ChangeProcessor thread, stopping VFS watcher), all WebSocket connections are cancelled. Nothing survives from the previous serve.
- **CLI-equivalent syncback**: Uses `syncback_loop` with `incremental: false` (clean mode), non-interactive. Identical behavior to `atlas syncback --input file.rbxl` with all defaults.
- **Same process, same port**: The `atlas serve` process never exits. After syncback completes, a fresh serve starts on the same address:port. The user sees log output in their terminal throughout.
- **Danger styling**: Uses existing `Danger` theme (`#E74C3C` red) defined in `[plugin/src/App/Theme.lua](plugin/src/App/Theme.lua)`.
- **Per-service serialization**: 14 visible services from `VISIBLE_SERVICES` constant, matching both Rust and Lua sides. Cross-service Ref properties are lost in transit but syncback's `Rojo_Ref_`* system resolves refs from filesystem structure.
## Tests
The core invariant: **live syncback must produce byte-identical filesystem output to CLI syncback given the same input data.** If it doesn't, something is broken in the serialize -> transmit -> reconstruct -> syncback pipeline. Every test that verifies data accuracy runs BOTH live syncback and CLI syncback on the same input, then compares output with `assert_dirs_equal` (byte-for-byte file comparison from `[tests/rojo_test/roundtrip_util.rs](tests/rojo_test/roundtrip_util.rs)`).
Tests go in a dedicated `tests/tests/live_syncback.rs`. They use `run_serve_test` from `[tests/rojo_test/serve_util.rs](tests/rojo_test/serve_util.rs)` plus the round-trip utilities.
### Test helpers (add to `serve_util.rs`)
`**post_api_syncback_raw`** -- Sends a raw msgpack body to `POST /api/syncback`. Returns the full `reqwest::blocking::Response` so tests can assert on status codes and error bodies.
`**post_api_syncback`** -- Convenience wrapper that fills in correct protocol version + server version. Takes `place_id: Option<u64>` and `services: Vec<ServiceChunk>`. Panics if response is not 200.
`**wait_to_come_back_online`** -- Polls `GET /api/rojo` with exponential backoff (longer timeout than initial startup since syncback runs between teardown and restart). Returns the new `ServerInfoResponse`. Panics if server doesn't come back.
`**make_service_chunk`** -- Builds a `ServiceChunk` from `InstanceBuilder` children:
```rust
fn make_service_chunk(class_name: &str, children: Vec<InstanceBuilder>) -> ServiceChunk {
let mut dom = WeakDom::new(InstanceBuilder::new("DataModel"));
let root = dom.root_ref();
for child in children {
dom.insert(root, child);
}
let refs: Vec<Ref> = dom.root().children().to_vec();
let mut buf = Vec::new();
rbx_binary::to_writer(&mut buf, &dom, &refs).unwrap();
ServiceChunk { class_name: class_name.to_string(), data: buf }
}
```
`**make_rbxl_from_chunks**` -- Builds a complete rbxl file from the same service chunks, for CLI parity testing. Creates a WeakDom with DataModel root + properly-classed service instances + children, serializes to rbxl. This gives CLI syncback the exact same input data as live syncback received.
`**run_cli_syncback_on_chunks**` -- End-to-end helper: copies a fixture to a fresh temp dir, calls `make_rbxl_from_chunks`, runs `run_rojo_syncback_clean`, returns the temp dir path. Used by every parity test as the CLI reference.
### Fixtures
`**rojo-test/serve-tests/live_syncback/**` -- Multi-service project:
```
default.project.json5 # RS -> src/shared, SSS -> src/server, Workspace -> src/workspace
src/shared/OldModule.luau
src/server/OldScript.server.luau
src/workspace/OldPart.model.json5
```
`**rojo-test/serve-tests/live_syncback_place_ids/**` -- Same layout + `servePlaceIds: [123]`, `blockedPlaceIds: [999]`.
`**rojo-test/serve-tests/live_syncback_complex/**` -- All 14 visible services mapped to separate `$path` dirs. Used for comprehensive parity tests.
### CLI parity tests (data accuracy)
Every parity test follows this pattern:
```rust
fn assert_live_matches_cli(
fixture_name: &str,
chunks: Vec<ServiceChunk>,
place_id: Option<u64>,
) {
run_serve_test(fixture_name, |session, _| {
// 1. Live syncback
session.post_api_syncback(place_id, chunks.clone());
session.wait_to_come_back_online();
// 2. CLI syncback on fresh copy of same fixture with same data
let cli_dir = run_cli_syncback_on_chunks(fixture_name, &chunks);
// 3. Byte-for-byte comparison of every file
assert_dirs_equal(session.path(), &cli_dir);
});
}
```
`**parity_scripts**` -- ModuleScript (`return {}`), Script/Server (`print("server")`), Script/Client (`print("client")`), LocalScript. Verifies Source content preserved exactly, correct file extensions (`.luau`, `.server.luau`, `.client.luau`, `.local.luau`), RunContext encoded in extension.
`**parity_models_with_properties**` -- Model containing Parts with: Anchored (Bool), Size (Vector3), Position (Vector3), Color (Color3), BrickColor, Shape (Enum), Transparency (Float64). Verifies every property value round-trips through rbxm -> reconstruct -> syncback -> meta/model files identically.
`**parity_ref_properties**` -- Model with PrimaryPart -> child Part. ObjectValue with Value -> sibling Part. Verifies `Rojo_Ref_*` attribute paths in meta/model files match CLI output exactly.
`**parity_special_names**` -- Instances named `"What?"`, `"Key:Value"`, `"A/B"`, `"<Angle>"`, `'He said "hi"'`. Verifies slugified filenames (`What_`, `Key_Value`, `A_B`, etc.) and meta `name` fields match CLI.
`**parity_duplicate_names**` -- 3x Folder named `"Data"`, 2x Script named `"Handler"`. Verifies dedup suffixes (`Data/`, `Data~2/`, `Data~3/`, `Handler.server.luau`, `Handler~2.server.luau`) and meta files match CLI.
`**parity_deep_hierarchy**` -- 6 levels: Folder > Folder > Folder > Folder > Model > Part. Verifies nested directory structure is identical.
`**parity_mixed_file_types**` -- StringValue (-> `.txt`), LocalizationTable (-> `.csv`), JSON data module (-> `.json5`), binary model (-> `.rbxm`), Folder (-> directory). All under one service. Byte-for-byte match with CLI.
`**parity_comprehensive_properties**` -- Single Part with every testable property type set to non-default values: Bool, Int32, Float32, Float64, String, Vector2, Vector3, Vector2int16, Vector3int16, UDim, UDim2, Color3, Color3uint8, BrickColor, NumberRange, Rect, Enum, Font, Tags, Attributes (with nested types). Verifies meta.json5 property serialization matches CLI exactly.
`**parity_multi_service**` -- Data under all 14 visible services simultaneously. Each service gets a unique set of instances (scripts, models, folders). Verifies every service's `$path` directory matches CLI output.
`**parity_empty_services_mixed**` -- Some services have children, some don't, some are absent from the payload entirely. Verifies populated services are correct and empty/absent services result in clean directories.
### Validation / rejection tests
`**rejects_bad_protocol**` -- Send `protocol_version: 999`. Expect non-200. GET /api/rojo still returns same session ID (server unaffected).
`**rejects_bad_version**` -- Send `server_version: "0.0.0"`. Expect non-200. Server unaffected.
`**rejects_blocked_place**` -- Fixture has `blockedPlaceIds: [999]`. Send `placeId: 999`. Expect 403. Server unaffected.
`**rejects_unlisted_place**` -- Fixture has `servePlaceIds: [123]`. Send `placeId: 456`. Expect 403. Server unaffected.
`**allows_whitelisted_place**` -- `servePlaceIds: [123]`, send `placeId: 123`. Expect 200. Syncback runs, server restarts.
`**allows_any_place_when_unrestricted**` -- No servePlaceIds/blockedPlaceIds. Send `placeId: 99999`. Expect 200.
### Lifecycle / edge case tests
`**server_comes_back_functional**` -- After live syncback, verify the restarted server is fully operational: GET /api/rojo works, can read instances via /api/read, WebSocket connects and receives patches when a file is modified on disk.
`**syncback_twice_different_data**` -- Run live syncback with data A. Wait for restart. Run live syncback AGAIN with data B. Wait for restart. Verify filesystem matches data B (not A). Verify server is functional. Tests the serve loop handles multiple iterations.
`**syncback_replaces_all_old_files**` -- Pre-populate the fixture with files that should NOT survive clean mode (orphan scripts, old models). After live syncback, verify none of the old files remain. Only files from the new data exist.
### Round-trip identity test
The ultimate accuracy test:
`**roundtrip_build_syncback_rebuild**` -- Using the `live_syncback_complex` fixture:
1. `atlas build` -> rbxl_A
2. Decompose rbxl_A into per-service chunks (parse with `rbx_binary::from_reader`, extract children of each service, re-serialize as rbxm chunks)
3. POST chunks to live syncback, wait for restart
4. `atlas build` -> rbxl_B
5. Parse both rbxl_A and rbxl_B into WeakDoms
6. Walk both trees recursively: compare instance names, classes, property values (using the same fuzzy equality from `variant_eq` for floats/CFrames), and child counts
7. Any divergence = test failure with detailed diff
This tests the full pipeline: build -> decompose into service chunks -> transmit -> reconstruct -> syncback -> build again. If the output matches, the live syncback system preserves 100% data fidelity.