diff --git a/.cursor/plans/matching_session_caching_cbfa7bda.plan.md b/.cursor/plans/matching_session_caching_cbfa7bda.plan.md new file mode 100644 index 00000000..5a3c7eb0 --- /dev/null +++ b/.cursor/plans/matching_session_caching_cbfa7bda.plan.md @@ -0,0 +1,609 @@ +--- +name: Matching session caching +overview: Rewrite the Lua matching module for native codegen performance (--!strict, --!native, --!optimize 2) with session-based caching, clean 2-function mutual recursion, concrete types, and local functions for inlining. Add session caching to both Rust implementations for consistency. +todos: + - id: lua-trueEquals + content: "Rewrite trueEquals.lua: --!strict/--!native/--!optimize 2, single typeof dispatch, dedicated helper functions per type (no temp arrays), adopt DeepEqualsPureUnsafe pattern for table comparison (rawlen/rawget/next, no checkedKeys set), keep fuzzy equality + null-ref + EnumItem cross-type" + status: completed + - id: lua-rewrite + content: "Rewrite matching.lua: --!strict/--!native/--!optimize 2, all functions as local function in dependency order (one forward declare for mutual recursion), 2-function mutual recursion (matchChildren + computePairCost), session caching, totalCost return, O(n) removeMatched, remove pcall in grouping" + status: completed + - id: lua-hydrate-threading + content: Thread session through hydrate.lua, Reconciler/init.lua, and ServeSession.lua (create session before hydrate call at line 628) + status: completed + - id: lua-verify-codegen + content: Run luau-compile --remarks on trueEquals.lua and matching.lua to verify inlining and native codegen quality, iterate until clean + status: completed + - id: lua-test + content: Build plugin, test Lua changes in Studio (hydrate + matching works correctly with session caching) before proceeding to Rust + status: completed + - id: rust-forward-session + content: "AFTER Lua passes: Add MatchingSession to src/snapshot/matching.rs with cost_cache, add total_cost to ForwardMatchResult, thread session through all functions and patch_compute.rs" + status: completed + - id: rust-syncback-session + content: "AFTER Lua passes: Add MatchingSession to src/syncback/matching.rs with cost_cache, add total_cost to MatchResult, thread session through all functions and dir.rs" + status: completed + - id: rust-test-updates + content: Update tests/tests/matching_fixtures.rs, run cargo test to verify Rust changes + status: completed +isProject: false +--- + +# Matching Session Caching + Native Codegen Rewrite + +## Problem + +1. **Redundant computation**: Scoring recurses into children, then hydrate re-calls matchChildren on the same children. The intermediate `_computeChildrenCost` calls matchChildren, iterates matched pairs calling `_computeChangeCount`, which recurses again -- all discarded after scoring. +2. **Poor native codegen structure**: All functions are on the `Matching` table (table lookup per call, uninlineable). No type annotations. No `--!strict`/`--!native`/`--!optimize 2`. A `pcall` with closure allocation in the hot grouping loop. +3. **4-function recursion chain** (`matchChildren -> _computeChildrenCost -> _computeChangeCount -> _computeChildrenCost -> matchChildren`) is complex and hard for both the optimizer and the developer to reason about. + +## Design + +### Conceptual Model + +`matchChildren` is fundamentally `computeSubtreeCost` -- it finds the cost-minimizing pairing of children and returns the total cost. The actual match assignments are a bonus output. The 2-function mutual recursion reflects this: + +- `**matchChildren(session, vChildren, sChildren, virtualInstances, parentVId, parentSInst)`**: Given two child sets, find optimal pairing. Returns `MatchResult` with `totalCost`. +- `**computePairCost(session, virtualId, studioInstance, virtualInstances, bestSoFar)`**: Given one virtual and one studio instance, compute total cost including subtree. Calls `matchChildren` for children cost. + +`_computeChildrenCost` and `_computeChangeCount` are eliminated. Their logic is absorbed into these two functions. + +### MatchingSession + +```lua +type MatchingSession = { + matchCache: { [string]: { [Instance]: MatchResult } }, + costCache: { [string]: { [Instance]: number } }, +} +``` + +- **matchCache**: keyed by `(parentVirtualId, parentStudioInstance)`. Parent uniquely determines children set. +- **costCache**: keyed by `(virtualId, studioInstance)`. Only stored when `cost < bestSoFar` (exact, not early-exited). + +Caching replaces the need for a depth limit: each unique pair/children-set is computed at most once. + +### bestSoFar Interaction + +- **costCache**: only store when computation was complete (`cost < bestSoFar`). If cached, return immediately regardless of caller's `bestSoFar` (the cached value IS the exact cost). +- **matchCache**: `matchChildren` always completes (no `bestSoFar` on it), so results are always cacheable. + +--- + +## Lua Plugin Rewrite: [matching.lua](plugin/src/Reconciler/matching.lua) + +### Module Header + +```lua +--!strict +--!native +--!optimize 2 +``` + +### Type Definitions (all concrete, minimal `any`) + +```lua +type MatchPair = { + virtualId: string, + studioInstance: Instance, +} + +type MatchResult = { + matched: { MatchPair }, + unmatchedVirtual: { string }, + unmatchedStudio: { Instance }, + totalCost: number, +} + +type MatchingSession = { + matchCache: { [string]: { [Instance]: MatchResult } }, + costCache: { [string]: { [Instance]: number } }, +} + +type VCache = { + props: { [string]: any }, + extraProps: { string }?, + tags: { [string]: boolean }?, + attrs: { [string]: any }?, + childCount: number, +} + +type SCache = { + instance: Instance, + props: { [string]: any }, + tags: { [string]: boolean }?, + attrs: { [string]: any }?, + children: { Instance }, + childCount: number, +} +``` + +### All Functions as `local function` in Dependency Order + +Every function is a `local function` defined in dependency order. No `Matching.foo()` table lookups. One forward declaration for the mutual recursion pair. + +```lua +local UNMATCHED_PENALTY: number = 10000 + +local function newSession(): MatchingSession + return { matchCache = {}, costCache = {} } +end + +local function cacheVirtual(vInst: any, classKeys: any): VCache + -- same logic as current _cacheVirtual, with return type annotation +end + +local function cacheStudio(studioInstance: Instance, classKeys: any, extraProps: { string }?): SCache + -- same logic as current _cacheStudio + -- ADDITION: store sCache.instance = studioInstance +end + +local function countOwnDiffs(vCache: VCache, sCache: SCache, classKeys: any): number + -- same logic as current _countOwnDiffsCached +end + +local function removeMatched(arr: { any }, matchedIndices: { [number]: boolean }): () + -- O(n) in-place compaction instead of O(n^2) table.remove loop + local write = 1 + for read = 1, #arr do + if not matchedIndices[read] then + arr[write] = arr[read] + write += 1 + end + end + for i = write, #arr do + arr[i] = nil + end +end +``` + +### Single Forward Declaration for Mutual Recursion + +`computePairCost` calls `matchChildren` and vice versa. Define `matchChildren` first (forward-declared), then `computePairCost` as a `local function`, then assign `matchChildren`: + +```lua +local matchChildren -- forward declare (matchChildren is defined after computePairCost) + +local function computePairCost(...) + -- can reference matchChildren (captured as upvalue, assigned before first call) +end + +matchChildren = function(...) + -- can reference computePairCost (already defined above) +end +``` + +### computePairCost (replaces `_computeChangeCount` + `_computeChildrenCost`) + +```lua +computePairCost = function( + session: MatchingSession, + virtualId: string, + studioInstance: Instance, + virtualInstances: { [string]: any }, + bestSoFar: number +): number + -- 1. Cache lookup + local vc = session.costCache[virtualId] + if vc then + local cached = vc[studioInstance] + if cached ~= nil then return cached end + end + + local vInst = virtualInstances[virtualId] + if not vInst then return UNMATCHED_PENALTY end + + -- 2. Own diffs + local classKeys = RbxDom.getClassComparisonKeys(vInst.ClassName) + local vCache = cacheVirtual(vInst, classKeys) + local sCache = cacheStudio(studioInstance, classKeys, vCache.extraProps) + local cost: number = countOwnDiffs(vCache, sCache, classKeys) + if cost >= bestSoFar then return cost end + + -- 3. Children cost (direct matchChildren call, no intermediate function) + local vChildren = vInst.Children + local studioKids = sCache.children + + if (not vChildren or #vChildren == 0) and #studioKids == 0 then + -- leaf: no children on either side + elseif not vChildren or #vChildren == 0 then + cost += #studioKids * UNMATCHED_PENALTY + elseif #studioKids == 0 then + for _, childId in vChildren do + if virtualInstances[childId] then cost += UNMATCHED_PENALTY end + end + else + local validVChildren: { string } = {} + for _, childId in vChildren do + if virtualInstances[childId] then + table.insert(validVChildren, childId) + end + end + local childResult = matchChildren( + session, validVChildren, studioKids, virtualInstances, + virtualId, studioInstance + ) + cost += childResult.totalCost + end + + -- 4. Cache if exact (no early exit) + if cost < bestSoFar then + if not session.costCache[virtualId] then + session.costCache[virtualId] = {} + end + session.costCache[virtualId][studioInstance] = cost + end + + return cost +end +``` + +### matchChildren (restructured, returns totalCost) + +```lua +matchChildren = function( + session: MatchingSession, + virtualChildren: { string }, + studioChildren: { Instance }, + virtualInstances: { [string]: any }, + parentVirtualId: string?, + parentStudioInstance: Instance? +): MatchResult + -- 1. Cache lookup + if parentVirtualId and parentStudioInstance then + local pc = session.matchCache[parentVirtualId] + if pc then + local cached = pc[parentStudioInstance] + if cached then return cached end + end + end + + local matched: { MatchPair } = {} + local remainingVirtual: { string } = table.clone(virtualChildren) + local remainingStudio: { Instance } = table.clone(studioChildren) + + -- 2. Group by (Name, ClassName) -- NO pcall, direct property access + local vByKey: { [string]: { number } } = {} + for i, id in remainingVirtual do + local vInst = virtualInstances[id] + if vInst then + local key = vInst.Name .. "\0" .. vInst.ClassName + local group = vByKey[key] + if not group then + group = {} + vByKey[key] = group + end + table.insert(group, i) + end + end + + local sByKey: { [string]: { number } } = {} + for i, inst in remainingStudio do + local key = inst.Name .. "\0" .. inst.ClassName + local group = sByKey[key] + if not group then + group = {} + sByKey[key] = group + end + table.insert(group, i) + end + + -- 3. 1:1 instant match + local matchedV: { [number]: boolean } = {} + local matchedS: { [number]: boolean } = {} + + for key, vIndices in vByKey do + local sIndices = sByKey[key] + if sIndices and #vIndices == 1 and #sIndices == 1 then + local vi, si = vIndices[1], sIndices[1] + if not matchedV[vi] and not matchedS[si] then + table.insert(matched, { + virtualId = remainingVirtual[vi], + studioInstance = remainingStudio[si], + }) + matchedV[vi] = true + matchedS[si] = true + end + end + end + + removeMatched(remainingVirtual, matchedV) + removeMatched(remainingStudio, matchedS) + + -- 4. Ambiguous groups (same structure as before but using computePairCost) + if #remainingVirtual > 0 and #remainingStudio > 0 then + -- Rebuild groups, pre-compute caches, N*M scoring... + -- Score via: ownDiffs (from pre-computed caches) + childrenCost + -- For childrenCost: call computePairCost (which calls matchChildren recursively) + -- BUT: the ambiguous loop uses pre-computed caches for ownDiffs directly + -- (cacheVirtual/cacheStudio/countOwnDiffs), then adds children cost via + -- a direct matchChildren call (same as computePairCost but with caches already built) + -- + -- The inner scoring for an ambiguous (vi, si) pair: + -- cost = countOwnDiffs(vCaches[vi], sCaches[si], classKeys) + -- if cost < bestSoFar then + -- local vInst = virtualInstances[remainingVirtual[vi]] + -- local studioKids = sCaches[si].children + -- -- filter valid virtual children, call matchChildren, add totalCost + -- end + -- + -- Greedy assign sorted by cost ascending (tie-break by insertion index) + end + + -- 5. Compute totalCost for ALL matched pairs (uses session cache) + local totalCost: number = 0 + for _, pair in matched do + totalCost += computePairCost( + session, pair.virtualId, pair.studioInstance, virtualInstances, math.huge + ) + end + totalCost += (#remainingVirtual + #remainingStudio) * UNMATCHED_PENALTY + + local result: MatchResult = { + matched = matched, + unmatchedVirtual = remainingVirtual, + unmatchedStudio = remainingStudio, + totalCost = totalCost, + } + + -- 6. Cache result + if parentVirtualId and parentStudioInstance then + if not session.matchCache[parentVirtualId] then + session.matchCache[parentVirtualId] = {} + end + session.matchCache[parentVirtualId][parentStudioInstance] = result + end + + return result +end +``` + +### pcall Removal in Studio Grouping + +The current code wraps `inst.Name, inst.ClassName` in a `pcall(function() ... end)`. This allocates a closure per studio child and prevents inlining. During matching, studio children come from `:GetChildren()` on a live instance within a single-threaded sync frame -- they cannot be destroyed mid-operation. Replace with direct property access. + +### O(n) removeMatched + +Current `_removeMatched` uses `table.remove` in a reverse loop -- O(n^2) due to element shifting. Replace with single-pass in-place compaction (shown above). + +### Module Export + +```lua +return { + newSession = newSession, + matchChildren = matchChildren, +} +``` + +Only two symbols exported. All helpers are local (invisible outside module, inlineable by compiler). + +--- + +## Lua Callers + +### [hydrate.lua](plugin/src/Reconciler/hydrate.lua) + +```lua +local Matching = require(script.Parent.matching) + +local function hydrate(instanceMap, virtualInstances, rootId, rootInstance, session) + -- ... + local result = Matching.matchChildren( + session, validVirtualIds, existingChildren, virtualInstances, + rootId, rootInstance + ) + for _, pair in result.matched do + hydrate(instanceMap, virtualInstances, pair.virtualId, pair.studioInstance, session) + end + -- ... +end +``` + +### [Reconciler/init.lua](plugin/src/Reconciler/init.lua) + +```lua +function Reconciler:hydrate(virtualInstances, rootId, rootInstance, session) + Timer.start("Reconciler:hydrate") + local result = hydrate(self.__instanceMap, virtualInstances, rootId, rootInstance, session) + Timer.stop() + return result +end +``` + +### [ServeSession.lua](plugin/src/ServeSession.lua) + +Create session before hydrate at line 628: + +```lua +local Matching = require(script.Parent.Reconciler.matching) +-- ... +local session = Matching.newSession() +self.__reconciler:hydrate(readResponseBody.instances, serverInfo.rootInstanceId, game, session) +``` + +--- + +## trueEquals Rewrite: [trueEquals.lua](plugin/src/Reconciler/trueEquals.lua) + +### Module Header + +```lua +--!strict +--!native +--!optimize 2 +``` + +### Problems in Current Implementation + +1. **Redundant type dispatch**: Every branch checks both `typeA` and `typeB` (`typeA == "CFrame" and typeB == "CFrame"`). Check type once, early-out if they differ. +2. **Heap allocations in hot path**: `{ a:GetComponents() }`, `{ a.X, a.Y, a.Z }`, `{ a.X, a.Y }` build temporary arrays for every CFrame/Vector3/Vector2 comparison. +3. `**checkedKeys` set**: Allocates a table for every table-vs-table comparison. Unnecessary -- second loop only needs to check for extra keys in `b`. +4. **No type annotations**, no `--!strict`, no `--!native`, no `--!optimize 2`. + +### Structure: Single typeof Dispatch + Dedicated Helpers + +Adopt the [DeepEqualsPureUnsafe](escape-tsunami reference) pattern: `rawequal` identity check, `rawlen` for quick array length, `rawget`/`next` for table traversal, no temporary allocations. Bake in fuzzy equality for the matching use case. + +```lua +local EPSILON: number = 0.0001 + +local function fuzzyEq(a: number, b: number): boolean + local diff = math.abs(a - b) + local maxVal = math.max(math.abs(a), math.abs(b), 1) + return diff < EPSILON or diff < maxVal * EPSILON +end + +local function color3Eq(a: Color3, b: Color3): boolean + return math.floor(a.R * 255) == math.floor(b.R * 255) + and math.floor(a.G * 255) == math.floor(b.G * 255) + and math.floor(a.B * 255) == math.floor(b.B * 255) +end + +local function vector2Eq(a: Vector2, b: Vector2): boolean + return fuzzyEq(a.X, b.X) and fuzzyEq(a.Y, b.Y) +end + +local function vector3Eq(a: Vector3, b: Vector3): boolean + return fuzzyEq(a.X, b.X) and fuzzyEq(a.Y, b.Y) and fuzzyEq(a.Z, b.Z) +end + +local function cframeEq(a: CFrame, b: CFrame): boolean + local ax, ay, az, aR00, aR01, aR02, aR10, aR11, aR12, aR20, aR21, aR22 = a:GetComponents() + local bx, by, bz, bR00, bR01, bR02, bR10, bR11, bR12, bR20, bR21, bR22 = b:GetComponents() + return fuzzyEq(ax, bx) and fuzzyEq(ay, by) and fuzzyEq(az, bz) + and fuzzyEq(aR00, bR00) and fuzzyEq(aR01, bR01) and fuzzyEq(aR02, bR02) + and fuzzyEq(aR10, bR10) and fuzzyEq(aR11, bR11) and fuzzyEq(aR12, bR12) + and fuzzyEq(aR20, bR20) and fuzzyEq(aR21, bR21) and fuzzyEq(aR22, bR22) +end +``` + +All helpers are small `local function`s -- the compiler can inline them into `trueEquals`. + +### Main Function: Single typeof Dispatch + +```lua +local function trueEquals(a: any, b: any): boolean + if rawequal(a, b) then return true end + + -- Null-ref equivalence: nil == {Ref = "000...0"} + if a == nil then + return type(b) == "table" and rawget(b, "Ref") == "00000000000000000000000000000000" + end + if b == nil then + return type(a) == "table" and rawget(a, "Ref") == "00000000000000000000000000000000" + end + + local t = typeof(a) + if t ~= typeof(b) then + -- EnumItem/number cross-type + if t == "number" and typeof(b) == "EnumItem" then return a == (b :: EnumItem).Value end + if t == "EnumItem" and typeof(b) == "number" then return (a :: EnumItem).Value == b end + return false + end + + -- Same-type dispatch (no redundant second type check) + if t == "table" then + -- DeepEqualsPureUnsafe pattern with fuzzy recursive equality + if rawlen(a) ~= rawlen(b) then return false end + for k, v in next, a do + local ov = rawget(b, k) + if ov == nil or not trueEquals(v, ov) then return false end + end + for k in next, b do + if rawget(a, k) == nil then return false end + end + return true + end + + if t == "number" then return fuzzyEq(a, b) end + if t == "Color3" then return color3Eq(a, b) end + if t == "Vector3" then return vector3Eq(a, b) end + if t == "Vector2" then return vector2Eq(a, b) end + if t == "CFrame" then return cframeEq(a, b) end + + -- NaN check (number case already handled above, this catches any remaining NaN-like values) + if a ~= a and b ~= b then return true end + + return false +end +``` + +**Key improvements:** + +- **Zero heap allocations**: No temporary arrays for CFrame/Vector3/Vector2. CFrame uses 12 direct `fuzzyEq` calls. Vector3/Vector2 use direct component access. +- **No `checkedKeys` set**: Second table loop only checks existence via `rawget`, doesn't compare values (already done in first loop). +- **Single `typeof` call path**: Check type of `a` once, verify `b` matches, then dispatch. No redundant `typeA == X and typeB == X` at every branch. +- `**rawequal` identity check first**: Catches same-reference case before any type checking. +- `**rawlen` fast-path for tables**: Different array lengths = not equal, no iteration needed. + +--- + +## Codegen Verification + +After each rewrite, run: + +```bash +luau-compile --remarks plugin/src/Reconciler/trueEquals.lua +luau-compile --remarks plugin/src/Reconciler/matching.lua +``` + +**trueEquals.lua** -- check that: + +- `fuzzyEq`, `color3Eq`, `vector2Eq`, `vector3Eq`, `cframeEq` are inlined into `trueEquals` +- No "can't inline" remarks on any helper + +**matching.lua** -- check that: + +- `countOwnDiffs`, `cacheVirtual`, `cacheStudio`, `removeMatched` are inlined at call sites +- `computePairCost` inlines or at minimum shows no optimization barriers +- No generic dispatch warnings for typed parameters + +Iterate on any functions that fail to inline (split if too large, simplify control flow). + +--- + +## Rust Forward Sync: [src/snapshot/matching.rs](src/snapshot/matching.rs) + +Add session struct and thread through. Rust already has the clean 2-function structure (`match_forward` + `compute_change_count` / `match_children_for_scoring`). + +```rust +pub struct MatchingSession { + cost_cache: RefCell>, +} +``` + +- Add `total_cost: u32` to `ForwardMatchResult` +- Add `session: &MatchingSession` param to `match_forward`, `compute_change_count`, `match_children_for_scoring` +- Cache `compute_change_count` results by `(snap.snapshot_id, tree_ref)`, only when `cost < best_so_far` +- Create session in [patch_compute.rs](src/snapshot/patch_compute.rs) `compute_patch_set`, thread through `compute_patch_set_internal` and `compute_children_patches` + +## Rust Syncback: [src/syncback/matching.rs](src/syncback/matching.rs) + +Same pattern. Session struct, `total_cost` on `MatchResult`, session param on all functions. + +- Create session in [dir.rs](src/snapshot_middleware/dir.rs) before `match_children` call + +## Test Updates: [matching_fixtures.rs](tests/tests/matching_fixtures.rs) + +Pass `&MatchingSession::new()` to `match_children` and `match_forward` calls. + +--- + +## Summary of Optimizations + + +| Change | Impact | +| ------------------------------------------------------- | ----------------------------------------------------------------- | +| **trueEquals**: single typeof dispatch | Eliminates redundant type checks at every branch | +| **trueEquals**: dedicated type helpers | Zero heap allocations (no temp arrays for CFrame/Vector3/Vector2) | +| **trueEquals**: DeepEqualsPureUnsafe table pattern | No checkedKeys set; rawlen/rawget/next | +| **trueEquals**: `--!strict`/`--!native`/`--!optimize 2` | Native codegen for the most-called function in matching | +| **matching**: local functions (not table methods) | Enables compiler inlining | +| **matching**: `--!strict`/`--!native`/`--!optimize 2` | Native codegen, type-guided optimization | +| **matching**: concrete types on all params/returns | Specialized native code paths | +| **matching**: remove pcall in grouping | Eliminates closure + pcall overhead per child | +| **matching**: 2-function recursion (from 4) | Simpler for optimizer, easier to reason about | +| **matching**: session caching (matchCache + costCache) | Eliminates redundant subtree scoring across recursion levels | +| **matching**: totalCost on matchChildren | Eliminates _computeChildrenCost re-iteration of matched pairs | +| **matching**: O(n) removeMatched | Linear vs quadratic element removal | + + diff --git a/.cursor/plans/syncback_via_rbxm_ad0231a7.plan.md b/.cursor/plans/syncback_via_rbxm_ad0231a7.plan.md new file mode 100644 index 00000000..8db42fca --- /dev/null +++ b/.cursor/plans/syncback_via_rbxm_ad0231a7.plan.md @@ -0,0 +1,394 @@ +--- +name: Syncback via RBXM +overview: Add a "Syncback" button to the plugin that serializes the DataModel per-service via SerializationService, sends the rbxm chunks to a totally segregated server endpoint, which tears down the entire serve system, runs CLI-equivalent clean-mode syncback, and restarts serve -- all within the same process. +todos: + - id: server-exit-reason + content: Modify LiveServer::start() to return ServerExitReason via tokio select + shutdown channel (src/web/mod.rs) + status: completed + - id: api-types + content: Add SyncbackPayload/ServiceChunk types to src/web/interface.rs + status: completed + - id: api-endpoint + content: Add POST /api/syncback endpoint handler in src/web/api.rs (fully segregated, no session validation) + status: completed + - id: serve-loop + content: Wrap ServeCommand::run() in a loop that handles SyncbackRequested (src/cli/serve.rs) + status: completed + - id: reconstruct-dom + content: "Implement run_live_syncback: reconstruct WeakDom from service chunks, run syncback_loop (clean mode, all CLI defaults), write files" + status: completed + - id: plugin-button + content: Add Syncback button (Danger style, layoutOrder 2) to NotConnected.lua + status: completed + - id: plugin-confirm + content: Add confirmation dialog (StudioPluginGui, danger-themed) in App/init.lua + status: completed + - id: plugin-serialize + content: Implement DataModel serialization per SYNCBACK_SERVICES list using SerializationService + status: completed + - id: plugin-api + content: Add standalone syncback HTTP post function (no ApiContext/session needed) + status: completed + - id: plugin-flow + content: "Wire up full flow in App/init.lua: confirm -> serialize -> POST /api/syncback -> done (fire and forget)" + status: completed + - id: test-helpers + content: Add post_api_syncback, wait_to_come_back_online, make_service_chunk, make_rbxl_from_chunks, run_cli_syncback_on_chunks to serve_util.rs + status: completed + - id: test-fixtures + content: Create live_syncback, live_syncback_place_ids, and live_syncback_complex serve-test fixtures + status: completed + - id: test-parity + content: "CLI parity tests (byte-for-byte): scripts, models+properties, refs, special names, duplicates, deep hierarchy, mixed types, comprehensive properties, multi-service" + status: completed + - id: test-validation + content: "Rejection tests: bad protocol, bad version, blocked place, unlisted place, whitelisted place, unrestricted" + status: completed + - id: test-lifecycle + content: "Lifecycle tests: server comes back functional (WebSocket + file watch), syncback twice with different data, replaces all old files" + status: completed + - id: test-roundtrip + content: "Round-trip identity test: build -> decompose -> live syncback -> rebuild -> compare instance trees" + status: completed +isProject: false +--- + +# Syncback via RBXM + +## Architecture + +```mermaid +sequenceDiagram + participant User + participant Plugin + participant Server + participant Filesystem + + User->>Plugin: Click "Syncback" button + Plugin->>Plugin: Show danger confirmation dialog + User->>Plugin: Confirm + Plugin->>Plugin: SerializeInstancesAsync per service + Plugin->>Server: POST /api/syncback (msgpack, segregated endpoint) + Server->>Plugin: 200 OK + Note over Server: Store payload, signal shutdown + Note over Server: Full teardown: drop ServeSession, VFS watcher, ChangeProcessor, WebSocket connections + Note over Server: Reconstruct WeakDom from service chunks + Note over Server: Run syncback_loop (clean mode, all CLI defaults) + Server->>Filesystem: Write files + Note over Server: Create fresh ServeSession + LiveServer + Note over Server: Bind to same address:port + Plugin->>Plugin: Poll GET /api/rojo until server is back + Plugin->>Server: Normal connect flow +``` + + + +## Server Changes + +### 1. Make `LiveServer::start()` return instead of blocking forever + +`[src/web/mod.rs](src/web/mod.rs)`: Change `start()` to return a `ServerExitReason` enum. Add a shutdown mechanism via `Arc` (wrapping a `Mutex>` + `tokio::sync::Notify`). The accept loop uses `tokio::select!` to watch for the notify signal alongside `listener.accept()`. When signaled, it breaks and returns. + +```rust +pub enum ServerExitReason { + SyncbackRequested(SyncbackPayload), +} + +pub struct SyncbackSignal { + payload: Mutex>, + notify: tokio::sync::Notify, +} +``` + +The `SyncbackSignal` is shared between `LiveServer` (which watches it) and the API handler (which deposits data into it). When the accept loop exits, the Tokio runtime drops, all spawned tasks (including active WebSocket connections) are cancelled, and the `Arc` drops -- tearing down the ChangeProcessor thread, VFS watcher, and all in-memory state. + +### 2. New segregated API endpoint: `POST /api/syncback` + +`[src/web/api.rs](src/web/api.rs)`: Totally separate from the connect/sync flow. No session ID needed -- the plugin doesn't need to be "connected" to use this. But it does validate version and place ID. + +Request body is msgpack-encoded: + +```rust +pub struct SyncbackRequest { + pub protocol_version: u64, + pub server_version: String, + pub place_id: Option, + pub services: Vec, +} +pub struct ServiceChunk { + pub class_name: String, + pub data: Vec, // raw rbxm bytes +} +// Payload passed to the serve loop (version info stripped after validation) +pub struct SyncbackPayload { + pub services: Vec, +} +``` + +Types live in `[src/web/interface.rs](src/web/interface.rs)`. + +Handler flow: + +1. Deserialize msgpack body into `SyncbackRequest` +2. **Validate protocol version** -- reject with 400 if `request.protocol_version != PROTOCOL_VERSION` +3. **Validate server version** -- reject with 400 if major.minor doesn't match `SERVER_VERSION` +4. **Validate place ID** against the project's `servePlaceIds` (whitelist) and `blockedPlaceIds` (blocklist). Reject with 403 if not allowed. +5. Extract `SyncbackPayload` (just the services), store in `SyncbackSignal` -> notify -> return 200 `{"status": "syncback_initiated"}` + +The response is sent before any teardown begins (the API handler is a separate tokio task from the accept loop). + +### 3. Wrap `ServeCommand::run()` in a loop + +`[src/cli/serve.rs](src/cli/serve.rs)`: The `run()` method becomes a loop. Each iteration creates a fresh VFS, ServeSession, and LiveServer. When `start()` returns `SyncbackRequested`, the old session is fully dropped (VFS watcher stops, ChangeProcessor thread joins, tree is freed), syncback runs, and the loop continues to create everything fresh: + +```rust +loop { + let (vfs, errors) = Vfs::new_default_with_errors(); + let session = Arc::new(ServeSession::new(vfs, project_path.clone(), Some(errors))?); + let server = LiveServer::new(session); + + match server.start(addr) { + ServerExitReason::SyncbackRequested(payload) => { + log::info!("Live syncback requested, running..."); + run_live_syncback(&project_path, payload)?; + log::info!("Syncback complete, restarting serve..."); + continue; + } + } +} +``` + +IP and port are resolved once before the loop (they don't change). + +### 4. Reconstruct WeakDom + run syncback (CLI-equivalent) + +New function `run_live_syncback(project_path: &Path, payload: SyncbackPayload) -> anyhow::Result<()>` in `src/cli/serve.rs`: + +1. **Build WeakDom** -- Create `DataModel` root, for each `ServiceChunk` parse rbxm via `rbx_binary::from_reader`, create a service instance of the correct class, deep-clone all children from the parsed dom into the target dom. Maintain a `HashMap` ref-map to fix up Ref properties across chunks. +2. **Run syncback** -- Identical to what CLI `atlas syncback` does: + - Create oneshot VFS: `Vfs::new_oneshot()` + - Load old tree: `ServeSession::new_oneshot(vfs, project_path)` + - Call `syncback_loop(vfs, &mut old_tree, new_dom, project, incremental: false)` -- clean mode + - Write: `fs_snapshot.write_to_vfs_parallel(base_path, vfs)` + - `refresh_git_index(base_path)` + +Same code path as CLI syncback, same defaults, same clean mode. The only difference is the WeakDom source. + +## Plugin Changes + +### 5. Syncback button on NotConnected page + +`[plugin/src/App/StatusPages/NotConnected.lua](plugin/src/App/StatusPages/NotConnected.lua)`: Add "Syncback" button with `style = "Danger"` and `layoutOrder = 2`, shift "Sync" to `layoutOrder = 3`: + +``` +[Settings] [Syncback] [Sync] +``` + +Fires `self.props.onSyncback`. + +### 6. Confirmation dialog + +`[plugin/src/App/init.lua](plugin/src/App/init.lua)`: Floating `StudioPluginGui` dialog (same pattern as `AcceptConfirm` in Confirming.lua): + +- Title: "Full Syncback" +- Message: "This will overwrite your project files with the current Studio state. This cannot be undone." +- Buttons: "Cancel" (Bordered) | "Syncback" (Danger) +- State: `self.state.showingSyncbackConfirm` + +### 7. DataModel serialization + +After confirmation, serialize per-service. Hardcode the same `VISIBLE_SERVICES` list from the Rust side: + +```lua +local SYNCBACK_SERVICES = { + "Lighting", "MaterialService", "ReplicatedFirst", "ReplicatedStorage", + "ServerScriptService", "ServerStorage", "SoundService", "StarterGui", + "StarterPack", "StarterPlayer", "Teams", "TextChatService", + "VoiceChatService", "Workspace", +} + +local services = {} +for _, className in SYNCBACK_SERVICES do + local service = game:FindService(className) + if service and #service:GetChildren() > 0 then + local buf = SerializationService:SerializeInstancesAsync(service:GetChildren()) + table.insert(services, { + className = className, + data = buf, -- msgpack-luau natively encodes buffer as bin format (zero-copy) + }) + end +end +``` + +The msgpack-luau library handles `buffer` types natively -- when `type(data) == "buffer"`, it encodes as msgpack bin 8/16/32 using `buffer.copy` directly. No string conversion needed. On the Rust side, `rmp_serde` deserializes msgpack bin into `Vec` automatically. + +### 8. Direct HTTP post (no ApiContext needed) + +Since this endpoint is fully segregated and doesn't need a session, the plugin just needs host:port from the address entry fields. A standalone function posts directly: + +```lua +local function postSyncback(host, port, services) + local url = ("http://%s:%s/api/syncback"):format(host, port) + local body = Http.msgpackEncode({ + protocolVersion = Config.protocolVersion, + serverVersion = Config.expectedServerVersionString, + placeId = game.PlaceId, + services = services, + }) + return Http.post(url, body):andThen(rejectFailedRequests) +end +``` + +No `ApiContext` instance needed. This keeps it completely independent from the connect flow. + +### 9. Full flow in App + +`[plugin/src/App/init.lua](plugin/src/App/init.lua)`: The `onSyncback` handler: + +1. Show confirmation dialog +2. On confirm: serialize DataModel per-service +3. POST to `/api/syncback` with the chunks +4. On 200: show a notification ("Syncback data sent. Server is processing.") +5. Done. Stay on NotConnected page. User clicks "Sync" manually whenever they're ready. + +Fire and forget -- the plugin's job ends after the POST succeeds. No polling, no auto-reconnect, no special state. + +## Key Design Decisions + +- **Fully segregated endpoint**: `/api/syncback` has zero coupling to the session/connect system. No session ID, no auth. Plugin fires and forgets. +- **Complete teardown**: When the signal fires, the entire serve system is destroyed -- Tokio runtime drops, ServeSession drops (joining the ChangeProcessor thread, stopping VFS watcher), all WebSocket connections are cancelled. Nothing survives from the previous serve. +- **CLI-equivalent syncback**: Uses `syncback_loop` with `incremental: false` (clean mode), non-interactive. Identical behavior to `atlas syncback --input file.rbxl` with all defaults. +- **Same process, same port**: The `atlas serve` process never exits. After syncback completes, a fresh serve starts on the same address:port. The user sees log output in their terminal throughout. +- **Danger styling**: Uses existing `Danger` theme (`#E74C3C` red) defined in `[plugin/src/App/Theme.lua](plugin/src/App/Theme.lua)`. +- **Per-service serialization**: 14 visible services from `VISIBLE_SERVICES` constant, matching both Rust and Lua sides. Cross-service Ref properties are lost in transit but syncback's `Rojo_Ref_`* system resolves refs from filesystem structure. + +## Tests + +The core invariant: **live syncback must produce byte-identical filesystem output to CLI syncback given the same input data.** If it doesn't, something is broken in the serialize -> transmit -> reconstruct -> syncback pipeline. Every test that verifies data accuracy runs BOTH live syncback and CLI syncback on the same input, then compares output with `assert_dirs_equal` (byte-for-byte file comparison from `[tests/rojo_test/roundtrip_util.rs](tests/rojo_test/roundtrip_util.rs)`). + +Tests go in a dedicated `tests/tests/live_syncback.rs`. They use `run_serve_test` from `[tests/rojo_test/serve_util.rs](tests/rojo_test/serve_util.rs)` plus the round-trip utilities. + +### Test helpers (add to `serve_util.rs`) + +`**post_api_syncback_raw`** -- Sends a raw msgpack body to `POST /api/syncback`. Returns the full `reqwest::blocking::Response` so tests can assert on status codes and error bodies. + +`**post_api_syncback`** -- Convenience wrapper that fills in correct protocol version + server version. Takes `place_id: Option` and `services: Vec`. Panics if response is not 200. + +`**wait_to_come_back_online`** -- Polls `GET /api/rojo` with exponential backoff (longer timeout than initial startup since syncback runs between teardown and restart). Returns the new `ServerInfoResponse`. Panics if server doesn't come back. + +`**make_service_chunk`** -- Builds a `ServiceChunk` from `InstanceBuilder` children: + +```rust +fn make_service_chunk(class_name: &str, children: Vec) -> ServiceChunk { + let mut dom = WeakDom::new(InstanceBuilder::new("DataModel")); + let root = dom.root_ref(); + for child in children { + dom.insert(root, child); + } + let refs: Vec = dom.root().children().to_vec(); + let mut buf = Vec::new(); + rbx_binary::to_writer(&mut buf, &dom, &refs).unwrap(); + ServiceChunk { class_name: class_name.to_string(), data: buf } +} +``` + +`**make_rbxl_from_chunks**` -- Builds a complete rbxl file from the same service chunks, for CLI parity testing. Creates a WeakDom with DataModel root + properly-classed service instances + children, serializes to rbxl. This gives CLI syncback the exact same input data as live syncback received. + +`**run_cli_syncback_on_chunks**` -- End-to-end helper: copies a fixture to a fresh temp dir, calls `make_rbxl_from_chunks`, runs `run_rojo_syncback_clean`, returns the temp dir path. Used by every parity test as the CLI reference. + +### Fixtures + +`**rojo-test/serve-tests/live_syncback/**` -- Multi-service project: + +``` +default.project.json5 # RS -> src/shared, SSS -> src/server, Workspace -> src/workspace +src/shared/OldModule.luau +src/server/OldScript.server.luau +src/workspace/OldPart.model.json5 +``` + +`**rojo-test/serve-tests/live_syncback_place_ids/**` -- Same layout + `servePlaceIds: [123]`, `blockedPlaceIds: [999]`. + +`**rojo-test/serve-tests/live_syncback_complex/**` -- All 14 visible services mapped to separate `$path` dirs. Used for comprehensive parity tests. + +### CLI parity tests (data accuracy) + +Every parity test follows this pattern: + +```rust +fn assert_live_matches_cli( + fixture_name: &str, + chunks: Vec, + place_id: Option, +) { + run_serve_test(fixture_name, |session, _| { + // 1. Live syncback + session.post_api_syncback(place_id, chunks.clone()); + session.wait_to_come_back_online(); + + // 2. CLI syncback on fresh copy of same fixture with same data + let cli_dir = run_cli_syncback_on_chunks(fixture_name, &chunks); + + // 3. Byte-for-byte comparison of every file + assert_dirs_equal(session.path(), &cli_dir); + }); +} +``` + +`**parity_scripts**` -- ModuleScript (`return {}`), Script/Server (`print("server")`), Script/Client (`print("client")`), LocalScript. Verifies Source content preserved exactly, correct file extensions (`.luau`, `.server.luau`, `.client.luau`, `.local.luau`), RunContext encoded in extension. + +`**parity_models_with_properties**` -- Model containing Parts with: Anchored (Bool), Size (Vector3), Position (Vector3), Color (Color3), BrickColor, Shape (Enum), Transparency (Float64). Verifies every property value round-trips through rbxm -> reconstruct -> syncback -> meta/model files identically. + +`**parity_ref_properties**` -- Model with PrimaryPart -> child Part. ObjectValue with Value -> sibling Part. Verifies `Rojo_Ref_*` attribute paths in meta/model files match CLI output exactly. + +`**parity_special_names**` -- Instances named `"What?"`, `"Key:Value"`, `"A/B"`, `""`, `'He said "hi"'`. Verifies slugified filenames (`What_`, `Key_Value`, `A_B`, etc.) and meta `name` fields match CLI. + +`**parity_duplicate_names**` -- 3x Folder named `"Data"`, 2x Script named `"Handler"`. Verifies dedup suffixes (`Data/`, `Data~2/`, `Data~3/`, `Handler.server.luau`, `Handler~2.server.luau`) and meta files match CLI. + +`**parity_deep_hierarchy**` -- 6 levels: Folder > Folder > Folder > Folder > Model > Part. Verifies nested directory structure is identical. + +`**parity_mixed_file_types**` -- StringValue (-> `.txt`), LocalizationTable (-> `.csv`), JSON data module (-> `.json5`), binary model (-> `.rbxm`), Folder (-> directory). All under one service. Byte-for-byte match with CLI. + +`**parity_comprehensive_properties**` -- Single Part with every testable property type set to non-default values: Bool, Int32, Float32, Float64, String, Vector2, Vector3, Vector2int16, Vector3int16, UDim, UDim2, Color3, Color3uint8, BrickColor, NumberRange, Rect, Enum, Font, Tags, Attributes (with nested types). Verifies meta.json5 property serialization matches CLI exactly. + +`**parity_multi_service**` -- Data under all 14 visible services simultaneously. Each service gets a unique set of instances (scripts, models, folders). Verifies every service's `$path` directory matches CLI output. + +`**parity_empty_services_mixed**` -- Some services have children, some don't, some are absent from the payload entirely. Verifies populated services are correct and empty/absent services result in clean directories. + +### Validation / rejection tests + +`**rejects_bad_protocol**` -- Send `protocol_version: 999`. Expect non-200. GET /api/rojo still returns same session ID (server unaffected). + +`**rejects_bad_version**` -- Send `server_version: "0.0.0"`. Expect non-200. Server unaffected. + +`**rejects_blocked_place**` -- Fixture has `blockedPlaceIds: [999]`. Send `placeId: 999`. Expect 403. Server unaffected. + +`**rejects_unlisted_place**` -- Fixture has `servePlaceIds: [123]`. Send `placeId: 456`. Expect 403. Server unaffected. + +`**allows_whitelisted_place**` -- `servePlaceIds: [123]`, send `placeId: 123`. Expect 200. Syncback runs, server restarts. + +`**allows_any_place_when_unrestricted**` -- No servePlaceIds/blockedPlaceIds. Send `placeId: 99999`. Expect 200. + +### Lifecycle / edge case tests + +`**server_comes_back_functional**` -- After live syncback, verify the restarted server is fully operational: GET /api/rojo works, can read instances via /api/read, WebSocket connects and receives patches when a file is modified on disk. + +`**syncback_twice_different_data**` -- Run live syncback with data A. Wait for restart. Run live syncback AGAIN with data B. Wait for restart. Verify filesystem matches data B (not A). Verify server is functional. Tests the serve loop handles multiple iterations. + +`**syncback_replaces_all_old_files**` -- Pre-populate the fixture with files that should NOT survive clean mode (orphan scripts, old models). After live syncback, verify none of the old files remain. Only files from the new data exist. + +### Round-trip identity test + +The ultimate accuracy test: + +`**roundtrip_build_syncback_rebuild**` -- Using the `live_syncback_complex` fixture: + +1. `atlas build` -> rbxl_A +2. Decompose rbxl_A into per-service chunks (parse with `rbx_binary::from_reader`, extract children of each service, re-serialize as rbxm chunks) +3. POST chunks to live syncback, wait for restart +4. `atlas build` -> rbxl_B +5. Parse both rbxl_A and rbxl_B into WeakDoms +6. Walk both trees recursively: compare instance names, classes, property values (using the same fuzzy equality from `variant_eq` for floats/CFrames), and child counts +7. Any divergence = test failure with detailed diff + +This tests the full pipeline: build -> decompose into service chunks -> transmit -> reconstruct -> syncback -> build again. If the output matches, the live syncback system preserves 100% data fidelity. \ No newline at end of file diff --git a/plugin/src/Reconciler/hydrate.lua b/plugin/src/Reconciler/hydrate.lua index 8be0319d..6ffd9702 100644 --- a/plugin/src/Reconciler/hydrate.lua +++ b/plugin/src/Reconciler/hydrate.lua @@ -13,7 +13,7 @@ local Log = require(Packages.Log) local invariant = require(script.Parent.Parent.invariant) local Matching = require(script.Parent.matching) -local function hydrate(instanceMap, virtualInstances, rootId, rootInstance) +local function hydrate(instanceMap, virtualInstances, rootId, rootInstance, session) local virtualInstance = virtualInstances[rootId] if virtualInstance == nil then @@ -40,12 +40,14 @@ local function hydrate(instanceMap, virtualInstances, rootId, rootInstance) end end - -- Use the 3-pass matching algorithm to pair virtual ↔ studio children - local result = Matching.matchChildren(validVirtualIds, existingChildren, virtualInstances) + local result = Matching.matchChildren( + session, validVirtualIds, existingChildren, virtualInstances, + rootId, rootInstance + ) -- Recursively hydrate matched pairs for _, pair in result.matched do - hydrate(instanceMap, virtualInstances, pair.virtualId, pair.studioInstance) + hydrate(instanceMap, virtualInstances, pair.virtualId, pair.studioInstance, session) end -- Log unmatched virtual children (no Studio counterpart found) diff --git a/plugin/src/Reconciler/init.lua b/plugin/src/Reconciler/init.lua index fe51478c..f7269a4f 100644 --- a/plugin/src/Reconciler/init.lua +++ b/plugin/src/Reconciler/init.lua @@ -33,9 +33,9 @@ function Reconciler:applyPatch(patch) return unappliedPatch end -function Reconciler:hydrate(virtualInstances, rootId, rootInstance) +function Reconciler:hydrate(virtualInstances, rootId, rootInstance, session) Timer.start("Reconciler:hydrate") - local result = hydrate(self.__instanceMap, virtualInstances, rootId, rootInstance) + local result = hydrate(self.__instanceMap, virtualInstances, rootId, rootInstance, session) Timer.stop() return result diff --git a/plugin/src/Reconciler/matching.lua b/plugin/src/Reconciler/matching.lua index 2a703dee..ea5fd386 100644 --- a/plugin/src/Reconciler/matching.lua +++ b/plugin/src/Reconciler/matching.lua @@ -1,20 +1,22 @@ +--!strict +--!native +--!optimize 2 --[[ Instance matching algorithm for the plugin. Pairs server virtual instances to Studio instances during hydration - by minimizing total reconciler changes. + by minimizing total reconciler changes. matchChildren is fundamentally + computeSubtreeCost -- it finds the cost-minimizing pairing of children + and returns the total cost. The match assignments are a bonus output. - Algorithm per parent: - 1. Ref pin: parent's Ref properties confirm identity of specific children - 2. Group remaining by (Name, ClassName) -- 1:1 groups instant-match - 3. Ambiguous groups: recursive change-count scoring + greedy assignment + Two-function mutual recursion: + matchChildren: given two child sets, find optimal pairing + totalCost + computePairCost: given one virtual + one studio instance, compute + total cost including subtree (calls matchChildren for children) - The change count for a pair = how many things the reconciler would need - to touch to turn instance A into instance B, including the entire subtree. - - Performance: all property decoding and Studio reads are pre-computed - BEFORE the N×M scoring loop. The inner comparison is pure table - lookups + trueEquals, with zero pcall/decode in the hot path. + A MatchingSession holds caches keyed by instance identity so that + recursive scoring results are reused when hydrate recurses into the + same children that were already scored. ]] local Packages = script.Parent.Parent.Parent.Packages @@ -22,27 +24,69 @@ local RbxDom = require(Packages.RbxDom) local trueEquals = require(script.Parent.trueEquals) -local UNMATCHED_PENALTY = 10000 +local UNMATCHED_PENALTY: number = 10000 -local Matching = {} +-- ================================================================ +-- Types +-- ================================================================ + +type MatchPair = { + virtualId: string, + studioInstance: Instance, +} + +type MatchResult = { + matched: { MatchPair }, + unmatchedVirtual: { string }, + unmatchedStudio: { Instance }, + totalCost: number, +} + +type MatchingSession = { + matchCache: { [string]: { [Instance]: MatchResult } }, + costCache: { [string]: { [Instance]: number } }, +} + +type VCache = { + props: { [string]: any }, + extraProps: { string }?, + tags: { [string]: boolean }?, + attrs: { [string]: any }?, + childCount: number, +} + +type SCache = { + instance: Instance, + props: { [string]: any }, + tags: { [string]: boolean }?, + attrs: { [string]: any }?, + children: { Instance }, + childCount: number, +} + +type ScoredPair = { + vi: number, + si: number, + cost: number, + idx: number, +} + +-- ================================================================ +-- Session constructor +-- ================================================================ + +local function newSession(): MatchingSession + return { + matchCache = {}, + costCache = {}, + } +end -- ================================================================ -- Pre-computation helpers (called once per instance per group) -- ================================================================ ---[[ - Pre-decode all virtual instance properties into native Roblox values. - Called once per virtual instance, reused across all M studio comparisons. - - Returns: { - props: { [propName]: decodedValue }, - extraProps: { propName, ... } | nil, -- props not in classKeys (rare) - tags: { [tag]: true } | nil, - attrs: { [key]: value } | nil, - childCount: number, - } -]] -function Matching._cacheVirtual(vInst: any, classKeys: any) +local function cacheVirtual(vInst: any, classKeys: any): VCache local decoded: { [string]: any } = {} local extraProps: { string }? = nil local vProps = vInst.Properties @@ -71,7 +115,6 @@ function Matching._cacheVirtual(vInst: any, classKeys: any) end end - -- Pre-decode tags into a set local decodedTags: { [string]: boolean }? = nil if vProps and vProps.Tags then local ok, tags = RbxDom.EncodedValue.decode(vProps.Tags) @@ -83,7 +126,6 @@ function Matching._cacheVirtual(vInst: any, classKeys: any) end end - -- Pre-decode attributes local decodedAttrs: { [string]: any }? = nil if vProps and vProps.Attributes then local ok, attrs = RbxDom.EncodedValue.decode(vProps.Attributes) @@ -101,23 +143,9 @@ function Matching._cacheVirtual(vInst: any, classKeys: any) } end ---[[ - Pre-read all comparable properties from a Studio instance. - Called once per Studio instance, reused across all N virtual comparisons. - - Returns: { - props: { [propName]: value }, - readable: { [propName]: true }, -- tracks which reads succeeded - tags: { [tag]: true } | nil, - attrs: { [key]: value } | nil, - children: { Instance } | nil, - childCount: number, - } -]] -function Matching._cacheStudio(studioInstance: Instance, classKeys: any, extraPropNames: { string }?) +local function cacheStudio(studioInstance: Instance, classKeys: any, extraPropNames: { string }?): SCache local props: { [string]: any } = {} - -- Read all comparison-key properties for _, propName in classKeys.propNames do local ok, value = pcall(function() return (studioInstance :: any)[propName] @@ -127,7 +155,6 @@ function Matching._cacheStudio(studioInstance: Instance, classKeys: any, extraPr end end - -- Read any extra properties the virtual side has (not in classKeys) if extraPropNames then for _, propName in extraPropNames do if props[propName] ~= nil then @@ -142,7 +169,6 @@ function Matching._cacheStudio(studioInstance: Instance, classKeys: any, extraPr end end - -- Read tags local tags: { [string]: boolean }? = nil local tagsOk, studioTags = pcall(function() return studioInstance:GetTags() @@ -154,7 +180,6 @@ function Matching._cacheStudio(studioInstance: Instance, classKeys: any, extraPr end end - -- Read attributes local attrs: { [string]: any }? = nil local attrsOk, studioAttrs = pcall(function() return studioInstance:GetAttributes() @@ -163,9 +188,8 @@ function Matching._cacheStudio(studioInstance: Instance, classKeys: any, extraPr attrs = studioAttrs end - -- Read children - local children: { Instance }? = nil - local childCount = 0 + local children: { Instance } = {} + local childCount: number = 0 local childOk, studioKids = pcall(function() return studioInstance:GetChildren() end) @@ -175,6 +199,7 @@ function Matching._cacheStudio(studioInstance: Instance, classKeys: any, extraPr end return { + instance = studioInstance, props = props, tags = tags, attrs = attrs, @@ -184,23 +209,15 @@ function Matching._cacheStudio(studioInstance: Instance, classKeys: any, extraPr end -- ================================================================ --- Hot-path scoring (ZERO pcall, ZERO decode, ZERO reflection lookup) +-- Hot-path scoring (ZERO decode, ZERO reflection lookup) -- ================================================================ ---[[ - Count own property diffs between pre-computed virtual and Studio caches. - Single loop over the class's comparable properties. Pure table lookups - and trueEquals calls -- no pcall, no decode, no reflection queries. -]] -function Matching._countOwnDiffsCached(vCache: any, sCache: any, classKeys: any): number - local cost = 0 +local function countOwnDiffs(vCache: VCache, sCache: SCache, classKeys: any): number + local cost: number = 0 local vProps = vCache.props local sProps = sCache.props local defaults = classKeys.defaults - -- Single loop over all comparable property names for this class. - -- For each property, the effective virtual value is the explicit value - -- if present, otherwise the class default (syncback strips defaults). for _, propName in classKeys.propNames do local vVal = vProps[propName] if vVal == nil then @@ -212,8 +229,6 @@ function Matching._countOwnDiffsCached(vCache: any, sCache: any, classKeys: any) end end - -- Handle virtual properties not in the class comparison set (rare: - -- unknown class or properties from a newer API version). if vCache.extraProps then for _, propName in vCache.extraProps do if not trueEquals(vProps[propName], sProps[propName]) then @@ -222,7 +237,6 @@ function Matching._countOwnDiffsCached(vCache: any, sCache: any, classKeys: any) end end - -- Tags: symmetric set diff (pre-computed sets) local vTags = vCache.tags local sTags = sCache.tags if vTags and sTags then @@ -238,7 +252,6 @@ function Matching._countOwnDiffsCached(vCache: any, sCache: any, classKeys: any) end end - -- Attributes: symmetric map diff (pre-decoded / pre-read) local vAttrs = vCache.attrs local sAttrs = sCache.attrs if vAttrs and sAttrs then @@ -254,7 +267,6 @@ function Matching._countOwnDiffsCached(vCache: any, sCache: any, classKeys: any) end end - -- Children count diff if vCache.childCount ~= sCache.childCount then cost += 1 end @@ -263,154 +275,155 @@ function Matching._countOwnDiffsCached(vCache: any, sCache: any, classKeys: any) end -- ================================================================ --- Recursive scoring (children matching) +-- Utilities -- ================================================================ ---[[ - Compute the recursive children cost for a matched pair. - Uses sCache.children (pre-read) to avoid repeated GetChildren(). -]] -function Matching._computeChildrenCost( - vInst: any, - sCache: any, - virtualInstances: { [string]: any }, - bestSoFar: number -): number - local vChildren = vInst.Children - local studioKids = sCache.children - - if not vChildren or #vChildren == 0 then - if studioKids then - return #studioKids * UNMATCHED_PENALTY - end - return 0 - end - - if not studioKids then - return 0 - end - - local validVChildren: { string } = {} - for _, childId in vChildren do - if virtualInstances[childId] then - table.insert(validVChildren, childId) +local function removeMatched(arr: { any }, matchedIndices: { [number]: boolean }): () + local write: number = 1 + for read = 1, #arr do + if not matchedIndices[read] then + arr[write] = arr[read] + write += 1 end end - - local childResult = Matching.matchChildren(validVChildren, studioKids, virtualInstances) - - local cost = 0 - for _, pair in childResult.matched do - cost += Matching._computeChangeCount(pair.virtualId, pair.studioInstance, virtualInstances, bestSoFar - cost) - if cost >= bestSoFar then - return cost - end + for i = write, #arr do + arr[i] = nil end - - cost += (#childResult.unmatchedVirtual + #childResult.unmatchedStudio) * UNMATCHED_PENALTY - return cost end ---[[ - Compute total change count between a virtual instance and a Studio instance. - Includes own property diffs + recursive children cost. - Early-exits if cost reaches or exceeds bestSoFar. +-- ================================================================ +-- Core: 2-function mutual recursion +-- ================================================================ - Used for recursive 1:1 scoring (matched child pairs). For the N×M - ambiguous-group loop, matchChildren uses pre-computed caches directly. -]] -function Matching._computeChangeCount( +local matchChildren -- forward declare + +local function computePairCost( + session: MatchingSession, virtualId: string, studioInstance: Instance, virtualInstances: { [string]: any }, bestSoFar: number ): number + local vc = session.costCache[virtualId] + if vc then + local cached = vc[studioInstance] + if cached ~= nil then + return cached + end + end + local vInst = virtualInstances[virtualId] if not vInst then return UNMATCHED_PENALTY end - -- Build caches inline (1:1 call, no N×M savings needed here, - -- but classKeys is still cached per class so it's free). local classKeys = RbxDom.getClassComparisonKeys(vInst.ClassName) - local vCache = Matching._cacheVirtual(vInst, classKeys) - local sCache = Matching._cacheStudio(studioInstance, classKeys, vCache.extraProps) + local vCache = cacheVirtual(vInst, classKeys) + local sCache = cacheStudio(studioInstance, classKeys, vCache.extraProps) - local cost = Matching._countOwnDiffsCached(vCache, sCache, classKeys) + local cost: number = countOwnDiffs(vCache, sCache, classKeys) if cost >= bestSoFar then return cost end - cost += Matching._computeChildrenCost(vInst, sCache, virtualInstances, bestSoFar - cost) + local vChildren = vInst.Children + local studioKids = sCache.children + + if (not vChildren or #vChildren == 0) and #studioKids == 0 then + -- leaf + elseif not vChildren or #vChildren == 0 then + cost += #studioKids * UNMATCHED_PENALTY + elseif #studioKids == 0 then + for _, childId in vChildren do + if virtualInstances[childId] then + cost += UNMATCHED_PENALTY + end + end + else + local validVChildren: { string } = {} + for _, childId in vChildren do + if virtualInstances[childId] then + table.insert(validVChildren, childId) + end + end + if #validVChildren > 0 then + local childResult: MatchResult = matchChildren( + session, validVChildren, studioKids, virtualInstances, + virtualId, studioInstance + ) + cost += childResult.totalCost + else + cost += #studioKids * UNMATCHED_PENALTY + end + end + + if cost < bestSoFar then + if not session.costCache[virtualId] then + session.costCache[virtualId] = {} + end + session.costCache[virtualId][studioInstance] = cost + end + return cost end --- ================================================================ --- Entry point --- ================================================================ - ---[[ - Match virtual children to Studio children under a parent. - - virtualChildren: array of virtual instance IDs - studioChildren: array of Studio Instance objects - virtualInstances: map of virtual ID → virtual instance data - - Returns {matched, unmatchedVirtual, unmatchedStudio} -]] -function Matching.matchChildren( +matchChildren = function( + session: MatchingSession, virtualChildren: { string }, studioChildren: { Instance }, - virtualInstances: { [string]: any } -) - local matched: { { virtualId: string, studioInstance: Instance } } = {} + virtualInstances: { [string]: any }, + parentVirtualId: string?, + parentStudioInstance: Instance? +): MatchResult + -- Cache lookup + if parentVirtualId and parentStudioInstance then + local pc = session.matchCache[parentVirtualId] + if pc then + local cached = pc[parentStudioInstance] + if cached then + return cached + end + end + end + + local matched: { MatchPair } = {} local remainingVirtual: { string } = table.clone(virtualChildren) local remainingStudio: { Instance } = table.clone(studioChildren) -- ============================================================ - -- Fast-path 1: Ref pin (confirmed identity, highest priority) - -- ============================================================ - -- (Ref pins are handled implicitly during hydration -- the parent - -- is already matched, and its Ref properties resolve via instanceMap - -- after children are matched. For initial hydration the instanceMap - -- is empty, so explicit Ref pinning at this stage is a no-op. - -- Ref correctness is maintained by the change-count scoring which - -- counts Ref property diffs.) - - -- ============================================================ - -- Fast-path 2: Group by (Name, ClassName) -- 1:1 instant match + -- Group by (Name, ClassName) -- direct property access, no pcall -- ============================================================ local vByKey: { [string]: { number } } = {} for i, id in remainingVirtual do local vInst = virtualInstances[id] if vInst then - local key = vInst.Name .. "\0" .. vInst.ClassName - if not vByKey[key] then - vByKey[key] = {} + local key: string = vInst.Name .. "\0" .. vInst.ClassName + local group = vByKey[key] + if not group then + group = {} + vByKey[key] = group end - table.insert(vByKey[key], i) + table.insert(group, i) end end local sByKey: { [string]: { number } } = {} for i, inst in remainingStudio do - local ok, name, class = pcall(function() - return inst.Name, inst.ClassName - end) - if ok then - local key = name .. "\0" .. class - if not sByKey[key] then - sByKey[key] = {} - end - table.insert(sByKey[key], i) + local key: string = inst.Name .. "\0" .. inst.ClassName + local group = sByKey[key] + if not group then + group = {} + sByKey[key] = group end + table.insert(group, i) end - -- Track which indices to remove (collected, then removed in reverse) + -- ============================================================ + -- 1:1 instant match + -- ============================================================ local matchedV: { [number]: boolean } = {} local matchedS: { [number]: boolean } = {} - -- 1:1 groups: instant match for key, vIndices in vByKey do local sIndices = sByKey[key] if sIndices and #vIndices == 1 and #sIndices == 1 then @@ -426,204 +439,213 @@ function Matching.matchChildren( end end - -- Remove 1:1 matched from remaining lists - Matching._removeMatched(remainingVirtual, matchedV) - Matching._removeMatched(remainingStudio, matchedS) - - -- Early exit if nothing left - if #remainingVirtual == 0 or #remainingStudio == 0 then - return { - matched = matched, - unmatchedVirtual = remainingVirtual, - unmatchedStudio = remainingStudio, - } - end + removeMatched(remainingVirtual, matchedV) + removeMatched(remainingStudio, matchedS) -- ============================================================ -- Ambiguous groups: change-count scoring + greedy assignment -- ============================================================ - -- Rebuild groups from remaining (only ambiguous ones are left) - local vByKey2: { [string]: { number } } = {} - for i, id in remainingVirtual do - local vInst = virtualInstances[id] - if vInst then - local key = vInst.Name .. "\0" .. vInst.ClassName - if not vByKey2[key] then - vByKey2[key] = {} - end - table.insert(vByKey2[key], i) - end - end - - local sByKey2: { [string]: { number } } = {} - for i, inst in remainingStudio do - local ok, name, class = pcall(function() - return inst.Name, inst.ClassName - end) - if ok then - local key = name .. "\0" .. class - if not sByKey2[key] then - sByKey2[key] = {} - end - table.insert(sByKey2[key], i) - end - end - - local matchedV2: { [number]: boolean } = {} - local matchedS2: { [number]: boolean } = {} - - for key, vIndices in vByKey2 do - local sIndices = sByKey2[key] - if not sIndices then - continue - end - - -- -------------------------------------------------------- - -- Pre-compute caches for this ambiguous group. - -- All instances share the same ClassName (grouped by key). - -- -------------------------------------------------------- - local firstVInst = virtualInstances[remainingVirtual[vIndices[1]]] - if not firstVInst then - continue - end - local classKeys = RbxDom.getClassComparisonKeys(firstVInst.ClassName) - - -- Pre-decode all virtual instances in this group - local vCaches: { [number]: any } = {} - local allExtraProps: { [string]: boolean }? = nil - for _, vi in vIndices do - if matchedV2[vi] then - continue - end - local vInst = virtualInstances[remainingVirtual[vi]] - if not vInst then - continue - end - local vCache = Matching._cacheVirtual(vInst, classKeys) - vCaches[vi] = vCache - -- Collect extra prop names across all virtuals for studio reads - if vCache.extraProps then - if not allExtraProps then - allExtraProps = {} - end - for _, propName in vCache.extraProps do - allExtraProps[propName] = true + if #remainingVirtual > 0 and #remainingStudio > 0 then + local vByKey2: { [string]: { number } } = {} + for i, id in remainingVirtual do + local vInst = virtualInstances[id] + if vInst then + local key: string = vInst.Name .. "\0" .. vInst.ClassName + local group = vByKey2[key] + if not group then + group = {} + vByKey2[key] = group end + table.insert(group, i) end end - -- Convert extra props set to array for studio caching - local extraPropNamesArray: { string }? = nil - if allExtraProps then - extraPropNamesArray = {} - for propName in allExtraProps do - table.insert(extraPropNamesArray, propName) + local sByKey2: { [string]: { number } } = {} + for i, inst in remainingStudio do + local key: string = inst.Name .. "\0" .. inst.ClassName + local group = sByKey2[key] + if not group then + group = {} + sByKey2[key] = group end + table.insert(group, i) end - -- Pre-read all Studio instances in this group - local sCaches: { [number]: any } = {} - for _, si in sIndices do - if not matchedS2[si] then - sCaches[si] = Matching._cacheStudio(remainingStudio[si], classKeys, extraPropNamesArray) - end - end + local matchedV2: { [number]: boolean } = {} + local matchedS2: { [number]: boolean } = {} - -- -------------------------------------------------------- - -- Score all (A, B) pairs using pre-computed caches. - -- Own diffs = pure table lookups (FAST). - -- Children cost = recursive (only if own diffs < bestSoFar). - -- -------------------------------------------------------- - local pairs: { { vi: number, si: number, cost: number, idx: number } } = {} - local pairIdx = 0 - local bestSoFar = math.huge - - for _, vi in vIndices do - if matchedV2[vi] then - continue - end - local vCache = vCaches[vi] - if not vCache then + for key, vIndices in vByKey2 do + local sIndices = sByKey2[key] + if not sIndices then continue end - for _, si in sIndices do - if matchedS2[si] then + local firstVInst = virtualInstances[remainingVirtual[vIndices[1]]] + if not firstVInst then + continue + end + local classKeys = RbxDom.getClassComparisonKeys(firstVInst.ClassName) + + local vCaches: { [number]: VCache } = {} + local allExtraProps: { [string]: boolean }? = nil + for _, vi in vIndices do + if matchedV2[vi] then continue end - local sCache = sCaches[si] - if not sCache then + local vInst = virtualInstances[remainingVirtual[vi]] + if not vInst then continue end - - pairIdx += 1 - - -- Own diffs: pure table lookups + trueEquals - local cost = Matching._countOwnDiffsCached(vCache, sCache, classKeys) - - if cost < bestSoFar then - -- Recursive children cost - local vInst = virtualInstances[remainingVirtual[vi]] - if vInst then - cost += Matching._computeChildrenCost(vInst, sCache, virtualInstances, bestSoFar - cost) + local vCache = cacheVirtual(vInst, classKeys) + vCaches[vi] = vCache + if vCache.extraProps then + if not allExtraProps then + allExtraProps = {} + end + for _, propName in vCache.extraProps do + allExtraProps[propName] = true end end + end - table.insert(pairs, { vi = vi, si = si, cost = cost, idx = pairIdx }) - if cost < bestSoFar then - bestSoFar = cost + local extraPropNamesArray: { string }? = nil + if allExtraProps then + extraPropNamesArray = {} + for propName in allExtraProps do + table.insert(extraPropNamesArray, propName) end end + + local sCaches: { [number]: SCache } = {} + for _, si in sIndices do + if not matchedS2[si] then + sCaches[si] = cacheStudio(remainingStudio[si], classKeys, extraPropNamesArray) + end + end + + -- Score all (A, B) pairs + local scoredPairs: { ScoredPair } = {} + local pairIdx: number = 0 + local bestSoFar: number = math.huge + + for _, vi in vIndices do + if matchedV2[vi] then + continue + end + local vCache = vCaches[vi] + if not vCache then + continue + end + + for _, si in sIndices do + if matchedS2[si] then + continue + end + local sCache = sCaches[si] + if not sCache then + continue + end + + pairIdx += 1 + + local cost: number = countOwnDiffs(vCache, sCache, classKeys) + + if cost < bestSoFar then + local vInst = virtualInstances[remainingVirtual[vi]] + if vInst then + local vChildren = vInst.Children + local studioKids = sCache.children + + if (not vChildren or #vChildren == 0) and #studioKids == 0 then + -- leaf + elseif not vChildren or #vChildren == 0 then + cost += #studioKids * UNMATCHED_PENALTY + elseif #studioKids == 0 then + for _, childId in vChildren do + if virtualInstances[childId] then + cost += UNMATCHED_PENALTY + end + end + else + local validVChildren: { string } = {} + for _, childId in vChildren do + if virtualInstances[childId] then + table.insert(validVChildren, childId) + end + end + if #validVChildren > 0 then + local childResult: MatchResult = matchChildren( + session, validVChildren, studioKids, virtualInstances, + remainingVirtual[vi], sCache.instance + ) + cost += childResult.totalCost + else + cost += #studioKids * UNMATCHED_PENALTY + end + end + end + end + + table.insert(scoredPairs, { vi = vi, si = si, cost = cost, idx = pairIdx }) + if cost < bestSoFar then + bestSoFar = cost + end + end + end + + table.sort(scoredPairs, function(a: ScoredPair, b: ScoredPair): boolean + if a.cost ~= b.cost then + return a.cost < b.cost + end + return a.idx < b.idx + end) + + for _, pair in scoredPairs do + if matchedV2[pair.vi] or matchedS2[pair.si] then + continue + end + table.insert(matched, { + virtualId = remainingVirtual[pair.vi], + studioInstance = remainingStudio[pair.si], + }) + matchedV2[pair.vi] = true + matchedS2[pair.si] = true + end end - -- Stable sort by cost ascending (Luau sort is not stable, - -- so break ties by insertion index to preserve child order) - table.sort(pairs, function(a, b) - if a.cost ~= b.cost then - return a.cost < b.cost - end - return a.idx < b.idx - end) - - -- Greedy assign - for _, pair in pairs do - if matchedV2[pair.vi] or matchedS2[pair.si] then - continue - end - table.insert(matched, { - virtualId = remainingVirtual[pair.vi], - studioInstance = remainingStudio[pair.si], - }) - matchedV2[pair.vi] = true - matchedS2[pair.si] = true - end + removeMatched(remainingVirtual, matchedV2) + removeMatched(remainingStudio, matchedS2) end - -- Remove matched from remaining - Matching._removeMatched(remainingVirtual, matchedV2) - Matching._removeMatched(remainingStudio, matchedS2) + -- ============================================================ + -- Compute totalCost for ALL matched pairs (session cache helps) + -- ============================================================ + local totalCost: number = 0 + for _, pair in matched do + totalCost += computePairCost( + session, pair.virtualId, pair.studioInstance, virtualInstances, math.huge + ) + end + totalCost += (#remainingVirtual + #remainingStudio) * UNMATCHED_PENALTY - return { + local result: MatchResult = { matched = matched, unmatchedVirtual = remainingVirtual, unmatchedStudio = remainingStudio, + totalCost = totalCost, } + + if parentVirtualId and parentStudioInstance then + if not session.matchCache[parentVirtualId] then + session.matchCache[parentVirtualId] = {} + end + session.matchCache[parentVirtualId][parentStudioInstance] = result + end + + return result end ---[[ - Remove items from an array by index set (removes in reverse order). -]] -function Matching._removeMatched(arr: { any }, matchedIndices: { [number]: boolean }) - local toRemove: { number } = {} - for idx in matchedIndices do - table.insert(toRemove, idx) - end - table.sort(toRemove, function(a, b) - return a > b - end) - for _, idx in toRemove do - table.remove(arr, idx) - end -end - -return Matching +return { + newSession = newSession, + matchChildren = matchChildren, +} diff --git a/plugin/src/Reconciler/trueEquals.lua b/plugin/src/Reconciler/trueEquals.lua index 43ccaed2..bc736063 100644 --- a/plugin/src/Reconciler/trueEquals.lua +++ b/plugin/src/Reconciler/trueEquals.lua @@ -1,3 +1,6 @@ +--!strict +--!native +--!optimize 2 --[[ Shared value equality check used by both the diff and matching modules. @@ -6,98 +9,96 @@ use this same function. ]] -local function fuzzyEq(a: number, b: number, epsilon: number): boolean +local NULL_REF: string = "00000000000000000000000000000000" +local EPSILON: number = 0.0001 + +local function fuzzyEq(a: number, b: number): boolean local diff = math.abs(a - b) - -- Use both absolute and relative epsilon for better precision handling - -- Absolute: for small numbers near zero - -- Relative: for larger numbers where absolute epsilon is too tight local maxVal = math.max(math.abs(a), math.abs(b), 1) - return diff < epsilon or diff < maxVal * epsilon + return diff < EPSILON or diff < maxVal * EPSILON end -local function trueEquals(a, b): boolean - -- Exit early for simple equality values - if a == b then +local function color3Eq(a: Color3, b: Color3): boolean + return math.floor(a.R * 255) == math.floor(b.R * 255) + and math.floor(a.G * 255) == math.floor(b.G * 255) + and math.floor(a.B * 255) == math.floor(b.B * 255) +end + +local function vector2Eq(a: Vector2, b: Vector2): boolean + return fuzzyEq(a.X, b.X) and fuzzyEq(a.Y, b.Y) +end + +local function vector3Eq(a: Vector3, b: Vector3): boolean + return fuzzyEq(a.X, b.X) and fuzzyEq(a.Y, b.Y) and fuzzyEq(a.Z, b.Z) +end + +local function cframeEq(a: CFrame, b: CFrame): boolean + local ax, ay, az, aR00, aR01, aR02, aR10, aR11, aR12, aR20, aR21, aR22 = a:GetComponents() + local bx, by, bz, bR00, bR01, bR02, bR10, bR11, bR12, bR20, bR21, bR22 = b:GetComponents() + return fuzzyEq(ax, bx) and fuzzyEq(ay, by) and fuzzyEq(az, bz) + and fuzzyEq(aR00, bR00) and fuzzyEq(aR01, bR01) and fuzzyEq(aR02, bR02) + and fuzzyEq(aR10, bR10) and fuzzyEq(aR11, bR11) and fuzzyEq(aR12, bR12) + and fuzzyEq(aR20, bR20) and fuzzyEq(aR21, bR21) and fuzzyEq(aR22, bR22) +end + +local function trueEquals(a: any, b: any): boolean + if rawequal(a, b) then return true end - -- Treat nil and { Ref = "000...0" } as equal - if - (a == nil and type(b) == "table" and b.Ref == "00000000000000000000000000000000") - or (b == nil and type(a) == "table" and a.Ref == "00000000000000000000000000000000") - then + if a == nil then + return type(b) == "table" and rawget(b, "Ref") == NULL_REF + end + if b == nil then + return type(a) == "table" and rawget(a, "Ref") == NULL_REF + end + + local t = typeof(a) + if t ~= typeof(b) then + if t == "number" and typeof(b) == "EnumItem" then + return a == (b :: EnumItem).Value + end + if t == "EnumItem" and typeof(b) == "number" then + return (a :: EnumItem).Value == b + end + return false + end + + if t == "table" then + if rawlen(a) ~= rawlen(b) then + return false + end + for k, v in next, a do + local ov = rawget(b, k) + if ov == nil or not trueEquals(v, ov) then + return false + end + end + for k in next, b do + if rawget(a, k) == nil then + return false + end + end return true end - local typeA, typeB = typeof(a), typeof(b) + if t == "number" then + return fuzzyEq(a, b) + end + if t == "Color3" then + return color3Eq(a, b) + end + if t == "Vector3" then + return vector3Eq(a, b) + end + if t == "Vector2" then + return vector2Eq(a, b) + end + if t == "CFrame" then + return cframeEq(a, b) + end - -- For tables, try recursive deep equality - if typeA == "table" and typeB == "table" then - local checkedKeys = {} - for key, value in a do - checkedKeys[key] = true - if not trueEquals(value, b[key]) then - return false - end - end - for key, value in b do - if checkedKeys[key] then - continue - end - if not trueEquals(value, a[key]) then - return false - end - end - return true - - -- For NaN, check if both values are not equal to themselves - elseif a ~= a and b ~= b then - return true - - -- For numbers, compare with epsilon of 0.0001 to avoid floating point inequality - elseif typeA == "number" and typeB == "number" then - return fuzzyEq(a, b, 0.0001) - - -- For EnumItem->number, compare the EnumItem's value - elseif typeA == "number" and typeB == "EnumItem" then - return a == b.Value - elseif typeA == "EnumItem" and typeB == "number" then - return a.Value == b - - -- For Color3s, compare to RGB ints to avoid floating point inequality - elseif typeA == "Color3" and typeB == "Color3" then - local aR, aG, aB = math.floor(a.R * 255), math.floor(a.G * 255), math.floor(a.B * 255) - local bR, bG, bB = math.floor(b.R * 255), math.floor(b.G * 255), math.floor(b.B * 255) - return aR == bR and aG == bG and aB == bB - - -- For CFrames, compare to components with epsilon of 0.0001 to avoid floating point inequality - elseif typeA == "CFrame" and typeB == "CFrame" then - local aComponents, bComponents = { a:GetComponents() }, { b:GetComponents() } - for i, aComponent in aComponents do - if not fuzzyEq(aComponent, bComponents[i], 0.0001) then - return false - end - end - return true - - -- For Vector3s, compare to components with epsilon of 0.0001 to avoid floating point inequality - elseif typeA == "Vector3" and typeB == "Vector3" then - local aComponents, bComponents = { a.X, a.Y, a.Z }, { b.X, b.Y, b.Z } - for i, aComponent in aComponents do - if not fuzzyEq(aComponent, bComponents[i], 0.0001) then - return false - end - end - return true - - -- For Vector2s, compare to components with epsilon of 0.0001 to avoid floating point inequality - elseif typeA == "Vector2" and typeB == "Vector2" then - local aComponents, bComponents = { a.X, a.Y }, { b.X, b.Y } - for i, aComponent in aComponents do - if not fuzzyEq(aComponent, bComponents[i], 0.0001) then - return false - end - end + if a ~= a and b ~= b then return true end diff --git a/plugin/src/ServeSession.lua b/plugin/src/ServeSession.lua index 8dd1098f..7f3f7fce 100644 --- a/plugin/src/ServeSession.lua +++ b/plugin/src/ServeSession.lua @@ -17,6 +17,7 @@ local encodePatchUpdate = require(script.Parent.ChangeBatcher.encodePatchUpdate) local encodeInstance = require(script.Parent.ChangeBatcher.encodeInstance) local InstanceMap = require(script.Parent.InstanceMap) local PatchSet = require(script.Parent.PatchSet) +local Matching = require(script.Parent.Reconciler.matching) local Reconciler = require(script.Parent.Reconciler) local strict = require(script.Parent.strict) local Settings = require(script.Parent.Settings) @@ -625,7 +626,8 @@ function ServeSession:__computeInitialPatch(serverInfo) -- tracking them in the reconciler. Log.trace("Matching existing Roblox instances to Rojo IDs") self:setLoadingText("Hydrating instance map...") - self.__reconciler:hydrate(readResponseBody.instances, serverInfo.rootInstanceId, game) + local matchingSession = Matching.newSession() + self.__reconciler:hydrate(readResponseBody.instances, serverInfo.rootInstanceId, game, matchingSession) -- Calculate the initial patch to apply to the DataModel to catch us -- up to what Rojo thinks the place should look like. diff --git a/src/cli/serve.rs b/src/cli/serve.rs index 831a4542..810a35a8 100644 --- a/src/cli/serve.rs +++ b/src/cli/serve.rs @@ -136,19 +136,39 @@ fn run_live_syncback(project_path: &Path, payload: SyncbackPayload) -> anyhow::R } fn build_dom_from_chunks(payload: SyncbackPayload) -> anyhow::Result { + use crate::syncback::VISIBLE_SERVICES; + let mut dom = WeakDom::new(InstanceBuilder::new("DataModel")); let root_ref = dom.root_ref(); let mut global_ref_map: HashMap = HashMap::new(); + let mut created_services: std::collections::HashSet = std::collections::HashSet::new(); for chunk in &payload.services { - let chunk_dom = rbx_binary::from_reader(Cursor::new(&chunk.data)) - .with_context(|| format!("Failed to parse rbxm for service {}", chunk.class_name))?; - let service_ref = dom.insert(root_ref, InstanceBuilder::new(&chunk.class_name)); + created_services.insert(chunk.class_name.clone()); - for &child_ref in chunk_dom.root().children() { - deep_clone_into(&chunk_dom, &mut dom, child_ref, service_ref, &mut global_ref_map); + if !chunk.data.is_empty() { + let chunk_dom = rbx_binary::from_reader(Cursor::new(&chunk.data)) + .with_context(|| { + format!("Failed to parse rbxm for service {}", chunk.class_name) + })?; + + for &child_ref in chunk_dom.root().children() { + deep_clone_into( + &chunk_dom, + &mut dom, + child_ref, + service_ref, + &mut global_ref_map, + ); + } + } + } + + for &service_name in VISIBLE_SERVICES { + if !created_services.contains(service_name) { + dom.insert(root_ref, InstanceBuilder::new(service_name)); } } diff --git a/src/snapshot/matching.rs b/src/snapshot/matching.rs index f98d002c..7b144431 100644 --- a/src/snapshot/matching.rs +++ b/src/snapshot/matching.rs @@ -10,6 +10,7 @@ //! The change count = how many things the reconciler would need to touch //! to turn instance A into instance B, including the entire subtree. +use std::cell::RefCell; use std::collections::HashMap; use rbx_dom_weak::types::Ref; @@ -25,6 +26,21 @@ const UNMATCHED_PENALTY: u32 = 10_000; /// O(n^k) explosion on deeply nested trees with many same-named instances. const MAX_SCORING_DEPTH: u32 = 3; +/// Session-scoped cache for the matching algorithm. Caches +/// `compute_change_count` results so that recursive scoring work is +/// reused across calls at different tree levels. +pub struct MatchingSession { + cost_cache: RefCell>, +} + +impl MatchingSession { + pub fn new() -> Self { + Self { + cost_cache: RefCell::new(HashMap::new()), + } + } +} + /// Result of the forward sync matching algorithm. pub struct ForwardMatchResult { /// Pairs of (snapshot_child, tree_ref) that were matched. @@ -33,6 +49,9 @@ pub struct ForwardMatchResult { pub unmatched_snapshot: Vec, /// Tree children with no match in the snapshot (to be removed). pub unmatched_tree: Vec, + /// Total cost of the matching (sum of all matched pair costs + + /// unmatched penalties). Zero means no changes across anything. + pub total_cost: u32, } /// Match snapshot children to tree children, minimizing total changes. @@ -40,12 +59,14 @@ pub fn match_forward( snapshot_children: Vec, tree_children: &[Ref], tree: &RojoTree, + session: &MatchingSession, ) -> ForwardMatchResult { if snapshot_children.is_empty() && tree_children.is_empty() { return ForwardMatchResult { matched: Vec::new(), unmatched_snapshot: Vec::new(), unmatched_tree: Vec::new(), + total_cost: 0, }; } @@ -95,7 +116,7 @@ pub fn match_forward( let snap_remaining = snap_matched.iter().filter(|&&m| !m).count(); let tree_remaining = tree_available.iter().filter(|&&a| a).count(); if snap_remaining == 0 || tree_remaining == 0 { - return build_result(snap_available, tree_children, &tree_available, matched); + return build_result(snap_available, tree_children, &tree_available, matched, tree, session); } // ================================================================ @@ -142,7 +163,8 @@ pub fn match_forward( None => continue, }; for &ti in &avail_tree { - let cost = compute_change_count(snap, tree_children[ti], tree, best_so_far, 0); + let cost = + compute_change_count(snap, tree_children[ti], tree, best_so_far, 0, session); pairs.push((cost, si, ti)); if cost < best_so_far { best_so_far = cost; @@ -164,19 +186,31 @@ pub fn match_forward( } } - build_result(snap_available, tree_children, &tree_available, matched) + build_result(snap_available, tree_children, &tree_available, matched, tree, session) } /// Build the final result, consuming the snapshot children. +/// Computes `total_cost` for all matched pairs (using session cache). fn build_result( mut snap_available: Vec>, tree_children: &[Ref], tree_available: &[bool], matched_indices: Vec<(usize, usize)>, + tree: &RojoTree, + session: &MatchingSession, ) -> ForwardMatchResult { let mut matched = Vec::with_capacity(matched_indices.len()); + let mut total_cost: u32 = 0; for (si, ti) in matched_indices { if let Some(snap) = snap_available[si].take() { + total_cost = total_cost.saturating_add(compute_change_count( + &snap, + tree_children[ti], + tree, + u32::MAX, + 0, + session, + )); matched.push((snap, tree_children[ti])); } } @@ -190,10 +224,15 @@ fn build_result( .map(|(_, r)| *r) .collect(); + total_cost = total_cost.saturating_add( + (unmatched_snapshot.len() + unmatched_tree.len()) as u32 * UNMATCHED_PENALTY, + ); + ForwardMatchResult { matched, unmatched_snapshot, unmatched_tree, + total_cost, } } @@ -214,6 +253,7 @@ fn match_children_for_scoring( tree_children: &[Ref], tree: &RojoTree, depth: u32, + session: &MatchingSession, ) -> ScoringMatchResult { let mut snap_matched = vec![false; snap_children.len()]; let mut tree_matched = vec![false; tree_children.len()]; @@ -298,6 +338,7 @@ fn match_children_for_scoring( tree, best_so_far, depth, + session, ); pairs.push((cost, si, ti)); if cost < best_so_far { @@ -341,7 +382,16 @@ fn compute_change_count( tree: &RojoTree, best_so_far: u32, depth: u32, + session: &MatchingSession, ) -> u32 { + let cacheable = snap.snapshot_id.is_some() && tree_ref.is_some(); + if cacheable { + let cache_key = (snap.snapshot_id, tree_ref); + if let Some(&cached) = session.cost_cache.borrow().get(&cache_key) { + return cached; + } + } + let inst = match tree.get_instance(tree_ref) { Some(i) => i, None => return UNMATCHED_PENALTY, @@ -356,10 +406,17 @@ fn compute_change_count( let tree_children = inst.children(); if snap_children.is_empty() && tree_children.is_empty() { + if cacheable && cost < best_so_far { + session + .cost_cache + .borrow_mut() + .insert((snap.snapshot_id, tree_ref), cost); + } return cost; } - let scoring = match_children_for_scoring(snap_children, tree_children, tree, depth + 1); + let scoring = + match_children_for_scoring(snap_children, tree_children, tree, depth + 1, session); for &(si, ti) in &scoring.matched { let remaining = best_so_far.saturating_sub(cost); @@ -369,6 +426,7 @@ fn compute_change_count( tree, remaining, depth + 1, + session, ); if cost >= best_so_far { return cost; @@ -379,6 +437,13 @@ fn compute_change_count( (scoring.unmatched_snap + scoring.unmatched_tree) as u32 * UNMATCHED_PENALTY, ); + if cacheable && cost < best_so_far { + session + .cost_cache + .borrow_mut() + .insert((snap.snapshot_id, tree_ref), cost); + } + cost } @@ -482,7 +547,7 @@ mod tests { ]; let (tree, children) = make_tree_with_children(&[("Alpha", "Folder"), ("Beta", "Script")]); - let result = match_forward(snaps, &children, &tree); + let result = match_forward(snaps, &children, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 2); assert!(result.unmatched_snapshot.is_empty()); assert!(result.unmatched_tree.is_empty()); @@ -497,7 +562,7 @@ mod tests { let (tree, children) = make_tree_with_children(&[("Handler", "ModuleScript"), ("Handler", "Script")]); - let result = match_forward(snaps, &children, &tree); + let result = match_forward(snaps, &children, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 2); for (snap, tree_ref) in &result.matched { @@ -514,7 +579,7 @@ mod tests { ]; let (tree, children) = make_tree_with_children(&[("A", "Folder"), ("OldOnly", "Folder")]); - let result = match_forward(snaps, &children, &tree); + let result = match_forward(snaps, &children, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 1); assert_eq!(result.unmatched_snapshot.len(), 1); assert_eq!(result.unmatched_tree.len(), 1); @@ -529,7 +594,7 @@ mod tests { ]; let (tree, children) = make_tree_with_children(&[("Data", "Folder"), ("Data", "Folder")]); - let result = match_forward(snaps, &children, &tree); + let result = match_forward(snaps, &children, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 2); assert!(result.unmatched_snapshot.is_empty()); assert!(result.unmatched_tree.is_empty()); @@ -540,7 +605,7 @@ mod tests { let snaps: Vec = vec![]; let (tree, children) = make_tree_with_children(&[]); - let result = match_forward(snaps, &children, &tree); + let result = match_forward(snaps, &children, &tree, &MatchingSession::new()); assert!(result.matched.is_empty()); } @@ -605,7 +670,7 @@ mod tests { let root_id = tree.get_root_id(); let tree_children: Vec = tree.get_instance(root_id).unwrap().children().to_vec(); - let result = match_forward(snaps, &tree_children, &tree); + let result = match_forward(snaps, &tree_children, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), count); assert!(result.unmatched_snapshot.is_empty()); @@ -635,7 +700,7 @@ mod tests { let (tree, children) = make_tree_with_children(&(0..count).map(|_| ("Line", "Part")).collect::>()); - let result = match_forward(snaps, &children, &tree); + let result = match_forward(snaps, &children, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), count); assert!(result.unmatched_snapshot.is_empty()); assert!(result.unmatched_tree.is_empty()); @@ -714,7 +779,7 @@ mod tests { let root_id = tree.get_root_id(); let tree_children: Vec = tree.get_instance(root_id).unwrap().children().to_vec(); - let result = match_forward(snaps, &tree_children, &tree); + let result = match_forward(snaps, &tree_children, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 2); assert!(result.unmatched_snapshot.is_empty()); @@ -852,7 +917,7 @@ mod tests { })); let (tree, tree_refs) = make_tree_from_snapshots(tree_children); - let result = match_forward(snaps, &tree_refs, &tree); + let result = match_forward(snaps, &tree_refs, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 6); assert!(result.unmatched_snapshot.is_empty()); @@ -945,7 +1010,7 @@ mod tests { tree_children.reverse(); let (tree, tree_refs) = make_tree_from_snapshots(tree_children); - let result = match_forward(snaps, &tree_refs, &tree); + let result = match_forward(snaps, &tree_refs, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 12); assert!(result.unmatched_snapshot.is_empty()); @@ -1017,7 +1082,7 @@ mod tests { tree_children.reverse(); let (tree, tree_refs) = make_tree_from_snapshots(tree_children); - let result = match_forward(snaps, &tree_refs, &tree); + let result = match_forward(snaps, &tree_refs, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 4); @@ -1074,7 +1139,7 @@ mod tests { make_snapshot_with_props("Wall", "Part", tree_props_b), make_snapshot_with_props("Wall", "Part", tree_props_a), ]); - let result = match_forward(vec![snap_a, snap_b], &tree_refs, &tree); + let result = match_forward(vec![snap_a, snap_b], &tree_refs, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 2, "Both should match"); @@ -1139,7 +1204,7 @@ mod tests { tree_children.reverse(); let (tree, tree_refs) = make_tree_from_snapshots(tree_children); - let result = match_forward(snaps, &tree_refs, &tree); + let result = match_forward(snaps, &tree_refs, &tree, &MatchingSession::new()); assert_eq!(result.matched.len(), 10); assert!(result.unmatched_snapshot.is_empty()); @@ -1208,7 +1273,7 @@ mod tests { let tree_children: Vec = tree.get_instance(root_id).unwrap().children().to_vec(); let start = std::time::Instant::now(); - let result = match_forward(snaps, &tree_children, &tree); + let result = match_forward(snaps, &tree_children, &tree, &MatchingSession::new()); let elapsed = start.elapsed(); assert_eq!(result.matched.len(), 2); diff --git a/src/snapshot/patch_compute.rs b/src/snapshot/patch_compute.rs index 3e9268ad..944574cf 100644 --- a/src/snapshot/patch_compute.rs +++ b/src/snapshot/patch_compute.rs @@ -19,12 +19,15 @@ use super::{ #[profiling::function] pub fn compute_patch_set(snapshot: Option, tree: &RojoTree, id: Ref) -> PatchSet { + use super::matching::MatchingSession; + let mut patch_set = PatchSet::new(); if let Some(snapshot) = snapshot { let mut context = ComputePatchContext::default(); + let session = MatchingSession::new(); - compute_patch_set_internal(&mut context, snapshot, tree, id, &mut patch_set); + compute_patch_set_internal(&mut context, snapshot, tree, id, &mut patch_set, &session); // Rewrite Ref properties to refer to instance IDs instead of snapshot IDs // for all of the IDs that we know about so far. @@ -80,6 +83,7 @@ fn compute_patch_set_internal( tree: &RojoTree, id: Ref, patch_set: &mut PatchSet, + session: &super::matching::MatchingSession, ) { if snapshot.snapshot_id.is_some() { context @@ -92,7 +96,7 @@ fn compute_patch_set_internal( .expect("Instance did not exist in tree"); compute_property_patches(&mut snapshot, &instance, patch_set, tree); - compute_children_patches(context, &mut snapshot, tree, id, patch_set); + compute_children_patches(context, &mut snapshot, tree, id, patch_set, session); } fn compute_property_patches( @@ -216,6 +220,7 @@ fn compute_children_patches( tree: &RojoTree, id: Ref, patch_set: &mut PatchSet, + session: &super::matching::MatchingSession, ) { use super::matching::match_forward; @@ -226,12 +231,11 @@ fn compute_children_patches( let instance_children = instance.children().to_vec(); let snapshot_children = take(&mut snapshot.children); - // Use the recursive change-count scoring + greedy assignment matching algorithm. - let match_result = match_forward(snapshot_children, &instance_children, tree); + let match_result = match_forward(snapshot_children, &instance_children, tree, session); // Matched pairs: recursively compute patches. for (snapshot_child, tree_child_id) in match_result.matched { - compute_patch_set_internal(context, snapshot_child, tree, tree_child_id, patch_set); + compute_patch_set_internal(context, snapshot_child, tree, tree_child_id, patch_set, session); } // Unmatched snapshots: new instances to be added. diff --git a/src/snapshot_middleware/dir.rs b/src/snapshot_middleware/dir.rs index 01e722af..ab2a41b6 100644 --- a/src/snapshot_middleware/dir.rs +++ b/src/snapshot_middleware/dir.rs @@ -208,6 +208,7 @@ pub fn syncback_dir_no_meta<'sync>( .collect(); // Run the 3-pass matching algorithm to pair new ↔ old children. + let matching_session = crate::syncback::matching::MatchingSession::new(); let match_result = match_children( &eligible_new, &eligible_old, @@ -215,6 +216,7 @@ pub fn syncback_dir_no_meta<'sync>( snapshot.old_tree(), None, // TODO: pass precomputed hashes for better similarity None, + &matching_session, ); // Pre-seed taken_names from ALL matched old children's filesystem diff --git a/src/syncback/matching.rs b/src/syncback/matching.rs index 01644eeb..6429cd50 100644 --- a/src/syncback/matching.rs +++ b/src/syncback/matching.rs @@ -11,6 +11,7 @@ //! The change count = how many things the reconciler would need to touch //! to turn instance A into instance B. +use std::cell::RefCell; use std::collections::{HashMap, HashSet}; use blake3::Hash; @@ -24,12 +25,26 @@ const UNMATCHED_PENALTY: u32 = 10_000; /// only flat property comparison is used (no subtree recursion). const MAX_SCORING_DEPTH: u32 = 3; +/// Session-scoped cache for the syncback matching algorithm. +pub struct MatchingSession { + cost_cache: RefCell>, +} + +impl MatchingSession { + pub fn new() -> Self { + Self { + cost_cache: RefCell::new(HashMap::new()), + } + } +} + /// Result of the matching algorithm. #[derive(Debug)] pub struct MatchResult { pub matched: Vec<(Ref, Ref)>, pub unmatched_new: Vec, pub unmatched_old: Vec, + pub total_cost: u32, } /// Match new children to old children, minimizing total changes. @@ -40,12 +55,14 @@ pub fn match_children( old_dom: &WeakDom, new_hashes: Option<&HashMap>, old_hashes: Option<&HashMap>, + session: &MatchingSession, ) -> MatchResult { if new_children.is_empty() && old_children.is_empty() { return MatchResult { matched: Vec::new(), unmatched_new: Vec::new(), unmatched_old: Vec::new(), + total_cost: 0, }; } @@ -102,10 +119,20 @@ pub fn match_children( remaining_old.retain(|r| !matched_old.contains(r)); if remaining_new.is_empty() || remaining_old.is_empty() { + let mut total_cost: u32 = 0; + for &(new_ref, old_ref) in &matched { + total_cost = total_cost.saturating_add(compute_change_count( + new_ref, old_ref, new_dom, old_dom, new_hashes, old_hashes, u32::MAX, 0, session, + )); + } + total_cost = total_cost.saturating_add( + (remaining_new.len() + remaining_old.len()) as u32 * UNMATCHED_PENALTY, + ); return MatchResult { matched, unmatched_new: remaining_new, unmatched_old: remaining_old, + total_cost, }; } @@ -151,14 +178,8 @@ pub fn match_children( for &new_ref in &avail_new { for &old_ref in &avail_old { let cost = compute_change_count( - new_ref, - old_ref, - new_dom, - old_dom, - new_hashes, - old_hashes, - best_so_far, - 0, + new_ref, old_ref, new_dom, old_dom, new_hashes, old_hashes, best_so_far, 0, + session, ); pairs.push((cost, new_ref, old_ref)); if cost < best_so_far { @@ -184,10 +205,21 @@ pub fn match_children( remaining_new.retain(|r| !matched_new.contains(r)); remaining_old.retain(|r| !matched_old.contains(r)); + let mut total_cost: u32 = 0; + for &(new_ref, old_ref) in &matched { + total_cost = total_cost.saturating_add(compute_change_count( + new_ref, old_ref, new_dom, old_dom, new_hashes, old_hashes, u32::MAX, 0, session, + )); + } + total_cost = total_cost.saturating_add( + (remaining_new.len() + remaining_old.len()) as u32 * UNMATCHED_PENALTY, + ); + MatchResult { matched, unmatched_new: remaining_new, unmatched_old: remaining_old, + total_cost, } } @@ -209,6 +241,7 @@ fn match_children_for_scoring( new_hashes: Option<&HashMap>, old_hashes: Option<&HashMap>, depth: u32, + session: &MatchingSession, ) -> ScoringMatchResult { let mut new_matched = vec![false; new_children.len()]; let mut old_matched = vec![false; old_children.len()]; @@ -298,6 +331,7 @@ fn match_children_for_scoring( old_hashes, best_so_far, depth, + session, ); pairs.push((cost, ni, oi)); if cost < best_so_far { @@ -342,11 +376,25 @@ fn compute_change_count( old_hashes: Option<&HashMap>, best_so_far: u32, depth: u32, + session: &MatchingSession, ) -> u32 { + let cacheable = new_ref.is_some() && old_ref.is_some(); + if cacheable { + if let Some(&cached) = session.cost_cache.borrow().get(&(new_ref, old_ref)) { + return cached; + } + } + // Hash fast-path: identical subtree = 0 cost if let (Some(nh), Some(oh)) = (new_hashes, old_hashes) { if let (Some(new_hash), Some(old_hash)) = (nh.get(&new_ref), oh.get(&old_ref)) { if new_hash == old_hash { + if cacheable { + session + .cost_cache + .borrow_mut() + .insert((new_ref, old_ref), 0); + } return 0; } } @@ -370,6 +418,12 @@ fn compute_change_count( let old_children = old_inst.children(); if new_children.is_empty() && old_children.is_empty() { + if cacheable && cost < best_so_far { + session + .cost_cache + .borrow_mut() + .insert((new_ref, old_ref), cost); + } return cost; } @@ -381,6 +435,7 @@ fn compute_change_count( new_hashes, old_hashes, depth + 1, + session, ); for &(ni, oi) in &scoring.matched { @@ -394,6 +449,7 @@ fn compute_change_count( old_hashes, remaining, depth + 1, + session, ); if cost >= best_so_far { return cost; @@ -403,6 +459,13 @@ fn compute_change_count( cost = cost .saturating_add((scoring.unmatched_new + scoring.unmatched_old) as u32 * UNMATCHED_PENALTY); + if cacheable && cost < best_so_far { + session + .cost_cache + .borrow_mut() + .insert((new_ref, old_ref), cost); + } + cost } @@ -482,7 +545,7 @@ mod tests { let new_children: Vec = new_dom.get_by_ref(new_root).unwrap().children().to_vec(); let old_children: Vec = old_dom.get_by_ref(old_root).unwrap().children().to_vec(); - let result = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None); + let result = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None, &MatchingSession::new()); assert_eq!(result.matched.len(), 2); assert!(result.unmatched_new.is_empty()); @@ -522,7 +585,7 @@ mod tests { let new_children: Vec = new_dom.get_by_ref(new_root).unwrap().children().to_vec(); let old_children: Vec = old_dom.get_by_ref(old_root).unwrap().children().to_vec(); - let result = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None); + let result = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None, &MatchingSession::new()); assert_eq!(result.matched.len(), 2); for (new_ref, old_ref) in &result.matched { @@ -553,7 +616,7 @@ mod tests { let new_children: Vec = new_dom.get_by_ref(new_root).unwrap().children().to_vec(); let old_children: Vec = old_dom.get_by_ref(old_root).unwrap().children().to_vec(); - let result = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None); + let result = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None, &MatchingSession::new()); assert_eq!(result.matched.len(), 1); assert_eq!(result.unmatched_new.len(), 1); @@ -575,7 +638,7 @@ mod tests { let new_children: Vec = new_dom.get_by_ref(new_root).unwrap().children().to_vec(); let old_children: Vec = old_dom.get_by_ref(old_root).unwrap().children().to_vec(); - let result = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None); + let result = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None, &MatchingSession::new()); assert_eq!(result.matched.len(), 2); assert!(result.unmatched_new.is_empty()); assert!(result.unmatched_old.is_empty()); @@ -585,7 +648,7 @@ mod tests { fn empty_children() { let new_dom = WeakDom::new(InstanceBuilder::new("DataModel")); let old_dom = WeakDom::new(InstanceBuilder::new("DataModel")); - let result = match_children(&[], &[], &new_dom, &old_dom, None, None); + let result = match_children(&[], &[], &new_dom, &old_dom, None, None, &MatchingSession::new()); assert!(result.matched.is_empty()); } @@ -604,8 +667,8 @@ mod tests { let new_children: Vec = new_dom.get_by_ref(new_root).unwrap().children().to_vec(); let old_children: Vec = old_dom.get_by_ref(old_root).unwrap().children().to_vec(); - let r1 = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None); - let r2 = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None); + let r1 = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None, &MatchingSession::new()); + let r2 = match_children(&new_children, &old_children, &new_dom, &old_dom, None, None, &MatchingSession::new()); assert_eq!(r1.matched.len(), r2.matched.len()); for (a, b) in r1.matched.iter().zip(r2.matched.iter()) { diff --git a/tests/rojo_test/serve_util.rs b/tests/rojo_test/serve_util.rs index a29d33c0..e7bbcb2b 100644 --- a/tests/rojo_test/serve_util.rs +++ b/tests/rojo_test/serve_util.rs @@ -538,6 +538,32 @@ pub fn make_rbxl_from_chunks(chunks: &[librojo::web_api::ServiceChunk]) -> Vec = chunks + .iter() + .map(|c| c.class_name.clone()) + .collect(); + for &svc in &visible_services { + if !existing.contains(svc) { + dom.insert(root_ref, rbx_dom_weak::InstanceBuilder::new(svc)); + } + } + let service_refs: Vec = dom.root().children().to_vec(); let mut buf = Vec::new(); rbx_binary::to_writer(&mut buf, &dom, &service_refs).unwrap(); diff --git a/tests/tests/matching_fixtures.rs b/tests/tests/matching_fixtures.rs index 817d6fcd..02614e6b 100644 --- a/tests/tests/matching_fixtures.rs +++ b/tests/tests/matching_fixtures.rs @@ -200,6 +200,7 @@ fn syncback_matching_ufowave_textures() { eprintln!("Expected face distribution: {:?}", expected_faces); // Run syncback matching + let session = librojo::syncback::matching::MatchingSession::new(); let result = librojo::syncback::matching::match_children( &input_textures, &expected_textures, @@ -207,6 +208,7 @@ fn syncback_matching_ufowave_textures() { &expected_dom, None, None, + &session, ); assert_eq!( @@ -306,7 +308,8 @@ fn forward_matching_ufowave_textures() { assert!(!snap_textures.is_empty(), "No Texture children found"); // Run forward sync matching - let result = librojo::snapshot::matching::match_forward(snap_textures, &tree_textures, &tree); + let session = librojo::snapshot::matching::MatchingSession::new(); + let result = librojo::snapshot::matching::match_forward(snap_textures, &tree_textures, &tree, &session); assert_eq!( result.matched.len(),