hakowork/rojo
3
0
Fork 0
mirror of https://github.com/rojo-rbx/rojo.git synced 2026-10-24 15:01:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #4 from UserGeneratedLLC/ambiguous-paths

Browse source 
Ambiguous paths
This commit is contained in:
CoderJoey 2026-02-19 06:29:33 -06:00 • committed by GitHub
commit 5a6a8d0b3d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
552 changed files with 45038 additions and 4607 deletions
Download patch file Download diff file Expand all files Collapse all files

325
.cursor/commands/audit.md
View file

@ -1,8 +1,10 @@
# /audit - Production-Grade Sync Feature Audit
**IMMEDIATE ACTION:** Before reading any files, running any commands, or performing any analysis, you MUST call `SwitchMode` with `target_mode_id: "plan"`. Do not proceed with any other step until you are in Plan mode. This is non-negotiable.
**Mode:** This command runs in **Plan mode** (read-only). It does NOT directly apply code changes. The deliverable is a `.cursor/plans/*.plan.md` file containing every approved fix, ready to be executed in a subsequent Agent-mode session.
**Workflow:** Analyze -> Report -> Quiz user on each fix -> Write plan file.
**Workflow:** Switch to Plan mode -> Analyze -> Report -> Quiz user on each fix -> Write plan file.
## Prerequisites
@ -31,7 +33,7 @@ Any divergence between the two-way sync path and the CLI syncback path is a bug
## Sync System Layout
The auditor MUST read the relevant files from each pipeline when tracing round-trip identity. This section maps every file involved in both sync systems. Read these files -- do not skip any that are in the scope of the feature being audited.
The auditor MUST read the relevant files from each pipeline when tracing round-trip identity. This section maps every file involved in all three sync systems. Read these files -- do not skip any that are in the scope of the feature being audited.
### CLI Syncback Pipeline (`atlas syncback`)
@ -52,17 +54,18 @@ Syncback Core
│ middleware context through the pipeline. Methods:
│ with_joined_path(), get_path_filtered_properties().
├── src/syncback/file_names.rs Filename generation. slugify_name() replaces forbidden
│ chars with '_'. deduplicate_name() appends ~1, ~2 on
│ collision. name_for_inst() is the main entry point.
│ adjacent_meta_path() computes .meta.json5 paths for
│ scripts (strips .server/.client suffixes from stem).
│ chars (incl. ~) with '_'. deduplicate_name_with_ext()
│ appends ~2, ~3 on collision. name_for_inst() is the
│ main entry point. adjacent_meta_path() computes
│ .meta.json5 paths for scripts (strips .server/.client
│ suffixes from stem).
├── src/syncback/property_filter.rs Property filtering. filter_properties() removes Name,
│ Parent, Source (scripts), Value (StringValue), Contents
│ (LocalizationTable), Ref/UniqueId, unscriptable props.
├── src/syncback/ref_properties.rs Ref property resolution. collect_referents() finds all
│ Ref properties. link_referents() writes Rojo_Ref_* or
│ Rojo_Target_* attributes. Path-based preferred, falls
│ back to ID-based for ambiguous paths.
│ Ref properties. link_referents() writes Rojo_Ref_*
│ (path-based) and Rojo_Id attributes to instances.
│ Rojo_Target_* is legacy (read-only during forward sync).
├── src/syncback/hash/mod.rs Content hashing. hash_tree() hashes entire tree bottom-
│ └── src/syncback/hash/variant.rs up with blake3. hash_instance() hashes class + filtered
│ properties + children hashes. Ref properties hashed as
@ -80,9 +83,9 @@ Syncback Core
Snapshot Middleware (syncback direction: instance → filesystem)
├── src/snapshot_middleware/mod.rs Dispatcher. Middleware enum routes to handler.
│ get_best_middleware() selects middleware for an instance.
│ Each middleware has syncback() producing SyncbackReturn
│ { fs_snapshot, children }.
│ { fs_snapshot, children }. get_best_middleware() lives
│ in src/syncback/mod.rs (selects middleware for an instance).
├── src/snapshot_middleware/dir.rs Directory handler. syncback_dir() processes children,
│ creates init.meta.json5, handles .gitkeep for empty dirs.
│ syncback_dir_no_meta() handles child dedup and recursion.
@ -146,12 +149,28 @@ Change Batching (Plugin)
│ Encodes a single property value using RbxDom.EncodedValue.
│ encode(). Maps Roblox values to wire format.
├── plugin/src/ChangeBatcher/encodeInstance.lua
│ Full instance encoding for additions. Duplicate detection
│ (skips instances with duplicate-named siblings).
│ Recursive children encoding. Attributes/Tags support.
│ Full instance encoding for additions. Uses encodeHelpers
│ for property iteration. All children encoded including
│ duplicates -- server handles dedup. Duplicate detection
│ helpers: findDuplicateNames(), hasDuplicateSiblings(),
│ isPathUnique(). Skips Ref properties during addition.
├── plugin/src/ChangeBatcher/encodeHelpers.lua
│ Shared encoding helpers used by encodeInstance and
│ encodeService. Exports: encodeAttributes(),
│ encodeTags(), forEachEncodableProperty(). Defines
│ SKIP_PROPERTIES (Parent, Name, Archivable). Filters
│ by scriptability, serialization, and data type.
├── plugin/src/ChangeBatcher/encodeService.lua
│ Service-level property encoding for live syncback.
│ Encodes properties via RbxDom.findClassDescriptor().
│ Skips: Parent, Name, Archivable, Attributes, Tags.
│ Ref properties: creates temporary ObjectValue carriers
│ (Name=propertyName, Value=target). Returns (chunk,
│ refTargets). Class-specific excludes (Lighting.ClockTime).
└── plugin/src/ChangeBatcher/propertyFilter.lua
Filters which properties are synced. Controls which data
types and property names are included/excluded.
Filters which properties are synced. Only UniqueId is
blocked. Ref properties are explicitly allowed (handled
by encodePatchUpdate via InstanceMap resolution).
Session & Transport (Plugin)
├── plugin/src/ServeSession.lua Orchestrates the session. onInstanceChanged → guards
@ -174,12 +193,29 @@ Server Write Processing (receives plugin writes)
│ model files. Sends PatchSet to tree_mutation_sender.
│ CRITICAL: file writes and PatchSet must stay consistent.
├── src/change_processor.rs handle_tree_event() receives PatchSet from api.rs.
│ Processes removals (tree update), updates (Source writes,
│ renames with slugify/dedup/meta lifecycle, ClassName
│ transitions), additions (tree insert). Applies patch via
│ apply_patch_set(). Broadcasts via message_queue.
│ Processes removals (tree update), dedup cleanup (with
│ removed_set to exclude co-removed siblings), updates
│ (Source writes, renames with slugify/dedup/meta lifecycle,
│ ClassName transitions, ref path updates via RefPathIndex),
│ additions (tree insert). Applies patch via apply_patch_set().
│ Broadcasts via message_queue.
├── src/git.rs Git integration. compute_git_metadata() identifies changed
│ files relative to HEAD. compute_blob_sha1() hashes content
│ using git blob format. git_add() stages files after writes.
│ Used by api.rs for stageIds and by serve_session for
│ GitMetadata in server info response.
├── src/rojo_ref.rs Ref path system. ref_target_path_from_tree() builds
│ filesystem-name paths for Rojo_Ref_* attributes. RefPathIndex
│ tracks which files contain which ref paths for efficient
│ updates on rename. Constants: REF_PATH_ATTRIBUTE_PREFIX,
│ REF_POINTER_ATTRIBUTE_PREFIX, REF_ID_ATTRIBUTE_NAME.
├── src/variant_eq.rs Property value comparison. variant_eq() compares Variant
│ values with fuzzy float matching (custom fuzzy_eq_f32/
│ fuzzy_eq_f64 with EPSILON 0.0001, absolute+relative).
│ Used by matching algorithms for property diff scoring.
└── src/serve_session.rs Server-side session. Owns RojoTree, ChangeProcessor,
MessageQueue. Coordinates serve lifecycle.
MessageQueue, RefPathIndex, git_repo_root. Coordinates
serve lifecycle.
SERVER → PLUGIN (Filesystem changes pushed to Studio)
=====================================================
@ -209,7 +245,12 @@ Snapshot Generation (Server, forward-sync direction: filesystem → instance)
│ DirectoryMetadata types used as overlays.
├── src/snapshot_middleware/project.rs snapshot_project() reads .project.json5. Resolves nodes,
│ infers class names for services, merges $properties.
└── (other middleware: csv.rs, txt.rs, json.rs, toml.rs, yaml.rs, rbxm.rs, rbxmx.rs)
├── (other middleware: csv.rs, txt.rs, json.rs, toml.rs, yaml.rs, rbxm.rs, rbxmx.rs)
│
├── src/snapshot/matching.rs Forward sync matching algorithm. match_forward() pairs
│ InstanceSnapshots to existing RojoTree children using
│ recursive change-count scoring + greedy assignment.
│ Constants: UNMATCHED_PENALTY=10000, MAX_SCORING_DEPTH=3.
Patch & Delivery (Server)
├── src/snapshot/patch_compute.rs compute_patch_set() diffs old snapshot vs tree →
@ -246,6 +287,16 @@ Patch Application (Plugin)
│ via instanceMap).
├── plugin/src/Reconciler/diff.lua diff() compares virtual instance data against Studio.
│ Used for confirmation dialog to show what will change.
├── plugin/src/Reconciler/matching.lua Recursive change-count scoring algorithm. matchChildren()
│ pairs virtual instances with Studio instances during
│ hydration. Signature: (virtualChildren, studioChildren,
│ virtualInstances). One of 3 parallel matching impls that
│ must produce identical pairings (see also src/snapshot/
│ matching.rs and src/syncback/matching.rs).
├── plugin/src/Reconciler/trueEquals.lua Shared value equality. Fuzzy floats (epsilon 0.0001),
│ Color3 via RGB ints, CFrame/Vector3 component-wise, NaN
│ handling, nil/null-ref equivalence. Used by matching.lua
│ and diff.lua.
├── plugin/src/Reconciler/setProperty.lua setProperty() applies a single property to an instance.
│ Handles special cases and error recovery.
├── plugin/src/Reconciler/getProperty.lua getProperty() reads a property from an instance.
@ -253,7 +304,93 @@ Patch Application (Plugin)
└── plugin/src/Reconciler/Error.lua Error types for reconciliation failures.
```
### Shared Infrastructure (used by both pipelines)
### Live Syncback Pipeline (`POST /api/syncback`)
One-click full sync from Studio to filesystem. Does NOT require an active WebSocket connection -- the plugin POSTs directly from the NotConnected page. The server restarts after completion.
```
PLUGIN SIDE (encoding and transport)
====================================
UI & Orchestration
├── plugin/src/App/init.lua SyncbackConfirm floating dialog (StudioPluginGui with
│ isEphemeral=true). Rendered when showingSyncbackConfirm
│ is true. SYNCBACK_SERVICES constant (14 services):
│ Lighting, MaterialService, ReplicatedFirst,
│ ReplicatedStorage, ServerScriptService, ServerStorage,
│ SoundService, StarterGui, StarterPack, StarterPlayer,
│ Teams, TextChatService, VoiceChatService, Workspace.
│ performSyncback() iterates services, encodes each via
│ encodeService(), serializes all children via
│ SerializationService:SerializeInstancesAsync() into a
│ single buffer, destroys temporary carriers, POSTs to
│ /api/syncback. Triggered from NotConnected page.
└── plugin/src/App/StatusPages/NotConnected.lua
"Syncback" button (Danger style) opens SyncbackConfirm.
Service Encoding
├── plugin/src/ChangeBatcher/encodeService.lua
│ Encodes service-level properties for a single service.
│ Returns (chunk, refTargets). chunk = { className,
│ childCount, refTargetCount, properties?, refs? }.
│ Ref properties use temporary ObjectValue carriers
│ (carrier.Name = propertyName, carrier.Value = target).
│ refs map stores 1-based carrier indices.
│ Class-specific excludes: Lighting.ClockTime.
│ Skips: Parent, Name, Archivable, Attributes, Tags.
└── plugin/src/ChangeBatcher/encodeHelpers.lua
Shared helpers used by encodeService. encodeAttributes(),
encodeTags(), forEachEncodableProperty().
Transport: HTTP POST /api/syncback (MessagePack body)
Request: { protocolVersion, serverVersion, placeId, data, services }
SERVER SIDE (validation, DOM reconstruction, syncback)
======================================================
API Handler
├── src/web/api.rs handle_api_syncback() validates:
│ (1) protocol version == PROTOCOL_VERSION (6)
│ (2) server version MAJOR.MINOR matches
│ (3) place ID passes servePlaceIds/blockedPlaceIds
│ Fires SyncbackSignal with payload.
│ Returns 400 (protocol/version mismatch), 403 (place ID),
│ or 409 (syncback already in progress).
Signal & Server Lifecycle
├── src/web/mod.rs SyncbackSignal: Mutex<Option<SyncbackPayload>> + Notify.
│ fire() deposits payload and notifies.
│ take_payload() retrieves and clears.
│ LiveServer::start() uses tokio::select! to wait for
│ new connections OR syncback signal. Exits accept loop
│ when signal fires.
Wire Types
├── src/web/interface.rs SyncbackRequest { protocol_version: f64,
│ server_version: String, place_id: Option<f64>,
│ data: Vec<u8>, services: Vec<ServiceChunk> }
│ ServiceChunk { class_name, child_count, ref_target_count,
│ properties: HashMap<String, Variant>,
│ refs: HashMap<String, u32> }
│ SyncbackPayload { data: Vec<u8>,
│ services: Vec<ServiceChunk> }
│ ServerExitReason::SyncbackRequested(SyncbackPayload)
DOM Reconstruction & Syncback
└── src/cli/serve.rs run_live_syncback() is the main handler.
build_dom_from_chunks() reconstructs WeakDom:
(1) parses rbxm blob with rbx_binary::from_reader()
(2) creates services from ServiceChunk metadata
(3) distributes children using child_count +
ref_target_count ranges per service
(4) resolves Ref properties via ObjectValue carriers
(5) ensures all VISIBLE_SERVICES exist
Runs syncback_loop(incremental=false) -- always clean
mode. write_to_vfs_parallel() writes files.
refresh_git_index() updates git. Serve loop restarts.
```
### Shared Infrastructure (used by all three pipelines)
```
Snapshot System
@ -264,11 +401,20 @@ Snapshot System
│ InstigatingSource (Path vs ProjectNode), SyncRule.
├── src/snapshot/tree.rs RojoTree: enhanced WeakDom with metadata. path_to_ids
│ mapping, specified_id_to_refs tracking, get_instance_
│ by_path() for path-based lookups.
│ by_path() for path-based lookups, filesystem_name_for()
│ for ref path resolution.
├── src/snapshot/matching.rs Forward sync matching: match_forward() pairs snapshot
│ children with existing tree children. Uses recursive
│ change-count scoring (same algorithm as syncback and
│ plugin matching). UNMATCHED_PENALTY=10000.
├── src/snapshot/patch.rs PatchSet, AppliedPatchSet, PatchAdd, PatchUpdate,
│ AppliedPatchUpdate data structures.
├── src/snapshot/patch_compute.rs compute_patch_set(): diffs snapshot vs tree.
└── src/snapshot/patch_apply.rs apply_patch_set(): applies patch to tree.
├── src/snapshot/patch_compute.rs compute_patch_set(): diffs snapshot vs tree. Uses
│ match_forward() for child pairing.
└── src/snapshot/patch_apply.rs apply_patch_set(): applies patch to tree. Deferred ref
resolution (path-based Rojo_Ref_* and ID-based
Rojo_Target_*). Cleans internal attributes before
sending to plugin.
Project System
└── src/project.rs Project loading. Project, ProjectNode, PathNode types.
@ -291,11 +437,45 @@ rbx-dom (Roblox format libraries, git submodule)
Plugin Shared Modules
├── plugin/src/InstanceMap.lua Instance ID ↔ Studio instance mapping. Signal management.
│ Also has onInstanceInserted callback for deferred Ref
│ resolution in ChangeBatcher.
├── plugin/src/PatchSet.lua PatchSet data structure and utilities.
├── plugin/src/Config.lua Protocol version, server version compatibility.
├── plugin/src/PatchTree.lua Builds tree for patch visualization. Accepts optional
│ gitMetadata to compute smart default selections (push/
│ pull/nil) per instance based on git change status and
│ committed script hashes.
├── plugin/src/Config.lua Protocol version (6), server version compatibility,
│ defaultPort ("34873").
├── plugin/src/Settings.lua Persistent settings (prefixed Atlas_ in Studio store).
│ Key defaults: oneShotSync=true, twoWaySync=true.
├── plugin/src/Types.lua Shared type definitions.
├── plugin/src/SHA1.luau SHA1 hashing. Computes git blob format hash ("blob
│ <size>\0<content>") for comparing script Source against
│ committed versions. Used by PatchTree for defaults.
├── plugin/src/XXH32.luau XXH32 hash function for plugin use.
├── plugin/src/ChangeMetadata.lua Change metadata tracking for sync operations.
├── plugin/src/strict.lua Module export wrapper (strict mode enforcement).
└── plugin/src/DiffUtil.lua Diff utilities for comparing instance data.
Integration Tests (tests/tests/)
├── tests/tests/build.rs Build tests (gen_build_tests! macro).
├── tests/tests/serve.rs Forward-sync serve tests.
├── tests/tests/connected_mode.rs Connected mode tests.
├── tests/tests/two_way_sync.rs Two-way sync via /api/write tests.
├── tests/tests/syncback.rs CLI syncback tests.
├── tests/tests/syncback_roundtrip.rs Full round-trip (syncback → build → compare).
├── tests/tests/syncback_format_transitions.rs File-to-folder and folder-to-file.
├── tests/tests/clean_mode.rs Clean mode syncback tests.
├── tests/tests/clean_mode_stress.rs Clean mode stress tests.
├── tests/tests/live_syncback.rs Live syncback parity tests (live output matches CLI),
│ validation tests (protocol/version/placeId), lifecycle
│ tests (server restart, sequential syncbacks), edge cases
│ (attributes, tags, cross-service refs, rapid sequential).
├── tests/tests/git_sync_defaults.rs Git-based sync default selection tests.
│ Hash computation parity, default selection logic,
│ auto-staging flow, no-git-repo fallback.
└── tests/tests/matching_fixtures.rs Matching algorithm fixture tests across all three
implementations (Rust syncback, Rust forward, Lua plugin).
```
### Audit Reading Order
@ -307,8 +487,11 @@ When auditing a feature, read files in this order to build understanding before
3. **Syncback path:** `src/syncback/mod.rs`, then the relevant middleware's `syncback()` function -- understand how instances become filesystem
4. **Two-way sync server:** `src/web/api.rs` (`handle_api_write`, `syncback_*` functions), then `src/change_processor.rs` (`handle_tree_event`) -- understand how plugin writes are processed
5. **Two-way sync plugin:** `plugin/src/ChangeBatcher/` (encoding path), then `plugin/src/Reconciler/` (decoding path) -- understand both directions in the plugin
6. **File naming:** `src/syncback/file_names.rs` -- understand slugification, dedup, meta path computation
7. **Ref properties:** `src/syncback/ref_properties.rs`, `src/snapshot/patch_apply.rs` (deferred ref section) -- understand cross-instance references
6. **File naming:** `src/syncback/file_names.rs` -- understand slugification, dedup, meta path computation (`adjacent_meta_path` strips script suffixes: `Foo.server.luau` pairs with `Foo.meta.json5`)
7. **Ref properties:** `src/rojo_ref.rs` (constants, `RefPathIndex`, `ref_target_path_from_tree`), `src/syncback/ref_properties.rs` (syncback ref linking), `src/snapshot/patch_apply.rs` (deferred ref resolution) -- understand cross-instance references
8. **Matching algorithms:** `src/snapshot/matching.rs` (forward sync), `src/syncback/matching.rs` (CLI syncback), `plugin/src/Reconciler/matching.lua` (plugin) -- three parallel implementations that must produce identical pairings. Also `src/variant_eq.rs` (Rust) and `plugin/src/Reconciler/trueEquals.lua` (Lua) for property comparison
9. **Git integration:** `src/git.rs` (server-side git metadata, blob hashing, auto-staging), `plugin/src/PatchTree.lua` (default selection logic using gitMetadata), `plugin/src/SHA1.luau` (plugin-side hash computation)
10. **Live syncback:** `plugin/src/ChangeBatcher/encodeService.lua` (service encoding), `plugin/src/App/init.lua` (`performSyncback`, `SyncbackConfirm`, `SYNCBACK_SERVICES`), `src/web/api.rs` (`handle_api_syncback`), `src/web/mod.rs` (`SyncbackSignal`, `LiveServer`), `src/cli/serve.rs` (`build_dom_from_chunks`, `run_live_syncback`), `src/web/interface.rs` (`SyncbackRequest`, `ServiceChunk`)
---
@ -559,7 +742,20 @@ Given the same instance tree, does the feature ALWAYS produce the same filesyste
Running the operation twice should produce zero changes on the second run. If re-running writes the same files again (even with identical content), something is unstable.
#### 11j. Edge Cases
#### 11j. Review Dialogue Invariant
**No two-way sync change may bypass the review dialogue.** In Always and Initial sync modes, every incoming change must be cataloged in the confirmation UI. The user must explicitly choose a resolution (Atlas / Skip / Studio) for each entry. There is no "fast path" or silent application.
**Verify:**
- Are there any code paths in `ServeSession`, `ApiContext`, or the Reconciler that apply patches without routing through the confirmation UI?
- Does the ChangeBatcher or any batch-processing logic silently apply changes that should appear in the review dialogue?
- When multiple changes arrive in a single WebSocket message or batch, does every individual change appear as a separate reviewable entry in the UI?
- Are there error/fallback paths that skip the dialogue and apply changes directly (e.g., "if dialogue fails, apply anyway")?
- Does the plugin correctly block patch application until the user has resolved every entry in the dialogue?
Any code path that writes to Studio instances without the user having explicitly approved it through the review UI is a **critical** bug. This applies to property updates, additions, removals, renames, and any other mutation — nothing is exempt.
#### 11k. Edge Cases
- **ProjectNode instances**: Can the feature interact with instances defined in project files? Is there a guard?
- **Nested projects**: Does data span project boundaries correctly?
@ -569,6 +765,75 @@ Running the operation twice should produce zero changes on the second run. If re
- **Script type transitions**: ClassName changes (ModuleScript -> Script) cause file extension changes. Does the feature handle this?
- **One-shot mode**: Does `Settings:get("oneShotSync")` correctly block the feature's outgoing writes?
#### 11l. Matching Algorithm Parity
The matching algorithm exists in **3 parallel implementations** that must produce identical pairings for the same input:
1. **Rust syncback** -- `src/syncback/matching.rs` (`match_children`, operates on `WeakDom` instances)
2. **Rust forward sync** -- `src/snapshot/matching.rs` (`match_forward`, operates on `InstanceSnapshot` vs `RojoTree`)
3. **Lua plugin** -- `plugin/src/Reconciler/matching.lua` (`Matching.matchChildren`, operates on virtual vs Studio instances)
**Verify:**
- If the feature modifies matching logic in one implementation, was the same change applied to all three?
- Do all three use the same constants? (`UNMATCHED_PENALTY=10000`, `MAX_SCORING_DEPTH=3`)
- Do all three use the same grouping strategy? (fast-path by `(Name, ClassName)`, then recursive scoring for ambiguous groups)
- Is sort stability preserved? (Rust `sort_by` is stable; Lua `table.sort` is NOT -- ties must be broken by insertion index)
- Are the property comparison functions consistent? (`variant_eq` in Rust, `trueEquals.lua` in Lua -- both use fuzzy float equality, both treat nil and null-ref as equal)
#### 11m. Git-Based Sync Defaults and Auto-Staging
The confirmation dialog uses git metadata for smart default selections (`PatchTree.lua` + `src/git.rs`). Verify:
- **Server-side `compute_git_metadata()`**: Does it correctly identify changed files vs HEAD? Does the two-phase tree lock (snapshot paths briefly, release before git subprocesses) prevent deadlocks?
- **Hash computation parity**: Server uses `compute_blob_sha1()` in `git.rs` and plugin uses `SHA1.luau` -- both must produce identical hashes for the same content using `SHA1("blob <byte_len>\0<content>")` git blob format.
- **Default selection logic** (`PatchTree.lua`): File has no git changes → default "pull". File has git changes AND is script AND Studio Source matches committed hash → default "push". Otherwise → `nil` (user must decide). Verify these rules are applied correctly.
- **Auto-staging (`stageIds`)**: After confirmation, `api.rs` receives `stageIds` and calls `git_add`. Push-accepted items are always staged. Pull-accepted items are only staged if auto-selected (`defaultSelection ~= nil`). Manually-chosen pulls are left unstaged. Is this flow correct?
- **Staging split**: `api.rs` stages additions/removals/push files directly. `change_processor` stages Source writes after they complete (via `stage_ids` on PatchSet). Verify both paths handle the `stage_ids` list correctly.
- **No git repo**: When `git_repo_root` is `None`, all defaults should be `nil` and no staging should be attempted.
#### 11n. Live Syncback Parity
Live syncback (`POST /api/syncback`) is a third sync path that must produce filesystem output identical to CLI syncback. It runs in clean mode (`incremental=false`) and restarts the server after completion.
**Verify plugin-side encoding:**
- Does `encodeService.lua` encode all service-level properties that CLI syncback captures? Compare against `filter_properties()` in `property_filter.rs`.
- Are `Attributes` and `Tags` handled correctly? (`encodeService` skips them from the property loop but `encodeHelpers.encodeAttributes/encodeTags` handle them separately.)
- Are class-specific excludes (`Lighting.ClockTime`) consistent with CLI syncback behavior?
- Is the `SKIP_PROPERTIES` list in `encodeHelpers.lua` (Parent, Name, Archivable) consistent with server-side filtering?
**Verify ObjectValue carrier lifecycle:**
- Are carriers created correctly? (`carrier.Name = propertyName`, `carrier.Value = targetInstance`)
- Are carriers included in the serialization buffer? (appended to service's children range)
- Are carriers destroyed after serialization? (cleanup in `performSyncback()`)
- Does `build_dom_from_chunks()` correctly parse the `ref_target_count` range and resolve carriers?
- Does the server destroy carriers after resolving Ref properties? (they must not appear in the synced tree)
**Verify rbxm blob reconstruction:**
- Does `build_dom_from_chunks()` distribute children correctly using `child_count` + `ref_target_count` ranges?
- Are services with zero children handled? (empty services still get created)
- Are `VISIBLE_SERVICES` in Rust consistent with `SYNCBACK_SERVICES` in Lua? (must list the same 14 services)
- Does the server handle out-of-range or zero-index carrier references gracefully?
**Verify clean mode semantics:**
- Live syncback always uses `incremental=false`. Does this match CLI syncback behavior for the same input?
- Are old files properly removed before new files are written?
- Is `refresh_git_index()` called after writes?
**Verify server restart behavior:**
- Does the server exit the accept loop cleanly when `SyncbackSignal` fires?
- Is the serve loop restarted with a new session after syncback completes?
- Can a new plugin connection be established after restart?
- Does `SyncbackSignal.fire()` return false if syncback is already in progress? (409 Conflict)
**Verify validation:**
- Protocol version mismatch → 400
- Server version MAJOR.MINOR mismatch → 400
- Place ID not in `servePlaceIds` whitelist → 403
- Place ID in `blockedPlaceIds` → 403
- No restrictions configured → any place ID allowed
**Verify round-trip:** Build → live syncback → rebuild → compare. The rebuilt instance tree must be bit-identical to the original (same as CLI syncback round-trip test).
### 12. Run Static Analysis
Run `cargo clippy` on the modified code to catch Rust issues the audit might miss:

33
.cursor/commands/format.md Normal file
View file

@ -0,0 +1,33 @@
# /format - Auto-fix Formatting & Static Analysis
Run formatting and static analysis (linting) across the entire project.
## Instructions
### 1. Run the Format Script
**Windows (PowerShell):**
```powershell
.\scripts\format.ps1
```
**macOS/Linux (Bash):**
```bash
bash scripts/format.sh
```
**Monitor the script output as it runs.**
### 2. Fix ALL Failures
If any step fails, you MUST fix the issues and re-run until everything passes. This applies to **every** failure -- Clippy, Selene, Prettier, all of them.
**Clippy is not optional.** Fix every Clippy warning, even if the warning is in code you did not write or that existed before your changes. There is no "pre-existing" or "before my time" exception. If Clippy reports it, fix it. The CI gate runs `cargo clippy` and will reject the PR regardless of who introduced the warning.
The same rule applies to Selene warnings on Lua code. If the linter flags it, fix it.
Keep re-running the format script until the overall result is **PASS** with zero failures.
### 3. Report
Once formatting passes clean, provide a brief summary and confirm the final result.

578
.cursor/commands/luau-optimize.md Normal file
View file

@ -0,0 +1,578 @@
# /luau-optimize - Roblox Luau Performance Optimization
Optimize Roblox Luau files for maximum performance, type safety, and native codegen quality using `luau-compile`, `luau-analyze`, `luau-ast`, and the `luau` CLI.
All scripts are **Roblox Luau**. Roblox APIs are always available. The vector type is `Vector3` (Roblox userdata). We are NOT using the new type solver. Studio testing uses `--!optimize 1` by default; live experiences use `--!optimize 2`. All `luau-compile` invocations must include `--vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3`. Prefer `.luau` extension over `.lua` everywhere.
**Workspace hygiene:** CLI tools generate residual files (`stats.json`, `profile.out`, `coverage.out`, `trace.json`). At the start of execution, create a temp working directory and use it for all tool output:
```
# Windows
$LUAU_TMP = New-Item -ItemType Directory -Path "$env:TEMP\luau-optimize-$(Get-Random)"
# Linux/Mac
LUAU_TMP=$(mktemp -d)
```
Route all output there: `--stats-file=$LUAU_TMP\stats.json`, write benchmark files to `$LUAU_TMP\bench.luau`, run profiling/coverage tools with `$LUAU_TMP` as working directory. The OS handles cleanup -- no manual deletion needed.
---
## Phase 0 -- Scope Resolution and Configuration
### Resolve target
- If user specifies a file: optimize that file.
- If user specifies a directory: discover all `.luau`/`.lua` files, enter Phase 6 (multi-file mode).
- If no argument: optimize the currently open file.
### Infer script context
Check in this order:
1. **Filename suffix** (highest priority):
- `*.server.luau` / `*.server.lua` -> **server** (`--!native` is high-value)
- `*.client.luau` / `*.client.lua` -> **client** (`--!native` less beneficial due to device diversity)
- `*.legacy.luau` / `*.legacy.lua` -> **ambiguous** (could be cloned/moved at runtime). Ask the user.
- `*.luau` / `*.lua` (no suffix) -> ModuleScript, fall through to path check.
2. **Path-based inference** (for ModuleScripts and ambiguous cases):
- `ServerScriptService`/`ServerStorage` in path -> server-side module
- `StarterPlayer`/`StarterGui`/`StarterCharacterScripts`/`ReplicatedFirst` in path -> client-side module
- `ReplicatedStorage`/`shared` in path -> shared module (conservative `--!native`)
- Ambiguous or outside Roblox project -> ask the user
### Quiz the user
Use the AskQuestion tool with these three questions:
**Question 1: "Compilation optimization intensity"**
- `minimal` -- Headers (`--!strict`/`--!native`/`--!optimize 2`), type annotations on all function signatures, deprecated pattern replacements only
- `moderate` -- + function restructuring for inlining, import hoisting, compound operators, fastcall enablement, allocation reduction
- `insane` -- + every micro-optimization from the pattern catalog, full bytecode analysis, register pressure optimization, closure caching analysis
**Question 2: "Algorithm optimization"**
- `none` -- Don't touch logic or data flow
- `low hanging fruit` -- Obvious O(n^2)->dict, redundant iterations, missing caches
- `moderate` -- + data structure changes, caching strategies, event-driven refactors, loop fusion
- `insane` -- + full algorithmic redesign where beneficial, dynamic programming, architectural restructuring
**Question 3: "Benchmarking"**
- `yes` -- Benchmark all non-trivial changes with `luau` CLI (when function is standalone)
- `no` -- Trust bytecode/codegen metrics only
- `+EV only` -- Benchmark only when algorithmic changes or ambiguous bytecode results warrant it
### Derived behavior
| Setting | minimal | moderate | insane |
|---|---|---|---|
| Headers | Always | Always | Always |
| Type annotations | Function signatures | + hot path locals | + all locals |
| Pattern replacements | Deprecated only | Full Priority 2 | + Priority 4 |
| Function restructuring | No | Yes (Priority 3) | Aggressive splitting |
| Bytecode verification | After all changes | After each priority | After each change |
| Algorithmic changes | Per algo quiz | Per algo quiz | Per algo quiz |
| `@native` vs `--!native` | `--!native` unless big file | Selective `@native` on hot functions | Full per-function analysis |
**Restructuring scope** scales with compilation intensity:
- **minimal**: No restructuring. Only additive changes (headers, annotations, pattern swaps).
- **moderate**: Break up monoliths, reorder locals for inlining, hoist imports. Preserve exports/API.
- **insane**: Aggressive restructuring permitted. May change internal function boundaries, rewrite data flow.
---
## Phase 1 -- Deep Code Read and Algorithmic Analysis
This phase comes FIRST. Algorithmic improvements produce order-of-magnitude speedups that dwarf any bytecode optimization. This is also the largest code change, so it must happen before any bytecode-level work.
Read the entire source file thoroughly before touching any CLI tools.
### Step 1 -- Structural decomposition
Monolithic functions hide their algorithmic structure behind interleaved concerns. Before analyzing complexity, understand what the code is *actually doing*:
- **Identify responsibilities**: A 200-line function usually contains 3-5 logical steps that should be separate functions.
- **Trace data flow**: What goes in, what comes out, what is mutated along the way.
- **Separate concerns**: Validation, transformation, I/O, state management, error handling are often tangled. Separating them reveals the core algorithm.
- **Name the steps**: If you can't name what a code block does in 3-5 words, it's doing too much.
- **Look for hidden abstractions**: Repeated "iterate, filter, transform, collect" patterns that could be a single helper.
When the quiz allows restructuring (moderate/insane):
- Physically decompose monoliths into focused local functions. Each does one thing.
- This decomposition often *reveals* optimization opportunities invisible in the monolith: step 2 recomputes what step 1 already knew, or an inner loop could be a lookup table built in the outer loop.
- Smaller functions are easier for the compiler to inline at `-O2` -- clearer code AND better bytecode.
- Preserve the original function as a thin orchestrator calling the decomposed pieces.
Even at "minimal" intensity, still perform this mental decomposition to inform the analysis below.
### Step 2 -- Algorithmic analysis
With the structure understood (or cleaned up), analyze:
**Complexity analysis** -- Identify time complexity of every significant function:
- O(n^2) or worse: nested loops, repeated linear searches, quadratic string building
- O(n) where O(1) is possible: linear search replaceable with dictionary/set lookup
- Repeated work: same computation done multiple times, cacheable/memoizable
- Unnecessary copies: cloning data that could be referenced or mutated in place
- Hidden quadratics: O(n) function that calls another O(n) function inside a loop
**Data structure fitness** -- Is the right data structure being used?
- Array for membership testing -> set (dictionary with `true` values)
- Linear scan for lookup by key -> dictionary
- Repeated sort of mostly-sorted data -> maintain sorted order on insert
- Flat list with frequent removal from middle -> swap-and-pop
- Large string built incrementally -> `table.concat` pattern or `buffer`
- Multiple parallel arrays -> single array of structs (table of tables)
- Unbounded growth without cleanup -> size limits, LRU eviction, periodic pruning
**Caching and memoization:**
- Pure functions called repeatedly with same args -> memoize
- Expensive property reads in loops -> cache in local before loop
- Computed values unchanged within a scope -> hoist out
- Derived state recomputed from scratch on every access -> maintain incrementally
**Redundant work elimination:**
- Multiple passes over same data -> fuse into one
- Recomputing derived state -> maintain incrementally
- Sort/filter that could be done once and reused
- Deep clone where shallow clone or reference suffices
- Intermediate result immediately discarded after extracting one field
**Architectural patterns:**
- Per-frame polling where event-driven would work (RunService events are expensive)
- Synchronous work that could be chunked across frames with `task.wait()`
- Undisconnected event connections (memory leak + wasted computation)
- Rebuilding entire state on small change -> incremental update
- God-object owning too much state -> split responsibilities
**Hot path identification:**
- Event handlers on RunService (PreAnimation, PreRender, PreSimulation, PostSimulation, Heartbeat)
- Functions called inside loops
- Recursive functions
- Functions called from other hot functions
### Step 3 -- Present findings and quiz the user
Before applying any changes, present a summary of all findings organized by impact. For each finding:
- **What**: One-line description of the issue
- **Where**: Function name / line range
- **Why**: What makes it slow (e.g., "O(n^2) nested scan on every frame")
- **Fix**: Proposed change in plain language
- **Impact**: Estimated improvement (order-of-magnitude, constant-factor, or clarity-only)
Then quiz the user using the AskQuestion tool:
- For fewer than 5 findings: one question per finding with options `apply` / `skip` / `modify`
- For 5+ findings: group by category with options `apply all` / `cherry-pick` / `skip all`:
- Structural decomposition
- Data structure changes
- Algorithm changes
- Architectural changes
If the algorithm quiz from Phase 0 was "none", present findings as informational only (no apply options). The user still sees what was found.
If the user selects "modify", wait for their input before proceeding with that specific change.
### Step 4 -- Apply approved changes
Apply only user-approved changes, in this order (dependencies flow downward):
1. **Structural decomposition** -- break monoliths into focused functions (foundation for everything else)
2. **Data structure changes** -- swap arrays for dicts, add indexes, etc.
3. **Algorithm changes** -- replace O(n^2) with O(n), add caching/memoization
4. **Architectural changes** -- event-driven refactors, incremental updates
Benchmark algorithmic changes if the benchmarking quiz warrants it (see Phase 4.5).
---
## Phase 2 -- Baseline Capture
Run all diagnostic tools on the code (post-algorithmic changes if any were applied in Phase 1). These metrics are the baseline for bytecode optimization phases.
### Tools to run
Detect OS: use `--target=x64_ms` on Windows, `--target=x64` on Linux/Mac. Route stats output to temp directory.
```
luau-analyze --mode=strict <file>
luau-analyze --annotate <file>
luau-compile --remarks -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>
luau-compile --text -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>
luau-compile --codegen --target=x64_ms --record-stats=function --stats-file=$LUAU_TMP\stats.json -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>
```
### Metrics to capture
- Total bytecode instruction count (from `--text`)
- Allocation count: NEWTABLE, NEWCLOSURE occurrences (from `--remarks`)
- Inlining success/failure count (from `--remarks`: "inlining succeeded"/"inlining failed")
- Type coverage: count of `any` types in `--annotate` output
- Register spills and skipped functions (from `--record-stats` JSON `lowerStats`)
- Lint warning count (from `--mode=strict`)
---
## Phase 3 -- Compilation-Level Code Analysis
Using both the source (already read in Phase 1) and tool output from Phase 2, identify bytecode-level optimization opportunities:
- **Missing types**: `any` inferences hurting native codegen (especially `Vector3`, `CFrame`, `buffer` params). JIT uses annotations directly -- no runtime type analysis. Unannotated params assumed to be tables.
- **Import patterns**: `math.max` resolved at load time via GETIMPORT. Broken by `getfenv`/`setfenv`/`loadstring` (marks env "impure").
- **Allocation in loops**: NEWTABLE, NEWCLOSURE in hot loops. High allocation rate = more GC assist work.
- **Closure caching**: Repeated function expressions cached when: no upvalues, or all upvalues immutable and module-scope. Mutable captures prevent caching.
- **Upvalue mutability**: ~90% immutable in typical code. Immutable = no allocation, no closing, faster access. Mutable = extra object.
- **Method call patterns**: `obj:Method()` uses fast method call instruction. Avoid `obj.Method(obj)`. `__index` should point at a table directly (not function or deep chain).
- **Inline caching**: Best when field name known at compile time, no metatables, uniform shapes.
- **pcall in hot paths**: Prevents native codegen optimization.
- **String concatenation**: `..` in loops -> `table.concat` or string interpolation.
- **Deoptimizing APIs**: `getfenv`/`setfenv` (even read-only!), `loadstring` disable ALL import optimization and fastcalls.
- **Builtin global writes**: `math = ...` disables fastcall. Lint `BuiltinGlobalWrite` catches this.
- **Metamethod cost**: `__eq` always called on `==`/`~=` even for rawequal values.
- **GC pressure**: Incremental GC with "GC assists" -- allocating code pays proportional GC work.
---
## Phase 4 -- Compilation Optimization
Apply changes in priority order. Re-run `luau-compile --remarks -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>` after structural changes to verify the compiler benefits.
### Priority 1 -- Headers and type safety (always apply)
**Add headers:**
- `--!strict` / `--!native` / `--!optimize 2`. (`--!optimize 2` is default in live but not Studio testing.)
**Remove deoptimizers:**
- Replace `getfenv`/`setfenv` -- disables builtins, imports, and optimizations globally.
- Replace `table.getn` -> `#t`, `table.foreach`/`table.foreachi` -> `for..in`.
**Type accuracy -- the first big battle:**
Getting the file to pass `--!strict` cleanly is the single most impactful compilation change. Every `any` type is a function the native codegen cannot specialize. But the goal is *accurate* types, not silencing errors with casts:
- Run `luau-analyze --annotate` and identify every `any` inference. These are the targets.
- Add real type annotations to function signatures first (parameters and return types). This has the highest leverage -- it propagates type info to every caller and every local inside the function.
- Then annotate locals in hot paths where inference still falls to `any`.
- Annotate `Vector3`, `CFrame`, `buffer` params explicitly -- native codegen generates specialized vector code. Unannotated params are assumed to be generic tables with extra type checks.
- **Do NOT paper over type errors with `:: any` casts.** A cast to `any` is worse than no annotation -- it actively tells the compiler to give up. If a type error is hard to fix, use the narrowest cast possible (`:: SpecificType`), or restructure the code so the type flows naturally.
- **Narrow `any` in dictionaries.** `{[string]: any}` is common but hurts codegen. Ask: what values actually go in there? If it's a known set of types, use a union: `{[string]: string | number | boolean}`. If the dictionary has known keys, use a typed table: `{name: string, score: number}`. Only fall back to `any` when the value type is genuinely unbounded (e.g., serialized data from an external source).
- Use type refinements (`type()`, `typeof()`, `:IsA()`, `assert()`) to provide type info from runtime checks instead of casts.
- For OOP patterns: use `typeof(setmetatable(...))` with explicit `self: ClassName` on methods (old typechecker compatible).
- For generic code: use proper generics (`<T>`) instead of `any`. If the function truly accepts anything, use `unknown` and narrow explicitly.
- Track type coverage: count `any` in `--annotate` output. The goal is zero (or as close as practical).
### Priority 2 -- Low-hanging structural wins
- Hoist library functions: `local floor = math.floor` enables GETIMPORT fastcall. Fastcall builtins: `assert`, `type`, `typeof`, `rawget`/`rawset`/`rawequal`, `getmetatable`/`setmetatable`, `tonumber`/`tostring`, most `math.*` (not `noise`, `random`/`randomseed`), `bit32.*`, some `string.*`/`table.*`. Partial specializations: `assert` (unused return + truthy), `bit32.extract` (constant field/width), `select(n, ...)` O(1).
- Replace `pairs(t)`/`ipairs(t)` -> `for k, v in t do`. Generalized iteration skips the `pairs()` call. `for i=1,#t` is slightly slower.
- `math.floor(a / b)` -> `a // b` (dedicated VM opcode, `//=` compound form).
- Use compound assignment (`+=`, `-=`, `*=`, `//=`, `..=`) -- LHS evaluated once.
- String concat in loops -> `table.concat` or backtick interpolation (lowers to optimized `string.format`).
- `table.create(n)` for known-size arrays. Sequential fill: `local t = table.create(N); for i=1,N do t[i] = ... end`.
- `table.insert(t, v)` for unknown-size append -- `#t` is O(1) cached, worst case O(log N).
- `a + (b - a) * t` -> `math.lerp(a, b, t)` (exact at endpoints, monotonic).
- Manual byte swap -> `bit32.byteswap(n)` (CPU bswap).
- Manual log2 -> `bit32.countlz(n)` (CPU instruction, ~8x faster).
- Manual linear search -> `table.find(t, v)`.
- Manual `pairs` clone -> `table.clone(t)`.
- Manual `string.byte`/`string.char` -> `string.pack`/`string.unpack`.
- `rawlen(t)` when metamethods not needed.
- Explicit `./`/`../`/`@` prefixes in `require()`.
### Priority 3 -- Function structure for inlining
- Break monoliths into small local functions (compiler inlines at `-O2`).
- Inlining requirements: local, non-mutated, non-recursive, not OOP (`:` syntax), not metamethod. Disabled by `getfenv`/`setfenv`.
- Inlining + constant folding cascade: `local function double(x) return x*2 end; local y = double(5)` folds to `y = 10`.
- Create local function wrappers for frequently-called imported module methods.
- Move `pcall` out of hot loops.
- Use `obj:Method()` not `obj.Method(obj)` -- fast method call instruction.
- `__index` should point at a table directly for inline caching.
- Reduce mutable upvalue captures in loop closures. Immutable upvalues = no allocation, no closing.
- Set all table fields at once (compiler infers hash capacity from subsequent assignments).
- Loop unrolling: constant bounds only, simple body.
- Native codegen size limits: 64K instructions/block, 32K blocks/function, 1M instructions/module. Split if exceeded.
### Priority 4 -- Micro-optimizations (hot paths only)
- Table shape consistency (same fields, same order) -- inline caching predicts hash slots.
- `buffer` for binary data (fixed-size, offset-based, efficient native lowering).
- `buffer.readu32` + `bit32` over `buffer.readbits` when schema known.
- Strength reduction: `* 2^n` -> `bit32.lshift`, `/ 2^n` -> `bit32.rshift`.
- Minimize `tostring`/`tonumber` in tight loops.
- `table.freeze` for readonly config (avoids proxy `__index` overhead).
- Keep `__eq` cheap -- fires on every `==`/`~=` and `table.find`.
- `math.isfinite`/`math.isnan`/`math.isinf` over `x ~= x`.
- Annotate `Vector3` params for native specialization.
- `if expr then A else B` over `cond and A or B` (one branch, falsy-safe).
---
## Phase 4.5 -- Benchmark with `luau` CLI
When a non-trivial optimization is applied to an isolated, pure-logic function (no Roblox API dependencies), write a small benchmark harness:
```luau
local function original(...)
-- paste original implementation
end
local function optimized(...)
-- paste optimized implementation
end
local ITERATIONS = 1_000_000
local clock = os.clock
for _ = 1, 1000 do original(...) end
for _ = 1, 1000 do optimized(...) end
local t0 = clock()
for _ = 1, ITERATIONS do original(...) end
local t1 = clock()
for _ = 1, ITERATIONS do optimized(...) end
local t2 = clock()
print(`Original: {t1 - t0:.4f}s`)
print(`Optimized: {t2 - t1:.4f}s`)
print(`Speedup: {(t1 - t0) / (t2 - t1):.2f}x`)
```
Run with both paths:
```
luau -O2 bench.luau
luau -O2 --codegen bench.luau
```
**When to benchmark:** Self-contained function, algorithmic/data structure change, ambiguous bytecode diff, tiebreak between approaches.
**When NOT to benchmark:** Roblox API dependencies (use Studio Script Profiler instead), purely additive changes, cold code.
Write the benchmark file to `$LUAU_TMP\bench.luau`, not the project directory. Run `luau` from `$LUAU_TMP` as working directory so `profile.out` and other residuals land there too.
---
## Phase 5 -- Verification
Re-run all Phase 2 tools and present a before/after comparison:
- Bytecode instruction count delta
- Allocation count delta
- Inlining success rate delta
- Type coverage improvement (fewer `any` types)
- Register spill delta
- Lint warning delta
- Benchmark results (if Phase 4.5 was used)
If any metric regressed, investigate and explain why (or revert).
---
## Phase 6 -- Multi-file Mode
When optimizing a directory:
1. Run Phase 1 (code read) on all files to identify algorithmic improvement potential.
2. Run Phase 2 (baseline) on all files to identify bytecode optimization potential.
3. Prioritize by: worst algorithmic complexity, most lint warnings, most `any` types, most allocations, most failed inlines.
4. Process files one at a time through Phases 1-5.
5. Present aggregate metrics at the end.
---
## Key Principles
These principles govern every optimization decision:
- **Same implementation, better performance** -- behavioral equivalence is non-negotiable. Never change what the code does, only how fast it does it.
- **Algorithms first, bytecode second** -- an O(n) algorithm with unoptimized bytecode beats an O(n^2) algorithm with perfect bytecode every time.
- **Low-hanging fruit first** -- headers and types before restructuring, restructuring before micro-optimizations.
- **Compiler feedback loop** -- verify each structural change with `luau-compile --remarks`. If the compiler didn't benefit, the change wasn't worth it.
- **Small functions over monoliths** -- easier for the compiler to inline, easier for humans to maintain, easier to reason about algorithmically.
- **Local functions for inlining** -- even at the cost of duplicating imported behavior in high-traffic code.
- **Quantify everything** -- before/after metrics for every optimization pass. No "I think this is faster" -- prove it.
- **Type annotations drive native codegen quality** -- every `any` type is a missed specialization opportunity. The JIT uses annotations directly with no runtime analysis.
- **Event-driven over polling** -- avoid per-frame calculations when events suffice.
---
## Reference: Type System (Old Typechecker Only)
**Available NOW:**
- `--!strict` / `--!nonstrict` / `--!nocheck`
- Basic annotations: `local x: number`, `function f(a: string): boolean`
- Optional: `string?`, Union: `number | string`, Intersection: `T1 & T2`
- Generics: `function id<T>(x: T): T`, explicit instantiation: `f<<number>>()`
- Cast: `x :: string`
- Array: `{number}`, Dict: `{[string]: number}`, Table: `type T = { field: type }`
- `typeof()`, `export type`, typed variadics, alias defaults, singletons
- `never`, `unknown`, `read`/`write` modifiers
- Type refinements: `type()`, `typeof()`, `:IsA()`, truthiness, equality, `assert()`
**DO NOT recommend (requires new typechecker):**
- `keyof<T>`, `index<T,K>`, `rawkeyof<T>`, `rawget<T,K>`
- `getmetatable<T>`, `setmetatable<T,M>`
- User-defined type functions, negation types (`~T`)
- Relaxed recursive type restrictions
## Reference: Type Refinement Patterns
The compiler narrows types after certain checks, improving both type safety and native codegen. Use refinements to provide type information without explicit annotations where the type is known from a runtime check:
- **Truthiness:** `if x then` narrows `x` from falsy (`nil`/`false`)
- **Type guards:** `if type(x) == "number" then` narrows `x` to `number`
- **Typeof guards (Roblox):** `if typeof(x) == "Vector3" then` narrows to `Vector3`; `x:IsA("TextLabel")` narrows `Instance` to subclass
- **Equality:** `if x == "hello" then` narrows `x` to singleton `"hello"`
- **Assert:** `assert(type(x) == "string")` narrows `x` to `string` after the call
- **Composition:** Supports `and`, `or`, `not` for compound refinements
## Reference: Deprecated Patterns
| Deprecated | Replacement | Impact |
|---|---|---|
| `getfenv()` / `setfenv()` | `debug.info(i, "snl")` | Disables ALL optimizations. Even read-only `getfenv()` deoptimizes. |
| `loadstring()` | Restructure with `require` | Marks env "impure", disables imports |
| `table.getn(t)` | `#t` | Slower |
| `table.foreach(t, f)` | `for k, v in t do` | Slower |
| `table.foreachi(t, f)` | `for i, v in t do` | Slower |
| `wait()` | `task.wait()` | Deprecated Roblox API |
| `obj.Method(obj, ...)` | `obj:Method(...)` | Misses fast method call instruction |
| `string:method()` | `string.method(s)` | Fastcall: `string.byte(s)` faster than `s:byte()` |
## Reference: Lint Rules
| Lint | Name | Optimization impact |
|---|---|---|
| 10 | BuiltinGlobalWrite | Overwriting builtins disables fastcall |
| 22 | DeprecatedApi | Deprecated APIs: perf/correctness issues |
| 23 | TableOperations | Wrong index, redundant insert, `#` on non-arrays |
| 3 | GlobalUsedAsLocal | Global in one function -> should be local |
| 7 | LocalUnused | Dead locals |
| 12 | UnreachableCode | Dead code |
| 25 | MisleadingAndOr | `a and false or c` bugs |
## Reference: Bytecode Pattern Catalog
| Slow | Fast | Why |
|---|---|---|
| `math.floor(a / b)` | `a // b` | Dedicated VM opcode |
| `data[i].x = data[i].x + 1` | `data[i].x += 1` | LHS evaluated once |
| `a + (b - a) * t` | `math.lerp(a, b, t)` | Exact at endpoints, monotonic |
| `"pre" .. v .. "suf"` | `` `pre{v}suf` `` | Optimized `string.format` |
| Manual byte swap | `bit32.byteswap(n)` | CPU bswap |
| Manual log2 loop | `bit32.countlz(n)` | CPU instruction, ~8x faster |
| `cond and A or B` | `if cond then A else B` | Safe for falsy, one branch |
## Reference: Allocation Reduction
| Slow | Fast | Why |
|---|---|---|
| `local t = {}` in loop | `table.clear(t)` + reuse | Avoids GC pressure |
| Repeated `table.insert` | `table.create(n)` + indexed writes | Preallocated |
| Manual `pairs` clone | `table.clone(t)` | Faster, copies layout |
| `string.byte`/`char` loops | `string.pack`/`unpack` | Native implementation |
| String binary data | `buffer` type | Fixed-size, efficient |
## Reference: Inlining Rules
| Blocks inlining | Enables inlining | Why |
|---|---|---|
| `function Module.foo()` | `local function foo()` | Mutable table prevents |
| `function Obj:method()` | `local function method(self)` | OOP syntax not inlineable |
| Recursive function | Split base/recursive | Recursion prevents |
| Deep upvalue captures | Pass as parameters | Reduces closure cost |
## Reference: Native Codegen
**`@native` vs `--!native`:**
- `--!native`: entire script. Good for math-heavy utility modules.
- `@native`: per-function. Better for selective hot functions or near the 1M instruction limit.
- Inner functions do NOT inherit `@native`.
- Top-level code runs once; `@native` has minimal benefit there.
**Hurts native:** `getfenv`/`setfenv`, wrong types to typed functions, non-numeric math args, breakpoints, size limits (64K instructions/block, 32K blocks/function, 1M/module).
**Helps native:** Type annotations (especially `Vector3`), small functions, consistent table shapes, `buffer` ops, `bit32` ops.
## Reference: CLI Tools
### `luau-compile` -- Static bytecode/codegen analysis (primary tool)
Modes (mutually exclusive): `binary` (raw bytecode), `text` (human-readable opcodes with source annotations), `remarks` (source with inlining/allocation comments -- **most useful**), `codegen` (native assembly, requires `--target`).
Options: `-O<n>` (0-2, default 1; `-O2` enables inlining), `-g<n>` (debug level 0-2), `--target=<arch>` (`x64`, `x64_ms`, `a64`, `a64_nf`; use `x64_ms` on Windows), `--record-stats=<total|file|function>` (JSON stats: bytecodeInstructionCount, lowerStats with spills/skipped/blocks/errors), `--bytecode-summary` (opcode distribution, requires `--record-stats=function`), `--stats-file=<name>` (default `stats.json`), `--timetrace` (trace.json), `--vector-lib`/`--vector-ctor`/`--vector-type` (always set for Roblox).
### `luau-analyze` -- Type checking and linting
Modes: omitted (typecheck + lint), `--annotate` (source with all inferred types inline -- **critical for finding `any`**).
Options: `--mode=strict` (force strict even without directive), `--formatter=plain` (machine-parseable), `--formatter=gnu` (grep-friendly), `--timetrace`.
### `luau` -- Runtime execution, profiling, and coverage
**Limited for Roblox scripts** (most depend on engine APIs). Only usable for pure-logic modules. For Roblox scripts, use Studio Script Profiler and `debug.dumpcodesize()`.
Options: `-O<n>`, `-g<n>`, `--codegen` (native execution), `--profile[=N]` (sampling at N Hz, default 10000, outputs `profile.out`), `--coverage` (outputs `coverage.out`), `--timetrace`, `-a` (program args).
### `luau-ast` -- AST dump
No flags. Outputs full JSON AST with node types, source locations, variable scopes, type annotations. Useful for programmatic analysis: function nesting depth, function body size (monolith detection), closure captures (upvalue analysis), table construction patterns, loop-invariant expression detection.
## Reference: VM Architecture
**Fastcall builtins:** `assert`, `type`, `typeof`, `rawget`/`rawset`/`rawequal`/`rawlen`, `getmetatable`/`setmetatable`, `tonumber`/`tostring`, most `math.*` (not `noise`/`random`/`randomseed`), `bit32.*`, some `string.*`/`table.*`. Partial: `assert` (unused return + truthy), `bit32.extract` (constant field/width), `select(n, ...)` O(1). With `-O2`: constant-arg builtins folded; `math.pi`, `math.huge` folded.
**Inline caching:** HREF-style for table fields. Compiler predicts hash slot; VM corrects. Best: compile-time field name, no metatables, uniform shapes.
**Import resolution:** `math.max` resolved at load time. Invalidated by `getfenv`/`setfenv`/`loadstring`.
**GC:** Incremental mark-sweep with "GC assists" (allocation pays proportional GC). Paged sweeper (16 KB pages). No `__gc` metamethod.
**Compiler:** Multi-pass. Constant folding across functions/locals. Upvalue optimization. `-O2`: inlining, loop unrolling (constant bounds), aggressive constant folding. Interprocedural limited to single modules.
**Table length:** `#t` is O(1) cached, worst case O(log N) branch-free binary search. `table.insert`/`table.remove` update the cache.
**Upvalues:** ~90% immutable. Immutable = no allocation, no closing, faster access. Mutable = extra object.
**Closures:** Cached when no upvalues, or all upvalues immutable and module-scope.
## Reference: Documentation to Consult
Read these docs during execution if needed for specific detail. Two doc trees:
- `.cursor/rules/luau/` -- Official Luau site docs (guides, types, reference, getting-started)
- `.cursor/rules/luau-rfcs/` -- Luau RFCs (language proposals, library additions)
**Luau site docs (primary):**
- `.cursor/rules/luau/guides/performance.md` -- **The official Luau performance guide.** Covers interpreter, compiler, fastcalls, inline caching, imports, table creation, GC, inlining, loop unrolling, upvalues, closures.
- `.cursor/rules/luau/guides/profile.md` -- Profiling guide (flame graphs, sampling)
- `.cursor/rules/luau/types/` -- Full type system docs (basic-types, tables, generics, refinements, unions-and-intersections, object-oriented-programs, roblox-types, considerations)
- `.cursor/rules/luau/getting-started/lint.md` -- All 28 lint rules
- `.cursor/rules/luau/reference/library.md` -- Complete library reference
- `.cursor/rules/luau/getting-started/compatibility.md` -- Luau vs Lua differences
**Luau RFCs (specific features):**
- `.cursor/rules/luau-rfcs/function-inlining.md` -- Inlining RFC (no user `@inline`, automatic at `-O2`)
- `.cursor/rules/luau-rfcs/syntax-attribute-functions-native.md` -- `@native` per-function attribute
- `.cursor/rules/luau-rfcs/function-table-create-find.md`, `function-table-clear.md`, `function-table-clone.md`, `function-table-freeze.md` -- Table operations
- `.cursor/rules/luau-rfcs/function-math-lerp.md` -- `math.lerp`
- `.cursor/rules/luau-rfcs/function-bit32-byteswap.md`, `function-bit32-countlz-countrz.md` -- CPU-level bit ops
- `.cursor/rules/luau-rfcs/function-buffer-bits.md`, `type-byte-buffer.md` -- Buffer type and ops
- `.cursor/rules/luau-rfcs/vector-library.md` -- Vector library and fastcalls
- `.cursor/rules/luau-rfcs/function-string-pack-unpack.md` -- Binary string operations
- `.cursor/rules/luau-rfcs/syntax-floor-division-operator.md` -- `//` dedicated opcode
- `.cursor/rules/luau-rfcs/syntax-compound-assignment.md` -- `+=` single LHS evaluation
- `.cursor/rules/luau-rfcs/deprecate-getfenv-setfenv.md` -- Why fenv disables optimizations
- `.cursor/rules/luau-rfcs/deprecate-table-getn-foreach.md` -- Deprecated table functions
- `.cursor/rules/luau-rfcs/generalized-iteration.md` -- Modern iteration patterns
- `.cursor/rules/luau-rfcs/generic-functions.md`, `never-and-unknown-types.md`, `syntax-type-ascription.md` -- Type system RFCs
**Cursor rules:**
- `.cursor/rules/luau.mdc` -- Luau language rules and optimization tips
**Roblox docs:**
- `.cursor/rules/roblox/en-us/luau/native-code-gen.md` -- Native codegen (size limits, Vector3 annotation impact)
- `.cursor/rules/roblox/en-us/luau/type-checking.md` -- Type annotation syntax and modes
- `.cursor/rules/roblox/en-us/performance-optimization/improve.md` -- Script computation, memory, physics, rendering
- `.cursor/rules/roblox/en-us/performance-optimization/design.md` -- Event-driven patterns, frame budgets
- `.cursor/rules/roblox/en-us/luau/variables.md` -- Local vs global performance
- `.cursor/rules/roblox/en-us/luau/scope.md` -- Scope performance implications

425
.cursor/plans/ambiguous_path_handling_f2eaef11.plan.md Normal file
View file

@ -0,0 +1,425 @@
---
name: Ambiguous Path Handling
overview: "Replace the current duplicate-name blacklist/skip system with a full ambiguous path handling system: a 3-pass matching algorithm (3 implementations: 2 Rust, 1 Lua), filesystem-collision-based dedup (~N suffixes), and filesystem-name-based ref paths. Implementation starts with the Rust syncback matching algorithm for testability, then expands outward."
todos:
- id: phase1a-remove-blacklist-rust
content: Remove duplicate-skip blacklist from Rust side (dir.rs, project.rs, syncback/mod.rs)
status: completed
- id: phase1b-remove-blacklist-plugin
content: Cherry-pick plugin relaxations from rbxm-ambiguous (diff.lua, encodeInstance.lua)
status: completed
- id: phase1c-two-pass-syncback
content: Restructure syncback_dir_no_meta to two-pass child processing (tentative names, then collision grouping)
status: completed
- id: phase1d-dedup-key-semantics
content: Change dedup key in file_names.rs from bare slug to full filesystem name component
status: completed
- id: phase1e-suffix-cleanup
content: Implement suffix cleanup rules (gap-tolerant, base-name promotion, group-to-1)
status: completed
- id: phase2a-matching-syncback
content: Implement 3-pass matching algorithm for syncback (src/syncback/matching.rs)
status: completed
- id: phase2b-integrate-syncback
content: Integrate matching into syncback_dir_no_meta (dir.rs)
status: completed
- id: phase2c-matching-forward
content: Implement 3-pass matching algorithm for forward sync (src/snapshot/matching.rs)
status: completed
- id: phase2d-integrate-forward
content: Integrate matching into compute_children_patches (patch_compute.rs)
status: completed
- id: phase2e-matching-cache
content: Add optional matching cache (parent_ref -> match_assignments) with invalidation
status: pending
- id: phase3a-fs-name-ref-paths
content: Switch ref_target_path() to filesystem-name-based paths in rojo_ref.rs
status: completed
- id: phase3b-tree-resolution
content: Update get_instance_by_path() in tree.rs to resolve filesystem-name segments
status: completed
- id: phase3c-ref-properties
content: Update ref_properties.rs for filesystem-name-based linking
status: completed
- id: phase3d-patch-apply
content: Update patch_apply.rs ref resolution for new path format
status: completed
- id: phase3e-ref-update-infra
content: Extend RefPathIndex and ref update infrastructure for all rename/dedup cases
status: completed
- id: phase4a-change-processor
content: Update change_processor.rs for VFS echo prevention on dedup renames and suffix cleanup
status: completed
- id: phase4b-api-write
content: Update api.rs for two-pass write, dedup in all 4 branches, dual write consistency
status: completed
- id: phase5a-lua-matching
content: Implement 3-pass matching algorithm in Lua (hydrate.lua + matching.lua + XXH32.luau)
status: completed
- id: phase5b-plugin-integration
content: Update ChangeBatcher and confirmation dialog for duplicate-named instances
status: completed
- id: phase6-cherry-picks
content: Cherry-pick roundtrip test improvements and macOS test ignore from rbxm-ambiguous
status: completed
- id: phase7-testing
content: "Implement comprehensive test suite: shared test vectors, all spec scenarios, roundtrip tests"
status: completed
isProject: false
---
# Ambiguous Path Handling
## Architecture
The feature introduces three interlocking subsystems that replace the current "skip duplicates" approach:
```mermaid
flowchart TD
subgraph matching [Matching Algorithm]
MA1[Pass 1: Unique Names]
MA2[Pass 2: Ref Discriminators]
MA3[Pass 3: Similarity Scoring]
MA1 --> MA2 --> MA3
end
subgraph dedup [Dedup System]
D1[Compute tentative FS name per child]
D2[Group by lowercased FS collision]
D3[Assign ~N suffixes within groups]
end
subgraph refs [Ref Path System]
R1[Paths use full FS names incl extensions]
R2[RefPathIndex tracks all refs]
R3[Batch update on rename/dedup change]
end
matching -->|"pairing determines which get new names"| dedup
dedup -->|"final FS names feed into"| refs
```
**Execution order per sync operation:** Matching runs first (pairs instances), dedup runs second (assigns filesystem names), ref paths are computed last (using final filesystem names).
**Performance target:** System must handle large games (10K+ instances) with <100ms latency per operation. Architecture must not have O(n^2) bottlenecks on the full tree. The matching algorithm should only run on the affected subtree when possible. Optimization can be deferred but the structure must support it.
---
## Phase 1: Foundation - Remove Blacklist and Restructure Dedup
### 1a. Remove duplicate-skip blacklist (Rust)
`**[src/snapshot_middleware/dir.rs](src/snapshot_middleware/dir.rs)`** -- Remove the duplicate child detection and skip logic in `syncback_dir_no_meta()`:
- Lines 151-181: `child_name_counts` / `duplicate_names` HashSet construction
- Lines 235-239, 321-324: `if duplicate_names.contains(...) { continue; }` guards
- Related stats recording (`record_duplicate_names_batch`)
`**[src/snapshot_middleware/project.rs](src/snapshot_middleware/project.rs)`** -- Replace `bail!` on duplicate names with dedup suffix handling. Adapt the rbxm-ambiguous branch's case-insensitive detection (skip + warn), but instead of skipping, route through the new dedup system. **Important:** project KEYS are unique by JSON definition and do NOT participate in dedup themselves. Only filesystem children under project nodes need dedup when they collide. Evaluate whether to remove the warn-and-skip guard from the rbxm branch or keep it as a safety net.
`**[src/syncback/mod.rs](src/syncback/mod.rs)`** -- Remove `has_duplicate_children` detection used for skipping. Keep the infrastructure for detecting duplicates since it feeds into the dedup system.
### 1b. Remove duplicate-skip blacklist (Plugin)
Take the rbxm-ambiguous branch changes directly:
`**[plugin/src/Reconciler/diff.lua](plugin/src/Reconciler/diff.lua)`** -- Remove `findDuplicateNames()`, `ambiguousIds`, `scanForDuplicates()`, all `if ambiguousIds[id] then` guards, and the `skippedDuplicateCount` tracking. Duplicate-named instances now flow through diff normally.
`**[plugin/src/ChangeBatcher/encodeInstance.lua](plugin/src/ChangeBatcher/encodeInstance.lua)`** -- Remove `skippedDuplicateCount`, remove child duplicate skip loop, keep `isPathUnique` as a debug log (not a skip). Return single function instead of wrapper.
### 1c. Restructure syncback to two-pass child processing
`**[src/snapshot_middleware/dir.rs](src/snapshot_middleware/dir.rs)`** `syncback_dir_no_meta()` -- Replace the current one-at-a-time `with_joined_path()` loop with:
**Pass 1 -- Tentative names:** Iterate ALL children, call `get_best_middleware()` for each, compute tentative filesystem name (slug + extension for files, directory name for dirs). Also determine whether each instance would be file or directory (for init-style collision detection).
**Pass 2 -- Dedup grouping:** Group children by lowercased tentative filesystem name. Within each collision group (size > 1), assign ~N suffixes. The matching algorithm's pairing (Phase 2) determines which instances preserve existing filenames and which get new suffixes.
### 1d. Change dedup key semantics in file_names.rs
`**[src/syncback/file_names.rs](src/syncback/file_names.rs)`** -- The current `dedup_key` is a bare slug (without extension). Change it to be the **full filesystem name component** (slug + extension for files, directory name for dirs), lowercased.
Current dedup key for `Script "Foo"` = `"foo"` (bare slug).
New dedup key for `Script "Foo"` = `"foo.server.luau"` (full filesystem name).
This means:
- `Folder "Foo"` (key: `"foo"`) and `Script "Foo"` (key: `"foo.server.luau"`) do NOT collide -- correct.
- `Script "Foo" (server)` and `Script "Foo" (server)` both key: `"foo.server.luau"` -- collision, dedup applied.
- `Folder "Foo"` and init-style `Script "Foo"` both key: `"foo"` (both directories) -- collision, dedup applied.
Update `deduplicate_name()` to accept the full filesystem name as input and apply the suffix to the stem before the compound extension: `Foo~1.server.luau`, not `Foo.server~1.luau`.
### 1e. Suffix cleanup rules
Implement in a shared helper (used by syncback, change_processor, and api.rs):
- **Gap-tolerant deletion:** Deleting `~1` from `{Foo, Foo~1, Foo~2}` leaves `{Foo, Foo~2}`. No renumbering.
- **Base-name promotion:** Deleting `Foo` (base) from `{Foo, Foo~1, Foo~2}` promotes `Foo~1` to `Foo`. `Foo~2` stays.
- **Group-to-1 cleanup:** Deleting from `{Foo, Foo~1}` leaves one instance -- remove its suffix entirely (rename to clean name).
---
## Phase 2: Matching Algorithm
### 2a. Rust syncback matching (`src/syncback/matching.rs` -- new file)
Core 3-pass algorithm operating on `WeakDom` instances (syncback context):
```
Input: new_children: Vec<Ref> (from Roblox file WeakDom)
old_children: Vec<Ref> (from filesystem WeakDom)
new_dom: &WeakDom, old_dom: &WeakDom
hashes: &HashMap<Ref, Hash> (precomputed by hash_tree)
Output: MatchResult { matched: Vec<(new_ref, old_ref)>,
unmatched_new: Vec<new_ref>,
unmatched_old: Vec<old_ref> }
```
**Pass 1 -- Unique names (O(n)):** Build name-count maps for both sides. Any name that appears exactly once on both sides is an instant match. **Within same-Name groups, ClassName is a fast first discriminator:** if two siblings share name "Part" but one is Part and one is MeshPart, and each (Name, ClassName) pair is unique on both sides, match them in Pass 1 without needing Pass 2/3. **Fast path:** if all names are unique (or all resolved by name+class), return immediately (skip P2+P3).
**Pass 2 -- Ref discriminators:** For remaining unmatched groups (same true Name), use pre-resolved Ref properties to differentiate. If instance A has a Ref pointing to a target already matched in P1, and only one candidate on the other side has the equivalent Ref, that's a match. Also consider incoming refs (other matched instances pointing TO a candidate). **Unmatched ref targets:** if a Ref target is itself in an unmatched ambiguous group (not resolved in Pass 1), treat it as a weaker signal -- compare target names/classes without requiring a confirmed match. Do NOT recursively match the target's group. If insufficient, the instance falls through to Pass 3.
**Pass 3 -- Similarity scoring:** For still-unmatched groups, narrow by ClassName first, then compute pairwise similarity using precomputed blake3 subtree hashes. Greedy best-match-first assignment: pick highest similarity pair, remove both, repeat. Tiebreaker: DOM child order (index in parent's children list).
**Ref pre-resolution (prerequisite):** Before any pass, resolve Ref properties on both sides. On the new side (Roblox file), Refs are WeakDom referents -- build a `Ref -> instance_path` map. On the old side (filesystem), Refs may be stored as `Rojo_Ref_`* attributes -- parse and resolve.
### 2b. Integrate into syncback_dir_no_meta (dir.rs)
In `syncback_dir_no_meta()`:
1. Call matching algorithm to pair `new_inst.children()` with old children
2. For matched pairs: preserve existing filesystem assignment (old instance keeps its filename)
3. For unmatched new: generate new filename via dedup system (Phase 1c)
4. For unmatched old: mark for deletion
### 2c. Forward sync matching (`src/snapshot/matching.rs` -- new file)
Same 3-pass algorithm but operating on `InstanceSnapshot` vs `RojoTree` instances:
```
Input: snapshot_children: Vec<InstanceSnapshot>
tree_children: Vec<Ref> (RojoTree instance IDs)
tree: &RojoTree
Output: MatchResult { matched, unmatched_snapshot, unmatched_tree }
```
Key differences from syncback version:
- **True Name resolution from filesystem:** On the filesystem side, resolving the true Instance.Name is non-trivial. Resolution chain: (1) `name` field in `.meta.json5` if present (overrides filename), (2) `name` field in `.model.json5`, (3) filename stem after stripping extensions and script suffixes (if no meta override). The matching algorithm must resolve all true Names before building groups. On the tree/plugin side, `Instance.Name` / `instance.name()` is readily available.
- **Ref pre-resolution fallback order:** Try the old RojoTree FIRST (most refs target existing instances). If the path can't be resolved in the old tree (target is a new instance), fall back to walking the snapshot tree. The snapshot is only a subtree (the re-snapshotted directory), so this fallback only works for refs within the same subtree. Refs to new instances outside the current snapshot remain unresolved and are skipped during matching (treated as nil).
- Snapshot children have `Rojo_Ref_`* as string paths in attributes -- resolve against tree (then snapshot) before matching
- Tree children have Refs as UIDs -- already resolved
- Hash infrastructure: snapshot side doesn't have precomputed hashes yet -- compute structural fingerprint on demand
### 2d. Integrate into patch_compute.rs
Replace the greedy name+class matching in `compute_children_patches()` with the forward sync matching algorithm. The current code:
```6:17:src/snapshot/patch_compute.rs
// Current: greedy first-match by name+class
.find(|(instance_index, instance_child_id)| {
if paired_instances[*instance_index] { return false; }
snapshot_child.name == instance_child.name()
&& snapshot_child.class_name == instance_child.class_name()
})
```
Becomes: call to matching algorithm, then iterate matched pairs for recursive `compute_patch_set_internal()`, add unmatched snapshots as `PatchAdd`, add unmatched tree children as `removed_instances`.
### 2e. Matching cache
Add an optional cache mapping `parent_ref -> (children_list_hash, match_assignments)` to avoid re-running the full algorithm when children haven't changed. Invalidate per-parent when a child is added, removed, or renamed under that parent. During two-way sync, each `/api/write` mutation invalidates the affected parent's cache.
---
## Phase 3: Ref Path System
### 3a. Switch to filesystem-name-based paths
`**[src/rojo_ref.rs](src/rojo_ref.rs)**` -- `ref_target_path()`:
Current: uses `escape_ref_path_segment(&inst.name)` producing instance-name paths like `"Workspace/A\/B"`.
New: uses the **full filesystem name** of each instance (slug + extension + dedup suffix for files; directory name + dedup suffix for dirs). Path from DataModel's children downward.
Examples:
- `Folder "Foo"` -> segment `"Foo"`
- `ModuleScript "Foo"` -> segment `"Foo.luau"`
- `Script "Foo"` (server) -> segment `"Foo.server.luau"`
- `Script "Foo"` with children (init-style) -> segment `"Foo"` (directory)
- Dedup'd `ModuleScript "Foo"` -> segment `"Foo~1.luau"`
Remove `escape_ref_path_segment()`, `unescape_ref_path_segment()`, `split_ref_path()`. Path splitting becomes a simple `path.split('/')` since filesystem names can't contain `/` (slugified to `_`).
### 3b. Filesystem-name resolution in tree.rs
`**[src/snapshot/tree.rs](src/snapshot/tree.rs)`** -- `get_instance_by_path()`:
Current: walks tree matching each segment against `child.name` (instance name).
New: walks tree matching each segment against the **filesystem name** of each child. Derive from:
1. `instigating_source` filename/dirname (for file-backed instances)
2. Fallback: compute what the filesystem name WOULD be (slugify name + middleware extension) for instances without instigating_source (e.g., inside .rbxm files, newly added)
### 3c. Update ref_properties.rs
`**[src/syncback/ref_properties.rs](src/syncback/ref_properties.rs)`:**
`link_referents()` currently writes path-based (`Rojo_Ref`_*) using instance-name paths. Update to use filesystem-name paths (the dedup'd filenames assigned during syncback).
`collect_referents()` currently uses `is_path_unique` to decide between path-based and ID-based linking. With the dedup system, ALL paths are unique (filesystem names are unique by definition). Simplify: always use path-based linking for file-backed instances. Fall back to `$id`/`Rojo_Target`_* only for instances without filesystem backing (inside .rbxm files).
### 3d. Update patch_apply.rs
`**[src/snapshot/patch_apply.rs](src/snapshot/patch_apply.rs)`:**
Update the ref resolution in `finalize_patch_application()` to use the new filesystem-name-based `get_instance_by_path()`.
### 3e. Ref path update infrastructure
Extend `RefPathIndex` and `update_ref_paths_after_rename()` in [change_processor.rs](src/change_processor.rs) to handle all cases:
- Instance rename -> slug change -> all refs to it and descendants update
- Dedup reassignment -> suffix change -> all refs update
- Reparenting -> entire ancestor path changes -> all refs update
- Script type transition -> extension change -> all refs through this instance update
- File-to-folder transition -> segment format change -> all refs update
- Batch writes: collect ALL affected ref updates, write each file once
---
## Phase 4: Two-Way Sync Integration
### 4a. change_processor.rs
`**[src/change_processor.rs](src/change_processor.rs)`:**
- **VFS echo prevention for dedup renames:** When suffix cleanup renames files (e.g., `Attachment~1` -> `Attachment`), call `suppress_path()` / `suppress_path_any()` for both old and new paths before the rename. **New instance additions must NOT be suppressed** -- the VFS watcher needs to pick up new files for forward sync.
- **Dedup suffix cleanup on deletion:** When a child deletion triggers suffix cleanup (base-name promotion or group-to-1), perform the rename and ref path updates.
- **Self-initiated rename tagging:** Tag dedup-triggered renames so `handle_vfs_event()` recognizes them as self-initiated and skips re-snapshotting.
### 4b. api.rs
`**[src/web/api.rs](src/web/api.rs)`:**
- **Two-pass /api/write:** First pass: apply ALL mutations (renames, property changes, adds, removes) to the tree. Second pass: recompute all affected ref slug paths based on final tree state. This ensures refs always use newest slugs.
- **4 file format branches in `syncback_updated_properties()`:** Each branch must compute the correct dedup'd filename for the meta/model file. When an instance has a dedup suffix, the meta path includes it: `Foo~1.meta.json5`, `Foo~1/init.meta.json5`, etc.
- **Added instances in dedup groups:** `syncback_added_instance()` must check for filesystem-name collisions with existing siblings and assign appropriate ~N suffix.
- **Dual write path consistency:** File writes and PatchSet sent to ChangeProcessor must use the same dedup'd names.
---
## Phase 5: Plugin Updates
### 5a. Lua matching algorithm (plugin/src/Reconciler/matching.lua -- new file)
`**[plugin/src/Reconciler/hydrate.lua](plugin/src/Reconciler/hydrate.lua)`:**
Replace the current name+class greedy matching with the 3-pass algorithm:
- **Pass 1:** Unique name matching (same as Rust)
- **Pass 2:** Ref discriminators. Server sends virtual instances with UIDs. Plugin resolves UIDs via instanceMap. If a Ref property on a virtual instance points to UID-7 (already matched in P1), and only one Studio candidate has the equivalent Ref, match.
- **Pass 3:** Structural fingerprinting with XXH32. Fingerprint = hash of (child count + sorted child name+class pairs). Do NOT hash full property values (too expensive). For actual similarity scoring, compare properties directly with early-exit ordering: ClassName -> Tags -> Attributes -> scalar properties -> Source (last, most expensive).
`**[plugin/src/XXH32.luau](plugin/src/XXH32.luau)`** -- New file containing the XXH32 implementation from the spec (lines 239-350 of the prompt).
**Caching:** Compute structural fingerprints LAZILY (only for instances entering Pass 3). Cache for duration of the matching call, not across calls.
### 5b. ChangeBatcher updates
`**[plugin/src/ChangeBatcher/init.lua](plugin/src/ChangeBatcher/init.lua)`:** Handle duplicate-named instances flowing through the batcher without skipping.
### 5c. Confirmation dialog
Dedup suffix renames, ref path updates, and meta file changes triggered by the matching algorithm must appear in the review UI in Always and Initial sync modes. No silent application.
---
## Phase 6: Cherry-Picks from rbxm-ambiguous
Selectively take these from the `rbxm-ambiguous` branch:
- **Plugin relaxations** (diff.lua, encodeInstance.lua) -- already described in Phase 1b
- **Roundtrip test improvements** (`[tests/tests/syncback_roundtrip.rs](tests/tests/syncback_roundtrip.rs)`) -- Vec-based `compare_children` (handles duplicates correctly), Tags comparison in `compare_properties`
- **project.rs case-insensitive detection** -- case-insensitive name counting for duplicate detection (adapt for dedup instead of skip)
- **macOS ref_properties test ignore** -- platform-specific test attribute
**Do NOT take:** `ambiguous_container` metadata, `find_rbxm_container`, `reserialize_rbxm_container`, rbxm fallback block, ambiguous test fixtures that output .rbxm files.
**Non-requirement: no .rbxm transition logic.** Syncback rewrites the filesystem from scratch (clean) or matches instances to existing files (incremental) -- either way, old .rbxm containers simply don't get recreated. User-intentional .rbxm files placed manually in their project still load fine via `snapshot_rbxm` during forward sync -- that's unchanged. Do not build migration code for converting old .rbxm files.
---
## Phase 7: Testing
### Shared test vectors
Create structured test data (JSON or Rust structs) defining:
- Two lists of instance descriptions (name, class, properties, refs, children)
- Expected match pairing output
Both Rust implementations and the Lua implementation must produce identical pairings for all vectors.
### Test scenarios (minimum required from spec)
**Dedup:** Two identical Folders, Folders with different children, Folders with identical content (tiebreaker test), Folder + Script same name (no dedup), Folder + init-style Script (dedup), natural tilde name ("Foo~1" slugifies differently), third duplicate added, suffix cleanup (group-to-1), gap tolerance, base-name promotion, idempotency, roundtrip.
**Ref paths:** Basic ref survives syncback, ref to dedup'd instance, ref survives gap-tolerant deletion, ref survives base-name promotion, ref survives group-to-1, ref survives rename, ref survives reparent, deeply nested through dedup layers, cross-file ref update, ref to deleted instance, tilde-named instance ref, two-way sync ref update, concurrent ref+rename, forward sync manual edit, fallback to $id, mass ref stress test (50+ instances).
**Matching:** ClassName discriminator, Ref discriminator, property similarity, matching stability, preserves existing pairings, fast path verification.
**Dedup collision:** Folder + ModuleScript (no collision), Folder + Folder (collision), Server + Client script (no collision), Server + Server (collision), ModuleScript + ModuleScript, slug collision ("Foo?" + "Foo*"), case-insensitive collision ("Foo" + "foo").
**Cross-system integration:** Full roundtrip with duplicates, incremental syncback after rename, two-way sync add/delete, forward sync edit dedup'd file, slugify+dedup interaction, project file + filesystem duplicates.
**Sync invariants:** CLI syncback parity, 4 file format branches, compound extension dedup, InstigatingSource after dedup, file-to-folder transition with dedup, script type transition in dedup group, dual write path consistency, review dialogue with dedup, atomicity, determinism, idempotency, pre-seeded dedup.
### Test infrastructure
- Syncback test fixtures in `[rojo-test/syncback-tests/](rojo-test/syncback-tests/)`
- Build test fixtures in `[rojo-test/build-tests/](rojo-test/build-tests/)`
- Integration tests in `[tests/tests/](tests/tests/)`
- Generate .rbxm fixtures programmatically (see `tests/generate_ambiguous_fixtures.rs` from rbxm-ambiguous branch)
---
## Key Invariants to Preserve
1. **Round-trip identity:** syncback -> build -> compare = bit-identical trees
2. **CLI syncback parity:** two-way sync produces byte-identical filesystem output to CLI syncback
3. **Determinism:** same input always produces same filesystem output (sorted child ordering, deterministic suffix assignment)
4. **INSTIGATING SOURCE consistency:** dedup'd paths reflected in InstigatingSource::Path
5. **Compound extensions:** suffix before compound extension (`Foo~1.server.luau`, not `Foo.server~1.luau`)
6. **Meta pairing:** `Foo~1.server.luau` pairs with `Foo~1.meta.json5` (strip script suffix, keep dedup suffix)
---
## Key Files Changed
| File | Change |
| --------------------------------------------- | ------------------------------------------------------------ |
| `src/syncback/file_names.rs` | Dedup key = full FS name; suffix on stem before compound ext |
| `src/syncback/dir.rs` (snapshot_middleware) | Two-pass child processing; remove blacklist |
| `src/syncback/matching.rs` (NEW) | 3-pass matching algorithm for syncback |
| `src/snapshot/matching.rs` (NEW) | 3-pass matching algorithm for forward sync |
| `src/snapshot/patch_compute.rs` | Replace greedy matching with algorithm |
| `src/snapshot/patch_apply.rs` | Update ref resolution for FS-name paths |
| `src/snapshot/tree.rs` | `get_instance_by_path()` uses FS names |
| `src/rojo_ref.rs` | FS-name paths; remove escape/split; update `ref_target_path` |
| `src/syncback/ref_properties.rs` | FS-name-based linking; simplify path uniqueness |
| `src/change_processor.rs` | VFS echo for dedup renames; suffix cleanup |
| `src/web/api.rs` | Two-pass write; dedup in all 4 branches |
| `src/snapshot_middleware/project.rs` | Replace bail with dedup; case-insensitive |
| `plugin/src/Reconciler/hydrate.lua` | 3-pass matching algorithm (Lua) |
| `plugin/src/Reconciler/diff.lua` | Remove blacklist |
| `plugin/src/ChangeBatcher/encodeInstance.lua` | Remove blacklist |
| `plugin/src/XXH32.luau` (NEW) | Hash function for Lua-side fingerprinting |

310
.cursor/plans/ambiguous_paths_audit_fixes_9d3ed6da.plan.md Normal file
View file

@ -0,0 +1,310 @@
---
name: Ambiguous paths audit fixes
overview: "Fix 6 issues found during the ambiguous-paths branch audit: make Rust matching recursive, fix ref path format in api.rs added_paths, restructure syncback ref collection for dedup-aware paths, improve dedup cleanup grouping precision, extract shared trueEquals module, and correct a misleading comment."
todos:
- id: fix1-recursive-matching
content: "Fix 1: Make Rust matching implementations recursive (both forward sync and syncback) to match the Lua plugin behavior. Add depth limit and best_so_far pruning."
status: completed
- id: fix2-added-paths
content: "Fix 2: Replace escape_ref_path_segment with slugify_name in api.rs added_paths to fix ref path format mismatch."
status: completed
- id: fix3-dedup-refs
content: "Fix 3: Restructure syncback ref collection to use final dedup'd filesystem names instead of tentative paths. Fix misleading comment."
status: completed
- id: fix4-dedup-cleanup
content: "Fix 4: Change dedup cleanup in change_processor to match by full filesystem name (stem + extension) instead of stem only."
status: completed
- id: fix5-shared-trueequals
content: "Fix 5: Extract trueEquals to a shared Lua module to eliminate duplication between matching.lua and diff.lua."
status: completed
- id: run-tests
content: Write all required tests and run CI to verify everything passes.
status: completed
isProject: false
---
# Ambiguous Path Handling Audit Fixes
> This plan was generated by `/audit` from a Plan-mode session.
> Refer to `.cursor/rules/atlas.mdc` for full project standards.
## Standards
These standards govern every fix in this plan. An implementer MUST read `.cursor/rules/atlas.mdc` before starting.
### Round-Trip Identity (from `atlas.mdc`)
Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a **bit-identical instance tree** -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.
### CLI Syncback Parity (from `atlas.mdc`)
Plugin-based sync must produce **exactly the same filesystem output** as `atlas syncback`. Any divergence is a bug until proven otherwise.
### Code Quality (from `atlas.mdc`)
DRY code that is easy to reason about and hard to get wrong. The same slugify/dedup/meta-update pattern should not be copy-pasted across files. Small refactors (2-3 call sites) are included as fixes. Major rewrites (10+ call sites) are deferred.
## Context
Auditing all changes on `ambiguous-paths` branch (10 commits, 41 files, +4275/-1020 lines) against `origin/master`. The feature replaces the duplicate-skip blacklist with a matching algorithm (3 implementations), filesystem-collision-based dedup, filesystem-name-based ref paths, dedup suffix cleanup, and plugin updates.
The audit traced round-trip identity through all sync paths and found 3 critical issues, 3 correctness concerns, and 2 code quality items.
---
## Fixes
Each fix below was approved by the user during the audit quiz. Implement them in order. Each fix includes its test requirements -- a fix is not complete until its tests pass.
### Fix 1: Make Rust matching implementations recursive
- **Status:** Approved
- **Finding:** C1/CC3 -- Rust matching is flat (non-recursive) while Lua is recursive, causing pairing divergence
- **Files:**
- [src/snapshot/matching.rs](src/snapshot/matching.rs) -- forward sync matching
- [src/syncback/matching.rs](src/syncback/matching.rs) -- syncback matching
- **Problem:** Both Rust `count_own_diffs` functions only compare top-level properties and add +1 for children count difference. They do NOT recursively score subtrees. The `UNMATCHED_PENALTY` constant (10,000) is defined but never used. For ambiguous groups where instances have identical top-level properties but different children, Rust scores them identically (0 cost), falling back to child order. The Lua implementation correctly recurses into subtrees and applies `UNMATCHED_PENALTY` for unmatched children.
- **Solution:**
**For `src/snapshot/matching.rs` (forward sync):**
1. Rename `count_own_diffs` to `count_own_diffs_only` (internal helper).
2. Create new `compute_change_count(snap: &InstanceSnapshot, inst: &InstanceWithMeta, tree: &RojoTree, best_so_far: u32) -> u32`:
- Call `count_own_diffs_only` for the flat property cost.
- If cost >= best_so_far, early return.
- Recursively call `match_forward_internal` on children (a new variant that returns matched pairs + unmatched counts without consuming snapshots -- use `&[InstanceSnapshot]` and `&[Ref]` refs).
- For each matched pair, recursively call `compute_change_count`. If cost >= best_so_far, early return.
- Add `UNMATCHED_PENALTY * unmatched_count` to cost.
3. In the ambiguous group scoring loop, replace `count_own_diffs(snap, &inst)` with `compute_change_count(snap, &inst, tree, best_so_far)`.
4. The `match_forward` function needs a variant that doesn't consume snapshots (borrows instead). Create `match_forward_scoring(snap_children: &[InstanceSnapshot], tree_children: &[Ref], tree: &RojoTree) -> ScoringResult` that returns `(matched_count, unmatched_snap_count, unmatched_tree_count, Vec<(usize, usize)>)` -- index pairs for recursive scoring.
5. **Depth limit:** Add a `depth: u32` parameter to `compute_change_count`. At depth > 3 (or configurable), fall back to flat comparison to prevent O(n^k) explosion on deeply nested identical trees.
**For `src/syncback/matching.rs` (syncback):**
Same transformation: create `compute_change_count` that recursively matches children and scores subtrees. The hash fast-path remains: if new_hash == old_hash, return 0 immediately (skips recursion). When hashes differ, recurse into children.
**Key constraint:** The recursive scoring functions must NOT allocate heavily in the inner loops. Use stack-based recursion with early exit on `best_so_far`. The hash fast-path in syncback makes this efficient -- most subtrees are identical and short-circuit at depth 0.
- **Risk:** Medium. Recursive matching is more expensive than flat matching. The `best_so_far` pruning and depth limit mitigate this. Hash fast-path in syncback means most comparisons never recurse. Forward sync doesn't have hashes but the (Name, ClassName) grouping limits the cross-product size.
- **Verify round-trip:** Create 10 same-named Parts with different subtrees, syncback them, rebuild. The pairings should be stable across syncback + forward sync.
- **Verify syncback parity:** N/A (matching algorithm is internal, output is the same files)
- **Tests required:**
- Rust unit test in `src/snapshot/matching.rs`: 10 Parts named "Line" with different Transparency values, reversed order in tree. Verify each pair has matching Transparency (proves recursive scoring produces correct pairing).
- Rust unit test in `src/syncback/matching.rs`: Same test with WeakDom instances.
- Rust unit test: Identical instances (all properties same, same children) -- verify greedy falls back to child order (stable).
- Rust unit test: Depth limit test -- deeply nested tree (depth > 5) completes in reasonable time.
### Fix 2: Replace `escape_ref_path_segment` with filesystem-compatible names in `added_paths`
- **Status:** Approved
- **Finding:** C2 -- `added_paths` mixes filesystem-name parent path with escape-encoded child name
- **Files:**
- [src/web/api.rs](src/web/api.rs) lines 685-686
- **Problem:** `crate::escape_ref_path_segment(&added.name)` produces escaped instance names (e.g., `"Hey\/Bro"`) but `get_instance_by_path` splits on `/` without unescaping. An instance named `"Hey/Bro"` gets path segment `"Hey\/Bro"` which splits into `["Hey\", "Bro"]`, failing resolution. Should use the same filesystem name format as `ref_target_path_from_tree`.
- **Solution:** Replace lines 685-686 with:
```rust
// Use slugified name (matching filesystem conventions) instead of
// escape_ref_path_segment (which uses legacy \/ escaping).
let fs_name = if crate::syncback::name_needs_slugify(&added.name) {
crate::syncback::slugify_name(&added.name)
} else {
added.name.clone()
};
let path = if parent_path.is_empty() {
fs_name
} else {
format!("{}/{}", parent_path, fs_name)
};
```
Note: This doesn't include the file extension (e.g., `.server.luau`) because added instances don't have middleware info at this point. The `get_instance_by_path` fallback uses case-insensitive instance name matching, which will handle the resolution. For a more complete fix, compute the tentative extension based on `added.class_name` and `added.children.is_empty()`, similar to `tentative_fs_name` in `ref_properties.rs`.
- **Risk:** Low. Only affects ref properties referencing just-added instances with `/` or `\` in their names.
- **Verify round-trip:** Add an instance named `"Hey/Bro"` via two-way sync, then verify a ref property pointing to it resolves correctly after a server restart.
- **Verify syncback parity:** N/A (two-way sync only, CLI syncback doesn't use added_paths)
- **Tests required:**
- Rust integration test in `tests/tests/two_way_sync.rs`: Send a `/api/write` with an added instance named `"A/B"` and an update that references it via Ref. Verify the Ref resolves to the correct tree instance after patch application.
### Fix 3: Restructure syncback ref collection for dedup-aware paths
- **Status:** Approved
- **Finding:** C3 -- `tentative_fs_path` doesn't include dedup suffixes, producing ambiguous paths for duplicate-named instances
- **Files:**
- [src/syncback/ref_properties.rs](src/syncback/ref_properties.rs) -- `tentative_fs_path`, `collect_referents`, comment
- [src/syncback/mod.rs](src/syncback/mod.rs) -- `syncback_loop` orchestration
- [src/syncback/file_names.rs](src/syncback/file_names.rs) -- `name_for_inst` (produces dedup'd names)
- [src/snapshot_middleware/dir.rs](src/snapshot_middleware/dir.rs) -- `syncback_dir_no_meta` (calls `name_for_inst`)
- **Problem:** `collect_referents()` calls `tentative_fs_path()` which computes paths BEFORE dedup suffix assignment. Two instances with the same name/class get identical paths (e.g., both `"Workspace/Data"`). The `Rojo_Ref`_* attribute stores an ambiguous path that resolves to the first matching child, potentially the wrong one.
The comment at line ~147 says "ALL filesystem paths are unique" which is misleading -- tentative paths aren't unique for duplicate-named instances.
- **Solution:**
**Phase 1: Correct the comment (immediate)**
Replace the misleading comment in `collect_referents` with:
```rust
// NOTE: tentative_fs_path() does not include dedup suffixes (~N).
// For instances with duplicate names under the same parent, this
// produces ambiguous paths. Resolution via get_instance_by_path()
// returns the first matching child, which may be wrong.
//
// To fully fix this, collect_referents must run AFTER dedup suffix
// assignment, with access to the final filesystem names. See Phase 2.
```
**Phase 2: Pass dedup assignments to ref collection**
1. In `syncback_loop()` (`src/syncback/mod.rs`), after the recursive `syncback_dir_no_meta` / middleware dispatch completes (which assigns filenames via `name_for_inst`), build a `HashMap<Ref, String>` mapping each instance's Ref to its FINAL dedup'd filesystem path.
2. Pass this map to `collect_referents()` as a new parameter `final_paths: &HashMap<Ref, String>`. In `collect_referents`, when building the target path, look up the target Ref in `final_paths` first. If found, use that path. If not found (external ref, pruned instance), fall back to `tentative_fs_path`.
3. The `final_paths` map is built bottom-up during the syncback walk. Each middleware's `syncback()` call returns a `SyncbackReturn` which contains the `fs_snapshot` with file paths. Extract the Ref→path mapping from the `SyncbackSnapshot` path + the child assignments.
**Implementation detail:** The simplest approach is to add a `ref_path_map: &mut HashMap<Ref, String>` parameter to `SyncbackSnapshot::with_joined_path` and record the mapping there. Each time a child gets a filesystem name assigned, record `(child_ref, parent_path + "/" + filename)` in the map.
- **Risk:** Medium. Changes the syncback flow to thread a map through the recursive walk. Must be careful not to break incremental mode. The `final_paths` map is only populated during syncback, so forward sync is unaffected.
- **Verify round-trip:** Create two Folders named "Data" under Workspace, add a Script with PrimaryPart pointing to the second "Data". Syncback, rebuild, verify PrimaryPart points to the correct "Data" (the one with dedup suffix `~1`).
- **Verify syncback parity:** The CLI syncback output must include the correct `Rojo_Ref_`* attribute with the dedup'd path (e.g., `"Workspace/Data~1"`). Two-way sync should produce the same path.
- **Tests required:**
- Rust integration test in `tests/tests/syncback_roundtrip.rs`: Two same-named Folders with a ref property pointing to the second one. Verify the `Rojo_Ref_`* attribute contains the dedup'd path (e.g., `"Workspace/Data~1"`).
- Rust unit test: `collect_referents` with `final_paths` containing dedup'd paths. Verify the output uses dedup'd paths instead of tentative paths.
### Fix 4: Dedup cleanup should match by full filesystem name, not stem alone
- **Status:** Approved
- **Finding:** CC1 -- Change processor groups dedup siblings by stem only, ignoring extension
- **Files:**
- [src/change_processor.rs](src/change_processor.rs) lines 1098-1125
- **Problem:** The dedup cleanup code extracts the stem by splitting on `'.'` and groups siblings by base stem. `Foo.server.luau` (stem `Foo`) and `Foo.luau` (stem `Foo`) are grouped together even though they have different dedup keys (`foo.server.luau` vs `foo.luau`) and are NOT in the same dedup group. This inflates `remaining_stems.len()` and can change the cleanup action.
- **Solution:** When finding siblings in the dedup group, also compare extensions. Extract the extension from the removed instance's filesystem name, and only include siblings whose extension matches. Specifically:
```rust
// Extract the compound extension (everything after the stem)
let removed_extension = removed_fs_name
.find('.')
.map(|i| &removed_fs_name[i + 1..]);
for &sibling_ref in parent_inst.children() {
if sibling_ref == removed_id { continue; }
let sibling_fs = tree.filesystem_name_for(sibling_ref);
let sibling_stem = sibling_fs.split('.').next().unwrap_or(&sibling_fs);
let sibling_ext = sibling_fs.find('.').map(|i| &sibling_fs[i + 1..]);
// Only group siblings with the same extension
if sibling_ext != removed_extension { continue; }
let sibling_base = match parse_dedup_suffix(sibling_stem) {
Some((b, _)) => b,
None => sibling_stem,
};
if sibling_base.eq_ignore_ascii_case(&base_stem) {
remaining_stems.push(sibling_stem.to_string());
}
}
```
- **Risk:** Very low. Makes grouping more precise, no behavior change for the common case (all siblings in a dedup group have the same extension by definition).
- **Verify round-trip:** Create siblings `Foo.server.luau` (Script) and `Foo.luau` (ModuleScript) and `Foo~1.server.luau` (Script). Delete `Foo.server.luau` via two-way sync. Verify `Foo~1.server.luau` is renamed to `Foo.server.luau` and `Foo.luau` is untouched.
- **Verify syncback parity:** N/A (dedup cleanup is a two-way sync operation)
- **Tests required:**
- Rust unit test in `src/syncback/dedup_suffix.rs`: Verify `compute_cleanup_action` with mixed-extension input (though this tests the helper, the real fix is in change_processor).
- Rust integration test or manual verification: Mixed-extension siblings deletion scenario as described above.
### Fix 5: Extract `trueEquals` to a shared module
- **Status:** Approved
- **Finding:** CC2 -- `trueEquals` (and `fuzzyEq`) duplicated between `matching.lua` and `diff.lua`
- **Files:**
- New: [plugin/src/Reconciler/trueEquals.lua](plugin/src/Reconciler/trueEquals.lua)
- [plugin/src/Reconciler/matching.lua](plugin/src/Reconciler/matching.lua) -- remove local copy, require shared
- [plugin/src/Reconciler/diff.lua](plugin/src/Reconciler/diff.lua) -- remove local copy, require shared
- **Problem:** `trueEquals` is copy-pasted between two files. If either is updated without the other, matching and diff will disagree on value equality, causing phantom changes (matching says "equal" but diff says "different").
- **Solution:**
1. Create `plugin/src/Reconciler/trueEquals.lua` containing `fuzzyEq` and `trueEquals`. Export `trueEquals` as the module return value.
2. In `matching.lua`, remove the local `fuzzyEq` and `trueEquals` definitions (lines 27-111). Add `local trueEquals = require(script.Parent.trueEquals)`.
3. In `diff.lua`, remove the local `fuzzyEq` and `trueEquals` definitions (lines 19-115). Add `local trueEquals = require(script.Parent.trueEquals)`.
- **Risk:** Very low. Pure refactor, no behavior change.
- **Verify round-trip:** Existing tests cover both callers.
- **Verify syncback parity:** N/A (plugin-only change)
- **Tests required:**
- Existing diff.spec.lua and diff.stress.spec.lua tests verify equality behavior. No new tests needed.
---
## Skipped Fixes
### Skipped: Remove unused `UNMATCHED_PENALTY` from Rust matching files
- **Finding:** Q1 -- dead constant in both Rust matching files
- **Problem:** `UNMATCHED_PENALTY` defined but never used
- **Reason skipped:** Fix 1 (recursive matching) will start using this constant, making removal unnecessary.
---
## Accepted Limitations
None -- all limitations flagged were approved for fixing.
---
## Deferred Refactors
- **Matching cache (Phase 2e from original plan):** Add optional cache mapping `parent_ref -> (children_list_hash, match_assignments)` to avoid re-running the full algorithm when children haven't changed. Not needed for correctness, optimization only.
- **Duplicated name-to-index grouping pattern:** All three matching implementations repeat the same "build HashMap of name to indices" pattern. Could be factored into a shared helper, but the three implementations operate on different types (InstanceSnapshot, WeakDom, Lua tables) making a shared abstraction non-trivial. Flagged for consideration after the matching implementations stabilize.
---
## Test Plan
Tests are not optional. Every approved fix must have test coverage. A fix is not complete until its tests pass (or intentionally fail, for known limitations).
### Rust Unit Tests (`#[cfg(test)]` blocks)
- **10 same-named Parts with different properties, reversed order:** Verify recursive matching pairs correctly by property content, not position. (Fix 1)
- **Identical instances (all same):** Verify stable child-order tiebreaker. (Fix 1)
- **Depth limit test:** Deeply nested tree (depth > 5) completes without hanging. (Fix 1)
- `**collect_referents` with `final_paths`:** Verify dedup'd paths used instead of tentative. (Fix 3)
### Rust Integration Tests (`tests/tests/`)
- **Two-way sync with added instance named `"A/B"`:** Verify ref resolution works with slugified names. (Fix 2)
- **Syncback roundtrip with duplicate-named Folders + ref:** Verify `Rojo_Ref`_* contains dedup'd path. (Fix 3)
### Lua Spec Tests (`.spec.lua` files)
- Existing diff.spec.lua and diff.stress.spec.lua cover `trueEquals` behavior. (Fix 5)
### Snapshot Tests (`insta` crate)
- Existing snapshot tests should be re-verified after recursive matching change. Run `cargo insta review` after Fix 1. (Fix 1)
---
## Test Rules
The implementer MUST follow these rules when writing tests for this plan:
### For bugs found and fixed
Every bug fix MUST have a test that:
1. Would have **FAILED** before the fix
2. **PASSES** after the fix
3. Prevents the bug from regressing
### For missing coverage identified
For each "missing test coverage" gap:
1. Write the test
2. Verify it passes against the current implementation
3. If it fails, investigate -- it may have found another bug
### Scope: test every affected layer
For each finding, determine which test layers are affected and write tests in ALL of them:
- Does the bug manifest at the plugin level? -> Lua spec test
- Does it manifest at the server level? -> Rust unit test or integration test
- Does it affect the round-trip? -> Integration test with filesystem verification
- Could it regress from a different code path? -> Test that code path too
## Final Step: Run CI
After ALL fixes and tests are implemented, run the `/ci` command (see `.cursor/commands/ci.md`) to execute the full CI pipeline. Every fix must pass CI before the plan is considered complete. Do not skip this step.

212
.cursor/plans/audit_ambiguous_path_handling_30e99d9a.plan.md Normal file
View file

@ -0,0 +1,212 @@
---
name: Audit Ambiguous Path Handling
overview: Production audit of the Ambiguous Path Handling feature (matching algorithm, dedup system, ref path system). Identifies 7 critical/correctness issues, 3 code quality items, and extensive missing test coverage."
todos:
- id: c1-pass3-double-match
content: "Fix: Forward sync pass3 can re-match pass1-matched items (snap_available not consumed)"
status: completed
- id: c2-case-sensitivity
content: "Fix: get_instance_by_path inconsistent case sensitivity between primary and fallback"
status: completed
- id: c3-projectnode-path
content: "Fix: ref_target_path_from_tree uses project filename instead of instance name for ProjectNode"
status: completed
- id: c5-return-value
content: "Verify: encodeInstance return value change doesn't break callers"
status: completed
- id: c6-dead-code
content: "Fix: Remove dead code in ref_properties.rs (compute_refs_with_duplicate_siblings, is_path_unique_with_cache)"
status: completed
- id: c7-ref-path-format
content: "Critical: ref_properties.rs writes instance-name paths but tree.rs expects filesystem-name paths"
status: completed
- id: q2-hashset-perf
content: "Fix: Use HashSet instead of Vec for matched tracking in syncback matching"
status: completed
- id: c8-matching-lua-require-crash
content: "Critical: matching.lua has wrong require path for Log module - crashes on load, breaks hydration entirely"
status: completed
- id: q3-unused-import
content: "Fix: Remove unused Log import in Lua matching module (after fixing c8, Log is still unused)"
status: completed
- id: t1-syncback-integration
content: "Test: Add syncback integration tests for duplicate-named instances"
status: completed
- id: t3-new-fn-tests
content: "Test: Add tests for ref_target_path_from_tree and filesystem_name_for"
status: completed
isProject: false
---
# Audit: Ambiguous Path Handling Feature
> This plan was generated by `/audit` from a Plan-mode session.
> Refer to `.cursor/rules/atlas.mdc` for full project standards.
## Standards
### Round-Trip Identity (from `atlas.mdc`)
Syncback writes a directory tree. Building an rbxl from that tree and forward-syncing it back must produce a **bit-identical instance tree**. Any deviation is a bug.
### CLI Syncback Parity (from `atlas.mdc`)
Plugin-based sync must produce **exactly the same filesystem output** as `atlas syncback`. Any divergence is a bug.
### Code Quality (from `atlas.mdc`)
DRY code, no duplicated slugify/dedup/meta patterns across files.
## Context
Auditing all changes on `ambiguous-paths` branch (4 commits, 24 files, +3560/-825 lines) against `origin/master`. The feature replaces the duplicate-skip blacklist with a 3-pass matching algorithm, filesystem-collision-based dedup, and filesystem-name-based ref paths.
---
## Critical Issues
### C1. Forward sync matching does not filter already-matched items in Pass 3
**File:** [src/snapshot/matching.rs](src/snapshot/matching.rs), `pass3_similarity()` line 199-262
**Problem:** `pass3_similarity` builds its own `snap_by_name` / `tree_by_name` maps from ALL items in `snap_available` / `tree_children`, including items already matched in Pass 1 (which are marked only via `snap_matched[si]=true` / `tree_available[ti]=false` in the caller). However, `snap_by_name` iterates `snap_available` checking `if let Some(snap) = snap_opt` -- but snap_available entries are never set to `None` after Pass 1 matching. This means Pass 3 **re-groups items already matched in Pass 1** into its name maps. The `snap_matched` guard inside the pair-building loop catches this for snap items, but the `tree_by_name` check is `if !tree_available[i]` which IS correctly applied. The snap side relies on `snap_matched[si]` which is initialized to `vec![false; ...]` at the top of pass3 -- it does NOT inherit the pass1 match state.
**Impact:** Pass 3 will attempt to re-match snapshots already matched in Pass 1, potentially creating duplicate matched pairs in the output. The `build_result()` function does `.take()` on snap entries, so the second `.take()` would return `None` and the pair would be silently dropped. This means the unmatched counts would be wrong (items appear in neither matched nor unmatched).
**Fix:** Either set `snap_available[si] = None` for pass1-matched items before calling pass3, or initialize pass3's `snap_matched` from the caller's `snap_matched` state. The simplest fix: filter `snap_by_name` to only include indices where `snap_available[i].is_some()` AND the index was not already matched. Since we use a separate `snap_matched` bool vec, we can check: the `snap_available` entries for pass1-matched items were never consumed (they're still `Some`). We need to either mark them `None` or pass the matched state.
### C2. `get_instance_by_path` uses case-insensitive comparison for filesystem names but case-SENSITIVE for fallback
**File:** [src/snapshot/tree.rs](src/snapshot/tree.rs), `get_instance_by_path()` line ~248
**Problem:** The primary lookup uses `fs_name.eq_ignore_ascii_case(segment)` but the fallback name lookup uses `child.name == *segment` (case-sensitive). This inconsistency means:
- A ref path `"workspace/Part"` would match filesystem name `"Workspace"` (case-insensitive) in the primary check
- But if the instance had no `instigating_source`, the fallback would fail because `"Workspace" != "workspace"`
More critically, this creates an asymmetry: `ref_target_path_from_tree()` builds paths from filesystem names preserving their original case, but resolution uses case-insensitive matching. This means two distinct instances `"Foo.luau"` and `"foo.luau"` (which should be dedup'd for cross-platform safety) would both match the same path segment.
**Fix:** Use case-insensitive comparison in BOTH the primary and fallback paths. Or use exact case matching in both (since filesystem names are deterministic and paths should round-trip exactly).
### C3. `ref_target_path_from_tree` falls back to instance name for ProjectNode instances, creating inconsistent paths
**File:** [src/rojo_ref.rs](src/rojo_ref.rs), `ref_target_path_from_tree()` line ~157-173
**Problem:** When an instance has `InstigatingSource::ProjectNode`, the code falls through to the `else` branch (no instigating_source) and uses `inst.name.clone()`. But `InstigatingSource::ProjectNode` DOES have a path -- `source.path()` returns the project file path, whose `file_name()` would be something like `default.project.json5`. This is wrong -- for project nodes, the segment should be the instance name (not the project filename).
The current code handles this correctly by accident: the `if let Some(source) = &meta.instigating_source` check matches `ProjectNode` (it has a source), so it uses `source.path().file_name()` which returns the project filename like `"default.project.json5"`. This produces WRONG ref path segments for project-sourced instances -- a service like "Workspace" would get segment `"default.project.json5"` instead of `"Workspace"`.
**Fix:** Check for `InstigatingSource::ProjectNode` specifically and use the instance name, not the project file path. The `instigating_source.path()` for ProjectNode returns the project file itself, not the instance's filesystem representation.
### C4. Lua matching `_pass1NameAndClass` has index invalidation bug when removing from arrays
**File:** [plugin/src/Reconciler/matching.lua](plugin/src/Reconciler/matching.lua), lines 168-190
**Problem:** After collecting matched indices, the code sorts them in reverse order and calls `table.remove(remainingVirtual, idx)` / `table.remove(remainingStudio, idx)` for each. However, if Pass 1 matched both a unique-name pair AND a class-narrowed pair from the same name group, the same index could theoretically appear twice in `matchedVirtualIndices`. The deduplication check (`seenV[idx]`) handles this.
But the more fundamental issue: indices collected during the matching loop refer to positions in `remainingVirtual` at the START of the function. As `table.remove` shifts elements, subsequent indices become invalid. The reverse-order sort mitigates this for sequential removals, but if indices from DIFFERENT name groups interleave (e.g., index 3 from name "A" and index 2 from name "B"), removing index 3 first then index 2 works. However, if the same index appears for both a unique match and a class match (shouldn't happen since unique match `continue`s), the dedup prevents double-removal.
**Real bug:** The virtual and studio index lists are independent. `matchedVirtualIndices` contains indices into `remainingVirtual`, `matchedStudioIndices` contains indices into `remainingStudio`. Removing from one array doesn't affect the other. But removing from `remainingVirtual` in reverse order IS correct for that array alone. The issue is that indices 1, 3, 5 removed in order 5, 3, 1 correctly removes the right elements.
Actually, this is correct IF no duplicates. But there's a subtle issue: the class narrowing loop can match items whose indices overlap with the unique-name match. Since the unique-name match does `continue` (skips class narrowing for that name), this shouldn't happen. **Low risk but fragile.**
### C5. `encodeInstance.lua` return value change breaks callers expecting 2 return values
**File:** [plugin/src/ChangeBatcher/encodeInstance.lua](plugin/src/ChangeBatcher/encodeInstance.lua), bottom
**Problem:** The old function returned `(result, skippedDuplicateCount)` -- two values. The new code returns just `encodeInstance(instance, parentId, false)` -- one value. Per the Lua silent argument mismatch rule, callers that destructured the second return value now silently get `nil` instead of a count. Need to verify all callers.
**Verification needed:** Search for all callers of `encodeInstance` / `encodeInstanceWithDuplicateTracking` to confirm none destructure the second return value.
### C6. Dead code left behind in `ref_properties.rs`
**File:** [src/syncback/ref_properties.rs](src/syncback/ref_properties.rs), lines 54-131
**Problem:** `compute_refs_with_duplicate_siblings()` and `is_path_unique_with_cache()` are now dead code (clippy confirms: "function is never used"). The `collect_referents()` function was simplified to always use path-based linking, but the old helper functions were not removed. This is dead code that will confuse future maintainers.
### C7. `ref_properties.rs` always uses instance-name paths, not filesystem-name paths
**File:** [src/syncback/ref_properties.rs](src/syncback/ref_properties.rs), `collect_referents()` line ~170
**Problem:** The simplified `collect_referents()` calls `inst_path(dom, *target_ref)` which uses the legacy `ref_target_path()` (instance-name-based with escaping). This means Rojo_Ref_* attributes written during syncback still use the OLD path format (instance names), not the new filesystem-name format. But `get_instance_by_path()` in tree.rs now expects filesystem-name segments. This creates a **format mismatch**: syncback writes instance-name paths, forward sync tries to resolve them as filesystem-name paths.
The fallback in `get_instance_by_path()` (case-sensitive instance name match) partially mitigates this, but it's fragile and won't work for instances whose name differs from their filesystem name (slugified names, dedup'd names).
**Impact:** Critical round-trip violation for any instance with a slugified or dedup'd name that is referenced by a Ref property. The ref path written during syncback won't resolve during forward sync.
**Fix:** `collect_referents()` needs to build filesystem-name paths. This requires access to the filesystem names assigned during syncback (via instigating_source or a Ref-to-filesystem-name map built during the syncback pass).
---
### C8. Lua matching module has wrong `require` path -- crashes on load, breaks hydration
**File:** [plugin/src/Reconciler/matching.lua](plugin/src/Reconciler/matching.lua), line 16
**Problem:** `local Log = require(script.Parent.Parent.Log)` resolves to `plugin/src/Log` which does not exist. Every other file in `plugin/src/Reconciler/` uses `local Packages = script.Parent.Parent.Parent.Packages` followed by `require(Packages.Log)`. Since Luau's `require()` executes immediately at module load time, this incorrect path causes a runtime error when `matching.lua` loads, which prevents the 3-pass matching algorithm from functioning and breaks hydration entirely (`hydrate.lua` line 15 requires `matching`).
**Impact:** The plugin cannot connect to the server at all. Hydration is the first thing that runs after the initial `/api/read` -- if matching.lua fails to load, the entire sync session crashes.
**Fix:** Replace line 16 with:
```lua
local Packages = script.Parent.Parent.Parent.Packages
local Log = require(Packages.Log)
```
---
## Code Quality Issues
### Q1. Duplicated name→index grouping pattern across 3 matching modules
All three matching implementations (syncback, forward sync, Lua plugin) repeat the same pattern: build `HashMap<name, Vec<index>>` for both sides, iterate names, check counts. This is ~40 lines repeated 6+ times across the codebase. Should be factored into a shared helper (or at minimum documented as intentional duplication per the spec requirement of separate implementations).
### Q2. `matched_new.contains(r)` uses O(n) linear scan in syncback matching
**File:** [src/syncback/matching.rs](src/syncback/matching.rs), lines 196, 262, 358, 400
The `matched_new.contains()` / `matched_old.contains()` checks are O(n) linear scans. For large ambiguous groups (10+ duplicates), this creates O(n^2) behavior within Pass 2 and Pass 3. Should use `HashSet` instead of `Vec` for the matched tracking sets.
### Q3. Unused `Log` import in Lua matching module
**File:** [plugin/src/Reconciler/matching.lua](plugin/src/Reconciler/matching.lua), line 16
`local Log = require(script.Parent.Parent.Log)` is imported but never used. Will trigger selene lint warning.
---
## Missing Test Coverage
### T1. No integration tests for duplicate-named instances through syncback
The feature's core value proposition -- syncing duplicate-named instances -- has zero integration tests. The only tests are unit tests for the matching algorithm and dedup suffix cleanup. Need syncback test fixtures with:
- Two Folders with same name
- Folder + Script same name (no collision)
- Init-style collision (Folder + Script with children)
- Roundtrip test with duplicates
### T2. No tests for forward sync with duplicate-named instances
`compute_children_patches` now uses the matching algorithm, but there are no tests verifying that editing one of two same-named files correctly pairs to the right tree instance.
### T3. No tests for `ref_target_path_from_tree` or `filesystem_name_for`
New functions with zero test coverage. Critical for ref path correctness.
### T4. No tests for dedup key semantics change
The `deduplicate_name_with_ext` function is tested via updated existing tests, but there are no tests specifically for the Folder+Script no-collision scenario or init-style collision scenario.
### T5. No Lua spec tests for the matching module
The Lua matching module has zero tests. The `hydrate.lua` changes are untested.
---
## Deferred Refactors
- **Syncback ref path format migration**: `collect_referents()` needs to build filesystem-name paths instead of instance-name paths. This requires restructuring the syncback flow to make filesystem name assignments available during ref collection. (Blocked on: syncback flow restructure)
- **Matching cache (Phase 2e)**: Deferred optimization, not needed for correctness.
- **change_processor dedup cleanup integration**: The `dedup_suffix.rs` module provides cleanup rules but they aren't integrated into the change processor's deletion handling yet.

280
.cursor/plans/audit_ambiguous_paths_d9495fcc.plan.md Normal file
View file

@ -0,0 +1,280 @@
---
name: Audit Ambiguous Paths
overview: "Audit of the \"Ambiguous Path Handling\" feature: three parallel matching algorithm implementations (2 Rust, 1 Lua), Reconciler integration, and encoding changes."
todos:
- id: fix-stale-comment
content: "RESOLVED: encodeInstance.lua header comment already updated in prior session"
status: pending
- id: align-tags-attrs-scoring
content: "RESOLVED by matching_algorithm_parity plan (Fix 2): Rust countOwnDiffs now has granular Tags/Attributes scoring"
status: pending
- id: ref-identity-rust
content: "RESOLVED by matching_algorithm_parity plan (Fix 1): Rust countOwnDiffs now has Ref-target-aware scoring"
status: pending
- id: double-cost-perf
content: "Fix: eliminate double cost computation in Lua matchChildren totalCost loop by tracking scored costs from ambiguous loop and reusing them"
status: completed
- id: trueequals-types
content: "Fix: add UDim, UDim2, NumberRange, NumberSequence, ColorSequence, Rect fuzzy equality to trueEquals.lua"
status: completed
isProject: false
---
# Audit: Ambiguous Path Handling (Reconciler Changes)
## Scope
Files audited:
- [plugin/src/Reconciler/matching.lua](plugin/src/Reconciler/matching.lua) -- Lua matching algorithm (631 lines)
- [plugin/src/Reconciler/matching.spec.lua](plugin/src/Reconciler/matching.spec.lua) -- Lua matching tests
- [plugin/src/Reconciler/hydrate.lua](plugin/src/Reconciler/hydrate.lua) -- Integration with matching
- [plugin/src/Reconciler/reify.lua](plugin/src/Reconciler/reify.lua) -- Nil-instance guards for ambiguous exclusions
- [plugin/src/Reconciler/trueEquals.lua](plugin/src/Reconciler/trueEquals.lua) -- Shared value equality
- [plugin/src/Reconciler/diff.lua](plugin/src/Reconciler/diff.lua) -- Uses shared trueEquals
- [plugin/src/ChangeBatcher/encodeInstance.lua](plugin/src/ChangeBatcher/encodeInstance.lua) -- Duplicate detection helpers
- [src/snapshot/matching.rs](src/snapshot/matching.rs) -- Rust forward sync matching
- [src/syncback/matching.rs](src/syncback/matching.rs) -- Rust syncback matching
- [src/variant_eq.rs](src/variant_eq.rs) -- Rust property comparison
---
## Architecture Summary
```mermaid
flowchart TD
subgraph luaPlugin ["Lua Plugin (Studio)"]
Hydrate["hydrate.lua"]
LuaMatch["matching.lua"]
TrueEquals["trueEquals.lua"]
Hydrate -->|"calls"| LuaMatch
LuaMatch -->|"uses"| TrueEquals
end
subgraph rustForward ["Rust Forward Sync"]
PatchCompute["patch_compute.rs"]
FwdMatch["snapshot/matching.rs"]
VariantEq["variant_eq.rs"]
PatchCompute -->|"calls"| FwdMatch
FwdMatch -->|"uses"| VariantEq
end
subgraph rustSyncback ["Rust Syncback"]
SbEntry["syncback entry"]
SbMatch["syncback/matching.rs"]
SbEntry -->|"calls"| SbMatch
SbMatch -->|"uses"| VariantEq
end
```
All three implementations use the same algorithm:
1. **Fast path**: Group by `(Name, ClassName)` -- 1:1 groups instant-match
2. **Ambiguous groups**: Pairwise recursive `computeChangeCount` scoring
3. **Greedy assignment**: Sort by cost ascending, pick lowest-cost unmatched pair
4. **Depth limit**: `MAX_SCORING_DEPTH = 3` prevents O(n^k) explosion
5. **Session caching**: `MatchingSession` avoids redundant recursive scoring
---
## Findings
### FINDING 1: Stale header comment in encodeInstance.lua (Low -- misleading docs)
The file header at lines 12-16 states:
```1:17:plugin/src/ChangeBatcher/encodeInstance.lua
--[[
// ...
DUPLICATE HANDLING:
- Instances with duplicate-named siblings at ANY level of the path are skipped
- This matches the behavior in ref_properties.rs::is_path_unique
- A path is ambiguous if duplicates exist anywhere from the instance to the root
- A warning is logged when duplicates are encountered
]]
```
But the actual code at lines 159-168 does **not** skip duplicates -- it logs a debug message and encodes normally:
```159:168:plugin/src/ChangeBatcher/encodeInstance.lua
encodeInstance = function(instance, parentId, _skipPathCheck)
-- Log duplicate-named siblings as a debug message (server handles them via
-- dedup suffixes) but do NOT skip encoding.
if not _skipPathCheck and not isPathUnique(instance) then
Log.debug(
"Instance '{}' ({}) has duplicate-named siblings in path (server will handle via dedup)",
instance:GetFullName(),
instance.ClassName
)
end
```
The inline comment (line 161) is correct; the file-level docblock is stale from a prior implementation. **Fix: update the header comment.**
---
### FINDING 2: Tags/Attributes granularity divergence across implementations (Medium -- scoring asymmetry) -- RESOLVED
> **Resolved by [matching_algorithm_parity](matching_algorithm_parity_01524d4c.plan.md) Fix 2** (status: completed). Rust `countOwnDiffs` in both forward sync and syncback now extracts Tags/Attributes and scores them granularly (per-tag, per-attribute), matching the Lua behavior. Parity tests added (Fix 6).
~~The three `countOwnDiffs` implementations score Tags and Attributes differently.~~
Original analysis preserved below for reference:
The Lua implementation (`matching.lua` lines 282-307) counts each individual tag/attribute diff as +1. The Rust implementations previously counted the entire `Tags`/`Attributes` Variant as a single +1 diff. This meant 3 of 10 tags differing scored +1 in Rust but +3 in Lua, potentially causing different pairings.
---
### FINDING 3: Lua Ref scoring uses identity heuristic not present in Rust (Medium -- divergence) -- RESOLVED
> **Resolved by [matching_algorithm_parity](matching_algorithm_parity_01524d4c.plan.md) Fix 1** (status: completed). Both Rust `countOwnDiffs` functions now resolve Ref targets to `(Name, ClassName)` and score mismatches at `UNMATCHED_PENALTY`, matching the Lua behavior. Rust unit tests `ref_scoring_forward_sync` and `ref_scoring_syncback` added.
~~The Rust implementations had no equivalent of Lua's Ref identity heuristic.~~
Original analysis preserved below for reference:
The Lua `countOwnDiffs` scored Ref property mismatches using `RefIdentity` (name + className) at a cost of `UNMATCHED_PENALTY` (10,000). The Rust implementations compared `Variant::Ref` values as opaque IDs (+1 per mismatch), which were always different between contexts, making Refs useless for disambiguation.
---
### FINDING 4: Performance -- double cost computation in Lua matchChildren (Low -- perf only)
In `matchChildren`, ambiguous group costs are computed inline (lines 534-557) for assignment decisions but never stored in `session.costCache`. Then lines 604-607 recompute total cost by calling `computePairCost` for every matched pair:
```604:608:plugin/src/Reconciler/matching.lua
local totalCost = 0
for _, pair in ipairs(matched) do
totalCost += computePairCost(session, pair.virtualId, pair.studioInstance, virtualInstances, math.huge, depth)
end
totalCost += (#unmatchedVirtual + #unmatchedStudio) * UNMATCHED_PENALTY
```
This means ambiguous group pairs have their full subtree cost computed **twice**: once during the scoring loop, once during totalCost. For 1:1 fast-path pairs, cost is computed **once** (only in totalCost), which is correct.
**Impact**: Wasted computation proportional to ambiguous group size squared. Since ambiguous groups are typically small (2-10 instances) and the depth limit caps recursion, this is acceptable. The `computePairCost` call does populate `session.costCache`, so subsequent recursive calls at deeper levels benefit.
**Fix (deferred)**: Store the computed cost from the ambiguous scoring loop and reuse it in the totalCost sum instead of calling `computePairCost` again.
---
### FINDING 5: `trueEquals` lacks fuzzy equality for compound float types (Low -- edge case)
`trueEquals.lua` provides fuzzy float comparison for: `number`, `Color3`, `Vector3`, `Vector2`, `CFrame`. It falls through to `return false` for all other types (line 111).
The Rust `variant_eq.rs` provides fuzzy comparison for additional float-containing types: `UDim`, `UDim2`, `NumberRange`, `NumberSequence`, `ColorSequence`, `Rect`, `PhysicalProperties`, `Ray`, `Region3`, `Float32`, `Float64`, `Font`.
In Luau, `UDim2 == UDim2` uses `__eq` which may or may not apply float tolerance. If Roblox's `__eq` does exact comparison, two UDim2 values that differ by float precision would fail `==` and then fail `trueEquals` (which returns false for unrecognized types). This would inflate match costs slightly.
**Impact**: Unlikely to cause wrong pairings since the inflation would be symmetric across all candidates. But it could trigger unnecessary reconciler updates (diff sees a change that isn't really a change).
**Fix (deferred)**: Add UDim2, NumberRange, NumberSequence, Rect handlers to `trueEquals`. These are uncommon in ambiguous matching scenarios but would improve diff accuracy.
---
### FINDING 6: Reconciler integration is clean (Positive)
**hydrate.lua** (lines 33-45) correctly filters out missing virtual children before calling `matchChildren`, and passes a shared `MatchingSession` through recursive hydration:
```32:48:plugin/src/Reconciler/hydrate.lua
-- Filter out virtual children missing from virtualInstances
local validVirtualIds: { string } = {}
for _, childId in ipairs(virtualChildIds) do
if virtualInstances[childId] then
table.insert(validVirtualIds, childId)
else
Log.warn(
"Hydration: virtualInstances missing child ID {} (parent: {} '{}')",
childId,
rootId,
virtualInstance.Name
)
end
end
local result =
Matching.matchChildren(session, validVirtualIds, existingChildren, virtualInstances, rootId, rootInstance)
```
**reify.lua** (lines 52-57) defensively handles nil virtual instances (excluded from patch):
```52:57:plugin/src/Reconciler/reify.lua
if virtualInstance == nil then
-- This child was intentionally excluded from the patch (e.g., due to
-- ambiguous paths with duplicate-named siblings). Skip it gracefully.
return
end
```
**diff.lua** (line 133) uses the shared `trueEquals` module, ensuring matching and diff agree on equality:
```133:133:plugin/src/Reconciler/diff.lua
if not trueEquals(existingValue, decodedValue) then
```
Session sharing through recursive hydration is correct. The `--!native` and `--!optimize 2` directives on `matching.lua` and `trueEquals.lua` ensure native codegen for hot paths.
---
### FINDING 7: Test coverage is solid but lacks cross-implementation parity tests (Low) -- RESOLVED
> **Resolved by [matching_algorithm_parity](matching_algorithm_parity_01524d4c.plan.md) Fix 6** (status: completed). Parity tests added: `parity_forward_vs_syncback` (Rust), `round_trip_matching_identity` (Rust integration), `parity_with_rust_fixture` (Lua spec).
**Lua tests** (`matching.spec.lua`): 15+ test cases covering session caching, totalCost computation, Ref scoring, 2x2 and 3x3 ambiguous groups, recursive matching, edge cases, and now cross-implementation parity fixtures.
**Rust forward sync tests** (`src/snapshot/matching.rs`): 12+ test cases including property-based matching, depth limit, session cache, default-omission tests, and Ref scoring.
**Rust syncback tests** (`src/syncback/matching.rs`): 6+ test cases covering basics, class grouping, unmatched, duplicates, stability, session cache, and Ref scoring.
**Integration test**: `ref_ambiguous_path_no_crash` and `ref_ambiguous_target_no_crash` in `tests/tests/two_way_sync.rs` verify that Ref properties with ambiguous paths don't crash.
---
## Summary Table
| # | Finding | Severity | Status | Fix Effort |
| --- | -------------------------------------------- | -------- | -------- | ------------------ |
| 1 | Stale header comment in encodeInstance.lua | Low | RESOLVED | -- (prior session) |
| 2 | Tags/Attributes granularity divergence | Medium | RESOLVED | -- (parity plan) |
| 3 | Lua Ref identity heuristic missing in Rust | Medium | RESOLVED | -- (parity plan) |
| 4 | Double cost computation in Lua matchChildren | Low | **Fix** | Small |
| 5 | trueEquals lacks compound float types | Low | **Fix** | Small |
| 6 | Reconciler integration is clean | Positive | N/A | N/A |
| 7 | No cross-implementation parity tests | Low | RESOLVED | -- (parity plan) |
Additionally, the parity plan also resolved a float epsilon divergence (Fix 3: Rust `variant_eq.rs` now uses a 0.0001 epsilon matching Lua's `trueEquals`), and a VCache session-level caching optimization (Fix 5), partially mitigating Finding 4.
**Overall assessment**: The ambiguous path handling feature is well-implemented and correct. The matching algorithm is sound, the three implementations are now fully aligned (via the parity plan fixes), and the integration points (hydrate, reify, diff) are clean. No round-trip identity violations found.
---
## Actionable Fixes
### Fix A: Eliminate double cost computation in Lua matchChildren (Finding 4)
- **File:** `plugin/src/Reconciler/matching.lua`
- **Problem:** In `matchChildren`, the ambiguous scoring loop (lines 528-575) computes full subtree costs for each pair, but these costs are only stored in `scoredPairs` for sorting. Then the totalCost loop (lines 615-618) calls `computePairCost` for **every** matched pair (including ambiguous ones), recomputing the same costs from scratch. The `computePairCost` function does check `session.costCache`, but the ambiguous scoring loop never writes to it, so nothing is cached.
- **Solution:** Track the winning cost for each matched pair during greedy assignment and sum them directly, instead of calling `computePairCost` again. For 1:1 fast-path matches (which have no pre-computed cost), call `computePairCost` only for those.
Concretely in `matchChildren`:
1. Add a `matchedCosts: { [number]: number }` table (keyed by match index in `matched` array)
2. In the ambiguous greedy assignment loop (lines 585-595), store the pair's cost from `scoredPairs` alongside each match
3. In the totalCost loop, use the stored cost for ambiguous pairs and only call `computePairCost` for 1:1 fast-path pairs (which don't have a pre-computed cost)
### Fix B: Add compound float type fuzzy equality to trueEquals (Finding 5)
- **File:** `plugin/src/Reconciler/trueEquals.lua`
- **Problem:** `trueEquals` handles `number`, `Color3`, `Vector3`, `Vector2`, `CFrame` with fuzzy float comparison, but falls through to `return false` for `UDim`, `UDim2`, `NumberRange`, `NumberSequence`, `ColorSequence`, `Rect`. Since these types contain floats, Luau's `==` operator may fail on precision differences, causing phantom diffs in both matching and diff.
- **Solution:** Add handlers before the final `return false` (line 111):
- `UDim2`: decompose via `.X.Scale`, `.X.Offset`, `.Y.Scale`, `.Y.Offset` -- fuzzyEq on Scale, exact on Offset (integer)
- `UDim`: decompose via `.Scale`, `.Offset` -- fuzzyEq on Scale, exact on Offset
- `NumberRange`: fuzzyEq on `.Min`, `.Max`
- `Rect`: fuzzyEq on `.Min.X`, `.Min.Y`, `.Max.X`, `.Max.Y`
- `NumberSequence`: compare `.Keypoints` length, then per-keypoint fuzzyEq on `.Time`, `.Value`, `.Envelope`
- `ColorSequence`: compare `.Keypoints` length, then per-keypoint fuzzyEq on `.Time` + color3Eq on `.Value`
These match the Rust `variant_eq.rs` coverage exactly.

231
.cursor/plans/audit_ambiguous_paths_followup_00831072.plan.md Normal file
View file

@ -0,0 +1,231 @@
---
name: audit ambiguous paths followup
overview: Production audit of the ambiguous-paths branch against origin/master. Documents critical round-trip issues, approved fixes, accepted limitations, and required regression tests for implementation in a follow-up Agent-mode session.
todos:
- id: fix-metadata-drift
content: "Fix 1: Update dedup cleanup rename flow to keep tree metadata/path indexes in sync"
status: completed
- id: fix-batch-removal-dedup
content: "Fix 2: Exclude co-removed siblings from dedup cleanup sibling enumeration"
status: completed
- id: fix-added-paths
content: "Fix 3: Extend added_paths in api.rs to include middleware extension and dedup suffix"
status: completed
- id: fix-ref-path-final
content: "Fix 4: Complete Phase 2 of previous audit Fix 3 -- pass final_paths to collect_referents instead of global text rewrite"
status: completed
- id: fix-regression-tests
content: "Fix 5: Add regression tests for Fixes 1-4"
status: completed
isProject: false
---
# Ambiguous Paths Audit Fix Plan
> This plan was generated by `/audit` from a Plan-mode session.
> Refer to `.cursor/rules/atlas.mdc` for full project standards.
## Standards
These standards govern every fix in this plan. An implementer MUST read `.cursor/rules/atlas.mdc` before starting.
### Round-Trip Identity (from `atlas.mdc` §Quality Standard)
Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a bit-identical instance tree -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.
### CLI Syncback Parity (from `atlas.mdc` §Two-Way Sync Strategy)
Plugin-based sync must produce exactly the same filesystem output that `atlas syncback` would give for the same input. Byte-for-byte identical files, identical directory structures, identical naming. Any divergence is a bug until proven otherwise.
### Code Quality (from `atlas.mdc` §Code Quality Standard)
DRY code that is easy to reason about and hard to get wrong. Small helper refactors are in scope; major pipeline redesign is deferred.
## Context
Audited branch `ambiguous-paths` against `origin/master` using commit range `origin/master..HEAD`.
Primary findings:
- Critical path drift in dedup cleanup rename flow (`[src/change_processor.rs](d:/UserGenerated/rojo/src/change_processor.rs)`).
- Critical batch-removal dedup cleanup counts co-removed siblings as "remaining", defeating group-to-1 cleanup (`[src/change_processor.rs](d:/UserGenerated/rojo/src/change_processor.rs)`).
- Critical same-batch Ref target path synthesis mismatch for added instances (`[src/web/api.rs](d:/UserGenerated/rojo/src/web/api.rs)`).
- Critical ref-path post-processing strategy relies on global text replacement (`[src/syncback/mod.rs](d:/UserGenerated/rojo/src/syncback/mod.rs)`, `[src/syncback/fs_snapshot.rs](d:/UserGenerated/rojo/src/syncback/fs_snapshot.rs)`).
- Known limitation accepted for this release: duplicate-named children under ProjectNode currently skipped in `[src/snapshot_middleware/project.rs](d:/UserGenerated/rojo/src/snapshot_middleware/project.rs)`.
- Known limitation accepted for this release: greedy matcher is not globally optimal (`[src/snapshot/matching.rs](d:/UserGenerated/rojo/src/snapshot/matching.rs)`, `[src/syncback/matching.rs](d:/UserGenerated/rojo/src/syncback/matching.rs)`, `[plugin/src/Reconciler/matching.lua](d:/UserGenerated/rojo/plugin/src/Reconciler/matching.lua)`).
## Fixes
Each fix below was approved by the user during the audit quiz. Implement in order.
### Fix 1: Keep tree metadata consistent after dedup cleanup rename
- **Status:** Approved
- **Finding:** Step 2d / Step 8 (rename/delete interactions + VFS suppression)
- **Files:** `[src/change_processor.rs](d:/UserGenerated/rojo/src/change_processor.rs)`
- **Problem:** In dedup cleanup (`~1073-1203`), the code renames files (`fs::rename`) and suppresses both old/new paths, but does not update the renamed sibling's `InstigatingSource::Path` or metadata path indexes. Subsequent writes can target stale paths.
- **Solution:** After successful `DedupCleanupAction::RemoveSuffix` / `PromoteLowest`, locate the sibling instance being renamed, update its metadata source path and relevant path indexes in-memory, and keep suppression counters consistent with the updated path.
- **Risk:** Medium; touches tree mutation bookkeeping and rename lifecycle.
- **Verify round-trip:** Create duplicate siblings (`Foo`, `Foo~1`), delete `Foo`, ensure `Foo~1 -> Foo` rename occurs, then issue a property update to the survivor in same session and verify disk write targets new path and survives rebuild.
- **Verify syncback parity:** Compare output of two-way deletion flow to CLI syncback on equivalent tree state; both must produce the same remaining filename/meta layout.
- **Tests required:**
- Rust integration test in `[tests/tests/two_way_sync.rs](d:/UserGenerated/rojo/tests/tests/two_way_sync.rs)` covering delete-triggered dedup cleanup followed by update.
- Optional focused unit test around cleanup metadata path update helper in `[src/change_processor.rs](d:/UserGenerated/rojo/src/change_processor.rs)`.
### Fix 2: Exclude co-removed siblings from dedup cleanup sibling enumeration
- **Status:** Approved
- **Files:** `[src/change_processor.rs](d:/UserGenerated/rojo/src/change_processor.rs)`
- **Problem:** In the dedup cleanup loop (lines 1077-1203), the sibling enumeration (line 1120) only skips `sibling_ref == removed_id` -- the single instance being processed in the current iteration. It does NOT skip other instances that are also in `patch_set.removed_instances`. When multiple siblings from the same dedup group are removed in a single patch, the co-removed instances are incorrectly counted as "remaining", inflating the sibling count and defeating cleanup rules.
- **Failure path:** Remove `Foo` (A) and `Foo~1` (B) from a group of 3, leaving `Foo~2` (C) as the sole survivor. api.rs deletes all files first. In the dedup cleanup loop, iteration for A finds B and C as remaining (count=2), so group-to-1 does not fire; instead `PromoteLowest` tries to rename the already-deleted `Foo~1` → `Foo` (fails). Iteration for B finds A and C as remaining (count=2), group-to-1 again does not fire. Result: C keeps its `~2` suffix permanently when it should have been renamed to `Foo`.
- **Solution:** Before the dedup cleanup loop, build a `HashSet<Ref>` from `patch_set.removed_instances`. In the sibling enumeration (line 1121), change the skip condition from `sibling_ref == removed_id` to `removed_set.contains(&sibling_ref)`:
```rust
let removed_set: HashSet<Ref> = patch_set.removed_instances.iter().copied().collect();
// ... inside sibling loop:
for &sibling_ref in parent_inst.children() {
if removed_set.contains(&sibling_ref) {
continue;
}
// ... rest of sibling processing
}
```
- **Risk:** Low; narrow, mechanical change. The HashSet construction is O(n) and the lookup is O(1).
- **Verify round-trip:** Remove 2 of 3 duplicate siblings via two-way sync. Verify the sole survivor has its dedup suffix removed and survives rebuild with the clean name.
- **Verify syncback parity:** N/A (dedup cleanup is a two-way sync operation only).
- **Tests required:**
- Rust integration test in `[tests/tests/two_way_sync.rs](d:/UserGenerated/rojo/tests/tests/two_way_sync.rs)`: batch-remove 2 of 3 same-named siblings, verify survivor is renamed from `Foo~2` to `Foo`.
- Edge test: batch-remove ALL members of a dedup group (no survivor); verify no spurious rename attempts.
### Fix 3: Build correct same-batch added Ref paths
- **Status:** Approved
- **Lineage:** Extends `ambiguous_paths_audit_fixes_9d3ed6da` Fix 2 (completed), which replaced `escape_ref_path_segment` with `slugify_name` in `added_paths` but explicitly acknowledged the missing extension as a limitation. The current code at lines 637-641 of `api.rs` uses `slugify_name`; this fix adds the remaining segments.
- **Finding:** Step 2a / Step 10 (plugin->server wire format and write processing)
- **Files:** `[src/web/api.rs](d:/UserGenerated/rojo/src/web/api.rs)`
- **Problem:** `added_paths` map (`~622-649`) uses only slugified instance name (from the previous fix); it still omits middleware extension and dedup suffix. `syncback_updated_properties` then writes incorrect `Rojo_Ref_`* when target is newly added in same request (`~2694-2706`).
- **Solution:** Extend the existing `added_paths` builder (lines 637-649) to derive the full filesystem name: slugified name + middleware extension based on `added.class_name` and `added.children.is_empty()` + dedup suffix if colliding with siblings. Reuse `tentative_fs_name` logic from `ref_properties.rs` or the shared helper from the Deferred Refactors section.
- **Risk:** Medium; affects same-request Ref persistence edge path.
- **Verify round-trip:** Send `/api/write` with add(target script/model) + update(ref property pointing to target) in one batch; verify persisted path includes correct segment (`.server.luau`, `.model.json5`, or dedup suffix) and resolves after rebuild.
- **Verify syncback parity:** Run equivalent scenario via CLI syncback fixture and compare written `Rojo_Ref_`* attribute string.
- **Tests required:**
- Rust integration test in `[tests/tests/two_way_sync.rs](d:/UserGenerated/rojo/tests/tests/two_way_sync.rs)` for same-batch add+ref on file-backed and directory-backed targets.
- Edge test for dedup suffix target (e.g., `Foo~1`).
### Fix 4: Complete attribute-scoped final paths (previously incomplete)
- **Status:** Approved
- **Finding:** Step 2a / Step 9 / Step 11h (round-trip identity, parity, determinism)
- **Lineage:** `ambiguous_paths_audit_fixes_9d3ed6da` Fix 3 defined a two-phase fix. Phase 1 (correct misleading comment) was completed -- visible at lines 194-199 of `ref_properties.rs`. Phase 2 (pass `final_paths` to `collect_referents`) was NOT completed despite the todo being marked "completed". The API was prepared (`final_paths: Option<&HashMap<Ref, String>>` parameter exists at line 188 of `ref_properties.rs`) but the call site at line 177 of `mod.rs` still passes `None`. The `ref_path_map` infrastructure was built by `ref_path_and_dedup_integration_81a47dfe` and is populated during the syncback walk (used at lines 614-637 of `mod.rs` for the text replacement). This fix completes that unfinished Phase 2.
- **Files:** `[src/syncback/mod.rs](d:/UserGenerated/rojo/src/syncback/mod.rs)`, `[src/syncback/ref_properties.rs](d:/UserGenerated/rojo/src/syncback/ref_properties.rs)`, `[src/syncback/fs_snapshot.rs](d:/UserGenerated/rojo/src/syncback/fs_snapshot.rs)`
- **Problem:** `collect_referents(&new_tree, &pre_prune_paths, None)` is called at line 177 of `mod.rs` BEFORE the syncback walk that populates `ref_path_map`. The `None` means tentative paths (without dedup suffixes) are written into `Rojo_Ref_`* attributes. Then the post-hoc `fix_ref_paths` (lines 614-637) performs global text replacement across entire meta/model file contents to substitute tentative paths with final dedup'd paths. This is not attribute-scoped and can alter unrelated strings that happen to contain the same substring as a tentative path.
- **Existing infrastructure to build on:**
1. `collect_referents` already accepts `final_paths: Option<&HashMap<Ref, String>>` (line 188 of `ref_properties.rs`).
2. `ref_path_map` is already populated during the syncback walk with `(Ref, final_path)` entries.
3. `tentative_fs_path_public` is already used at line 623 of `mod.rs` to compute tentative paths for the substitution diff.
- **Solution:** Move the `collect_referents` call from line 177 (before the syncback walk) to after the walk completes (after line 612). Pass `Some(&ref_path_map.borrow())` instead of `None`. Then remove or guard the `fix_ref_paths` block (lines 614-637) -- it should no longer be needed for any path that has a `final_paths` entry.
- **Ordering constraint:** `collect_referents` is currently called before property filtering (lines 179-198). Moving it after the syncback walk means it runs after filtering. Verify that `collect_referents` does not depend on properties that are removed by filtering (it should not -- it only reads Ref-type properties, which are not in the filtered set unless `ignore_referents` is true, in which case refs are skipped entirely).
- **Risk:** Medium; changes the ordering of the syncback pipeline. The infrastructure is already in place, reducing implementation risk.
- **Verify round-trip:** Scenario with duplicate-named ref target and non-ref string property containing same substring as tentative path; ensure only `Rojo_Ref_`* changes and non-ref string is untouched.
- **Verify syncback parity:** Compare CLI syncback and two-way outputs for refs targeting dedup-suffixed instances; paths must be byte-identical.
- **Tests required:**
- Rust integration test in `[tests/tests/syncback_roundtrip.rs](d:/UserGenerated/rojo/tests/tests/syncback_roundtrip.rs)` validating non-ref string preservation while refs are rewritten correctly.
- Rust integration test in `[tests/tests/two_way_sync.rs](d:/UserGenerated/rojo/tests/tests/two_way_sync.rs)` for dedup-aware ref persistence without text rewrite side effects.
### Fix 5: Add regression coverage for approved fixes
- **Status:** Approved
- **Finding:** Step 13 (missing coverage)
- **Files:** `[tests/tests/two_way_sync.rs](d:/UserGenerated/rojo/tests/tests/two_way_sync.rs)`, `[tests/tests/syncback_roundtrip.rs](d:/UserGenerated/rojo/tests/tests/syncback_roundtrip.rs)`, optionally helper fixtures under `[rojo-test/syncback-tests](d:/UserGenerated/rojo/rojo-test/syncback-tests)`.
- **Problem:** Current tests do not fully guard the four approved bug classes.
- **Solution:** Add targeted regression tests tied to Fixes 1-4 and ensure they fail pre-fix and pass post-fix.
- **Risk:** Low.
- **Verify round-trip:** Covered by added integration scenarios.
- **Verify syncback parity:** Add assertions comparing resulting file paths/attribute values against expected CLI-format output.
- **Tests required:**
- All tests listed under Fixes 1-4.
- One deterministic re-run assertion (second run no changes) for affected scenarios.
## Skipped Fixes
### Skipped: ProjectNode duplicate-child handling rewrite
- **Finding:** Step 2d / Step 11k
- **Problem:** Duplicate-named children under ProjectNode are currently skipped with warnings (`[src/snapshot_middleware/project.rs](d:/UserGenerated/rojo/src/snapshot_middleware/project.rs)` around `~587-683`).
- **Reason skipped:** User accepted as release limitation for now.
### Skipped: Globally optimal matcher replacement
- **Finding:** Step 11h (determinism/assignment quality)
- **Problem:** Greedy assignment may produce suboptimal pairing in rare ambiguous groups.
- **Reason skipped:** User accepted greedy behavior for this release.
## Accepted Limitations
- **Limitation:** ProjectNode duplicate-named children are skipped during syncback.
- **Scenario:** Under project-backed nodes, duplicate names are detected and omitted from child maps; warning logged.
- **User decision:** Accepted for this release.
- **Mitigation:** Keep explicit warning logs; add failing acceptance test (below) to track until fixed.
- **Limitation:** Greedy matching is not globally optimal.
- **Scenario:** Ambiguous duplicate groups can choose a local minimum causing rename churn.
- **User decision:** Accepted for this release.
- **Mitigation:** Add stress/failing test for known counterexample and keep deterministic tie behavior.
## Deferred Refactors
- Consolidate filesystem-segment derivation into one shared helper used by `[src/rojo_ref.rs](d:/UserGenerated/rojo/src/rojo_ref.rs)`, `[src/snapshot/tree.rs](d:/UserGenerated/rojo/src/snapshot/tree.rs)`, and `[src/web/api.rs](d:/UserGenerated/rojo/src/web/api.rs)` to eliminate duplicate naming logic. *(Continuing from `ambiguous_paths_audit_fixes_9d3ed6da` Deferred Refactors -- still open.)*
- Revisit optimal assignment algorithm for large ambiguous groups (major rewrite; out of current scope). *(Continuing from `ambiguous_paths_audit_fixes_9d3ed6da` Deferred Refactors -- still open.)*
- Phase 2e matching cache from the original plan (`ambiguous_path_handling_f2eaef11`): add optional cache mapping `parent_ref -> (children_list_hash, match_assignments)` to avoid re-running the full algorithm when children haven't changed. Optimization only, not needed for correctness. *(Never completed or cancelled across any subsequent plan -- explicitly deferred here.)*
## Test Plan
### Rust Unit Tests (`#[cfg(test)]`)
- Dedup cleanup metadata update helper behavior after rename (Fix 1).
- Added-path segment generation helper returns correct extension/suffix across middleware types (Fix 3).
### Rust Integration Tests (`tests/tests/`)
- Delete base dedup name then update promoted sibling in same serve session; verify writes hit new path and persist after rebuild (Fix 1).
- Batch-remove 2 of 3 same-named siblings; verify sole survivor is renamed from `Foo~2` to `Foo` (Fix 2).
- Batch-remove all members of a dedup group; verify no spurious rename attempts (Fix 2).
- Same `/api/write` request: add target + set Ref to target, verify persisted path includes expected extension/suffix (Fix 3).
- Ref rewrite safety: non-ref string fields are unchanged while `Rojo_Ref_`* gets final dedup path (Fix 4).
### Lua Spec Tests (`.spec.lua`)
- No mandatory Lua-side changes for approved fixes; optional guard test for same-batch add+ref behavior if plugin-side helper touched.
### Snapshot Tests (`insta`)
- Add or update syncback snapshot fixture showing final dedup-aware `Rojo_Ref_`* output without incidental string changes (Fix 4).
### Failing Tests for Known Limitations
- ProjectNode duplicate-named child round-trip identity test asserting correct behavior (expected to fail until limitation removed).
- Ambiguous-group matching counterexample test asserting globally optimal pairing (expected to fail with greedy algorithm).
## Test Rules
### For bugs found and fixed
- Each approved bug fix must have a regression test that fails pre-fix and passes post-fix.
### For missing coverage identified
- Add coverage in affected layer(s); investigate any unexpected failure as a potential additional bug.
### For known limitations (Accepted Limitations section)
- Write correctness-asserting tests even if they fail now.
- Leave them failing (do not ignore), to serve as future acceptance criteria.
### Scope
- Cover both persistence correctness (filesystem outputs) and in-memory tree consistency where applicable.
## Final Step: Run CI
After implementing all approved fixes and tests, run `/ci` and require a clean pass before considering this plan complete.

185
.cursor/plans/audit_ref_path_system_c21fc130.plan.md Normal file
View file

@ -0,0 +1,185 @@
---
name: Audit Ref Path System
overview: Audit the Luau-style ref path system (rojo_ref.rs, tree.rs, patch_apply.rs, change_processor.rs, ref_properties.rs, api.rs, meta.rs) for round-trip correctness, resolution consistency, and edge-case safety.
todos:
- id: roundtrip
content: Verify compute_relative_ref_path <-> resolve_ref_path_to_absolute round-trip for all prefix classes
status: completed
- id: dual-resolution
content: Verify resolve_ref_path (tree.rs) and resolve_ref_path_to_absolute (rojo_ref.rs) produce identical results
status: completed
- id: priority
content: Audit Rojo_Ref_ vs Rojo_Target_ priority handling in patch_apply.rs
status: completed
- id: cleanup
content: Verify attribute cleanup completeness after ref resolution
status: completed
- id: fs-name-consistency
content: Audit filesystem_name_for vs ref_target_path_from_tree consistency
status: completed
- id: index-correctness
content: Audit RefPathIndex operations (populate, find_by_prefix, update_prefix, rename_file)
status: completed
- id: rename-fixup
content: Audit update_ref_paths_after_rename path reconstruction logic
status: completed
- id: placeholder
content: Audit placeholder system in CLI syncback (uniqueness, substitution order, fallback)
status: completed
- id: added-paths
content: Audit added_paths same-batch ref handling in api.rs (extensions, dedup suffixes)
status: completed
- id: test-gaps
content: Identify test coverage gaps and flag missing scenarios
status: completed
isProject: false
---
# Audit: Ref Path System (Luau-Style Ref Paths)
## Scope
The ref path system converts Roblox `Ref` properties (like `Model.PrimaryPart`) into filesystem-path-based `Rojo_Ref_*` attributes using Luau require-by-string syntax (`@self`, `./`, `../`, `@game/`). This audit covers:
- Path generation: `[src/rojo_ref.rs](src/rojo_ref.rs)` (`compute_relative_ref_path`, `ref_target_path_from_tree`)
- Path resolution: `[src/rojo_ref.rs](src/rojo_ref.rs)` (`resolve_ref_path_to_absolute`), `[src/snapshot/tree.rs](src/snapshot/tree.rs)` (`resolve_ref_path`, `walk_segments`, `get_instance_by_path`, `filesystem_name_for`)
- Forward sync application: `[src/snapshot/patch_apply.rs](src/snapshot/patch_apply.rs)` (`finalize_patch_application`, `defer_ref_properties`)
- Syncback generation: `[src/syncback/ref_properties.rs](src/syncback/ref_properties.rs)` (`collect_referents`, `tentative_fs_path`)
- Two-way sync: `[src/web/api.rs](src/web/api.rs)` (`syncback_updated_properties`, `added_paths`)
- Rename updates: `[src/change_processor.rs](src/change_processor.rs)` (`update_ref_paths_after_rename`), `[src/syncback/meta.rs](src/syncback/meta.rs)` (`update_ref_paths_in_file`)
- Index: `[src/rojo_ref.rs](src/rojo_ref.rs)` (`RefPathIndex`)
## Audit Methodology
Evaluate each area against the **round-trip identity invariant**: syncback writes `Rojo_Ref_`* paths, forward sync resolves them back to the same `Ref` targets. Any code path where a ref target can be lost, silently changed, or fail to resolve is a bug.
## Audit Areas
### 1. Round-Trip: `compute_relative_ref_path` <-> `resolve_ref_path_to_absolute`
**Files:** `src/rojo_ref.rs` lines 205-289
Verify the compute/resolve pair is bijective for all prefix classes:
- `@self` / `@self/...` -- descendant paths
- `./...` -- sibling paths (1 up)
- `../...` chains -- 2+ levels up within same service
- `@game/...` -- cross-service / ancestor refs
- Bare paths (no prefix) -- legacy fallback
**Specific checks:**
- `compute_relative_ref_path` uses **case-sensitive** common-prefix detection (line 222: `a == b`), but `resolve_ref_path` / `walk_segments` uses **case-insensitive** matching (line 365: `eq_ignore_ascii_case`). This asymmetry is safe only if path generation always uses canonical filesystem case. Verify this invariant holds across all callers of `ref_target_path_from_tree`.
- When `remaining_parts.is_empty()` (target is ancestor of source), falls through to `@game/` (line 231-233). Verify `resolve_ref_path_to_absolute` handles this correctly.
- Empty `source_abs` or `target_abs` edge cases.
### 2. Dual Resolution: `resolve_ref_path` (tree.rs) vs `resolve_ref_path_to_absolute` (rojo_ref.rs)
**Files:** `src/snapshot/tree.rs` lines 300-389, `src/rojo_ref.rs` lines 254-289
Two independent implementations resolve relative paths:
- `resolve_ref_path` (tree.rs) -- walks the live `RojoTree`, used during forward sync
- `resolve_ref_path_to_absolute` (rojo_ref.rs) -- pure string manipulation, used by `RefPathIndex` and `update_ref_paths_in_file`
Verify they handle **all prefix variants identically**:
- `../` prefix: tree.rs pops to grandparent THEN walks rest. rojo_ref.rs pops source + parent from parts THEN processes rest. Both handle subsequent `..` in rest via loop.
- Edge case: `../` on a root-level instance (1 segment in `source_abs`). `resolve_ref_path_to_absolute` does `p.pop(); p.pop()?` -- the second pop returns `None`. `resolve_ref_path` checks `grandparent.is_none()`. Both return None. Consistent.
- Empty rest after prefix strip: e.g., `"@self/"` → empty rest. tree.rs: `walk_segments(source, "")` returns `Some(source)`. rojo_ref.rs: no segments to iterate, returns `source_abs`. Consistent (equivalent to `@self`).
### 3. Priority: `Rojo_Ref_`* vs `Rojo_Target_`* in Forward Sync
**File:** `src/snapshot/patch_apply.rs` lines 110-205, 321-360
In `defer_ref_properties`: both `Rojo_Ref_`* and `Rojo_Target_`* are collected into **separate** maps without deduplication. In `finalize_patch_application`: `attribute_refs_to_rewrite` (Target) runs first (line 131), then `path_refs_to_rewrite` (Ref) runs second (line 144), **overwriting** the Target result.
**Finding:** Priority is correct (`Rojo_Ref_`* wins) but implemented via **execution order**, not explicit logic. If the blocks are reordered, priority silently flips. Consider whether this warrants a code comment or explicit dedup in `defer_ref_properties`.
### 4. Attribute Cleanup Completeness
**File:** `src/snapshot/patch_apply.rs` lines 171-202
After resolution, `instances_needing_attr_cleanup` triggers removal of all `Rojo_Id`, `Rojo_Target_`*, and `Rojo_Ref_`* attributes. Verify:
- Both `defer_ref_properties` code paths (Rojo_Target_ and Rojo_Ref_) add to `instances_needing_attr_cleanup`.
- Rojo_Id is also added (line 357).
- Cleanup iterates ALL attributes and removes matches by prefix -- catches attributes even if they weren't in the rewrite maps (e.g., orphaned attributes).
- Check: does an update patch also trigger defer_ref_properties? Yes (line 310).
### 5. `filesystem_name_for` Consistency
**File:** `src/snapshot/tree.rs` lines 394-418
This function is the **ground truth** for path segment names during forward sync. Verify:
- `InstigatingSource::Path` -- returns filename from disk. Correct: includes extension, dedup suffix, slugified name.
- `InstigatingSource::ProjectNode` -- returns instance name. Correct: project nodes are services or named containers, not files.
- No `instigating_source` -- falls back to instance name. This happens for newly-added instances (two-way sync). Verify this fallback doesn't break ref path resolution for refs pointing at just-added instances.
- `ref_target_path_from_tree` (rojo_ref.rs lines 140-191) independently reimplements the same logic. Check for consistency: both use `meta.instigating_source`, both have the same ProjectNode vs Path branching.
### 6. `RefPathIndex` Correctness
**File:** `src/rojo_ref.rs` lines 339-511
- `populate_from_dir` (line 400): scans filesystem directly (bypassing VFS), reads JSON5 files, resolves relative paths to absolute. The `source_abs` comes from `tree.get_ids_at_path`. **Edge case:** orphaned meta file (no tree entry) gets `source_abs = ""`, which corrupts relative path resolution (e.g., `"./Sibling"` resolves to `"Sibling"` instead of `"Service/Parent/Sibling"`). Low severity since orphaned meta files shouldn't exist during normal serve sessions.
- `find_by_prefix` (line 464): checks exact match OR prefix-with-slash. Correct. No false positives from segment-prefix overlap (e.g., `"Workspace/Foo"` won't match `"Workspace/FooBar"`).
- `update_prefix` (line 491): merges file sets when new key already exists. Correct for convergent renames.
- `rename_file` (line 481): updates filesystem paths in all entries. O(n) scan -- acceptable since called rarely.
### 7. `update_ref_paths_after_rename` Path Fixup
**File:** `src/change_processor.rs` lines 429-540
When the indexed file path no longer exists (directory was renamed), this code tries to reconstruct the new path by replacing the old directory segment with the new one in the PathBuf components (lines 462-497).
**Checks:**
- Only replaces the **first** occurrence. If the old segment appears in multiple path components (e.g., `src/Foo/Foo/file.meta.json5`), only the first gets replaced. This is correct for single-level renames but could fail for deeply nested same-named directories. **Low severity** since dedup suffixes prevent truly identical sibling names.
- Handles both plain name and slugified name variants (lines 448-484).
- After updating files, updates the index with both `update_prefix` and `rename_file` (lines 532-540). The zip iteration assumes `original_paths` and `files_to_check` have the same length and order -- they do since `files_to_check` is a 1:1 map of the originals.
### 8. Placeholder System (CLI Syncback)
**Files:** `src/syncback/ref_properties.rs`, `src/syncback/mod.rs`, `src/syncback/fs_snapshot.rs`
Placeholders (`__ROJO_REF_<source>_TO_<target>`__) are written during the syncback walk before dedup suffixes are known, then substituted with final relative paths.
**Checks:**
- Placeholder uniqueness: encodes both source and target Refs. Unique per (source, target) pair.
- Substitution in `fix_ref_paths`: verify it handles placeholders appearing in both meta and model files.
- Length-descending sort prevents partial prefix matches (e.g., `__ROJO_REF_A_TO_B`__ won't partially match `__ROJO_REF_A_TO_BC_`_ if the longer one is processed first).
- Fallback: if final path not in `ref_path_map`, falls back to `tentative_fs_path_public()`. Verify this fallback produces a valid path.
### 9. Two-Way Sync: `added_paths` for Same-Batch Refs
**File:** `src/web/api.rs`
When the plugin adds instances and sets Ref properties targeting them in the same `/api/write` request, `added_paths` pre-computes filesystem paths. Verify:
- `added_instance_fs_segment` mirrors `tentative_fs_name` logic from ref_properties.rs.
- Paths include correct extensions (`.server.luau`, `.luau`, `.model.json5`, no extension for folders).
- Dedup suffixes are NOT included in `added_paths` (the instance hasn't been written yet), which could cause the ref path to mismatch if the actual write adds a dedup suffix.
### 10. Test Coverage Assessment
**Files:** `src/rojo_ref.rs` (tests), `tests/tests/two_way_sync.rs`, `tests/tests/syncback.rs`
Existing tests cover: basic ref write, nil ref, target change, rename update, same-batch add, dedup cleanup, RefPathIndex population, relative path round-trips (100+ cases).
**Gaps to check:**
- Multi-level ancestor renames (rename grandparent, verify ref 3 levels down updates)
- Cross-nested-project refs
- Refs where source and target are in the same dedup group
- `../` chains exceeding 2 levels
- Unicode instance names in ref paths
## Severity Scale
- **Critical** -- Round-trip identity violated; ref targets silently lost or changed
- **High** -- Refs fail to resolve in specific but reproducible scenarios
- **Medium** -- Code fragility or missing error handling that could cause future regressions
- **Low** -- Minor edge cases, code quality, or documentation gaps

211
.cursor/plans/changebatcher_encoding_audit_bade61c8.plan.md Normal file
View file

@ -0,0 +1,211 @@
---
name: ChangeBatcher Encoding Audit
overview: Audit of the ChangeBatcher encoding refactors covering encodeService.lua (new), encodeInstance.lua (modified), and related test changes across 4 files (132 insertions, 63 deletions).
todos:
- id: fix-header-comment
content: "Fix stale header comment in encodeInstance.lua (Finding 1): update DUPLICATE HANDLING section to say duplicates are encoded and delegated to server"
status: completed
- id: extract-shared-filter
content: Extract shared property encoding filter from encodeService.lua and encodeInstance.lua into a common helper to eliminate duplication (Finding 2)
status: completed
- id: add-encodeService-tests
content: Create encodeService.spec.lua with tests for property encoding, Ref carrier logic, Attributes/Tags, EXCLUDE_PROPERTIES, and empty services (Finding 3)
status: completed
- id: document-clocktime-exclude
content: Add comment to EXCLUDE_PROPERTIES in encodeService.lua explaining why Lighting.ClockTime is excluded (Finding 4)
status: completed
- id: fix-servicechunk-doc
content: Fix misleading doc comment on ServiceChunk in src/web/interface.rs to correctly describe refs as 1-based carrier indices (Finding 5)
status: completed
isProject: false
---
# ChangeBatcher Encoding Refactors -- Audit Report
## Scope
4 files changed: +132 / -63 across 13 commits.
| File | Change Type | Lines |
| -------------------------------- | --------------- | -------------------------------- |
| `encodeService.lua` | New (108 lines) | +108 |
| `encodeInstance.lua` | Modified | +24 / -53 |
| `encodeInstance.spec.lua` | Modified | +3 / -4 |
| `encodeInstance.stress.spec.lua` | Modified | -6 (removed skippedCount checks) |
---
## Architecture Summary
```mermaid
flowchart TD
subgraph TwoWaySync [Two-Way Sync Path]
InstanceMap["InstanceMap signals"] --> ChangeBatcher
ChangeBatcher --> createPatchSet
createPatchSet --> encodePatchUpdate
encodePatchUpdate --> encodeProperty
end
subgraph LiveSyncback [Live Syncback Path]
performSyncback["App:performSyncback()"] --> encodeService
encodeService --> encodeProperty
performSyncback --> encodeInstance
encodeInstance --> encodeProperty
end
encodeProperty --> RbxDom
encodeService --> propertyFilter
encodeInstance --> propertyFilter
encodePatchUpdate --> propertyFilter
```
`encodeService` is new and serves the live syncback feature. `encodeInstance` was refactored to delegate duplicate handling to the server.
---
## Finding 1: Stale Header Comment (Bug)
**Severity: Moderate** | **File:** `[encodeInstance.lua](plugin/src/ChangeBatcher/encodeInstance.lua)` lines 1-17
The module header comment claims:
```
DUPLICATE HANDLING:
- Instances with duplicate-named siblings at ANY level of the path are skipped
```
But the code was changed to do the opposite -- duplicates are now **encoded** (not skipped), with a debug log, and the server handles dedup. The header directly contradicts the code behavior.
**Fix:** Update the header to reflect the new behavior.
---
## Finding 2: Duplicated Encoding Filter Logic (Code Quality)
**Severity: Moderate (small refactor candidate)** | **Files:** `[encodeService.lua](plugin/src/ChangeBatcher/encodeService.lua)`, `[encodeInstance.lua](plugin/src/ChangeBatcher/encodeInstance.lua)`
The property encoding filter is copy-pasted across both files with only minor differences:
| Logic | `encodeService.lua` | `encodeInstance.lua` |
| --------------------------------------------- | ------------------- | -------------------- |
| SKIP_PROPERTIES (Parent, Name, Archivable) | Lines 7-11 | Lines 34-38 |
| Skip Attributes/Tags (handled separately) | Line 33 | Lines 217-219 |
| Scriptability check (ReadWrite, Read, Custom) | Lines 37-39 | Lines 224-226 |
| DoesNotSerialize check | Line 40 | Line 227 |
| UNENCODABLE_DATA_TYPES filter | Line 45 | Line 233 |
| Attributes encoding (pcall + descriptor) | Lines 70-81 | Lines 121-138 |
| Tags encoding (pcall + descriptor) | Lines 83-94 | Lines 141-157 |
If either filter is updated (e.g., adding a new skip property, changing the scriptability check), the other must be updated identically or a divergence bug is introduced.
**Fix:** Extract a shared `iterEncodableProperties(className, callback)` helper that handles the filter logic once. Both `encodeService` and `encodeInstance` would call it. The Attributes/Tags encoding blocks could also be extracted to a shared `encodeAttributesAndTags(instance, properties)` function (the `encodeAttributes`/`encodeTags` functions in `encodeInstance.lua` already do this but aren't exported).
---
## Finding 3: No Tests for `encodeService.lua` (Test Gap)
**Severity: Moderate** | **File:** `encodeService.lua` (new, 108 lines, no spec file)
Every other encoding module has a `.spec.lua` test:
- `encodeInstance.spec.lua` (exists)
- `encodeInstance.stress.spec.lua` (exists)
- `encodePatchUpdate.spec.lua` (exists)
- `encodeProperty.spec.lua` (exists)
- `**encodeService.spec.lua` -- MISSING**
`encodeService` handles Ref property encoding via ObjectValue carriers (a tricky serialization path), service-level property filtering, and the carrier index calculation. These all warrant test coverage.
**Fix:** Add `encodeService.spec.lua` covering:
- Basic property encoding for a service (e.g., Lighting)
- Ref property carrier creation and index correctness
- Attributes/Tags encoding on services
- EXCLUDE_PROPERTIES filtering (Lighting.ClockTime)
- Empty service (no properties, no children)
---
## Finding 4: `EXCLUDE_PROPERTIES` Undocumented (Documentation)
**Severity: Low** | **File:** `[encodeService.lua](plugin/src/ChangeBatcher/encodeService.lua)` lines 13-15
```lua
local EXCLUDE_PROPERTIES = {
Lighting = { ClockTime = true },
}
```
`ClockTime` (Float32) and `TimeOfDay` (String) are linked properties on Lighting -- they represent the same value in different formats, and both have `Serialization: "Serializes"`. Excluding `ClockTime` avoids sending a redundant/conflicting representation alongside `TimeOfDay`. This is a reasonable design choice, but there is no comment explaining the reasoning. If someone sees `TimeOfDay` behaving oddly, they won't know `ClockTime` was intentionally excluded or why.
**Fix:** Add a comment explaining the linked-property reason.
---
## Finding 5: Misleading `ServiceChunk` Doc Comment (Documentation)
**Severity: Low** | **File:** `[src/web/interface.rs](src/web/interface.rs)` lines 30-42
The doc comment on `ServiceChunk` states:
> `refs` maps property names to 1-based indices into this combined range (0 = nil). Indices <= child_count point directly to a child instance. Indices > child_count point to an ObjectValue carrier.
But the server code in `[serve.rs](src/cli/serve.rs)` lines 198-204 treats `refs` values as **1-based indices into the carrier slice only** (not the combined range):
```rust
let carriers = &service_range[carrier_start..carrier_end];
for (prop_name, &idx) in &chunk.refs {
if idx == 0 || (idx as usize) > carriers.len() { continue; }
ref_entries.push((prop_name.clone(), carriers[idx as usize - 1]));
}
```
And the plugin sets `refs[propertyName] = #refTargets` (1-based into refTargets only). The "indices into combined range" and "indices <= child_count point to a child" parts of the comment are incorrect -- carriers never point to children.
**Fix:** Update the doc comment to say `refs` maps property names to 1-based indices into the carrier slice.
---
## Positive Changes (No Action Needed)
### P1: Module-Level Mutable State Removed (encodeInstance.lua)
The old code had `local skippedDuplicateCount = 0` at module scope, mutated across calls and reset inside a wrapper function. This was a shared mutable state pattern that risked stale counts if the wrapper wasn't used. The new code removes this entirely -- clean improvement.
### P2: Duplicate Handling Delegated to Server
Previously, `encodeInstance` skipped any instance whose path contained duplicate-named siblings, silently losing data. Now all instances are encoded and sent to the server, which handles filesystem name collisions via the dedup suffix system (`~2`, `~3`, etc.). This is the correct design -- the server has the filesystem context needed to resolve collisions, while the plugin should send complete data.
### P3: `Custom` Scriptability Added
The filter now includes `propertyMeta.scriptability == "Custom"` alongside `ReadWrite` and `Read`. The reflection database has 9 `Custom` scriptability properties. After accounting for those handled separately (Attributes, Tags, Source, Contents), this adds coverage for `Scale` (NumberValue), `WorldPivotData` (Model), `PropertiesSerialize` (StyleSheet), and `MaterialColors` (Workspace) -- all of which serialize and were previously silently dropped.
### P4: Simplified Wrapper Function
The `encodeInstanceWithDuplicateTracking` wrapper was replaced with a simple `return function(instance, parentId)` that calls `encodeInstance(instance, parentId, false)`. Tests updated to match (no more second `skippedCount` return value).
### P5: Tests Updated Correctly
Both `encodeInstance.spec.lua` and `encodeInstance.stress.spec.lua` are updated to assert that duplicates ARE included (e.g., `expect(#encoded.children).to.equal(3)` instead of `1`). All `skippedCount` assertions removed.
---
## Summary
| # | Finding | Severity | Type |
| --- | ------------------------------------------ | -------- | --------------------- |
| 1 | Stale header comment in encodeInstance.lua | Moderate | Bug (misleading docs) |
| 2 | Duplicated encoding filter logic | Moderate | Code quality |
| 3 | No tests for encodeService.lua | Moderate | Test gap |
| 4 | EXCLUDE_PROPERTIES undocumented | Low | Documentation |
| 5 | Misleading ServiceChunk doc comment | Low | Documentation |
No critical or round-trip-identity issues found. The encoding logic is correct, the server-side consumption matches, and the refactors improve the codebase. The main action items are updating stale comments, adding a test file, and extracting the duplicated filter logic.

328
.cursor/plans/compact_json5_arrays_36a8cfab.plan.md Normal file
View file

@ -0,0 +1,328 @@
---
name: Compact JSON5 arrays
overview: Arrays with <12 non-compound elements render on one line. Recursive -- inner arrays compact independently.
todos:
- id: modify-array-arm
content: "Modify the Array arm of write_to() in src/json.rs: inline if <12 elements and no element is Array/Object"
status: completed
- id: snapshot-tests
content: Run cargo test, then cargo insta review to update affected snapshots
status: completed
isProject: false
---
# Compact JSON5 Array Formatting
## Rule
An array renders inline if:
1. Fewer than 12 direct elements
2. No direct element is an `Array` or `Object`
Format: `[ elem, elem, elem ]` (spaces inside brackets). Recursive -- each nested array independently decides.
## Change
One file: [src/json.rs](src/json.rs), the Array arm of `Json5Value::write_to()` (lines 154-169). ~10 lines added.
---
## Expected Output Examples
### Color3 (3 floats)
Before:
```json5
Color3: [
0.41568630933761597,
0,
1
]
```
After:
```json5
Color3: [ 0.41568630933761597, 0, 1 ]
```
### NumberRange (2 floats)
Before:
```json5
Lifetime: {
NumberRange: [
0.25,
0.6000000238418579
]
}
```
After:
```json5
Lifetime: {
NumberRange: [ 0.25, 0.6000000238418579 ]
}
```
### Tags (string array)
Before:
```json5
Tags: [
"SoundGroupEffects"
]
```
After:
```json5
Tags: [ "SoundGroupEffects" ]
```
Multiple tags:
```json5
Tags: [ "_VFX", "Collectible", "PowerUp" ]
```
### CFrame (object with position + orientation)
Before:
```json5
CFrame: {
position: [
-0.09659165,
1.517803,
-0.83597946
],
orientation: [
[
0.10630996,
-0.0024746126,
-0.9943304
],
[
0.021991052,
0.999758,
-0.0021534874
],
[
0.9941102,
-0.021623673,
0.1063388
]
]
}
```
After:
```json5
CFrame: {
position: [ -0.09659165, 1.517803, -0.83597946 ],
orientation: [
[ 0.10630996, -0.0024746126, -0.9943304 ],
[ 0.021991052, 0.999758, -0.0021534874 ],
[ 0.9941102, -0.021623673, 0.1063388 ]
]
}
```
- `position` (3 numbers) -> inline
- `orientation` (3 arrays) -> multi-line because elements are arrays
- Each inner orientation row (3 numbers) -> inline
### UDim2 (2 arrays of 2 numbers)
Before:
```json5
Size: [
[
1,
0
],
[
0,
100
]
]
```
After:
```json5
Size: [
[ 1, 0 ],
[ 0, 100 ]
]
```
Outer has array elements -> multi-line. Inner arrays (2 numbers each) -> inline.
### Rect (2 Vector2s)
Before:
```json5
ImageRectOffset: [
[
100,
100
],
[
200,
200
]
]
```
After:
```json5
ImageRectOffset: [
[ 100, 100 ],
[ 200, 200 ]
]
```
### ColorSequence keypoints (array of objects -- NO CHANGE)
Before and after (objects always expand):
```json5
Color: {
ColorSequence: {
keypoints: [
{
color: [ 0.364705890417099, 0.11372549086809158, 1 ],
time: 0
},
{
color: [ 0.364705890417099, 0.11372549086809158, 1 ],
time: 1
}
]
}
}
```
- `keypoints` array has objects -> stays multi-line
- `color` arrays inside each keypoint (3 numbers) -> inline
### NumberSequence with 20 keypoints (array of objects -- NO CHANGE)
Stays fully expanded. Both because elements are objects AND count >= 12.
### Full file: Texture.model.json5
Before:
```json5
{
className: "Texture",
properties: {
Color3: [
0.41568630933761597,
0,
1
],
Face: "Top",
StudsPerTileU: 200,
StudsPerTileV: 200,
TextureContent: "rbxassetid://10025668350",
Transparency: 0.6000000238418579
}
}
```
After:
```json5
{
className: "Texture",
properties: {
Color3: [ 0.41568630933761597, 0, 1 ],
Face: "Top",
StudsPerTileU: 200,
StudsPerTileV: 200,
TextureContent: "rbxassetid://10025668350",
Transparency: 0.6000000238418579
}
}
```
### Full file: Water.model.json5
Before:
```json5
{
attributes: {
Rojo_Ref_SoundGroup: "ServerStorage/__server_muted.model.json5"
},
className: "Sound",
properties: {
AudioContent: "rbxassetid://9120549564",
Looped: true,
Playing: true,
SourceAssetId: 9120549564,
Tags: [
"SoundGroupEffects"
],
Volume: 3
}
}
```
After:
```json5
{
attributes: {
Rojo_Ref_SoundGroup: "ServerStorage/__server_muted.model.json5"
},
className: "Sound",
properties: {
AudioContent: "rbxassetid://9120549564",
Looped: true,
Playing: true,
SourceAssetId: 9120549564,
Tags: [ "SoundGroupEffects" ],
Volume: 3
}
}
```
### Edge: empty array (NO CHANGE)
```json5
Tags: []
```
### Edge: 12+ primitive elements (stays multi-line)
```json5
SomeList: [
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12
]
```

96
.cursor/plans/consolidate_filesystem_name_for_f2bdef69.plan.md Normal file
View file

@ -0,0 +1,96 @@
---
name: Consolidate filesystem_name_for
overview: Eliminate the duplicated InstigatingSource branching logic by having `ref_target_path_from_tree` call `tree.filesystem_name_for()` instead of reimplementing the same logic inline.
todos:
- id: consolidate
content: Replace inline InstigatingSource branching in ref_target_path_from_tree with tree.filesystem_name_for() call
status: completed
- id: verify
content: Run cargo test and cargo clippy to confirm no regressions
status: completed
isProject: false
---
# Consolidate `filesystem_name_for` Duplication
## Problem
`ref_target_path_from_tree` (`[src/rojo_ref.rs](src/rojo_ref.rs)` lines 140-191) reimplements the same `InstigatingSource` branching logic that `filesystem_name_for` (`[src/snapshot/tree.rs](src/snapshot/tree.rs)` lines 394-418) already provides. The [ref path audit](c:/Users/Joe/.cursor/plans/audit_ref_path_system_c21fc130.plan.md) confirmed these produce identical results for all branches.
## Approach
`ref_target_path_from_tree` already receives `&RojoTree` and `filesystem_name_for` is already `pub` on `RojoTree`. The fix is to replace the inline branching with a call to the existing method. No new functions, no signature changes, no new files.
## Change
**Single file: `[src/rojo_ref.rs](src/rojo_ref.rs)`** -- Replace the 25-line `let fs_name = if let Some(meta) ...` block (lines 158-183) inside `ref_target_path_from_tree` with a single call to `tree.filesystem_name_for(current)`:
Before (lines 140-191):
```rust
pub fn ref_target_path_from_tree(tree: &crate::snapshot::RojoTree, target_ref: Ref) -> String {
let dom = tree.inner();
let root_ref = dom.root_ref();
let mut components: Vec<String> = Vec::new();
let mut current = target_ref;
loop {
if current == root_ref || current.is_none() { break; }
let inst = match dom.get_by_ref(current) {
Some(i) => i,
None => break,
};
// 25 lines of InstigatingSource branching (duplicated from tree.rs)
let fs_name = if let Some(meta) = tree.get_metadata(current) {
// ... Path branch ...
// ... ProjectNode branch ...
// ... no source fallback ...
} else {
inst.name.clone()
};
components.push(fs_name);
current = inst.parent();
}
components.reverse();
components.join("/")
}
```
After:
```rust
pub fn ref_target_path_from_tree(tree: &crate::snapshot::RojoTree, target_ref: Ref) -> String {
let dom = tree.inner();
let root_ref = dom.root_ref();
let mut components: Vec<String> = Vec::new();
let mut current = target_ref;
loop {
if current == root_ref || current.is_none() { break; }
let inst = match dom.get_by_ref(current) {
Some(i) => i,
None => break,
};
components.push(tree.filesystem_name_for(current));
current = inst.parent();
}
components.reverse();
components.join("/")
}
```
## Scope
- **1 file** changed
- **~25 lines** removed (duplicated branching logic)
- **1 line** added (call to existing method)
- **0 call sites** affected (function signature unchanged)
- **0 test changes** needed (behavior is identical per the audit)
## Verification
- `cargo test` -- all existing ref path tests (100+ unit, 30+ integration) exercise this function
- `cargo clippy` -- no new warnings expected

89
.cursor/plans/dedup_suffix_start_at_2_89e66bf2.plan.md Normal file
View file

@ -0,0 +1,89 @@
---
name: Dedup suffix start at 2
overview: Change the dedup suffix numbering to start at `~2` instead of `~1`, so the sequence is `Something.luau`, `Something~2.luau`, `Something~3.luau` -- consistent with Windows duplicate naming conventions.
todos:
- id: core-loops
content: Change `for i in 1..` to `for i in 2..` in both `deduplicate_name_with_ext` and `deduplicate_name` in `src/syncback/file_names.rs`
status: completed
- id: unit-tests
content: Update unit test expectations in `file_names.rs` for generated suffix assertions (~1 -> ~2, ~2 -> ~3)
status: completed
- id: integration-tests
content: Update integration test assertions in `two_way_sync.rs` and `connected_mode.rs`
status: completed
- id: fixtures-rename
content: "Rename test fixture files: connected_slugify, ref_ambiguous_path, UFOWave_matching expected outputs"
status: completed
- id: snapshot-tests
content: Run cargo test + cargo insta review to update snapshot expectations
status: completed
- id: docs-update
content: Update atlas-dedup.mdc examples to reflect ~2 starting suffix
status: completed
isProject: false
---
# Dedup Suffix: Start at ~2
## Core Change
The only production logic change is in `[src/syncback/file_names.rs](src/syncback/file_names.rs)` -- two `for` loops that generate dedup suffixes:
`**deduplicate_name_with_ext` (line ~297):**
```rust
// Before:
for i in 1.. {
// After:
for i in 2.. {
```
`**deduplicate_name` (line ~315):**
```rust
// Before:
for i in 1.. {
// After:
for i in 2.. {
```
These are the only two places where dedup suffix numbers are generated. All other code (`parse_dedup_suffix`, `compute_cleanup_action`, `build_dedup_name`, cleanup logic in `change_processor.rs`, `api.rs`) operates on whatever numbers exist on disk and needs no changes.
## Backward Compatibility
- `**parse_dedup_suffix**` in `[src/syncback/dedup_suffix.rs](src/syncback/dedup_suffix.rs)` (line ~~53): keep `if n > 0` unchanged. Old filesystems may have `~~1` files; forward sync must still read them correctly.
- `**build_dedup_name**`: takes an explicit number, no change needed.
- **Cleanup rules**: operate on parsed suffix numbers dynamically, work with any starting value.
## Test Updates
### Unit Tests
- `[src/syncback/file_names.rs](src/syncback/file_names.rs)` test module (~~line 389+): update assertions that expect generated `~~1`to`~~2`,`~~ 2`to`~3`, etc. Key tests:` dedup_single_collision`,` dedup_multiple_collisions`,` dedup_skips_taken_suffix`,` dedup_gap_in_suffixes`.
- `[src/syncback/dedup_suffix.rs](src/syncback/dedup_suffix.rs)` test module: `parse_suffix_basic` still tests `~1` parsing (backward compat) -- no change. `build_name_file` references `~1` as explicit input to `build_dedup_name` -- no change (function takes arbitrary numbers). Cleanup tests (`cleanup_group_to_one`, `cleanup_base_deleted_promote_lowest`, etc.) use `~1`/`~2` as existing filesystem state -- no change needed since they test cleanup of pre-existing suffixes.
### Integration Tests
- `[tests/tests/two_way_sync.rs](tests/tests/two_way_sync.rs)`: update all generation assertions: `X_Y~1` to `X_Y~2`, `X_Y~2` to `X_Y~3`. Affected tests: `add_two_colliding_instances_deduplicates`, `add_three_colliding_instances_deduplicates`, `delete_deduped_instance_group_to_1_cleanup`, `delete_base_deduped_instance_promotes_lowest`, `delete_middle_deduped_instance_tolerates_gap`, `add_same_named_folders_deduplicates`, `ref_through_deduped_instance`, `ref_path_updated_after_dedup_cleanup`.
- `[tests/tests/connected_mode.rs](tests/tests/connected_mode.rs)`: update tests that interact with generated dedup files. `add_third_collision` expects `Hey_Bro~2` -- should become `Hey_Bro~3`. Tests that read existing fixture files (`edit_dedup_file_with_meta`, `remove_one_of_two_colliding_files`) need fixture updates (see below).
### Test Fixtures (rename files)
- `rojo-test/serve-tests/connected_slugify/src/Hey_Bro~1.luau` -> `Hey_Bro~2.luau`
- `rojo-test/serve-tests/connected_slugify/src/Hey_Bro~1.meta.json5` -> `Hey_Bro~2.meta.json5`
- `rojo-test/serve-tests/ref_ambiguous_path/src/Workspace/DupParent/Child~1.model.json5` -> `Child~2.model.json5` (and update any ref paths inside)
- `rojo-test/syncback-tests/UFOWave_matching/expected/TsunamiWave/`: rename all `~N` files to `~(N+1)` (e.g., `Texture~1` -> `Texture~2`, ..., `Texture~11` -> `Texture~12`)
**Keep unchanged (backward compat reading):**
- `rojo-test/build-tests/dedup_suffix_with_meta/src/Foo~1.`* -- tests that forward sync reads old `~1` files correctly
- `rojo-test/build-tests/tilde_no_meta/src/Foo~1.luau` -- tests literal tilde name (no meta)
### Snapshot Tests
- Run `cargo test` then `cargo insta review` to accept updated snapshot expectations. The `.snap` files in `rojo-test/build-test-snapshots/` and `rojo-test/serve-test-snapshots/` will auto-update.
## Documentation Update
- `[.cursor/rules/atlas-dedup.mdc](.cursor/rules/atlas-dedup.mdc)`: update examples that reference `~1` as generated output (e.g., `"Foo~1.server.luau"` -> `"Foo~2.server.luau"` where it represents generated names; keep `~1` where it represents parseable/legacy names).

284
.cursor/plans/extension-free_ref_paths_b9a68637.plan.md Normal file
View file

@ -0,0 +1,284 @@
---
name: extension-free ref paths
overview: Remove file extensions from ref path segments and enforce dedup suffixes for ALL instances sharing an Instance.Name, regardless of middleware/file type. Ref paths change from `Workspace/Foo.server.luau` to `Workspace/Foo`, and siblings like `LocalScript.local.luau` + `LocalScript.client.luau` now become `LocalScript.local.luau` + `LocalScript~2.client.luau`.
todos:
- id: strip-ext
content: Add `strip_known_extensions()` utility to `file_names.rs` with unit tests
status: pending
- id: dedup-bare-slug
content: Change `deduplicate_name_with_ext()` and `name_for_inst()` to use bare slug for collision detection; update dedup_key semantics
status: pending
- id: taken-names-sites
content: Update ALL taken_names seeding sites (dir.rs, project.rs, api.rs) to use bare slugs via strip_known_extensions
status: pending
- id: ref-segment
content: Add `ref_segment_for()` to tree.rs; update `walk_segments()` and `get_instance_by_path()` to match bare stems
status: pending
- id: ref-construction
content: Update `ref_target_path_from_tree()`, `tentative_fs_name()`, `added_instance_fs_segment()`, `record_ref_path()` to use bare stems
status: pending
- id: cleanup-grouping
content: Update change_processor.rs cleanup to group by bare stem (remove extension filter), refactor compute_cleanup_action for mixed-extension groups
status: pending
- id: unit-tests-dedup
content: Update all file_names.rs unit tests for new bare-slug taken_names semantics; add same-name-different-type tests
status: pending
- id: unit-tests-ref
content: Update rojo_ref.rs unit tests for extension-free ref path format
status: pending
- id: forward-sync-constraint
content: "Add forward sync handling for bare-stem uniqueness constraint: walk_segments backward compat fallback for old-format refs, and warning/auto-dedup for legacy same-bare-stem files"
status: pending
- id: integration-tests
content: Add build test fixture for same-name-different-type dedup; add connected_mode cleanup test; update snapshot expectations with `cargo insta review`
status: pending
isProject: false
---
# Extension-Free Ref Paths
## Core Semantic Change
**Old behavior:** Dedup key = full filesystem name (slug + extension). `Foo.server.luau` and `Foo.luau` do NOT collide. Ref paths include extensions: `Workspace/Foo.server.luau`.
**New behavior:** Dedup key = bare slug only (no extension). ALL instances with the same slug collide and get `~2`, `~3`, etc. Ref paths use bare stems: `Workspace/Foo`, `Workspace/Foo~2`.
`~1` is never produced (already the case, maintained).
## Hard Constraint: Bare Stem Uniqueness
**Files in the same directory can NEVER share a primary name (bare stem), even if they have different types/extensions.** This is the foundational invariant of this change. Previously, `Foo.server.luau` and `Foo.luau` could coexist because their full filenames differed. Now they CANNOT -- one must get a dedup suffix (`Foo~2`).
This constraint must be enforced at multiple levels:
1. **Writeout (syncback/two-way sync):** The dedup system uses bare slugs in `taken_names`, so same-named instances of different types always get `~2`, `~3`, etc. This is the primary enforcement mechanism.
2. **Forward sync (reading files):** When reading a directory, if two files share a bare stem (legacy state from before this change), forward sync needs to handle it. The `walk_segments` resolver must support a **backward compatibility fallback** that matches old-format refs (with extensions) so existing projects don't break. Additionally, we should log a warning when same-bare-stem siblings are detected, guiding users to re-run syncback to fix the state.
3. **Ref path resolution backward compat:** `walk_segments()` should have a 3-tier fallback:
- **Primary:** Match against `ref_segment_for()` (bare stem) -- new format
- **Fallback 1:** Match against `filesystem_name_for()` (full filename) -- old format backward compat
- **Fallback 2:** Match against instance name -- existing fallback
This ensures old ref paths like `./Foo.server.luau` still resolve correctly until the project is re-synced.
## Phase 1: Core Dedup Change
### 1a. Add `strip_known_extensions()` to [file_names.rs](src/syncback/file_names.rs)
New utility function that strips known middleware extensions from a filename to recover the bare stem (slug + dedup suffix). Must check compound extensions before simple ones:
```rust
pub fn strip_known_extensions(filename: &str) -> &str {
const COMPOUND: &[&str] = &[
".server.luau", ".client.luau", ".plugin.luau", ".local.luau", ".legacy.luau",
".server.lua", ".client.lua", ".plugin.lua",
".model.json5", ".model.json",
".project.json5", ".project.json",
".meta.json5", ".meta.json",
];
const SIMPLE: &[&str] = &[
".luau", ".lua", ".json5", ".json", ".csv", ".txt",
".rbxm", ".rbxmx", ".toml", ".yml", ".yaml",
];
for ext in COMPOUND { if let Some(s) = filename.strip_suffix(ext) { return s; } }
for ext in SIMPLE { if let Some(s) = filename.strip_suffix(ext) { return s; } }
filename
}
```
### 1b. Modify `deduplicate_name_with_ext()` in [file_names.rs](src/syncback/file_names.rs)
Change collision check from full filesystem name to bare slug:
```rust
// OLD: checks "foo.server.luau" against taken_names
// NEW: checks "foo" (bare slug) against taken_names
if !taken_names.contains(&base.to_lowercase()) { ... }
```
`taken_names` now stores bare slugs (lowercased, no extensions).
### 1c. Modify `name_for_inst()` in [file_names.rs](src/syncback/file_names.rs)
- **New instance branch:** `dedup_key` = bare slug (with dedup suffix if applied), NOT full filesystem name.
- **Old instance branch:** `dedup_key` = `strip_known_extensions(filename)` (bare stem from existing file).
- Doc comments updated to reflect new semantics.
### 1d. Update all `taken_names` seeding sites
These files seed `taken_names` from existing filesystem entries and must switch to bare slugs:
- [dir.rs](src/snapshot_middleware/dir.rs) lines 224-242: `taken_names.insert(filename.to_lowercase())` must become `taken_names.insert(strip_known_extensions(filename).to_lowercase())`
- [project.rs](src/snapshot_middleware/project.rs) lines 736-744: same change
- [api.rs](src/web/api.rs): any taken_names seeding from tree siblings
## Phase 2: Ref Path Segment Change
### 2a. Add `ref_segment_for()` to [tree.rs](src/snapshot/tree.rs)
New method on `RojoTree` that returns the bare stem for use in ref paths. Keep `filesystem_name_for()` unchanged for any non-ref-path callers (verify first).
```rust
pub fn ref_segment_for(&self, id: Ref) -> String {
if let Some(meta) = self.metadata_map.get(&id) {
if let Some(source) = &meta.instigating_source {
match source {
InstigatingSource::Path(_) => {
if let Some(name) = source.path().file_name().and_then(|f| f.to_str()) {
return strip_known_extensions(name).to_string();
}
}
InstigatingSource::ProjectNode { .. } => {
if let Some(inst) = self.inner.get_by_ref(id) {
return inst.name.clone();
}
}
}
}
}
self.inner.get_by_ref(id).map(|i| i.name.clone()).unwrap_or_default()
}
```
For `Foo~2.server.luau` this returns `Foo~2`. For `Foo/` (dir) this returns `Foo`. For project nodes, returns instance name.
### 2b. Update `walk_segments()` and `get_instance_by_path()` in [tree.rs](src/snapshot/tree.rs)
Replace `filesystem_name_for()` with `ref_segment_for()` in the segment matching loop. The fallback to instance name matching remains.
### 2c. Update `ref_target_path_from_tree()` in [rojo_ref.rs](src/rojo_ref.rs)
Use `tree.ref_segment_for()` instead of `tree.filesystem_name_for()` when building path segments.
### 2d. Update `tentative_fs_name()` in [ref_properties.rs](src/syncback/ref_properties.rs)
Return just the bare slug (slugified instance name, no extension). This produces tentative ref-path segments like `Foo` instead of `Foo.server.luau`.
### 2e. Update `added_instance_fs_segment()` in [api.rs](src/web/api.rs)
Return bare slug (slugified instance name, no extension) for same-batch ref target paths.
### 2f. Update `record_ref_path()` in [snapshot.rs](src/syncback/snapshot.rs)
Currently passes the full filename (e.g., `"Foo~2.server.luau"`) as the child segment. Must pass the bare stem instead (e.g., `"Foo~2"`). The `name` variable from `name_for_inst()` is the full filename, so we need to derive the bare stem -- use `strip_known_extensions(&name)`.
## Phase 2.5: Forward Sync Constraint Enforcement
### 2.5a. Update `walk_segments()` with 3-tier fallback in [tree.rs](src/snapshot/tree.rs)
The current 2-tier fallback (filesystem_name_for -> instance name) becomes 3-tier:
```rust
// Tier 1: match ref segment (bare stem) -- new format
for &child_ref in children {
let ref_seg = self.ref_segment_for(child_ref);
if ref_seg.eq_ignore_ascii_case(segment) { ... break; }
}
// Tier 2: match full filesystem name -- backward compat for old refs
if !found {
for &child_ref in children {
let fs_name = self.filesystem_name_for(child_ref);
if fs_name.eq_ignore_ascii_case(segment) { ... break; }
}
}
// Tier 3: match instance name -- existing fallback
if !found {
for &child_ref in children {
if child.name.eq_ignore_ascii_case(segment) { ... break; }
}
}
```
Same structure for `get_instance_by_path()`.
### 2.5b. Add bare-stem collision warning in forward sync
In [dir.rs](src/snapshot_middleware/dir.rs) `snapshot_dir_no_meta()`, after collecting all child snapshots, check for bare-stem collisions among siblings. If found, log a warning like:
```
Warning: Files "Foo.server.luau" and "Foo.luau" share bare stem "Foo" --
this is ambiguous for ref paths. Run syncback to fix.
```
This is a non-fatal warning. The forward sync still works (instances get their correct names/classes), but ref resolution may be ambiguous until syncback adds dedup suffixes.
## Phase 3: Dedup Cleanup Update
### 3a. Update cleanup grouping in [change_processor.rs](src/change_processor.rs) (lines 1090-1156)
Currently groups siblings by matching extension (`sibling_ext != removed_extension`). Remove this extension filter -- all siblings with the same bare stem should be in the same dedup group regardless of extension.
- `removed_stem` derivation: use `strip_known_extensions(removed_fs_name)` instead of splitting on first `.`
- `sibling_stem` derivation: use `strip_known_extensions(sibling_fs)` instead of splitting on first `.`
- Remove the `sibling_ext != removed_extension` guard entirely
- `compute_cleanup_action()` `extension` parameter: still needed for building filesystem paths. Pass `None` for the extension since cleanup now operates on bare stems. The renamed file keeps its original extension -- we just need to rename `Foo~2.server.luau` to `Foo.server.luau`. The extension is derived from the actual file's path, not from the dedup group.
Wait -- `compute_cleanup_action()` builds rename paths using `build_dedup_name(base_stem, suffix, extension)`. It needs the extension to know the full filename. Since siblings now have DIFFERENT extensions within the same group, the survivor's extension must be read from its actual filesystem name. This requires a small refactor of `compute_cleanup_action()` or the caller: instead of passing a single `extension`, the cleanup should track each remaining sibling's full filename so renames produce correct paths.
**Approach:** In the caller loop in `change_processor.rs`, track `remaining_entries: Vec<(stem, full_fs_name)>` instead of just `remaining_stems: Vec<String>`. The cleanup action can then derive the correct filesystem path from the survivor's actual full name.
### 3b. Update `compute_cleanup_action()` in [dedup_suffix.rs](src/syncback/dedup_suffix.rs)
Modify to accept `remaining_entries` with both stem and full filename info, so it can build correct rename paths for mixed-extension groups.
## Phase 4: Tests
### 4a. Update unit tests in [file_names.rs](src/syncback/file_names.rs)
**All tests using full filesystem names in `taken_names`** must switch to bare slugs:
- `nfi_dedup_collision`: `"foo.luau"` -> `"foo"`
- `nfi_slug_collision_file_middleware`: `"hey_bro.luau"` -> `"hey_bro"`
- `disk_seed_file_middleware_*`: entries like `"mymodule.luau"` -> `"mymodule"`
- `nightmare_mixed_middleware_*`: These tests will fundamentally change. `Folder "Shared"` and `ModuleScript "Shared"` now DO collide (same bare slug "shared") where they previously didn't.
**New tests to add:**
- `same_name_different_type_dedup`: Two instances "Foo" (ModuleScript + ServerScript) get dedup: `Foo.luau`, `Foo~2.server.luau`
- `same_name_dir_and_file_dedup`: Folder "Foo" and Script "Foo" get dedup: `Foo/`, `Foo~2.server.luau`
- `three_way_mixed_type_dedup`: Three instances "X" (ModuleScript, ServerScript, Folder) all dedup
- `dedup_key_is_bare_slug`: Verify dedup_key returned by `name_for_inst()` is the bare slug
### 4b. Add `strip_known_extensions` unit tests in [file_names.rs](src/syncback/file_names.rs)
Test all compound and simple extension stripping, including edge cases (dots in instance names, no extension, etc.).
### 4c. Update ref path unit tests in [rojo_ref.rs](src/rojo_ref.rs)
Update expected ref path formats: `"Workspace/Foo.luau"` -> `"Workspace/Foo"`, etc.
### 4d. Add build test fixture
New test in [build-tests/](rojo-test/build-tests/) for same-name-different-type instances. Verify the build output snapshot shows correct dedup suffixes.
### 4e. Add connected_mode test
Test dedup cleanup when a mixed-type group member is deleted. Verify the survivor's suffix is cleaned correctly even though siblings have different extensions.
### 4f. Update ALL existing snapshot test expectations
Run `cargo test` and `cargo insta review` to update snapshot files affected by the new ref path format and dedup behavior.
## Key Files Summary
| File | Change Type |
| ------------------------------------------------------------------------ | ------------------------------------------------------------------------- |
| [src/syncback/file_names.rs](src/syncback/file_names.rs) | Add `strip_known_extensions`, modify dedup logic, update tests |
| [src/snapshot/tree.rs](src/snapshot/tree.rs) | Add `ref_segment_for`, update `walk_segments`, `get_instance_by_path` |
| [src/rojo_ref.rs](src/rojo_ref.rs) | Update `ref_target_path_from_tree`, update tests |
| [src/syncback/ref_properties.rs](src/syncback/ref_properties.rs) | Update `tentative_fs_name` to return bare slug |
| [src/web/api.rs](src/web/api.rs) | Update `added_instance_fs_segment`, taken_names seeding |
| [src/change_processor.rs](src/change_processor.rs) | Remove extension-based grouping, update ref paths |
| [src/syncback/dedup_suffix.rs](src/syncback/dedup_suffix.rs) | Update `compute_cleanup_action` for mixed-extension groups |
| [src/syncback/snapshot.rs](src/syncback/snapshot.rs) | Update `record_ref_path` to use bare stems |
| [src/snapshot_middleware/dir.rs](src/snapshot_middleware/dir.rs) | Update taken_names seeding to bare slugs; add bare-stem collision warning |
| [src/snapshot_middleware/project.rs](src/snapshot_middleware/project.rs) | Update taken_names seeding to bare slugs |
## Risk Areas
1. `**compute_cleanup_action` refactor** -- The most complex change. Currently assumes all group members share an extension. Must now handle mixed extensions and still produce correct filesystem rename paths.
2. **Custom sync rules** -- `strip_known_extensions()` only handles built-in extensions. Custom `syncRules` patterns won't be stripped. Acceptable for initial implementation; the ref segment will include the custom extension, and resolution will match correctly since both sides use the same derivation.
3. **Snapshot test churn** -- Many snapshot tests will have new expected output. Use `cargo insta review` to batch-accept.
4. **Legacy project migration** -- Projects with same-bare-stem files (e.g., `Foo.server.luau` + `Foo.luau` in same dir) are now in an invalid state. The 3-tier `walk_segments` fallback ensures old ref paths still resolve, but the ambiguity warning encourages users to re-run syncback. The first syncback after this change will automatically fix the filesystem by applying dedup suffixes.

68
.cursor/plans/fix_infinity_comparison_71fd9057.plan.md Normal file
View file

@ -0,0 +1,68 @@
---
name: Fix infinity comparison
overview: Make `fuzzyEq` self-contained for all IEEE 754 special values (infinity and NaN), eliminating phantom diffs on properties like `UISizeConstraint.MaxSize` and fixing NaN component comparison in compound types.
todos:
- id: fix-lua-fuzzyeq
content: Add exact-equality and NaN checks to `fuzzyEq` in trueEquals.lua
status: completed
- id: add-tests
content: Add infinity and NaN test cases to trueEquals.spec.lua
status: completed
isProject: false
---
# Fix Infinity and NaN Comparison in fuzzyEq
## Root Cause
Two bugs in `fuzzyEq`:
**1. Infinity:** `fuzzyEq(inf, inf)` returns `false` because `inf - inf = NaN`, and `NaN < x` is always `false`. Affects `UISizeConstraint.MaxSize` which defaults to `Vector2(inf, inf)`.
**2. NaN in compound types:** NaN equality is handled in the `trueEquals` number branch (line 93), but `fuzzyEq` is called directly from `vector2Eq`, `vector3Eq`, `cframeEq`, etc. A Vector2/Vector3/CFrame with a NaN component silently returns `false` instead of `true` because `fuzzyEq` doesn't check for NaN.
## Fix
Make `fuzzyEq` self-contained by adding exact-equality and NaN checks at the top:
```lua
local function fuzzyEq(a: number, b: number): boolean
if a == b then
return true
end
if a ~= a then
return b ~= b
end
local diff = math.abs(a - b)
local maxVal = math.max(math.abs(a), math.abs(b), 1)
return diff < EPSILON or diff < maxVal * EPSILON
end
```
Behavior:
- `inf == inf` -> `true` (caught by `a == b`)
- `-inf == -inf` -> `true` (caught by `a == b`)
- `inf == -inf` -> `false` (falls through, `inf - (-inf) = inf`, fails comparison)
- `NaN == NaN` -> `true` (`a == b` is false for NaN, then `a ~= a` catches it)
- `NaN == 5` -> `false` (`a ~= a` is true but `b ~= b` is false)
Remove the now-redundant NaN check in `trueEquals` line 93 (the number branch `if a ~= a then return b ~= b end`) since `fuzzyEq` handles it directly.
## Files
### 1. [plugin/src/Reconciler/trueEquals.lua](plugin/src/Reconciler/trueEquals.lua) (line 15)
Replace `fuzzyEq` with the version above.
### 2. [plugin/src/Reconciler/trueEquals.spec.lua](plugin/src/Reconciler/trueEquals.spec.lua)
Add test cases for:
- `trueEquals(math.huge, math.huge)` -> true
- `trueEquals(-math.huge, -math.huge)` -> true
- `trueEquals(math.huge, -math.huge)` -> false
- `trueEquals(Vector2.new(math.huge, math.huge), Vector2.new(math.huge, math.huge))` -> true
- `trueEquals(Vector3.new(0/0, 1, 2), Vector3.new(0/0, 1, 2))` -> true (NaN component)
- `trueEquals(Vector2.new(0/0, 0/0), Vector2.new(0/0, 0/0))` -> true (NaN components)

235
.cursor/plans/fix_live_syncback_service_properties_d7c2c927.plan.md Normal file
View file

@ -0,0 +1,235 @@
---
name: Fix Live Syncback Service Properties
overview: Fix live syncback to preserve service-level properties and respect syncCurrentCamera by encoding service properties/attributes/tags/refs alongside the existing rbxm children data.
todos:
- id: plugin-encode-service
content: Create encodeService helper in plugin that returns className, data, properties, attributes, tags, refs
status: completed
- id: plugin-use-helper
content: Update performSyncback to use encodeService, remove children-only guard
status: completed
- id: server-service-chunk
content: Extend ServiceChunk struct with properties, attributes, tags, refs fields
status: completed
- id: server-apply-props
content: Update build_dom_from_chunks to apply service properties/attributes/tags and resolve refs
status: completed
- id: test-helper
content: Update make_service_chunk helpers and make_rbxl_from_chunks
status: completed
- id: fixture-update
content: Update live_syncback fixture project with service $path entries
status: completed
- id: roundtrip-update
content: Update roundtrip test to include service properties
status: completed
- id: new-tests
content: Add parity tests for service properties, childless services, and Camera removal
status: completed
isProject: false
---
# Fix Live Syncback Service Properties
## Root Cause
1. **Service properties lost** -- Plugin serializes `service:GetChildren()` only. Service-level properties never sent.
2. **Childless services skipped** -- `#service:GetChildren() > 0` guard skips services like VoiceChatService.
3. **Camera not removed** -- `syncback_loop` reads `Workspace.CurrentCamera` to find/remove Camera, but Workspace has no properties in the DOM.
## Constraint
`SerializationService:SerializeInstancesAsync` throws if services are in the top-level list. Children-only rbxm stays; service metadata must be encoded separately.
## Changes
### 1. Plugin -- New `encodeService` helper
Create `[plugin/src/ChangeBatcher/encodeService.lua](plugin/src/ChangeBatcher/encodeService.lua)`. Returns a full `ServiceChunk` table:
```lua
-- Returns:
-- {
-- className = "Lighting",
-- data = buffer, -- rbxm of children (empty buffer if no children)
-- properties = {}, -- non-Ref encoded properties (via RbxDom)
-- attributes = {}, -- encoded attributes map
-- tags = {}, -- tags array
-- refs = {}, -- Ref properties: { PropName = { name, className } }
-- }
```
Implementation reuses existing `encodeProperty`, `RbxDom.findClassDescriptor`, `RbxDom.findCanonicalPropertyDescriptor`, and the `UNENCODABLE_DATA_TYPES` filter from `[plugin/src/ChangeBatcher/propertyFilter.lua](plugin/src/ChangeBatcher/propertyFilter.lua)`.
**Property encoding** -- same loop as `[encodeInstance.lua](plugin/src/ChangeBatcher/encodeInstance.lua)` lines 200-249 but without script-specific skips:
```lua
for propertyName, propertyMeta in classDescriptor.properties do
-- skip Parent, Name, Archivable, Attributes, Tags
if isReadable and doesSerialize then
if descriptor.dataType == "Ref" then
-- Read ref target; if it's a direct child, record name+className
local ok, target = descriptor:read(service)
if ok and target and target.Parent == service then
refs[propertyName] = { name = target.Name, className = target.ClassName }
end
elseif not UNENCODABLE_DATA_TYPES[descriptor.dataType] then
local ok, encoded = encodeProperty(service, propertyName, descriptor)
if ok and encoded ~= nil then
properties[propertyName] = encoded
end
end
end
end
```
**Attributes** -- encoded via the Attributes property descriptor (same as `encodeAttributes` in `encodeInstance.lua` lines 121-137), but returned as a separate field.
**Tags** -- read via `service:GetTags()`, returned as a plain array.
**Refs** -- Ref properties where the target is a direct child of the service. Encoded as `{ name = target.Name, className = target.ClassName }` so the server can resolve them by finding the matching child after cloning from rbxm.
**Children rbxm** -- `SerializationService:SerializeInstancesAsync(service:GetChildren())` if children exist, otherwise empty buffer.
### 2. Plugin -- Update `performSyncback` in `[plugin/src/App/init.lua](plugin/src/App/init.lua)` (lines 641-651)
```lua
local encodeService = require(Plugin.ChangeBatcher.encodeService)
local services = {}
for _, className in SYNCBACK_SERVICES do
local ok, service = pcall(game.FindService, game, className)
if ok and service then
table.insert(services, encodeService(service))
end
end
```
No children guard. Every found service is encoded.
### 3. Server -- Extend `ServiceChunk` in `[src/web/interface.rs](src/web/interface.rs)` (lines 30-37)
```rust
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct ServiceChunk {
pub class_name: String,
#[serde(with = "serde_bytes")]
pub data: Vec<u8>,
#[serde(default)]
pub properties: HashMap<String, Variant>,
#[serde(default)]
pub attributes: HashMap<String, Variant>,
#[serde(default)]
pub tags: Vec<String>,
#[serde(default)]
pub refs: HashMap<String, ServiceRef>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ServiceRef {
pub name: String,
pub class_name: String,
}
```
All new fields use `#[serde(default)]` for backwards compatibility.
### 4. Server -- Update `build_dom_from_chunks` in `[src/cli/serve.rs](src/cli/serve.rs)` (lines 140-179)
After creating the service instance, apply properties/attributes/tags:
```rust
for chunk in &payload.services {
let mut builder = InstanceBuilder::new(&chunk.class_name);
// Apply encoded properties
for (key, value) in &chunk.properties {
builder = builder.with_property(key.as_str(), value.clone());
}
// Apply attributes
if !chunk.attributes.is_empty() {
let attrs = rbx_dom_weak::types::Attributes::new();
// ... convert HashMap<String, Variant> to Attributes ...
builder = builder.with_property("Attributes", Variant::Attributes(attrs));
}
// Apply tags
if !chunk.tags.is_empty() {
builder = builder.with_property("Tags", Variant::Tags(chunk.tags.clone().into()));
}
let service_ref = dom.insert(root_ref, builder);
created_services.insert(chunk.class_name.clone());
// Clone children from rbxm
if !chunk.data.is_empty() {
let chunk_dom = rbx_binary::from_reader(Cursor::new(&chunk.data))?;
for &child_ref in chunk_dom.root().children() {
deep_clone_into(&chunk_dom, &mut dom, child_ref, service_ref, &mut global_ref_map);
}
}
// Resolve Ref properties by finding matching children
if !chunk.refs.is_empty() {
let children: Vec<Ref> = dom.get_by_ref(service_ref).unwrap().children().to_vec();
for (prop_name, target) in &chunk.refs {
let found = children.iter().find(|&&child_ref| {
let child = dom.get_by_ref(child_ref).unwrap();
child.name.as_str() == target.name && child.class.as_str() == target.class_name
});
if let Some(&child_ref) = found {
let service = dom.get_by_ref_mut(service_ref).unwrap();
service.properties.insert(prop_name.as_str().into(), Variant::Ref(child_ref));
}
}
}
}
```
This handles `Workspace.CurrentCamera` naturally -- the Camera is cloned from the rbxm, then the Ref is resolved by matching `name="Camera", className="Camera"` among children. `syncback_loop` can then find it and destroy it when `syncCurrentCamera: false`.
### 5. Test helpers -- `[tests/rojo_test/serve_util.rs](tests/rojo_test/serve_util.rs)`
Update `make_service_chunk` to accept optional properties/attributes/tags/refs. Add companion:
```rust
pub fn make_service_chunk_full(
class_name: &str,
properties: Vec<(&str, Variant)>,
attributes: Vec<(&str, Variant)>,
tags: Vec<String>,
refs: Vec<(&str, &str, &str)>, // (prop_name, target_name, target_class)
children: Vec<InstanceBuilder>,
) -> ServiceChunk
```
Update `make_rbxl_from_chunks` to apply properties/attributes/tags/refs to the service in the DOM (mirrors server logic).
Update `roundtrip_build_syncback_rebuild` to extract service properties from the built rbxl and populate `ServiceChunk` fields.
### 6. Fixture -- `[rojo-test/serve-tests/live_syncback/default.project.json5](rojo-test/serve-tests/live_syncback/default.project.json5)`
Add service entries with `$path` for Lighting, SoundService, StarterPlayer, TextChatService, VoiceChatService. Create empty `src/` subdirectories.
### 7. New tests -- `[tests/tests/live_syncback.rs](tests/tests/live_syncback.rs)`
**Service property parity tests** (use `assert_live_matches_cli`):
- `**parity_lighting_properties`** -- Lighting with `Ambient`, `ClockTime`, `Brightness` + child
- `**parity_soundservice_properties`** -- SoundService with `AmbientReverb`, `DistanceFactor`
- `**parity_starterplayer_properties`** -- StarterPlayer with `CameraMaxZoomDistance`
- `**parity_textchatservice_properties**` -- TextChatService with `ChatVersion`
- `**parity_childless_service_with_properties**` -- Service with properties but zero children
- `**parity_service_properties_with_children**` -- Service with both properties AND children
- `**parity_multiple_services_with_properties**` -- Multiple services in one request
**Camera removal test:**
- `**parity_camera_not_synced`** -- Workspace with Camera child + `CurrentCamera` ref. Verify Camera does NOT appear on disk (default `syncCurrentCamera: false`).
**Round-trip test:**
- `**roundtrip_service_properties`** -- Build -> live syncback -> rebuild. Compare service properties survive the cycle.

104
.cursor/plans/fix_test_port_toctou_86726cb4.plan.md Normal file
View file

@ -0,0 +1,104 @@
---
name: Fix test port TOCTOU
overview: Two-layer fix for port-in-use race conditions. (1) Server-side -- retry bind with backoff when a known port is busy. (2) Test-side -- retry with a new random port when the serve process dies from AddrInUse.
todos:
- id: server-bind-retry
content: Add bind retry with backoff in `LiveServer::start()` for known ports
status: completed
- id: respawn-method
content: Add `respawn_with_new_port()` method to `TestServeSession`
status: completed
- id: wait-retry
content: Modify `wait_to_come_online()` to detect AddrInUse and retry via respawn
status: completed
- id: fresh-rebuild-retry
content: Add port-retry logic to `fresh_rebuild_read()`
status: completed
isProject: false
---
# Fix Port-in-Use Race Conditions
## Problem
Two failure modes:
1. **Tests (random ports):** `get_port_number()` in `[tests/rojo_test/serve_util.rs](tests/rojo_test/serve_util.rs)` binds port 0, reads the assigned port, drops the listener, then passes the port to the spawned serve process. Another test can grab the same port in that TOCTOU window.
2. **Server (known ports):** `TcpListener::bind(address).await.unwrap()` in `[src/web/mod.rs:80](src/web/mod.rs)` panics immediately if the port is in use, with no retry. A previous server instance shutting down or a transient conflict gives no grace period.
## Layer 1: Server-side bind retry (`src/web/mod.rs`)
Replace the `.unwrap()` on `TcpListener::bind()` at line 80 with a retry loop:
- On `AddrInUse`: retry up to 5 times with exponential backoff (200ms, 400ms, 800ms, 1600ms, 3200ms -- ~6s total)
- On any other bind error, or after retries exhausted: return `anyhow::Error` (propagates up cleanly instead of panicking)
```rust
let listener = {
const MAX_BIND_RETRIES: u32 = 5;
const BASE_BACKOFF_MS: u64 = 200;
let mut last_err = None;
for attempt in 0..MAX_BIND_RETRIES {
match TcpListener::bind(address).await {
Ok(listener) => { last_err = None; break; }
Err(err) if err.kind() == std::io::ErrorKind::AddrInUse => {
let delay = BASE_BACKOFF_MS * 2u64.pow(attempt);
log::warn!(
"Port {} in use, retrying in {}ms (attempt {}/{})",
address.port(), delay, attempt + 1, MAX_BIND_RETRIES
);
tokio::time::sleep(Duration::from_millis(delay)).await;
last_err = Some(err);
}
Err(err) => { last_err = Some(err); break; }
}
}
// unwrap or propagate
};
```
This helps both tests (brief TOCTOU window) and production (previous process releasing a port).
## Layer 2: Test-side retry with new random port (`tests/rojo_test/serve_util.rs`)
If the serve process still dies after the server-side retries (e.g., the port is held long-term by another test), the test infrastructure retries with a completely new port.
### 2a. Add `respawn_with_new_port()` to `TestServeSession`
Kills the current process, picks a new ephemeral port, and respawns:
```rust
fn respawn_with_new_port(&mut self) {
let port = get_port_number();
let port_string = port.to_string();
let working_dir = get_working_dir_path();
let rojo_process = Command::new(ROJO_PATH)
.args(["serve", self.project_path.to_str().unwrap(), "--port", &port_string])
.current_dir(working_dir)
.stderr(Stdio::piped())
.spawn()
.expect("Couldn't start Rojo");
self.rojo_process = KillOnDrop(rojo_process);
self.port = port;
}
```
### 2b. Modify `wait_to_come_online()`
In the `Ok(Some(status))` branch (line ~266-276), detect `AddrInUse` in stderr and retry:
- Add a `port_retries` counter (max 3 attempts)
- On `AddrInUse`: call `self.respawn_with_new_port()`, reset the backoff counter, continue
- On other errors: panic as before
### 2c. Apply same pattern to `fresh_rebuild_read()`
`fresh_rebuild_read()` (line ~929) has its own inline wait loop. Wrap the spawn + wait in a retry loop that detects `AddrInUse` and retries with a new port.
## Files changed
- `[src/web/mod.rs](src/web/mod.rs)` -- bind retry with backoff (Layer 1)
- `[tests/rojo_test/serve_util.rs](tests/rojo_test/serve_util.rs)` -- respawn with new port (Layer 2)

329
.cursor/plans/git-based_sync_defaults_be6371b6.plan.md Normal file
View file

@ -0,0 +1,329 @@
---
name: Git-Based Sync Defaults
overview: Add git-aware default sync direction selection to the two-way sync confirmation UI. Server sends changed-instance IDs + SHA1 hashes (both staged and HEAD) to the plugin for auto-selecting defaults. Accepted files are auto-staged via git add.
todos:
- id: sha1-dep
content: Add sha1 crate to Cargo.toml
status: completed
- id: sha1-luau
content: Add SHA1.luau to plugin/src/ from user-provided file
status: completed
- id: git-module
content: Create src/git.rs with git CLI helpers (repo root, changed files, committed content, SHA1 computation, git add)
status: completed
- id: interface-rs
content: Add GitMetadata struct and field to ServerInfoResponse, add stage_ids to WriteRequest and PatchSet
status: completed
- id: api-rs
content: Call compute_git_metadata in handle_api_rojo (two-phase lock), wire up project root, handle stage_ids in handle_api_write
status: completed
- id: patchtree-lua
content: Update PatchTree.build to accept gitMetadata and set defaultSelection based on git status + SHA1 hash comparison
status: completed
- id: app-init-lua
content: Pass serverInfo.gitMetadata to PatchTree.build call sites in App/init.lua, build stage_ids in confirmation flow
status: completed
- id: auto-stage
content: "Auto git-add: all push-accepted files + auto-selected pull-accepted files. Split staging between api.rs and change_processor."
status: completed
- id: test-verify
content: "Build and verify: cargo build, cargo test, cargo clippy"
status: completed
isProject: false
---
# Git-Based Sync Direction Defaults
## Summary
When the plugin shows the initial sync confirmation dialog, currently all items default to `nil` (unselected). This feature adds smart defaults based on git status:
- **File has no git changes** (same as HEAD): default to "Studio" (pull) -- any diff is because Studio was changed
- **File has git changes AND is a script AND Studio Source matches committed version**: default to "Atlas" (push) -- developer changed the file, Studio hasn't diverged from the last commit
- **Otherwise**: `nil` -- user must decide (both sides changed from committed, or non-script with changes)
- **No git repo**: all defaults remain `nil` (feature is no-op)
These defaults only apply to **Updated** items (where both Studio and filesystem have the instance). Added/Removed items remain `nil` to avoid destructive auto-selections.
**Hash comparison is scripts-only for now.** Non-script property/misc changes that are git-changed remain `nil` (deferred feature -- would require reproducing a bit-for-bit state on the Roblox side to compare).
Recomputed fresh on **each connect** (not cached).
## User Stories
1. **Scripts changed locally, hashes match Studio Source**: Developer changed scripts in git. Plugin receives committed SHA1 hashes. Studio's `script.Source` (non-draft version) matches the committed hashes. Result: **"Atlas" auto-selected** for those scripts.
2. **Scripts changed by someone else too**: Developer pulled commits that changed files, AND someone edited in Studio. Committed SHA1 won't match Studio Source. Result: **No bubble selected** -- user must decide.
3. **Property/misc changes in git**: Non-script files flagged as changed. Result: **No auto-selection** (deferred feature).
4. **No git changes on a file**: File is identical to HEAD. Any diff is because Studio was changed. Result: **"Studio" auto-selected**.
## Architecture
```mermaid
sequenceDiagram
participant Git as Git CLI
participant Server as Rust Server
participant Tree as RojoTree
participant Plugin as Lua Plugin
participant UI as PatchTree / UI
Note over Server: On each GET /api/rojo
Server->>Tree: snapshot path_to_ids + class names (brief lock)
Server->>Git: git rev-parse --show-toplevel
Git-->>Server: repo root (or fail = no git)
Server->>Git: git diff HEAD --name-only
Git-->>Server: list of changed files
Server->>Server: map paths to Refs using snapshot
loop For each changed script with committed version
Server->>Git: git show HEAD:path + git show :0:path
Git-->>Server: committed + staged content
Server->>Server: SHA1 in blob format (no normalization)
end
Server->>Plugin: ServerInfoResponse + gitMetadata
Note over Plugin: Building PatchTree
Plugin->>UI: For each Updated node
alt Not in changedIds
UI->>UI: defaultSelection = "pull"
else Script with committed hash
Plugin->>Plugin: SHA1 blob format of script.Source
alt Any hash matches
UI->>UI: defaultSelection = "push"
else No hash matches
UI->>UI: defaultSelection = nil
end
else Non-script or no committed hash
UI->>UI: defaultSelection = nil
end
```
## Server Side (Rust)
### 1. Add `sha1` crate dependency
In [Cargo.toml](Cargo.toml), add under `[dependencies]`:
```toml
sha1 = "0.10"
```
We compute hashes ourselves using the git blob format scheme (`"blob <size>\0<content>"`) so both sides produce identical hashes from identical content.
### 2. New module: `src/git.rs`
All git interaction via `std::process::Command`.
- `**git_repo_root(project_root: &Path) -> Option<PathBuf>**`: Runs `git -C <dir> rev-parse --show-toplevel`. Returns `None` if not a git repo or git not installed.
- `**git_changed_files(repo_root: &Path) -> Option<HashSet<PathBuf>>**`: Runs `git -C <dir> diff HEAD --name-only` to get files differing from HEAD (covers staged + unstaged). Also runs `git -C <dir> ls-files --others --exclude-standard` for untracked files. Returns paths relative to repo root. **Edge case**: if HEAD doesn't exist (no commits yet), treat all tracked files as changed or return None gracefully.
- `**git_show_head(repo_root: &Path, rel_path: &Path) -> Option<String>`**: Runs `git -C <dir> show HEAD:<path>`. Returns committed content at HEAD. Returns `None` for new files.
- `**git_show_staged(repo_root: &Path, rel_path: &Path) -> Option<String>`**: Runs `git -C <dir> show :0:<path>`. Returns staged (index) content. Returns `None` if not staged or file doesn't exist in index.
- `**compute_blob_sha1(content: &str) -> String`**: Computes `SHA1("blob <byte_len>\0<content>")`. Returns 40-char lowercase hex string. No normalization.
- `**git_add(repo_root: &Path, paths: &[PathBuf])`**: Runs `git -C <dir> add -- <paths...>`. Stages files for commit. No-op if paths is empty.
- `**compute_git_metadata(path_snapshot: &PathSnapshot, project_root: &Path) -> Option<GitMetadata>`**: Takes a pre-extracted `PathSnapshot` (see section 4). Orchestrates git commands:
1. Find repo root
2. Get changed files set
3. Map changed file paths to instance Refs using the path snapshot
4. For each changed script: collect up to 2 hashes -- HEAD version and staged version (skip staged if identical to HEAD)
5. Returns `None` if no git repo
### 3. Update data structures in [src/web/interface.rs](src/web/interface.rs)
**GitMetadata** (new struct for ServerInfoResponse):
```rust
#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct GitMetadata {
/// Instance IDs whose backing files have changes relative to HEAD
pub changed_ids: Vec<Ref>,
/// For changed script instances: SHA1 hashes of prior versions (git blob format).
/// Contains 1-2 hashes per script: HEAD version and staged version (if different).
/// Plugin checks if Studio Source matches ANY hash to default "Atlas".
pub script_committed_hashes: HashMap<Ref, Vec<String>>,
}
```
Add to `ServerInfoResponse`:
```rust
#[serde(default, skip_serializing_if = "Option::is_none")]
pub git_metadata: Option<GitMetadata>,
```
**WriteRequest** (add `stage_ids` field):
```rust
pub struct WriteRequest {
pub session_id: SessionId,
pub removed: Vec<Ref>,
#[serde(default)]
pub added: HashMap<Ref, AddedInstance>,
pub updated: Vec<InstanceUpdate>,
/// Instance IDs whose backing files should be staged via git add.
/// Includes: all push-accepted IDs + auto-selected pull-accepted IDs.
#[serde(default)]
pub stage_ids: Vec<Ref>,
}
```
**PatchSet** (add `stage_ids` field for change_processor):
Add to the existing PatchSet struct in [src/snapshot/patch.rs](src/snapshot/patch.rs):
```rust
/// Instance IDs whose backing files should be staged after writing.
/// Populated from WriteRequest.stage_ids, consumed by change_processor.
#[serde(default, skip)]
pub stage_ids: HashSet<Ref>,
```
### 4. Update `handle_api_rojo()` in [src/web/api.rs](src/web/api.rs) -- two-phase lock
**Critical**: Do NOT hold the tree lock while running git subprocesses.
**Phase 1 (with lock)**: Snapshot the data needed from the tree:
- `path_to_ids` entries (clone the relevant mappings)
- Instance class names for each Ref
- `instigating_source` paths
**Phase 2 (no lock)**: Run git commands and compute hashes using the snapshot.
```rust
// Phase 1: brief lock
let path_snapshot = {
let tree = self.serve_session.tree();
extract_path_snapshot(&tree) // Clone path_to_ids, class names, instigating sources
};
// Lock released here
// Phase 2: slow git work, no lock held
let git_metadata = compute_git_metadata(&path_snapshot, project_root);
```
### 5. Wire up in [src/serve_session.rs](src/serve_session.rs)
Expose `project_root()` from `ServeSession` (the project file's parent directory). Compute and store `git_repo_root` as `Option<PathBuf>` at session start for use by auto-staging in `handle_api_write` and change_processor.
### 5b. Auto-stage accepted files
After the user confirms sync selections, automatically `git add` affected files based on these rules:
- **Push (Atlas) accepted**: Always stage the backing file. The filesystem changes are being finalized into Studio, so they're ready to commit. Applies to ALL push items (auto-selected and manually-selected), including non-scripts.
- **Pull (Studio) accepted**: Only stage if the item was auto-selected (had a non-nil `defaultSelection`). Manually-chosen pulls are left unstaged so the user can diff them.
**Staging is split between two components** to handle the async Source write timing:
1. `**handle_api_write()` in [src/web/api.rs](src/web/api.rs)**: Stages files that api.rs writes directly -- additions (new files), removals (deleted files), and push-accepted files (already on disk, just need staging). Resolves Ref IDs to filesystem paths via `instigating_source` in the tree. Calls `git_add` in one batch. Also copies `stage_ids` into the PatchSet before sending to change_processor.
2. `**handle_tree_event()` in [src/change_processor.rs](src/change_processor.rs)**: After writing Source to disk (line ~1807), checks if the instance's Ref is in `patch.stage_ids`. If so, calls `git_add` on the written path. The change_processor receives `repo_root: Option<PathBuf>` during construction (computed once at session start).
**Why split?** Source writes are async -- api.rs sends the PatchSet to change_processor via a channel and returns immediately. If api.rs staged those files, it would stage the OLD content before change_processor writes the new content. Staging must happen AFTER the write.
**Plugin side** ([plugin/src/ServeSession.lua](plugin/src/ServeSession.lua)): In `__confirmAndApplyInitialPatch`, build the `stage_ids` list:
- For each push-selected item: always include in `stage_ids`
- For each pull-selected item: include only if the PatchTree node had `defaultSelection ~= nil`
- Always send the write request if there are either pull changes OR stage IDs (currently only sent when pullPatch is non-empty -- expand the condition to also fire when `stage_ids` is non-empty)
If not in a git repo (`repo_root` is `None`), all stage calls are no-ops.
## Plugin Side (Lua)
### 6. Add SHA1.luau to plugin
Copy the user-provided [SHA1.luau](c:\Users\Joe\Downloads\SHA1.luau) to `plugin/src/SHA1.luau`. This is a native-optimized SHA1 implementation that takes a `buffer` and returns a 40-char hex string.
### 7. Update `PatchTree.build()` in [plugin/src/PatchTree.lua](plugin/src/PatchTree.lua)
Add a 4th parameter `gitMetadata` (the `gitMetadata` table from `serverInfo`, may be nil).
At the start of `build()`, convert `changedIds` array to a Set for O(1) lookup:
```lua
local changedIdSet = nil
if gitMetadata and gitMetadata.changedIds then
changedIdSet = {}
for _, id in gitMetadata.changedIds do
changedIdSet[id] = true
end
end
```
For **Updated** nodes (around line 308-318), replace `defaultSelection = nil` with:
```lua
local defaultSelection = nil
if changedIdSet then
if not changedIdSet[change.id] then
defaultSelection = "pull"
elseif instance:IsA("LuaSourceContainer")
and gitMetadata.scriptCommittedHashes then
local hashes = gitMetadata.scriptCommittedHashes[change.id]
if hashes then
local source = instance.Source
local gitBlob = "blob " .. tostring(#source) .. "\0" .. source
local studioHash = SHA1(buffer.fromstring(gitBlob))
for _, hash in hashes do
if studioHash == hash then
defaultSelection = "push"
break
end
end
end
end
end
```
**Key**: `script.Source` returns the non-draft (published) version when Script Drafts are enabled. Both sides use the git blob format `"blob <byte_length>\0<content>"` as SHA1 input. No normalization -- committed content is LF (repo standard), Studio Source is LF.
**Added** and **Removed** nodes: keep `defaultSelection = nil` (unchanged).
### 8. Update `PatchTree.build()` call sites in [plugin/src/App/init.lua](plugin/src/App/init.lua)
Must support both **"Initial"** and **"Always"** confirmation modes. In "Always" mode, every incoming WebSocket patch triggers the confirmation dialog via `__onWebSocketMessage` -> `__userConfirmCallback(instanceMap, patch, self.__serverInfo)`, so `serverInfo.gitMetadata` is available for ongoing patches too.
**Note**: Per-item push/pull/ignore selections for ongoing WebSocket patches (Always mode) are a pre-existing limitation (see `ServeSession.lua` line 311-314). The git defaults will be set in the PatchTree but the existing "Accept"/"Abort" handling for ongoing patches doesn't yet support them. This is not introduced by this feature.
Three call sites in `App:startSession()` and one approach to wire them:
**Cache `serverInfo` in a closure-scoped local** accessible to all callbacks:
```lua
local cachedServerInfo = nil
```
- **Line 867** (`setConfirmCallback`, initial + ongoing confirms): Has `serverInfo` directly. Cache it: `cachedServerInfo = serverInfo`. Pass `serverInfo.gitMetadata` to `PatchTree.build`. Works for both Initial and Always modes since `__onWebSocketMessage` passes `self.__serverInfo`.
- **Line 918** (`setPatchUpdateCallback`, patches merging during active confirmation): Doesn't receive `serverInfo`. Use cached: `cachedServerInfo and cachedServerInfo.gitMetadata`. This is safe because the confirm callback always fires first (sets `cachedServerInfo`) before any patch updates can arrive.
- **Line 659** (precommit hook, auto-accepted patches): For patches that bypass confirmation (e.g., "Never" mode), this builds a tree for the Connected page display. Use `cachedServerInfo and cachedServerInfo.gitMetadata`. Will be `nil` if no confirmation has happened yet (first connect stores it).
### 9. Import SHA1 in PatchTree.lua
```lua
local SHA1 = require(Plugin.SHA1)
```
## Key Design Decisions
- **Self-computed SHA1 with git blob format scheme**: Read content via `git show HEAD:<path>` and `git show :0:<path>` (staged), compute SHA1 using `"blob <size>\0<content>"`. Send both hashes (HEAD + staged if different) so the plugin can match against either version.
- **Two-phase tree lock**: Snapshot path mappings under brief lock, then release before running git subprocesses. Prevents blocking all API requests during I/O-bound git operations.
- **Auto-staging with asymmetric rules**: Push-accepted items are always staged (filesystem changes finalized). Pull-accepted items are only staged if auto-selected (manually-chosen pulls left unstaged for manual diff). Plugin sends `stage_ids` in the WriteRequest; server resolves to paths and runs `git add`.
- **Split staging responsibility**: api.rs stages additions, removals, and push files directly. change_processor stages Source writes after they complete (via `stage_ids` on PatchSet). This avoids staging old content before async writes finish.
- **Only Updated items get defaults**: Added (would destructively delete files if "pull") and Removed (IDs aren't in server tree) are left as `nil`
- **Scripts-only hash comparison**: Non-script property changes are deferred (see note below)
- **Git via subprocess**: No git2/libgit2 dependency, just `std::process::Command`. Graceful fallback if git not installed or not a repo. Per-file `git show` calls are acceptable for typical changed file counts; `git cat-file --batch` is a future optimization for large changesets.
- **Draft handling**: `script.Source` returns the non-draft version when Script Drafts are enabled, which is exactly what we compare against the committed version
- **Fresh computation per connect**: Not cached -- each `GET /api/rojo` recomputes git state
- **No normalization anywhere**: Committed content is LF (repo standard), Studio Source is LF, VFS normalizes reads. Both sides hash identical LF content.
- **Init-style scripts (init.luau) handled correctly**: `dir.rs` registers all init file paths (init.luau, init.server.luau, etc.) in `relevant_paths`, so `path_to_ids` maps `MyFolder/init.luau` to the correct Ref. `instigating_source` is the actual init file path (not the directory), so Source writes and git-add target the correct file. Git reports file-level changes (`MyFolder/init.luau`), matching the indexed path.
- **No-HEAD edge case**: If the repo has no commits (`git diff HEAD` fails), treat as "no git metadata" and return None. All defaults remain nil.
## Deferred: Non-Script (Attribute/Property/Tag) Hash Comparison
Hash comparison for non-script changes (properties, attributes, tags) is **not included in this feature** and would be significantly harder:
- Would require building a **deterministic payload** of the instance's properties/attributes/tags (possibly msgpack with guaranteed key ordering, including default values) to hash against the committed file content (meta.json5, model.json5, etc.)
- Even with a deterministic serialization, there is **no "draft" concept** for non-script properties in Studio. `script.Source` has the draft/non-draft distinction, but property values are always live. There's no way to get "the version before the user started editing" for properties.
- This makes the comparison fundamentally less useful: if a user drags a Part in Studio, the property is immediately the new value with no way to recover the "original" for comparison.
- For now, non-script changes that are git-changed simply get `defaultSelection = nil` (user must decide).

272
.cursor/plans/git_sync_defaults_audit_557237e0.plan.md Normal file
View file

@ -0,0 +1,272 @@
---
name: Git Sync Defaults Audit
overview: Audit of the git-based sync defaults feature spanning git.rs (server), PatchTree.lua (plugin default selection logic), SHA1.luau (plugin hashing), and the auto-staging pipeline.
todos: []
isProject: false
---
# Audit: Git-Based Sync Defaults
## Feature Summary
When the plugin connects during `atlas serve`, the server computes git metadata (which files changed, SHA1 hashes of committed/staged script content) and sends it to the plugin. The plugin uses this to auto-select sync directions in the confirmation dialog:
- **File unchanged in git** -> default "pull" (Studio changed it)
- **File changed + script + Studio Source matches committed hash** -> default "push" (developer changed file, Studio still has old version)
- **Otherwise** -> nil (user must decide)
After confirmation, accepted files are auto-staged via `git add`.
```mermaid
sequenceDiagram
participant Plugin
participant Server
participant Git
Plugin->>Server: GET /api/rojo
Server->>Git: git diff HEAD --name-only
Server->>Git: git ls-files --others
Note over Server: Brief tree lock: resolve paths to Refs
Server->>Git: git show HEAD:path (per changed script)
Server->>Git: git show :0:path (per changed script)
Note over Server: compute_blob_sha1 for each version
Server-->>Plugin: ServerInfoResponse with gitMetadata
Note over Plugin: PatchTree.build() computes defaultSelection per node
Note over Plugin: User reviews/overrides defaults
Plugin->>Server: POST /api/write with stageIds
Note over Server: api.rs resolves stageIds to paths
Note over Server: change_processor writes Source, appends paths
Server->>Git: git add (consolidated, single call)
```
## Files Audited
| File | Lines | Purpose |
| ---------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------ |
| `[src/git.rs](src/git.rs)` | 766 | Server-side git helpers: changed files, blob SHA1, auto-staging |
| `[plugin/src/PatchTree.lua](plugin/src/PatchTree.lua)` | 715 | Default selection logic, tree building |
| `[plugin/src/SHA1.luau](plugin/src/SHA1.luau)` | 112 | Native SHA1 implementation |
| `[src/web/interface.rs](src/web/interface.rs)` | ~20 lines | `GitMetadata` struct, `stage_ids` on `WriteRequest` |
| `[src/web/api.rs](src/web/api.rs)` | ~30 lines | `handle_api_rojo` metadata computation, `handle_api_write` staging |
| `[src/change_processor.rs](src/change_processor.rs)` | ~40 lines | Source write staging, rename path tracking |
| `[src/snapshot/patch.rs](src/snapshot/patch.rs)` | ~10 lines | `stage_ids`/`stage_paths` fields on `PatchSet` |
| `[plugin/src/App/init.lua](plugin/src/App/init.lua)` | ~30 lines | `autoSelectedIds` construction, `cachedServerInfo` |
| `[plugin/src/ServeSession.lua](plugin/src/ServeSession.lua)` | ~80 lines | `stageIds` construction per selection type |
| `[plugin/src/App/StatusPages/Confirming.lua](plugin/src/App/StatusPages/Confirming.lua)` | 459 | Selection state management, `didUpdate` merge logic |
| `[tests/tests/git_sync_defaults.rs](tests/tests/git_sync_defaults.rs)` | 952 | Integration tests |
| `[plugin/src/PatchTree.spec.lua](plugin/src/PatchTree.spec.lua)` | ~200 lines | Plugin unit tests for git defaults |
---
## Verdict: No Critical Issues Found
The feature is well-designed and correctly implemented. The round-trip identity invariant is not directly affected by this feature -- git metadata is read-only advisory data that influences UI defaults, not filesystem writes. The auto-staging is purely additive (`git add`) and cannot corrupt the filesystem.
---
## Findings
### Finding 1: Performance -- O(N) Subprocess Spawning [Medium]
`compute_git_metadata` spawns **2 + 2N** git subprocesses for N changed scripts:
```35:35:src/git.rs
// 1 call
```
- `git diff HEAD --name-only` (1 process)
- `git ls-files --others --exclude-standard` (1 process)
- Per changed script: `git show HEAD:<path>` (N processes)
- Per changed script: `git show :0:<path>` (N processes)
On Windows, process creation is 10-100x slower than Linux. For 50 changed scripts, this is 102 subprocess spawns on every plugin connect.
**Recommendation**: Replace per-file `git show` calls with a single `git cat-file --batch` invocation that reads all file contents in one subprocess. This collapses 2N calls to 1. Not a correctness issue, but could cause noticeable connect latency on large repos with many changes. Defer unless users report slow connects.
### Finding 2: Windows Path Normalization Edge Case [Low]
Git commands return paths with `/` separators. `PathBuf::from("src/module.luau")` on Windows preserves the forward slashes. The tree lookup may fail if the tree stores paths with `\` separators.
```188:199:src/git.rs
for rel_path in &changed_files {
let abs_path = repo_root.join(rel_path);
let canonical_path = std::fs::canonicalize(&abs_path).ok();
let first = tree.get_ids_at_path(&abs_path);
let ids = if !first.is_empty() {
first
} else if let Some(ref canon) = canonical_path {
tree.get_ids_at_path(canon)
} else {
first
};
```
The `canonicalize` fallback mitigates this, but `canonicalize` can fail (file deleted, permissions). When it fails, the instance won't appear in `changedIds` -- it degrades to "no default" which is safe.
**Recommendation**: No action needed. The fallback is sufficient and the failure mode is safe (user decides manually).
### Finding 3: `refresh_git_index` Duplicates Repo Detection [Low -- Code Quality]
`[src/git.rs:252-260](src/git.rs)` walks up the directory tree to find `.git`, but the repo root is already cached in `ServeSession.git_repo_root`.
```251:260:src/git.rs
pub fn refresh_git_index(project_dir: &Path) {
let mut check_dir = Some(project_dir);
let mut is_git_repo = false;
while let Some(dir) = check_dir {
if dir.join(".git").exists() {
is_git_repo = true;
break;
}
check_dir = dir.parent();
}
```
Both call sites (`[src/cli/serve.rs:133](src/cli/serve.rs)` and `[src/cli/syncback.rs:282](src/cli/syncback.rs)`) could pass the known repo root instead. However, `syncback.rs` doesn't have a `ServeSession` -- it runs standalone. The current approach is correct for both call sites.
**Recommendation**: Accept as-is. The directory walk is negligible cost and keeps the function self-contained for both serve and CLI syncback contexts.
### Finding 4: Hash Parity Verified [No Issue]
The critical correctness requirement is that the Rust server and Luau plugin produce identical SHA1 hashes for the same content. Both implementations use the git blob format `"blob <byte_length>\0<content>"`:
**Rust** (`[src/git.rs:118-124](src/git.rs)`):
```118:124:src/git.rs
pub fn compute_blob_sha1(content: &str) -> String {
let header = format!("blob {}\0", content.len());
let mut hasher = Sha1::new();
hasher.update(header.as_bytes());
hasher.update(content.as_bytes());
format!("{:x}", hasher.finalize())
}
```
**Luau** (`[plugin/src/PatchTree.lua:363-364](plugin/src/PatchTree.lua)`):
```363:364:plugin/src/PatchTree.lua
local gitBlob = "blob " .. tostring(#source) .. "\0" .. source
local studioHash = SHA1(buffer.fromstring(gitBlob))
```
- Rust `content.len()` = byte length. Luau `#source` = byte length. Consistent.
- Both use UTF-8 encoding. `\0` is the null byte in both languages.
- The `sha1` crate (Rust) and `SHA1.luau` (native Luau implementation with `--!native` + `--!optimize 2`) are standard SHA1.
- Integration tests in `[tests/tests/git_sync_defaults.rs](tests/tests/git_sync_defaults.rs)` verify against real `git hash-object` output.
### Finding 5: Draft Source Fallback is a Nice Touch [No Issue]
```372:384:plugin/src/PatchTree.lua
if defaultSelection == nil then
local ok, draft = pcall(ScriptEditorService.GetEditorSource, ScriptEditorService, instance)
if ok and draft ~= source then
local draftBlob = "blob " .. tostring(#draft) .. "\0" .. draft
local draftHash = SHA1(buffer.fromstring(draftBlob))
for _, hash in hashes do
if draftHash == hash then
defaultSelection = "push"
break
end
end
end
end
```
When Script Drafts is enabled, `instance.Source` may not reflect the user's current editor content until they "commit" the draft. This fallback checks the draft source against committed hashes, ensuring the default selection is still accurate even with Script Drafts.
### Finding 6: Consolidated git_add Prevents index.lock Races [No Issue]
The staging pipeline correctly consolidates all paths into a single `git add` call:
```1875:1880:src/change_processor.rs
// Consolidated git staging: one git_add call for all paths
// ...
if !pending_stage_paths.is_empty() {
if let Some(ref repo_root) = self.git_repo_root {
crate::git::git_add(repo_root, &pending_stage_paths);
}
}
```
Pre-resolved paths from `api.rs` are merged with Source-write paths from `change_processor.rs`. Rename handling correctly updates `pending_stage_paths` (retains old path, pushes new path) at lines 1363-1368, 1531-1534, and 1713-1717. The batch staging test (`batch_stage_20_files_no_index_lock_race`) confirms this works for 20 files.
### Finding 7: Stale Git Metadata After Connect [Low -- By Design]
Git metadata is computed once per `GET /api/rojo` call (plugin connect/reconnect). If git state changes while the user reviews the confirmation dialog, the defaults may be stale. This is acceptable because:
1. The defaults are advisory -- the user can always override
2. Recomputing during the dialog would require a round-trip and could confuse the user if defaults change mid-review
3. Reconnecting (which triggers a fresh `GET /api/rojo`) gets fresh metadata
### Finding 8: Selection State Management in Confirming.lua is Correct [No Issue]
The `didUpdate` logic in `[Confirming.lua:107-132](plugin/src/App/StatusPages/Confirming.lua)` correctly handles streaming patch updates during confirmation:
1. Preserves existing selections for unchanged items
2. Unselects items in `changedIds` (forces re-review)
3. Merges new default selections for newly added items
4. Only overwrites if the item doesn't already have a selection
This prevents the confusing UX of defaults overriding conscious user choices during streaming updates.
### Finding 9: `autoSelectedIds` Staging Logic is Sound [No Issue]
The staging rules in `[ServeSession.lua:719-886](plugin/src/ServeSession.lua)`:
- **Push-accepted**: Always staged (lines 732, 792, 837) -- the user accepted the filesystem version
- **Pull-accepted + auto-selected**: Staged (lines 759-761, 806-808) -- high confidence automatic selection
- **Pull-accepted + manually selected**: NOT staged -- deliberate design choice: manually chosen pulls may need review before staging
- **Ignored**: Never staged
This is a thoughtful design that avoids staging files the user chose manually (they might want to `git diff` first).
---
## Test Coverage Assessment
**Excellent coverage across all three layers:**
| Layer | File | Tests | Coverage |
| ------------------ | ---------------------------------- | --------- | -------------------------------------------------------- |
| Unit (Rust) | `src/git.rs` | 20+ tests | SHA1 parity, changed files, staging, edge cases |
| Integration (Rust) | `tests/tests/git_sync_defaults.rs` | 20 tests | Full pipeline: metadata, staging, renames, batch 20-file |
| Unit (Luau) | `plugin/src/PatchTree.spec.lua` | ~15 tests | Default selections, hash matching, edge cases |
**Notable test scenarios covered:**
- No git repo (feature disabled)
- No commits yet (empty HEAD)
- Untracked, staged, unstaged, deleted files
- Init-style scripts (`init.luau` in directories)
- All script types (Module, Server, Client)
- Non-script changes (model files)
- Batch staging of 20 files (index.lock race prevention)
- Source write + staging interaction
- Rename + staging (new path staged, not old)
- Reconnect gets fresh metadata
- Draft source fallback
- Mixed push/pull batch operations
---
## Summary
The git-based sync defaults feature is **well-architected and correctly implemented**. Key design strengths:
1. **Three-phase tree lock** in `compute_git_metadata` minimizes lock contention
2. **Consolidated `git add`** prevents index.lock races
3. **Safe degradation** at every level (no git = no defaults, hash mismatch = user decides)
4. **Draft source fallback** handles Script Drafts edge case
5. **Thorough test coverage** across all three layers
6. **Clear separation of concerns**: server computes data, plugin makes UI decisions, staging is consolidated
The only actionable recommendation is the subprocess spawning optimization (Finding 1), which is a performance concern, not a correctness issue. It should be deferred unless users report slow plugin connects on repos with many changed files.

373
.cursor/plans/live_syncback_audit_fixes_561d17c7.plan.md Normal file
View file

@ -0,0 +1,373 @@
---
name: Live Syncback Audit Fixes
overview: "Fix 6 issues found during the live syncback audit: resource leaks from mem::forget, crash-on-error instead of recovery, concurrent syncback races, silent malformed input corruption, misleading doc comments, and missing test coverage."
todos:
- id: fix1-mem-forget
content: "Fix 1: Replace `mem::forget` with `drop()` in serve.rs (lines 61, 135)"
status: completed
- id: fix2-error-recovery
content: "Fix 2: Catch syncback errors and log+recover instead of crashing (serve.rs lines 84-88)"
status: completed
- id: fix3-concurrent-guard
content: "Fix 3: Reject concurrent syncback requests (SyncbackSignal::fire returns bool, api.rs returns 409)"
status: completed
- id: fix4-validate-counts
content: "Fix 4: Validate chunk counts match rbxm blob size (build_dom_from_chunks)"
status: completed
- id: fix5-doc-comment
content: "Fix 5: Correct misleading ServiceChunk doc comment (interface.rs lines 30-42)"
status: completed
- id: fix6-tests
content: "Fix 6: Add 5 new integration tests (attributes/tags, multiple refs, concurrent, malformed, properties-only)"
status: completed
- id: run-ci
content: Run CI after all fixes to verify everything passes
status: completed
isProject: false
---
# Live Syncback Audit Fixes
> This plan was generated by `/audit` from a Plan-mode session.
> Refer to `.cursor/rules/atlas.mdc` for full project standards.
## Standards
These standards govern every fix in this plan. An implementer MUST read `.cursor/rules/atlas.mdc` before starting.
### Round-Trip Identity (from `atlas.mdc` Quality Standard)
Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a **bit-identical instance tree** -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.
### CLI Syncback Parity (from `atlas.mdc` Two-Way Sync Strategy)
Plugin-based sync must produce **exactly the same filesystem output** that `atlas syncback` would give for the same input. Byte-for-byte identical files, identical directory structures, identical naming. Any divergence is a bug until proven otherwise.
### Code Quality (from `atlas.mdc` Code Quality Standard)
DRY code that is easy to reason about and hard to get wrong. The same slugify/dedup/meta-update pattern should not be copy-pasted across files. Small refactors (2-3 call sites) are included as fixes. Major rewrites (10+ call sites) are deferred.
## Context
Audited the "Live syncback" feature on the `ambiguous-paths` branch. This feature adds a one-click full sync from Studio to filesystem via `POST /api/syncback`. The plugin serializes all services (properties via msgpack, children via a single rbxm blob), sends them to the server, which reconstructs a `WeakDom`, runs `syncback_loop(incremental=false)`, and writes the result. The server shuts down and restarts after each syncback.
**Key files audited:**
- `src/cli/serve.rs` -- serve loop, `run_live_syncback()`, `build_dom_from_chunks()`
- `src/web/api.rs` -- `handle_api_syncback()` endpoint
- `src/web/mod.rs` -- `SyncbackSignal`, `LiveServer`
- `src/web/interface.rs` -- `ServiceChunk`, `SyncbackRequest`, `SyncbackPayload`
- `plugin/src/App/init.lua` -- `performSyncback()`, `SYNCBACK_SERVICES`
- `plugin/src/ChangeBatcher/encodeService.lua` -- service property/ref encoding
**No critical round-trip identity violations found.** The CLI parity tests (`assert_live_matches_cli`) provide strong guarantees. The round-trip test (`roundtrip_build_syncback_rebuild`) covers the full lifecycle. All 6 fixes below are correctness hardening, resilience, and test coverage.
## Fixes
Each fix below was approved by the user during the audit quiz. Implement them in order. Each fix includes its test requirements -- a fix is not complete until its tests pass.
---
### Fix 1: Replace `mem::forget` with proper `drop`
- **Status:** Approved
- **Finding:** Step 14 (Challenged Limitations) -- resource leak accumulating per syncback invocation
- **Files:** `[src/cli/serve.rs](src/cli/serve.rs)` (lines 61, 135)
- **Problem:** `forget(session_old)` at line 135 leaks the entire oneshot `ServeSession` (tree, VFS, project) after every live syncback. `forget(session)` at line 61 leaks the initial config-reading session at startup. The oneshot session has `change_processor: None`, `suppressed_paths: None`, `ref_path_index: None`, `git_repo_root: None` -- all `Option` types that drop cleanly as `None`. The `Vfs` destructor will stop its watcher thread. There is no reason to use `forget()`.
- **Solution:**
- Line 135: Replace `forget(session_old);` with `drop(session_old);`
- Line 61: Replace `forget(session);` with `drop(session);`
- Remove `use std::mem::forget;` from the imports (line 4) if no other uses remain
- **Risk:** Very low. The oneshot session's fields are all `Option::None` or standard types. The VFS watcher thread is dormant (no watches registered). Existing live syncback tests exercise this path.
- **Verify round-trip:** Run `parity_scripts` and `roundtrip_build_syncback_rebuild` -- both must pass.
- **Verify syncback parity:** Run `assert_live_matches_cli` tests -- filesystem output unchanged.
- **Tests required:**
- No new tests needed. Existing tests `syncback_twice_different_data` and `server_comes_back_functional` already exercise repeated syncback and verify the server recovers correctly.
---
### Fix 2: Recover from syncback errors instead of crashing
- **Status:** Approved
- **Finding:** Step 14 (Challenged Limitations) -- server crashes permanently on syncback failure
- **Files:** `[src/cli/serve.rs](src/cli/serve.rs)` (lines 84-88)
- **Problem:** `run_live_syncback(&project_path, payload)?` propagates errors via `?`, crashing the serve process. The plugin shows "Syncback data sent" but the server never comes back.
- **Solution:** Catch the error and log it, then continue to restart the server:
```rust
ServerExitReason::SyncbackRequested(payload) => {
log::info!("Live syncback requested, running...");
match run_live_syncback(&project_path, payload) {
Ok(()) => log::info!("Syncback complete, restarting serve..."),
Err(err) => log::error!("Live syncback failed: {err:#}. Restarting serve..."),
}
continue;
}
```
- **Risk:** Very low. The server restarts fresh regardless (new VFS, new session). An error just means the filesystem wasn't updated -- the old state persists.
- **Verify round-trip:** Existing tests still pass. The error path is tested by Fix 6 test 4.
- **Verify syncback parity:** N/A -- this fix doesn't change the syncback logic, only error handling.
- **Tests required:**
- **Rust integration test** (Fix 6, test 4): Send malformed chunk counts that trigger an error in `build_dom_from_chunks`. Verify the server recovers and comes back online.
---
### Fix 3: Reject concurrent syncback requests
- **Status:** Approved
- **Finding:** Step 14 (Challenged Limitations) -- second syncback silently overwrites first payload
- **Files:** `[src/web/mod.rs](src/web/mod.rs)` (`SyncbackSignal::fire`, lines 43-46), `[src/web/api.rs](src/web/api.rs)` (`handle_api_syncback`, line 259)
- **Problem:** `fire()` unconditionally overwrites any existing payload. If two requests arrive before the accept loop processes the first, the first payload is silently lost.
- **Solution:**
1. Change `SyncbackSignal::fire()` to return `bool`:
```rust
pub fn fire(&self, payload: SyncbackPayload) -> bool {
let mut guard = self.payload.lock().unwrap_or_else(|e| e.into_inner());
if guard.is_some() {
return false;
}
*guard = Some(payload);
self.notify.notify_one();
true
}
```
1. In `handle_api_syncback`, check the return value:
```rust
if !syncback_signal.fire(payload) {
return msgpack(
ErrorResponse::bad_request("A syncback is already in progress"),
StatusCode::CONFLICT,
);
}
```
- **Risk:** Very low. Happy path unchanged. Only adds rejection for the concurrent case.
- **Verify round-trip:** Existing tests pass (single syncback). New test verifies rejection.
- **Verify syncback parity:** N/A -- this fix doesn't change syncback output.
- **Tests required:**
- **Rust integration test** (Fix 6, test 3): Fire two syncback requests rapidly, verify the second returns 409 Conflict and the server still processes the first successfully.
---
### Fix 4: Validate chunk counts match rbxm blob size
- **Status:** Approved
- **Finding:** Step 11g (Audit Step 4, server-side processing) -- silent cursor clamping corrupts subsequent services
- **Files:** `[src/cli/serve.rs](src/cli/serve.rs)` (`build_dom_from_chunks`, lines 184-210)
- **Problem:** Line 187 `let end = (cursor + total).min(cloned_children.len());` silently clamps when counts exceed blob size. This corrupts cursor position for all subsequent services.
- **Solution:**
1. Inside the service loop, replace the `.min()` clamp with an early error:
```rust
let end = cursor + total;
anyhow::ensure!(
end <= cloned_children.len(),
"Service '{}' claims {} children + {} carriers but only {} instances remain in blob",
chunk.class_name, child_count, ref_count,
cloned_children.len().saturating_sub(cursor)
);
```
1. After the service loop, validate all instances were consumed:
```rust
anyhow::ensure!(
cursor == cloned_children.len(),
"Service chunks account for {} instances but rbxm blob contains {}",
cursor, cloned_children.len()
);
```
- **Risk:** Very low. Only triggers on malformed input. Normal requests are unaffected. With Fix 2, the error is caught and the server recovers.
- **Verify round-trip:** Existing tests pass (all have correct counts). New test verifies rejection.
- **Verify syncback parity:** N/A -- this fix doesn't change output for valid input.
- **Tests required:**
- **Rust integration test** (Fix 6, test 4): Send a request where service chunks claim more children than the blob contains. Verify the server logs an error and recovers.
---
### Fix 5: Correct misleading `ServiceChunk` doc comment
- **Status:** Approved
- **Finding:** Step 4 (server-side processing audit) -- doc describes behavior that doesn't exist
- **Files:** `[src/web/interface.rs](src/web/interface.rs)` (lines 30-42)
- **Problem:** Doc says "Indices <= child_count point directly to a child instance" but the implementation only indexes into the carrier subrange. This misleads future maintainers.
- **Solution:** Replace the doc comment block (lines 30-42) with:
```rust
/// Metadata for a single service in the live syncback request.
///
/// Children are NOT stored here -- they live in the top-level `data` blob
/// (a single rbxm containing all children from all services).
///
/// Each service's range in the blob is `child_count + ref_target_count`
/// consecutive entries. The first `child_count` are real children (parented
/// to the service). The next `ref_target_count` are ObjectValue carriers
/// whose `Value` property holds a Ref to the actual target.
///
/// `refs` maps property names to 1-based indices into the carrier subrange
/// (NOT the full service range). Index 1 = first carrier, index 2 = second,
/// etc. Index 0 or out-of-range values are ignored (treated as nil).
```
- **Risk:** Zero. Documentation-only change.
- **Verify round-trip:** N/A
- **Verify syncback parity:** N/A
- **Tests required:** None.
---
### Fix 6: Add missing test coverage
- **Status:** Approved
- **Finding:** Step 13 (audit report) -- several edge cases lack test coverage
- **Files:** `[tests/tests/live_syncback.rs](tests/tests/live_syncback.rs)`
- **Problem:** Missing tests for service attributes/tags, multiple Ref properties, concurrent requests, malformed input, and properties-only services.
- **Solution:** Add the following tests to `tests/tests/live_syncback.rs`:
**Test 1: `parity_service_attributes_and_tags`**
- Create a Lighting service with custom Attributes (e.g., `{"CustomFloat": Float32(1.5)}`) and Tags (e.g., `["TestTag"]`)
- Use `make_service_chunk_full` with Attributes and Tags encoded as `Variant::Attributes` and `Variant::Tags` in the properties map
- Run `assert_live_matches_cli`
**Test 2: `parity_multiple_service_refs`**
- Create a service (Workspace) with two Ref properties: one pointing to a Part child, one pointing to a Light child
- Use `make_service_chunk_full` with two carrier ObjectValues and two `refs` entries
- Run `assert_live_matches_cli`
**Test 3: `rejects_concurrent_syncback`** (for Fix 3)
- Use `run_serve_test` to start a server
- Send a valid syncback request (don't wait for restart)
- Immediately send a second syncback request
- Verify the second returns 409 Conflict
- Wait for the server to come back online (first syncback completes)
**Test 4: `recovers_from_malformed_chunks`** (for Fixes 2+4)
- Use `run_serve_test` to start a server
- Record the initial session info
- Send a syncback request with mismatched counts (e.g., `child_count: 999` but empty data blob)
- Verify the server comes back online (new session ID)
- Verify the filesystem is unchanged (the malformed syncback was rejected before writing)
**Test 5: `parity_properties_only_no_children`**
- Create 3 services (Lighting, SoundService, TextChatService) all with properties but zero children
- Verify empty rbxm blob case works with properties
- Run `assert_live_matches_cli`
- **Risk:** Zero. Test-only additions.
- **Verify round-trip:** The parity tests verify round-trip by design.
- **Verify syncback parity:** The parity tests verify CLI parity by design.
- **Tests required:** Self-contained -- the fix IS the tests.
---
## Skipped Fixes
None -- all identified fixes were approved.
## Accepted Limitations
### rbxm Child Ordering Assumption
- **Limitation:** Live syncback relies on `SerializationService:SerializeInstancesAsync()` preserving the array order of top-level instances in the resulting rbxm blob. If Roblox ever changes the serializer to reorder by class, the cursor-based partitioning in `build_dom_from_chunks` would break.
- **Scenario:** A future Roblox engine update changes serialization order, causing children to be assigned to wrong services.
- **User decision:** Accepted for this release. This is the standard approach used by all Rojo-family tools. The rbxm format preserves parent-child relationships and child ordering through the PRNT chunk. Roblox has never changed this behavior.
- **Mitigation:** Fix 4 adds count validation that would catch the symptom (cursor doesn't consume all instances) even if not the root cause.
### Asymmetric ClockTime Exclusion
- **Limitation:** `encodeService.lua` excludes `Lighting.ClockTime` (alias for `TimeOfDay`) but the server-side `build_dom_from_chunks` doesn't have this exclusion.
- **Scenario:** A crafted request with both `ClockTime` and `TimeOfDay` could produce conflicting properties. This never happens in normal usage since the plugin prevents it.
- **User decision:** Accepted. The plugin is the only intended client, and it already filters correctly.
- **Mitigation:** None needed beyond the existing plugin-side filter.
### No Session ID on `/api/syncback`
- **Limitation:** Unlike `/api/write`, the syncback endpoint has no session ID validation. Any client that knows the host/port can trigger a syncback.
- **Scenario:** On a shared network with `serveAddress: "0.0.0.0"`, someone could overwrite project files.
- **User decision:** Accepted by design. The syncback is intended to work from the NotConnected page (no active session). The `servePlaceIds`/`blockedPlaceIds` whitelist provides protection for team environments.
- **Mitigation:** Existing place ID validation. The serve address defaults to `127.0.0.1` (localhost only).
## Deferred Refactors
### Duplicate Service Lists
The service list exists in 3 places that must stay synchronized:
1. `SYNCBACK_SERVICES` in `plugin/src/App/init.lua` (lines 7-22)
2. `VISIBLE_SERVICES` in `src/syncback/mod.rs` (lines 73-88)
3. Implicitly in test helper code (`serve_util.rs`)
Currently they match exactly (14 services). If these diverge, Ref properties targeting instances in a missing service would be silently lost. A cross-language shared constant is impossible (Lua vs Rust), but a CI check or a test that compares the lists (e.g., parsing the Lua source) could prevent drift. Deferred because it's a build/CI infrastructure change, not a code change.
### `build_dom_from_chunks` Duplication with Test Helper
`build_dom_from_chunks` in `serve.rs` and `make_rbxl_from_chunks` in `serve_util.rs` implement nearly identical DOM reconstruction logic. The test helper was written to mirror the production code for CLI parity tests. If the production code changes, the test helper must be updated in lockstep. Consider extracting the shared logic into a `librojo` function that both can call. Deferred because it requires making `build_dom_from_chunks` public and adjusting its API.
## Test Plan
Tests are not optional. Every approved fix must have test coverage. A fix is not complete until its tests pass.
### Rust Integration Tests (`tests/tests/live_syncback.rs`)
- `parity_service_attributes_and_tags` -- service with Attributes + Tags survives round-trip (Fix 6, test 1)
- `parity_multiple_service_refs` -- service with 2+ Ref properties, all carriers resolved correctly (Fix 6, test 2)
- `rejects_concurrent_syncback` -- second syncback returns 409 Conflict (Fix 6, test 3)
- `recovers_from_malformed_chunks` -- mismatched counts cause error, server recovers (Fix 6, test 4)
- `parity_properties_only_no_children` -- 3 services with properties, zero children, empty blob (Fix 6, test 5)
### Existing Tests (Must Continue Passing)
All existing tests in `tests/tests/live_syncback.rs` must pass after all fixes:
- `parity_scripts`, `parity_models_with_properties`, `parity_special_names`, `parity_duplicate_names`
- `parity_deep_hierarchy`, `parity_mixed_file_types`, `parity_multi_service`
- `parity_empty_services_mixed`, `parity_lighting_properties`, `parity_soundservice_properties`
- `parity_starterplayer_properties`, `parity_textchatservice_properties`
- `parity_childless_service_with_properties`, `parity_service_properties_with_children`
- `parity_multiple_services_with_properties`, `parity_cross_service_refs`
- `parity_camera_not_synced`, `rejects_bad_protocol`, `rejects_bad_version`
- `rejects_blocked_place`, `rejects_unlisted_place`, `allows_whitelisted_place`
- `allows_any_place_when_unrestricted`, `server_comes_back_functional`
- `syncback_twice_different_data`, `syncback_replaces_all_old_files`
- `roundtrip_build_syncback_rebuild`
## Test Rules
The implementer MUST follow these rules when writing tests for this plan:
### For bugs found and fixed
Every bug fix MUST have a test that:
1. Would have **FAILED** before the fix
2. **PASSES** after the fix
3. Prevents the bug from regressing
### For missing coverage identified
For each missing test coverage gap:
1. Write the test
2. Verify it passes against the current implementation
3. If it fails, investigate -- it may have found another bug
### Scope: test every affected layer
For each finding, determine which test layers are affected and write tests in ALL of them:
- Fixes 1-5 affect the Rust server -- integration tests in `tests/tests/live_syncback.rs`
- No Lua spec tests needed (plugin encoding is unchanged)
- No snapshot tests needed (syncback output format is unchanged)
## Final Step: Run CI
After ALL fixes and tests are implemented, run the `/ci` command (see `.cursor/commands/ci.md`) to execute the full CI pipeline. Every fix must pass CI before the plan is considered complete. Do not skip this step.

195
.cursor/plans/live_syncback_float_precision_1be79350.plan.md Normal file
View file

@ -0,0 +1,195 @@
---
name: Live syncback fixes
overview: "Fix five issues: lost Workspace attributes, spurious ClockTime, Rojo_Ref_CurrentCamera leak, and float precision noise via f32 truncation."
todos:
- id: fix-attributes-encoding
content: Add explicit Attributes/Tags encoding to encodeService.lua (mirroring encodeInstance.lua)
status: completed
- id: fix-clocktime
content: Filter non-serializable properties server-side in build_dom_from_chunks using the Rust reflection DB
status: completed
- id: fix-currentcamera-ref
content: "Stop Rojo_Ref_CurrentCamera from leaking: filter deferred_referents after camera destruction in syncback_loop"
status: completed
- id: fix-float-rounding
content: Truncate all float values to f32 in UnresolvedValue::from_variant for rbxl parity
status: completed
- id: fix-index-refs
content: Change service-level refs to index-based with ObjectValue carriers (childCount + refTargetCount, 1-based index, 0=nil)
status: completed
isProject: false
---
# Live Syncback Fixes
## Issues 1-4: DONE
## Issue 5: Index-Based Service Refs via ObjectValue Carriers
**Problem:** Service Ref targets aren't always direct children. Can't include arbitrary instances directly in the rbxm without duplicating or breaking serialization.
**Solution:** Create temporary ObjectValue instances as Ref carriers. The ObjectValue's `Value` property IS a Ref that gets preserved in the rbxm because both the ObjectValue and the target are in the same blob.
**Layout:**
```
allChildren = [
...svc1_children, ...svc1_objectValues,
...svc2_children, ...svc2_objectValues,
...
]
```
### Plugin -- `[plugin/src/ChangeBatcher/encodeService.lua](plugin/src/ChangeBatcher/encodeService.lua)`
Return metadata + list of ObjectValue carrier instances to append:
```lua
local children = service:GetChildren()
local refTargets = {} -- ObjectValue carriers
for propertyName ... do
if descriptor.dataType == "Ref" then
local readOk, target = descriptor:read(service)
if readOk and target then
-- Check if target is a direct child
for i, child in children do
if child == target then
refs[propertyName] = i -- 1-based into children range
goto found
end
end
-- Not a child: create ObjectValue carrier
local carrier = Instance.new("ObjectValue")
carrier.Name = propertyName
carrier.Value = target
table.insert(refTargets, carrier)
refs[propertyName] = #children + #refTargets
::found::
end
continue
end
end
chunk.childCount = #children
chunk.refTargetCount = #refTargets
return chunk, refTargets
```
### Plugin -- `[plugin/src/App/init.lua](plugin/src/App/init.lua)` `performSyncback`
Append children + ObjectValue carriers, clean up after serialization:
```lua
local allRefTargets = {}
for _, className in SYNCBACK_SERVICES do
local ok, service = pcall(game.FindService, game, className)
if ok and service then
local chunk, refTargets = encodeService(service)
for _, child in service:GetChildren() do
table.insert(allChildren, child)
end
for _, carrier in refTargets do
table.insert(allChildren, carrier)
table.insert(allRefTargets, carrier)
end
table.insert(services, chunk)
end
end
local data = buffer.create(0)
if #allChildren > 0 then
data = SerializationService:SerializeInstancesAsync(allChildren)
end
-- Clean up temporary ObjectValue carriers
for _, carrier in allRefTargets do
carrier:Destroy()
end
```
### Server -- `[src/web/interface.rs](src/web/interface.rs)`
```rust
pub struct ServiceChunk {
pub class_name: String,
#[serde(default)]
pub child_count: u32,
#[serde(default)]
pub ref_target_count: u32,
#[serde(default)]
pub properties: HashMap<String, Variant>,
#[serde(default)]
pub refs: HashMap<String, u32>, // prop_name -> 1-based index, 0=nil
}
```
Remove `ServiceRef` struct.
### Server -- `[src/cli/serve.rs](src/cli/serve.rs)` `build_dom_from_chunks`
```rust
let child_count = chunk.child_count as usize;
let ref_count = chunk.ref_target_count as usize;
let total = child_count + ref_count;
let end = (cursor + total).min(cloned_children.len());
let service_range = &cloned_children[cursor..end];
// Parent only the real children
for &child_ref in &service_range[..child_count.min(service_range.len())] {
dom.transfer_within(child_ref, service_ref);
}
// Resolve refs using the full range
let mut carriers_to_destroy = Vec::new();
if !chunk.refs.is_empty() {
for (prop_name, &idx) in &chunk.refs {
if idx == 0 || (idx as usize - 1) >= service_range.len() {
continue;
}
let carrier_ref = service_range[idx as usize - 1];
// If in the ref target range, it's an ObjectValue carrier
if (idx as usize) > child_count {
let carrier = dom.get_by_ref(carrier_ref).unwrap();
if let Some(Variant::Ref(actual_target)) = carrier.properties.get("Value") {
let service = dom.get_by_ref_mut(service_ref).unwrap();
service.properties.insert(
prop_name.as_str().into(),
Variant::Ref(*actual_target),
);
}
carriers_to_destroy.push(carrier_ref);
} else {
// Direct child ref
let service = dom.get_by_ref_mut(service_ref).unwrap();
service.properties.insert(
prop_name.as_str().into(),
Variant::Ref(carrier_ref),
);
}
}
}
// Destroy ObjectValue carriers
for carrier_ref in carriers_to_destroy {
dom.destroy(carrier_ref);
}
cursor = end;
```
### Test helpers -- `[tests/rojo_test/serve_util.rs](tests/rojo_test/serve_util.rs)`
- `ServiceEntry` gets `ref_targets: Vec<InstanceBuilder>` for ObjectValue carriers
- `make_service_chunk_full` refs param: `Vec<(&str, u32)>` (prop_name, 1-based index)
- `build_syncback_request` appends ref_targets after children per service
- `make_rbxl_from_chunks` resolves refs: direct children by index, carriers by reading `Value` property
### Tests
- Update `parity_camera_not_synced` to use index-based ref with ObjectValue carrier
- Existing tests that don't use refs continue to work (empty refs map, refTargetCount=0)
## Deferred: Matching Parity
Revisit after these fixes land.

743
.cursor/plans/luau-optimize_cursor_command_ce1448a8.plan.md Normal file
View file

@ -0,0 +1,743 @@
---
name: luau-optimize cursor command
overview: Create a `/luau-optimize` Cursor command at `.cursor/commands/luau-optimize.md` that uses the `luau-compile`, `luau-analyze`, and `luau-ast` CLI tools to analyze and optimize Luau files for maximum performance, type safety, and native codegen quality.
todos:
- id: create-command
content: Create `.cursor/commands/luau-optimize.md` with the full multi-phase optimization workflow
status: completed
isProject: false
---
# /luau-optimize Cursor Command
## Deliverable
Single file: `[.cursor/commands/luau-optimize.md](.cursor/commands/luau-optimize.md)`
## Context
**This command is always for Roblox Luau.** All scripts being optimized run in the Roblox engine (Studio / live servers / clients). This means:
- Roblox APIs (`game`, `workspace`, `Players`, `RunService`, etc.) are always available
- The vector type is `Vector3` (Roblox userdata), not the `vector` primitive
- Native codegen (`--!native`) primarily benefits server-side Scripts; client LocalScripts benefit less due to device diversity
- The `luau` standalone runtime cannot execute most Roblox scripts (they depend on engine APIs); it is only useful for pure-logic library modules with no Roblox dependencies
- `luau-compile` should use `--vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3` for accurate Roblox vector optimization analysis
- Roblox performance patterns (event-driven design, frame budgets, RunService events, instance streaming, replication) are always relevant
- Studio testing uses `--!optimize 1` by default; live experiences use `--!optimize 2`
## Design
The command is a structured prompt file that instructs the AI agent through a multi-phase optimization workflow. It supports both single-file and multi-file/directory modes.
### Phase Structure
**Phase 0 -- Scope Resolution and Configuration**
Resolve the target:
- If user specifies a file: optimize that file
- If user specifies a directory: discover all `.luau`/`.lua` files, prioritize by traffic/complexity
- If no argument: optimize the currently open file
Infer script context from filename and path (checked in this order):
1. **Filename suffix** (highest priority, overrides path):
- `*.server.luau` / `*.server.lua` -> always server Script (`--!native` is high-value)
- `*.client.luau` / `*.client.lua` -> always client Script/LocalScript (`--!native` less beneficial due to device diversity)
- `*.legacy.luau` / `*.legacy.lua` -> **ambiguous** (could be cloned/moved at runtime). Ask the user where it runs.
- `*.luau` / `*.lua` (no suffix) -> ModuleScript, context depends on who requires it. Fall through to path check.
2. **Path-based inference** (for ModuleScripts and ambiguous cases):
- Path contains `ServerScriptService`/`ServerStorage` -> server-side module
- Path contains `StarterPlayer`/`StarterGui`/`StarterCharacterScripts`/`ReplicatedFirst` -> client-side module
- Path contains `ReplicatedStorage`/`shared` -> shared module (runs on both; conservative `--!native` recommendation)
- If path is ambiguous or outside a Roblox project structure -> ask the user
**Quiz the user using AskQuestion tool (combined, structured):**
```
Title: "/luau-optimize Configuration"
Question 1: "Compilation optimization intensity"
- minimal -- Headers (--!strict/--!native/--!optimize 2), type annotations on all function signatures, simple deprecated pattern replacements
- moderate -- + function restructuring for inlining, import hoisting, compound operators, fastcall enablement, allocation reduction
- insane -- + every micro-optimization from the pattern catalog, full bytecode analysis, register pressure optimization, closure caching analysis
- (other: freeform)
Question 2: "Algorithm optimization"
- none -- Don't touch logic or data flow
- low hanging fruit -- Obvious O(n^2)->dict, redundant iterations, missing caches
- moderate -- + data structure changes, caching strategies, event-driven refactors, loop fusion
- insane -- + full algorithmic redesign where beneficial, dynamic programming, architectural restructuring
- (other: freeform)
Question 3: "Benchmarking"
- yes -- Benchmark all non-trivial changes with luau CLI (when function is standalone)
- no -- Trust bytecode/codegen metrics only
- +EV only -- Benchmark only when algorithmic changes or ambiguous bytecode results warrant it
```
**Derived behavior from quiz answers:**
| Setting | minimal | moderate | insane |
| ------------------------ | --------------------------- | ------------------------------------- | ---------------------------------------- |
| Headers | Always | Always | Always |
| Type annotations | Function signatures | Function signatures + hot path locals | Function signatures + all locals |
| Pattern replacements | Deprecated only | Full Priority 2 catalog | Full Priority 2 + Priority 5 |
| Function restructuring | No | Yes (Priority 3) | Yes + aggressive splitting |
| Algorithmic changes | Per algo quiz | Per algo quiz | Per algo quiz |
| Bytecode verification | After all changes | After each priority | After each individual change |
| `@native` vs `--!native` | `--!native` unless big file | Selective `@native` on hot functions | Full analysis of which functions benefit |
**Restructuring scope** scales with compilation intensity:
- minimal: No restructuring. Only additive changes (headers, annotations, pattern swaps).
- moderate: Break up monoliths, reorder locals for inlining, hoist imports. Preserve existing exports/API.
- insane: Aggressive restructuring permitted. May change internal function boundaries, introduce helper modules, rewrite data flow.
**Phase 1 -- Deep Code Read and Algorithmic Analysis (FIRST)**
Read the entire source file thoroughly before touching any tools. This is the highest-leverage phase -- algorithmic improvements produce order-of-magnitude speedups that dwarf any bytecode-level optimization. This is also the largest code change, so it must come first (everything else builds on top of it).
Understand the code's purpose, data flow, and architecture.
**Step 1 -- Structural decomposition (read and mentally refactor)**
Monolithic functions hide their actual algorithmic structure behind walls of interleaved concerns. Before analyzing complexity, first understand what the code is *actually doing* by decomposing it mentally (and then physically if restructuring is allowed):
- **Identify responsibilities**: What distinct jobs does each large function perform? A 200-line function usually contains 3-5 logical steps that should be separate functions.
- **Trace data flow**: What goes in, what comes out, what is mutated along the way? Map the pipeline.
- **Separate concerns**: Validation, transformation, I/O, state management, error handling -- these are often tangled together. Separating them reveals the core algorithm.
- **Name the steps**: If you can't name what a code block does in 3-5 words, it's doing too much.
- **Look for hidden abstractions**: Repeated patterns of "iterate, filter, transform, collect" that could be a single well-named helper.
When the quiz allows restructuring (moderate/insane compilation intensity):
- Physically decompose monoliths into focused local functions. Each function should do one thing.
- This decomposition often *reveals* optimization opportunities that were invisible in the monolith: you can now see that step 2 recomputes what step 1 already knew, or that the inner loop could be replaced with a lookup table built in the outer loop.
- Smaller functions are also easier for the compiler to inline at `-O2`, so this restructuring pays double -- clearer code AND better bytecode.
- Preserve the original function as a thin orchestrator that calls the decomposed pieces.
Even when the quiz says "minimal" (no restructuring), still perform this mental decomposition to inform the analysis below.
**Step 2 -- Algorithmic analysis**
With the structure understood (or cleaned up), analyze:
- **Complexity analysis**: Identify the time complexity of every significant function. Look for:
- O(n^2) or worse: nested loops over the same data, repeated linear searches, quadratic string building
- O(n) where O(1) is possible: linear search replaceable with dictionary/set lookup
- Repeated work: same computation done multiple times when it could be cached/memoized
- Unnecessary copies: cloning data that could be referenced or mutated in place
- Hidden quadratics: a function that looks O(n) but calls another O(n) function inside a loop
- **Data structure fitness**: Is the right data structure being used?
- Array used for membership testing -> should be a set (dictionary with `true` values)
- Linear scan for lookup by key -> should be a dictionary
- Repeated sort of mostly-sorted data -> consider maintaining sorted order on insert
- Flat list with frequent removal from middle -> consider swap-and-pop
- Large string built incrementally -> `table.concat` pattern or `buffer`
- Multiple parallel arrays tracking related data -> single array of structs (table of tables)
- Unbounded growth without cleanup -> consider size limits, LRU eviction, or periodic pruning
- **Caching and memoization**:
- Pure functions called with the same args repeatedly -> memoize
- Expensive property reads in loops -> cache in local before loop
- Computed values that don't change within a scope -> hoist out
- Derived state recomputed from scratch on every access -> maintain incrementally
- **Redundant work elimination**:
- Multiple passes over the same data that could be fused into one
- Recomputing derived state that could be maintained incrementally
- Sorting or filtering that could be done once and reused
- Deep cloning where shallow clone or reference suffices
- Building an intermediate result that is immediately discarded after extracting one field
- **Architectural patterns**:
- Per-frame polling where event-driven would work (RunService events are expensive)
- Synchronous work that could be chunked across frames with `task.wait()`
- Undisconnected event connections (memory leak + wasted computation)
- Rebuilding entire state on small change where incremental update would work
- God-object that owns too much state -> split responsibilities
- **Hot path identification**: Which functions are called most frequently?
- Event handlers connected to RunService (PreAnimation, PreRender, PreSimulation, PostSimulation, Heartbeat)
- Functions called inside loops
- Recursive functions
- Functions called from other hot functions
**Step 3 -- Present findings and quiz the user**
Before applying any changes, present a summary of all findings organized by impact:
```
For each finding, present:
- What: One-line description of the issue
- Where: Function name / line range
- Why: What makes it slow (e.g., "O(n^2) nested scan on every frame")
- Fix: Proposed change in plain language
- Impact: Estimated improvement (order-of-magnitude, constant-factor, or clarity-only)
```
Then quiz the user using AskQuestion tool to let them control the flow:
```
Title: "Phase 1 Findings -- Select which changes to apply"
For each finding, one question with options:
- apply -- Apply this change
- skip -- Skip this change
- modify -- I want a different approach (wait for user input)
If there are many findings (5+), group them by category and present one question per category:
- "Structural decomposition: [list of proposed decompositions]. Apply all / cherry-pick / skip all"
- "Data structure changes: [list]. Apply all / cherry-pick / skip all"
- "Algorithm changes: [list]. Apply all / cherry-pick / skip all"
- "Architectural changes: [list]. Apply all / cherry-pick / skip all"
```
If the algorithm quiz from Phase 0 was "none", present findings as informational only (no apply option). The user still sees what was found.
**Step 4 -- Apply approved changes**
Apply only the user-approved changes, in this order:
1. Structural decomposition first (if approved) -- break monoliths into focused functions. This is the foundation; other changes may depend on it.
2. Data structure changes -- swap arrays for dicts, add indexes, etc.
3. Algorithm changes -- replace O(n^2) with O(n), add caching/memoization
4. Architectural changes -- event-driven refactors, incremental updates
If the user selected "modify" on any finding, wait for their input before proceeding with that specific change.
Benchmark algorithmic changes if the benchmarking quiz answer warrants it (Phase 4.5 pattern).
**Phase 2 -- Baseline Capture (read-only)**
Run all diagnostic tools on the code (post-algorithmic changes if any were made in Phase 1) and capture baseline metrics for the bytecode optimization phases:
1. `luau-analyze --mode=strict <file>` -- type errors and lint warnings
2. `luau-analyze --annotate <file>` -- inferred types (count `any` occurrences for type coverage score)
3. `luau-compile --remarks -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>` -- inlining success/failure, allocation remarks (with Roblox vector optimization)
4. `luau-compile --text -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>` -- full bytecode listing (instruction count, opcodes)
5. `luau-compile --codegen --target=x64_ms --record-stats=function -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>` -- native codegen stats (spills, skipped functions, block counts). Use `--target=x64` on Linux/Mac.
Parse and summarize:
- Total bytecode instruction count
- Allocation count (NEWTABLE, NEWCLOSURE from remarks)
- Inlining success/failure count (from remarks)
- Type coverage: count of `any` types in annotate output
- Register spills and skipped functions from codegen stats
- Lint warnings count
**Phase 3 -- Compilation-Level Code Analysis**
Using both the source (already read in Phase 1) and the tool output from Phase 2, identify bytecode-level optimization opportunities:
- **Missing types**: `any` inferences that hurt native codegen (especially `Vector3`, `CFrame`, `buffer` params)
- **Import patterns**: Global chains like `math.max` are resolved at load time (not execution) via GETIMPORT. Broken by `getfenv`/`setfenv`/`loadstring` which mark environment "impure" and disable this.
- **Allocation in loops**: Table/closure creation inside hot loops (NEWTABLE, NEWCLOSURE). High allocation rate = more GC assist work. Avoid temporary tables and userdata in hot loops.
- **Closure caching**: Repeated function expressions can be cached (same closure reused) when: no upvalues, or all upvalues are immutable and declared at module scope. Mutable upvalue captures prevent caching.
- **Upvalue mutability**: ~90% of upvalues are immutable in typical code. Immutable upvalues = no extra allocation, no closing, faster access, better locality. Mutable upvalues need extra objects.
- **Method call patterns**: `obj:Method()` uses specialized fast method call instruction. Avoid `obj.Method(obj)`. For tables: `__index` should point at a table directly (not a function or deep chain). Ideal: metatable whose `__index` points to itself.
- **Inline caching**: VM uses inline caching for table field access. Best when field name is known at compile time, no metatables, and object shapes are uniform. Varying keys/shapes degrade performance.
- **pcall in hot paths**: Prevents native codegen optimization
- **String concatenation patterns**: `..` in loops vs `table.concat`
- **Deprecated/deoptimizing API usage**: `getfenv`/`setfenv` (marks env "impure", disables ALL import optimization and fastcalls), `loadstring` (same deoptimization), `table.getn`/`table.foreach`/`table.foreachi`, `wait()`. Even read-only `getfenv()` deoptimizes.
- **Builtin global writes**: Overwriting builtins (e.g., `math = ...`) disables fastcall optimization. Lint rule `BuiltinGlobalWrite` catches this.
- **Metamethod cost**: `__eq` is always called on `==`/`~=` even for rawequal values; ensure cheap implementations
- **GC pressure**: Incremental GC uses "GC assists" -- allocating code pays for collection work proportionally. Fewer allocations = less GC overhead.
**Phase 4 -- Compilation Optimization (graduated priority)**
Apply changes in priority order, re-running `luau-compile --remarks -O2` after structural changes to verify the compiler actually benefits:
**Priority 1 -- Headers and type safety (always apply)**
- Add `--!strict` / `--!native` / `--!optimize 2` header (note: `--!optimize 2` is default in live experiences but not in Studio testing)
- Fix type errors reported by `luau-analyze`
- Add missing type annotations where `--annotate` shows `any` (focus on function signatures first, then locals in hot paths)
- Annotate `Vector3`, `CFrame`, `buffer` parameters explicitly -- native codegen generates specialized vector code when types are known; unannotated params are assumed to be tables with extra checks
- Replace `getfenv`/`setfenv` usage -- these disable builtin calls, imports, and optimizations globally for the affected function
- Replace `table.getn` with `#t` or `rawlen(t)`, `table.foreach`/`table.foreachi` with `for..in` loops
**Priority 2 -- Low-hanging structural wins**
- Hoist frequently-used library functions: `local floor = math.floor` enables `GETIMPORT` fastcall. Fastcall builtins include: `assert`, `type`, `typeof`, `rawget`/`rawset`/`rawequal`, `getmetatable`/`setmetatable`, `tonumber`/`tostring`, most `math.`* (not `noise`, `random`/`randomseed`), `bit32.`*, and some `string.`*/`table.*`. Partial specializations: `assert` when return value unused and condition truthy; `bit32.extract` when field/width constant; `select(n, ...)` is O(1) via builtin dispatch.
- Replace `pairs(t)` / `ipairs(t)` with generalized `for k, v in t do` iteration. Generalized iteration skips the `pairs()` call, which matters for very short tables. Performance of all three is similar for longer tables. Note: `for i=1,#t` is slightly slower due to extra element access cost.
- Use `//` floor division instead of `math.floor(a / b)` -- dedicated VM opcode, supports `__idiv` metamethod, also `//=` compound form
- Use compound assignment (`+=`, `-=`, `*=`, `//=`, `..=`) -- LHS evaluated once, e.g. `data[index].cost += 1` avoids double-indexing
- Replace string concatenation in loops with `table.concat` or string interpolation (backticks lower to `string.format` with no extra overhead)
- Use `table.create(n)` for known-size arrays -- preallocates capacity, much faster than repeated `table.insert`
- For sequential fill with known index, use indexed writes with `table.create`: `local t = table.create(N); for i=1,N do t[i] = ... end` (fastest pattern)
- Use `table.insert(t, v)` for appending when size unknown -- `#t` is usually O(1) (cached, updated by `table.insert`/`table.remove`; worst case O(log N) branch-free binary search, "50%+ faster" than Lua's O(N))
- Use `math.lerp(a, b, t)` instead of `a + (b - a) * t` -- exact for t=0/1, monotonic, bounded
- Use `bit32.byteswap(n)` instead of manual shifting/masking for endian conversion (uses CPU bswap instruction)
- Use `bit32.countlz`/`bit32.countrz` instead of manual log2 loops (uses CPU instructions, ~8x faster)
- Use `table.find(t, v)` instead of manual linear search loops
- Use `table.clone(t)` instead of manual `pairs` copy loops
- Use `string.pack`/`string.unpack` instead of manual `string.byte`/`string.char`/`bit32` for binary data
- Use `rawlen(t)` when metamethods are not needed for table length
- Use explicit `./`/`../`/`@` prefixes in `require()` to avoid extra path-resolution work
**Priority 3 -- Function structure for inlining**
- Break monolith functions into smaller, focused local functions (compiler inlines automatically at `-O2`; small local functions = best inlining candidates)
- Inlining requirements: function must be local, non-mutated, non-recursive, not an OOP method (`Foo:bar()`), not a metamethod. Exported functions in mutable tables CANNOT be inlined. Disabled in modules using `getfenv`/`setfenv`.
- Inlining + constant folding: inlined functions with constant args can enable further constant folding (e.g., `local function double(x) return x*2 end; local y = double(5)` folds to `y = 10`)
- For high-traffic functions that call imported module methods frequently, create local function wrappers at module top-level to enable inlining
- Move `pcall` out of hot loops where possible (wrap the loop, not the body) -- pcall prevents native codegen optimization
- Use `obj:Method()` not `obj.Method(obj)` -- colon syntax gets specialized fast method call instruction
- Keep `__index` pointing at a table directly (not a function or deep chain) for optimal inline caching
- Reduce upvalue captures in closures created inside loops. Immutable upvalues (never reassigned) are much cheaper -- no extra allocation, no closing. Mutable upvalues need an extra object.
- Prefer object-like table construction with all fields set at once (enables compiler capacity inference): `local v = {}; v.x = 1; v.y = 2; v.z = 3` -- compiler infers hash capacity from subsequent assignments
- Loop unrolling: only for compile-time constant bounds (`for i=1,4 do`); body must be simple. Unrolling enables further constant folding on the loop index.
- Watch for native codegen size limits: 64K instructions per code block, 32K blocks per function, 1M instructions per module. Split large functions if limits are hit.
**Priority 4 -- Micro-optimizations (hot paths only)**
- Table shape consistency (same fields, same order across constructors) -- VM uses inline caching (HREF-style) that predicts hash slots; uniform shapes = better cache hit rates
- Use `buffer` for binary data manipulation instead of string ops (fixed-size, offsets start at 0, far more space/time efficient)
- Use `buffer.readu32` + `bit32` operations instead of `buffer.readbits` when schema is known (faster due to fewer bounds checks)
- Strength reduction: multiplication by power-of-2 to `bit32.lshift`, division by power-of-2 to `bit32.rshift`
- Minimize `tostring`/`tonumber` in tight loops
- Use `table.freeze` for readonly config tables (avoids need for proxy tables and `__index` overhead)
- Keep `__eq` metamethods cheap -- they fire on every `==`/`~=` and `table.find` comparison
- Use `math.isfinite`/`math.isnan`/`math.isinf` instead of `x ~= x` patterns for clarity (no perf difference)
- For vector math: use `Vector3.new(x, y, z)` (Roblox userdata, not the `vector` primitive); annotate params as `Vector3` for native specialization
- Use `if expr then A else B` instead of `cond and A or B` ternary (only evaluates one branch; avoids falsy-value bugs)
**Phase 4.5 -- Benchmark with `luau` CLI (when warranted)**
When a non-trivial optimization is applied to an isolated, pure-logic function (no Roblox API dependencies), write a small benchmark harness and run it with the `luau` CLI to measure actual wall-clock impact. This is especially valuable for:
- Algorithmic changes (Priority 4) where bytecode metrics alone can't prove the win
- Micro-optimizations (Priority 5) where the improvement is ambiguous
- Competing approaches where both look reasonable in bytecode but one may be faster at runtime
**Benchmark harness pattern:**
```luau
-- bench.luau (temporary file, deleted after benchmarking)
local function original(...)
-- paste original implementation
end
local function optimized(...)
-- paste optimized implementation
end
local ITERATIONS = 1_000_000
local clock = os.clock
-- Warmup
for _ = 1, 1000 do original(...) end
for _ = 1, 1000 do optimized(...) end
local t0 = clock()
for _ = 1, ITERATIONS do original(...) end
local t1 = clock()
for _ = 1, ITERATIONS do optimized(...) end
local t2 = clock()
print(`Original: {t1 - t0:.4f}s`)
print(`Optimized: {t2 - t1:.4f}s`)
print(`Speedup: {(t1 - t0) / (t2 - t1):.2f}x`)
```
**Run with both interpreter and native codegen to compare:**
```
luau -O2 bench.luau # Interpreter at -O2
luau -O2 --codegen bench.luau # Native codegen at -O2
```
**When to benchmark:**
- The function is self-contained (no `game`, `workspace`, Roblox services)
- The optimization changes algorithm or data structure (not just headers/types)
- The bytecode diff is ambiguous (instruction count similar, or more instructions but fewer allocations)
- Two candidate approaches need a tiebreak
**When NOT to benchmark:**
- The function depends on Roblox APIs (use Studio Script Profiler instead)
- The change is purely additive (headers, type annotations) -- trust the bytecode metrics
- The function is not hot (benchmarking cold code is misleading)
**Cleanup:** Delete the temporary benchmark file after capturing results. Do not leave bench files in the project.
**Phase 5 -- Verification**
Re-run all Phase 2 tools and present a before/after comparison:
- Bytecode instruction count delta
- Allocation count delta
- Inlining success rate delta
- Type coverage improvement (fewer `any` types)
- Register spill delta
- Lint warning delta
- Benchmark results (if Phase 4.5 was used) -- wall-clock speedup for interpreter and native codegen
If any metric regressed, investigate and explain why (or revert that change).
**Phase 6 -- Multi-file mode (directory scope only)**
When optimizing a directory:
- Run Phase 1 (code read) on all files first to identify which files have the most algorithmic improvement potential
- Run Phase 2 (baseline capture) on all files to identify which have the most bytecode optimization potential
- Prioritize files by: worst algorithmic complexity, most lint warnings, most `any` types, most allocations, most failed inlines
- Process files one at a time through Phases 1-5
- Present aggregate metrics at the end
### Key Principles (embedded in command)
- **Same implementation, better performance** -- behavioral equivalence is non-negotiable
- **Low-hanging fruit first** -- headers and types before restructuring
- **Compiler feedback loop** -- verify each structural change with `--remarks`
- **Small functions over monoliths** -- easier for compiler, easier to maintain
- **Local functions for inlining** -- even at cost of duplicating imported behavior in high-traffic code
- **Quantify everything** -- before/after metrics for every optimization pass
- **Type annotations drive native codegen quality** -- every `any` is a missed specialization opportunity
- **Event-driven over polling** -- avoid per-frame calculations when events suffice
### Type System Availability (Old Typechecker)
The command must only recommend type features available with the OLD typechecker (we are NOT using the new type solver).
**Available NOW (old typechecker):**
- `--!strict` / `--!nonstrict` / `--!nocheck` directives
- Basic type annotations: `local x: number`, `function f(a: string): boolean`
- Optional types: `string?` (sugar for `string | nil`)
- Union types: `number | string`
- Intersection types: `type1 & type2`
- Generic functions: `function id<T>(x: T): T`
- Explicit generic instantiation: `f<<number>>()`
- Type ascription (cast): `x :: string`
- Array types: `{number}`, dictionary types: `{[string]: number}`
- Typed table constructors: `type Car = { Speed: number, Drive: (Car) -> () }`
- `typeof()` for inferring types from values
- `export type` for cross-module type sharing
- Typed variadics: `function f(...: number)`
- Type alias defaults: `type Map<K, V = string> = {[K]: V}`
- Singleton/literal types: `"hello"`, `true`
- `never` and `unknown` types (exhaustiveness checking, safe top type)
- `read` / `write` property modifiers
- Sealed/unsealed table subtyping
- `table.freeze` type narrowing
**REQUIRES new typechecker (DO NOT recommend):**
- `keyof<T>`, `index<T, K>`, `rawkeyof<T>`, `rawget<T, K>` type functions
- `getmetatable<T>`, `setmetatable<T, M>` type functions
- User-defined type functions (`type function ...`)
- Negation types (`~T`)
- Relaxed recursive type restrictions
- Local type inference improvements (better bounds tracking)
- `new-nonstrict` unified engine
### Native Codegen Knowledge Base
Embedded in the command for reference during optimization:
**When to use `@native` vs `--!native`:**
- `--!native`: entire script compiled natively. Good for math-heavy utility modules.
- `@native`: per-function. Better when only specific hot functions benefit, or when script is close to the 1M instruction module limit.
- Inner functions do NOT inherit `@native` -- each needs its own annotation.
- Top-level module code usually runs once; `@native` on it has minimal benefit.
**What hurts native execution:**
- `getfenv()` / `setfenv()` -- disables optimizations
- Passing wrong types to typed functions (triggers type guard failures)
- Non-numeric args to math builtins
- Breakpoints disable native execution for affected functions
- Functions exceeding size limits: 64K instructions/block, 32K blocks/function, 1M instructions/module
**What helps native execution:**
- Type annotations, especially `Vector3` parameters (specialized vector codegen). JIT uses annotations directly -- no runtime type analysis.
- Small, focused functions (fewer register spills, better block optimization)
- Consistent table shapes (predictable field access, inline caching hits)
- `buffer` operations (efficient native lowering)
- `bit32` operations (map directly to CPU instructions)
- Native 3-component 32-bit float vector type: 16-byte tagged values, VM-level 3-wide SIMD, lower GC pressure than userdata Vector3
### VM Architecture Knowledge
The command should understand how the Luau VM works to give accurate advice:
**Bytecode interpreter:**
- Highly tuned portable bytecode interpreter in C. Core dispatch loop ~16 KB on x64 (cache-friendly).
- Can match LuaJIT interpreter on some workloads.
- Computationally intensive scripts mostly use the interpreter core + builtins.
**Compiler pipeline:**
- Multi-pass: frontend parses AST, backend emits bytecode.
- Without type info: deep constant folding (across functions and locals), upvalue optimization for non-mutated upvalues, builtin usage analysis, multi-assignment optimization, peephole optimizations.
- With `-O2`: function inlining, loop unrolling (constant bounds only), more aggressive constant folding after inlining.
- Type-directed: currently small peephole optimizations; potential for CSE and allocation hoisting.
- Interprocedural: limited to single modules. Local function calls optimized with known arg/return counts.
**Inline caching:**
- Table field access uses HREF-style inline caching. Compiler predicts hash slot; VM corrects dynamically.
- Best when: field name known at compile time, no metatables, uniform object shapes.
- `table.field` and `table["constantString"]` both get inline caching.
**Import resolution:**
- Global chains like `math.max` resolved at load time, not execution time.
- Invalidated by `getfenv`/`setfenv`/`loadstring` (marks environment "impure").
**Fastcall builtins (complete list):**
- `assert`, `type`, `typeof`, `rawget`, `rawset`, `rawequal`, `rawlen`
- `getmetatable`, `setmetatable`
- `tonumber`, `tostring`
- Most `math.`* (NOT `math.noise`, `math.random`, `math.randomseed`)
- `bit32.`*
- Some `string.`*, `table.`*
- Partial specializations: `assert` (unused return + truthy), `bit32.extract` (constant field/width), `select(n, ...)` (O(1) via builtin dispatch)
- `math.floor` and similar use SSE4.1 when available
- With `-O2`: builtins with constant args and single return can be constant-folded; `math.pi`, `math.huge` etc. are folded
**GC architecture:**
- Incremental mark-sweep with "GC assists" (allocating code pays proportional GC work)
- Paged sweeper: 16 KB pages by size class, 2-3x faster than linked-list, saves 16 bytes/object on 64-bit
- No `__gc` metamethod (removed for performance)
- Weak tables with `__mode = "s"` are "shrinkable" (resized during GC)
### Type Refinement Patterns
The compiler narrows types after certain checks, improving both type safety and native codegen quality:
- **Truthiness:** `if x then` narrows `x` from falsy (`nil`/`false`)
- **Type guards:** `if type(x) == "number" then` narrows `x` to `number`
- **Typeof guards (Roblox):** `if typeof(x) == "Vector3" then` narrows to `Vector3`; `x:IsA("TextLabel")` narrows `Instance` to subclass
- **Equality:** `if x == "hello" then` narrows `x` to singleton `"hello"`
- **Assert:** `assert(type(x) == "string")` narrows `x` to `string` after the call
- **Composition:** Supports `and`, `or`, `not` for compound refinements
Use refinements to provide type information to the compiler without explicit annotations in contexts where the type is known from a runtime check.
### Comprehensive Pattern Catalog
The command will embed this as the reference for pattern replacement:
**Bytecode-level improvements:**
| Slow pattern | Fast pattern | Why |
| ----------------------------- | ------------------------------ | ----------------------------------- |
| `math.floor(a / b)` | `a // b` | Dedicated VM opcode |
| `data[i].x = data[i].x + 1` | `data[i].x += 1` | LHS evaluated once |
| `a + (b - a) * t` | `math.lerp(a, b, t)` | Exact at endpoints, monotonic |
| `"prefix" .. var .. "suffix"` | `prefix{var}suffix` | Lowers to optimized `string.format` |
| Manual byte swap | `bit32.byteswap(n)` | CPU bswap instruction |
| Manual log2 loop | `bit32.countlz(n)` | CPU instruction, ~8x faster |
| `table.getn(t)` | `#t` or `rawlen(t)` | Deprecated, slower |
| `table.foreach(t, f)` | `for k, v in t do f(k, v) end` | Deprecated, slower |
| `cond and A or B` | `if cond then A else B` | Safe for falsy values, one branch |
| `x ~= x` (NaN check) | `math.isnan(x)` | Clearer, same performance |
**Allocation reduction:**
| Slow pattern | Fast pattern | Why |
| --------------------------------- | ---------------------------------- | ----------------------------------- |
| `local t = {}` in loop | `table.clear(t)` + reuse | Avoids GC pressure |
| `table.insert(t, v)` repeated | `table.create(n)` + indexed writes | Preallocated capacity |
| Manual `pairs` clone | `table.clone(t)` | Faster, copies layout |
| `string.byte`/`string.char` loops | `string.pack`/`string.unpack` | Native implementation |
| String-based binary data | `buffer` type | Fixed-size, offset-based, efficient |
**Inlining enablers:**
| Blocks inlining | Enables inlining | Why |
| ----------------------- | ----------------------------- | ---------------------------------------- |
| `function Module.foo()` | `local function foo()` | Exported/mutable table prevents inlining |
| `function Obj:method()` | `local function method(self)` | OOP colon syntax not inlineable |
| Recursive function | Split base/recursive cases | Recursion prevents inlining |
| Deep upvalue captures | Pass as parameters | Reduces closure cost |
**Deprecated patterns to detect and replace:**
| Deprecated | Replacement | Impact |
| ------------------------- | -------------------------- | -------------------------------------------------------------------------------------------------- |
| `getfenv()` / `setfenv()` | `debug.info(i, "snl")` | Disables ALL optimizations (imports, fastcalls, inlining). Even read-only `getfenv()` deoptimizes. |
| `loadstring()` | Restructure with `require` | Marks environment "impure", disables import optimization |
| `table.getn(t)` | `#t` | Slower |
| `table.foreach(t, f)` | `for k, v in t do` | Slower |
| `table.foreachi(t, f)` | `for i, v in t do` | Slower |
| `wait()` | `task.wait()` | Deprecated Roblox API |
| `obj.Method(obj, ...)` | `obj:Method(...)` | Misses fast method call instruction |
| `string:method()` | `string.method(s)` | `string.byte(s)` is faster than `s:byte()` for builtins |
**Lint rules to check (via `luau-analyze`):**
The command should flag these lint warnings as optimization opportunities:
| Lint | Name | Optimization impact |
| ---- | ------------------ | ------------------------------------------------------------------------------------------------ |
| 10 | BuiltinGlobalWrite | Overwriting builtins disables fastcall optimization |
| 22 | DeprecatedApi | Deprecated APIs may have performance/correctness issues |
| 23 | TableOperations | `table.insert(t, 0, x)` (wrong index), `table.insert(t, #t+1, x)` (redundant), `#` on non-arrays |
| 3 | GlobalUsedAsLocal | Global used in one function -- should be local for performance |
| 7 | LocalUnused | Dead locals (cleanup candidate) |
| 12 | UnreachableCode | Dead code (removal candidate) |
| 25 | MisleadingAndOr | `a and false or c` bugs -- use `if a then false else c` |
### CLI Tool Reference
All four tools live at `C:\Program Files\Atlas\` and are in PATH.
#### `luau-compile` -- Static bytecode/codegen analysis (primary tool)
**Modes** (mutually exclusive, controls output format):
- `binary` -- Raw bytecode blob (default, not human-readable)
- `text` -- Human-readable bytecode listing with source line annotations. Shows every opcode, register, constant, and jump label per function.
- `remarks` -- Original source with compiler remarks injected as `-- remark:` comments. Shows inlining success/failure (with cost/profit), allocation tracking (`table hash N`, `table array N`). **Single most useful mode for optimization.**
- `codegen` -- Native assembly output. Shows full register allocation, memory ops, branch structure, and outlined helpers. Requires `--target`.
**Options:**
- `-O<n>` -- Optimization level 0-2 (default 1). `**-O2` enables automatic inlining.** Always use `-O2` to see production compiler behavior. `-O0` useful as baseline comparison.
- `-g<n>` -- Debug info level 0-2 (default 1). `-g0` strips debug info for smallest bytecode.
- `--target=<arch>` -- Architecture for codegen mode: `x64` (Linux/Mac), `x64_ms` (Windows x64 calling convention), `a64` (ARM64), `a64_nf` (ARM64 no-FP). **Use `x64_ms` on Windows.**
- `--record-stats=<granularity>` -- JSON compilation stats at `total`, `file`, or `function` granularity. At `function` level shows: `bytecodeInstructionCount`, codegen size, parse/compile time, and `lowerStats` (spillsToSlot, spillsToRestore, maxSpillSlotsUsed, skippedFunctions, blocksPreOpt, blocksPostOpt, regAllocErrors, loweringErrors).
- `--bytecode-summary` -- Bytecode operation distribution. Requires `--record-stats=function`. Shows which opcodes dominate.
- `--stats-file=<filename>` -- Where to write stats JSON (default `stats.json`). Useful when analyzing multiple files.
- `--timetrace` -- Writes `trace.json` with compiler phase timings (Chrome `chrome://tracing` compatible).
- `--vector-lib=<name>` -- Library name for vector ops (e.g., `"Vector3"`). Tells compiler about vector type for better optimization.
- `--vector-ctor=<name>` -- Constructor function name (e.g., `"new"`). Paired with `--vector-lib` for constant-folding.
- `--vector-type=<name>` -- Vector type name. Completes the vector triplet for full vector optimization.
**Key invocations for the command (always include Roblox vector flags):**
```
luau-compile --remarks -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>
luau-compile --text -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>
luau-compile --codegen --target=x64_ms --record-stats=function -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file> # Windows
luau-compile --codegen --target=x64 --record-stats=function -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file> # Linux/Mac
luau-compile --bytecode-summary --record-stats=function -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>
```
#### `luau-analyze` -- Type checking and linting
**Modes:**
- *(omitted)* -- Default: typecheck + lint. Reports type errors and lint warnings to stderr.
- `--annotate` -- Outputs source with all inferred types written inline. Every variable, parameter, and return gets its inferred type. **Critical for finding `any` types that hurt native codegen.**
**Options:**
- `--mode=strict` -- Forces `--!strict` type checking even if file lacks the directive. Catches type errors silent in nonstrict mode.
- `--formatter=plain` -- Luacheck-compatible output (machine-parseable, good for counting errors).
- `--formatter=gnu` -- GNU-compatible output (`file:line:col: message`), grep-friendly.
- `--timetrace` -- Writes `trace.json` with analysis phase timings.
**Key invocations:**
```
luau-analyze --mode=strict <file> # Type errors + lint warnings
luau-analyze --annotate <file> # Inferred types (find 'any' types)
```
#### `luau` -- Runtime execution, profiling, and coverage
**Limited utility for Roblox scripts.** Most Roblox scripts depend on engine APIs (`game`, `workspace`, `Players`, etc.) that the standalone `luau` runtime does not provide. This tool is only usable for pure-logic library modules with zero Roblox dependencies.
**Options:**
- `-O<n>` -- Optimization level 0-2 (default 1). Same as compile.
- `-g<n>` -- Debug level 0-2 (default 1). Same as compile.
- `--codegen` -- Execute using native code generation.
- `--profile[=N]` -- CPU profiling at N Hz sampling (default 10000). Outputs `profile.out`.
- `--coverage` -- Code coverage collection. Outputs `coverage.out`. Can identify dead code paths.
- `--timetrace` -- Compiler time tracing to `trace.json`.
- `-i, --interactive` -- REPL after script execution.
- `-a, --program-args` -- Pass arguments to the Luau program.
**Key invocations (only for pure-logic modules without Roblox API dependencies):**
```
luau --codegen --profile -O2 <file> # Profile with native codegen
luau --coverage -O2 <file> # Dead code detection
```
**For Roblox scripts:** Use Roblox Studio's Script Profiler and `debug.dumpcodesize()` instead. The command should note this when the target file uses Roblox APIs.
#### `luau-ast` -- AST dump
**No flags.** Takes a single file, outputs full JSON AST.
- Node types, source locations, variable scopes, type annotations
- Programmatically parseable for: function nesting depth, function body size (monolith detection), closure captures (upvalue analysis), table construction patterns, loop-invariant expression detection
**Key invocation:**
```
luau-ast <file> # Full JSON AST
```
### Reference Documentation
The command will reference these docs (read during execution if needed for detail):
**Core optimization:**
- `.cursor/rules/luau.mdc` -- Luau language rules, optimization tips, performance patterns
- `.cursor/rules/luau/function-inlining.md` -- Inlining RFC (no user `@inline`, automatic at `-O2`, requirements for inlineability)
- `.cursor/rules/luau/syntax-attribute-functions-native.md` -- `@native` per-function attribute RFC
- `.cursor/rules/roblox/en-us/luau/native-code-gen.md` -- Native codegen docs (size limits, type annotation impact, gotchas)
**Type system (old typechecker only):**
- `.cursor/rules/roblox/en-us/luau/type-checking.md` -- Type annotation syntax and modes
- `.cursor/rules/luau/generic-functions.md` -- Generic function syntax
- `.cursor/rules/luau/never-and-unknown-types.md` -- `never`/`unknown` for exhaustiveness
- `.cursor/rules/luau/syntax-type-ascription.md` -- `::` cast operator
**Library optimizations:**
- `.cursor/rules/luau/function-table-create-find.md` -- `table.create`, `table.find`
- `.cursor/rules/luau/function-table-clear.md` -- `table.clear` for reuse
- `.cursor/rules/luau/function-table-clone.md` -- `table.clone` vs manual copy
- `.cursor/rules/luau/function-table-freeze.md` -- `table.freeze` for readonly
- `.cursor/rules/luau/function-math-lerp.md` -- `math.lerp` properties and correctness
- `.cursor/rules/luau/function-bit32-byteswap.md` -- CPU-level byte swap
- `.cursor/rules/luau/function-bit32-countlz-countrz.md` -- CPU-level bit counting
- `.cursor/rules/luau/function-buffer-bits.md` -- Buffer bit operations
- `.cursor/rules/luau/type-byte-buffer.md` -- `buffer` type for binary data
- `.cursor/rules/luau/vector-library.md` -- Vector library and fastcalls
- `.cursor/rules/luau/function-string-pack-unpack.md` -- Binary string operations
- `.cursor/rules/luau/syntax-floor-division-operator.md` -- `//` dedicated opcode
- `.cursor/rules/luau/syntax-compound-assignment.md` -- `+=` single LHS evaluation
**Deprecation/avoidance:**
- `.cursor/rules/luau/deprecate-getfenv-setfenv.md` -- Why fenv disables optimizations
- `.cursor/rules/luau/deprecate-table-getn-foreach.md` -- Deprecated table functions
- `.cursor/rules/luau/generalized-iteration.md` -- Modern iteration patterns
**Roblox performance (always applicable):**
- `.cursor/rules/roblox/en-us/performance-optimization/improve.md` -- Script computation, memory, physics, rendering, networking, assets
- `.cursor/rules/roblox/en-us/performance-optimization/design.md` -- Event-driven patterns, frame budgets (~16.67ms at 60 FPS), multithreading
- `.cursor/rules/roblox/en-us/luau/variables.md` -- Local vs global: "global in a time-critical loop can make it perform more than 10% slower"
- `.cursor/rules/roblox/en-us/luau/scope.md` -- Scope performance implications
- `.cursor/rules/roblox/en-us/luau/native-code-gen.md` -- `@native` attribute, Vector3 annotation impact, size limits, `debug.dumpcodesize()`

1034
.cursor/plans/luau-style_ref_paths_845083e1.plan.md Normal file
View file

File diff suppressed because it is too large Load diff

275
.cursor/plans/matching_algorithm_parity_01524d4c.plan.md Normal file
View file

@ -0,0 +1,275 @@
---
name: matching algorithm parity
overview: Fix scoring parity between the 3 matching algorithm implementations (Rust forward sync, Rust syncback, Lua plugin) and add parity/round-trip tests.
todos:
- id: fix-1-ref-scoring
content: "Fix 1: Add Ref-target-aware scoring to Rust countOwnDiffs in both forward sync and syncback matching"
status: completed
- id: fix-2-tags-attrs
content: "Fix 2: Add granular Tags/Attributes scoring to Rust countOwnDiffs in both implementations"
status: completed
- id: fix-3-epsilon
content: "Fix 3: Replace float_cmp::approx_eq! with custom fuzzy_eq matching Lua's 0.0001 epsilon in variant_eq.rs"
status: completed
- id: fix-4-sort
content: "Fix 4: Add matched_indices sort to forward sync build_result; remove TODO from syncback"
status: completed
- id: fix-5-vcache
content: "Fix 5: Cache VCache at session level in Lua matching.lua to avoid redundant decoding"
status: completed
- id: fix-6-tests
content: "Fix 6: Add parity tests (Rust forward vs syncback), round-trip matching test, and Lua parity spec test"
status: completed
- id: run-ci
content: Run /ci after all fixes to verify CI passes
status: completed
isProject: false
---
# Matching Algorithm Parity Audit Fixes
> This plan was generated by `/audit` from a Plan-mode session.
> Refer to `.cursor/rules/atlas.mdc` for full project standards.
## Standards
These standards govern every fix in this plan. An implementer MUST read `.cursor/rules/atlas.mdc` before starting.
### Round-Trip Identity (from `atlas.mdc` Quality Standard)
Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a **bit-identical instance tree** -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.
### CLI Syncback Parity (from `atlas.mdc` Two-Way Sync Strategy)
Plugin-based sync must produce **exactly the same filesystem output** that `atlas syncback` would give for the same input. Byte-for-byte identical files, identical directory structures, identical naming. Any divergence is a bug until proven otherwise.
### Code Quality (from `atlas.mdc` Code Quality Standard)
DRY code that is easy to reason about and hard to get wrong. The same slugify/dedup/meta-update pattern should not be copy-pasted across files. Small refactors (2-3 call sites) are included as fixes. Major rewrites (10+ call sites) are deferred.
## Context
The matching algorithm was audited across all 3 parallel implementations:
- `src/snapshot/matching.rs` (Rust forward sync)
- `src/syncback/matching.rs` (Rust syncback)
- `plugin/src/Reconciler/matching.lua` (Lua plugin)
The audit found **scoring parity issues** that could cause different implementations to produce different pairings for the same logical input. The most critical is Ref property scoring (UNMATCHED_PENALTY in Lua vs +1 in Rust), followed by Tags/Attributes granularity and float epsilon differences.
## Fixes
### Fix 1: Add Ref-Target-Aware Scoring to Rust `countOwnDiffs`
- **Status:** Approved
- **Finding:** Step 11l: Matching Algorithm Parity -- Ref property scoring
- **Files:** `src/snapshot/matching.rs`, `src/syncback/matching.rs`
- **Problem:** Lua `countOwnDiffs` (matching.lua lines 309-325) scores Ref property mismatches at `UNMATCHED_PENALTY = 10000` by comparing target Name+ClassName. Rust `count_own_diffs` in both files treats Ref properties as regular Variant properties scoring +1 per mismatch. In syncback matching, Ref IDs are always different between WeakDom instances, so ALL Ref properties score +1 regardless of logical equivalence.
- **Solution:**
- In **forward sync** `count_own_diffs` (matching.rs line 452-496): When comparing properties, detect `Variant::Ref` values. For each Ref property:
- If both sides have a Ref value, resolve both Refs to their target instances (via `tree.get_instance(ref)` for tree side, look up snapshot children by snapshot_id for snap side). Compare target Name+ClassName. If different, add `UNMATCHED_PENALTY` instead of 1.
- If one side has Ref and other doesn't (or Ref::none vs valid ref), add `UNMATCHED_PENALTY`.
- Skip `Variant::Ref` from regular property comparison to avoid double-counting.
- In **syncback** `count_own_diffs` (matching.rs line 501-548): Same approach using `new_dom.get_by_ref(ref)` and `old_dom.get_by_ref(ref)` to resolve targets.
- Both functions need access to the DOM/tree for Ref resolution -- add the necessary parameters.
- **Risk:** Adding DOM access to `count_own_diffs` increases its parameter count. The function is `#[inline]`, so ensure the extra parameter doesn't cause performance regression. The Ref resolution is a simple hash lookup (O(1)).
- **Verify round-trip:** Create 3 Models named "Weapon" with different PrimaryPart targets. Syncback, then forward sync. Verify all 3 match to their correct counterparts.
- **Verify syncback parity:** Compare CLI syncback output with two-way sync output for a tree with Ref-differentiated instances.
- **Tests required:**
- Rust unit test: `ref_scoring_forward_sync` -- 2 Models with different PrimaryPart targets, verify correct pairing
- Rust unit test: `ref_scoring_syncback` -- same scenario for syncback matching
- Lua spec test: already has ref scoring test in matching.spec.lua, verify it passes
### Fix 2: Add Granular Tags/Attributes Scoring to Rust `countOwnDiffs`
- **Status:** Approved
- **Finding:** Step 11l: Matching Algorithm Parity -- Tags/Attributes granularity
- **Files:** `src/snapshot/matching.rs`, `src/syncback/matching.rs`
- **Problem:** Lua `countOwnDiffs` counts per-tag (+1 each in symmetric difference) and per-attribute (+1 each diff). Rust counts Tags as one Variant property (+1 total) and Attributes as one Variant property (+1 total). An instance with 5 extra tags scores +5 in Lua but +1 in Rust.
- **Solution:**
- In both Rust `count_own_diffs` functions:
- When encountering `Variant::Tags(tags)`, extract and compare granularly: +1 per tag in symmetric difference. Skip from regular property comparison.
- When encountering `Variant::Attributes(attrs)`, extract and compare granularly: +1 per attribute whose value differs (using `variant_eq`), +1 per attribute present only on one side. Skip from regular property comparison.
- The key check is `key == "Tags"` (for `Variant::Tags`) and `key == "Attributes"` (for `Variant::Attributes`).
- **Risk:** Low. Only changes the cost value for Tags/Attributes, making it more discriminating. Existing tests should still pass since they don't depend on exact cost values, only on correct pairings.
- **Verify round-trip:** Instances differing by tags should be paired correctly.
- **Verify syncback parity:** N/A -- Tags/Attributes rarely distinguish instances in typical syncback scenarios.
- **Tests required:**
- Rust unit test: `tags_granular_scoring` -- 2 Parts with different tag sets, verify scoring matches Lua behavior
- Rust unit test: `attributes_granular_scoring` -- 2 Parts with different attribute maps, verify scoring
### Fix 3: Loosen Rust Float Epsilon to Match Lua
- **Status:** Approved (Modified: loosen Rust to match Lua)
- **Finding:** Step 11l: Matching Algorithm Parity -- Float epsilon
- **Files:** `src/variant_eq.rs`
- **Problem:** Lua `trueEquals` uses epsilon=0.0001 with absolute+relative check. Rust `variant_eq` uses `float_cmp::approx_eq!` defaults (epsilon~1.19e-7, ulps=4), which is ~1000x tighter. This means Rust considers values different that Lua considers equal, potentially causing different match scores and different pairings.
- **Solution:**
- Replace the `approx_eq!` macro wrapper in `variant_eq.rs` with a custom `fuzzy_eq` function that matches Lua's dual epsilon approach:
```rust
const EPSILON: f32 = 0.0001;
fn fuzzy_eq_f32(a: f32, b: f32) -> bool {
if a.is_nan() && b.is_nan() { return true; }
let diff = (a - b).abs();
let max_val = a.abs().max(b.abs()).max(1.0);
diff < EPSILON || diff < max_val * EPSILON
}
```
- Apply similar `fuzzy_eq_f64` for f64 comparisons.
- Update all float comparison call sites in `variant_eq` (Color3, CFrame, Vector3, UDim, etc.) to use the new function instead of `approx_eq!`.
- Remove the `float-cmp` dependency from `Cargo.toml` if no other code uses it.
- **Risk:** Medium. Loosening epsilon means values that were previously considered different are now equal. This affects:
- `variant_eq` used in matching (intended -- this fix)
- `variant_eq` used in `patch_compute.rs` for property diff detection (side effect -- could suppress legitimate property patches for near-equal values)
- Verify that the 0.0001 epsilon doesn't suppress meaningful Studio property changes.
- **Verify round-trip:** Float properties should round-trip without phantom diffs (the looser epsilon should REDUCE phantom diffs, not create them).
- **Verify syncback parity:** N/A
- **Tests required:**
- Rust unit test: update existing `variant_eq` float tests to verify the new epsilon behavior
- Rust unit test: `fuzzy_eq_matches_lua` -- verify same boundary values as `trueEquals.spec.lua` (0.0001 threshold, relative scaling, NaN handling)
### Fix 4: Add Sort to Forward Sync `build_result` for Consistency
- **Status:** Approved
- **Finding:** Code quality -- inconsistent sort between implementations
- **Files:** `src/snapshot/matching.rs`, `src/syncback/matching.rs`
- **Problem:** Syncback `build_result` (line 217) sorts `matched_indices` by new_ref index. Forward sync `build_result` (line 206) does NOT sort. The Lua `matchChildren` returns matched pairs in insertion order (which is group iteration order, then sorted within groups).
- **Solution:**
- Add `matched_indices.sort_by_key(|&(si, _)| si);` to the forward sync `build_result` function (line 203, before the matched Vec construction).
- Remove the TODO comment from the syncback version (line 217): the sort is intentional for consistent downstream iteration order.
- **Risk:** Very low. Only changes the ORDER of matched pairs in the result, not WHICH pairs are matched. Downstream code iterates matched pairs for patch computation, and order should not affect correctness.
- **Verify round-trip:** N/A -- order of matched pairs doesn't affect round-trip identity.
- **Verify syncback parity:** N/A
- **Tests required:**
- No new tests needed. Existing tests verify correct pairings, not ordering.
### Fix 5: Cache VCache at Session Level in Lua Matching
- **Status:** Approved
- **Finding:** Performance -- redundant `cacheVirtual` calls
- **Files:** `plugin/src/Reconciler/matching.lua`
- **Problem:** `cacheVirtual` is called during ambiguous group scoring (line 492) AND again inside `computePairCost` (line 362). The same virtual instance's properties are decoded multiple times per session.
- **Solution:**
- Add a `vCacheStore: { [string]: VCache }` field to `MatchingSession`.
- In `cacheVirtual`, check `session.vCacheStore[virtualId]` first. If present, return cached. Otherwise, compute and store.
- Update `computePairCost` to use the session-level cache instead of re-calling `cacheVirtual`.
- The `classKeys` parameter varies by group, but for a given virtualId the classKeys is always the same (determined by ClassName). So caching by virtualId is safe.
- **Risk:** Very low. Pure performance optimization. The VCache content is deterministic for a given virtualId.
- **Verify round-trip:** N/A
- **Verify syncback parity:** N/A
- **Tests required:**
- No new tests needed. Existing matching.spec.lua tests verify correctness. Could add a timing test but not required.
### Fix 6: Add Parity Tests Between All 3 Implementations
- **Status:** Approved
- **Finding:** Steps 5-6: Missing parity and round-trip tests
- **Files:** `tests/tests/matching_fixtures.rs` (extend), `plugin/src/Reconciler/matching.spec.lua` (extend)
- **Problem:** No tests verify that the 3 implementations produce identical pairings for the same input. Given Fixes 1-3 close the scoring gaps, parity tests are needed to prevent future divergence.
- **Solution:**
- **Rust integration test** (`tests/tests/matching_fixtures.rs`): Add a test `parity_forward_vs_syncback` that:
1. Creates a WeakDom with an ambiguous group (e.g., 4 Parts named "Line" with different Transparency values, in reversed order)
2. Runs syncback matching on `(new_children, old_children)`
3. Builds a RojoTree from `new_children` snapshots, runs forward sync matching on `(old_children_as_snapshots, tree_children)`
4. Verifies both produce the same pairings (by comparing matched Name+Property values)
- **Lua spec test** (`matching.spec.lua`): Add a test `parity_with_rust_fixture` that:
1. Uses the same input data as the Rust test (4 Parts with specific Transparency values)
2. Runs Lua matching
3. Verifies the same pairings as the Rust tests (by property comparison)
- **Round-trip matching test** (`tests/tests/matching_fixtures.rs`): Add `round_trip_matching_identity`:
1. Uses the UFOWave fixture (12 Textures with default-omitted Face)
2. Runs syncback to produce filesystem
3. Runs forward sync on the filesystem output
4. Verifies all 12 Textures are matched to their correct counterparts (by Face value)
- **Risk:** None. Tests only.
- **Verify round-trip:** This IS the round-trip test.
- **Verify syncback parity:** N/A (these are matching-level tests, not filesystem-level)
- **Tests required:** (this fix IS the test)
## Skipped Fixes
None. All identified fixes were approved.
## Accepted Limitations
### Greedy Matching Not Globally Optimal
- **Limitation:** The greedy assignment algorithm is not globally optimal. Edge cases exist where a locally-best pick forces a globally-worse assignment.
- **Scenario:** Large ambiguous groups (10+ instances) with specific cost patterns where the Hungarian algorithm would produce different (better) pairings.
- **User decision:** Accepted for all releases (documented in `atlas-dedup.mdc`).
- **Mitigation:** Ambiguous groups are typically small (2-10 instances). The depth limit (MAX_SCORING_DEPTH=3) bounds worst-case performance. Monitoring via test coverage for large groups.
## Deferred Refactors
- **Shared matching core:** The 3 implementations share the same algorithm but differ in input types (InstanceSnapshot vs WeakDom vs Studio Instance). A generic/trait-based approach could consolidate the duplicated logic (~2700 total lines), but the type differences are fundamental to each context. Estimated scope: 10+ files, significant API redesign. Deferred.
- **Duplicated grouping pattern:** All 3 implementations repeat the same HashMap-based `(Name, ClassName) -> Vec<index>` grouping. Could be factored into a shared helper, but types differ. Minor DRY violation. Deferred.
## Test Plan
### Rust Unit Tests (`#[cfg(test)]` blocks)
- `ref_scoring_forward_sync` -- 2 Models with different PrimaryPart targets, verify correct pairing (Fix 1)
- `ref_scoring_syncback` -- same scenario for syncback matching (Fix 1)
- `tags_granular_scoring` -- 2 Parts with different tag sets, verify scoring (Fix 2)
- `attributes_granular_scoring` -- 2 Parts with different attribute maps, verify scoring (Fix 2)
- Update existing `variant_eq` float tests for new epsilon behavior (Fix 3)
- `fuzzy_eq_matches_lua` -- verify same boundary values as `trueEquals.spec.lua` (Fix 3)
### Rust Integration Tests (`tests/tests/`)
- `parity_forward_vs_syncback` -- same ambiguous group through both Rust matchers, verify identical pairings (Fix 6)
- `round_trip_matching_identity` -- UFOWave fixture through syncback then forward sync, verify correct pairings (Fix 6)
### Lua Spec Tests (`.spec.lua` files)
- `parity_with_rust_fixture` -- same input as Rust parity test, verify identical pairings (Fix 6)
### Snapshot Tests (`insta` crate)
- None needed for matching fixes.
### Failing Tests for Known Limitations
- None. The greedy limitation is accepted and no test is needed to demonstrate it.
## Test Rules
The implementer MUST follow these rules when writing tests for this plan:
### For bugs found and fixed
Every bug fix MUST have a test that:
1. Would have **FAILED** before the fix
2. **PASSES** after the fix
3. Prevents the bug from regressing
### For missing coverage identified
For each "missing test coverage" gap:
1. Write the test
2. Verify it passes against the current implementation
3. If it fails, investigate -- it may have found another bug
### For known limitations (Accepted Limitations section)
No known-limitation tests needed for this plan.
### Scope: test every affected layer
For each finding, determine which test layers are affected and write tests in ALL of them:
- Does the bug manifest at the plugin level? -> Lua spec test
- Does it manifest at the server level? -> Rust unit test or integration test
- Does it affect the round-trip? -> Integration test with `/api/write` + filesystem verification
- Does it affect forward-sync? -> Serve test with `recv_socket_packet`
- Could it regress from a different code path? -> Test that code path too
## Final Step: Run CI
After ALL fixes and tests are implemented, run the `/ci` command (see `.cursor/commands/ci.md`) to execute the full CI pipeline. Every fix must pass CI before the plan is considered complete. Do not skip this step.

324
.cursor/plans/matching_cost_semantics_376e45e2.plan.md Normal file
View file

@ -0,0 +1,324 @@
---
name: matching cost semantics
overview: "Split `variant_eq` into two variants: keep fuzzy comparison for Studio-facing operations (patch_compute, api, property_filter), and add a new disk-representation comparison for both Rust matching files where cost should reflect actual file changes."
todos:
- id: expose-fns
content: Make format_f32/format_f64 pub(crate) in json.rs and cleanup_f32 pub(crate) in resolution.rs
status: completed
- id: disk-eq-helpers
content: Add disk_eq_f32, disk_eq_f32_cleaned, disk_eq_f64 zero-allocation helpers to variant_eq.rs
status: completed
- id: variant-eq-disk
content: Implement variant_eq_disk in variant_eq.rs dispatching per Variant type (Float32 standalone vs composite cleanup)
status: completed
- id: update-fwd-matching
content: Replace variant_eq with variant_eq_disk in src/snapshot/matching.rs (count_own_diffs, diff_variant_pair, count_attributes_diff)
status: completed
- id: update-sb-matching
content: Replace variant_eq with variant_eq_disk in src/syncback/matching.rs (count_own_diffs, diff_variant_pair, attribute diff)
status: completed
- id: tests-veq
content: "Add variant_eq_disk unit tests: fuzzy-vs-disk divergence, cleanup normalization, NaN/Inf/-0, all composite types"
status: completed
- id: tests-fwd-large
content: Add 50+ same-named instance matching tests to src/snapshot/matching.rs with 5 variation groups
status: completed
- id: tests-sb-large
content: Add 50+ same-named instance matching tests to src/syncback/matching.rs with 5 variation groups
status: completed
- id: tests-disk-boundary
content: Add tests with values at format_f32 boundaries (close-but-different-on-disk, different-but-same-on-disk)
status: completed
- id: tests-update-existing
content: Update existing matching test assertions to match variant_eq_disk semantics
status: completed
isProject: false
---
# Matching Cost Semantics Fix
## Problem
`variant_eq` currently uses fuzzy epsilon comparison (absolute + relative, `EPSILON = 0.0001`) in both Rust matching files. This is wrong because:
- **Rust cost = "will this value result in a file change?"** Two values that format identically on disk should score 0 cost, even if their raw f32 bits differ. Conversely, two values within epsilon but with different disk representations should score 1.
- **Lua cost = "will this change be applied in Studio?"** Fuzzy epsilon is correct here -- it prevents oscillation from Studio float precision rounding.
The fuzzy epsilon can diverge from disk reality in both directions:
- `format_f32(1.00005)` = `"1.00005"`, `format_f32(1.00015)` = `"1.00015"` -- different on disk, but `fuzzy_eq_f32` says "equal" (diff = 0.0001)
- `format_f32(0.123456)` could round both `0.1234561` and `0.1234559` to the same string -- same on disk, but `fuzzy_eq_f32` might say "different"
## Data Flow Analysis
### Disk serialization pipeline (how floats reach files)
```
Variant value
|
v
resolution.rs::from_variant()
|-- Standalone Float32 → AmbiguousValue::Number32(v) [NO cleanup]
|-- Standalone Float64 → AmbiguousValue::Number(v as f64) [NO cleanup]
|-- Composite types (Vector3, Color3, CFrame, UDim, etc.)
| → cleanup_f32(component) then into Array/FullyQualified
v
json.rs::Json5ValueSerializer
|-- serialize_f32 → format_f32(v) [6 significant digits, lexical-write-float]
|-- serialize_f64 → format_f64(v) [full precision]
v
.meta.json5 / .model.json5 on disk
```
`**cleanup_f32` matters for composites.** It rounds to 6 decimal places (via `format!("{:.6}")` round-trip), which zeros out very small values. Example: `cleanup_f32(0.0000001)` = `0.0`, so `format_f32(0.0)` = `"0"` vs `format_f32(0.0000001)` = `"1e-7"` -- completely different disk output.
### Where values come from in each matching file
`**[src/syncback/matching.rs](src/syncback/matching.rs)`:**
- `new_dom` = from rbxl/rbxm binary parse. Raw Studio values, NOT through resolution.rs
- `old_dom` = from filesystem build. Values already in disk-normalized form
`**[src/snapshot/matching.rs](src/snapshot/matching.rs)`:**
- `snap` = from filesystem re-snapshot. Values read from disk, already in disk form
- `tree` = from RojoTree. Could be from initial forward sync (disk form) or two-way sync push (raw Studio values)
In both cases, at least one side may not have been through `cleanup_f32 + format_f32`. The comparison must normalize both sides through the full disk pipeline.
### Other callers of `variant_eq` (KEEP fuzzy -- different semantics)
| File | Purpose | Why fuzzy is correct |
| ------------------------------------------------------- | ----------------------------------- | -------------------------------------------------------------- |
| `[patch_compute.rs](src/snapshot/patch_compute.rs)` | Compute patches to send to plugin | Must agree with Lua `diff.lua` (which uses fuzzy `trueEquals`) |
| `[api.rs](src/web/api.rs)` | Filter defaults during two-way sync | "Is this at default?" -- not a disk question |
| `[property_filter.rs](src/syncback/property_filter.rs)` | Filter defaults during syncback | Same rationale |
| `[hash/mod.rs](src/syncback/hash/mod.rs)` | Exclude defaults from content hash | Same rationale |
| `[project.rs](src/snapshot_middleware/project.rs)` | Check project node value drift | Same rationale |
## Implementation
### 1. Expose formatting functions
- `**[src/json.rs](src/json.rs)`:** Make `format_f32`, `format_f64` and the `Options` constants `pub(crate)`
- `**[src/resolution.rs](src/resolution.rs)`:** Make `cleanup_f32` `pub(crate)`
### 2. Add disk-representation comparison to `[src/variant_eq.rs](src/variant_eq.rs)`
Add zero-allocation comparison helpers that write to stack buffers:
```rust
fn disk_eq_f32(a: f32, b: f32) -> bool {
if a.to_bits() == b.to_bits() { return true; }
// format both into stack buffers, compare byte slices
}
fn disk_eq_f32_cleaned(a: f32, b: f32) -> bool {
disk_eq_f32(cleanup_f32(a), cleanup_f32(b))
}
fn disk_eq_f64(a: f64, b: f64) -> bool {
if a.to_bits() == b.to_bits() { return true; }
// format both into stack buffers, compare byte slices
}
```
Then `pub fn variant_eq_disk(a: &Variant, b: &Variant) -> bool` that dispatches:
| Variant type | Comparison |
| ----------------------------------------------- | -------------------------------------------------- |
| Float32 | `disk_eq_f32` (no cleanup -- standalone) |
| Float64 | `disk_eq_f64` (no cleanup) |
| Vector3, Color3, CFrame components | `disk_eq_f32_cleaned` (cleanup -- composite) |
| UDim scale / UDim2 scales | `disk_eq_f32_cleaned` (cleanup) |
| NumberRange, Rect min/max | `disk_eq_f32_cleaned` (cleanup) |
| NumberSequence, ColorSequence keypoint floats | `disk_eq_f32_cleaned` (cleanup) |
| PhysicalProperties floats | `disk_eq_f32_cleaned` (cleanup) |
| Ray, Region3 components | `disk_eq_f32_cleaned` (cleanup) |
| Non-float types (Bool, Int, String, Enum, etc.) | Exact equality (no epsilon) |
| Tags | Sorted set comparison (same as now) |
| Attributes | Recursive with `variant_eq_disk` for nested values |
### 3. Update both matching files
`**[src/snapshot/matching.rs](src/snapshot/matching.rs)`:** Replace all `variant_eq` calls in `count_own_diffs`, `diff_variant_pair`, and `count_attributes_diff` with `variant_eq_disk`.
`**[src/syncback/matching.rs](src/syncback/matching.rs)`:** Same replacements in `count_own_diffs`, `diff_variant_pair`, and attribute comparison.
### 4. Lua side -- no changes needed (verified)
The plugin's cost model is correct as-is:
- `trueEquals` is shared by both `matching.lua` (scoring) and `diff.lua` (patch generation), so they cannot disagree -- if matching says "equal", diff will also say "equal", and the change won't appear in the patch visualizer
- The fuzzy epsilon prevents Studio float-precision oscillation (set 0.5 -> Studio stores 0.49999999 -> reads back as ~0.5 -> trueEquals says "equal" -> no phantom re-diff)
- `matching.lua`'s `countOwnDiffs` fills defaults from `classKeys.defaults` for missing virtual properties, matching what diff/hydrate will do
The Lua side's cost is correctly "will this change be flagged in the patch visualizer / applied to Studio."
### 5. Testing Suite
Existing `variant_eq` tests stay unchanged (they test the fuzzy function which is still used by patch_compute, api, etc.). All new tests below are additive.
#### 5a. `variant_eq_disk` unit tests (in `src/variant_eq.rs`)
**Fuzzy-vs-disk divergence tests** -- these are the cases that prove fuzzy is wrong for file-change detection:
```rust
// Fuzzy says "equal" but disk would differ
assert!(fuzzy_eq_f32(1.00005, 1.00015)); // within epsilon
assert!(!disk_eq_f32(1.00005, 1.00015)); // format_f32 gives different strings
// Fuzzy says "different" but disk would be identical
assert!(!fuzzy_eq_f32(0.1234561, 0.1234559)); // outside relative epsilon
assert!(disk_eq_f32(0.1234561, 0.1234559)); // format_f32 rounds both the same
```
**cleanup_f32 normalization tests** -- composite types get cleaned before formatting:
```rust
// Very small values zeroed by cleanup
assert!(disk_eq_f32_cleaned(0.0000001, 0.0)); // cleanup makes both 0.0
assert!(!disk_eq_f32(0.0000001, 0.0)); // without cleanup, different
// Full Vector3 comparison
assert!(variant_eq_disk(
&Variant::Vector3(Vector3::new(1.0, 2.0, 0.0000001)),
&Variant::Vector3(Vector3::new(1.0, 2.0, 0.0)),
)); // cleanup zeros the tiny z component
```
**Edge cases:**
- NaN == NaN (both format to "NaN")
- Inf == Inf (both format to "Infinity")
- -0.0 == 0.0 (both format to "0")
- All composite types: Vector2, Vector3, Color3, CFrame, UDim, UDim2, Rect, Ray, Region3, NumberRange, NumberSequence, ColorSequence, PhysicalProperties
- Non-float types: exact equality (Bool, Int32, Int64, String, Enum, BrickColor)
- Tags: sorted set comparison
- Attributes: recursive disk comparison of nested Variants
#### 5b. Large ambiguous group matching -- forward sync (`src/snapshot/matching.rs`)
**Test: `fifty_same_name_parts_five_variation_groups`**
50 Parts all named `"Line"` with 5 variation groups of 10 exact duplicates each:
| Group | Position | Color | Count |
| ----- | ------------ | ------------------- | ----- |
| A | `[0, 0, 0]` | `[1, 0, 0]` (red) | 10 |
| B | `[0, 5, 0]` | `[1, 0, 0]` (red) | 10 |
| C | `[0, 10, 0]` | `[0, 1, 0]` (green) | 10 |
| D | `[0, 0, 0]` | `[0, 1, 0]` (green) | 10 |
| E | `[0, 5, 0]` | `[0, 0, 1]` (blue) | 10 |
- Groups A vs D: same position, different color (color-only diff)
- Groups A vs B: same color, different position (position-only diff)
- Groups B vs E: same position, different color
- Within each group: 10 exact duplicates (any pairing is correct)
Build snapshots in order A,B,C,D,E. Build tree in **reversed** order E,D,C,B,A.
**Assertions:**
1. All 50 matched, 0 unmatched on either side
2. For every matched pair: `Position` values match (within disk equality)
3. For every matched pair: `Color` values match (within disk equality)
4. Completes in < 5 seconds (depth limit prevents exponential blowup)
**Test: `fifty_parts_position_only_variations`**
50 Parts named `"Block"` -- same Color on all, 5 distinct Position values (10 duplicates each). Tree reversed. Verify all 50 pair correctly by Position.
**Test: `fifty_parts_color_only_variations`**
50 Parts named `"Block"` -- same Position on all, 5 distinct Color values (10 duplicates each). Tree reversed. Verify all 50 pair correctly by Color.
**Test: `sixty_parts_with_near_float_values`**
60 Parts named `"Segment"` with Transparency values chosen at `format_f32` boundaries:
| Group | Transparency | Count | Notes |
| ----- | ------------ | ----- | ------------------------------------------------ |
| F | `0.100000` | 12 | disk repr: `"0.1"` |
| G | `0.100001` | 12 | disk repr: `"0.100001"` -- different from F |
| H | `0.500000` | 12 | disk repr: `"0.5"` |
| I | `0.500001` | 12 | disk repr differs from H |
| J | `0.999999` | 12 | disk repr rounds to same or different from `1.0` |
The exact Transparency values will be chosen by testing `format_f32` output during implementation to find values that:
- Are within fuzzy epsilon of each other but format differently (must NOT cross-pair)
- Are outside fuzzy epsilon but format the same (must cross-pair)
This test specifically catches the bug where fuzzy equality would match Group F with Group G instances, causing 12 unnecessary file rewrites.
#### 5c. Large ambiguous group matching -- syncback (`src/syncback/matching.rs`)
Same test matrix as 5b but using `WeakDom` + `InstanceBuilder`:
**Test: `fifty_same_name_parts_five_groups_syncback`**
Same 5 variation groups (A-E) of 10 Parts each. `new_dom` in order, `old_dom` reversed.
```rust
let mut new_dom = WeakDom::new(InstanceBuilder::new("DataModel"));
let new_root = new_dom.root_ref();
for (pos, color) in &groups {
for _ in 0..10 {
new_dom.insert(new_root, InstanceBuilder::new("Part")
.with_name("Line")
.with_property("Position", Variant::Vector3(*pos))
.with_property("Color", Variant::Color3(*color)));
}
}
```
Assertions identical to 5b: all 50 paired, Position+Color match on every pair.
**Test: `sixty_parts_near_float_syncback`**
Same boundary test as 5b's `sixty_parts_with_near_float_values`, but in syncback context.
#### 5d. Disk boundary stress test
**Test: `disk_representation_boundary_matching`** (in both matching files)
Explicitly construct instances where fuzzy and disk comparison disagree, and verify matching picks the disk-correct pairing:
2 Parts named `"Edge"`:
- Snap/New A: Transparency = `1.00005` (format_f32 = `"1.00005"`)
- Snap/New B: Transparency = `1.00015` (format_f32 = `"1.00015"`)
Tree/Old (reversed):
- Tree/Old X: Transparency = `1.00015`
- Tree/Old Y: Transparency = `1.00005`
With fuzzy equality: both (A,X) and (A,Y) score 0 (within epsilon) -- random pairing, 50% chance of wrong.
With disk equality: (A,Y) scores 0 and (A,X) scores 1 -- deterministically correct.
Verify A matches Y and B matches X.
#### 5e. Update existing tests
Existing matching tests that assert `variant_eq` behavior in matching contexts will continue to pass because:
- Most use exact integer/enum values (not affected by fuzzy vs disk)
- Float tests use values that are either identical or far apart
Tests that use float values at fuzzy epsilon boundaries may need value adjustments if the old assertion relied on fuzzy tolerance. Specifically review:
- `many_same_name_parts_matched_by_properties` (uses `i * 0.1` increments -- safe, these format distinctly)
- `ten_ambiguous_textures_stress` (uses Enum values -- safe, not float)
- `variant_eq_float32_with_new_epsilon` (tests `variant_eq` not `variant_eq_disk` -- unchanged)
No existing test should break, but verify during implementation.

609
.cursor/plans/matching_session_caching_cbfa7bda.plan.md Normal file
View file

@ -0,0 +1,609 @@
---
name: Matching session caching
overview: Rewrite the Lua matching module for native codegen performance (--!strict, --!native, --!optimize 2) with session-based caching, clean 2-function mutual recursion, concrete types, and local functions for inlining. Add session caching to both Rust implementations for consistency.
todos:
- id: lua-trueEquals
content: "Rewrite trueEquals.lua: --!strict/--!native/--!optimize 2, single typeof dispatch, dedicated helper functions per type (no temp arrays), adopt DeepEqualsPureUnsafe pattern for table comparison (rawlen/rawget/next, no checkedKeys set), keep fuzzy equality + null-ref + EnumItem cross-type"
status: completed
- id: lua-rewrite
content: "Rewrite matching.lua: --!strict/--!native/--!optimize 2, all functions as local function in dependency order (one forward declare for mutual recursion), 2-function mutual recursion (matchChildren + computePairCost), session caching, totalCost return, O(n) removeMatched, remove pcall in grouping"
status: completed
- id: lua-hydrate-threading
content: Thread session through hydrate.lua, Reconciler/init.lua, and ServeSession.lua (create session before hydrate call at line 628)
status: completed
- id: lua-verify-codegen
content: Run luau-compile --remarks on trueEquals.lua and matching.lua to verify inlining and native codegen quality, iterate until clean
status: completed
- id: lua-test
content: Build plugin, test Lua changes in Studio (hydrate + matching works correctly with session caching) before proceeding to Rust
status: completed
- id: rust-forward-session
content: "AFTER Lua passes: Add MatchingSession to src/snapshot/matching.rs with cost_cache, add total_cost to ForwardMatchResult, thread session through all functions and patch_compute.rs"
status: completed
- id: rust-syncback-session
content: "AFTER Lua passes: Add MatchingSession to src/syncback/matching.rs with cost_cache, add total_cost to MatchResult, thread session through all functions and dir.rs"
status: completed
- id: rust-test-updates
content: Update tests/tests/matching_fixtures.rs, run cargo test to verify Rust changes
status: completed
isProject: false
---
# Matching Session Caching + Native Codegen Rewrite
## Problem
1. **Redundant computation**: Scoring recurses into children, then hydrate re-calls matchChildren on the same children. The intermediate `_computeChildrenCost` calls matchChildren, iterates matched pairs calling `_computeChangeCount`, which recurses again -- all discarded after scoring.
2. **Poor native codegen structure**: All functions are on the `Matching` table (table lookup per call, uninlineable). No type annotations. No `--!strict`/`--!native`/`--!optimize 2`. A `pcall` with closure allocation in the hot grouping loop.
3. **4-function recursion chain** (`matchChildren -> _computeChildrenCost -> _computeChangeCount -> _computeChildrenCost -> matchChildren`) is complex and hard for both the optimizer and the developer to reason about.
## Design
### Conceptual Model
`matchChildren` is fundamentally `computeSubtreeCost` -- it finds the cost-minimizing pairing of children and returns the total cost. The actual match assignments are a bonus output. The 2-function mutual recursion reflects this:
- `**matchChildren(session, vChildren, sChildren, virtualInstances, parentVId, parentSInst)`**: Given two child sets, find optimal pairing. Returns `MatchResult` with `totalCost`.
- `**computePairCost(session, virtualId, studioInstance, virtualInstances, bestSoFar)`**: Given one virtual and one studio instance, compute total cost including subtree. Calls `matchChildren` for children cost.
`_computeChildrenCost` and `_computeChangeCount` are eliminated. Their logic is absorbed into these two functions.
### MatchingSession
```lua
type MatchingSession = {
matchCache: { [string]: { [Instance]: MatchResult } },
costCache: { [string]: { [Instance]: number } },
}
```
- **matchCache**: keyed by `(parentVirtualId, parentStudioInstance)`. Parent uniquely determines children set.
- **costCache**: keyed by `(virtualId, studioInstance)`. Only stored when `cost < bestSoFar` (exact, not early-exited).
Caching replaces the need for a depth limit: each unique pair/children-set is computed at most once.
### bestSoFar Interaction
- **costCache**: only store when computation was complete (`cost < bestSoFar`). If cached, return immediately regardless of caller's `bestSoFar` (the cached value IS the exact cost).
- **matchCache**: `matchChildren` always completes (no `bestSoFar` on it), so results are always cacheable.
---
## Lua Plugin Rewrite: [matching.lua](plugin/src/Reconciler/matching.lua)
### Module Header
```lua
--!strict
--!native
--!optimize 2
```
### Type Definitions (all concrete, minimal `any`)
```lua
type MatchPair = {
virtualId: string,
studioInstance: Instance,
}
type MatchResult = {
matched: { MatchPair },
unmatchedVirtual: { string },
unmatchedStudio: { Instance },
totalCost: number,
}
type MatchingSession = {
matchCache: { [string]: { [Instance]: MatchResult } },
costCache: { [string]: { [Instance]: number } },
}
type VCache = {
props: { [string]: any },
extraProps: { string }?,
tags: { [string]: boolean }?,
attrs: { [string]: any }?,
childCount: number,
}
type SCache = {
instance: Instance,
props: { [string]: any },
tags: { [string]: boolean }?,
attrs: { [string]: any }?,
children: { Instance },
childCount: number,
}
```
### All Functions as `local function` in Dependency Order
Every function is a `local function` defined in dependency order. No `Matching.foo()` table lookups. One forward declaration for the mutual recursion pair.
```lua
local UNMATCHED_PENALTY: number = 10000
local function newSession(): MatchingSession
return { matchCache = {}, costCache = {} }
end
local function cacheVirtual(vInst: any, classKeys: any): VCache
-- same logic as current _cacheVirtual, with return type annotation
end
local function cacheStudio(studioInstance: Instance, classKeys: any, extraProps: { string }?): SCache
-- same logic as current _cacheStudio
-- ADDITION: store sCache.instance = studioInstance
end
local function countOwnDiffs(vCache: VCache, sCache: SCache, classKeys: any): number
-- same logic as current _countOwnDiffsCached
end
local function removeMatched(arr: { any }, matchedIndices: { [number]: boolean }): ()
-- O(n) in-place compaction instead of O(n^2) table.remove loop
local write = 1
for read = 1, #arr do
if not matchedIndices[read] then
arr[write] = arr[read]
write += 1
end
end
for i = write, #arr do
arr[i] = nil
end
end
```
### Single Forward Declaration for Mutual Recursion
`computePairCost` calls `matchChildren` and vice versa. Define `matchChildren` first (forward-declared), then `computePairCost` as a `local function`, then assign `matchChildren`:
```lua
local matchChildren -- forward declare (matchChildren is defined after computePairCost)
local function computePairCost(...)
-- can reference matchChildren (captured as upvalue, assigned before first call)
end
matchChildren = function(...)
-- can reference computePairCost (already defined above)
end
```
### computePairCost (replaces `_computeChangeCount` + `_computeChildrenCost`)
```lua
computePairCost = function(
session: MatchingSession,
virtualId: string,
studioInstance: Instance,
virtualInstances: { [string]: any },
bestSoFar: number
): number
-- 1. Cache lookup
local vc = session.costCache[virtualId]
if vc then
local cached = vc[studioInstance]
if cached ~= nil then return cached end
end
local vInst = virtualInstances[virtualId]
if not vInst then return UNMATCHED_PENALTY end
-- 2. Own diffs
local classKeys = RbxDom.getClassComparisonKeys(vInst.ClassName)
local vCache = cacheVirtual(vInst, classKeys)
local sCache = cacheStudio(studioInstance, classKeys, vCache.extraProps)
local cost: number = countOwnDiffs(vCache, sCache, classKeys)
if cost >= bestSoFar then return cost end
-- 3. Children cost (direct matchChildren call, no intermediate function)
local vChildren = vInst.Children
local studioKids = sCache.children
if (not vChildren or #vChildren == 0) and #studioKids == 0 then
-- leaf: no children on either side
elseif not vChildren or #vChildren == 0 then
cost += #studioKids * UNMATCHED_PENALTY
elseif #studioKids == 0 then
for _, childId in vChildren do
if virtualInstances[childId] then cost += UNMATCHED_PENALTY end
end
else
local validVChildren: { string } = {}
for _, childId in vChildren do
if virtualInstances[childId] then
table.insert(validVChildren, childId)
end
end
local childResult = matchChildren(
session, validVChildren, studioKids, virtualInstances,
virtualId, studioInstance
)
cost += childResult.totalCost
end
-- 4. Cache if exact (no early exit)
if cost < bestSoFar then
if not session.costCache[virtualId] then
session.costCache[virtualId] = {}
end
session.costCache[virtualId][studioInstance] = cost
end
return cost
end
```
### matchChildren (restructured, returns totalCost)
```lua
matchChildren = function(
session: MatchingSession,
virtualChildren: { string },
studioChildren: { Instance },
virtualInstances: { [string]: any },
parentVirtualId: string?,
parentStudioInstance: Instance?
): MatchResult
-- 1. Cache lookup
if parentVirtualId and parentStudioInstance then
local pc = session.matchCache[parentVirtualId]
if pc then
local cached = pc[parentStudioInstance]
if cached then return cached end
end
end
local matched: { MatchPair } = {}
local remainingVirtual: { string } = table.clone(virtualChildren)
local remainingStudio: { Instance } = table.clone(studioChildren)
-- 2. Group by (Name, ClassName) -- NO pcall, direct property access
local vByKey: { [string]: { number } } = {}
for i, id in remainingVirtual do
local vInst = virtualInstances[id]
if vInst then
local key = vInst.Name .. "\0" .. vInst.ClassName
local group = vByKey[key]
if not group then
group = {}
vByKey[key] = group
end
table.insert(group, i)
end
end
local sByKey: { [string]: { number } } = {}
for i, inst in remainingStudio do
local key = inst.Name .. "\0" .. inst.ClassName
local group = sByKey[key]
if not group then
group = {}
sByKey[key] = group
end
table.insert(group, i)
end
-- 3. 1:1 instant match
local matchedV: { [number]: boolean } = {}
local matchedS: { [number]: boolean } = {}
for key, vIndices in vByKey do
local sIndices = sByKey[key]
if sIndices and #vIndices == 1 and #sIndices == 1 then
local vi, si = vIndices[1], sIndices[1]
if not matchedV[vi] and not matchedS[si] then
table.insert(matched, {
virtualId = remainingVirtual[vi],
studioInstance = remainingStudio[si],
})
matchedV[vi] = true
matchedS[si] = true
end
end
end
removeMatched(remainingVirtual, matchedV)
removeMatched(remainingStudio, matchedS)
-- 4. Ambiguous groups (same structure as before but using computePairCost)
if #remainingVirtual > 0 and #remainingStudio > 0 then
-- Rebuild groups, pre-compute caches, N*M scoring...
-- Score via: ownDiffs (from pre-computed caches) + childrenCost
-- For childrenCost: call computePairCost (which calls matchChildren recursively)
-- BUT: the ambiguous loop uses pre-computed caches for ownDiffs directly
-- (cacheVirtual/cacheStudio/countOwnDiffs), then adds children cost via
-- a direct matchChildren call (same as computePairCost but with caches already built)
--
-- The inner scoring for an ambiguous (vi, si) pair:
-- cost = countOwnDiffs(vCaches[vi], sCaches[si], classKeys)
-- if cost < bestSoFar then
-- local vInst = virtualInstances[remainingVirtual[vi]]
-- local studioKids = sCaches[si].children
-- -- filter valid virtual children, call matchChildren, add totalCost
-- end
--
-- Greedy assign sorted by cost ascending (tie-break by insertion index)
end
-- 5. Compute totalCost for ALL matched pairs (uses session cache)
local totalCost: number = 0
for _, pair in matched do
totalCost += computePairCost(
session, pair.virtualId, pair.studioInstance, virtualInstances, math.huge
)
end
totalCost += (#remainingVirtual + #remainingStudio) * UNMATCHED_PENALTY
local result: MatchResult = {
matched = matched,
unmatchedVirtual = remainingVirtual,
unmatchedStudio = remainingStudio,
totalCost = totalCost,
}
-- 6. Cache result
if parentVirtualId and parentStudioInstance then
if not session.matchCache[parentVirtualId] then
session.matchCache[parentVirtualId] = {}
end
session.matchCache[parentVirtualId][parentStudioInstance] = result
end
return result
end
```
### pcall Removal in Studio Grouping
The current code wraps `inst.Name, inst.ClassName` in a `pcall(function() ... end)`. This allocates a closure per studio child and prevents inlining. During matching, studio children come from `:GetChildren()` on a live instance within a single-threaded sync frame -- they cannot be destroyed mid-operation. Replace with direct property access.
### O(n) removeMatched
Current `_removeMatched` uses `table.remove` in a reverse loop -- O(n^2) due to element shifting. Replace with single-pass in-place compaction (shown above).
### Module Export
```lua
return {
newSession = newSession,
matchChildren = matchChildren,
}
```
Only two symbols exported. All helpers are local (invisible outside module, inlineable by compiler).
---
## Lua Callers
### [hydrate.lua](plugin/src/Reconciler/hydrate.lua)
```lua
local Matching = require(script.Parent.matching)
local function hydrate(instanceMap, virtualInstances, rootId, rootInstance, session)
-- ...
local result = Matching.matchChildren(
session, validVirtualIds, existingChildren, virtualInstances,
rootId, rootInstance
)
for _, pair in result.matched do
hydrate(instanceMap, virtualInstances, pair.virtualId, pair.studioInstance, session)
end
-- ...
end
```
### [Reconciler/init.lua](plugin/src/Reconciler/init.lua)
```lua
function Reconciler:hydrate(virtualInstances, rootId, rootInstance, session)
Timer.start("Reconciler:hydrate")
local result = hydrate(self.__instanceMap, virtualInstances, rootId, rootInstance, session)
Timer.stop()
return result
end
```
### [ServeSession.lua](plugin/src/ServeSession.lua)
Create session before hydrate at line 628:
```lua
local Matching = require(script.Parent.Reconciler.matching)
-- ...
local session = Matching.newSession()
self.__reconciler:hydrate(readResponseBody.instances, serverInfo.rootInstanceId, game, session)
```
---
## trueEquals Rewrite: [trueEquals.lua](plugin/src/Reconciler/trueEquals.lua)
### Module Header
```lua
--!strict
--!native
--!optimize 2
```
### Problems in Current Implementation
1. **Redundant type dispatch**: Every branch checks both `typeA` and `typeB` (`typeA == "CFrame" and typeB == "CFrame"`). Check type once, early-out if they differ.
2. **Heap allocations in hot path**: `{ a:GetComponents() }`, `{ a.X, a.Y, a.Z }`, `{ a.X, a.Y }` build temporary arrays for every CFrame/Vector3/Vector2 comparison.
3. `**checkedKeys` set**: Allocates a table for every table-vs-table comparison. Unnecessary -- second loop only needs to check for extra keys in `b`.
4. **No type annotations**, no `--!strict`, no `--!native`, no `--!optimize 2`.
### Structure: Single typeof Dispatch + Dedicated Helpers
Adopt the [DeepEqualsPureUnsafe](escape-tsunami reference) pattern: `rawequal` identity check, `rawlen` for quick array length, `rawget`/`next` for table traversal, no temporary allocations. Bake in fuzzy equality for the matching use case.
```lua
local EPSILON: number = 0.0001
local function fuzzyEq(a: number, b: number): boolean
local diff = math.abs(a - b)
local maxVal = math.max(math.abs(a), math.abs(b), 1)
return diff < EPSILON or diff < maxVal * EPSILON
end
local function color3Eq(a: Color3, b: Color3): boolean
return math.floor(a.R * 255) == math.floor(b.R * 255)
and math.floor(a.G * 255) == math.floor(b.G * 255)
and math.floor(a.B * 255) == math.floor(b.B * 255)
end
local function vector2Eq(a: Vector2, b: Vector2): boolean
return fuzzyEq(a.X, b.X) and fuzzyEq(a.Y, b.Y)
end
local function vector3Eq(a: Vector3, b: Vector3): boolean
return fuzzyEq(a.X, b.X) and fuzzyEq(a.Y, b.Y) and fuzzyEq(a.Z, b.Z)
end
local function cframeEq(a: CFrame, b: CFrame): boolean
local ax, ay, az, aR00, aR01, aR02, aR10, aR11, aR12, aR20, aR21, aR22 = a:GetComponents()
local bx, by, bz, bR00, bR01, bR02, bR10, bR11, bR12, bR20, bR21, bR22 = b:GetComponents()
return fuzzyEq(ax, bx) and fuzzyEq(ay, by) and fuzzyEq(az, bz)
and fuzzyEq(aR00, bR00) and fuzzyEq(aR01, bR01) and fuzzyEq(aR02, bR02)
and fuzzyEq(aR10, bR10) and fuzzyEq(aR11, bR11) and fuzzyEq(aR12, bR12)
and fuzzyEq(aR20, bR20) and fuzzyEq(aR21, bR21) and fuzzyEq(aR22, bR22)
end
```
All helpers are small `local function`s -- the compiler can inline them into `trueEquals`.
### Main Function: Single typeof Dispatch
```lua
local function trueEquals(a: any, b: any): boolean
if rawequal(a, b) then return true end
-- Null-ref equivalence: nil == {Ref = "000...0"}
if a == nil then
return type(b) == "table" and rawget(b, "Ref") == "00000000000000000000000000000000"
end
if b == nil then
return type(a) == "table" and rawget(a, "Ref") == "00000000000000000000000000000000"
end
local t = typeof(a)
if t ~= typeof(b) then
-- EnumItem/number cross-type
if t == "number" and typeof(b) == "EnumItem" then return a == (b :: EnumItem).Value end
if t == "EnumItem" and typeof(b) == "number" then return (a :: EnumItem).Value == b end
return false
end
-- Same-type dispatch (no redundant second type check)
if t == "table" then
-- DeepEqualsPureUnsafe pattern with fuzzy recursive equality
if rawlen(a) ~= rawlen(b) then return false end
for k, v in next, a do
local ov = rawget(b, k)
if ov == nil or not trueEquals(v, ov) then return false end
end
for k in next, b do
if rawget(a, k) == nil then return false end
end
return true
end
if t == "number" then return fuzzyEq(a, b) end
if t == "Color3" then return color3Eq(a, b) end
if t == "Vector3" then return vector3Eq(a, b) end
if t == "Vector2" then return vector2Eq(a, b) end
if t == "CFrame" then return cframeEq(a, b) end
-- NaN check (number case already handled above, this catches any remaining NaN-like values)
if a ~= a and b ~= b then return true end
return false
end
```
**Key improvements:**
- **Zero heap allocations**: No temporary arrays for CFrame/Vector3/Vector2. CFrame uses 12 direct `fuzzyEq` calls. Vector3/Vector2 use direct component access.
- **No `checkedKeys` set**: Second table loop only checks existence via `rawget`, doesn't compare values (already done in first loop).
- **Single `typeof` call path**: Check type of `a` once, verify `b` matches, then dispatch. No redundant `typeA == X and typeB == X` at every branch.
- `**rawequal` identity check first**: Catches same-reference case before any type checking.
- `**rawlen` fast-path for tables**: Different array lengths = not equal, no iteration needed.
---
## Codegen Verification
After each rewrite, run:
```bash
luau-compile --remarks plugin/src/Reconciler/trueEquals.lua
luau-compile --remarks plugin/src/Reconciler/matching.lua
```
**trueEquals.lua** -- check that:
- `fuzzyEq`, `color3Eq`, `vector2Eq`, `vector3Eq`, `cframeEq` are inlined into `trueEquals`
- No "can't inline" remarks on any helper
**matching.lua** -- check that:
- `countOwnDiffs`, `cacheVirtual`, `cacheStudio`, `removeMatched` are inlined at call sites
- `computePairCost` inlines or at minimum shows no optimization barriers
- No generic dispatch warnings for typed parameters
Iterate on any functions that fail to inline (split if too large, simplify control flow).
---
## Rust Forward Sync: [src/snapshot/matching.rs](src/snapshot/matching.rs)
Add session struct and thread through. Rust already has the clean 2-function structure (`match_forward` + `compute_change_count` / `match_children_for_scoring`).
```rust
pub struct MatchingSession {
cost_cache: RefCell<HashMap<(Ref, Ref), u32>>,
}
```
- Add `total_cost: u32` to `ForwardMatchResult`
- Add `session: &MatchingSession` param to `match_forward`, `compute_change_count`, `match_children_for_scoring`
- Cache `compute_change_count` results by `(snap.snapshot_id, tree_ref)`, only when `cost < best_so_far`
- Create session in [patch_compute.rs](src/snapshot/patch_compute.rs) `compute_patch_set`, thread through `compute_patch_set_internal` and `compute_children_patches`
## Rust Syncback: [src/syncback/matching.rs](src/syncback/matching.rs)
Same pattern. Session struct, `total_cost` on `MatchResult`, session param on all functions.
- Create session in [dir.rs](src/snapshot_middleware/dir.rs) before `match_children` call
## Test Updates: [matching_fixtures.rs](tests/tests/matching_fixtures.rs)
Pass `&MatchingSession::new()` to `match_children` and `match_forward` calls.
---
## Summary of Optimizations
| Change | Impact |
| ------------------------------------------------------- | ----------------------------------------------------------------- |
| **trueEquals**: single typeof dispatch | Eliminates redundant type checks at every branch |
| **trueEquals**: dedicated type helpers | Zero heap allocations (no temp arrays for CFrame/Vector3/Vector2) |
| **trueEquals**: DeepEqualsPureUnsafe table pattern | No checkedKeys set; rawlen/rawget/next |
| **trueEquals**: `--!strict`/`--!native`/`--!optimize 2` | Native codegen for the most-called function in matching |
| **matching**: local functions (not table methods) | Enables compiler inlining |
| **matching**: `--!strict`/`--!native`/`--!optimize 2` | Native codegen, type-guided optimization |
| **matching**: concrete types on all params/returns | Specialized native code paths |
| **matching**: remove pcall in grouping | Eliminates closure + pcall overhead per child |
| **matching**: 2-function recursion (from 4) | Simpler for optimizer, easier to reason about |
| **matching**: session caching (matchCache + costCache) | Eliminates redundant subtree scoring across recursion levels |
| **matching**: totalCost on matchChildren | Eliminates _computeChildrenCost re-iteration of matched pairs |
| **matching**: O(n) removeMatched | Linear vs quadratic element removal |

161
.cursor/plans/merge_plugin_upload_workflow_0943cf6c.plan.md Normal file
View file

@ -0,0 +1,161 @@
---
name: Merge plugin upload workflow
overview: Manually apply the upstream commit "Upload plugin as part of release workflow" to the Atlas fork, adapting all file references and metadata to match the fork's diverged codebase.
todos:
- id: new-files
content: Create .lune/.config.luau, .lune/scripts/plugin-upload.luau, .lune/upload-plugin.luau (adapted for atlas), plugin/UploadDetails.json (with Atlas values)
status: completed
- id: submodule
content: Add opencloud-execute submodule at .lune/opencloud-execute
status: completed
- id: build-rs
content: Add UploadDetails.json to plugin snapshot in build.rs
status: completed
- id: project-json
content: Add UploadDetails node to plugin.project.json
status: completed
- id: rokit
content: Add lune tool to rokit.toml
status: completed
- id: release-yml
content: Add Roblox upload step to release.yml build-plugin job
status: completed
- id: plugin-rs
content: Apply trivial test change in src/cli/plugin.rs
status: completed
- id: local-scripts
content: Replace scripts/publish-plugin.ps1 and add scripts/publish-plugin.sh to use lune-based upload
status: completed
isProject: false
---
# Merge Plugin Upload Release Workflow
Cherry-picking will produce conflicts in 6+ files due to heavy divergence. Manual application is the cleanest path.
## Upstream bugs to fix
**Swapped env vars in `release.yml`:** The upstream commit has `RBX_UNIVERSE_ID` mapped to `vars.PLUGIN_CI_PLACE_ID` and `RBX_PLACE_ID` mapped to `vars.PLUGIN_CI_UNIVERSE_ID` -- clearly crossed. We will use the correct mapping.
## Local scripts replacement
`scripts/publish-plugin.ps1` currently uses `cargo run -- upload`. Replace it (and add missing `.sh`) to use the same Lune-based upload pipeline locally. The `.lune/upload-plugin.luau` script already handles the local case (builds the plugin via `atlas build` when `CI != "true"`), so the wrapper scripts just load `.env` and run `lune run upload-plugin Atlas.rbxm`.
`.env` uses the same names as GitHub Actions secrets/vars:
```
PLUGIN_UPLOAD_TOKEN=your-opencloud-api-key
PLUGIN_CI_UNIVERSE_ID=your-universe-id
PLUGIN_CI_PLACE_ID=your-place-id
```
The local scripts map these to `RBX_API_KEY`/`RBX_UNIVERSE_ID`/`RBX_PLACE_ID` before calling lune, matching what the CI workflow `env:` block does.
## What the commit adds
Automated plugin upload to Roblox as part of the GitHub release workflow, using [Lune](https://lune-org.github.io/docs) + OpenCloud Luau Execute to publish the built `.rbxm` to Roblox's plugin marketplace.
## New files (create as-is, with Atlas adaptations)
### `.lune/.config.luau`
Lune config with strict mode and typedefs alias. Copy verbatim from upstream.
### `.lune/scripts/plugin-upload.luau`
Roblox engine script that deserializes the plugin `.rbxm`, reads `UploadDetails`, and calls `AssetService:CreateAssetVersionAsync`. Copy verbatim -- it's generic (reads values from UploadDetails at runtime).
### `.lune/upload-plugin.luau`
Lune entry point that reads the `.rbxm`, sends it to OpenCloud for execution. One adaptation needed: change `process.exec("rojo", ...)` to `process.exec("atlas", ...)` on the local-testing path (line 25 in upstream).
### `plugin/UploadDetails.json`
Atlas-specific values per user input:
```json
{
"assetId": 111151139098227,
"name": "Atlas",
"description": "The plugin portion of Atlas, a tool to enable professional tooling for Roblox developers.",
"creatorId": 422368266,
"creatorType": "Group"
}
```
## Submodule addition
### `.lune/opencloud-execute`
Add git submodule: `https://github.com/Dekkonot/opencloud-luau-execute-lune.git` at `.lune/opencloud-execute`. This also adds an entry to `[.gitmodules](.gitmodules)`.
## Existing file modifications
### `[.github/workflows/release.yml](.github/workflows/release.yml)`
Two additions to the `build-plugin` job:
1. **Add Rokit install step** (after "Install Rust", before "Build Atlas") so `lune` is on PATH:
```yaml
- name: Install Rokit
uses: CompeyDev/setup-rokit@v0.1.2
```
1. **Add upload step** (after "Upload Plugin to Artifacts" at line 47) with **corrected** env var mapping (upstream had these swapped):
```yaml
- name: Upload Plugin to Roblox
env:
RBX_API_KEY: ${{ secrets.PLUGIN_UPLOAD_TOKEN }}
RBX_UNIVERSE_ID: ${{ vars.PLUGIN_CI_UNIVERSE_ID }}
RBX_PLACE_ID: ${{ vars.PLUGIN_CI_PLACE_ID }}
run: lune run upload-plugin Atlas.rbxm
```
### `[build.rs](build.rs)`
Add `"UploadDetails.json"` entry to the plugin snapshot hashmap, after `"Version.txt"` on line 98:
```rust
"UploadDetails.json" => snapshot_from_fs_path(&plugin_dir.join("UploadDetails.json"))?,
```
### `[plugin.project.json](plugin.project.json)`
Add `UploadDetails` child node after `Version` (line 25):
```json
"UploadDetails": {
"$path": "plugin/UploadDetails.json"
}
```
### `[rokit.toml](rokit.toml)`
Add lune tool (latest version) after the existing entries:
```toml
lune = "lune-org/lune@0.10.4"
```
### `[src/cli/plugin.rs](src/cli/plugin.rs)`
Trivial test change on line 101: `assert!(initialize_plugin().is_ok())` becomes `let _ = initialize_plugin().unwrap();`
### `[scripts/publish-plugin.ps1](scripts/publish-plugin.ps1)`
Replace entirely. New version: load `.env`, export `RBX_API_KEY`/`RBX_UNIVERSE_ID`/`RBX_PLACE_ID`, run `lune run upload-plugin Atlas.rbxm`.
### `scripts/publish-plugin.sh` (new)
Bash equivalent: source `.env`, export vars, run `lune run upload-plugin Atlas.rbxm`.
## Required GitHub secrets/vars
After merging, the repo needs these configured in GitHub settings:
- **Secret:** `PLUGIN_UPLOAD_TOKEN` -- Roblox OpenCloud API key
- **Variable:** `PLUGIN_CI_UNIVERSE_ID` -- Universe ID for the place used by OpenCloud Execute
- **Variable:** `PLUGIN_CI_PLACE_ID` -- Place ID within that universe

343
.cursor/plans/parallel_filesystem_io_39399341.plan.md Normal file
View file

@ -0,0 +1,343 @@
---
name: Parallel filesystem IO
overview: Add parallel file reading via VFS prefetch cache, parallel snapshot building via rayon at directory level, and parallel RefPathIndex population to speed up all project loading paths.
todos:
- id: vfs-prefetch-cache
content: Add PrefetchCache struct (files + canonical maps), prefetch_cache field on VfsInner, read_raw() helper, wire into read()/read_to_string()/canonicalize()
status: completed
- id: vfs-prefetch-api
content: Add Vfs::set_prefetch_cache() and Vfs::clear_prefetch_cache() public methods
status: completed
- id: vfs-prefetch-tests
content: Add unit tests for prefetch cache (file hit/miss, read_to_string, canonicalize hit/miss, watch registration, clear)
status: completed
- id: prefetch-builder
content: Write prefetch_project_files() using walkdir + rayon to parallel-read all files under root
status: completed
- id: parallel-snapshot-build
content: Add rayon par_iter to snapshot_dir_no_meta in dir.rs for parallel child processing
status: completed
- id: env-var-fallback
content: Add ATLAS_SEQUENTIAL env var to disable all parallelization for debugging
status: completed
- id: integrate-serve-session
content: Call prefetch in ServeSession::init_tree() before snapshot_from_vfs()
status: completed
- id: parallel-refpathindex
content: Parallelize RefPathIndex::populate_from_dir() using walkdir + rayon for file reads/parsing
status: completed
- id: test-determinism
content: Add parallel snapshot determinism test - run snapshot_from_vfs N times, verify identical output
status: completed
- id: test-and-verify
content: Run full cargo test, verify no regressions, add timing logs to confirm speedup
status: completed
isProject: false
---
# Parallel Filesystem I/O for Project Loading
## Problem
Every file read goes through `Vfs` -> `Mutex<VfsInner>` -> `StdBackend::read()` -> `fs_err::read()`. This serializes all I/O. For a 3500-file project, startup is bottlenecked by ~3500 sequential syscalls, plus sequential CPU-bound parsing of each file.
**Affected commands:** `serve`, `build`, `upload`, `syncback`, `sourcemap` -- all call `snapshot_from_vfs()` which recursively reads every file through the VFS one at a time.
## Current Sequential Hotspots
1. `**snapshot_from_vfs()` initial tree build** ([src/serve_session.rs](src/serve_session.rs) line 141) -- The biggest bottleneck. Recursive directory traversal in [src/snapshot_middleware/dir.rs](src/snapshot_middleware/dir.rs) lines 54-63, each child read goes through VFS mutex.
2. `**RefPathIndex::populate_from_dir()`** ([src/rojo_ref.rs](src/rojo_ref.rs) lines 372-431) -- Sequential walk + `std::fs::read()` for every `.meta.json5`/`.model.json5` file. Runs after tree build on `serve`.
3. **Two-way sync writes in `api.rs`** -- Sequential `fs::write()` calls (low priority, small batches).
## Strategy: Full Pipeline
Three layers, all using `rayon` (already a dependency):
1. **VFS Prefetch Cache** -- parallel I/O: read all files upfront
2. **Parallel Snapshot Build** -- parallel CPU: parse files concurrently at directory level
3. **Parallel RefPathIndex** -- parallel I/O+CPU: read and parse meta/model files concurrently
Combined timeline (vs current sequential):
```
Current: [read→parse→read→parse→read→parse→read→parse→...] (serial everything)
Pipelined: [=====parallel read all files=====] (Phase 1: I/O)
......[==parallel parse+build snapshots==] (Phase 2: CPU)
```
### Part 1: VFS Prefetch Cache
**Concept:** Before `snapshot_from_vfs()` runs, walk the directory tree and read all file contents in parallel using `std::fs` + rayon (bypassing the VFS lock). Store results in a HashMap on VfsInner. The subsequent parallel snapshot build hits the cache (HashMap lookup, no I/O) instead of disk.
**Changes to [crates/memofs/src/lib.rs](crates/memofs/src/lib.rs):**
- Add `PrefetchCache` struct with `files: HashMap<PathBuf, Vec<u8>>` and `canonical: HashMap<PathBuf, PathBuf>`
- Add `prefetch_cache: Option<PrefetchCache>` field to `VfsInner`
- Extract a shared `read_raw(&mut self, path) -> io::Result<Vec<u8>>` helper that checks `prefetch_cache.files` first (using `HashMap::remove` to free memory on hit), falls back to backend
- Wire BOTH `VfsInner::read()` AND `VfsInner::read_to_string()` through `read_raw()`
- Add cache check to `VfsInner::canonicalize()`: check `prefetch_cache.canonical` first (using `HashMap::remove`), fall back to backend
- Add `Vfs::set_prefetch_cache(cache)` and `Vfs::clear_prefetch_cache()`
```rust
pub struct PrefetchCache {
pub files: HashMap<PathBuf, Vec<u8>>,
pub canonical: HashMap<PathBuf, PathBuf>,
}
```
**Critical: `read_to_string_lf_normalized` coverage.** The `Vfs::read_to_string_lf_normalized()` method (used by ALL `.luau` files via `lua.rs:55`) calls `VfsInner::read_to_string()` internally. Since we wire `VfsInner::read_to_string()` through `read_raw()`, LF-normalized reads hit the cache automatically. The flow: `Vfs::read_to_string_lf_normalized` -> lock -> `VfsInner::read_to_string` -> `read_raw` (cache hit) -> UTF-8 convert -> unlock -> CRLF->LF normalize.
**Critical: `canonicalize` coverage.** Every middleware calls `vfs.canonicalize(path)` once, storing the result in `InstanceMetadata::relevant_paths` for file-change-to-instance mapping. During init, these are parallelized in the prefetch phase. After init, the cache is cleared and `change_processor.rs` calls `canonicalize()` through the backend for fresh results.
**What is NOT cached** (and why that's acceptable):
- `read_dir()`: directory listing syscall, ~10us each, ~500 dirs = ~5ms total
- `metadata()`: stat syscall, ~~1-10us each. `get_dir_middleware` probes up to 13 init files per directory (~~6500 calls for 500 dirs = ~6.5ms)
- `exists()`: only used by `Project::load_initial_project` (1-2 calls before prefetch starts)
**No stale canonicalize entries:** The cache exists only during `init_tree()`. During this window, the filesystem is static (no watcher running, no mutations). After `clear_prefetch_cache()`, all canonicalize calls go through the backend -- the `ChangeProcessor` always gets fresh results for rename/move detection. **No memory leak:** `clear_prefetch_cache()` drops the entire `PrefetchCache` struct.
**New helper (in `src/` or inline in `serve_session.rs`):**
```rust
fn prefetch_project_files(root: &Path) -> io::Result<PrefetchCache> {
use memofs::PrefetchCache;
use rayon::prelude::*;
use walkdir::WalkDir;
let entries: Vec<walkdir::DirEntry> = WalkDir::new(root)
.follow_links(true) // Match VFS behavior: StdBackend::read() follows symlinks
.into_iter()
.filter_map(|e| e.ok())
.collect();
// Parallel file reads (files only)
let file_data: Vec<_> = entries
.par_iter()
.filter(|e| e.file_type().is_file())
.filter_map(|e| {
let path = e.path().to_path_buf();
std::fs::read(&path).ok().map(|c| (path, c))
})
.collect();
// Parallel canonicalize (files AND directories)
let canonical_data: Vec<_> = entries
.par_iter()
.filter_map(|e| {
let path = e.path().to_path_buf();
std::fs::canonicalize(&path).ok().map(|c| (path, c))
})
.collect();
Ok(PrefetchCache {
files: file_data.into_iter().collect(),
canonical: canonical_data.into_iter().collect(),
})
}
```
`walkdir` (already a dependency) handles the directory walk. `.follow_links(true)` ensures symlinked files are prefetched (walkdir handles symlink loop detection). `rayon::par_iter` parallelizes both file reads AND canonicalize syscalls. The canonicalize cache covers both files and directories since `dir.rs:66` canonicalizes directory paths.
**Integration in [src/serve_session.rs](src/serve_session.rs) `init_tree()`:**
```rust
fn init_tree(vfs: &Vfs, start_path: &Path) -> Result<...> {
let root_project = Project::load_initial_project(vfs, start_path)?;
if std::env::var("ATLAS_SEQUENTIAL").is_err() {
let cache = prefetch_project_files(root_project.folder_location())?;
vfs.set_prefetch_cache(cache);
}
let snapshot = snapshot_from_vfs(&instance_context, vfs, start_path)?;
vfs.clear_prefetch_cache();
// ... rest unchanged
}
```
**Note:** The serve command calls `init_tree()` twice -- once via `new_oneshot()` for config (port/address), once via `new()` for the real session. Both get prefetch. The oneshot is immediately dropped so its prefetch is "wasted", but the overhead is small (~50ms) and the code stays simple.
**Why this is safe:** The prefetch reads files with `std::fs` (no VFS lock). The cache is loaded into VfsInner behind the mutex. The snapshot build then acquires the mutex per-read as before, but each read is a HashMap remove (nanoseconds) instead of a disk syscall. Existing behavior is preserved for cache misses (falls through to backend). Watch registration still happens normally (gated by `watch_enabled`).
`**StdBackend::read_dir()` is eager:** It collects and sorts all directory entries before returning the `ReadDir` iterator (`std_backend.rs:283-296`). The iterator does not hold any reference into the backend. This means `read_dir` results are safe to use from rayon threads after the VFS lock is released.
### Part 2: Parallel Snapshot Build at Directory Level
**Concept:** In `snapshot_dir_no_meta`, process each directory's children in parallel using rayon. With the prefetch cache warm, each rayon thread acquires the VFS mutex briefly for a cache hit (~1us), then parses the file content lock-free (the expensive part). Mutex contention is negligible since critical sections are just HashMap removes.
**Changes to [src/snapshot_middleware/dir.rs](src/snapshot_middleware/dir.rs) `snapshot_dir_no_meta()`:**
```rust
// Current (sequential):
for entry in vfs.read_dir(path)? {
let entry = entry?;
if !passes_filter_rules(&entry) { continue; }
if let Some(child) = snapshot_from_vfs(context, vfs, entry.path())? {
snapshot_children.push(child);
}
}
// Parallel (with order preservation):
use rayon::prelude::*;
let entries: Vec<_> = vfs.read_dir(path)?
.filter_map(|e| e.ok())
.collect();
let results: Vec<anyhow::Result<Option<InstanceSnapshot>>> = entries
.par_iter()
.map(|entry| {
if !passes_filter_rules(entry) {
return Ok(None);
}
snapshot_from_vfs(context, vfs, entry.path())
})
.collect();
let mut snapshot_children = Vec::new();
for result in results {
if let Some(snapshot) = result? {
snapshot_children.push(snapshot);
}
}
```
**Why `map` instead of `filter_map`:** rayon's `par_iter().map().collect()` preserves original order (map is an indexed operation). `filter_map` converts to an unindexed iterator where order is implementation-dependent. Using `map` + sequential flatten ensures children appear in the same order as `read_dir` returns them.
**Why this is safe for concurrent VFS access:** Each VFS call (metadata, read, read_dir) independently acquires/releases the Mutex -- no nested locking, so no deadlock risk. `&InstanceContext` is `Sync` (contains `Arc<Vec<PathIgnoreRule>>` + `Vec<SyncRule>`). `&Vfs` is `Sync` (wraps `Mutex<VfsInner>`). `anyhow::Error` is `Send` (required for rayon's `collect::<Vec<Result<_>>>()`).
**Multiple VFS calls per middleware:** Each middleware makes several VFS calls during snapshot building. For example, `lua.rs` calls `read_to_string_lf_normalized` (cache hit), `canonicalize` (backend), and reads companion `.meta.json5` (cache hit). With parallel builds, the cached reads are fast (~1us lock + HashMap remove), while `metadata` and `canonicalize` still serialize on the mutex with real syscalls. This limits maximum parallelism but the file-read savings dominate.
**Recursion:** Since `snapshot_from_vfs` is recursive, a directory with 10 subdirectories kicks off 10 parallel subtrees, each of which parallelizes their own children. rayon's work-stealing keeps all cores busy across the entire tree.
**Rayon thread pool:** The `sourcemap` command configures a global rayon pool with `max(num_cpus, 6)` threads at `src/cli/sourcemap.rs:93`, but this happens AFTER `init_tree()` completes. Our parallel snapshot build runs before that configuration, using rayon's default pool (CPU count). No conflict.
**Environment variable gate:**
```rust
if std::env::var("ATLAS_SEQUENTIAL").is_ok() {
// Sequential fallback (existing code)
} else {
// Parallel path
}
```
### Part 3: Parallel RefPathIndex Population
**Changes to [src/rojo_ref.rs](src/rojo_ref.rs) `populate_from_dir()`:**
Current: sequential stack-based walk + `std::fs::read()` per file.
New approach:
1. Walk tree with `walkdir`, collect all `.meta.json5`/`.model.json5` paths (sequential, fast -- just filename checks)
2. Read + parse + extract `Rojo_Ref_*` entries in parallel with rayon. `&RojoTree` is read-only and safe to share across threads (`get_ids_at_path()` and `ref_target_path_from_tree()` are in-memory lookups).
3. Collect `Vec<(String, PathBuf)>` results, then insert into index (sequential)
## Race Condition Analysis
### No risk (by design)
- **VFS data races:** Impossible. `Mutex<VfsInner>` serializes all access. Multiple rayon threads contend briefly on cache lookups (~1us critical sections), which is safe.
- **Deadlock from re-entrant locking:** Impossible. Each VFS call (metadata, read, read_dir) acquires and releases the lock independently. No VFS call internally calls another VFS method. `ReadDir` iterator is fully materialized before lock release.
- **InMemoryFs thread safety:** `InMemoryFs` wraps `Arc<Mutex<InMemoryFsInner>>`, additionally protected by VFS's outer Mutex. Unit tests using InMemoryFs are safe with parallel snapshot building.
### Mitigated risks
- **Child ordering non-determinism:** `par_iter().filter_map().collect()` does NOT guarantee order (filter_map makes the iterator unindexed). We use `par_iter().map().collect()` instead, which preserves input order because `map` on an indexed iterator stays indexed. The sequential flatten step then filters Nones. A determinism test (run N times, compare output) validates this.
- **Prefetch cache stale reads:** If a file is modified between prefetch and snapshot_from_vfs, the cached content is stale. In practice this cannot happen during the few milliseconds of startup. Same time-of-check-time-of-use window exists with sequential reads. No additional risk.
- **Error propagation in parallel:** `par_iter().map().collect::<Vec<Result<_>>>()` collects all results. Errors are propagated sequentially in the flatten step via `?`. First error wins, consistent with sequential behavior.
### Covered by environment variable fallback
`ATLAS_SEQUENTIAL=1` disables both prefetch cache and parallel snapshot building, reverting to the exact current sequential code path. Useful for:
- Debugging if parallelization causes platform-specific issues
- A/B performance comparison
- CI environments requiring deterministic sequential execution
## Testing Strategy
### Existing tests that automatically exercise the new system
All integration tests spawn the CLI binary, which goes through `init_tree()` -> prefetch + parallel snapshot:
- **Build tests** ([tests/tests/build.rs](tests/tests/build.rs)) -- Run `rojo build` on real filesystem fixtures. Exercise full prefetch + parallel snapshot pipeline.
- **Serve tests** ([tests/rojo_test/serve_util.rs](tests/rojo_test/serve_util.rs) `TestServeSession`) -- Spawn `atlas serve` on temp dirs. Exercise prefetch + parallel snapshot + RefPathIndex.
- **Syncback/roundtrip tests** ([tests/tests/syncback.rs](tests/tests/syncback.rs), [tests/tests/syncback_roundtrip.rs](tests/tests/syncback_roundtrip.rs)) -- Full CLI syncback pipeline.
- **Two-way sync tests** ([tests/tests/two_way_sync.rs](tests/tests/two_way_sync.rs)) -- Serve session + write API.
- **Matching fixture tests** ([tests/tests/matching_fixtures.rs](tests/tests/matching_fixtures.rs)) -- Call `snapshot_from_vfs()` directly with `Vfs::new_default()`. Exercise parallel snapshot building in dir.rs but NOT prefetch (no init_tree call).
- **Unit tests in dir.rs** -- Use `InMemoryFs`. Exercise parallel snapshot building but not prefetch. This is correct -- InMemoryFs files don't exist on the real filesystem.
### New tests to add
1. **VFS prefetch cache unit tests** (in [crates/memofs/src/lib.rs](crates/memofs/src/lib.rs) test module):
- Cache hit: `set_prefetch_cache` with known path -> `read()` returns cached data
- Cache depletion: second `read()` of same path falls through to backend (entry was removed on first read)
- Cache miss: `read()` of path not in cache falls through to backend
- `read_to_string` uses cache: verify UTF-8 conversion from cached bytes
- Watch registration: if `watch_enabled`, watches are registered even on cache hits
- `clear_prefetch_cache`: after clear, all reads go through backend
- Empty/None cache behaves identically to no cache
2. **Parallel snapshot determinism test** (in [tests/tests/build.rs](tests/tests/build.rs) or new file):
- Run `snapshot_from_vfs` on a non-trivial fixture 5 times
- Assert all 5 snapshots are identical (catches ordering bugs from rayon)
- Use a fixture with multiple children per directory to stress the parallel path
3. **Prefetch + parallel integration sanity** (implicit):
- All existing insta snapshot tests serve as regression tests. If prefetch returns wrong data or parallel build reorders children, snapshot comparisons fail immediately.
### Tests we do NOT need
- **VFS mutex stress test:** The Mutex is a well-tested primitive. Contention with ~1us critical sections and rayon's thread pool is a solved problem.
- **Prefetch TOCTOU test:** The time-of-check-time-of-use window between prefetch and read is identical to the existing sequential window. No additional test.
- **InMemoryFs concurrency test:** InMemoryFs is protected by both its own Mutex and the VFS's outer Mutex. Double-locked by design.
## What About the File Watcher?
The ChangeProcessor handles individual file events one at a time. Single-file reads don't benefit from parallelization. Two scenarios where deeper parallelism could help:
- **Batch events after `git checkout`**: Many files change at once, processed sequentially. Could batch and parallelize, but adds complexity. Deferred.
- **Rescan after watcher critical error**: Re-snapshots the full tree. Would benefit from prefetch, but rescan is rare. Can be added later.
## Other Sequential I/O (Lower Priority, Not in This Plan)
- **Two-way sync writes** (`api.rs`): Small batches (1-10 files). Not worth the complexity.
- **Build output encoding**: Single file write. Cannot parallelize.
- **Upload encoding**: Single buffer + HTTP upload. Network-bound.
- `**validate_tree()` / `check_tree_freshness()`**: Re-snapshots full tree. Could use prefetch. Secondary benefit, can add later.
## Files Changed
| File | Change |
| ---------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| [crates/memofs/src/lib.rs](crates/memofs/src/lib.rs) | Add prefetch cache field, `read_raw()` helper, `set/clear_prefetch_cache()` methods |
| [src/snapshot_middleware/dir.rs](src/snapshot_middleware/dir.rs) | Add rayon `par_iter` for parallel child snapshot building |
| [src/serve_session.rs](src/serve_session.rs) | Call prefetch before snapshot build in `init_tree()` |
| [src/rojo_ref.rs](src/rojo_ref.rs) | Parallelize `populate_from_dir()` with rayon |
No changes to VFS backend trait. No new dependencies (rayon, walkdir already present).
## Stack Compatibility Checklist
Verified across the full stack:
- `Vfs` is `Sync` (wraps `Mutex<VfsInner>`, `VfsBackend: Send`)
- `InstanceContext` is `Sync` (contains `Arc<Vec<...>>` + `Vec<SyncRule>`)
- `anyhow::Error` is `Send` (required for rayon error collection)
- `StdBackend::read_dir()` eagerly collects entries before returning (`std_backend.rs:283-296`) -- no lazy iteration holding the lock
- No `vfs.lock()` calls in any snapshot middleware -- only syncback writes use `VfsLock`
- `InMemoryFs` is thread-safe (`Arc<Mutex<...>>`) -- unit tests with parallel dir.rs are safe
- `read_to_string_lf_normalized` routes through `VfsInner::read_to_string` -> `read_raw` -- cache hit for all `.luau` files
- `VfsInner::canonicalize` checks `prefetch_cache.canonical` -- eliminates ~400ms Windows serialization point
- Canonicalize cache safe: only alive during `init_tree()` (no mutations), `ChangeProcessor` always gets fresh backend results after cache clear
- `walkdir` with `.follow_links(true)` matches `StdBackend::read` symlink behavior; handles loop detection
- `Project::load_initial_project` reads via VFS BEFORE prefetch starts -- correct, only reads 1 project file
- Rayon global pool not configured until after `init_tree` completes (sourcemap configures at `sourcemap.rs:93`)
- `ReadDir` iterator fully materialized -- safe to use from rayon threads after lock release
- No stateful side-effects in any snapshot middleware -- all are pure functions of (context, vfs, path)

267
.cursor/plans/parallel_io_audit_fixes_35f49f83.plan.md Normal file
View file

@ -0,0 +1,267 @@
---
name: Parallel IO audit fixes
overview: "Fix three issues found during audit of the parallelization/optimization commits: a data loss bug in the orphan scanner, a duplicate-children bug in the prefetch cache, and a nested-detection correctness issue, plus a small DRY refactor."
todos:
- id: fix1-hidden-dirs
content: Fix orphan scanner walkdir to prune hidden directories using filter_entry()
status: completed
- id: fix2-dedup-roots
content: Deduplicate overlapping $path roots in prefetch_project_files before walking
status: completed
- id: fix3-nested-detection
content: Replace broken binary search nested detection with O(n log n) ancestor-tracking approach
status: completed
- id: fix4-unc-refactor
content: Extract shared strip_unc_prefix helper from 3 inline copies
status: completed
- id: test-hidden-dir
content: Add integration test for hidden directory survival in clean syncback
status: completed
- id: test-overlap
content: Add test for overlapping $path roots with prefetch cache
status: completed
- id: test-nested
content: Add unit test for nested path detection correctness
status: completed
- id: run-ci
content: Run full CI pipeline (cargo test, clippy, fmt)
status: completed
isProject: false
---
# Parallel I/O Audit Fixes
> This plan was generated by `/audit` from a Plan-mode session.
> Refer to `.cursor/rules/atlas.mdc` for full project standards.
## Standards
These standards govern every fix in this plan. An implementer MUST read `.cursor/rules/atlas.mdc` before starting.
### Round-Trip Identity (from `atlas.mdc` - Quality Standard)
Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a **bit-identical instance tree** -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.
### CLI Syncback Parity (from `atlas.mdc` - Two-Way Sync Strategy)
Plugin-based sync must produce **exactly the same filesystem output** that `atlas syncback` would give for the same input. Byte-for-byte identical files, identical directory structures, identical naming. Any divergence is a bug until proven otherwise.
### Code Quality (from `atlas.mdc` - Code Quality Standard)
DRY code that is easy to reason about and hard to get wrong. The same slugify/dedup/meta-update pattern should not be copy-pasted across files. Small refactors (2-3 call sites) are included as fixes. Major rewrites (10+ call sites) are deferred.
## Context
Audited the last 7 commits on the current branch (from `1ffed121` through `56e1ab4a`), which add:
- VFS prefetch cache for parallel file reading at startup
- Parallel snapshot building via rayon in `dir.rs`
- Parallel RefPathIndex population via rayon + walkdir
- Parallel tree hashing in syncback
- Orphan scanner rewrite using walkdir
- Placeholder-based ref path substitution optimization
- ServeCommand refactor and dead code cleanup
## Fixes
### Fix 1: Prune hidden directories from walkdir orphan scanner
- **Status:** Approved
- **Finding:** Orphan scanner walkdir replacement doesn't prune hidden directories
- **Files:** `[src/syncback/mod.rs](src/syncback/mod.rs)`
- **Problem:** Lines ~406-424. The `walkdir::WalkDir` iterator yields ALL entries including children of hidden directories. The hidden-name filter only checks `file_name().starts_with('.')`, so `settings.json` inside `.vscode/` passes the filter and gets added to `existing_paths`. In clean mode, these files would be deleted.
The old recursive `scan_directory` function skipped hidden directories AND never recursed into them. The walkdir replacement must replicate this behavior.
- **Solution:** Replace the flat `.flatten()` + post-hoc hidden check with `filter_entry()` to prune the walk at hidden directories:
```rust
for entry in walkdir::WalkDir::new(&canonical_dir)
.follow_links(true)
.into_iter()
.filter_entry(|e| {
if e.depth() == 0 {
return true;
}
e.file_name()
.to_str()
.map_or(true, |n| !n.starts_with('.') || n == ".gitkeep")
})
.flatten()
{
if entry.depth() == 0 {
continue;
}
let path = entry.path().to_path_buf();
if !is_valid_path(&ignore_patterns, project_path, &path) {
continue;
}
// Hidden check no longer needed here -- filter_entry handles it
paths.insert(path);
}
```
`filter_entry` controls which directories walkdir descends into. When it returns `false` for a directory, walkdir skips the directory AND all its descendants. This exactly matches the old recursive behavior.
- **Risk:** Low -- only touches the orphan scanner loop. `filter_entry` is walkdir's standard pruning mechanism.
- **Verify round-trip:** Clean mode syncback with a `.vscode/` directory inside a `$path` source dir should NOT delete `.vscode/` contents.
- **Verify syncback parity:** N/A -- this only affects clean mode orphan removal, not the sync pipeline.
- **Tests required:**
- Integration test: create a temp project with `src/.hidden/keep.txt`, run clean-mode syncback, verify `.hidden/keep.txt` survives.
### Fix 2: Deduplicate overlapping roots in prefetch cache
- **Status:** Approved
- **Finding:** Overlapping `$path` roots cause duplicate children in prefetch cache
- **Files:** `[src/serve_session.rs](src/serve_session.rs)`
- **Problem:** Lines ~136-156 in `prefetch_project_files`. `collect_path_roots` collects ALL `$path` entries including nested ones. For a project with `$path: "src"` and `$path: "src/shared"`, walkdir walks `src/shared` twice -- once as part of `src` and once as its own root. The `children_map` accumulates duplicate entries for `src/shared`'s children. When consumed by VFS `read_dir`, these duplicates cause duplicate instance snapshots.
- **Solution:** After collecting roots in `prefetch_project_files`, deduplicate by removing any root that is a descendant of another root:
```rust
let mut roots: Vec<std::path::PathBuf> = Vec::new();
collect_path_roots(&project.tree, folder, &mut roots);
// Remove roots that are descendants of other roots to avoid
// walking the same directory twice and producing duplicate children.
roots.sort();
let mut deduped: Vec<PathBuf> = Vec::new();
for root in roots {
if !deduped.iter().any(|existing| root.starts_with(existing)) {
deduped.push(root);
}
}
let roots = deduped;
```
After sorting, a parent path always sorts before its descendants. By iterating in order and only keeping roots that are NOT descendants of already-kept roots, we eliminate all overlaps in O(n^2) worst case but typically O(n) since most projects have few `$path` entries.
- **Risk:** Low -- only modifies which directories are walked during prefetch. The dedup logic is simple and well-understood.
- **Verify round-trip:** Build a project with overlapping `$path` entries. Verify snapshot tree has no duplicate instances. Run build 5 times, verify deterministic output.
- **Verify syncback parity:** N/A -- prefetch cache only affects forward sync initial load.
- **Tests required:**
- Unit test in `tests/tests/build.rs`: create a project with `$path: "src"` and a child with `$path: "src/shared"`. Run `snapshot_from_vfs` with prefetch cache. Verify no duplicate children under the `shared` directory instance.
### Fix 3: Fix nested path detection in orphan removal
- **Status:** Approved (O(n log n) approach)
- **Finding:** Binary search nested detection misses sibling paths
- **Files:** `[src/syncback/mod.rs](src/syncback/mod.rs)`
- **Problem:** Lines ~869-888. The optimization uses `binary_search` to find the current path's position, then checks only `sorted_removals[idx - 1]`. This misses cases where a sibling path sits between the parent and the current path in sorted order.
Example: `sorted = ["a/b", "a/b/c", "a/b/d"]`. For `a/b/d` at idx 2, the code checks `sorted[1] = "a/b/c"` (a sibling, not an ancestor). The actual ancestor `"a/b"` at idx 0 is not checked.
- **Solution:** Replace binary search with a sorted single-pass using an `current_ancestor` tracker:
```rust
let mut sorted_removals: Vec<_> = paths_to_remove.iter().cloned().collect();
sorted_removals.sort();
let mut current_ancestor: Option<&PathBuf> = None;
for old_path in &sorted_removals {
if let Some(ancestor) = current_ancestor {
if old_path.starts_with(ancestor) && old_path != ancestor {
continue; // nested under current ancestor, skip
}
}
current_ancestor = Some(old_path);
let relative_path = old_path
.strip_prefix(&canonical_project_path)
.unwrap_or(old_path);
log::debug!("Removing orphaned path: {}", relative_path.display());
if old_path.is_dir() {
fs_snapshot.remove_dir(relative_path);
} else {
fs_snapshot.remove_file(relative_path);
}
}
```
In sorted path order, all descendants of a path appear consecutively after it (because `Path` comparison is component-based). So a single variable tracking the current top-level ancestor is sufficient. When we encounter a non-descendant, it becomes the new ancestor. This is O(n log n) for the sort + O(n) for the walk.
This also removes the intermediate `paths_vec` and the `binary_search` code, simplifying the function.
- **Risk:** Low -- the logic is simpler than what it replaces. The correctness argument is straightforward: sorted paths guarantee all descendants appear consecutively.
- **Verify round-trip:** Clean mode syncback with nested directories. Verify parent-only removal (no redundant child removals).
- **Verify syncback parity:** N/A -- orphan removal is internal to clean mode.
- **Tests required:**
- Unit test: create paths `["a/b", "a/b/c", "a/b/d", "x/y", "x/y/z"]`, verify only `"a/b"` and `"x/y"` are top-level removals.
### Fix 4: Extract shared `strip_unc_prefix` helper
- **Status:** Approved
- **Finding:** Duplicated UNC prefix stripping logic in 3 locations
- **Files:** `[src/syncback/mod.rs](src/syncback/mod.rs)`
- **Problem:** The same UNC prefix stripping pattern appears at:
1. Lines ~396-404 (orphan scanner: canonicalize `dirs_to_scan` entries)
2. Lines ~649-656 (`strip_unc_prefix` helper definition inside orphan removal block)
3. Lines ~703-711 (inline canonicalize in `collect_all_path_refs_and_mappings`)
Location 2 defines a proper `fn strip_unc_prefix`, but locations 1 and 3 inline the same logic.
- **Solution:** Move `strip_unc_prefix` to module level (or just above the clean-mode block), and replace all 3 inline instances with calls to it. The function is already defined at location 2:
```rust
fn strip_unc_prefix(p: PathBuf) -> PathBuf {
let path_str = p.to_string_lossy();
if path_str.starts_with(r"\\?\") || path_str.starts_with("//?/") {
PathBuf::from(&path_str[4..])
} else {
p
}
}
```
Move this definition before the `existing_paths` block (around line ~258), then replace the inline code at lines ~396-404 with `strip_unc_prefix(std::fs::canonicalize(dir).unwrap_or_else(|_| dir.clone()))` and at lines ~703-711 with `strip_unc_prefix(p)`.
- **Risk:** Zero -- pure refactor, no behavior change.
- **Verify round-trip:** N/A -- refactor only.
- **Verify syncback parity:** N/A -- refactor only.
- **Tests required:** None -- existing tests cover this code path.
## Skipped Fixes
### Skipped: Serve command loads project twice
- **Finding:** `src/cli/serve.rs` loads the project file via `Project::load_initial_project` for port/address, then `ServeSession::new` loads it again in `init_tree`.
- **Problem:** Minor performance -- project file read twice from disk at startup.
- **Reason skipped:** Acceptable overhead. Project files are small and this only happens once.
### Skipped: Commented-out git config in init.rs
- **Finding:** Four commented-out git config lines in `src/cli/init.rs` lines 134-137.
- **Reason skipped:** User indicated these are planned for future enablement.
## Accepted Limitations
- **Limitation:** Symlinks within project `$path` directories may cause path mismatches
- **Scenario:** If a symlink inside a `$path` source directory points elsewhere, walkdir-collected paths won't match `added_paths` during clean-mode orphan removal. Files behind the symlink could be incorrectly removed.
- **User decision:** Accepted for this release
- **Mitigation:** Symlinks within source directories are not a supported configuration in Roblox development. Root-level symlinks (project folder itself) still work because `canonicalize` is called on the root.
## Deferred Refactors
None identified beyond the small `strip_unc_prefix` extraction (included as Fix 4).
## Test Plan
### Rust Integration Tests (`tests/tests/`)
- **Hidden directory survival in clean syncback** (Fix 1): Create temp project with `src/.hidden/keep.txt`, run clean-mode syncback, assert `.hidden/keep.txt` still exists after syncback.
- **Overlapping `$path` roots don't create duplicates** (Fix 2): Create project with overlapping `$path` entries (`"src"` and `"src/shared"`), run `snapshot_from_vfs` with prefetch cache, verify no duplicate children.
### Rust Unit Tests
- **Nested path detection correctness** (Fix 3): Test the ancestor-tracking algorithm with paths `["a/b", "a/b/c", "a/b/d", "x/y", "x/y/z"]`, verify only `"a/b"` and `"x/y"` are emitted as top-level.
## Test Rules
- Every bug fix MUST have a test that would have FAILED before the fix and PASSES after.
- Fix 4 (refactor) does not need new tests -- existing test coverage is sufficient.
## Final Step: Run CI
After ALL fixes and tests are implemented, run the full CI pipeline (`cargo test --locked`, `cargo clippy`, `cargo fmt --check`). Every fix must pass CI before the plan is considered complete.

286
.cursor/plans/reconciler_robustness_overhaul_2f440e06.plan.md Normal file
View file

@ -0,0 +1,286 @@
---
name: Reconciler robustness overhaul
overview: Fix the Atlas plugin Reconciler to handle MeshPart creation via AssetService, MeshId swapping, and CSG reconstruction -- modeled after the proven rbxsync Sync.luau / CSGHandler.luau implementation.
todos:
- id: tags-attrs-verify
content: Verify Tags/Attributes work through existing RbxDom customProperties.lua path (read-only audit, no code changes)
status: pending
- id: meshpart-create
content: Add MeshPart creation via AssetService:CreateMeshPartAsync in reify.lua, skip MeshId in property loop
status: pending
- id: meshid-swap
content: "Add MeshId change detection in applyPatch.lua: recreate MeshPart, copy children/properties, swap in instanceMap"
status: pending
- id: csg-handler
content: Port CSGHandler from rbxsync to Atlas plugin (Plugin:Separate for decomposition, UnionAsync/SubtractAsync for reconstruction)
status: pending
- id: csg-twoway-decompose
content: "Add CSG decomposition to ChangeBatcher/encodeInstance: when encoding a CSG part, call Plugin:Separate and send component parts to server"
status: pending
- id: csg-server-store
content: "Server-side: store decomposed CSG component parts on disk and send them back to plugin during forward sync"
status: pending
- id: csg-reify-reconstruct
content: "Plugin-side: detect CSG component data in patches, reconstruct via CSGHandler in reify.lua"
status: pending
- id: csg-warning
content: Add warning for .rbxm-sourced CSG parts that have never been decomposed (appear as empty shells until first two-way sync)
status: pending
isProject: false
---
# Reconciler Robustness Overhaul
## Tags/Attributes: Already Working (Verification Only)
Tags and Attributes ARE correctly handled through the RbxDom `customProperties.lua` system ([plugin/rbx_dom_lua/customProperties.lua](plugin/rbx_dom_lua/customProperties.lua) lines 51-120):
- **Tags** (lines 97-120): Uses `CollectionService:GetTags()` for read, `CollectionService:AddTag()/RemoveTag()` for write. Properly diffs existing tags against new tags.
- **Attributes** (lines 51-96): Uses `instance:GetAttributes()` for read, `instance:SetAttribute()` for write. Clears removed attributes. Skips reserved `RBX` prefixed names.
They arrive in the `Properties` map from the server (e.g. `Properties.Tags`, `Properties.Attributes`) and flow through `setProperty` -> RbxDom `PropertyDescriptor:write` -> custom handler. The property application order within a single instance doesn't matter per user confirmation.
No code changes needed -- just verify during implementation that this path is exercised.
---
## Problem 1: MeshPart Creation Fails Silently
**File:** [plugin/src/Reconciler/reify.lua](plugin/src/Reconciler/reify.lua) line 63
```lua
local createSuccess, instance = pcall(Instance.new, virtualInstance.ClassName)
```
`Instance.new("MeshPart")` creates an instance with empty MeshId. **MeshId is read-only** after creation in plugins. The server sends MeshId in Properties, but `setProperty` fails silently (property added to `unappliedProperties`). Result: every MeshPart synced from server has no mesh geometry.
**rbxsync reference:** [Sync.luau](d:\UserGenerated\rbxsync\plugin\src\Sync.luau) lines 624-681 use `AssetService:CreateMeshPartAsync(meshId)`, then apply remaining properties with MeshId excluded.
**Fix in reify.lua:** Before `Instance.new`, check for MeshPart and use `AssetService:CreateMeshPartAsync`:
```lua
local instance
local meshIdAlreadySet = false
if virtualInstance.ClassName == "MeshPart" then
local meshIdValue = virtualInstance.Properties.MeshId
if meshIdValue then
local decodeSuccess, meshId = decodeValue(meshIdValue, instanceMap)
if decodeSuccess and type(meshId) == "string" and meshId ~= "" then
local AssetService = game:GetService("AssetService")
local ok, meshPart = pcall(function()
return AssetService:CreateMeshPartAsync(meshId)
end)
if ok and meshPart then
instance = meshPart
meshIdAlreadySet = true
end
end
end
end
if instance == nil then
local createSuccess
createSuccess, instance = pcall(Instance.new, virtualInstance.ClassName)
if not createSuccess then
addAllToPatch(unappliedPatch, virtualInstances, id)
return
end
end
```
Then in the property loop, skip MeshId if already set:
```lua
if meshIdAlreadySet and propertyName == "MeshId" then
continue
end
```
---
## Problem 2: MeshId Swap on Existing MeshPart
**File:** [plugin/src/Reconciler/applyPatch.lua](plugin/src/Reconciler/applyPatch.lua) lines 250-277
When an update changes MeshId on an existing MeshPart, `setProperty` fails (read-only). The property ends up in `unappliedUpdate` -- the MeshPart keeps the old mesh.
**rbxsync reference:** [Sync.luau](d:\UserGenerated\rbxsync\plugin\src\Sync.luau) lines 852-941 detect MeshId change, recreate via `CreateMeshPartAsync`, copy properties/children, destroy old.
**Fix in applyPatch.lua:** Add a MeshPart recreation block before the normal update logic:
```lua
-- Before the normal property update loop:
if instance:IsA("MeshPart") and update.changedProperties
and update.changedProperties.MeshId then
local decodeSuccess, newMeshId = decodeValue(update.changedProperties.MeshId, instanceMap)
if decodeSuccess and type(newMeshId) == "string" and newMeshId ~= "" then
local AssetService = game:GetService("AssetService")
local ok, newMeshPart = pcall(function()
return AssetService:CreateMeshPartAsync(newMeshId)
end)
if ok and newMeshPart then
-- Apply name
newMeshPart.Name = update.changedName or instance.Name
-- Apply changed properties (excluding MeshId)
for propName, propValue in pairs(update.changedProperties) do
if propName ~= "MeshId" then
local ds, dv = decodeValue(propValue, instanceMap)
if ds then setProperty(newMeshPart, propName, dv) end
end
end
-- Move children
for _, child in ipairs(instance:GetChildren()) do
child.Parent = newMeshPart
end
-- Swap in instanceMap
local parent = instance.Parent
newMeshPart.Parent = parent
instanceMap:insert(update.id, newMeshPart)
instance.Parent = nil -- Not Destroy(), allows undo
continue -- Skip normal update path
end
end
end
```
---
## Problem 3: CSG Parts Created as Empty Shells
### Root Cause
The server filters out CSG geometry data before sending to the plugin:
- `**MeshData**` (BinaryString) -- can't be serialized to JSON/MessagePack ([src/web/api.rs](src/web/api.rs) line 3606-3608)
- `**PhysicalConfigData**` (SharedString) -- explicitly filtered ([src/web/interface.rs](src/web/interface.rs) lines 147-153)
Even if these bytes reached the plugin, they're **not scriptable** -- no Roblox API exists to set `MeshData` or `PhysicalConfigData` on instances from Lua. No plugin API exists to load raw .rbxm bytes into instances either.
**The only way to create CSG geometry in Studio is `BasePart:UnionAsync()` / `BasePart:SubtractAsync()` from component parts.** This is exactly what rbxsync does.
### CSG Data Flow
```
┌─────────────────────────────────────────────────────────────────────┐
│ TWO-WAY SYNC (Studio → Server → Studio) │
│ │
│ 1. User edits CSG in Studio │
│ 2. Plugin detects change via ChangeBatcher │
│ 3. Plugin calls Plugin:Separate() to decompose → component parts │
│ 4. Plugin encodes component parts + CSG metadata │
│ 5. Server stores component data alongside instance │
│ 6. On next forward sync: server sends component data │
│ 7. Plugin reconstructs via UnionAsync/SubtractAsync │
│ │
│ LIMITATION: .rbxm-sourced CSG that was NEVER synced from Studio │
│ will appear as empty shells until first two-way sync. │
│ A warning is logged for these cases. │
└─────────────────────────────────────────────────────────────────────┘
```
### Implementation: 4 parts
#### Part A: Port CSGHandler module
Create [plugin/src/CSGHandler.lua](plugin/src/CSGHandler.lua), ported from [rbxsync CSGHandler.luau](d:\UserGenerated\rbxsync\plugin\src\CSGHandler.luau).
Key functions:
- `CSGHandler.isCSGOperation(instance)` -- check if UnionOperation/IntersectOperation/NegateOperation
- `CSGHandler.separate(pluginInstance, union)` -- decompose via `Plugin:Separate()`, returns `{addParts, subtractParts}`
- `CSGHandler.reconstruct(basePart, additionalParts, subtractParts)` -- rebuild via `UnionAsync/SubtractAsync`
#### Part B: CSG decomposition in two-way sync (Plugin → Server)
**File:** [plugin/src/ChangeBatcher/encodeInstance.lua](plugin/src/ChangeBatcher/encodeInstance.lua)
When encoding a CSG instance for syncback, decompose it:
1. Call `CSGHandler.separate(plugin, instance)` to get component parts
2. Encode each component part as a child instance under a special `_csg` key
3. Include CSG metadata (operation types: add vs subtract)
4. Send to server as part of the normal write patch
The server write handler ([src/web/api.rs](src/web/api.rs)) needs to recognize the `_csg` data and store it.
#### Part C: Server-side CSG storage and retrieval
**File:** [src/web/api.rs](src/web/api.rs) and [src/change_processor.rs](src/change_processor.rs)
When the server receives CSG component data via `/api/write`:
1. Store component part snapshots alongside the CSG instance (e.g., as metadata or in a `_csg/` subdirectory in the snapshot tree)
2. During forward sync, when sending a CSG instance to the plugin, include the component data in the instance's properties/metadata
This needs a new mechanism -- CSG component data is not a normal property. Options:
- Add a `csgComponents` field to the `Instance` struct in [src/web/interface.rs](src/web/interface.rs)
- Or encode as a special metadata entry in `InstanceMetadata`
#### Part D: CSG reconstruction in reify.lua
**File:** [plugin/src/Reconciler/reify.lua](plugin/src/Reconciler/reify.lua) and [plugin/src/Reconciler/applyPatch.lua](plugin/src/Reconciler/applyPatch.lua)
When reifying a CSG instance that has component data:
1. Instead of `Instance.new("UnionOperation")`, queue it for deferred CSG reconstruction
2. Create the component parts first (as normal instances, unparented)
3. After all instances are created, reconstruct CSG via `CSGHandler.reconstruct()`
4. Apply properties (skip CFrame -- geometry defines position, per rbxsync line 1499-1509)
5. Parent the reconstructed union
For CSG instances WITHOUT component data (from .rbxm, never two-way synced):
- Create the empty shell via `Instance.new()` as today
- Log a warning: "CSG instance 'X' has no component data -- appears as empty shell"
---
## Implementation Order
The changes are independent enough to implement incrementally:
1. **MeshPart creation** (reify.lua) -- immediate high-value fix, no server changes
2. **MeshId swap** (applyPatch.lua) -- immediate high-value fix, no server changes
3. **CSGHandler module** (new file) -- pure plugin code, self-contained
4. **CSG warning** (reify.lua) -- trivial addition
5. **CSG decomposition** (encodeInstance.lua + server) -- requires server changes
6. **CSG server storage** (api.rs + interface.rs) -- requires protocol thinking
7. **CSG reconstruction** (reify.lua) -- requires steps 5+6
Steps 1-4 can ship as one PR. Steps 5-7 require more design and could be a separate PR.
---
## Files Modified
| File | Changes |
| ------------------------------------------------------------------------------------------ | ----------------------------------------------------------- |
| [plugin/src/Reconciler/reify.lua](plugin/src/Reconciler/reify.lua) | MeshPart creation, CSG warning, CSG deferred reconstruction |
| [plugin/src/Reconciler/applyPatch.lua](plugin/src/Reconciler/applyPatch.lua) | MeshId swap detection + recreation |
| [plugin/src/CSGHandler.lua](plugin/src/CSGHandler.lua) | **NEW** -- ported from rbxsync |
| [plugin/src/ChangeBatcher/encodeInstance.lua](plugin/src/ChangeBatcher/encodeInstance.lua) | CSG decomposition during two-way sync |
| [src/web/interface.rs](src/web/interface.rs) | CSG component data in Instance struct |
| [src/web/api.rs](src/web/api.rs) | CSG storage on write, CSG data on read |
## Files Verified (no changes)
| File | Status |
| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| [plugin/rbx_dom_lua/customProperties.lua](plugin/rbx_dom_lua/customProperties.lua) | Tags (lines 97-120) and Attributes (lines 51-96) correctly handled |
| [plugin/src/Reconciler/setProperty.lua](plugin/src/Reconciler/setProperty.lua) | Routes to RbxDom descriptors, including custom handlers |
| [plugin/src/Reconciler/decodeValue.lua](plugin/src/Reconciler/decodeValue.lua) | Decodes all property types via RbxDom |
| [plugin/src/Reconciler/diff.lua](plugin/src/Reconciler/diff.lua) | Compares Tags/Attributes via Properties map |

88
.cursor/plans/ref_path_and_dedup_integration_81a47dfe.plan.md Normal file
View file

@ -0,0 +1,88 @@
---
name: Ref path and dedup integration
overview: "Two deferred items from the ambiguous path handling feature: (1) migrate collect_referents() to build filesystem-name-compatible paths so ref round-trips work for slugified/dedup'd instances, and (2) integrate dedup_suffix.rs cleanup rules into the change_processor deletion handler.\""
todos:
- id: ref-path-tentative-fs
content: Create tentative_fs_path() function and replace inst_path() calls in collect_referents/collect_all_paths
status: completed
- id: dedup-cleanup-change-processor
content: Integrate dedup_suffix cleanup rules into change_processor handle_tree_event removal handling
status: completed
isProject: false
---
# Syncback Ref Path Migration and Dedup Cleanup Integration
## Problem 1: Ref paths break for slugified/dedup'd instance names
### Root cause
`collect_referents()` in [src/syncback/ref_properties.rs](src/syncback/ref_properties.rs) calls `inst_path()` which delegates to the legacy `ref_target_path()`. This produces escaped instance-name paths like `"Workspace/Hey\/Bro"`. But `get_instance_by_path()` in [src/snapshot/tree.rs](src/snapshot/tree.rs) now uses simple `path.split('/')` without escape handling. An instance named `"Hey/Bro"` gets path `"Workspace/Hey\/Bro"`, which splits into `["Workspace", "Hey\\/Bro"]` -- the escaped slash is NOT unescaped, so resolution fails.
The fallback in `get_instance_by_path()` tries case-insensitive instance name matching, but the segment `"Hey\\/Bro"` does not match instance name `"Hey/Bro"`.
### Fix: Build filesystem-compatible paths in collect_referents
Create a `tentative_fs_path()` function that builds paths compatible with the new `get_instance_by_path()`:
- Walk ancestors from target to root (same as `ref_target_path`)
- For each instance, compute a **tentative filesystem name**: `slugify(name)` + extension based on class/children
- Join with `/` (no escaping needed since slugified names can't contain `/`)
The extension mapping uses the same class-to-middleware logic as `get_best_middleware()`:
- `Script` -> check RunContext: `.server.luau` / `.client.luau` / `.plugin.luau` / `.legacy.luau` / `.local.luau`; if has children, directory (no ext)
- `LocalScript` -> `.local.luau` or directory
- `ModuleScript` -> `.luau` or directory
- `Folder`, `Configuration`, `Tool`, GUI classes -> directory (no ext)
- `StringValue` -> `.txt`
- `LocalizationTable` -> `.csv`
- Everything else -> `.model.json5` or directory if has children
**Files to change:**
- [src/syncback/ref_properties.rs](src/syncback/ref_properties.rs): Replace `inst_path()` calls with new `tentative_fs_path()`. Update `collect_all_paths()` similarly.
- [src/syncback/snapshot.rs](src/syncback/snapshot.rs): Update `inst_path()` to use the new path builder (or deprecate it for ref contexts).
### How resolution works after this fix
A ref path like `"Workspace/Hey_Bro.server.luau"`:
1. Primary lookup: `filesystem_name_for()` returns `"Hey_Bro.server.luau"` (from instigating_source) -- **match**
2. If no instigating_source: fallback matches instance name `"Hey/Bro"` against segment `"Hey_Bro.server.luau"` -- no match, but this is expected for file-backed instances
For directory instances like services: `"Workspace"` matches both filesystem name and instance name -- works as before.
---
## Problem 2: Dedup cleanup not triggered on deletion in change_processor
### Current state
[src/syncback/dedup_suffix.rs](src/syncback/dedup_suffix.rs) provides `compute_cleanup_action()` with gap-tolerant, base-name-promotion, and group-to-1 rules. But `handle_tree_event()` in [src/change_processor.rs](src/change_processor.rs) (line 1043) only logs removals -- it doesn't check if a deletion leaves a dedup group needing suffix cleanup.
### Fix: Add dedup cleanup after removals
In `handle_tree_event()`, after the removal loop (line ~1070), for each removed instance:
1. Find the parent instance (before the removal is applied to the tree)
2. Enumerate the parent's remaining children (excluding the removed one)
3. Group remaining children by dedup base name (parse `~N` suffixes from filesystem names via `parse_dedup_suffix()`)
4. For each dedup group affected by the removal, call `compute_cleanup_action()`
5. If the action is `RemoveSuffix` or `PromoteLowest`:
a. Call `suppress_path()` for both old and new paths
b. Rename the file on disk
c. Update meta file `name` field if needed
d. Update `RefPathIndex` via `update_ref_paths_after_rename()`
**Files to change:**
- [src/change_processor.rs](src/change_processor.rs): Add dedup cleanup logic after the removal loop in `handle_tree_event()`. Import `dedup_suffix::{parse_dedup_suffix, compute_cleanup_action}`.
### Edge cases
- The removed instance's parent must still exist in the tree when we enumerate siblings
- ProjectNode instances are excluded (already guarded)
- The cleanup rename must happen BEFORE `apply_patch_set()` removes the instance from the tree (since we need the parent relationship)
- VFS suppression must cover both the old path (being renamed from) and the new path (being renamed to)

101
.cursor/plans/remove_plugin_animations_f4a43647.plan.md Normal file
View file

@ -0,0 +1,101 @@
---
name: Remove Plugin Animations
overview: Replace every `Flipper.Spring.new(...)` with `Flipper.Instant.new(...)` across all plugin components to make all UI transitions instant, and remove the 0.3s double-click delay in DomLabel expand/collapse.
todos:
- id: domlabel
content: "DomLabel.lua: Replace 2 Spring goals with Instant, remove task.delay(0.3) wrapper from expand logic"
status: completed
- id: page
content: "Page.lua: Replace Spring goal with Instant"
status: completed
- id: touchripple
content: "TouchRipple.lua: Replace 3 Spring goals with Instant, remove EXPAND_SPRING constant"
status: completed
- id: textbutton
content: "TextButton.lua: Replace 3 Spring goals with Instant, remove SPRING_PROPS constant"
status: completed
- id: textinput
content: "TextInput.lua: Replace 3 Spring goals with Instant, remove SPRING_PROPS constant"
status: completed
- id: notification
content: "Notification.lua: Replace 2 Spring goals with Instant"
status: completed
- id: dropdown
content: "Dropdown.lua: Replace Spring goal with Instant"
status: completed
- id: iconbutton
content: "IconButton.lua: Replace 2 Spring goals with Instant, remove HOVER_SPRING_PROPS constant"
status: completed
- id: checkbox
content: "Checkbox.lua: Replace Spring goal with Instant"
status: completed
isProject: false
---
# Remove All Animations From Plugin
## Strategy
Every animation in the plugin uses Flipper motors with `Flipper.Spring.new(value, { frequency, dampingRatio })` goals. The Flipper library already has `Flipper.Instant.new(value)` which sets the motor value immediately with no interpolation. The fix is a mechanical replacement of every `Flipper.Spring.new(...)` with `Flipper.Instant.new(...)`.
This preserves the binding/motor infrastructure (no structural refactoring needed) while making every transition instant.
Additionally, the DomLabel expand/collapse has a `task.delay(0.3)` that waits to distinguish single-click from double-click before expanding. This delay should be removed so expansion is immediate on click.
## Files to Change (10 files, ~20 replacements)
### 1. [DomLabel.lua](plugin/src/App/Components/PatchVisualizer/DomLabel.lua) -- Expand/collapse
- Line 436: `Flipper.Spring.new(24, {...})` -> `Flipper.Instant.new(24)`
- Line 602: `Flipper.Spring.new(goalHeight, {...})` -> `Flipper.Instant.new(goalHeight)`
- Lines 545-608: Remove the `task.delay(0.3)` wrapper so expansion logic runs immediately on single click (keep double-click detection, just don't delay the expand)
### 2. [Page.lua](plugin/src/App/Page.lua) -- Page transitions
- Line 65: `Flipper.Spring.new(...)` -> `Flipper.Instant.new(...)`
### 3. [TouchRipple.lua](plugin/src/App/Components/TouchRipple.lua) -- Click ripple effect
- Line 84: `Flipper.Spring.new(1, EXPAND_SPRING)` -> `Flipper.Instant.new(1)`
- Line 85: `Flipper.Spring.new(1, EXPAND_SPRING)` -> `Flipper.Instant.new(1)`
- Line 93: `Flipper.Spring.new(0, {...})` -> `Flipper.Instant.new(0)`
- Remove unused `EXPAND_SPRING` constant
### 4. [TextButton.lua](plugin/src/App/Components/TextButton.lua) -- Hover/enabled
- Line 36: `Flipper.Spring.new(...)` -> `Flipper.Instant.new(...)`
- Line 64: `Flipper.Spring.new(1, SPRING_PROPS)` -> `Flipper.Instant.new(1)`
- Line 70: `Flipper.Spring.new(0, SPRING_PROPS)` -> `Flipper.Instant.new(0)`
- Remove unused `SPRING_PROPS` constant
### 5. [TextInput.lua](plugin/src/App/Components/TextInput.lua) -- Hover/enabled
- Line 34: `Flipper.Spring.new(...)` -> `Flipper.Instant.new(...)`
- Line 101: `Flipper.Spring.new(1, SPRING_PROPS)` -> `Flipper.Instant.new(1)`
- Line 107: `Flipper.Spring.new(0, SPRING_PROPS)` -> `Flipper.Instant.new(0)`
- Remove unused `SPRING_PROPS` constant
### 6. [Notification.lua](plugin/src/App/Components/Notifications/Notification.lua) -- Appear/dismiss
- Line 43: `Flipper.Spring.new(0, {...})` -> `Flipper.Instant.new(0)`
- Line 50: `Flipper.Spring.new(1, {...})` -> `Flipper.Instant.new(1)`
### 7. [Dropdown.lua](plugin/src/App/Components/Dropdown.lua) -- Open/close
- Line 39: `Flipper.Spring.new(...)` -> `Flipper.Instant.new(...)`
### 8. [IconButton.lua](plugin/src/App/Components/IconButton.lua) -- Hover circle
- Line 41: `Flipper.Spring.new(1, HOVER_SPRING_PROPS)` -> `Flipper.Instant.new(1)`
- Line 45: `Flipper.Spring.new(0, HOVER_SPRING_PROPS)` -> `Flipper.Instant.new(0)`
- Remove unused `HOVER_SPRING_PROPS` constant
### 9. [Checkbox.lua](plugin/src/App/Components/Checkbox.lua) -- Active state
- Line 26: `Flipper.Spring.new(...)` -> `Flipper.Instant.new(...)`
## Not Touched
- **Spinner.lua** -- Uses `RenderStepped` rotation, not Flipper springs. This is a loading indicator, not a decorative animation. Keeping it.
- **bindingUtil.lua** -- `mapLerp`, `blendAlpha`, `fromMotor` are utility functions still needed by the instant-value bindings. No changes needed.
- **Flipper package** -- Vendored, not modified.

156
.cursor/plans/rewrite_matching_algorithm_df45bbe1.plan.md Normal file
View file

@ -0,0 +1,156 @@
---
name: Rewrite matching algorithm
overview: "Replace the multi-pass matching algorithm in all 3 implementations with a recursive change-count scoring function and greedy assignment. Sort key is a tuple: (change_count, child_index_A, child_index_B). Fast-paths for Ref pins and (Name, ClassName) grouping."
todos:
- id: rewrite-lua-matching
content: "Rewrite Lua matching.lua: recursive computeChangeCount + greedy assign with Ref pin and (Name, ClassName) fast-paths"
status: completed
- id: rewrite-rust-forward
content: "Rewrite Rust forward sync matching.rs: recursive count_changes + greedy assign using variant_eq"
status: completed
- id: rewrite-rust-syncback
content: "Rewrite Rust syncback matching.rs: recursive count_changes + greedy assign with hash fast-path"
status: completed
- id: verify-tests
content: Run cargo test to verify all 714 tests still pass
status: completed
isProject: false
---
# Rewrite Matching Algorithm
Replace all 3 matching implementations with one recursive scoring function and greedy assignment.
## matchChildren (entry point, called per parent)
```
matchChildren(sideA, sideB, parentA, parentB) -> (matched, unmatchedA, unmatchedB):
// Fast-path 1: Ref pin (confirmed identity, highest priority)
// Scan parentA and parentB for Ref properties (PrimaryPart, Value, etc.)
// that point to children. If parentA.PrimaryPart = childX on sideA and
// parentB.PrimaryPart = childY on sideB, pin (X, Y) as matched.
// Remove pinned children from sideA/sideB.
for each Ref property name shared by parentA and parentB:
targetA = parentA[refProp]
targetB = parentB[refProp]
if targetA in sideA and targetB in sideB:
match (targetA, targetB)
remove both from sideA/sideB
// Fast-path 2: Group remaining by (Name, ClassName)
// 1:1 groups get instant-matched with zero scoring.
groups = group sideA and sideB by (Name, ClassName)
for each group:
if exactly 1 on each side: instant match, remove both
// Ambiguous groups: score + greedy assign
for each group with multiple on at least one side:
pairs = []
bestSoFar = infinity
for each (A, B) in cross-product of group: // iteration order = child order
cost = computeChangeCount(A, B, bestSoFar) // RECURSIVE, early-exits
pairs.append((cost, A, B))
if cost < bestSoFar: bestSoFar = cost
stable sort pairs by cost ascending
// Stable sort preserves insertion order on ties = original child order
greedy assign: pick first unmatched, remove both, repeat
// Leftovers = new/deleted
```
The two fast-paths resolve the vast majority of children with zero scoring. Only truly ambiguous groups (same Name, same ClassName, multiple candidates) ever call `computeChangeCount`.
## computeChangeCount (recursive scoring function)
One function. Returns the total number of changes the reconciler would need to make to turn A into B, **including the entire subtree**.
```
computeChangeCount(A, B, bestSoFar) -> number:
// Cheap: all own diffs in one shot (flat loops, no branching)
cost = countOwnDiffs(A, B) // properties + tags + attributes
if cost >= bestSoFar: return cost
// Expensive: recursive children matching
childResult = matchChildren(A.children, B.children, A, B) // RECURSE
for each (childA, childB) in childResult.matched:
cost += computeChangeCount(childA, childB, bestSoFar - cost)
if cost >= bestSoFar: return cost
cost += count(childResult.unmatched) * UNMATCHED_PENALTY
return cost
countOwnDiffs(A, B) -> number:
cost = 0
for each property key in union(A.properties, B.properties):
if values differ or key missing on one side: cost += 1
for each tag in symmetricDifference(A.Tags, B.Tags): cost += 1
for each attr key in union(A.Attributes, B.Attributes):
if missing or value differs: cost += 1
return cost
```
`computeChangeCount` and `matchChildren` are **mutually recursive**. Base case: leaf nodes (no children).
Name and ClassName are NOT in the change count -- the (Name, ClassName) grouping in `matchChildren` already guarantees they match for any pair that reaches `computeChangeCount`.
## Unmatched Child Penalty
Each unmatched child (create or delete an entire instance) adds a **large constant** (`UNMATCHED_PENALTY`) to the parent pair's cost. No need to recursively calculate the subtree -- creating/deleting an instance is categorically more expensive than tweaking properties, so a flat constant correctly prioritizes pairings that avoid creates/deletes.
`UNMATCHED_PENALTY` should be large enough that any number of property diffs on matched children is still cheaper than one unmatched child. A value like `10,000` works (no real instance has 10K properties).
## Sort Key and Stability
Sort key is `change_count`, ascending. On ties, original child order is preserved.
- **Rust:** `slice::sort_by` is stable natively. No extra work needed.
- **Lua:** `table.sort` is NOT stable (Luau uses non-stable quicksort). Use a simple ad-hoc stable sort: pairs are built with their insertion index, and the Lua sort comparator breaks ties by index. Something like `if a.cost == b.cost then return a.idx < b.idx end; return a.cost < b.cost`.
## Known Limitation: Greedy vs Optimal
The greedy algorithm (pick lowest-cost pair first) is NOT globally optimal. Edge cases exist where a locally-best pick forces a globally-worse assignment. The Hungarian algorithm (O(N^3)) would give optimal results, but ambiguous groups are typically small (2-10 instances) and the greedy works correctly for the common case. Flagged for revisiting if real-world games hit this edge case.
## What Counts as a Change (+1 each)
- **Each differing property:** iterate union of property keys on both sides. Each key where values differ or one side missing = +1. Includes CFrame, Size, Color3, Source, Material, Ref properties (PrimaryPart, Value), everything. Tags and Attributes are excluded from this loop (counted separately below).
- **Tags:** count individual tag operations. Each tag that needs to be added or removed = +1. If A has `{"Foo", "Bar"}` and B has `{"Bar", "Baz"}`, cost = 2 (remove "Foo", add "Baz").
- **Attributes:** count individual attribute operations. Each attribute that needs to be added, removed, or changed = +1. If A has `{X=1, Y=2}` and B has `{Y=3, Z=4}`, cost = 3 (remove X, change Y, add Z).
- **Recursive children cost:** total from `matchChildren` (matched pair costs + unmatched subtree costs).
Things NOT in the change count:
- Name -- grouping guarantees match.
- ClassName -- grouping guarantees match.
- Parent -- structural.
## Files to Rewrite
### 1. Lua -- `[plugin/src/Reconciler/matching.lua](plugin/src/Reconciler/matching.lua)`
Complete rewrite. Three functions: `matchChildren`, `computeChangeCount`, `costOfEntireInstance`. Property comparison via pcall reads from Studio instances vs virtual `.Properties`. Ref pin scans parent's Ref properties. (Name, ClassName) grouping as fast-path.
### 2. Rust forward sync -- `[src/snapshot/matching.rs](src/snapshot/matching.rs)`
Complete rewrite. Same three functions. Property comparison using `variant_eq()` from `[src/variant_eq.rs](src/variant_eq.rs)`. Operates on `InstanceSnapshot` vs `RojoTree` instances.
### 3. Rust syncback -- `[src/syncback/matching.rs](src/syncback/matching.rs)`
Complete rewrite. Same three functions. Hash-match fast-path: if `new_hash == old_hash`, return 0 without checking properties. Otherwise `variant_eq()` per property. Operates on `WeakDom` instances.
## What Gets Deleted
- All multi-pass infrastructure (Pass 1/2/3, `_pass1NameAndClass`, `_pass3Similarity`, etc.)
- `compute_similarity` / `compute_forward_similarity` / `_computeSimilarity`
- `collect_ref_property_targets`, `pass2_ref_discriminators`
- Replaced by: `matchChildren`, `computeChangeCount`, `costOfEntireInstance`
## Correctness
- Ref-pinned pairs are pulled out first -- confirmed identity, never wrong
- (Name, ClassName) grouping prevents cross-class or cross-name pairing
- Within ambiguous groups, recursive change counting picks the pair requiring fewest total reconciler operations across the entire subtree
- Child order tiebreaker ensures determinism on equal scores
- Unmatched subtree cost correctly penalizes pairings that cause cascading creates/deletes

390
.cursor/plans/rokit_install_audit_fixes_e465275a.plan.md Normal file
View file

@ -0,0 +1,390 @@
---
name: Rokit Install Audit Fixes
overview: Fix the EPERM crash when deleting a globally installed atlas.exe, and harden the entire rokit installation pipeline in the VS Code extension for reliability across Windows, macOS, and Linux.
todos:
- id: bug1-elevated-delete
content: Add elevatedDelete() helper and fix the 'Switch to Rokit' flow in openMenu.ts to handle EPERM/EACCES/EBUSY with platform-specific elevation
status: completed
- id: bug2-progress
content: Wrap installRojo() in vscode.window.withProgress() with stage reporting
status: completed
- id: bug3-github-api
content: Replace eryn.io proxy URL with official GitHub API URL in installRojo.ts
status: completed
- id: bug4-temp-cleanup
content: Delete temp rokit binary after successful self-install
status: completed
- id: bug5-path-dedup
content: Deduplicate PATH entry and verify rokit accessibility before rokit add
status: completed
- id: bug6-stream
content: Replace manual stream piping with stream.promises.pipeline()
status: completed
- id: bug7-rokit-add-existing
content: Use 'rokit add --global --force' + 'rokit trust' + verify atlas --version + non-fatal plugin install
status: completed
- id: feat-check-updates
content: Add 'Check for Updates' menu item in openMenu.ts for rokit-managed installs, handle project-level + global rokit update, reinstall Studio plugin after update
status: completed
- id: feat-auto-update-check
content: Add background update check on activation using rokit update --check (once per day, project + global), prompt user with Update button that triggers rokit update + plugin reinstall
status: completed
isProject: false
---
# Rokit Installation Audit and Fixes
## Audit Findings
### How Rokit Installation Works (No Elevation Needed)
Rokit installs entirely to `~/.rokit/` (user-level directory). `rokit self-install` modifies user-level PATH environment variables (HKCU on Windows, shell rc files on Unix). **Neither rokit installation nor `rokit add` requires elevation.**
The only scenario requiring elevation is the "Switch to Rokit" migration flow, which tries to delete an existing globally installed `atlas.exe` from a protected directory like `C:\Program Files\Atlas` (written by [install.ps1](install.ps1)) or `/usr/local/bin/atlas` (written by [install.sh](install.sh)).
### Issues Found (by severity)
---
**BUG 1 -- CRITICAL: `fs.unlink` fails without elevation, crashes the entire flow**
In [vscode-rojo/src/commands/openMenu.ts](vscode-rojo/src/commands/openMenu.ts) line 133:
```typescript
return fs.unlink(install.resolvedPath)
```
When atlas.exe is in `C:\Program Files\Atlas` or `/usr/local/bin/`, this throws EPERM/EACCES. The `.then()` error handler on line 139 catches it but just shows a generic message and the flow dies. Additionally, on Windows the exe may be **locked by a running process** (atlas serve), causing EBUSY even with elevation.
**Fix:** Add an `elevatedDelete()` helper that:
1. Tries `fs.unlink()` first
2. On EPERM/EACCES, uses platform-specific elevated delete:
- **Windows:** `powershell -Command "Start-Process powershell -ArgumentList '-Command Remove-Item \"path\"' -Verb RunAs -Wait"` (triggers UAC prompt)
- **macOS:** `osascript -e 'do shell script "rm path" with administrator privileges'` (triggers password dialog)
- **Linux:** `pkexec rm path` (triggers PolicyKit dialog)
3. On EBUSY (Windows exe locked), shows a specific error message telling the user to stop atlas serve first
---
**BUG 2 -- HIGH: No progress indicator during installation**
In [vscode-rojo/src/installRojo.ts](vscode-rojo/src/installRojo.ts) `installRojo()`:
The entire flow (download rokit zip ~3.5MB, extract, self-install, `rokit add` atlas) runs silently. Users see zero feedback and may think the extension froze.
**Fix:** Wrap the body of `installRojo()` in `vscode.window.withProgress()` with `ProgressLocation.Notification` and `cancellable: false`. Report progress at each stage:
- "Downloading Rokit..."
- "Installing Rokit..."
- "Installing Atlas via Rokit..."
---
**BUG 3 -- HIGH: Third-party release URL is a single point of failure**
In [vscode-rojo/src/installRojo.ts](vscode-rojo/src/installRojo.ts) line 113:
```typescript
const latestReleaseResponse = await fetch(
"https://latest-github-release.eryn.io/rojo-rbx/rokit",
)
```
This third-party proxy (`eryn.io`) is not under our control. If it goes down, installation breaks. The official rokit install scripts use the GitHub API directly.
**Fix:** Replace with the official GitHub API URL: `https://api.github.com/repos/rojo-rbx/rokit/releases/latest`. Verified: response schema is identical, both return the same `GitHubRelease` shape. Add an `Accept: application/vnd.github+json` header per GitHub API best practice.
---
**BUG 4 -- MEDIUM: Temp files never cleaned up**
In [vscode-rojo/src/installRojo.ts](vscode-rojo/src/installRojo.ts) line 143-170:
The rokit binary is downloaded to `os.tmpdir()` but never deleted after `rokit self-install` succeeds.
**Fix:** Add `fs.promises.unlink(tempPath).catch(() => {})` after the `exec(self-install)` call succeeds.
---
**BUG 5 -- MEDIUM: PATH update is fragile**
In [vscode-rojo/src/installRojo.ts](vscode-rojo/src/installRojo.ts) lines 177-181:
```typescript
if ("PATH" in process.env) {
const envPath = process.env.PATH!.split(path.delimiter)
envPath.push(path.join(os.homedir(), ".rokit", "bin"))
process.env.PATH = envPath.join(path.delimiter)
}
```
This only updates the extension's own process environment. But the subsequent `rokit add --global` on line 184 runs via `exec()` which inherits this env, so it should work. The concern is:
- It doesn't check if the path is already present (could add duplicates on repeated attempts)
- The rokit binary path might differ on some systems
**Fix:** Check for duplicates before appending. Also verify the rokit binary is actually accessible after the PATH update by doing a quick `which("rokit")` check before proceeding to `rokit add`.
---
**BUG 6 -- LOW: Stream error handling is fragile**
In [vscode-rojo/src/installRojo.ts](vscode-rojo/src/installRojo.ts) lines 51-60, 149-163:
`promisifyStream` resolves on whichever of "close"/"finish"/"end" fires first. The pipeline/pipe chain has multiple potential failure modes that aren't properly propagated.
**Fix:** Use `stream.promises.pipeline()` (available in Node 15+, which VS Code ships with) instead of manually piping and tracking events. This gives proper error propagation and backpressure handling. Alternatively, simplify to `await promisify(stream.pipeline)(download.body!, unzipper.ParseOne(), fs.createWriteStream(tempPath))`.
---
**BUG 7 -- HIGH: `rokit add --global` crashes if atlas is already configured**
In [vscode-rojo/src/installRojo.ts](vscode-rojo/src/installRojo.ts) line 184:
```typescript
await exec("rokit add --global UserGeneratedLLC/rojo atlas")
```
This is the **only** action after rokit is confirmed installed. It fails if atlas is already in the project's `rokit.toml` or already added globally. The scenario:
1. Project has `rokit.toml` with atlas listed but `rokit install` hasn't been run yet
2. `which("atlas")` finds the rokit shim at `~/.rokit/bin/atlas`
3. `atlas --version` fails (rokit says tool not installed), stderr contains "rokit" -> `getRojoInstall` returns null
4. User sees "Install Atlas now", clicks it
5. `isRokitInstalled()` returns true (skips rokit download)
6. `rokit add --global` fails because atlas is already configured -> **onboarding dies**
**Fix:** Use `--force` to handle "already exists", trust the tool, then verify and install the plugin:
```typescript
await exec("rokit add --global --force UserGeneratedLLC/rojo atlas")
await exec("rokit trust UserGeneratedLLC/rojo")
await exec("atlas --version")
// Install the Studio plugin (non-fatal -- don't crash onboarding if Studio isn't installed)
try {
await exec("atlas plugin install")
} catch (e: any) {
vscode.window.showWarningMessage(
`Atlas installed successfully, but the Studio plugin could not be installed: ${e.stderr || e}. ` +
`You can install it later from the Atlas menu or by running "atlas plugin install".`
)
}
```
- `--force` on `rokit add` handles the "already exists" case cleanly -- no try/catch needed, it re-adds/updates regardless
- `rokit trust` marks `UserGeneratedLLC/rojo` as trusted so rokit won't prompt the user for confirmation when running atlas (without trust, `rokit install` and first-run shims can block waiting for user input)
- `atlas --version` verifies the tool is actually usable after add
- `atlas plugin install` writes the embedded Studio plugin to disk (non-fatal if Studio isn't installed yet)
---
---
**FEATURE: "Check for Updates" menu item for Rokit-managed installs**
In [vscode-rojo/src/commands/openMenu.ts](vscode-rojo/src/commands/openMenu.ts), the `generateProjectMenu()` return array (line 315) has action items like Studio, Syncback, Sourcemap, Install Plugin. There is no way to update Atlas from the extension.
`generateProjectMenu()` already tracks `installType` (line 168) and whether it's `InstallType.rokit`. We use this to conditionally show the item.
**Implementation:**
1. Add a menu item in `generateProjectMenu()` return array, conditionally included when `installType === InstallType.rokit`:
```typescript
...(installType === InstallType.rokit
? [
{
label: "$(sync) Check for Updates",
description: `Atlas v${allRojoVersions[0]}`,
action: "checkUpdates",
projectFile: projectFiles[0],
},
]
: []),
```
1. Add `"checkUpdates"` case in the `onDidAccept` handler (after the `"install"` case). The handler needs to support both project-level `rokit.toml` and global installations:
```typescript
case "checkUpdates": {
if (!selectedItem.projectFile) return
const updateFolder = path.dirname(selectedItem.projectFile.path.fsPath)
input.hide()
await vscode.window.withProgress(
{ location: vscode.ProgressLocation.Notification, title: "Checking for Atlas updates..." },
async (progress) => {
// Try project-level update first, fall back to global
let updateOutput: string
try {
const result = await exec("rokit update atlas", { cwd: updateFolder })
updateOutput = result.stdout || result.stderr
} catch {
try {
const result = await exec("rokit update --global atlas")
updateOutput = result.stdout || result.stderr
} catch (e: any) {
vscode.window.showErrorMessage(`Could not update Atlas: ${e.stderr || e}`)
return
}
}
// Re-install the Studio plugin so it matches the new binary.
// The plugin is embedded in the atlas binary at compile time
// (include_bytes! in src/cli/plugin.rs), so updating the binary
// without re-installing leaves a stale AtlasManagedPlugin.rbxm
// in Studio's plugins folder. Mismatched versions break live sync
// due to protocol version enforcement.
progress.report({ message: "Reinstalling Studio plugin..." })
try {
await exec("atlas plugin install")
} catch (e: any) {
vscode.window.showWarningMessage(
`Atlas updated, but plugin reinstall failed: ${e.stderr || e}. ` +
`Run "atlas plugin install" manually to avoid version mismatch.`
)
}
vscode.window.showInformationMessage(updateOutput.trim() || "Atlas is up to date.")
},
)
break
}
```
The strategy: try `rokit update atlas` in the project directory first. If atlas isn't in the project's `rokit.toml`, this fails and we fall back to `rokit update --global atlas`. Rokit's output tells the user whether an update was applied or they're already current.
After a successful update, immediately runs `atlas plugin install` to overwrite the stale `AtlasManagedPlugin.rbxm` in Studio's plugins folder. The plugin is baked into the atlas binary via `include_bytes!` at compile time (`src/cli/plugin.rs:12`), so the `.rbxm` in Studio never auto-updates -- it must be explicitly reinstalled. If the plugin reinstall fails (e.g. Studio not installed), a warning is shown with manual instructions rather than failing the whole update.
---
**FEATURE: Background update check on extension activation**
In [vscode-rojo/src/extension.ts](vscode-rojo/src/extension.ts) `activate()`:
Rokit has `rokit update atlas --check` which checks for updates without installing. Supports both project-level (`rokit update atlas --check` in the project dir, reads `rokit.toml`) and global (`rokit update --global atlas --check`).
**Implementation:** Add a new file `vscode-rojo/src/checkForUpdates.ts` with a `checkForAtlasUpdates(context)` function, called fire-and-forget from `activate()`:
```typescript
export async function checkForAtlasUpdates(context: vscode.ExtensionContext) {
// Throttle: once per day
const lastCheck = context.globalState.get<number>("atlas::lastUpdateCheck")
const ONE_DAY = 24 * 60 * 60 * 1000
if (lastCheck && Date.now() - lastCheck < ONE_DAY) return
// Check if rokit is available
const rokitPath = await which("rokit").catch(() => null)
if (!rokitPath) return
const workspaceFolder = vscode.workspace.workspaceFolders?.[0]?.uri.fsPath
// rokit update --check prints update info if available, or "up to date" if current.
// Check both project-level and global.
let checkOutput: string | null = null
let updateScope: "project" | "global" = "global"
// Try project-level first (if workspace is open)
if (workspaceFolder) {
try {
const result = await exec("rokit update atlas --check", { cwd: workspaceFolder })
const output = (result.stdout + result.stderr).trim()
if (output && !output.toLowerCase().includes("up to date")) {
checkOutput = output
updateScope = "project"
}
} catch {
// atlas not in project rokit.toml -- fall through
}
}
// Try global if no project-level update found
if (!checkOutput) {
try {
const result = await exec("rokit update --global atlas --check")
const output = (result.stdout + result.stderr).trim()
if (output && !output.toLowerCase().includes("up to date")) {
checkOutput = output
updateScope = "global"
}
} catch {
// atlas not installed globally via rokit
}
}
context.globalState.update("atlas::lastUpdateCheck", Date.now())
if (!checkOutput) return
const choice = await vscode.window.showInformationMessage(
`An Atlas update is available. ${checkOutput}`,
"Update",
"Dismiss",
)
if (choice !== "Update") return
await vscode.window.withProgress(
{ location: vscode.ProgressLocation.Notification, title: "Updating Atlas..." },
async (progress) => {
try {
if (updateScope === "project" && workspaceFolder) {
await exec("rokit update atlas", { cwd: workspaceFolder })
} else {
await exec("rokit update --global atlas")
}
} catch (e: any) {
vscode.window.showErrorMessage(`Could not update Atlas: ${e.stderr || e}`)
return
}
progress.report({ message: "Reinstalling Studio plugin..." })
try {
await exec("atlas plugin install")
} catch {
vscode.window.showWarningMessage(
"Atlas updated, but Studio plugin reinstall failed. Run 'atlas plugin install' manually.",
)
}
vscode.window.showInformationMessage("Atlas updated successfully.")
},
)
}
```
Called from `activate()` as fire-and-forget (no `await`) so it never blocks extension startup:
```typescript
// In activate(), after all synchronous setup is done:
checkForAtlasUpdates(context).catch(() => {})
```
Key design decisions:
- **Uses `rokit update --check`**: Lets rokit handle version resolution natively -- no GitHub API calls, no manual semver parsing
- **Project-level first, then global**: Checks the project's `rokit.toml` first (if workspace is open), falls back to global. When updating, uses the same scope that found the update
- **Throttle**: Once per day via `globalState`, so frequent restarts don't spam the user
- **Silent failures**: Every step is wrapped in try/catch that returns early. Missing rokit, no atlas configured, network errors -- all silently ignored
- **Non-blocking**: Called without `await` so extension activation is never delayed
---
### Summary of What's Working Correctly
- Asset name regex (`-(?<platform>\w+)-(?<arch>\w+)\.zip$`) correctly matches all 6 rokit 1.2.0 release assets (linux/macos/windows x x86_64/aarch64, all `.zip`)
- Platform/arch mapping covers all common Node.js platforms (win32, darwin, linux) and architectures (x64, arm64)
- `rokit self-install` correctly sets up `~/.rokit/bin` in user PATH on all platforms
- `chmod 0o755` is correctly applied on Unix after extraction
- `getRojoInstall` correctly detects rokit-managed vs globally installed atlas
## Files to Change
- [vscode-rojo/src/installRojo.ts](vscode-rojo/src/installRojo.ts) -- Bugs 2, 3, 4, 5, 6, 7
- [vscode-rojo/src/commands/openMenu.ts](vscode-rojo/src/commands/openMenu.ts) -- Bug 1, Check for Updates menu item
- [vscode-rojo/src/checkForUpdates.ts](vscode-rojo/src/checkForUpdates.ts) -- NEW: background update check logic
- [vscode-rojo/src/extension.ts](vscode-rojo/src/extension.ts) -- Call checkForAtlasUpdates() on activation

117
.cursor/plans/strip_dedup_suffix_automatically_3d35c847.plan.md Normal file
View file

@ -0,0 +1,117 @@
---
name: Strip dedup suffix automatically
overview: Eliminate unnecessary `.meta.json5` files for dedup'd instances by having forward sync automatically strip `~N` dedup suffixes from filenames when deriving instance names, and having syncback skip writing meta `name` fields for dedup-only cases.
todos:
- id: strip-fn
content: Add `strip_dedup_suffix()` to `dedup_suffix.rs` and ensure it is importable from snapshot code
status: completed
- id: forward-files
content: Modify `file_name_for_path()` in `metadata.rs` to strip `~N` dedup suffixes
status: completed
- id: forward-dirs
content: Modify `get_dir_middleware()` in `snapshot_middleware/mod.rs` to strip `~N` from directory names
status: completed
- id: syncback-meta
content: Change `name_for_inst()` needs_meta condition to `needs_slugify` only (remove dedup check)
status: completed
- id: change-processor
content: Update file rename handler in `change_processor.rs` to use `slugified_new_name` for meta name conditions
status: completed
- id: update-tests
content: Update existing tests and fixtures for new dedup-only behavior (no meta name)
status: completed
- id: new-tests
content: "Add new tests: forward sync stripping, edge cases, syncback dedup-only, roundtrip without meta"
status: completed
isProject: false
---
# Auto-Strip Dedup Suffixes to Eliminate Redundant Meta Files
## Problem
When syncback encounters duplicate instance names (e.g., two instances named "LocalScript"), it creates:
- `LocalScript.local.luau` (first)
- `LocalScript~2.local.luau` (second, with dedup suffix)
- `LocalScript~2.meta.json5` with `name: "LocalScript"` (meta file to recover real name)
The meta file is redundant because `~N` is a well-defined system convention -- the `~` character is forbidden in instance names (always slugified to `_`), so `~N` can only come from the dedup system. Forward sync can strip it automatically.
## Safety Argument
- The `~` character is in `SLUGIFY_CHARS` (`[file_names.rs](src/syncback/file_names.rs)`), so any instance literally named `Foo~2` would be slugified to `Foo_2` during syncback. Natural tilde names cannot produce the `~N` pattern.
- Existing projects with meta files still work: meta `name` field takes priority over filename-derived name (`[meta_file.rs](src/snapshot_middleware/meta_file.rs)` `apply_name`).
- Non-dedup patterns like `~0`, `~abc`, `~` are NOT stripped (only `~N` where N > 0).
## Changes
### 1. Add `strip_dedup_suffix()` utility
Add to `[src/syncback/dedup_suffix.rs](src/syncback/dedup_suffix.rs)` alongside `parse_dedup_suffix`:
```rust
pub fn strip_dedup_suffix(name: &str) -> &str {
parse_dedup_suffix(name).map_or(name, |(base, _)| base)
}
```
### 2. Forward sync: strip dedup from file names
In `[src/snapshot/metadata.rs](src/snapshot/metadata.rs)`, `SyncRule::file_name_for_path()` (line 330) -- after stripping the middleware extension, also strip `~N`:
```rust
pub fn file_name_for_path<'a>(&self, path: &'a Path) -> anyhow::Result<&'a str> {
let name = /* existing logic to strip middleware extension */;
Ok(strip_dedup_suffix(name))
}
```
The return type stays `&'a str` because stripping `~N` returns a sub-slice of the same string.
### 3. Forward sync: strip dedup from directory names
In `[src/snapshot_middleware/mod.rs](src/snapshot_middleware/mod.rs)`, `get_dir_middleware()` (line 116) -- strip `~N` from `dir_name` before returning:
```rust
let dir_name = strip_dedup_suffix(dir_name);
```
Return type stays `&'path str` (sub-slice of the original path).
### 4. Syncback: don't require meta name for dedup-only
In `[src/syncback/file_names.rs](src/syncback/file_names.rs)`, `name_for_inst()` line 79 -- change:
```rust
// Before:
let needs_meta = needs_slugify || deduped_slug != base;
// After:
let needs_meta = needs_slugify;
```
When only dedup happened (no slugification), forward sync will recover the correct name by stripping `~N`.
### 5. Two-way sync: update meta name conditions in change_processor
In `[src/change_processor.rs](src/change_processor.rs)`, the file rename handler (lines 1569, 1581) currently uses `deduped_new_name != *new_name` to decide whether meta name is needed. Change both to `slugified_new_name != *new_name`, matching the directory handler (line 1401) which already does this correctly.
### 6. Module visibility
Ensure `strip_dedup_suffix` is importable from snapshot code. May need to adjust `pub` visibility on the `syncback::dedup_suffix` module, or alternatively place the function in a shared location if cross-module import is problematic.
### 7. Update existing tests and fixtures
- `**src/syncback/file_names.rs` tests:** Update tests like `name_for_inst_dedup_collision` (line 826) and `nfi_clean_name_dedup_still_needs_meta_dir` (line 1175) -- these should now assert `needs_meta = false` for dedup-only cases.
- `**rojo-test/build-tests/dedup_suffix_with_meta/`:** Update fixture to remove now-unnecessary meta files for dedup-only cases.
- `**tests/tests/syncback_roundtrip.rs`:** The `dedup_suffix_with_meta` roundtrip test (line 299) and idempotency test (line 362) may need updated expected output.
### 8. Add new tests
- **Forward sync unit test:** File `Foo~2.luau` without meta -> instance name is `"Foo"`
- **Forward sync unit test:** Directory `Foo~2/` without meta -> instance name is `"Foo"`
- **Forward sync edge cases:** `Foo~0.luau` -> `"Foo~0"`, `Foo~abc.luau` -> `"Foo~abc"` (not valid dedup, kept as-is)
- **Syncback test:** Dedup-only case produces no meta `name` field
- **Syncback test:** Slugify + dedup still produces meta `name` field (e.g., `"Hey/Bro"` -> `Hey_Bro~2.luau` with meta name `"Hey/Bro"`)
- **Roundtrip test:** Dedup'd names without meta files survive build -> syncback -> rebuild

394
.cursor/plans/syncback_via_rbxm_ad0231a7.plan.md Normal file
View file

@ -0,0 +1,394 @@
---
name: Syncback via RBXM
overview: Add a "Syncback" button to the plugin that serializes the DataModel per-service via SerializationService, sends the rbxm chunks to a totally segregated server endpoint, which tears down the entire serve system, runs CLI-equivalent clean-mode syncback, and restarts serve -- all within the same process.
todos:
- id: server-exit-reason
content: Modify LiveServer::start() to return ServerExitReason via tokio select + shutdown channel (src/web/mod.rs)
status: completed
- id: api-types
content: Add SyncbackPayload/ServiceChunk types to src/web/interface.rs
status: completed
- id: api-endpoint
content: Add POST /api/syncback endpoint handler in src/web/api.rs (fully segregated, no session validation)
status: completed
- id: serve-loop
content: Wrap ServeCommand::run() in a loop that handles SyncbackRequested (src/cli/serve.rs)
status: completed
- id: reconstruct-dom
content: "Implement run_live_syncback: reconstruct WeakDom from service chunks, run syncback_loop (clean mode, all CLI defaults), write files"
status: completed
- id: plugin-button
content: Add Syncback button (Danger style, layoutOrder 2) to NotConnected.lua
status: completed
- id: plugin-confirm
content: Add confirmation dialog (StudioPluginGui, danger-themed) in App/init.lua
status: completed
- id: plugin-serialize
content: Implement DataModel serialization per SYNCBACK_SERVICES list using SerializationService
status: completed
- id: plugin-api
content: Add standalone syncback HTTP post function (no ApiContext/session needed)
status: completed
- id: plugin-flow
content: "Wire up full flow in App/init.lua: confirm -> serialize -> POST /api/syncback -> done (fire and forget)"
status: completed
- id: test-helpers
content: Add post_api_syncback, wait_to_come_back_online, make_service_chunk, make_rbxl_from_chunks, run_cli_syncback_on_chunks to serve_util.rs
status: completed
- id: test-fixtures
content: Create live_syncback, live_syncback_place_ids, and live_syncback_complex serve-test fixtures
status: completed
- id: test-parity
content: "CLI parity tests (byte-for-byte): scripts, models+properties, refs, special names, duplicates, deep hierarchy, mixed types, comprehensive properties, multi-service"
status: completed
- id: test-validation
content: "Rejection tests: bad protocol, bad version, blocked place, unlisted place, whitelisted place, unrestricted"
status: completed
- id: test-lifecycle
content: "Lifecycle tests: server comes back functional (WebSocket + file watch), syncback twice with different data, replaces all old files"
status: completed
- id: test-roundtrip
content: "Round-trip identity test: build -> decompose -> live syncback -> rebuild -> compare instance trees"
status: completed
isProject: false
---
# Syncback via RBXM
## Architecture
```mermaid
sequenceDiagram
participant User
participant Plugin
participant Server
participant Filesystem
User->>Plugin: Click "Syncback" button
Plugin->>Plugin: Show danger confirmation dialog
User->>Plugin: Confirm
Plugin->>Plugin: SerializeInstancesAsync per service
Plugin->>Server: POST /api/syncback (msgpack, segregated endpoint)
Server->>Plugin: 200 OK
Note over Server: Store payload, signal shutdown
Note over Server: Full teardown: drop ServeSession, VFS watcher, ChangeProcessor, WebSocket connections
Note over Server: Reconstruct WeakDom from service chunks
Note over Server: Run syncback_loop (clean mode, all CLI defaults)
Server->>Filesystem: Write files
Note over Server: Create fresh ServeSession + LiveServer
Note over Server: Bind to same address:port
Plugin->>Plugin: Poll GET /api/rojo until server is back
Plugin->>Server: Normal connect flow
```
## Server Changes
### 1. Make `LiveServer::start()` return instead of blocking forever
`[src/web/mod.rs](src/web/mod.rs)`: Change `start()` to return a `ServerExitReason` enum. Add a shutdown mechanism via `Arc<SyncbackSignal>` (wrapping a `Mutex<Option<SyncbackPayload>>` + `tokio::sync::Notify`). The accept loop uses `tokio::select!` to watch for the notify signal alongside `listener.accept()`. When signaled, it breaks and returns.
```rust
pub enum ServerExitReason {
SyncbackRequested(SyncbackPayload),
}
pub struct SyncbackSignal {
payload: Mutex<Option<SyncbackPayload>>,
notify: tokio::sync::Notify,
}
```
The `SyncbackSignal` is shared between `LiveServer` (which watches it) and the API handler (which deposits data into it). When the accept loop exits, the Tokio runtime drops, all spawned tasks (including active WebSocket connections) are cancelled, and the `Arc<ServeSession>` drops -- tearing down the ChangeProcessor thread, VFS watcher, and all in-memory state.
### 2. New segregated API endpoint: `POST /api/syncback`
`[src/web/api.rs](src/web/api.rs)`: Totally separate from the connect/sync flow. No session ID needed -- the plugin doesn't need to be "connected" to use this. But it does validate version and place ID.
Request body is msgpack-encoded:
```rust
pub struct SyncbackRequest {
pub protocol_version: u64,
pub server_version: String,
pub place_id: Option<u64>,
pub services: Vec<ServiceChunk>,
}
pub struct ServiceChunk {
pub class_name: String,
pub data: Vec<u8>, // raw rbxm bytes
}
// Payload passed to the serve loop (version info stripped after validation)
pub struct SyncbackPayload {
pub services: Vec<ServiceChunk>,
}
```
Types live in `[src/web/interface.rs](src/web/interface.rs)`.
Handler flow:
1. Deserialize msgpack body into `SyncbackRequest`
2. **Validate protocol version** -- reject with 400 if `request.protocol_version != PROTOCOL_VERSION`
3. **Validate server version** -- reject with 400 if major.minor doesn't match `SERVER_VERSION`
4. **Validate place ID** against the project's `servePlaceIds` (whitelist) and `blockedPlaceIds` (blocklist). Reject with 403 if not allowed.
5. Extract `SyncbackPayload` (just the services), store in `SyncbackSignal` -> notify -> return 200 `{"status": "syncback_initiated"}`
The response is sent before any teardown begins (the API handler is a separate tokio task from the accept loop).
### 3. Wrap `ServeCommand::run()` in a loop
`[src/cli/serve.rs](src/cli/serve.rs)`: The `run()` method becomes a loop. Each iteration creates a fresh VFS, ServeSession, and LiveServer. When `start()` returns `SyncbackRequested`, the old session is fully dropped (VFS watcher stops, ChangeProcessor thread joins, tree is freed), syncback runs, and the loop continues to create everything fresh:
```rust
loop {
let (vfs, errors) = Vfs::new_default_with_errors();
let session = Arc::new(ServeSession::new(vfs, project_path.clone(), Some(errors))?);
let server = LiveServer::new(session);
match server.start(addr) {
ServerExitReason::SyncbackRequested(payload) => {
log::info!("Live syncback requested, running...");
run_live_syncback(&project_path, payload)?;
log::info!("Syncback complete, restarting serve...");
continue;
}
}
}
```
IP and port are resolved once before the loop (they don't change).
### 4. Reconstruct WeakDom + run syncback (CLI-equivalent)
New function `run_live_syncback(project_path: &Path, payload: SyncbackPayload) -> anyhow::Result<()>` in `src/cli/serve.rs`:
1. **Build WeakDom** -- Create `DataModel` root, for each `ServiceChunk` parse rbxm via `rbx_binary::from_reader`, create a service instance of the correct class, deep-clone all children from the parsed dom into the target dom. Maintain a `HashMap<Ref, Ref>` ref-map to fix up Ref properties across chunks.
2. **Run syncback** -- Identical to what CLI `atlas syncback` does:
- Create oneshot VFS: `Vfs::new_oneshot()`
- Load old tree: `ServeSession::new_oneshot(vfs, project_path)`
- Call `syncback_loop(vfs, &mut old_tree, new_dom, project, incremental: false)` -- clean mode
- Write: `fs_snapshot.write_to_vfs_parallel(base_path, vfs)`
- `refresh_git_index(base_path)`
Same code path as CLI syncback, same defaults, same clean mode. The only difference is the WeakDom source.
## Plugin Changes
### 5. Syncback button on NotConnected page
`[plugin/src/App/StatusPages/NotConnected.lua](plugin/src/App/StatusPages/NotConnected.lua)`: Add "Syncback" button with `style = "Danger"` and `layoutOrder = 2`, shift "Sync" to `layoutOrder = 3`:
```
[Settings] [Syncback] [Sync]
```
Fires `self.props.onSyncback`.
### 6. Confirmation dialog
`[plugin/src/App/init.lua](plugin/src/App/init.lua)`: Floating `StudioPluginGui` dialog (same pattern as `AcceptConfirm` in Confirming.lua):
- Title: "Full Syncback"
- Message: "This will overwrite your project files with the current Studio state. This cannot be undone."
- Buttons: "Cancel" (Bordered) | "Syncback" (Danger)
- State: `self.state.showingSyncbackConfirm`
### 7. DataModel serialization
After confirmation, serialize per-service. Hardcode the same `VISIBLE_SERVICES` list from the Rust side:
```lua
local SYNCBACK_SERVICES = {
"Lighting", "MaterialService", "ReplicatedFirst", "ReplicatedStorage",
"ServerScriptService", "ServerStorage", "SoundService", "StarterGui",
"StarterPack", "StarterPlayer", "Teams", "TextChatService",
"VoiceChatService", "Workspace",
}
local services = {}
for _, className in SYNCBACK_SERVICES do
local service = game:FindService(className)
if service and #service:GetChildren() > 0 then
local buf = SerializationService:SerializeInstancesAsync(service:GetChildren())
table.insert(services, {
className = className,
data = buf, -- msgpack-luau natively encodes buffer as bin format (zero-copy)
})
end
end
```
The msgpack-luau library handles `buffer` types natively -- when `type(data) == "buffer"`, it encodes as msgpack bin 8/16/32 using `buffer.copy` directly. No string conversion needed. On the Rust side, `rmp_serde` deserializes msgpack bin into `Vec<u8>` automatically.
### 8. Direct HTTP post (no ApiContext needed)
Since this endpoint is fully segregated and doesn't need a session, the plugin just needs host:port from the address entry fields. A standalone function posts directly:
```lua
local function postSyncback(host, port, services)
local url = ("http://%s:%s/api/syncback"):format(host, port)
local body = Http.msgpackEncode({
protocolVersion = Config.protocolVersion,
serverVersion = Config.expectedServerVersionString,
placeId = game.PlaceId,
services = services,
})
return Http.post(url, body):andThen(rejectFailedRequests)
end
```
No `ApiContext` instance needed. This keeps it completely independent from the connect flow.
### 9. Full flow in App
`[plugin/src/App/init.lua](plugin/src/App/init.lua)`: The `onSyncback` handler:
1. Show confirmation dialog
2. On confirm: serialize DataModel per-service
3. POST to `/api/syncback` with the chunks
4. On 200: show a notification ("Syncback data sent. Server is processing.")
5. Done. Stay on NotConnected page. User clicks "Sync" manually whenever they're ready.
Fire and forget -- the plugin's job ends after the POST succeeds. No polling, no auto-reconnect, no special state.
## Key Design Decisions
- **Fully segregated endpoint**: `/api/syncback` has zero coupling to the session/connect system. No session ID, no auth. Plugin fires and forgets.
- **Complete teardown**: When the signal fires, the entire serve system is destroyed -- Tokio runtime drops, ServeSession drops (joining the ChangeProcessor thread, stopping VFS watcher), all WebSocket connections are cancelled. Nothing survives from the previous serve.
- **CLI-equivalent syncback**: Uses `syncback_loop` with `incremental: false` (clean mode), non-interactive. Identical behavior to `atlas syncback --input file.rbxl` with all defaults.
- **Same process, same port**: The `atlas serve` process never exits. After syncback completes, a fresh serve starts on the same address:port. The user sees log output in their terminal throughout.
- **Danger styling**: Uses existing `Danger` theme (`#E74C3C` red) defined in `[plugin/src/App/Theme.lua](plugin/src/App/Theme.lua)`.
- **Per-service serialization**: 14 visible services from `VISIBLE_SERVICES` constant, matching both Rust and Lua sides. Cross-service Ref properties are lost in transit but syncback's `Rojo_Ref_`* system resolves refs from filesystem structure.
## Tests
The core invariant: **live syncback must produce byte-identical filesystem output to CLI syncback given the same input data.** If it doesn't, something is broken in the serialize -> transmit -> reconstruct -> syncback pipeline. Every test that verifies data accuracy runs BOTH live syncback and CLI syncback on the same input, then compares output with `assert_dirs_equal` (byte-for-byte file comparison from `[tests/rojo_test/roundtrip_util.rs](tests/rojo_test/roundtrip_util.rs)`).
Tests go in a dedicated `tests/tests/live_syncback.rs`. They use `run_serve_test` from `[tests/rojo_test/serve_util.rs](tests/rojo_test/serve_util.rs)` plus the round-trip utilities.
### Test helpers (add to `serve_util.rs`)
`**post_api_syncback_raw`** -- Sends a raw msgpack body to `POST /api/syncback`. Returns the full `reqwest::blocking::Response` so tests can assert on status codes and error bodies.
`**post_api_syncback`** -- Convenience wrapper that fills in correct protocol version + server version. Takes `place_id: Option<u64>` and `services: Vec<ServiceChunk>`. Panics if response is not 200.
`**wait_to_come_back_online`** -- Polls `GET /api/rojo` with exponential backoff (longer timeout than initial startup since syncback runs between teardown and restart). Returns the new `ServerInfoResponse`. Panics if server doesn't come back.
`**make_service_chunk`** -- Builds a `ServiceChunk` from `InstanceBuilder` children:
```rust
fn make_service_chunk(class_name: &str, children: Vec<InstanceBuilder>) -> ServiceChunk {
let mut dom = WeakDom::new(InstanceBuilder::new("DataModel"));
let root = dom.root_ref();
for child in children {
dom.insert(root, child);
}
let refs: Vec<Ref> = dom.root().children().to_vec();
let mut buf = Vec::new();
rbx_binary::to_writer(&mut buf, &dom, &refs).unwrap();
ServiceChunk { class_name: class_name.to_string(), data: buf }
}
```
`**make_rbxl_from_chunks**` -- Builds a complete rbxl file from the same service chunks, for CLI parity testing. Creates a WeakDom with DataModel root + properly-classed service instances + children, serializes to rbxl. This gives CLI syncback the exact same input data as live syncback received.
`**run_cli_syncback_on_chunks**` -- End-to-end helper: copies a fixture to a fresh temp dir, calls `make_rbxl_from_chunks`, runs `run_rojo_syncback_clean`, returns the temp dir path. Used by every parity test as the CLI reference.
### Fixtures
`**rojo-test/serve-tests/live_syncback/**` -- Multi-service project:
```
default.project.json5 # RS -> src/shared, SSS -> src/server, Workspace -> src/workspace
src/shared/OldModule.luau
src/server/OldScript.server.luau
src/workspace/OldPart.model.json5
```
`**rojo-test/serve-tests/live_syncback_place_ids/**` -- Same layout + `servePlaceIds: [123]`, `blockedPlaceIds: [999]`.
`**rojo-test/serve-tests/live_syncback_complex/**` -- All 14 visible services mapped to separate `$path` dirs. Used for comprehensive parity tests.
### CLI parity tests (data accuracy)
Every parity test follows this pattern:
```rust
fn assert_live_matches_cli(
fixture_name: &str,
chunks: Vec<ServiceChunk>,
place_id: Option<u64>,
) {
run_serve_test(fixture_name, |session, _| {
// 1. Live syncback
session.post_api_syncback(place_id, chunks.clone());
session.wait_to_come_back_online();
// 2. CLI syncback on fresh copy of same fixture with same data
let cli_dir = run_cli_syncback_on_chunks(fixture_name, &chunks);
// 3. Byte-for-byte comparison of every file
assert_dirs_equal(session.path(), &cli_dir);
});
}
```
`**parity_scripts**` -- ModuleScript (`return {}`), Script/Server (`print("server")`), Script/Client (`print("client")`), LocalScript. Verifies Source content preserved exactly, correct file extensions (`.luau`, `.server.luau`, `.client.luau`, `.local.luau`), RunContext encoded in extension.
`**parity_models_with_properties**` -- Model containing Parts with: Anchored (Bool), Size (Vector3), Position (Vector3), Color (Color3), BrickColor, Shape (Enum), Transparency (Float64). Verifies every property value round-trips through rbxm -> reconstruct -> syncback -> meta/model files identically.
`**parity_ref_properties**` -- Model with PrimaryPart -> child Part. ObjectValue with Value -> sibling Part. Verifies `Rojo_Ref_*` attribute paths in meta/model files match CLI output exactly.
`**parity_special_names**` -- Instances named `"What?"`, `"Key:Value"`, `"A/B"`, `"<Angle>"`, `'He said "hi"'`. Verifies slugified filenames (`What_`, `Key_Value`, `A_B`, etc.) and meta `name` fields match CLI.
`**parity_duplicate_names**` -- 3x Folder named `"Data"`, 2x Script named `"Handler"`. Verifies dedup suffixes (`Data/`, `Data~2/`, `Data~3/`, `Handler.server.luau`, `Handler~2.server.luau`) and meta files match CLI.
`**parity_deep_hierarchy**` -- 6 levels: Folder > Folder > Folder > Folder > Model > Part. Verifies nested directory structure is identical.
`**parity_mixed_file_types**` -- StringValue (-> `.txt`), LocalizationTable (-> `.csv`), JSON data module (-> `.json5`), binary model (-> `.rbxm`), Folder (-> directory). All under one service. Byte-for-byte match with CLI.
`**parity_comprehensive_properties**` -- Single Part with every testable property type set to non-default values: Bool, Int32, Float32, Float64, String, Vector2, Vector3, Vector2int16, Vector3int16, UDim, UDim2, Color3, Color3uint8, BrickColor, NumberRange, Rect, Enum, Font, Tags, Attributes (with nested types). Verifies meta.json5 property serialization matches CLI exactly.
`**parity_multi_service**` -- Data under all 14 visible services simultaneously. Each service gets a unique set of instances (scripts, models, folders). Verifies every service's `$path` directory matches CLI output.
`**parity_empty_services_mixed**` -- Some services have children, some don't, some are absent from the payload entirely. Verifies populated services are correct and empty/absent services result in clean directories.
### Validation / rejection tests
`**rejects_bad_protocol**` -- Send `protocol_version: 999`. Expect non-200. GET /api/rojo still returns same session ID (server unaffected).
`**rejects_bad_version**` -- Send `server_version: "0.0.0"`. Expect non-200. Server unaffected.
`**rejects_blocked_place**` -- Fixture has `blockedPlaceIds: [999]`. Send `placeId: 999`. Expect 403. Server unaffected.
`**rejects_unlisted_place**` -- Fixture has `servePlaceIds: [123]`. Send `placeId: 456`. Expect 403. Server unaffected.
`**allows_whitelisted_place**` -- `servePlaceIds: [123]`, send `placeId: 123`. Expect 200. Syncback runs, server restarts.
`**allows_any_place_when_unrestricted**` -- No servePlaceIds/blockedPlaceIds. Send `placeId: 99999`. Expect 200.
### Lifecycle / edge case tests
`**server_comes_back_functional**` -- After live syncback, verify the restarted server is fully operational: GET /api/rojo works, can read instances via /api/read, WebSocket connects and receives patches when a file is modified on disk.
`**syncback_twice_different_data**` -- Run live syncback with data A. Wait for restart. Run live syncback AGAIN with data B. Wait for restart. Verify filesystem matches data B (not A). Verify server is functional. Tests the serve loop handles multiple iterations.
`**syncback_replaces_all_old_files**` -- Pre-populate the fixture with files that should NOT survive clean mode (orphan scripts, old models). After live syncback, verify none of the old files remain. Only files from the new data exist.
### Round-trip identity test
The ultimate accuracy test:
`**roundtrip_build_syncback_rebuild**` -- Using the `live_syncback_complex` fixture:
1. `atlas build` -> rbxl_A
2. Decompose rbxl_A into per-service chunks (parse with `rbx_binary::from_reader`, extract children of each service, re-serialize as rbxm chunks)
3. POST chunks to live syncback, wait for restart
4. `atlas build` -> rbxl_B
5. Parse both rbxl_A and rbxl_B into WeakDoms
6. Walk both trees recursively: compare instance names, classes, property values (using the same fuzzy equality from `variant_eq` for floats/CFrames), and child counts
7. Any divergence = test failure with detailed diff
This tests the full pipeline: build -> decompose into service chunks -> transmit -> reconstruct -> syncback -> build again. If the output matches, the live syncback system preserves 100% data fidelity.

110
.cursor/plans/update_audit_and_rules_0625423a.plan.md Normal file
View file

@ -0,0 +1,110 @@
---
name: Update audit and rules
overview: Update `.cursor/commands/audit.md` to cover the live syncback pipeline, new files (encodeHelpers, encodeService, new test suites), and refactored encoding architecture. Fix discrepancies in rule MDC files where they don't reflect recent code changes.
todos:
- id: audit-live-syncback
content: Add Live Syncback Pipeline section to audit.md with full file tree covering plugin encoding, transport, server handling, and serve loop restart
status: completed
- id: audit-encode-helpers
content: Add encodeHelpers.lua and encodeService.lua entries to the Plugin->Server ChangeBatcher section in audit.md
status: completed
- id: audit-new-tests
content: Add live_syncback.rs, git_sync_defaults.rs, matching_fixtures.rs to the Shared Infrastructure test section in audit.md
status: completed
- id: audit-11n-live-syncback
content: Add section 11n (Live Syncback Parity) audit checklist to audit.md
status: completed
- id: fix-atlas-api-mdc
content: Add encodeHelpers.lua to atlas-api.mdc ChangeBatcher table and update encodeInstance.lua description
status: completed
- id: fix-atlas-plugin-mdc
content: Add encodeHelpers.lua to atlas-plugin.mdc ChangeBatcher listing and fix SyncbackConfirm placement
status: completed
- id: fix-atlas-dedup-mdc
content: Add encodeHelpers.lua to atlas-dedup.mdc Key Files Map table
status: completed
isProject: false
---
# Update Audit Command and Rule Files
## Summary of Discrepancies Found
After comparing all 8 rule MDC files and the audit command against the actual source code and recent git history (40+ commits including live syncback, matching session caching, encodeHelpers refactor, float precision, git sync defaults), here are the issues:
### audit.md - Major Gaps
1. **No live syncback pipeline** -- The audit command describes only two pipelines (CLI syncback, two-way sync). The live syncback pipeline (`POST /api/syncback`) is a third distinct flow involving `encodeService.lua`, `SyncbackConfirm` UI, `handle_api_syncback`, `build_dom_from_chunks`, and `run_live_syncback`. Not covered at all.
2. **Missing `encodeHelpers.lua`** -- New shared helper module (`plugin/src/ChangeBatcher/encodeHelpers.lua`) that consolidates `encodeAttributes`, `encodeTags`, and `forEachEncodableProperty`. Both `encodeInstance.lua` and `encodeService.lua` import from it. Not mentioned anywhere in audit.md.
3. **Missing `encodeService.lua`** -- New file (`plugin/src/ChangeBatcher/encodeService.lua`) for service-level property encoding during live syncback. Not mentioned.
4. **Missing 3 new test suites** -- `tests/tests/live_syncback.rs`, `tests/tests/git_sync_defaults.rs`, `tests/tests/matching_fixtures.rs` are not referenced in the audit's test infrastructure section.
5. **Missing `SyncbackConfirm` UI** -- The floating confirmation dialog for live syncback (rendered in `App/init.lua`) is not mentioned in the plugin UI or data flow sections.
6. **Missing live syncback audit checklist** -- No equivalent of sections 11a-11m for live syncback-specific concerns (service property encoding, ObjectValue carrier lifecycle, rbxm blob reconstruction, clean-mode-only semantics, server restart behavior).
### atlas-api.mdc - Issues
1. **Missing `encodeHelpers.lua`** in the ChangeBatcher file table (section "ChangeBatcher"). The table lists `init.lua`, `createPatchSet.lua`, `encodePatchUpdate.lua`, `encodeInstance.lua`, `encodeService.lua`, `encodeProperty.lua`, `propertyFilter.lua` -- but `encodeHelpers.lua` is missing.
2. `**encodeInstance.lua` description** says "includes duplicate detection helpers" but doesn't note that property iteration was refactored to use `encodeHelpers.forEachEncodableProperty()`.
### atlas-plugin.mdc - Issues
1. **Missing `encodeHelpers.lua`** from the file listing under "4. ChangeBatcher" section. The `encodeHelpers.lua` file exports `encodeAttributes`, `encodeTags`, `forEachEncodableProperty`.
2. `**SyncbackConfirm**` is listed under StatusPages (`SyncbackConfirm` - "Floating dialog for live syncback confirmation (rendered by `App/init.lua`)") which is slightly misleading. It's NOT a StatusPage file -- it's rendered inline in `App/init.lua` as a `StudioPluginGui` with `isEphemeral = true`. The parenthetical note is correct but the placement under StatusPages implies a separate file.
### atlas-dedup.mdc - Issues
1. **Missing `encodeHelpers.lua`** from the Key Files Map table. This file is relevant because it contains the shared property filtering logic (`forEachEncodableProperty`) used during instance encoding for both two-way sync additions and live syncback.
## Changes
### Changes to `audit.md`
**Section "Sync System Layout" -- add Live Syncback Pipeline** after the Two-Way Sync section:
Add a third pipeline section covering:
- Plugin-side: `SYNCBACK_SERVICES` in `App/init.lua`, `encodeService.lua`, `performSyncback()`, `SyncbackConfirm` dialog
- Transport: `POST /api/syncback` (MessagePack)
- Server-side: `handle_api_syncback` in `api.rs`, `SyncbackSignal`/`LiveServer` in `mod.rs`, `build_dom_from_chunks`/`run_live_syncback` in `serve.rs`
- Wire types: `SyncbackRequest`, `ServiceChunk`, `SyncbackPayload` in `interface.rs`
**Section "Plugin --> Server" -- add `encodeHelpers.lua`** to the Change Batching subsection:
- Add entry for `plugin/src/ChangeBatcher/encodeHelpers.lua` describing `encodeAttributes`, `encodeTags`, `forEachEncodableProperty`
- Add entry for `plugin/src/ChangeBatcher/encodeService.lua` describing service-level encoding for live syncback
**Section "Shared Infrastructure" -- add new test files:**
- `tests/tests/live_syncback.rs` -- Live syncback parity, validation, lifecycle, edge cases
- `tests/tests/git_sync_defaults.rs` -- Git-based sync default selection tests
- `tests/tests/matching_fixtures.rs` -- Matching algorithm fixture tests
**Add new audit section 11n: Live Syncback Parity** covering:
- Service property encoding completeness (`encodeService.lua` vs `filter_properties`)
- ObjectValue carrier lifecycle (created, serialized, destroyed)
- rbxm blob reconstruction (`build_dom_from_chunks` child distribution)
- Clean mode semantics (always `incremental=false`)
- Server restart behavior (session invalidation, reconnection)
- `SYNCBACK_SERVICES` list consistency with `VISIBLE_SERVICES` in Rust
- Protocol/version/placeId validation
- Round-trip: live syncback output must match CLI syncback output
**Update Audit Reading Order** (step 9 area) to include live syncback flow.
### Changes to Rule MDC Files
`**atlas-api.mdc`:**
- Add `encodeHelpers.lua` row to ChangeBatcher file table with description: "Shared encoding helpers: `encodeAttributes`, `encodeTags`, `forEachEncodableProperty`. Used by `encodeInstance.lua` and `encodeService.lua`."
- Update `encodeInstance.lua` row to note it uses `encodeHelpers.forEachEncodableProperty()` for property iteration
`**atlas-plugin.mdc`:**
- Add `encodeHelpers.lua` to the "4. ChangeBatcher" file listing with description: "Shared property encoding helpers (`encodeAttributes`, `encodeTags`, `forEachEncodableProperty`)"
- Move `SyncbackConfirm` from the StatusPages list to its own note under the App section, clarifying it's rendered inline in `App/init.lua` as a floating `StudioPluginGui`, not a separate StatusPage file
`**atlas-dedup.mdc`:**
- Add `plugin/src/ChangeBatcher/encodeHelpers.lua` to the Key Files Map table with subsystem "Encoding" and purpose "Shared property filtering/encoding helpers used by `encodeInstance.lua` and `encodeService.lua`"

350
.cursor/prompts/Ambiguous Path Handling.txt Normal file
View file

@ -0,0 +1,350 @@
Ambiguous Path Handling
We're looking to finally add support for Ambiguous Paths.
Naming
* Duplicate file names are saved as ~1, ~2 etc - an expansion of the slugify system.
- ~1, ~2 suffixes: the existing deduplicate_name() in file_names.rs already does this. But currently it's only used for slug collisions (e.g., What?Script and What_Script both slugify to What_Script). I'm proposing to extend it to true duplicate names. The meta name field then stores the real name.
- Note: `~` is in SLUGIFY_CHARS, so instance names naturally containing `~` get it replaced with `_` BEFORE dedup runs. "Foo~1" (natural name) slugifies to "Foo_1", which is always distinct from the dedup suffix "Foo~1". This invariant is already tested in file_names.rs.
* Name field will be populated in the meta/model definitions
* IMPORTANT: Matching groups and dedup groups are DIFFERENT concepts with different grouping criteria:
- Matching groups (for pairing instances during sync): built by true Instance.Name. Two siblings with the same Name are in the same matching group regardless of ClassName. Used by the matching algorithm in all three locations.
- Dedup groups (for filesystem ~N suffixes): built by the FILESYSTEM SLUG that the middleware would produce. Only apply ~N suffixes when the filesystem representation would actually collide. A Folder "Foo" (becomes Foo/) and a Script "Foo" (becomes Foo.server.luau) do NOT collide on disk and do NOT need dedup suffixes. Two Folders both named "Foo" DO collide (both become Foo/) and need dedup. The dedup key is the lowercased final filesystem path component (slug + extension or directory name) that the middleware assigns.
- Init-style collisions: when a directory-style instance (Folder) and an init-style instance (Script with children) both want the same directory name, they DO collide -- apply dedup suffixes to the directories (Foo/ and Foo~1/). The middleware must compute whether each instance would be a directory or file BEFORE dedup grouping runs.
- In short: compute what each instance's filesystem name WOULD be (via middleware rules), group by lowercased collision on that name, then apply ~N suffixes within each collision group.
* Suffix cleanup is GAP-TOLERANT: when a dedup'd sibling is deleted, do NOT renumber remaining suffixes to fill gaps. If you have Attachment, Attachment~1, Attachment~2 and delete ~1, keep Attachment and Attachment~2 as-is. Gaps are harmless and avoiding renumbering prevents cascading file renames + mass ref path updates.
- The ONE exception: when the dedup group shrinks to exactly 1 remaining instance, remove its suffix entirely (rename to clean name). This keeps the filesystem clean for the common case. The ref path update system must handle this single rename.
- If the base-name holder (no suffix) is deleted and multiple remain, the lowest-numbered suffix (~1, or ~2 if ~1 is already gone) is promoted to base name. This is the only rename that occurs on deletion besides the group→1 cleanup.
* Base-name assignment: the matching algorithm should preserve existing filesystem assignments. Whichever instance currently holds the base name (no suffix) on disk keeps it when updating. New instances get the next available ~N suffix. In clean syncback (no existing files), Roblox child order determines base-name assignment (first child in the DOM children list gets the base name).
* Tiebreaker ordering: when the matching algorithm has tied candidates (identical similarity scores), Roblox child order (Instance:GetChildren() index on plugin side, DOM child order on Rust side) is the canonical tiebreaker for deterministic results.
* VFS event handling: when change_processor renames files for dedup suffix cleanup (e.g., Attachment~1 → Attachment after base-name holder deleted), tag these as self-initiated renames. compute_patch_set should recognize and ignore self-initiated VFS events to prevent feedback loops.
Refs
* PREREQUISITE: Ref resolution must happen BEFORE the matching algorithm runs. Ref resolution is cheap on the Rust side (pointer lookups). On the Lua plugin side, reading Ref properties from Studio instances requires pcall (some instances have restricted access) -- still fast but add error handling. Missing or invalid Refs should log a warning and default to nil -- don't error, don't block matching.
* FILE FORMAT: Keep the Rojo_Ref_* attribute prefix with slug paths. Ref paths are stored as attributes: `Rojo_Ref_PrimaryPart: "Workspace/Model.model.json5/Attachment~1"`. Each path segment is the FULL FILESYSTEM NAME of the instance (slug + dedup suffix + extension for files, directory name + dedup suffix for directories). Examples:
- Folder "Foo" (directory) → segment `"Foo"`
- ModuleScript "Foo" → segment `"Foo.luau"`
- Script "Foo" (server) → segment `"Foo.server.luau"`
- Script "Foo" with children (init-style directory) → segment `"Foo"`
- Dedup'd ModuleScript "Foo" → segment `"Foo~1.luau"`
This guarantees ref path segments are always unique among siblings (because filesystem names are unique). The tradeoff: ref paths are coupled to middleware/extension choices, so script type transitions (ModuleScript→Script changes .luau→.server.luau) and file-to-folder transitions ALSO require ref path updates for all refs passing through that instance. This is additional maintenance but eliminates the ambiguity that extension-free paths would create for same-name siblings with different classes.
The path starts from the DataModel's children (DataModel excluded, so `game.Workspace.Model.Part` becomes `"Workspace/Model.model.json5/Part"`). This keeps the existing prefix-based detection (scan for keys starting with `Rojo_Ref_`) which is simple and doesn't require reflection data.
* NOTE: The Rojo_Ref_* system did NOT exist on origin/master -- it was added on the rbxm-ambiguous branch. So this is effectively a new addition, not a modification. The original Rojo $id / Rojo_Target_* / Rojo_Id system stays for backward compatibility with existing projects. When the slug-path system is available, prefer it. Fall back to the $id system for instances without filesystem backing (see below).
* The ref path is always unambiguous because each segment is the full filesystem name, which is unique among siblings by definition (filesystem can't have two files/dirs with the same name). Two Folders named "Attachment" become "Attachment" and "Attachment~1" on disk, producing distinct segments. A Folder "Foo" and Script "Foo" produce segments "Foo" and "Foo.server.luau" -- distinct without dedup.
* Ref path resolution:
- Syncback (ref_properties.rs): When writing Rojo_Ref_* attributes, build the ref path from the full filesystem names assigned during syncback (the same names used for files/directories on disk, INCLUDING extensions).
- Forward sync (patch_apply.rs): When resolving Rojo_Ref_* paths, walk the tree matching each segment against the full filesystem name of each child (derived from instigating_source filename/dirname). get_instance_by_path() in tree.rs currently resolves using instance names -- it needs a filesystem-name resolver instead. For instances without instigating_source (e.g., inside user .rbxm files, newly added during session), fall back to computing what the filesystem name WOULD be (slugify name + middleware extension).
- Forward sync Ref pre-resolution for matching (patch_compute.rs): When pre-resolving Refs before the matching algorithm, try the old RojoTree FIRST (most refs target existing instances). If the path can't be resolved in the old tree (target is a new instance), fall back to walking the snapshot tree. The snapshot is only a subtree (the re-snapshotted directory), so this fallback only works for refs within the same subtree. Refs to new instances outside the current snapshot remain unresolved and are skipped during matching (treated as nil).
- Two-way sync (change_processor.rs, api.rs): When the plugin sends a Ref property change, the server receives a UID. The server must compute the slug path for the target to write it to disk. For file-backed instances, derive the slug from instigating_source. For instances without filesystem backing (inside .rbxm files, newly added instances before VFS catches up), fall back to the $id / Rojo_Target_* system instead of slug paths.
* Cleanup: Remove the escape_ref_path_segment() / split_ref_path() infrastructure in rojo_ref.rs. With filesystem-name-based paths, `/` in instance names is already slugified to `_` by slugify_name(). The `\/` escaping system was needed because the old system used raw instance names -- filesystem-name paths don't need it.
* IMPORTANT: Any time an instance is moved, renamed, or has its dedup suffix reassigned, ALL Rojo_Ref_* paths across ALL meta/model files that reference that instance (or any of its descendants) must be updated. This includes:
- Renames: instance name changes -> slug changes -> all refs to it and its children update
- Dedup reassignment: matching algorithm reassigns ~1 to ~2, or suffix removed entirely -> all refs update
- Reparenting: instance moves to a different parent -> entire ancestor path changes -> all refs update
- Suffix cleanup: dedup suffix removed because sibling was deleted -> all refs update
- Script type transitions: ClassName change (e.g., ModuleScript→Script) changes extension (.luau→.server.luau), which changes the ref segment -> all refs through this instance update
- File-to-folder transitions: instance gains/loses children, flipping between file (Foo.server.luau) and directory (Foo/) representation -> ref segment changes -> all refs update
- The existing RefPathIndex infrastructure tracks which files contain refs to which targets. Extend it to handle all cases above.
- Batch ref path writes: when dedup reassignment causes mass ref path updates (many siblings renamed, each referenced by multiple files), collect ALL affected ref updates first, then write all affected files in one batch. This avoids redundant I/O when multiple refs in the same file need updating.
Matching System
* We need to create a Rust and Lua matching algorithm that produces the same pairing given equivalent instance data. The data shapes differ (Rust: UstrMap<Variant>, Lua: Studio property access via pcall) but the algorithm logic and output must be equivalent.
* The matching algorithm attempts to assign a list of child instances to the current corresponding state, whether that is the actual Instances in the tree in Roblox Studio or the files on the file system (the dom).
* It tries it's BEST to find what will effectively requires the fewest number of changes of each child, when duplicates are encountered. Things which are 1:1 can be considered an instant match.
* You need to consider literally everything about the instance. Name, ClassName, Properties, Attributes, Tags, ITS children, etc. To avoid combinatorial explosion, use a bottom-up subtree hash computed once and cached. The hash infrastructure already exists in syncback/hash/.
- Rust side: blake3 (existing infrastructure). Full subtree hash of name+class+sorted property hashes+children hashes.
- Lua plugin side: use XXH32 (impl attached at bottom of this file) for the STRUCTURAL subtree fingerprint only (child count + sorted child name+class pairs). Do NOT hash full property values (especially Source) -- the serialize-to-buffer cost dominates. For actual property comparison in Pass 3 similarity scoring, compare properties directly with early-exit ordering: ClassName → Tags → Attributes → scalar properties (numbers, bools, enums) → Source last (most expensive, least likely needed after cheaper signals differentiate).
- Hashes are a fast-path signal, NOT a definitive equality proof. If hashes differ → definitely different. If hashes match → very likely identical, score as max similarity. The matching algorithm must not rely solely on hash equality for correctness -- other discriminators (Refs, ClassName, Tags) always participate. This makes 32-bit collision risk irrelevant in practice (ambiguous groups are small, typically 2-10 instances).
- Lua-side caching: compute structural fingerprints LAZILY -- only for instances that actually enter Pass 3 (ambiguous groups not resolved by Refs). Cache fingerprints for the duration of the matching call (not across calls). On the Rust side, hash_tree() precomputes all hashes bottom-up in one pass before matching begins -- this is cheap with blake3.
* Ambiguous groups are built by the TRUE Instance.Name (as finally resolved for each instance). Two siblings with the same true Name are in the same ambiguous group regardless of ClassName. On the plugin/Rust-tree side, Instance.Name / instance.name() is readily available. On the filesystem side, resolving the true Name is non-trivial: it may come from the `name` field in .meta.json5 (if present, overrides filename), the `name` field in .model.json5, or the filename stem after stripping extensions and script suffixes (if no meta override). The matching algorithm must resolve all true Names before it can build groups.
* The matching algorithm should work in passes (Refs must be pre-resolved before any pass runs -- see Refs section above):
- Pass 1: Match all instances that have a unique true Name among their siblings (no ambiguity, O(n)). Within a same-Name group, ClassName is a fast first discriminator for narrowing candidates.
- Pass 2: For remaining unmatched instances (same-Name groups), use the pre-resolved Refs as discriminators. Refs are the strongest signal because they carry structural identity -- if instance A has a Ref property pointing to UID-7, and UID-7 was already matched in Pass 1, then we know instance A's Ref "means" something specific. If only one candidate on the other side also has a Ref resolving to that same matched target, that's a near-certain match. On the Rust side, Refs are already resolved to specific Refs in the tree. On the Lua side, Refs sent from the server are literal UIDs that map to specific Studio instances via the instanceMap. Also consider incoming Refs TO these instances (other instances pointing at them) -- if instance X is the only candidate that other already-matched instances reference, that's equally strong signal.
- Unmatched Ref targets: if a Ref target is itself in an unmatched ambiguous group (not resolved in Pass 1), treat it as a weaker signal -- compare target names/classes without requiring a confirmed match. Do NOT recursively match the target's group (adds complexity for an extremely rare edge case). If the weaker signal is insufficient, the instance falls through to Pass 3 naturally.
- Pass 3: For still-unmatched same-Name groups, use ClassName as a fast discriminator to narrow candidates, then compute pairwise similarity scores (using precomputed subtree hashes on Rust side, direct comparison on Lua side) and greedily assign best-match-first (pick highest score, remove both from pool, repeat).
* Instances which were extra from the incoming changes are considered NEW and then extra from the existing are considered DELETED
* The server assigns every instance a unique Ref (UID) in the RojoTree. When the plugin connects, the matching algorithm pairs server UIDs to Studio instances and caches this mapping in the instanceMap. Once matched, that pairing is stable for the duration of the sync session -- the matching algorithm should not reshuffle existing matched pairs unnecessarily. This cache allows subsequent operations (property changes, renames) to be fast lookups rather than re-running the full matching algorithm.
* We don't need a "UUID" system saved as attributes on instances because of this matching system, even though that would allow this algorithm to work as-is but faster, it's not desired as it adds a lot of bloat.
* Performance of this is really important. Precompute hashes and resolve Refs once before entering the matching loop. Avoid recomputing anything per-pair in Pass 3 -- all signals should be cached before pairwise comparison begins.
* FAST PATH: if Pass 1 matches ALL children (no ambiguous groups exist among siblings), skip Passes 2 and 3 entirely. This is the common case -- most parents have uniquely-named children. The fast path makes forward sync (called on every VFS event) effectively free for non-ambiguous subtrees, degrading to the same cost as the current name+class matching.
* Matching priority within an ambiguous group (same true Name): Refs (pre-resolved -- strongest signal), ClassName, Tags, Attributes, Properties (non-Ref), Children hash.
* The matching needs to handle partial matches -- instance exists in both but has property changes. In this case, just pick the best of each. If candidates are completely equal (identical similarity scores), use Roblox child order as the deterministic tiebreaker (first in DOM children list / Instance:GetChildren() index wins).
* Matching should be run in 3 locations. These are 3 SEPARATE IMPLEMENTATIONS of the same algorithm pattern (2 Rust, 1 Lua), NOT shared code. The data shapes are too different to share implementation. Equivalence is guaranteed by shared test vectors (see Testing section), not by code reuse.
- hydrate.lua -- plugin connects, matches server virtual instances (encoded properties, UIDs for Refs) to Studio instances (read via pcall, direct Instance Ref pointers). Lua implementation.
- compute_patch_set in patch_compute.rs -- VFS event triggers re-snapshot, matches snapshot children (InstanceSnapshot with Rojo_Ref_* as string paths) to tree children (RojoTree instances with Refs as UIDs). Rust implementation #1.
- syncback_dir_no_meta in dir.rs -- syncback matches new instance children (WeakDom from Roblox file) to existing filesystem children (WeakDom from old tree). Rust implementation #2.
* IDEA: We can build a cache progressively by assigning children to specific UIDs. This can help on running time of the algorithm. We need to ensure things don't become stale in this LUT cache. The cache can be built up during actual application. The cache invalidates per-parent when a child is added, removed, or renamed under that parent. During two-way sync, each /api/write mutation should invalidate the affected parent's cache entry. The cache maps parent_ref -> (children_list, match_assignments) and is cheap to rebuild for a single parent.
Syncback:
* ARCHITECTURAL CHANGE: the current syncback flow processes children one-at-a-time via with_joined_path(). Collision-based dedup requires knowing ALL children's filesystem names before assigning suffixes. This requires restructuring to TWO PASSES:
1. First pass: iterate all children, select middleware for each (get_best_middleware), compute tentative filesystem name (slug + extension or directory name).
2. Second pass: group by lowercased collision on tentative name, assign ~N suffixes within each collision group, then actually process each child with its final dedup'd name.
The current with_joined_path() does middleware selection + filename generation + dedup in one call per child. This must be split so dedup grouping has the full picture before any suffix is assigned.
* Needs to be able to output the duplicate files with ~N dedup suffixes
* The matching algorithm runs FIRST to pair incoming instances to existing filesystem children. Then files are written/updated based on the pairing:
- Matched pairs: update the existing file/directory in place (overwrite content, update meta). The matching algorithm should preserve existing base-name assignments -- whoever held the base name on disk keeps it.
- Unmatched incoming instances: create new files (with dedup suffixes if Name collides)
- Unmatched existing files: delete them (instance was removed)
* When a dedup suffix changes (base-name holder deleted → lowest suffix promoted, or group shrinks to 1 → suffix removed), rename the file on disk and update all ref paths that referenced the old slug. Gaps in suffix numbers are left as-is (no renumbering).
* Clean syncback rebuilds everything from scratch -- matching is only needed in incremental mode to preserve existing file paths where possible. In clean mode, base-name assignment for duplicate names uses Roblox child order (first child in DOM children list gets the base name, second gets ~1, etc.).
* It should not just keep "adding" more files each time a syncback happens, it needs to CORRECT/UPDATE them, utilizing the matching algorithm
* When a third duplicate is added (e.g., existing "Attachment" and "Attachment~1", now a third "Attachment" appears), the matching algorithm pairs the two existing ones, and the new one gets "Attachment~2". Existing suffixes are never renumbered.
Forward Sync:
* When a user edits Attachment~1/init.meta.json5 on disk, the server re-snapshots. The snapshot produces two children both named "Attachment" (from their meta name fields). compute_patch_set must use the matching algorithm to pair them correctly against the tree's existing Attachment instances, rather than greedy first-match which may cross-pair them.
API
* Leverage the UID we are sending it from the rust side, and vice-versa for newly created instances - make sure those are session persistant and utilized. We need to utilize this to keep things straight when we send data back to the rust side so it knows where to place things.
* The UID I'm speaking of is the Ref that the server assigns to each instance in the RojoTree. These are sent to the plugin via the API when connecting, and the plugin sends them back for newly created instances. This is the existing system -- no new UID mechanism is needed.
* Processing order in /api/write: use a two-pass approach. First pass: apply ALL mutations (renames, property changes, adds, removes) to the tree. Second pass: recompute all affected ref slug paths based on the final tree state. This ensures refs always use the newest slugs, even when a rename and a ref update targeting the renamed instance arrive in the same request.
Notes
* Case-insensitive instances need to be differentiated on the filesystem to properly support all operating systems. On Linux (case-sensitive FS), "Script" and "script" can coexist as separate files, but for cross-platform compatibility they must still be deduped with ~N suffixes. The matching algorithm handles these trivially since the true Names are different.
* This system needs to work for all sync directions/systems - syncback, two-way sync, forward sync, one-shot, rbxl construction from project, etc.
* Project file instances: project keys are unique by JSON definition (duplicate keys in JSON/JSON5 are not valid), so project-sourced instances do NOT participate in the dedup system. They always have unique names within their sibling group.
* Performance target: system must handle large games (10K+ instances) with <100ms latency per operation. Optimization can be deferred but the architecture must not have O(n^2) bottlenecks on the full tree. The matching algorithm should only run on the affected subtree when possible.
* We need to literally remove the ENTIRE system responsible for blacklisting duplicately named paths on the plugin and in the rust code, it is no longer relevant.
rbxm-ambiguous
* Additionally, you should review the changes done in the branch rbxm-ambiguous. I'm abandoning that branch, but these two problems are closely related. I don't really ever want projects to have rbxm files being outputted for syncback or two-way sync anymore, but I still want to fully support loading them, but they should also get removed/replaced properly when you do a syncback on clean mode.
* Best to do a full diff from origin/master
* We specifically mention / probably want to take these features which were mentioned specifically in this plan:
- The plugin relaxations (diff.lua, encodeInstance.lua duplicate-skip removal) -- these are needed regardless of approach
- The has_duplicate_children helper and case-insensitive detection
- The project.rs case-insensitive duplicate detection. Note: with the dedup-suffix approach, ProjectNode children with duplicate names CAN now be written as individual files with ~N suffixes -- the warn-and-skip guard from the rbxm branch may no longer be needed. The implementer should evaluate whether to remove it or keep it as a safety net.
- The roundtrip test improvements (compare_children Vec-based, Tags check)
- The macOS ref_properties test ignore
- The DuplicateChildrenError typed error pattern (or equivalent)
* Don't take: ambiguous_container metadata, find_rbxm_container, reserialize_rbxm_container, rbxm fallback block, all ambiguous test fixtures. We are specifically trying to avoid rbxm-ing things.
* No special transition logic needed for old .rbxm files. Syncback rewrites the filesystem from scratch (clean) or matches instances to existing files (incremental) -- either way, the rbxm just doesn't get recreated. User-intentional .rbxm files that users placed manually in their project still load fine via snapshot_rbxm during forward sync -- that's unchanged.
Sync System Invariants (from audit.md -- these apply to EVERY change in this feature)
* ROUND-TRIP IDENTITY: Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a bit-identical instance tree -- same names, same classes, same properties, same hierarchy, same ref targets. The dedup system, matching algorithm, and ref path system must ALL preserve this invariant. Any deviation is a bug.
* CLI SYNCBACK PARITY: Plugin-based two-way sync must produce EXACTLY the same filesystem output that `atlas syncback` would give for the same input. Byte-for-byte identical files, identical directory structures, identical naming, identical dedup suffixes. This means the dedup logic, matching algorithm, and ref path computation must be shared or equivalent between the two paths. Any divergence is a bug.
* 4 FILE FORMAT BRANCHES: syncback_updated_properties() in api.rs has 4 branches for writing property changes. The dedup/naming system must work correctly in ALL 4:
1. Directory format: init.meta.json5 inside the instance's directory
2. Standalone script: adjacent ScriptName.meta.json5 (strips .server/.client/.plugin/.local/.legacy suffix from stem)
3. Model file: inline in .model.json5 (writes directly into the file, NOT adjacent meta)
4. Other file types (.txt, .csv, .toml): adjacent FileName.meta.json5
Each branch must: compute the correct dedup'd filename, write the correct meta name field, and produce ref paths consistent with the dedup assignment.
* INSTIGATING SOURCE CONSISTENCY: every instance in the RojoTree tracks where it came from via InstigatingSource. When creating files with dedup suffixes, the InstigatingSource path must reflect the dedup'd path (including ~N suffix). Pre-seed taken_names from existing tree children via their instigating_source paths (using strip_middleware_extension), NOT from instance names. This prevents slug collisions with existing files on disk.
* FILE-TO-FOLDER TRANSITIONS: instances can flip between file representation (Foo.server.luau) and directory representation (Foo/init.server.luau + children). This transition changes the dedup group key (file stem vs directory name) and must update:
- The dedup suffix (may change or be added/removed)
- The meta file location (adjacent → init.meta.json5 inside directory, or vice versa)
- All Rojo_Ref_* paths that pass through this instance (slug changes)
- The InstigatingSource path
Round-trip: adding a child to a file instance then removing it must produce identical filesystem state to the original. Treat any code touching these transitions as high-risk.
* SCRIPT TYPE TRANSITIONS: ClassName changes (ModuleScript → Script) cause file extension changes (.luau → .server.luau). This changes the dedup group key since the filesystem representation changes. A ModuleScript "Foo" (.luau) and Script "Foo" (.server.luau) don't collide -- but if the ModuleScript becomes a Script, it now collides with the existing Script. The dedup system must handle this mid-session during two-way sync.
* DUAL WRITE PATH CONSISTENCY: handle_api_write in api.rs does TWO things: (1) writes files to disk, and (2) sends a PatchSet to the ChangeProcessor to update the in-memory tree. These MUST stay consistent. If the file write uses dedup suffix ~1 but the PatchSet doesn't reflect that, the tree and filesystem diverge silently.
* MULTIPLE PLUGIN ENTRY POINTS: property changes flow through the plugin via TWO separate paths:
1. ChangeBatcher flow (live two-way sync): Instance.Changed → InstanceMap → ChangeBatcher → encodePatchUpdate → ApiContext:write()
2. Confirmation dialog flow (initial sync pull): ServeSession.__confirmAndApplyInitialPatch → encodePatchUpdate → ApiContext:write()
Both paths must handle duplicate-named instances correctly. A function modified in path 1 may not have been updated in path 2.
* REVIEW DIALOGUE INVARIANT: No two-way sync change may bypass the review dialogue in Always and Initial sync modes. Dedup suffix renames, ref path updates, and meta file changes triggered by the matching algorithm must all be cataloged in the confirmation UI. No silent application.
* VFS ECHO PREVENTION: the existing suppress_path() system uses per-path counters. For dedup operations: suppress_path() MUST be called before any file write/rename during dedup suffix changes. New instance additions must NOT suppress (VFS watcher needs to pick up new files). The VFS event tagging mentioned in the Naming section should integrate with this existing mechanism.
* ATOMICITY: dedup suffix cleanup involves multiple filesystem operations (rename file, update meta, update ref paths). There is a window where the VFS watcher could fire between operations and snapshot an inconsistent intermediate state. Minimize race windows by ordering operations carefully and using suppress_path() to cover all intermediate states.
* DETERMINISM: given the same instance tree, the dedup system must ALWAYS produce the same filesystem output. Child ordering must be stable (sorted/deterministic), dedup suffix assignment must be deterministic, and generated identifiers must be reproducible. Non-determinism = different builds from same source = git churn = invariant violation.
* COMPOUND FILE EXTENSIONS: .server.luau, .client.luau, .model.json5, .meta.json5, .project.json5 are multi-part extensions. Dedup suffixes are applied to the STEM before the compound extension: Foo~1.server.luau (correct), NOT Foo.server~1.luau. Meta pairing: Foo~1.server.luau pairs with Foo~1.meta.json5 (strip script suffix from stem, then add .meta.json5). The suffix list in change_processor.rs must include all compound parts for rename operations.
Testing
* This system is extremely critical -- subtle bugs here silently corrupt user projects. Test the hell out of it at every layer: e2e, plugin, api, syncback, forward sync, two-way sync, build, roundtrip.
* Ensure we try all combinations, folder/meta swaps, etc. Try to make the system conflict with the existing slugify system.
* Define shared test vectors as structured data (e.g., two lists of instance descriptions -> expected match pairing) that both the Rust and Lua implementations must produce identical results for. This is the strongest guarantee that the two implementations don't diverge.
* The scenarios below are MINIMUM REQUIRED tests, not exhaustive. The implementer should add additional edge cases they discover during implementation. Think adversarially -- what combinations of operations could break invariants?
* Specific scenarios to test:
- Two identical Folders with same Name (simplest dedup case)
- Two Folders same Name, different children (matching must pair correctly by content)
- Two Folders same Name, identical content (tiebreaker: Roblox child order wins -- first in DOM children list gets base name)
- Folder "Foo" next to Script "Foo" (same Name, different ClassName -- NO dedup suffix because Foo/ and Foo.server.luau don't collide on disk. Both keep clean names.)
- Folder "Foo" next to Script "Foo" where Script has children (init-style: Foo/ vs Foo/init.server.luau -- these DO collide, dedup suffix applied)
- Instance naturally named "Foo~1" next to two "Foo" instances -- "Foo~1" slugifies to "Foo_1" (tilde replaced with underscore), which is distinct from the dedup suffix "Foo~1". Verify filesystem has Foo/, Foo~1/, Foo_1/ (or Foo_1.server.luau etc.)
- Third duplicate added to existing dedup group (no renumbering of existing suffixes)
- Dedup suffix cleanup (group→1): only two instances "Foo" and "Foo~1", base deleted, ~1 renamed to clean "Foo" (group shrank to 1)
- Gap tolerance: three instances Foo, Foo~1, Foo~2 -- delete ~1 -- verify Foo and Foo~2 remain unchanged (no renumbering)
- Base-name promotion: three instances Foo, Foo~1, Foo~2 -- delete Foo (base) -- lowest suffix ~1 promoted to base name, ~2 stays as ~2
- Gap tolerance with refs: Foo~2 has refs pointing to it, ~1 deleted, verify Foo~2's refs are untouched (no path changes since no rename occurred)
- Idempotency: syncback twice produces identical filesystem output
- Roundtrip: syncback -> build -> compare trees
* Ref path integrity tests (CRITICAL -- refs becoming stale/broken is the highest-risk failure mode):
- Basic ref survives syncback: Model.PrimaryPart points to a Part, syncback writes the ref path, build resolves it back to the same Part. Roundtrip identity preserved.
- Ref to a dedup'd instance: PrimaryPart points to "Attachment~1". Syncback writes the slug path. Build resolves it correctly (not to the other Attachment).
- Ref survives gap-tolerant deletion: Three "Attachment" instances (Attachment, Attachment~1, Attachment~2). Delete the middle one (~1). Attachment and Attachment~2 remain unchanged -- NO ref path updates needed. Verify all Rojo_Ref_* paths still resolve correctly.
- Ref survives base-name promotion: Three "Attachment" instances. Delete the base (Attachment). Attachment~1 is promoted to base name "Attachment". ALL Rojo_Ref_* paths that pointed to the old "Attachment~1" must now say "Attachment". Attachment~2 is unchanged. Verify by building.
- Ref survives group→1 cleanup: Two "Attachment" instances (Attachment, Attachment~1). Delete Attachment. Attachment~1 renamed to Attachment (group shrank to 1). ALL Rojo_Ref_* paths must update. Verify.
- Ref survives rename: Instance "Foo" is renamed to "Bar". All ref paths containing "Foo" segment must update to "Bar". Verify across meta files, model files, and project files.
- Ref survives reparent: Instance moves from Parent_A to Parent_B. The entire ancestor path changes. All ref paths must update. Verify.
- Ref to deeply nested instance through multiple dedup layers: A/B~1/C/D~2 where both B and D have dedup suffixes. Ref resolves correctly through the full chain.
- Cross-file ref update: File_A.meta.json5 has a ref to an instance whose slug changes. File_B.model.json5 also has a ref to the same instance. BOTH files must be updated. Verify by reading both files after the operation.
- Ref to instance that gets deleted: Ref path points to an instance that no longer exists after syncback. Should resolve to nil, not crash, not point to wrong instance.
- Ref path with instance whose natural name contains tilde: Instance named "Foo~1" (real name). Slug is "Foo_1" (tilde slugified). If there are also two "Foo" instances, filesystem has Foo/, Foo~1/ (dedup), Foo_1/ (slugified). Ref to the "Foo~1" instance uses slug path "Parent/Foo_1". Verify the resolver finds the correct instance.
- Two-way sync ref update: Plugin sends a property change that triggers a dedup suffix reassignment on the server. Server must update all Rojo_Ref_* paths in affected files AND the patch sent back to the plugin must reflect the correct ref targets.
- Concurrent ref + rename in same batch: In a single /api/write request, one instance is renamed (changing its slug) and another instance has a Ref property update pointing to the renamed instance. The ref path must use the NEW slug, not the old one.
- Forward sync ref resolution after manual edit: User manually edits a ref path in a meta file on disk (changes "Part/Attachment~1" to "Part/Attachment~2"). Server re-snapshots and resolves the ref. Verify it points to the correct instance or logs a warning if invalid.
- Fallback to $id system: Instance inside a user .rbxm file has no instigating_source. Ref to this instance should use the $id/Rojo_Target_* system, not a slug path. Verify the ref resolves correctly after build.
- Mass ref update stress test: 50+ instances all referencing the same target. Target gets renamed. All 50 Rojo_Ref_* paths must update. Verify none are missed.
* Matching algorithm tests (CRITICAL -- wrong pairing silently corrupts projects):
- Matching with ClassName discriminator: two "Part" instances, one is Part and one is MeshPart. Match by ClassName in Pass 1 (unique within same-Name group per class).
- Matching with Ref discriminator: two identical Folders "Data", one has a Ref from an ObjectValue pointing to it. Match via Ref in Pass 2.
- Matching with property similarity: two Scripts "Handler" with different Source. Match the one with closer Source in Pass 3.
- Matching stability: run matching on same input twice, verify identical pairing both times.
- Matching preserves existing pairings: during two-way sync, matched pairs should not reshuffle when a new sibling is added.
- Fast path verification: parent with all uniquely-named children skips Passes 2-3 entirely (verify via timing or instrumentation).
* Dedup group collision tests:
- Folder "Foo" + ModuleScript "Foo" → no collision (Foo/ vs Foo.luau), no dedup suffix
- Folder "Foo" + Folder "Foo" → collision (both Foo/), dedup applied
- Script "Foo" (server) + Script "Foo" (client) → no collision (Foo.server.luau vs Foo.client.luau), no dedup suffix
- Script "Foo" (server) + Script "Foo" (server) → collision (both Foo.server.luau), dedup applied
- ModuleScript "Foo" + ModuleScript "Foo" → collision (both Foo.luau), dedup applied
- Instance "Foo?" + Instance "Foo*" → both slugify to "Foo_", filesystem collision, dedup applied. Different matching groups (different true names) but same dedup group (same slug).
- Instance "Foo" + Instance "foo" → case-insensitive collision on Windows/macOS, dedup applied for cross-platform safety.
* Cross-system integration tests:
- Full roundtrip with duplicates: create place with duplicate-named instances → syncback → build → compare trees bit-for-bit
- Incremental syncback after rename: syncback, rename one duplicate in Studio, syncback again -- verify correct file is updated (not the wrong sibling)
- Two-way sync add duplicate: during live sync, add a third instance with same name -- verify correct ~N suffix assigned and refs intact
- Two-way sync delete from middle: during live sync, delete an instance from a dedup group -- verify gap tolerance (no renumbering)
- Forward sync edit dedup'd file: edit Attachment~1/init.meta.json5 on disk -- verify compute_patch_set pairs it to the correct tree instance
- Slugify + dedup interaction: instance named "What?Script" alongside "What_Script" alongside another "What?Script" -- first two slugify to same thing, third is a true duplicate of the first. Verify all three get correct unique filesystem names and refs resolve.
- Project file + filesystem duplicates: project defines child "Foo" via $path, filesystem sibling also named "Foo" -- project key wins (unique by definition), filesystem child coexists without dedup.
* Sync invariant tests (from audit.md concepts):
- CLI syncback parity: run CLI syncback on a place with duplicates, then run two-way sync on the same place -- verify byte-identical filesystem output. Any divergence is a bug.
- 4 file format branches: verify dedup works for (1) directory with init.meta.json5, (2) standalone script with adjacent .meta.json5, (3) model file with inline name, (4) .txt/.csv with adjacent .meta.json5. Each must produce correct dedup'd names and meta name fields.
- Compound extension dedup: Script~1.server.luau pairs with Script~1.meta.json5 (not Script.server~1.luau or Script~1.server.meta.json5). Verify meta pairing is correct.
- InstigatingSource after dedup: verify that instances written with ~N suffixes have InstigatingSource::Path pointing to the dedup'd path (e.g., src/Foo~1.server.luau, not src/Foo.server.luau).
- File-to-folder transition with dedup: Script "Foo" (flat, Foo.server.luau) gets a child added → becomes Foo/init.server.luau. If another "Foo" folder already exists, dedup suffix must be applied to the new directory. All ref paths must update.
- Script type transition in dedup group: ModuleScript "Foo" and Script "Foo" don't collide (Foo.luau vs Foo.server.luau). Change ModuleScript to Script → now both are .server.luau → dedup collision created mid-session. Verify dedup suffix added and refs updated.
- Dual write path consistency: during /api/write, verify the file written to disk uses the same dedup suffix as the PatchSet sent to ChangeProcessor. Kill the server mid-sync and verify build from filesystem matches the in-memory tree state.
- Review dialogue with dedup: in Always sync mode, adding a duplicate-named instance shows the dedup assignment in the confirmation UI before applying.
- Atomicity: rename Attachment~1 to Attachment (suffix cleanup) while VFS watcher is active. Verify no intermediate state is snapshotted where both Attachment and Attachment~1 exist or neither exists.
- Determinism: build the same project 3 times from scratch -- verify all 3 outputs are bit-identical. No non-deterministic suffix assignment or child ordering.
- Idempotency: syncback twice on same input with no changes in between -- verify zero filesystem modifications on second run.
- Pre-seeded dedup: add a new instance "Foo" to a parent that already has Foo.server.luau on disk. Verify taken_names is seeded from instigating_source paths and the new instance gets ~1 (not overwriting the existing file).
This is a large refactor, and I may be missing some critical details in this - ask many questions to make sure there is no ambiguity.
XXH32.luau:
--!strict
--!optimize 2
--!native
local function XXH32(Message: buffer, Seed: number?): number
local PRIME_1, PRIME_1_HIGH, PRIME_1_LOW = 0x9e3779B1, 40503, 31153
local PRIME_2, PRIME_2_HIGH, PRIME_2_LOW = 0x85ebca77, 34283, 51831
local PRIME_3_HIGH, PRIME_3_LOW = 49842, 44605
local PRIME_4_HIGH, PRIME_4_LOW = 10196, 60207
local PRIME_5, PRIME_5_HIGH, PRIME_5_LOW = 0x165667b1, 5718, 26545
local UsedSeed = Seed or 0
local MessageLength = buffer.len(Message)
local Digest: number
local CurrentOffset = 0
if MessageLength >= 16 then
local Accumulator1 = UsedSeed + PRIME_1 + PRIME_2
local Accumulator2 = UsedSeed + PRIME_2
local Accumulator3 = UsedSeed
local Accumulator4 = UsedSeed - PRIME_1
while CurrentOffset <= MessageLength - 16 do
local Word1 = buffer.readu32(Message, CurrentOffset)
local Word2 = buffer.readu32(Message, CurrentOffset + 4)
local Word3 = buffer.readu32(Message, CurrentOffset + 8)
local Word4 = buffer.readu32(Message, CurrentOffset + 12)
local AHigh1, ALow1 = bit32.rshift(Word1, 16), bit32.band(Word1, 65535)
local Mult1 = bit32.lshift((AHigh1 * PRIME_2_LOW) + (ALow1 * PRIME_2_HIGH), 16) + (ALow1 * PRIME_2_LOW)
local Temp1 = bit32.lrotate(Accumulator1 + Mult1, 13)
local AHigh1_2, ALow1_2 = bit32.rshift(Temp1, 16), bit32.band(Temp1, 65535)
Accumulator1 = bit32.lshift((AHigh1_2 * PRIME_1_LOW) + (ALow1_2 * PRIME_1_HIGH), 16) + (ALow1_2 * PRIME_1_LOW)
local AHigh2, ALow2 = bit32.rshift(Word2, 16), bit32.band(Word2, 65535)
local Mult2 = bit32.lshift((AHigh2 * PRIME_2_LOW) + (ALow2 * PRIME_2_HIGH), 16) + (ALow2 * PRIME_2_LOW)
local Temp2 = bit32.lrotate(Accumulator2 + Mult2, 13)
local AHigh2_2, ALow2_2 = bit32.rshift(Temp2, 16), bit32.band(Temp2, 65535)
Accumulator2 = bit32.lshift((AHigh2_2 * PRIME_1_LOW) + (ALow2_2 * PRIME_1_HIGH), 16) + (ALow2_2 * PRIME_1_LOW)
local AHigh3, ALow3 = bit32.rshift(Word3, 16), bit32.band(Word3, 65535)
local Mult3 = bit32.lshift((AHigh3 * PRIME_2_LOW) + (ALow3 * PRIME_2_HIGH), 16) + (ALow3 * PRIME_2_LOW)
local Temp3 = bit32.lrotate(Accumulator3 + Mult3, 13)
local AHigh3_2, ALow3_2 = bit32.rshift(Temp3, 16), bit32.band(Temp3, 65535)
Accumulator3 = bit32.lshift((AHigh3_2 * PRIME_1_LOW) + (ALow3_2 * PRIME_1_HIGH), 16) + (ALow3_2 * PRIME_1_LOW)
local AHigh4, ALow4 = bit32.rshift(Word4, 16), bit32.band(Word4, 65535)
local Mult4 = bit32.lshift((AHigh4 * PRIME_2_LOW) + (ALow4 * PRIME_2_HIGH), 16) + (ALow4 * PRIME_2_LOW)
local Temp4 = bit32.lrotate(Accumulator4 + Mult4, 13)
local AHigh4_2, ALow4_2 = bit32.rshift(Temp4, 16), bit32.band(Temp4, 65535)
Accumulator4 = bit32.lshift((AHigh4_2 * PRIME_1_LOW) + (ALow4_2 * PRIME_1_HIGH), 16) + (ALow4_2 * PRIME_1_LOW)
CurrentOffset += 16
end
Digest = bit32.lrotate(Accumulator1, 1) + bit32.lrotate(Accumulator2, 7) + bit32.lrotate(Accumulator3, 12) + bit32.lrotate(Accumulator4, 18)
else
Digest = UsedSeed + PRIME_5
end
Digest += MessageLength
while CurrentOffset <= MessageLength - 4 do
if CurrentOffset + 4 <= buffer.len(Message) then
local Word = buffer.readu32(Message, CurrentOffset)
local AHigh_w, ALow_w = bit32.rshift(Word, 16), bit32.band(Word, 65535)
local Mult_w = bit32.lshift((AHigh_w * PRIME_3_LOW) + (ALow_w * PRIME_3_HIGH), 16) + (ALow_w * PRIME_3_LOW)
Digest += Mult_w
local Rotated = bit32.lrotate(Digest, 17)
local AHigh_r, ALow_r = bit32.rshift(Rotated, 16), bit32.band(Rotated, 65535)
Digest = bit32.lshift((AHigh_r * PRIME_4_LOW) + (ALow_r * PRIME_4_HIGH), 16) + (ALow_r * PRIME_4_LOW)
end
CurrentOffset += 4
end
while CurrentOffset < MessageLength do
if CurrentOffset < buffer.len(Message) then
local ByteValue = buffer.readu8(Message, CurrentOffset)
local AHigh_b, ALow_b = bit32.rshift(ByteValue, 16), bit32.band(ByteValue, 65535)
local Mult_b = bit32.lshift((AHigh_b * PRIME_5_LOW) + (ALow_b * PRIME_5_HIGH), 16) + (ALow_b * PRIME_5_LOW)
Digest += Mult_b
local Rotated_b = bit32.lrotate(Digest, 11)
local AHigh_rb, ALow_rb = bit32.rshift(Rotated_b, 16), bit32.band(Rotated_b, 65535)
Digest = bit32.lshift((AHigh_rb * PRIME_1_LOW) + (ALow_rb * PRIME_1_HIGH), 16) + (ALow_rb * PRIME_1_LOW)
end
CurrentOffset += 1
end
local XorResult1 = bit32.bxor(Digest, bit32.rshift(Digest, 15))
local AHigh_f1, ALow_f1 = bit32.rshift(XorResult1, 16), bit32.band(XorResult1, 65535)
Digest = bit32.lshift((AHigh_f1 * PRIME_2_LOW) + (ALow_f1 * PRIME_2_HIGH), 16) + (ALow_f1 * PRIME_2_LOW)
local XorResult2 = bit32.bxor(Digest, bit32.rshift(Digest, 13))
local AHigh_f2, ALow_f2 = bit32.rshift(XorResult2, 16), bit32.band(XorResult2, 65535)
Digest = bit32.lshift((AHigh_f2 * PRIME_3_LOW) + (ALow_f2 * PRIME_3_HIGH), 16) + (ALow_f2 * PRIME_3_LOW)
return bit32.bxor(Digest, bit32.rshift(Digest, 16))
end
return XXH32

309
.cursor/rules/atlas-api.mdc
View file

@ -1,11 +1,6 @@
---
description: "Web API and plugin communication: WebSocket protocol, live sync architecture, two-way sync, patch format, ServeSession state management, and ChangeBatcher."
globs:
- "src/web/**/*.rs"
- "src/serve_session.rs"
- "plugin/src/ServeSession.lua"
- "plugin/src/ChangeBatcher/**/*.lua"
- "plugin/src/ApiContext.lua"
globs: src/web/**,src/serve_session.rs,src/git.rs,,plugin/src/**
alwaysApply: false
---
# Atlas API & Communication
@ -50,6 +45,7 @@ Studio → ChangeBatcher → Plugin → WebSocket → Server → Filesystem
| `/api/write` | POST | Two-way sync: plugin sends changes |
| `/api/open/:id` | POST | Open script in editor (`{"cursor": [line, col]}`) |
| `/api/ref-patch/:id` | GET | Fix Ref properties after syncback |
| `/api/syncback` | POST | Live syncback: full Roblox → filesystem sync |
| `/api/socket/:cursor` | GET | WebSocket upgrade for live patch streaming |
### Server Info Response (`GET /api/rojo`)
@ -68,10 +64,22 @@ Studio → ChangeBatcher → Plugin → WebSocket → Server → Filesystem
"visibleServices": ["Workspace", "ServerScriptService"],
"placeId": 123456,
"gameId": 789012,
"ignoreHiddenServices": true
"ignoreHiddenServices": true,
"gitMetadata": {
"changedIds": ["ref-1", "ref-2"],
"scriptCommittedHashes": {
"ref-1": ["a1b2c3...40-char-sha1-hex", "d4e5f6...staged-sha1"]
}
}
}
```
**`gitMetadata`** is `null` when the project is not in a git repo. Contains:
- `changedIds`: Instance Refs whose backing files have changes relative to HEAD (staged + unstaged + untracked)
- `scriptCommittedHashes`: For changed script instances, 1-2 SHA1 hashes (HEAD version and staged version if different) computed using the git blob format (`"blob <size>\0<content>"`). Plugin checks if Studio `script.Source` matches ANY hash to default "Atlas".
Recomputed fresh on each `GET /api/rojo` connect. Uses a two-phase tree lock: snapshot path mappings briefly, then release before running git subprocesses.
### Write Request (`POST /api/write`)
```json
@ -79,12 +87,108 @@ Studio → ChangeBatcher → Plugin → WebSocket → Server → Filesystem
"sessionId": "uuid",
"removed": ["id-1"],
"added": { "id-2": { "parent": "parent-id", "className": "Folder", "name": "New", "properties": {}, "children": [] } },
"updated": [{ "id": "id-3", "changedName": "Renamed", "changedClassName": "Script", "changedProperties": { "Source": "print('hi')" }, "changedMetadata": {} }]
"updated": [{ "id": "id-3", "changedName": "Renamed", "changedClassName": "Script", "changedProperties": { "Source": "print('hi')" }, "changedMetadata": {} }],
"stageIds": ["ref-1", "ref-2"]
}
```
**`stageIds`**: Instance IDs whose backing files should be staged via `git add` after writing. Includes all push-accepted IDs + auto-selected pull-accepted IDs. The server resolves these to filesystem paths and runs `git add`. Staging is split: `api.rs` stages additions/removals/push files directly; `change_processor` stages Source writes after they complete (via `stage_ids` on PatchSet).
**Format detection:** `Name/init.luau` → directory | `Name.luau` → file | neither → use `has_children` (`.lua` also supported)
### Live Syncback Endpoint (`POST /api/syncback`)
One-click full sync from Studio to filesystem. Does **not** require an active WebSocket connection -- the plugin POSTs directly from the NotConnected page using the host/port fields.
**Request (`SyncbackRequest`):**
```rust
pub struct SyncbackRequest {
pub protocol_version: f64, // f64 because Luau has no integer type
pub server_version: String,
pub place_id: Option<f64>,
pub data: Vec<u8>, // Single rbxm blob containing ALL children from ALL services
pub services: Vec<ServiceChunk>,
}
```
**`ServiceChunk`:**
```rust
pub struct ServiceChunk {
pub class_name: String,
pub child_count: u32,
pub ref_target_count: u32,
pub properties: HashMap<String, Variant>, // Service-level properties (omitted if empty)
pub refs: HashMap<String, u32>, // Ref property name → 1-based carrier index (omitted if empty)
}
```
**Data format:** The `data` field is a single rbxm binary blob containing all children from all services concatenated. Each service's range in the blob is `child_count + ref_target_count` consecutive entries. First `child_count` entries are real children (parented to service). Next `ref_target_count` entries are temporary ObjectValue carriers encoding Ref properties (destroyed after resolving). The `refs` map connects property names to carrier indices.
**Validation:**
- Protocol version must match `PROTOCOL_VERSION` (6) → 400 Bad Request
- Server version MAJOR.MINOR must match → 400 Bad Request
- Place ID must pass `servePlaceIds`/`blockedPlaceIds` checks if configured → 403 Forbidden
**Response:** `{"status": "syncback_initiated"}`
**Server-side types:**
```rust
pub struct SyncbackPayload {
pub data: Vec<u8>,
pub services: Vec<ServiceChunk>,
}
pub enum ServerExitReason {
SyncbackRequested(SyncbackPayload),
}
```
**`SyncbackSignal` (`src/web/mod.rs`):** Thread-safe signal for API handler → serve loop communication. Uses `Mutex<Option<SyncbackPayload>>` + `Notify`. `fire()` deposits payload and notifies; `take_payload()` retrieves it. `LiveServer::start()` uses `tokio::select!` to wait for either new connections OR the syncback signal, exiting the accept loop when fired.
### Live Syncback Data Flow
```
Plugin (NotConnected page)
│ User clicks "Syncback" button → SyncbackConfirm dialog
│ User confirms → performSyncback()
│ Encodes services via encodeService() + serializes children via SerializationService
▼
HTTP POST /api/syncback (MessagePack body)
│
▼
handle_api_syncback() [src/web/api.rs]
│ Validates protocol, version, place ID
│ Fires SyncbackSignal
▼
LiveServer::start() receives notification [src/web/mod.rs]
│ Exits accept loop
│ Returns ServerExitReason::SyncbackRequested(payload)
▼
run_live_syncback() [src/cli/serve.rs]
│ build_dom_from_chunks(payload) → reconstructs DOM from rbxm + service chunks
│ syncback_loop(incremental = false) → clean mode, full filesystem rewrite
│ write_to_vfs_parallel() → writes files
│ refresh_git_index() → updates git index
▼
Serve loop restarts (new session, ready for connections)
```
### Live Syncback vs Two-Way Sync
| Aspect | Live Syncback (`/api/syncback`) | Two-Way Sync (`/api/write`) |
|--------|--------------------------------|---------------------------|
| **Purpose** | Full Roblox → filesystem sync | Incremental property/instance changes |
| **Scope** | Entire instance tree (all services) | Individual instance changes |
| **Mode** | Clean mode (replaces everything) | Incremental (add/update/remove) |
| **Server restart** | Yes (restarts serve loop) | No (continues serving) |
| **Data format** | Single rbxm blob + service chunks | Patch-based (added/updated/removed) |
| **Connection** | No WebSocket needed (HTTP POST) | Requires active WebSocket session |
| **When used** | User-initiated one-click syncback | Real-time bidirectional sync |
| **Session** | Creates new session after restart | Uses existing live session |
## WebSocket Protocol (v6)
**Connection:** `ws://host:port/api/socket/:cursor`
@ -177,7 +281,7 @@ for id, data in patch.added do instanceMap:insert(id, Reconciler.reify(data)) en
for _, update in patch.updated do Reconciler.hydrate(instanceMap:fromId(update.id), update) end
```
## Two-Way Sync (Experimental)
## Two-Way Sync
**Setting:** `twoWaySync` (default: true) | Syncs Studio changes → filesystem in real-time
@ -229,13 +333,21 @@ for _, update in patch.updated do Reconciler.hydrate(instanceMap:fromId(update.i
│ │ Looks up InstigatingSource::Path → fs::remove_file() │
│ ▼ │
│ For each added: syncback_added_instance() │
│ │ Groups by parent, pre-seeds sibling_slugs, dedup on collision │
│ │ Determines file format, writes to filesystem │
│ ▼ │
│ Build added_paths map (for same-batch Ref resolution) │
│ │ Maps Ref → filesystem path via added_instance_fs_segment() │
│ ▼ │
│ For each updated: syncback_updated_properties() │
│ │ Extracts Ref props → Rojo_Ref_* attrs (path from tree/added_paths) │
│ │ merge_or_build_meta() writes props/attrs (supports attr removal) │
│ ▼ │
│ Sends PatchSet to tree_mutation_sender channel │
│ ▼ │
│ ChangeProcessor::handle_tree_event() [src/change_processor.rs] │
│ │ For removed: fs::remove_file(path) + remove meta │
│ │ For updated: Source writes, renames, ClassName changes, properties │
│ │ For removed: dedup cleanup (gap-tolerant, base-name promotion) │
│ │ For updated: Source writes, renames, ClassName changes, ref paths │
│ ▼ │
│ apply_patch_set() → Updates RojoTree │
│ ▼ │
@ -293,17 +405,21 @@ end
| File | Purpose |
|------|---------|
| `init.lua` | Main batcher: accumulator, RenderStepped loop, pause/resume |
| `createPatchSet.lua` | Converts pending changes → PatchSet |
| `encodePatchUpdate.lua` | Encodes property updates for existing instances |
| `encodeInstance.lua` | Full instance encoding (for additions/syncback) |
| `init.lua` | Main batcher: accumulator, RenderStepped loop, pause/resume, deferred refs |
| `createPatchSet.lua` | Converts pending changes → PatchSet (passes instanceMap for Ref resolution) |
| `encodePatchUpdate.lua` | Encodes property updates including Ref properties via InstanceMap |
| `encodeInstance.lua` | Full instance encoding (for additions/syncback); uses `encodeHelpers` for property iteration; includes duplicate detection helpers (`findDuplicateNames`, `hasDuplicateSiblings`, `isPathUnique`) |
| `encodeHelpers.lua` | Shared encoding helpers: `encodeAttributes`, `encodeTags`, `forEachEncodableProperty`. Used by `encodeInstance.lua` and `encodeService.lua`. Defines `SKIP_PROPERTIES` (Parent, Name, Archivable). |
| `encodeService.lua` | Encodes service-level properties/attributes/refs for live syncback; uses ObjectValue carriers for Ref properties |
| `encodeProperty.lua` | Single property value encoding via RbxDom |
| `propertyFilter.lua` | Property type filtering (only `UniqueId` blocked; `Ref` is allowed) |
**Core State:**
```lua
ChangeBatcher = {
__accumulator = 0, -- Time since last flush
__pendingPropertyChanges = {}, -- Instance → { PropertyName → true }
__pendingUnresolvedRefs = {}, -- targetInstance → [{sourceInstance, propertyName}]
__syncSourceOnly = false, -- Only sync Source property
__paused = false, -- Pause during confirmation
__renderSteppedConnection = conn, -- Frame loop connection
@ -338,10 +454,20 @@ end
- Reduces server load and WebSocket traffic
- Still feels responsive to users
**Deferred Ref Mechanism:**
When a Ref property targets an instance not yet in the InstanceMap (e.g., newly pulled instance whose server ID hasn't arrived), the property is deferred:
```lua
-- In encodePatchUpdate: onUnresolvedRef callback defers the property
-- In ChangeBatcher: __pendingUnresolvedRefs[targetInstance] = {sourceInstance, propertyName}
-- When target appears (onInstanceInserted), re-queues the property change for next batch
```
### createPatchSet.lua: Patch Generation
```lua
function createPatchSet(instanceMap, propertyChanges, syncSourceOnly)
function createPatchSet(instanceMap, propertyChanges, syncSourceOnly, onUnresolvedRef)
local patch = PatchSet.newEmpty()
for instance, properties in propertyChanges do
@ -353,20 +479,18 @@ function createPatchSet(instanceMap, propertyChanges, syncSourceOnly)
if properties.Parent then
if instance.Parent == nil then
-- Instance was deleted in Studio
table.insert(patch.removed, instanceId)
else
Log.warn("Cannot sync non-nil Parent changes yet") -- Reparenting not supported
Log.warn("Cannot sync non-nil Parent changes yet")
end
else
if syncSourceOnly then
-- Server in source-only mode: only sync Source property
if properties.Source then
local update = encodePatchUpdate(instance, instanceId, { Source = true })
local update = encodePatchUpdate(instance, instanceId, { Source = true }, instanceMap, onUnresolvedRef)
if update then table.insert(patch.updated, update) end
end
else
local update = encodePatchUpdate(instance, instanceId, properties)
local update = encodePatchUpdate(instance, instanceId, properties, instanceMap, onUnresolvedRef)
if update then table.insert(patch.updated, update) end
end
end
@ -378,8 +502,10 @@ end
### encodePatchUpdate.lua: Update Encoding
Encodes property updates for existing instances. Now handles Ref properties via InstanceMap lookup.
```lua
function encodePatchUpdate(instance, instanceId, properties)
function encodePatchUpdate(instance, instanceId, properties, instanceMap, onUnresolvedRef)
local update = {
id = instanceId,
changedProperties = {},
@ -392,15 +518,30 @@ function encodePatchUpdate(instance, instanceId, properties)
local descriptor = RbxDom.findCanonicalPropertyDescriptor(instance.ClassName, propertyName)
if not descriptor then continue end
local success, encoded = encodeProperty(instance, propertyName, descriptor)
if success then
update.changedProperties[propertyName] = encoded
if descriptor.dataType == "Ref" then
-- Ref property: resolve target via InstanceMap
local target = instance[propertyName]
if target == nil then
update.changedProperties[propertyName] = { Ref = NULL_REF }
else
local targetId = instanceMap.fromInstances[target]
if targetId then
update.changedProperties[propertyName] = { Ref = targetId }
elseif onUnresolvedRef then
onUnresolvedRef(target, instance, propertyName)
end
end
else
local success, encoded = encodeProperty(instance, propertyName, descriptor)
if success then
update.changedProperties[propertyName] = encoded
end
end
end
end
if next(update.changedProperties) == nil and update.changedName == nil then
return nil -- No actual changes
return nil
end
return update
@ -409,65 +550,31 @@ end
### encodeInstance.lua: Full Instance Encoding (Syncback/Additions)
Used when pulling new instances from Studio to filesystem. **Critical: handles duplicate detection.**
Used when pulling new instances from Studio to filesystem. Uses `encodeHelpers.forEachEncodableProperty()` for property iteration (shared with `encodeService.lua`). Duplicate-named instances are encoded and sent to the server; the server handles filesystem collisions via the dedup suffix system (`~N`). Includes duplicate detection helpers (`findDuplicateNames`, `hasDuplicateSiblings`, `isPathUnique`).
```lua
-- Duplicate handling (matches server's ref_properties.rs::is_path_unique)
local function isPathUnique(instance)
local current = instance
while current do
if hasDuplicateSiblings(current) then
return false -- Any duplicate in ancestor chain = ambiguous path
end
current = current.Parent
if current == game then break end
end
return true
end
function encodeInstance(instance, parentId, _skipPathCheck)
-- Skip instances with ambiguous paths (duplicate-named siblings)
if not _skipPathCheck and not isPathUnique(instance) then
Log.warn("Skipped '{}' - path contains duplicate-named siblings", instance:GetFullName())
return nil
end
local properties = {}
-- Encode Attributes and Tags (work on all instance types)
encodeAttributes(instance, properties)
encodeTags(instance, properties)
Helpers.encodeAttributes(instance, properties)
Helpers.encodeTags(instance, properties)
-- For scripts, Source is required
if SCRIPT_CLASS_NAMES[instance.ClassName] then
local success, encoded = encodeProperty(instance, "Source", descriptor)
if not success then return nil end
properties.Source = encoded
end
-- Encode all serializable properties
local classDescriptor = RbxDom.findClassDescriptor(instance.ClassName)
for propertyName, meta in classDescriptor.properties do
if SKIP_PROPERTIES[propertyName] then continue end -- Parent, Name, Archivable
if SKIP_PROPERTIES_BY_CLASS[instance.ClassName]?[propertyName] then continue end -- Source, ScriptGuid, etc.
local isReadable = meta.scriptability == "ReadWrite" or meta.scriptability == "Read"
local doesSerialize = meta.serialization ~= "DoesNotSerialize"
if isReadable and doesSerialize then
local success, encoded = encodeProperty(instance, propertyName, descriptor)
if success and encoded ~= nil then
properties[propertyName] = encoded
end
Helpers.forEachEncodableProperty(instance.ClassName, SKIP_PROPERTIES_BY_CLASS, function(propertyName, descriptor)
local success, encoded = encodeProperty(instance, propertyName, descriptor)
if success and encoded ~= nil then
properties[propertyName] = encoded
end
end
end)
-- Recursively encode children (skipping duplicates at each level)
local children = {}
local childDuplicates = findDuplicateNames(instance:GetChildren())
for _, child in instance:GetChildren() do
if childDuplicates[child.Name] then continue end
local encoded = encodeInstance(child, nil, true) -- Skip path check, already validated
local encoded = encodeInstance(child, nil, true)
if encoded then table.insert(children, encoded) end
end
@ -521,19 +628,31 @@ true, false, 42, "string"
### Server-Side: ChangeProcessor (`src/change_processor.rs`)
Receives patches from plugin via `tree_mutation_receiver` channel. Handles removals, Source writes, renames (with slugify/dedup/meta lifecycle), ClassName changes (script type transitions via file renames), and property writes to meta/model files.
Receives patches from plugin via `tree_mutation_receiver` channel. Handles removals (with dedup cleanup), Source writes, renames (with slugify/dedup/meta lifecycle), ClassName changes (script type transitions via file renames), property writes to meta/model files, and ref path updates.
**Capabilities:**
- **Removals:** Deletes file + associated meta file via `InstigatingSource::Path`
- **Dedup cleanup after removals:** When a deletion leaves a dedup group needing suffix cleanup, performs renames (gap-tolerant, base-name promotion, group-to-1). Uses `removed_set` (HashSet) to exclude co-removed siblings from sibling enumeration. Calls `compute_cleanup_action()` from `dedup_suffix.rs`.
- **Source writes:** Updates script file contents
- **Renames:** Slugifies new name, deduplicates against siblings, renames file on disk, creates/updates/removes meta `name` field as needed. Handles both standalone files and init-file directories.
- **ClassName changes:** Script-to-script transitions (e.g. ModuleScript → Script) via file extension renames
- **Property writes:** Attributes and other properties written to meta/model files via `syncback_updated_properties` in `api.rs`
- **Ref path updates:** `RefPathIndex` tracks which meta/model files contain `Rojo_Ref_*` paths. `update_ref_paths_after_rename()` efficiently updates all affected files when an instance is renamed or dedup-cleaned.
- **VFS suppression for dedup renames:** `suppress_path_any(&from)` + `suppress_path(&to)` prevents the VFS watcher from re-triggering on files renamed during dedup cleanup.
- **InstigatingSource updates:** After dedup cleanup renames, the renamed sibling's `InstigatingSource::Path` and metadata path indexes are updated in-memory.
**InstigatingSource:** Every instance in RojoTree tracks where it came from:
- `InstigatingSource::Path(PathBuf)` - Created from a file, can be written back
- `InstigatingSource::ProjectNode { path, name, node, parent_class }` - Defined in project file, **cannot be modified via two-way sync**
**`handle_tree_event()` flow:**
1. Process removals (log but don't delete -- API already deleted files)
2. Dedup cleanup (runs before `apply_patch_set` to use parent/sibling relationships)
3. Process updates: renames, ClassName changes, Source writes, ref path updates
4. `apply_patch_set()` to update RojoTree
5. Apply metadata updates (InstigatingSource paths for both regular and dedup renames)
6. Send applied patch to message queue for plugin broadcast
### Pause Mechanism: Critical for Consistency
**Why pause is essential:**
@ -584,6 +703,25 @@ end
- Safe for production environments
- Team viewing same project simultaneously
### Git-Based Sync Direction Defaults
When the plugin shows the sync confirmation dialog, items get smart default selections based on git status:
- **File has no git changes**: default to "Studio" (pull) -- any diff is because Studio was changed
- **File has git changes AND is a script AND Studio Source matches committed/staged hash**: default to "Atlas" (push) -- developer changed the file, Studio hasn't diverged
- **Otherwise**: `nil` -- user must decide (both sides changed, or non-script with changes)
- **No git repo**: all defaults remain `nil` (feature is no-op)
Only **Updated** items get defaults. Added/Removed items remain `nil`.
**Hash computation**: Both sides use `SHA1("blob <byte_len>\0<content>")` (git blob format). Server sends up to 2 hashes per changed script (HEAD + staged if different). Plugin computes hash of `script.Source` (non-draft version) and checks if ANY hash matches.
**Auto-staging**: After confirmation, accepted files are staged via `git add`:
- Push-accepted items: always staged (all types, including non-scripts)
- Pull-accepted items: only staged if auto-selected (`defaultSelection ~= nil`); manually-chosen pulls are left unstaged for manual diff
**Implementation files**: `src/git.rs` (server git helpers), `plugin/src/PatchTree.lua` (default selection logic), `plugin/src/SHA1.luau` (hash computation), `src/web/interface.rs` (`GitMetadata` struct, `stage_ids` on WriteRequest/PatchSet)
### Known Limitations (Detailed)
| Issue | Cause | Impact |
@ -592,8 +730,23 @@ end
| **Rapid changes interleave** | 200ms batch window may split logically related changes | Partial updates can leave inconsistent state |
| **Reparenting not supported** | `Parent` property changes to non-nil values are ignored | Moving instances in Studio doesn't sync |
| **Project node instances immutable** | InstigatingSource::ProjectNode cannot be modified | Instances from project file can't be changed |
| **Duplicate-named siblings skipped** | Path ambiguity makes reliable sync impossible | Instances with same-name siblings ignored |
| **Ref properties complex** | Instance references may become invalid when instances move | ObjectValue.Value may not sync correctly |
| **Greedy matching not globally optimal** | Ambiguous groups (same Name+ClassName) use greedy assignment | Rare edge cases may produce suboptimal pairing and rename churn |
| **Ref paths with ancestor-level ambiguity** | `Rojo_Ref_*` path-based refs written to disk may resolve to wrong sibling if duplicates exist at any ancestor level | Ref may point to wrong instance after rebuild; in-memory tree is correct during session |
### Reconciler Matching Algorithm (`plugin/src/Reconciler/`)
The plugin uses a recursive change-count scoring algorithm (implemented in `matching.lua`) to pair incoming virtual instances with existing Studio instances during hydration.
**Entry point:** `Matching.matchChildren(virtualChildren, studioChildren, virtualInstances)`
**Algorithm:**
1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring.
2. **Ambiguous groups:** For groups with multiple candidates, compute pairwise `computePairCost()` (recursive, scores entire subtree). Greedy assignment sorted by cost ascending, child-order tiebreaker.
3. **Leftovers:** Unmatched virtual = new instances, unmatched Studio = deleted.
**Performance:** File-level `--!native` and `--!optimize 2` directives for native codegen. `MatchingSession` caches scoring results across recursive calls (`matchCache` for match results, `costCache` for pair costs), avoiding redundant computation when hydrate re-calls `matchChildren` on the same children. `cacheVirtual()` filters children to only those existing in `virtualInstances`, preventing crashes on incomplete virtual instance trees.
**Value equality:** Shared `trueEquals.lua` module used by both `matching.lua` and `diff.lua` (fuzzy equality for floats with epsilon 0.0001, Color3, CFrame, Vector3; NaN handling; nil/null-ref equivalence).
## Protocol Enforcement
@ -635,6 +788,10 @@ Browser: `http://localhost:34872` → Network tab → WebSocket messages
| Server | Plugin |
|--------|--------|
| `src/web/api.rs` - Endpoints | `plugin/src/ServeSession.lua` - Connection |
| `src/web/interface.rs` - Packets | `plugin/src/ApiContext.lua` - HTTP/WS client |
| `src/serve_session.rs` - Session | `plugin/src/ChangeBatcher/` - Two-way sync |
| | `plugin/src/Reconciler/` - Patch application |
| `src/web/interface.rs` - Packets, syncback types | `plugin/src/ApiContext.lua` - HTTP/WS client |
| `src/web/mod.rs` - SyncbackSignal, LiveServer | `plugin/src/ChangeBatcher/` - Two-way sync |
| `src/serve_session.rs` - Session | `plugin/src/ChangeBatcher/encodeService.lua` - Live syncback encoding |
| `src/cli/serve.rs` - Live syncback handler | `plugin/src/Reconciler/` - Patch application |
| `src/change_processor.rs` - FS writer | `plugin/src/Reconciler/matching.lua` - Matching |
| `src/rojo_ref.rs` - Ref paths | `plugin/src/PatchTree.lua` - Default selections |
| `src/git.rs` - Git integration | `plugin/src/App/init.lua` - Live syncback UI |

579
.cursor/rules/atlas-dedup.mdc Normal file
View file

@ -0,0 +1,579 @@
---
globs: src/syncback/file_names.rs,src/syncback/dedup_suffix.rs,src/syncback/matching.rs,src/syncback/ref_properties.rs,src/snapshot/matching.rs,src/snapshot/tree.rs,src/snapshot/patch_compute.rs,src/rojo_ref.rs,src/variant_eq.rs,src/change_processor.rs,src/web/api.rs,plugin/src/Reconciler/matching.lua,plugin/src/Reconciler/trueEquals.lua,plugin/src/Reconciler/hydrate.lua,plugin/src/Reconciler/diff.lua,plugin/src/XXH32.luau,tests/tests/two_way_sync.rs,tests/tests/syncback_roundtrip.rs
alwaysApply: false
---
# Atlas Dedup System
> Atlas is a fork of Rojo. Many internal identifiers, API endpoints, and struct names still reference the original "Rojo" naming.
## Quality Standard
This is a production-grade synchronization system. The filesystem must represent a 1:1 copy of the Studio place. The core invariant:
**Round-trip identity:** Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a bit-identical instance tree -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.
**CLI syncback parity:** Plugin-based two-way sync must produce exactly the same filesystem output as `atlas syncback`. Byte-for-byte identical files, identical directory structures, identical naming. Any divergence is a bug until proven otherwise.
## Architecture Overview
The dedup system replaced the old "skip duplicates" blacklist with three interlocking subsystems:
```
Matching Algorithm → Dedup System → Ref Path System
(pairs instances) (assigns FS names) (paths use final FS names)
```
**Execution order per sync operation:** Matching runs first (pairs instances), dedup runs second (assigns filesystem names), ref paths are computed last (using final filesystem names).
### Three Implementations of Matching
The matching algorithm exists in 3 parallel implementations that must produce identical pairings:
1. **Rust syncback** -- `src/syncback/matching.rs` (operates on `WeakDom` instances)
2. **Rust forward sync** -- `src/snapshot/matching.rs` (operates on `InstanceSnapshot` vs `RojoTree`)
3. **Lua plugin** -- `plugin/src/Reconciler/matching.lua` (operates on Studio instances vs virtual instances)
All three use the same recursive change-count scoring + greedy assignment algorithm. Changes to the algorithm must be applied to all three.
All three now have **`MatchingSession` caching** to avoid redundant recursive scoring:
- **Rust:** `cost_cache: RefCell<HashMap<(Ref, Ref), u32>>` -- caches `compute_change_count` results
- **Lua:** `matchCache` (parent virtual ID → Studio instance → MatchResult), `costCache` (virtual ID → Studio instance → cost)
### Key Files Map
| File | Subsystem | Purpose |
|------|-----------|---------|
| `src/syncback/file_names.rs` | Dedup | `slugify_name`, `deduplicate_name_with_ext`, `name_for_inst`, `name_needs_slugify` |
| `src/syncback/dedup_suffix.rs` | Dedup | `compute_cleanup_action`, `DedupCleanupAction`, `parse_dedup_suffix`, `build_dedup_name` |
| `src/syncback/matching.rs` | Matching | Syncback matching: `match_children` (WeakDom) |
| `src/snapshot/matching.rs` | Matching | Forward sync matching: `match_forward` (Snapshot vs Tree) |
| `plugin/src/Reconciler/matching.lua` | Matching | Lua matching: `Matching.matchChildren` (Virtual vs Studio) |
| `src/rojo_ref.rs` | Ref Paths | `ref_target_path_from_tree`, `compute_relative_ref_path`, `resolve_ref_path_to_absolute`, `RefPathIndex`, `Rojo_Ref_*` constants |
| `src/snapshot/tree.rs` | Ref Paths | `get_instance_by_path`, `filesystem_name_for` |
| `src/syncback/ref_properties.rs` | Ref Paths | `collect_referents`, `tentative_fs_path`, `RefLinks` |
| `src/variant_eq.rs` | Matching | `variant_eq` property comparison (Rust) |
| `plugin/src/Reconciler/trueEquals.lua` | Matching | `trueEquals` property comparison (Lua) |
| `src/change_processor.rs` | Integration | Dedup cleanup on deletion, VFS suppression, ref path updates |
| `src/web/api.rs` | Integration | `added_paths`, `syncback_updated_properties`, ref-to-attribute conversion |
| `plugin/src/ChangeBatcher/encodeHelpers.lua` | Encoding | Shared property filtering/encoding helpers (`encodeAttributes`, `encodeTags`, `forEachEncodableProperty`). Used by `encodeInstance.lua` and `encodeService.lua`. |
---
## Subsystem 1: Slugification and Dedup (`file_names.rs`)
### slugify_name
Pure, stateless. Replaces forbidden filesystem characters with `_`.
**Constants:**
```rust
const FORBIDDEN_CHARS: [char; 9] = ['<', '>', ':', '"', '/', '|', '?', '*', '\\'];
const SLUGIFY_CHARS: [char; 10] = ['<', '>', ':', '"', '/', '|', '?', '*', '\\', '~'];
const DANGEROUS_SUFFIXES: [&str; 8] = [".server", ".client", ".plugin", ".local", ".legacy", ".meta", ".model", ".project"];
const INVALID_WINDOWS_NAMES: [&str; 22] = ["CON", "PRN", "AUX", "NUL", "COM1".."COM9", "LPT1".."LPT9"];
```
**Behavior:**
- Replaces each `SLUGIFY_CHARS` char with `_` (including `~` to prevent natural names from looking like dedup suffixes)
- Strips leading/trailing spaces and dots
- Neutralizes Windows reserved names (`CON` → `CON_`)
- Neutralizes dangerous suffixes (`.server` etc.) that would be misinterpreted as middleware extensions
**Key invariant:** `slugify_name` is idempotent. Applying it twice produces the same result.
### name_needs_slugify
Returns `true` if the name requires slugification. Checks for: `SLUGIFY_CHARS`, control characters, leading/trailing spaces, trailing dots, dangerous suffixes (`.server` etc.), Windows reserved names (`CON` etc.), and empty string. Used to decide whether metadata `name` field is needed.
### deduplicate_name_with_ext
Stateful collision resolver. Takes a base slug and extension, checks against `taken_names` set (case-insensitive).
```rust
pub fn deduplicate_name_with_ext(
base: &str,
extension: Option<&str>,
taken_names: &HashSet<String>,
) -> (String, String)
// Returns: (deduped_slug, full_fs_name)
// "Foo" + ".server.luau" with collision → ("Foo~2", "Foo~2.server.luau")
```
**Key invariant:** Suffix goes on the stem BEFORE the compound extension: `Foo~2.server.luau`, NOT `Foo.server~2.luau`.
### Dedup Key Semantics
The dedup key is the **full filesystem name component** (slug + extension), lowercased. This means:
- `Folder "Foo"` (key: `"foo"`) and `Script "Foo"` (key: `"foo.server.luau"`) do NOT collide
- `Script "Foo" (server)` and `Script "Foo" (server)` both key `"foo.server.luau"` → collision, dedup applied
- `Folder "Foo"` and init-style `Script "Foo"` both key `"foo"` (both directories) → collision, dedup applied
### name_for_inst
The primary entry point for generating filesystem names during syncback:
```rust
pub fn name_for_inst<'a>(
middleware: Middleware,
new_inst: &'a Instance,
old_inst: Option<InstanceWithMeta<'a>>,
taken_names: &HashSet<String>,
) -> anyhow::Result<(Cow<'a, str>, bool, String)>
// Returns: (filename, needs_meta_name, dedup_key)
```
- In incremental mode (`old_inst` is `Some`): preserves existing filesystem path
- In clean mode: slugifies name, adds middleware extension, deduplicates against `taken_names`
- Returns `needs_meta_name = true` when slug differs from real name (including dedup suffix)
### parse_dedup_suffix (in `dedup_suffix.rs`)
Parses `~N` suffix from a stem: `"Foo~3"` → `Some(("Foo", 3))`, `"Foo"` → `None`. Returns `None` for `~0` (invalid) and `~abc` (non-numeric).
### Metadata `name` Field
When the filesystem name differs from the instance name (due to slugification or dedup), the real name is stored in `.meta.json5` or `.model.json5` `name` field. This is the authoritative instance name during forward sync.
```
Instance "Hey/Bro" → Hey_Bro.luau + meta: {"name": "Hey/Bro"}
Instance "Folder" → Folder/ (no meta needed, name == slug)
Instance "Folder" → Folder~2/ + meta: {"name": "Folder"}
```
---
## Subsystem 2: Dedup Suffix Cleanup (`dedup_suffix.rs`)
When instances are deleted, the remaining dedup group may need suffix adjustments.
### DedupCleanupAction
```rust
pub enum DedupCleanupAction {
None,
RemoveSuffix { from: PathBuf, to: PathBuf },
PromoteLowest { from: PathBuf, to: PathBuf },
}
```
### Three Cleanup Rules
1. **Gap-tolerant deletion:** Deleting `~2` from `{Foo, Foo~2, Foo~3}` leaves `{Foo, Foo~3}`. No renumbering. Gaps are tolerated.
2. **Base-name promotion:** Deleting `Foo` (base) from `{Foo, Foo~2, Foo~3}` promotes `Foo~2` to `Foo`. `Foo~3` stays. The lowest-numbered suffix becomes the new base.
3. **Group-to-1 cleanup:** Reducing the group to 1 member removes the suffix entirely. Deleting from `{Foo, Foo~2}` leaves one instance → renamed to clean `Foo`.
### compute_cleanup_action
```rust
pub fn compute_cleanup_action(
base_stem: &str,
extension: Option<&str>,
remaining_stems: &[String],
deleted_was_base: bool,
parent_dir: &Path,
) -> DedupCleanupAction
```
**Critical integration detail:** The `remaining_stems` input must exclude co-removed siblings (instances being deleted in the same batch). The `change_processor.rs` builds a `removed_set: HashSet<Ref>` and skips those refs during sibling enumeration.
---
## Subsystem 3: Matching Algorithm
### Entry Points
**Rust syncback:**
```rust
pub fn match_children(
new_children: &[Ref], old_children: &[Ref],
new_dom: &WeakDom, old_dom: &WeakDom,
new_hashes: Option<&HashMap<Ref, Hash>>,
old_hashes: Option<&HashMap<Ref, Hash>>,
) -> MatchResult
```
**Rust forward sync:**
```rust
pub fn match_forward(
snapshot_children: Vec<InstanceSnapshot>,
tree_children: &[Ref],
tree: &RojoTree,
) -> ForwardMatchResult
```
**Lua plugin:**
```lua
function Matching.matchChildren(
virtualChildren: { string }, -- array of virtual instance IDs
studioChildren: { Instance }, -- array of Studio Instance objects
virtualInstances: { [string]: any } -- map of virtual ID → virtual instance data
) -> { matched, unmatchedVirtual, unmatchedStudio }
```
### Algorithm (All Three Implementations)
**Fast-path: (Name, ClassName) Grouping.** Group children by `(Name, ClassName)`. 1:1 groups get instant-matched with zero scoring.
**Ambiguous Groups: Recursive Scoring + Greedy Assignment.** For groups with multiple candidates on at least one side:
```
for each (A, B) in cross-product:
cost = computeChangeCount(A, B, bestSoFar)
if cost < bestSoFar: bestSoFar = cost
stable sort by cost ascending (ties broken by child order)
greedy assign: pick lowest-cost pair, remove both, repeat
```
### computeChangeCount (Recursive)
`computeChangeCount` and `matchChildren` are **mutually recursive**:
```
computeChangeCount(A, B, bestSoFar):
cost = countOwnDiffs(A, B) // properties + tags + attributes
if cost >= bestSoFar: return cost // early exit
childResult = matchChildren(A.children, B.children) // RECURSE
for matched pair: cost += computeChangeCount(childA, childB, bestSoFar - cost)
cost += count(unmatched) * UNMATCHED_PENALTY
return cost
```
### Constants
```rust
const UNMATCHED_PENALTY: u32 = 10_000; // Unmatched child = categorically expensive
const MAX_SCORING_DEPTH: u32 = 3; // Depth limit prevents O(n^k) explosion
```
### countOwnDiffs
Counts property-level changes needed to turn A into B:
- Each differing property = +1 (using `variant_eq` / `trueEquals`)
- Each tag in symmetric difference = +1
- Each differing attribute = +1
- Name and ClassName NOT counted (grouping guarantees match)
### Hash Fast-Path (Syncback Only)
Syncback matching has precomputed blake3 subtree hashes. If `new_hash == old_hash`, return cost 0 immediately (skips property comparison and recursion).
### Sort Stability
- **Rust:** `slice::sort_by` is natively stable. Ties preserve child order.
- **Lua:** `table.sort` is NOT stable. Comparator breaks ties by insertion index: `if a.cost == b.cost then return a.idx < b.idx end`.
### Property Comparison
**Rust (`variant_eq`):** Type check first; float comparison uses `approx_eq!` with epsilon. Special handling for CFrame, Color3, Vector3, ColorSequence, NumberSequence, Attributes (sorted), Tags (sorted).
**Lua (`trueEquals`):** Recursive deep equality. Treats `nil` and null ref (`{Ref = "000...0"}`) as equal. Floats use epsilon 0.0001 (absolute + relative). CFrame/Vector3 component-wise. Color3 via RGB ints. EnumItem via `.Value`.
### Session Caching
All three implementations wrap scoring in a `MatchingSession` that caches results:
- **Rust:** `compute_change_count` checks `cost_cache` before scoring. If `(ref_a, ref_b)` was scored before, returns cached value.
- **Lua:** `computePairCost` checks `costCache`; `matchChildren` checks `matchCache`. Avoids redundant work when hydrate re-calls matching on the same children.
The Lua matching file uses `--!native` and `--!optimize 2` file-level directives for native codegen performance. `cacheVirtual()` filters children to only those existing in `virtualInstances`, preventing crashes on incomplete virtual instance trees.
### Known Limitation: Greedy vs Optimal
The greedy algorithm is NOT globally optimal. Edge cases exist where a locally-best pick forces a globally-worse assignment. The Hungarian algorithm (O(N^3)) would give optimal results, but ambiguous groups are typically small (2-10 instances). Accepted for current release.
---
## Subsystem 4: Ref Path System
### Path Format
Ref paths use **full filesystem names** including extensions and dedup suffixes:
```
Folder "Foo" → segment "Foo"
ModuleScript "Foo" → segment "Foo.luau"
Script "Foo" (server) → segment "Foo.server.luau"
Dedup'd ModuleScript → segment "Foo~2.luau"
Init-style Script → segment "Foo" (directory)
```
Absolute paths are slash-separated from DataModel's children downward: `"Workspace/TestModel/Part1.model.json5"`.
### Luau-Style Relative Ref Paths
Syncback now writes **relative** ref paths using Luau require-by-string semantics via `compute_relative_ref_path()`:
- `@self` -- target is the source instance itself
- `@self/Child.luau` -- target is a descendant of source
- `./Sibling.luau` -- target is a same-parent sibling
- `../Uncle.luau` -- target is a parent's sibling
- `@game/Service/Path` -- cross-service reference
Forward sync resolves both absolute and relative formats via `resolve_ref_path_to_absolute()` + `get_instance_by_path()`.
**Placeholder system:** During syncback, before dedup suffixes are finalized, ref paths are written as `__ROJO_REF_<source>_TO_<target>__` placeholders. After the syncback walk assigns final filesystem names, `fix_ref_paths()` substitutes placeholders with actual relative paths.
### Key Functions
**Building paths:**
```rust
// From RojoTree (two-way sync, change_processor):
pub fn ref_target_path_from_tree(tree: &RojoTree, target_ref: Ref) -> String
// Uses filesystem_name_for() for each ancestor
// From WeakDom (CLI syncback, legacy instance-name paths):
pub fn ref_target_path(dom: &WeakDom, target_ref: Ref) -> String
// Uses dom.ancestors_of() with escape_ref_path_segment (instance names, NOT filesystem names)
// Tentative path (before dedup suffixes assigned):
fn tentative_fs_path(dom: &WeakDom, target_ref: Ref) -> String
// Uses slugified name + extension based on class
```
**Resolving paths:**
```rust
// In tree.rs:
pub fn get_instance_by_path(&self, path: &str) -> Option<Ref>
// Splits by "/", matches segments case-insensitive against filesystem_name_for()
// Fallback: matches against instance name (case-insensitive)
pub fn filesystem_name_for(&self, id: Ref) -> String
// Returns filename from instigating_source, falls back to instance name
```
**Computing filesystem names for refs:**
```rust
// In ref_properties.rs:
fn tentative_fs_name(inst: &Instance) -> String
// Slugified name + extension based on class and children count
// Used before dedup suffixes are assigned
pub fn collect_referents(
dom: &WeakDom,
pre_prune_paths: &HashMap<Ref, String>,
final_paths: Option<&HashMap<Ref, String>>,
) -> RefLinks
// Uses final_paths (dedup-aware) when available, otherwise tentative_fs_path
```
### Attribute Format
Ref properties are stored as `Rojo_Ref_*` attributes in meta/model files:
```json5
{
"attributes": {
"Rojo_Ref_PrimaryPart": "Workspace/TestModel/Part1.model.json5"
}
}
```
Constants in `rojo_ref.rs`:
```rust
pub const REF_PATH_ATTRIBUTE_PREFIX: &str = "Rojo_Ref_";
pub const REF_POINTER_ATTRIBUTE_PREFIX: &str = "Rojo_Target_";
pub const REF_ID_ATTRIBUTE_NAME: &str = "Rojo_Id";
```
### RefPathIndex
Tracks which filesystem files contain which ref paths, enabling efficient updates on rename:
```rust
pub struct RefPathIndex {
paths_to_files: HashMap<String, HashSet<PathBuf>>,
}
```
Used by `update_ref_paths_after_rename` in `change_processor.rs` to find and update all files containing a given ref path prefix when a dedup rename occurs.
---
## Integration Point 1: change_processor.rs (Dedup Cleanup)
### Deletion Flow
When instances are removed via two-way sync, `handle_tree_event` runs dedup cleanup **before** `apply_patch_set` removes instances from the tree:
1. Build `removed_set: HashSet<Ref>` from ALL removals in the batch
2. For each removed instance:
a. Get parent ref and filesystem name
b. Parse dedup suffix from stem
c. Enumerate remaining siblings (skipping co-removed via `removed_set`)
d. Group by extension (prevents cross-middleware false matches)
e. Call `compute_cleanup_action`
3. Execute cleanup action (rename file, suppress VFS, update ref paths)
### VFS Suppression
Dedup renames must suppress VFS events to prevent feedback loops:
- `suppress_path_any(&old_path)` -- old path may trigger REMOVE (Linux/Windows) or stale CREATE (macOS)
- `suppress_path(&new_path)` -- new path triggers Create/Write
- `unsuppress_path(&new_path)` -- rollback if rename fails
**Critical:** New instance additions must NOT be suppressed. The VFS watcher needs to pick up new files for forward sync.
### Ref Path Updates After Dedup Rename
After successful dedup rename (e.g., `Foo~2` → `Foo`):
1. Compute old and new ref path segments
2. Call `update_ref_paths_after_rename(&old_prefix, &new_prefix)`
3. This function uses `RefPathIndex` to find affected files
4. Updates `Rojo_Ref_*` attribute values in each affected file
5. Updates the index: both path keys and filesystem paths
6. Suppresses VFS events for each file write
### Ref Path Updates After User-Initiated Rename
User-initiated renames (via the plugin's two-way sync) also update `Rojo_Ref_*` paths. The flow:
1. `old_ref_path` is captured BEFORE the rename (line ~1268, `ref_target_path_from_tree`)
2. Rename handler sets `new_ref_segment` to the new filesystem name (directory rename line ~1350, file rename line ~1512)
3. After rename handling, `update_ref_paths_after_rename(old_ref_path, new_ref_path)` is called (line ~1812) to update all meta/model files containing refs to the renamed instance
This ensures both dedup-triggered and user-initiated renames maintain ref path integrity.
### Metadata Consistency
After dedup rename, the survivor's metadata must be updated:
- `InstigatingSource::Path` updated to new filesystem path
- Path indexes updated in-memory
- Suppression counters kept consistent with updated path
---
## Integration Point 2: api.rs (Two-Way Sync Write)
### added_paths (Same-Batch Ref Targets)
When `/api/write` adds instances and sets Ref properties targeting them in the same batch, `added_paths` pre-computes filesystem paths:
```rust
let added_paths: HashMap<Ref, String> = request.added.iter()
.filter_map(|(guid, added)| {
let parent_path = ref_target_path_from_tree(&tree, added.parent?);
let fs_name = added_instance_fs_segment(added); // slug + extension
Some((*guid, format!("{}/{}", parent_path, fs_name)))
})
.collect();
```
`added_instance_fs_segment` mirrors `tentative_fs_name` logic but operates on `AddedInstance` fields (class_name, children, properties).
### Ref-to-Attribute Conversion
`syncback_updated_properties` converts `Variant::Ref` properties to `Rojo_Ref_*` attributes:
1. Extract Ref properties from `changed_properties` BEFORE `filter_properties_for_meta`
2. For each Ref property:
- **Nil ref:** Mark `Rojo_Ref_PropertyName` for removal
- **Valid target in tree:** Compute path via `ref_target_path_from_tree`, add to `ref_attributes`
- **Target in `added_paths`:** Use pre-computed path
- **Target not found:** Log warning and skip
3. Merge `ref_attributes` into attributes map
4. Pass `remove_attributes` to `merge_or_build_meta`
### merge_or_build_meta Attribute Lifecycle
```
1. Remove stale attributes (remove_attributes list)
2. Merge new attributes (including Rojo_Ref_* from ref properties)
3. Remove wins over add if same key in both lists? No -- adds happen AFTER removes, so add wins
```
### syncback_added_instance
When adding instances, dedup is handled via `sibling_slugs: &mut HashSet<String>`:
- Tracks filesystem names claimed in the current batch
- Prevents duplicate slug assignment when multiple instances added simultaneously
- Passed to `syncback_instance_to_path_with_stats` for dedup suffix assignment
---
## Key Invariants
1. **Round-trip identity:** syncback → build → compare = bit-identical trees
2. **CLI syncback parity:** two-way sync produces byte-identical filesystem output to CLI syncback
3. **Determinism:** same input always produces same filesystem output (sorted child ordering, deterministic suffix assignment)
4. **InstigatingSource consistency:** dedup'd paths reflected in `InstigatingSource::Path`
5. **Compound extensions:** suffix before compound extension (`Foo~2.server.luau`, NOT `Foo.server~2.luau`)
6. **Meta pairing:** `Foo~2.server.luau` pairs with `Foo~2.meta.json5` (strip script suffix, keep dedup suffix)
7. **Dedup key = full FS name:** collisions checked using complete filesystem name (slug + extension), lowercased
8. **Co-removed exclusion:** batch removal skips co-removed siblings when computing cleanup actions
---
## Known Limitations (Accepted)
### ProjectNode Duplicate-Named Children
Duplicate-named children under ProjectNode are currently skipped during syncback with warnings (`src/snapshot_middleware/project.rs`). Project keys are unique by JSON definition, but filesystem children under project nodes could collide. Accepted for current release.
### Greedy Matching Not Globally Optimal
The greedy assignment may produce suboptimal pairing in rare ambiguous groups. A globally optimal algorithm (Hungarian, O(N^3)) would be better but is deferred since ambiguous groups are typically small.
### ~~Matching Cache Not Implemented~~ (Resolved)
`MatchingSession` caching is now implemented in all three matching implementations. See "Session Caching" in Subsystem 3. The per-parent invalidation cache (mapping `parent_ref → (children_list_hash, match_assignments)`) remains deferred.
### ~~Stale Ref Paths After Rename~~ (Resolved)
Both dedup-triggered renames and user-initiated renames now update `Rojo_Ref_*` paths automatically. See "Ref Path Updates" in the Integration Point 1 section for details.
---
## Deferred Refactors
- **Shared filesystem-segment derivation:** Consolidate tentative filesystem name computation into one shared helper used by `rojo_ref.rs`, `tree.rs`, `api.rs`, and `ref_properties.rs`. Currently duplicated across 4+ call sites.
- **Optimal assignment algorithm:** Replace greedy with Hungarian for large ambiguous groups.
- **Per-parent matching invalidation cache:** `MatchingSession` cost/match caching is implemented, but the per-parent cache with `children_list_hash` invalidation (avoids re-running matching when children haven't changed) remains deferred.
- **Duplicated name→index grouping:** All three matching implementations repeat the same HashMap pattern. Could be factored but types differ.
---
## Testing Strategy
### Test Locations
- Rust unit tests: `#[cfg(test)]` blocks in source files
- Rust integration tests: `tests/tests/two_way_sync.rs`, `tests/tests/syncback_roundtrip.rs`
- Test fixtures: `rojo-test/syncback-tests/`, `rojo-test/build-tests/`
- Lua spec tests: `.spec.lua` files
- Snapshot tests: `cargo insta review`
### Critical Test Scenarios
**Dedup:** Same-named Folders, Folders with different children, Folder + Script same name (no dedup), Folder + init-style Script (dedup), natural tilde name, third duplicate added, suffix cleanup (group-to-1), gap tolerance, base-name promotion, idempotency, roundtrip.
**Ref paths:** Basic ref survives syncback, ref to dedup'd instance, ref survives cleanup rename, ref survives base-name promotion, ref survives group-to-1, deeply nested through dedup layers, cross-file ref update, tilde-named instance ref, two-way sync ref update, forward sync manual edit.
**Matching:** ClassName discriminator, property similarity scoring, matching stability across runs, preserves existing pairings, fast path verification, recursive depth limit, hash fast-path short circuit.
**Dedup collision:** Folder + ModuleScript (no collision, different keys), Server + Client script (no collision), ModuleScript + ModuleScript (collision), slug collision ("Foo?" + "Foo*"), case-insensitive collision ("Foo" + "foo").
**Cross-system:** Full roundtrip with duplicates, two-way sync add/delete, forward sync edit dedup'd file, project file + filesystem duplicates, batch-remove 2 of 3 same-named siblings, same-batch add + ref.
---
## Historical Context
The dedup system was built across these plans (chronological):
1. **nuke_path_encoding** -- Replaced `%NAME%` encoding with slugify + dedup + metadata `name` field foundation
2. **two-way_sync_refs** -- Added Ref property support to two-way sync with `Rojo_Ref_*` path-based attributes
3. **atlas_fork_isolation** -- Renamed binary/plugin for coexistence (preserved `Rojo_Ref_*` attribute names for backward compatibility)
4. **ambiguous_path_handling** -- Core feature: 3-pass matching algorithm, filesystem-collision dedup, filesystem-name ref paths, dedup suffix cleanup
5. **ref_path_and_dedup_integration** -- Migrated `collect_referents` to filesystem-name paths, integrated cleanup rules into change_processor
6. **rewrite_matching_algorithm** -- Replaced multi-pass with recursive change-count scoring + greedy assignment
7. **audit_ambiguous_path_handling** -- First audit: fixed forward sync re-matching, case sensitivity, ProjectNode paths, Lua require crash, dead code, ref format mismatch
8. **ambiguous_paths_audit_fixes** -- Second audit: recursive matching, `added_paths` format, dedup-aware ref collection, full-FS-name cleanup grouping, shared `trueEquals`
9. **audit_ambiguous_paths_followup** -- Third audit: metadata drift after cleanup rename, co-removed sibling exclusion, `added_paths` extension/suffix, attribute-scoped final paths
10. **audit_ref_two-way_sync** -- Ref audit: round-trip fidelity, path computation correctness, meta lifecycle, VFS suppression, wire format
11. **matching_session_caching** -- Session-scoped caching for all three matching implementations, `--!native`/`--!optimize 2` for Lua, valid children tracking
12. **luau-style_ref_paths** -- Relative ref paths (`@self`, `./`, `../`, `@game/`), source-aware placeholder system
13. **live_syncback** -- One-click full sync from Studio via `POST /api/syncback`, service property encoding, server restart

373
.cursor/rules/atlas-plugin.mdc Normal file
View file

@ -0,0 +1,373 @@
---
globs: plugin/**
alwaysApply: false
---
# Rojo Studio Plugin
Built with Roact (React-like), handles WebSocket sync with server, patch application to Studio, and UI.
**Entry:** `plugin/src/init.server.lua` | **Protocol:** WebSocket v6 | **Must match server MAJOR.MINOR**
## Architecture
### Key Components
**1. UI Layer** (`App/`)
- **init.lua** - Root component, manages connection state, auto-reconnect, sync reminders, playtest server auto-connect. Also renders `SyncbackConfirm` floating dialog (StudioPluginGui with `isEphemeral=true`, not a StatusPage file). Contains `performSyncback()` and `SYNCBACK_SERVICES` constant.
- **Theme.lua** - Studio-matched color palette (auto light/dark)
- **Page.lua** - Page wrapper component
- **Components/** - Reusable UI components
- `PatchVisualizer/` - Tree view of changes with property diffs (`init.lua`, `DomLabel.lua`, `DisplayValue.lua`, `ChangeList.lua`)
- `StringDiffVisualizer/` - Side-by-side script diffs (`init.lua`, `StringDiff.lua`)
- `TableDiffVisualizer/` - Attributes and Tags diffs (`init.lua`, `Dictionary.lua`, `Array.lua`)
- `Notifications/` - Toast notifications (`init.lua`, `Notification.lua`, `FullscreenNotification.lua`)
- `Studio/` - Studio API wrappers (`StudioPluginAction`, `StudioPluginContext`, `StudioPluginGui`, `StudioToggleButton`, `StudioToolbarContext`, `StudioToolbar`)
- General: `Header`, `TextInput`, `TextButton`, `Checkbox`, `Dropdown`, `Tag`, `IconButton`, `Tooltip`, `TouchRipple`, `VirtualScroller`, `ScrollingFrame`, `Spinner`, `BorderedContainer`, `EditableImage`, `SlicedImage`, `ClassIcon`
- **StatusPages/** - Main UI pages
- `NotConnected.lua` - Connection form (host/port input), "Syncback" button (Danger style)
- `Connecting.lua` - Loading screen
- `Confirming.lua` - Patch confirmation dialog with push/pull/ignore per-node selections
- `Connected.lua` - Status display with patch history
- `Error.lua` - Error display
- `Settings/` - Settings UI (`init.lua`, `Setting.lua`)
- Uses Roact for declarative UI
**2. Communication** (`ServeSession.lua`, `ApiContext.lua`)
- WebSocket connection to server via `/api/socket/:cursor`
- Handles patches, errors, reconnection
- Protocol version checking
- Confirmation flow with push/pull/ignore
- One-shot mode support
**3. Reconciler** (`Reconciler/`)
- **applyPatch.lua** - Applies server patches to Studio tree
- **reify.lua** - Creates instances from snapshots (parent last)
- **hydrate.lua** - Updates InstanceMap with existing Studio instances
- **diff.lua** - Computes patch between virtual and Studio state (for two-way sync)
- **matching.lua** - Recursive change-count scoring algorithm for pairing virtual instances with Studio instances
- **trueEquals.lua** - Shared value equality (fuzzy floats, Color3, CFrame, Vector3; NaN handling; nil/null-ref equivalence)
- **decodeValue.lua** - Converts encoded values to Roblox types
- **setProperty.lua** - Sets properties safely (handles unscriptable)
- **getProperty.lua** - Gets properties with fallbacks
- **Error.lua** - Reconciler error handling
**4. ChangeBatcher** (`ChangeBatcher/`)
- **init.lua** - Main batcher: accumulates changes every 200ms, pause/resume, deferred Ref resolution
- **createPatchSet.lua** - Converts pending changes to PatchSet
- **encodePatchUpdate.lua** - Encodes property updates including Ref properties via InstanceMap
- **encodeInstance.lua** - Full instance encoding (for additions/syncback); uses `encodeHelpers` for property iteration; includes duplicate detection helpers (`findDuplicateNames`, `hasDuplicateSiblings`, `isPathUnique`)
- **encodeHelpers.lua** - Shared property encoding helpers (`encodeAttributes`, `encodeTags`, `forEachEncodableProperty`). Used by `encodeInstance.lua` and `encodeService.lua`. Defines `SKIP_PROPERTIES` (Parent, Name, Archivable).
- **encodeService.lua** - Encodes service-level properties/attributes/refs for live syncback; uses ObjectValue carriers for Ref properties
- **encodeProperty.lua** - Single property value encoding via RbxDom
- **propertyFilter.lua** - Property type filtering (only `UniqueId` blocked; `Ref` is allowed)
**5. State Management**
- **Settings.lua** - Persistent plugin settings (prefixed `Atlas_` in plugin store)
- **InstanceMap.lua** - Bidirectional Server ID <-> Studio Instance mapping with change detection
- **PatchSet.lua** / **PatchTree.lua** - Patch representation and visualization tree (PatchTree.build accepts optional `gitMetadata` for smart default selections)
- **Types.lua** - Type definitions and runtime validation
- **ChangeMetadata.lua** - Change metadata tracking
**6. Utilities**
- **Timer.lua** - Performance timing
- **DiffUtil.lua** - Diff utilities
- **Version.lua** - Version parsing and comparison
- **Dictionary.lua** - Dictionary utilities
- **Assets.lua** - Asset references
- **SHA1.luau** - SHA1 hashing (used for git blob format hash comparison in PatchTree default selections)
- **XXH32.luau** - XXH32 hash function
- **soundPlayer.lua** - Sound effects
- **preloadAssets.lua** - Asset preloading
- **ignorePlaceIds.lua** - Place ID ignore list
- **timeUtil.lua** - Time formatting
- **strict.lua** - Strict module wrapper
- **invariant.lua** - Assertion utilities
**7. Testing**
- Unit tests: `.spec.lua` files (TestEZ) throughout all modules
- Integration tests: `integration/` directory (`confirmBehavior`, `oneShotSync`, `syncflow`, `twoWaySync`)
- Stress tests: `.stress.spec.lua` files (applyPatch, diff, reify, encodeInstance, PatchSet)
- Chaos tests: `chaos/` directory (`stressPatterns.spec.lua`)
- Test utilities: `testUtils/` (`MockApiContext`, `PatchGenerator`, `LargeTreeGenerator`)
- Test runner: `runTests.lua`
## Build Structure (`plugin.project.json`)
The plugin is built from `plugin.project.json` at the repo root:
```json
{
"name": "Rojo",
"tree": {
"$className": "Folder",
"Plugin": { "$path": "plugin/src" },
"Packages": {
"$path": "plugin/Packages",
"Log": { "$path": "plugin/log" },
"Http": { "$path": "plugin/http" },
"Fmt": { "$path": "plugin/fmt" },
"RbxDom": { "$path": "plugin/rbx_dom_lua" }
},
"Version": { "$path": "plugin/Version.txt" }
}
}
```
Vendored packages live under `plugin/Packages/`. Custom modules (`Log`, `Http`, `Fmt`, `RbxDom`) are injected into the Packages tree from their own directories at build time.
## Key Configuration (`Config.lua`)
```lua
{
isDevBuild = false, -- Detected from ROJO_DEV_BUILD instance
codename = "Epiphany",
version = { major, minor, patch, metadata }, -- Parsed from Version.txt
expectedServerVersionString = "7.7 or newer",
protocolVersion = 6, -- MUST match server
defaultHost = "localhost",
defaultPort = "34873",
ignoredClassNames = { -- Auto-created instances to skip
TouchTransmitter = true,
Camera = true,
},
}
```
**When changing protocol:** Bump `protocolVersion` in both `plugin/src/Config.lua` AND `src/web/interface.rs`.
## Settings
Persistent settings stored with `Atlas_` prefix in Studio plugin store:
```lua
{
openScriptsExternally = false,
twoWaySync = true,
oneShotSync = true,
autoReconnect = false,
showNotifications = false,
enableSyncFallback = false,
syncReminderMode = "Notify", -- "None" | "Notify" | "Fullscreen"
syncReminderPolling = true,
checkForUpdates = false,
checkForPrereleases = false,
autoConnectPlaytestServer = false,
confirmationBehavior = "Always", -- "Never" | "Initial" | "Always" | "Large Changes" | "Unlisted PlaceId"
largeChangesConfirmationThreshold = 5,
playSounds = true,
typecheckingEnabled = false,
logLevel = "Info",
timingLogsEnabled = false,
priorEndpoints = {},
}
```
## WebSocket Protocol (v6)
1. Connect to `ws://host:port/api/socket/:cursor`
2. Receive `ServerInfo` handshake (protocol version, project name, session ID)
3. Plugin sends: `{type = "Subscribe"}`
4. Server sends patches on filesystem changes (MessagePack encoded)
**Patch structure:**
```lua
{
type = "ApplyPatch",
sessionId = "uuid",
patch = {
removed = {"id-1", "id-2"},
added = {
["id-3"] = {
id = "id-3", className = "Folder", name = "New",
properties = {}, metadata = { ignoreUnknownInstances = false },
children = {"child-1"},
},
},
updated = {
{ id = "id-4", changedName = "Renamed", changedProperties = { Source = "code" } },
},
}
}
```
## ServeSession (`ServeSession.lua`)
Manages the lifecycle of a connection to the server.
**Status flow:** `NotStarted` -> `Connecting` -> `Connected` -> `Disconnected`
**Key methods:**
- `new(options)` - Creates session with `ApiContext` and `twoWaySync` flag
- `start()` - Connects, computes initial patch, shows confirmation
- `stop()` - Disconnects and cleans up
- `__onWebSocketMessage()` - Handles incoming patches
- `__applyPatch()` - Applies patches via Reconciler
- `__confirmAndApplyInitialPatch()` - Handles user confirmation with push/pull/ignore
- `__computeInitialPatch()` - Computes catch-up patch
- `hookPrecommit()` / `hookPostcommit()` - Lifecycle hooks
**Confirmation flow:**
1. Receive patch (initial or streaming)
2. Check `confirmationBehavior` setting
3. If confirmation needed: pause ChangeBatcher, build PatchTree with git-based default selections (from `serverInfo.gitMetadata`), show Confirming page with PatchVisualizer
4. User reviews defaults and selects push/pull/ignore per node
5. Build `stageIds` list (all push-accepted + auto-selected pull-accepted), send with write request
6. Apply via Reconciler, resume ChangeBatcher
**One-shot mode (`oneShotSync: true`):**
- ChangeBatcher still collects changes (for diff visualization)
- Automatic outgoing writes are blocked
- Only explicit "pull" selections in confirmation dialog can write
- Disconnects after initial sync completes
## Reconciler (`Reconciler/`)
**applyPatch.lua** - Main patch application:
1. Remove instances in `removed`
2. Create instances in `added` via `reify`
3. Update instances in `updated` via `hydrate`
4. Handle deferred Ref properties
5. Collect errors, return unapplied patch
**reify.lua** - Create instances (parent last - Roblox best practice)
**hydrate.lua** - Update InstanceMap with existing instances (uses matching algorithm)
**diff.lua** - Compute patch between virtual instances and Studio state
**setProperty.lua** / **getProperty.lua** - Handle unscriptable properties
### Matching Algorithm (`matching.lua`)
Pairs incoming virtual instances with existing Studio instances during hydration.
**Entry point:** `Matching.matchChildren(virtualChildren, studioChildren, virtualInstances)`
**Algorithm:**
1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring.
2. **Ambiguous groups:** For groups with multiple candidates, compute pairwise `computePairCost()` (recursive, scores entire subtree). Greedy assignment sorted by cost ascending, child-order tiebreaker for stability.
3. **Leftovers:** Unmatched virtual = new instances, unmatched studio = deleted.
**Performance:** File-level `--!native` and `--!optimize 2` directives for native codegen. `MatchingSession` caches results across recursive calls:
```lua
type MatchingSession = {
matchCache: { [string]: { [Instance]: MatchResult } },
costCache: { [string]: { [Instance]: number } },
}
```
**Valid children tracking:** `cacheVirtual()` filters children to only those existing in `virtualInstances`, preventing crashes on incomplete virtual instance trees.
**Sort stability note:** `table.sort` is NOT stable in Luau. Comparator breaks ties by insertion index to ensure deterministic ordering.
**Value equality:** Shared `trueEquals.lua` module used by both `matching.lua` and `diff.lua` -- fuzzy equality for floats (epsilon 0.0001, absolute + relative), Color3 via RGB ints, CFrame/Vector3 component-wise, NaN handling, nil/null-ref equivalence, EnumItem/number interop.
## InstanceMap (`InstanceMap.lua`)
Bidirectional Server ID <-> Studio Instance mapping with integrated change detection:
```lua
InstanceMap = {
fromIds = {}, -- Server ID -> Studio Instance
fromInstances = {}, -- Studio Instance -> Server ID
pausedUpdateInstances = {}, -- Set of instances to ignore changes from
instancesToSignal = {}, -- Instance -> RBXScriptConnection(s)
onInstanceChanged = fn, -- Callback: (instance, propertyName) -> void
}
```
**Key methods:**
- `insert(id, instance)` - Add mapping and connect change signals
- `removeId(id)` / `removeInstance(instance)` - Remove mapping and disconnect signals
- `destroyInstance(instance)` - Parent to nil (preserves undo history)
- `fromId(id)` / `fromInstance(instance)` - Look up mapping
- `pauseInstance(instance)` / `unpauseAllInstances()` - Pause/resume change detection
**Signal connection (`__connectSignals`):**
- Regular instances: `instance.Changed` event (fires for any property)
- ValueBase instances: Explicit `GetPropertyChangedSignal` for Name, Value, Parent (ValueBase.Changed only fires for Value)
**Change guards (`__maybeFireInstanceChanged`):**
- Skipped if instance is in `pausedUpdateInstances`
- Skipped if `onInstanceChanged` callback is nil
- Skipped if `RunService:IsRunning()` (Play mode)
**Pause mechanism:** During incoming patch application, instances are paused to prevent their changes from being sent back to the server (feedback loop prevention). `unpauseAllInstances()` is called each frame in `ChangeBatcher:__cycle()`.
## Two-Way Sync
Watches Studio changes and sends them to the server in real-time. Enabled by default (`twoWaySync: true`).
**Data flow:** Studio property change -> InstanceMap signal -> ServeSession callback -> ChangeBatcher -> 200ms batch flush -> `POST /api/write` (MessagePack) -> Server writes to filesystem
**ChangeBatcher state:**
```lua
{
__accumulator = 0, -- Time since last flush
__pendingPropertyChanges = {}, -- Instance -> { PropertyName -> true }
__pendingUnresolvedRefs = {}, -- targetInstance -> [{sourceInstance, propertyName}]
__syncSourceOnly = false, -- Only sync Source property
__paused = false, -- Pause during confirmation
__renderSteppedConnection = conn, -- Frame loop connection
__instanceMap = instanceMap,
__onChangesFlushed = callback,
}
```
**Deferred Ref resolution:** When a Ref property targets an instance not yet in the InstanceMap, the property is deferred in `__pendingUnresolvedRefs`. When the target appears via `onInstanceInserted`, the property change is re-queued for the next batch.
**Guards:**
- `oneShotSync` setting blocks automatic outgoing writes
- `__paused` flag blocks flushes during confirmation dialog
- `RunService:IsRunning()` blocks during Play mode
## Live Syncback
One-click full sync from Studio to filesystem. Available from the **NotConnected page** -- does not require an active WebSocket session. Sends directly to the server via HTTP POST using the host/port fields from the connection form.
**`SYNCBACK_SERVICES` constant (14 services):** Lighting, MaterialService, ReplicatedFirst, ReplicatedStorage, ServerScriptService, ServerStorage, SoundService, StarterGui, StarterPack, StarterPlayer, Teams, TextChatService, VoiceChatService, Workspace.
**Flow:**
1. User clicks "Syncback" button on NotConnected page → `SyncbackConfirm` floating dialog appears
2. Dialog warns: "This will overwrite your project files with the current Studio state. This cannot be undone."
3. User confirms → `performSyncback()`:
a. Iterates `SYNCBACK_SERVICES`, calls `encodeService(service)` for each
b. Collects all children instances and ObjectValue Ref carriers
c. Serializes all children via `SerializationService:SerializeInstancesAsync()` into a single buffer
d. Destroys temporary ObjectValue carriers
e. POSTs to `http://host:port/api/syncback` (MessagePack): `{protocolVersion, serverVersion, placeId, data, services}`
4. Shows success/error notification
**`encodeService(service)`** (`ChangeBatcher/encodeService.lua`):
- Encodes service-level properties via `RbxDom.findClassDescriptor()`
- Skips: `Parent`, `Name`, `Archivable`, `Attributes`, `Tags` (handled separately), class-specific excludes (`Lighting.ClockTime`)
- Ref properties: creates temporary `ObjectValue` carriers (Name = property name, Value = target instance), stores 1-based index in `refs` map
- Returns `(chunk, refTargets)` where `chunk = {className, childCount, refTargetCount, properties?, refs?}`
## Dependencies
### Vendored Packages (`plugin/Packages/`)
Roact, Flipper, Promise, TestEZ, t, Highlighter, msgpack-luau, json5. **Don't edit vendored packages.**
### Custom Modules (injected via `plugin.project.json`)
| Module | Path | Purpose |
|--------|------|---------|
| Log | `plugin/log` | Structured logging with configurable levels |
| Http | `plugin/http` | HTTP client utilities |
| Fmt | `plugin/fmt` | String formatting with debug output |
| RbxDom | `plugin/rbx_dom_lua` | Reflection database, property descriptors, encoded value encode/decode |
## Common Issues
**Plugin not reloading:** Enable Studio setting "Reload plugins on file change"
**Connection fails:** Check server running, host/port match, protocol versions match
**Properties not syncing:** Verify scriptable in rbx_reflection_database
**Duplicates:** InstanceMap corruption - disconnect, restart Studio
**Two-way sync feedback loops:** Check pause mechanism is working; instances must be paused during reconciliation
## Debugging
Settings -> Log Level -> "Trace" for verbose output. `ROJO_DEV_BUILD` instance enables auto-tests via TestEZ.
**Timing:** Enable `timingLogsEnabled` setting to see `Timer.start`/`Timer.stop` measurements for Reconciler and other hot paths.

46
.cursor/rules/atlas-project.mdc
View file

@ -1,12 +1,5 @@
---
description: "Project file format specification: .project.json5 schema, file extensions, middleware types, init files, sync rules, and property reference."
globs:
- "**/*.project.json5"
- "**/*.project.json"
- "**/*.meta.json5"
- "**/*.meta.json"
- "**/*.model.json5"
- "**/*.model.json"
alwaysApply: true
---
# Atlas Project File Format
@ -198,7 +191,7 @@ Configure `syncbackRules` object for Roblox → filesystem sync:
| `ignoreProperties` | object | `{}` | Properties to skip per class: `{"ClassName": ["Prop1", "Prop2"]}` |
| `ignoreClasses` | string[] | `[]` | Instance classes to skip entirely |
| `syncCurrentCamera` | bool | `false` | Whether to sync `Workspace.CurrentCamera` |
| `syncUnscriptable` | bool | `true` | Whether to sync unscriptable properties |
| `syncUnscriptable` | bool | `false` | Whether to sync unscriptable properties |
| `ignoreReferents` | bool | `false` | Whether to skip Ref properties (e.g., `Model.PrimaryPart`) |
| `createIgnoreDirPaths` | bool | `false` | Whether `/**` in ignore paths also matches the directory itself |
| `ignoreHiddenServices` | bool | — | Override root-level `ignoreHiddenServices` for syncback |
@ -215,7 +208,7 @@ Configure `syncbackRules` object for Roblox → filesystem sync:
},
"ignoreClasses": ["Terrain", "Camera"],
"syncCurrentCamera": false,
"syncUnscriptable": true,
"syncUnscriptable": false,
"ignoreReferents": false,
"createIgnoreDirPaths": false,
"warnDuplicateNames": false
@ -254,7 +247,9 @@ Add metadata to file-based instances. Meta files use **camelCase keys** (no `$`
**Key differences from ProjectNode:** Meta uses `className`, `properties`, `attributes` (camelCase, no `$` prefix). Adjacent meta does NOT support `className`.
**`name` field (slugify support):** When an instance name contains forbidden filesystem characters (`: / ? * < > | " \`), syncback slugifies the filename (forbidden chars → `_`) and stores the real name in the `name` field. On forward sync, `name` overrides the filename-derived instance name. On collisions (e.g. `A/B` and `A_B` both slugify to `A_B`), `~1`, `~2` suffixes are appended and meta `name` fields distinguish them.
**`name` field (slugify support):** When an instance name contains forbidden filesystem characters (`: / ? * < > | " \`), syncback slugifies the filename (forbidden chars → `_`) and stores the real name in the `name` field. On forward sync, `name` overrides the filename-derived instance name. On collisions (e.g. `A/B` and `A_B` both slugify to `A_B`), `~2`, `~3` suffixes are appended and meta `name` fields distinguish them.
**`Rojo_Ref_*` attributes:** Meta files may also contain `Rojo_Ref_PropertyName` attributes written by syncback/two-way sync for Ref property linking. These are managed automatically and should not be hand-edited.
**Meta files CAN'T:** Set `$path`, have children.
@ -331,6 +326,33 @@ Same format as properties. Limitations: no Ref, no NumberSequence/ColorSequence,
## Ref Property Linking
Two systems for linking instance references, both supported:
### Path-Based: `Rojo_Ref_*` (preferred, written by syncback/two-way sync)
Uses filesystem-name paths (slugified, dedup'd) to reference target instances:
```json5
// In meta file for an instance with PrimaryPart pointing to Handle
{
"attributes": {
"Rojo_Ref_PrimaryPart": "Workspace/MyModel/Handle.model.json5"
}
}
```
**Attribute format:** `Rojo_Ref_<PropertyName> = "<path>"`
Syncback writes **Luau-style relative paths**:
- `@self` / `@self/Child.luau` -- target is same instance or a descendant
- `./Sibling.luau` -- target is a same-parent sibling
- `../Uncle.luau` -- target is a parent's sibling
- `@game/Service/Path` -- cross-service reference
Forward sync resolves both relative and absolute formats via `resolve_ref_path_to_absolute()` + `get_instance_by_path()`. Path segments use full filesystem names (slug + extension + dedup suffix), matched case-insensitive with instance name fallback.
### ID-Based: `Rojo_Target_*` + `$id` (legacy, still read by forward sync)
Link instances via `$id` and special attributes:
```json5
@ -347,6 +369,8 @@ Link instances via `$id` and special attributes:
**Attribute format:** `Rojo_Target_<PropertyName>`
**Priority:** When both `Rojo_Ref_*` and `Rojo_Target_*` exist for the same property, `Rojo_Ref_*` takes precedence.
## Property Types
`String`, `Bool`, `Int32`, `Int64`, `Float32`, `Float64`, `Vector2`, `Vector3`, `Vector2int16`, `Vector3int16`, `UDim`, `UDim2`, `CFrame`, `Color3`, `Color3uint8`, `BrickColor`, `NumberRange`, `Rect`, `NumberSequence`, `ColorSequence`, `Enum`, `Ref`, `Content`, `Font`, `PhysicalProperties`, `Axes`, `Faces`, `Tags`, `Attributes`

7
.cursor/rules/rbx-dom.mdc → .cursor/rules/atlas-rbx-dom.mdc
View file

@ -1,9 +1,6 @@
---
description: "rbx-dom submodule: separate Git repo for Roblox format libraries. Has own CI/CD and release cycle."
globs:
- "rbx-dom/**/*.rs"
- "rbx-dom/**/*.lua"
- "rbx-dom/**/*.toml"
globs: rbx-dom/**
alwaysApply: false
---
# rbx-dom Submodule

62
.cursor/rules/src.mdc → .cursor/rules/atlas-src.mdc
View file

@ -1,8 +1,6 @@
---
description: "Rust server/CLI implementation: snapshot system, middleware, commands, syncback, and web API architecture."
globs:
- "src/**/*.rs"
- "crates/**/*.rs"
globs: src/**,crates/**
alwaysApply: false
---
# Rojo Server/CLI (Rust)
@ -19,8 +17,9 @@ Pure functions: filesystem → instance tree → minimal patches → apply chang
**Key files:**
- `mod.rs` - System overview with detailed explanation
- `patch_compute.rs` - Generate diffs
- `patch_apply.rs` - Apply changes
- `patch_compute.rs` - Generate diffs (uses matching algorithm)
- `patch_apply.rs` - Apply changes (resolves `Rojo_Ref_*` and `Rojo_Target_*` refs)
- `matching.rs` - Forward sync matching algorithm (recursive change-count scoring, `MatchingSession` caching)
- `instance_snapshot.rs` - Core snapshot type
```rust
@ -73,14 +72,14 @@ pub struct InstanceContext {
**Filename handling (syncback/two-way sync):**
Instance names with forbidden filesystem chars (`<>:"/\|?*`) are handled by:
Instance names with forbidden filesystem chars (`<>:"/\|?*~`) are handled by:
1. `slugify_name()` -- replaces forbidden chars with `_` (pure, stateless)
2. `deduplicate_name()` -- appends `~1`, `~2`, etc. on collision (stateful)
2. `deduplicate_name_with_ext()` -- dedup key = full filesystem name (slug + extension), case-insensitive; appends `~2`, `~3`, etc. on collision (stateful)
3. Metadata `name` field -- stores the real instance name in `.meta.json5`/`.model.json5` when the slug differs
### 3. Commands (`src/cli/`)
- `serve.rs` - Live sync with WebSocket
- `serve.rs` - Live sync with WebSocket; live syncback via `run_live_syncback()` + `build_dom_from_chunks()` (server restarts after syncback)
- `build.rs` - Generate rbxl/rbxm/rbxmx/rbxlx
- `upload.rs` - Upload to Roblox.com
- `sourcemap.rs` - Generate sourcemaps
@ -93,13 +92,19 @@ Reverse: Roblox file → filesystem
- `snapshot.rs` - Convert instances to snapshots
- `fs_snapshot.rs` - Filesystem representation
- `matching.rs` - Syncback matching algorithm (recursive change-count scoring, `MatchingSession` caching)
- `dedup_suffix.rs` - Dedup cleanup rules (gap-tolerant, base-name promotion, group-to-1)
- `meta.rs` - Meta file read/write helpers (upsert/remove name, update ref paths)
- `ref_properties.rs` - Ref property path linking (`Rojo_Ref_*`, tentative_fs_path, placeholder system)
- `hash/` - Content hashing for change detection
- `property_filter.rs` - Apply syncback rules
- `fs_snapshot.rs` - Filesystem representation, parallel writes (`write_to_vfs_parallel`), ref path post-processing (`fix_ref_paths`)
- `property_filter.rs` - Apply syncback rules (default: `sync_unscriptable = false`)
### 5. Web Server (`src/web/`)
- `api.rs` - HTTP/WebSocket endpoints
- `interface.rs` - Browser UI
- `api.rs` - HTTP/WebSocket endpoints, including `POST /api/syncback` handler (`handle_api_syncback`)
- `interface.rs` - Wire types: patches, `SyncbackRequest`, `ServiceChunk`, `SyncbackPayload`, `ServerExitReason`
- `mod.rs` - `SyncbackSignal` (Mutex + Notify pattern), `LiveServer` (uses `tokio::select!` to wait for connections or syncback signal)
- Protocol version enforcement
### 6. Project System (`src/project.rs`)
@ -120,6 +125,8 @@ Parse/validate `.project.json5` (or legacy `.project.json`), handle nested proje
**Key:** `src/serve_session.rs` manages active sessions
**Live syncback integration:** The serve loop checks for `ServerExitReason::SyncbackRequested(payload)`. When received, calls `run_live_syncback()` which reconstructs the DOM from the plugin's rbxm blob + service chunks via `build_dom_from_chunks()`, runs `syncback_loop(incremental=false)` (clean mode), writes files via `write_to_vfs_parallel()`, refreshes git index, then restarts the serve loop.
### `rojo build`
1. Load project
@ -146,7 +153,7 @@ pub struct SyncbackRules {
ignore_properties: IndexMap<Ustr, Vec<Ustr>>, // Class → properties
ignore_classes: Vec<String>, // Classes to skip entirely
sync_current_camera: Option<bool>, // Default: false
sync_unscriptable: Option<bool>, // Default: true
sync_unscriptable: Option<bool>, // Default: false
ignore_referents: Option<bool>, // Skip Ref properties, default: false
}
```
@ -217,13 +224,14 @@ Review: `cargo insta review`
**HTTP:**
- `GET /` - Browser UI
- `GET /api/rojo` - Server info (version, project name, protocol)
- `POST /api/syncback` - Live syncback: full Roblox → filesystem sync (validates protocol, version, place ID)
**WebSocket:** `ws://host:port/api/rojo`
**Protocol v5:**
**Protocol v6:**
```json
// Handshake
{"type": "ServerInfo", "protocolVersion": 5, "serverVersion": "7.7.0"}
{"type": "ServerInfo", "protocolVersion": 6, "serverVersion": "7.7.0"}
// Patch
{
@ -309,6 +317,7 @@ to_writer(&mut buffer, &dom, dom.root())?;
**Core:** rbx_binary, rbx_xml, rbx_dom_weak, memofs (local crate)
**Async:** tokio, hyper, hyper-tungstenite
**Serialization:** serde, json5, toml, yaml-rust2, csv
**Float precision:** lexical-write-float (for roundtrip-safe f32/f64 formatting in JSON)
**Utilities:** clap, anyhow, thiserror, crossbeam-channel, rayon, globset
## Module Organization
@ -367,13 +376,22 @@ RUST_LOG=rojo=debug rojo build
```
src/
cli/ - Commands
snapshot/ - Core system
snapshot_middleware/ - File handlers
syncback/ - Reverse sync
web/ - HTTP/WebSocket
lib.rs - Public API
main.rs - CLI entry
cli/ - Commands
snapshot/ - Core system
matching.rs - Forward sync matching algorithm
snapshot_middleware/ - File handlers
syncback/ - Reverse sync
matching.rs - Syncback matching algorithm
dedup_suffix.rs - Dedup cleanup rules
meta.rs - Meta file helpers
web/ - HTTP/WebSocket (incl. SyncbackSignal, LiveServer)
interface.rs - Wire types (patches, syncback requests)
change_processor.rs - Two-way sync filesystem writer
git.rs - Git integration (changed files, SHA1 hashing, auto-staging)
rojo_ref.rs - Ref path system (Rojo_Ref_*, RefPathIndex, Luau-style relative paths)
variant_eq.rs - Property value comparison
lib.rs - Public API
main.rs - CLI entry
crates/
memofs/ - Virtual filesystem

191
.cursor/rules/atlas-syncback.mdc
View file

@ -1,8 +1,6 @@
---
description: "Syncback architecture: Roblox→filesystem synchronization, incremental vs clean modes, hash-based change detection, middleware selection, and property filtering."
globs:
- "src/syncback/**/*.rs"
- "src/cli/syncback.rs"
globs: src/syncback/**,src/cli/syncback.rs
alwaysApply: false
---
# Atlas Syncback Architecture
@ -48,15 +46,17 @@ Filesystem (Lua files, models, directories)
```
syncback/
mod.rs - Main loop, orchestration
snapshot.rs - Convert instances → snapshots
fs_snapshot.rs - Filesystem representation
file_names.rs - Generate valid filenames
meta.rs - Meta file read/write helpers
mod.rs - Main loop, orchestration, ref path post-processing
snapshot.rs - Convert instances → snapshots, record_ref_path()
fs_snapshot.rs - Filesystem representation, fix_ref_paths()
file_names.rs - slugify_name(), deduplicate_name_with_ext(), name_for_inst()
matching.rs - Recursive change-count matching algorithm
dedup_suffix.rs - Dedup cleanup rules (gap-tolerant, base-name promotion, group-to-1)
meta.rs - Meta file read/write helpers (upsert/remove name, update ref paths)
property_filter.rs - Filter properties per syncback rules
ref_properties.rs - Handle Ref properties (cross-references)
ref_properties.rs - Ref property path linking (Rojo_Ref_*, tentative_fs_path)
hash/ - Content hashing for change detection
mod.rs - Hash variants
mod.rs - Hash tree computation
variant.rs - Variant-specific hashing
stats.rs - Statistics tracking
```
@ -66,13 +66,16 @@ syncback/
**Flow:**
1. **Parse input file** - Read rbxl/rbxm with `rbx_binary`
2. **Prune tree** - Remove instances not in original project
3. **Filter hidden services** - Remove internal Roblox services (AdService, Analytics, etc.)
4. **Collect pre-prune paths** - Track all instances before pruning (for external refs)
5. **Hash comparison** - Compare hashes of old tree vs new tree
6. **Recursive syncback** - Walk trees, matching instances to files
7. **Write filesystem** - Create/update files based on changes
8. **Link Ref properties** - Resolve cross-references between instances
2. **Collect pre-prune paths** - `collect_all_paths()` maps all Refs to filesystem-name paths before pruning (for external refs)
3. **Prune tree** - Remove instances not in original project
4. **Filter hidden services** - Remove internal Roblox services (AdService, Analytics, etc.)
5. **Collect referents** - `collect_referents()` finds Ref properties and writes `Rojo_Ref_*` attributes using tentative paths
6. **Filter properties** - Apply syncback rules (ignore classes, properties, etc.)
7. **Hash comparison** - Compare hashes of old tree vs new tree
8. **Recursive syncback walk** - Match instances to files via matching algorithm, assign filenames with dedup
9. **Record ref path map** - During the walk, `record_ref_path()` maps each Ref to its final dedup'd filesystem path
10. **Post-process ref paths** - `fix_ref_paths()` compares tentative paths vs final paths, substitutes in written meta/model files
11. **Write filesystem** - Apply FsSnapshot to disk (sequential or parallel)
**Signature:**
```rust
@ -111,6 +114,67 @@ pub fn syncback_loop(
**Implementation detail:** `old_ref` is ignored in `SyncbackSnapshot::with_joined_path` when `incremental=false`.
### Live Syncback Mode (via `atlas serve`)
Triggered via `POST /api/syncback` during an active `atlas serve` session. The plugin encodes the Studio state (service properties + children as rbxm blob) and POSTs directly -- no WebSocket connection required.
**Server-side flow:**
1. `handle_api_syncback()` validates protocol, version, place ID
2. Fires `SyncbackSignal` → `LiveServer` exits accept loop
3. `run_live_syncback()` in `src/cli/serve.rs`:
a. `build_dom_from_chunks()` reconstructs DOM from rbxm blob + `ServiceChunk` metadata
b. Runs `syncback_loop(incremental=false)` -- always clean mode
c. `write_to_vfs_parallel()` writes files with Windows retry logic
d. `refresh_git_index()` updates git
4. Serve loop restarts (new session)
**Key differences from CLI syncback:**
- Runs during an active serve session (not standalone)
- Receives data via HTTP (not file-based)
- Always clean mode (replaces everything)
- Server restarts after completion
**Plugin-side encoding:** `encodeService()` encodes service properties/attributes/refs (ObjectValue carriers for Ref), `SerializationService:SerializeInstancesAsync()` serializes all children into a single rbxm buffer.
See `atlas-api.mdc` for wire format details (`SyncbackRequest`, `ServiceChunk`).
## Matching Algorithm (`matching.rs`)
Pairs new instances (from Roblox file) with old instances (from filesystem) to determine which files to update vs create.
**Entry point:** `match_children(new_children, old_children, new_dom, old_dom, hashes)`
**Algorithm:**
1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring.
2. **Ambiguous groups:** For groups with multiple candidates on either side, compute pairwise `compute_change_count()` -- a recursive scoring function that counts property diffs + recursive children cost. Greedy assignment sorted by cost ascending, child-order tiebreaker for stability.
3. **Hash fast-path:** If `new_hash == old_hash`, return 0 cost immediately (skips property comparison and recursion).
**Constants:**
- `UNMATCHED_PENALTY = 10,000` - Cost per unmatched child (create/delete is categorically more expensive than property tweaks)
- `MAX_SCORING_DEPTH = 3` - Depth limit for recursive scoring to prevent O(n^k) explosion
**`compute_change_count()` and `match_children()` are mutually recursive.** Base case: leaf nodes (no children).
**Output:** `MatchResult { matched: Vec<(new_ref, old_ref)>, unmatched_new: Vec<new_ref>, unmatched_old: Vec<old_ref> }`
**Session caching:** `MatchingSession` wraps a `cost_cache: RefCell<HashMap<(Ref, Ref), u32>>` to cache `compute_change_count` results, avoiding redundant recursive scoring when the same pairs are encountered at different tree levels.
**Known limitation:** Greedy assignment is not globally optimal. The Hungarian algorithm (O(N^3)) would give optimal results, but ambiguous groups are typically small (2-10 instances).
## Dedup Suffix Management (`dedup_suffix.rs`)
Manages `~N` suffix cleanup after instance deletions during two-way sync.
**Key functions:**
- `parse_dedup_suffix(stem) -> Option<(base, N)>` - Parse `~N` from filename stem
- `build_dedup_name(base, suffix, ext)` - Build filename with optional `~N` suffix
- `compute_cleanup_action(removed_stem, remaining_stems) -> DedupCleanupAction` - Determine what to do after a deletion
**Cleanup rules:**
- **Gap-tolerant:** Deleting `~2` from `{Foo, Foo~2, Foo~3}` leaves `{Foo, Foo~3}`. No renumbering.
- **Base-name promotion:** Deleting `Foo` (base) from `{Foo, Foo~2, Foo~3}` promotes `Foo~2` to `Foo`. `Foo~3` stays.
- **Group-to-1 cleanup:** If group shrinks to 1 member, remove its suffix entirely.
## Syncback Rules
Configured in `syncbackRules` section of project file:
@ -133,7 +197,7 @@ pub struct SyncbackRules {
/// Whether to sync Workspace.CurrentCamera (default: false)
pub sync_current_camera: Option<bool>,
/// Whether to sync unscriptable properties (default: true)
/// Whether to sync unscriptable properties (default: false)
pub sync_unscriptable: Option<bool>,
/// Whether to skip Ref properties (default: false)
@ -246,12 +310,13 @@ snapshot.with_new_path(path, new_ref, old_ref)
```
**Filename generation** (`file_names.rs`):
- Slugifies instance name (`slugify_name()`: forbidden chars → `_`)
- Deduplicates against taken names (`deduplicate_name()`: appends `~1`, `~2`, etc.)
- Adds appropriate extension based on middleware
- Stores real name in `.meta.json5` `name` field when slug differs
- `slugify_name()`: forbidden chars (`<>:"/\|?*~`) → `_`. Also strips leading/trailing spaces/dots, neutralizes dangerous suffixes (`.server`, `.meta`, etc.), handles Windows reserved names (`CON`, `PRN`, etc.)
- `deduplicate_name_with_ext()`: Dedup key = full filesystem name (slug + extension), case-insensitive. `Folder "Foo"` (key `"foo"`) and `Script "Foo"` (key `"foo.server.luau"`) do NOT collide. Appends `~2`, `~3`, etc. on collision, with suffix inserted before compound extension: `Foo~2.server.luau`
- `name_for_inst()` returns `(filename, needs_meta_name, dedup_key)`. `needs_meta_name` is true when the filesystem name differs from the real instance name.
- `~` is in `SLUGIFY_CHARS` so natural tilde names (e.g., `"Foo~1"`) get slugified to `"Foo_1"`, avoiding confusion with dedup suffixes.
- Example: `Script` named "My Script" → `My Script.server.luau`
- Example: `Script` named "Hey/Bro" → `Hey_Bro.server.luau` + meta with `name: "Hey/Bro"`
- Example: Two `Folder "Data"` → `Data/` + `Data~2/` (meta: `name: "Data"`)
### Property Filtering
@ -301,6 +366,10 @@ fs_snapshot.merge(other) // Combine snapshots
// Apply to filesystem
fs_snapshot.write_to_vfs(base_path, vfs)?;
fs_snapshot.write_to_vfs_parallel(base_path, vfs)?; // Parallel writes (see below)
// Post-process ref paths
fs_snapshot.fix_ref_paths(&substitutions); // Replace placeholder paths with final relative paths
// Query
fs_snapshot.is_empty()
@ -310,26 +379,57 @@ fs_snapshot.removed_paths()
## Ref Properties (`ref_properties.rs`)
**Challenge:** Ref properties (like `ObjectValue.Value`) point to other instances by Ref, but Refs are different between old and new trees.
**Challenge:** Ref properties (like `ObjectValue.Value`, `Model.PrimaryPart`) point to other instances by Ref, but Refs are different between old and new trees.
**Solution:**
**Solution: Path-based `Rojo_Ref_*` attributes (preferred)**
1. **Collect paths** - Map every Ref → full instance path (e.g., `Workspace.Part1`)
2. **Collect referents** - Find all properties that are Refs, store as (source_path, target_path)
3. **Link referents** - After syncback, use Atlas's `$id` / `Rojo_Target_PropertyName` system to link instances
Syncback writes **Luau-style relative paths** as `Rojo_Ref_PropertyName` attributes via `compute_relative_ref_path()`:
**Atlas linking system:**
```json
```json5
{
"id": "some-unique-id",
"properties": {},
"attributes": {
"Rojo_Target_Value": "some-unique-id"
"Rojo_Ref_PrimaryPart": "./Handle.model.json5"
}
}
```
Plugin reads `Rojo_Target_PropertyName` attributes and sets corresponding Ref properties.
Relative path prefixes: `@self` (same/descendant), `./` (sibling), `../` (parent's sibling), `@game/` (cross-service).
Forward sync resolves both relative and absolute formats via `resolve_ref_path_to_absolute()` + `get_instance_by_path()`.
**Placeholder system:** During the syncback walk, before dedup suffixes are finalized, ref paths are written as `__ROJO_REF_<source>_TO_<target>__` placeholders (encoding both source and target refs). After the walk assigns final filesystem names, `fix_ref_paths()` substitutes placeholders with actual relative paths. Substitutions are sorted by length descending to prevent partial prefix matches.
**Legacy: `Rojo_Target_*` / `$id` system (still supported for reading)**
```json5
{
"id": "some-unique-id",
"attributes": { "Rojo_Target_Value": "some-unique-id" }
}
```
Forward sync also reads `Rojo_Target_*` attributes and resolves via `get_specified_id()`. Priority: `Rojo_Ref_*` takes precedence over `Rojo_Target_*`.
**Key functions:**
- `collect_all_paths(dom)` - Maps all Refs to filesystem-name-compatible paths (called pre-prune for external refs)
- `collect_referents(dom, paths, final_paths)` - Finds all Ref properties, stores as (source, target_path) pairs
- `tentative_fs_path(dom, ref)` - Computes pre-dedup path: slugified name + extension based on class/children
- `tentative_fs_path_public()` - Public wrapper used in post-processing
- `link_referents(dom, referents)` - Writes `Rojo_Ref_*` attributes to instances
**Ref path post-processing (in `mod.rs`):**
During the syncback walk, `record_ref_path()` records each instance's final dedup'd filesystem path in `ref_path_map`. After the walk, `fix_ref_paths()` compares tentative paths to final paths and substitutes in all written meta/model files. Substitutions are sorted by length descending to prevent partial prefix matches.
## Meta File Helpers (`meta.rs`)
Shared helpers for reading/writing `.meta.json5` and `.model.json5` files:
- `upsert_meta_name(path, name)` - Upsert `name` field in `.meta.json5`
- `remove_meta_name(path)` - Remove `name` field; returns `RemoveNameOutcome` (NoOp, FieldRemoved, FileDeleted)
- `upsert_model_name(path, name)` - Upsert `name` field in `.model.json5`
- `update_ref_paths_in_file(path, old_prefix, new_prefix)` - Update `Rojo_Ref_*` attribute paths in a meta/model file
## Statistics Tracking (`stats.rs`)
@ -428,12 +528,9 @@ Filesystem:
## Integration with Plugin
**Plugin does NOT handle syncback.** It only handles:
1. Downloading place file from Studio
2. Sending file to server
3. Server runs syncback command
**Live syncback (plugin-initiated):** The plugin encodes the Studio state via `encodeService()` + `SerializationService` and POSTs to `/api/syncback`. The server reconstructs the DOM, runs `syncback_loop(incremental=false)`, writes files, and restarts. Available from the NotConnected page -- no active WebSocket session required. See `atlas-api.mdc` for wire format.
**Two-way sync (experimental)** is different - see `atlas-api.mdc`.
**Two-way sync** is a separate system for incremental bidirectional changes -- see `atlas-api.mdc`.
## Performance Considerations
@ -441,7 +538,9 @@ Filesystem:
**Incremental benefit:** On large projects, incremental mode with hashing can be 10-100x faster than rewriting everything.
**Parallelization:** Not currently parallelized - filesystem writes are sequential.
**Parallelization:** File writes can be parallelized via `write_to_vfs_parallel()`:
- 4-phase approach: create dirs (sequential) → write files (parallel via rayon) → remove files (parallel) → remove dirs (sequential via VFS)
- Windows retry logic: exponential backoff (10ms initial, doubles each retry, max 3 retries) for OS errors 5 (Access denied) and 32 (Sharing violation)
## Debugging
@ -473,14 +572,24 @@ diff place.rbxl rebuilt.rbxl
**From code comments:**
- Better detection of when root is directory vs file
- More sophisticated middleware selection
- Parallel file writes
- Better handling of Roblox format changes
- Support for partial syncback (only changed instances)
- Globally optimal matching (Hungarian algorithm) for large ambiguous groups
- ~~Matching cache (parent_ref -> match_assignments) for avoiding redundant scoring~~ (session-level caching now implemented via `MatchingSession`; per-parent invalidation cache still deferred)
## Key Files to Reference
- `src/syncback/mod.rs` - Main orchestration loop
- `src/syncback/mod.rs` - Main orchestration loop, ref path post-processing
- `src/syncback/snapshot.rs` - SyncbackSnapshot implementation
- `src/syncback/matching.rs` - Recursive change-count matching algorithm
- `src/syncback/file_names.rs` - slugify_name(), deduplicate_name_with_ext()
- `src/syncback/dedup_suffix.rs` - Dedup cleanup rules
- `src/syncback/meta.rs` - Meta file read/write helpers
- `src/syncback/ref_properties.rs` - Ref property path linking
- `src/syncback/hash/` - Hashing algorithm
- `src/syncback/fs_snapshot.rs` - Filesystem operations, parallel writes, ref path post-processing
- `src/syncback/property_filter.rs` - Property filtering logic
- `src/cli/syncback.rs` - CLI command implementation
- `src/cli/serve.rs` - Live syncback handler (`run_live_syncback`, `build_dom_from_chunks`)
- `src/web/interface.rs` - `SyncbackRequest`, `ServiceChunk`, `SyncbackPayload` types
- `src/rojo_ref.rs` - Luau-style relative ref paths (`compute_relative_ref_path`, `resolve_ref_path_to_absolute`)

20
.cursor/rules/atlas.mdc
View file

@ -1,5 +1,4 @@
---
description: "Essential Atlas context: architecture, contribution workflow, and code conventions for this project."
alwaysApply: true
---
@ -33,6 +32,7 @@ Atlas bridges filesystem and Roblox Studio, enabling professional development to
- Build/upload rbxl/rbxm files from command line
- Syncback: pull instances from Roblox → filesystem (7.7.0+)
- Two-way sync (plugin-based): live bidirectional sync during `atlas serve` sessions
- Live syncback: one-click full sync from Studio to filesystem during `atlas serve` (no WebSocket connection required)
### Two-Way Sync Strategy
@ -41,7 +41,7 @@ Atlas bridges filesystem and Roblox Studio, enabling professional development to
**Current state:**
- `atlas syncback` (CLI) is the **ground truth** for Roblox → filesystem syncing. It is battle-tested and handles edge cases reliably.
- Plugin-based two-way sync (via `atlas serve` + the Studio plugin) is under active development to reach feature parity and match the resilience of CLI syncback.
- The bar for replacing syncback is high: plugin two-way sync must handle all the same scenarios (encoded names, nested projects, property conflicts, large trees, concurrent edits) just as reliably before syncback can be deprecated.
- The bar for replacing syncback is high: plugin two-way sync must handle all the same scenarios (slugified names, dedup suffixes, nested projects, property conflicts, large trees, concurrent edits) just as reliably before syncback can be deprecated.
**When working on two-way sync:** Plugin-based sync must produce **exactly the same filesystem output** that `atlas syncback` would give for the same input. Byte-for-byte identical files, identical directory structures, identical naming. If the plugin path produces different results, treat syncback as the correct reference unless there's an explicit, documented reason to diverge. Any divergence is a bug until proven otherwise.
@ -207,7 +207,7 @@ pub struct InstanceContext {
}
```
**Filename handling:** Instance names with forbidden filesystem chars are slugified (forbidden chars → `_`), deduplicated (`~1`, `~2`, etc. on collision), and the real name is stored in `.meta.json5`/`.model.json5` `name` fields. See `src/syncback/file_names.rs`.
**Filename handling:** Instance names with forbidden filesystem chars are slugified (forbidden chars → `_`), deduplicated (`~2`, `~3`, etc. on collision), and the real name is stored in `.meta.json5`/`.model.json5` `name` fields. See `src/syncback/file_names.rs`.
**VFS passed separately to snapshot functions:**
```rust
@ -316,16 +316,30 @@ Reconciler.applyPatch(instanceMap, patch)
src/ - Rust server/CLI
cli/ - Command implementations
snapshot/ - Core snapshot system
matching.rs - Forward sync matching algorithm
snapshot_middleware/ - File type handlers
syncback/ - Roblox → filesystem
matching.rs - Syncback matching algorithm
dedup_suffix.rs - Dedup cleanup rules
meta.rs - Meta file read/write helpers
web/ - HTTP/WebSocket server
interface.rs - Wire types (patches, syncback requests)
change_processor.rs - Two-way sync filesystem writer
git.rs - Git integration (changed files, SHA1 hashing, auto-staging)
rojo_ref.rs - Ref path system (Rojo_Ref_*, RefPathIndex)
variant_eq.rs - Property value comparison
plugin/src/ - Lua Studio plugin
App/ - Roact UI components
Reconciler/ - Patch application logic
matching.lua - Lua matching algorithm
trueEquals.lua - Shared value equality module
ChangeBatcher/ - Two-way sync patch creation
encodeService.lua - Service-level property encoding for live syncback
Config.lua - Version and protocol config
PatchTree.lua - Patch visualization tree (git-based default selections)
SHA1.luau - SHA1 hash for git blob format comparison
XXH32.luau - Hash function for plugin
rbx-dom/ - Git submodule (separate repo)
rbx_binary/ - Binary format (rbxm/rbxl)

0
.cursor/rules/luau/CNAME → .cursor/rules/luau-rfcs/CNAME
View file

0
.cursor/rules/luau/_config.yml → .cursor/rules/luau-rfcs/_config.yml
View file

0
.cursor/rules/luau/abstract-module-paths-and-init-dot-luau.md → .cursor/rules/luau-rfcs/abstract-module-paths-and-init-dot-luau.md
View file

0
.cursor/rules/luau/amended-require-resolution.md → .cursor/rules/luau-rfcs/amended-require-resolution.md
View file

0
.cursor/rules/luau/behavior-eq-metamethod.md → .cursor/rules/luau-rfcs/behavior-eq-metamethod.md
View file

0
.cursor/rules/luau/behavior-stricter-utf8-library.md → .cursor/rules/luau-rfcs/behavior-stricter-utf8-library.md
View file

0
.cursor/rules/luau/change-global-version.md → .cursor/rules/luau-rfcs/change-global-version.md
View file

0
.cursor/rules/luau/config-luauconfig.md → .cursor/rules/luau-rfcs/config-luauconfig.md
View file

0
.cursor/rules/luau/config-luaurc.md → .cursor/rules/luau-rfcs/config-luaurc.md
View file

0
.cursor/rules/luau/deprecate-getfenv-setfenv.md → .cursor/rules/luau-rfcs/deprecate-getfenv-setfenv.md
View file

0
.cursor/rules/luau/deprecate-table-getn-foreach.md → .cursor/rules/luau-rfcs/deprecate-table-getn-foreach.md
View file

0
.cursor/rules/luau/disallow-proposals-leading-to-ambiguity-in-grammar.md → .cursor/rules/luau-rfcs/disallow-proposals-leading-to-ambiguity-in-grammar.md
View file

0
.cursor/rules/luau/explicit-type-parameter-instantiation.md → .cursor/rules/luau-rfcs/explicit-type-parameter-instantiation.md
View file

0
.cursor/rules/luau/function-bit32-byteswap.md → .cursor/rules/luau-rfcs/function-bit32-byteswap.md
View file

0
.cursor/rules/luau/function-bit32-countlz-countrz.md → .cursor/rules/luau-rfcs/function-bit32-countlz-countrz.md
View file

0
.cursor/rules/luau/function-buffer-bits.md → .cursor/rules/luau-rfcs/function-buffer-bits.md
View file

0
.cursor/rules/luau/function-coroutine-close.md → .cursor/rules/luau-rfcs/function-coroutine-close.md
View file

0
.cursor/rules/luau/function-debug-info.md → .cursor/rules/luau-rfcs/function-debug-info.md
View file

0
.cursor/rules/luau/function-inlining.md → .cursor/rules/luau-rfcs/function-inlining.md
View file

0
.cursor/rules/luau/function-math-lerp.md → .cursor/rules/luau-rfcs/function-math-lerp.md
View file

0
.cursor/rules/luau/function-math-map.md → .cursor/rules/luau-rfcs/function-math-map.md
View file

0
.cursor/rules/luau/function-string-pack-unpack.md → .cursor/rules/luau-rfcs/function-string-pack-unpack.md
View file

0
.cursor/rules/luau/function-table-clear.md → .cursor/rules/luau-rfcs/function-table-clear.md
View file

0
.cursor/rules/luau/function-table-clone.md → .cursor/rules/luau-rfcs/function-table-clone.md
View file

0
.cursor/rules/luau/function-table-create-find.md → .cursor/rules/luau-rfcs/function-table-create-find.md
View file

0
.cursor/rules/luau/function-table-freeze.md → .cursor/rules/luau-rfcs/function-table-freeze.md
View file

0
.cursor/rules/luau/function-vector-lerp.md → .cursor/rules/luau-rfcs/function-vector-lerp.md
View file

0
.cursor/rules/luau/generalized-iteration.md → .cursor/rules/luau-rfcs/generalized-iteration.md
View file

0
.cursor/rules/luau/generic-function-subtyping.md → .cursor/rules/luau-rfcs/generic-function-subtyping.md
View file

0
.cursor/rules/luau/generic-functions.md → .cursor/rules/luau-rfcs/generic-functions.md
View file

0
.cursor/rules/luau/index-type-operator.md → .cursor/rules/luau-rfcs/index-type-operator.md
View file

0
.cursor/rules/luau/index.md → .cursor/rules/luau-rfcs/index.md
View file

0
.cursor/rules/luau/keyof-type-operator.md → .cursor/rules/luau-rfcs/keyof-type-operator.md
View file

0
.cursor/rules/luau/len-metamethod-rawlen.md → .cursor/rules/luau-rfcs/len-metamethod-rawlen.md
View file

0
.cursor/rules/luau/local-type-inference.md → .cursor/rules/luau-rfcs/local-type-inference.md
View file

0
.cursor/rules/luau/lower-bounds-calculation.md → .cursor/rules/luau-rfcs/lower-bounds-calculation.md
View file

0
.cursor/rules/luau/luau-mascot.md → .cursor/rules/luau-rfcs/luau-mascot.md
View file

0
.cursor/rules/luau/mascot.png → .cursor/rules/luau-rfcs/mascot.png
View file

Side by side Swipe Overlay

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

Before After
Before After

56
.cursor/rules/luau-rfcs/math-constants.md Normal file
View file

@ -0,0 +1,56 @@
# Math constants
## Summary
Expand the constants provided in the `math` library.
## Motivation
With the introduction of [`math.isnan`](https://rfcs.luau.org/math-isnan-isfinite-isinf.html), Luau has the opportunity to add a new constant representing `NaN` (Not a Number).
As linting support for Luau continues becoming more robust, we can discourage patterns like `0/0` to generate `NaN` and instead provide the constant directly through the math library.
Without the `math.isnan` API, this could have created confusion in the past:
```luau
if maybeNan == math.nan do
print("This line will never be hit: comparing to NaN always returns false.")
end
```
With the new API, however, this is now easier to use:
```luau
local maybeNan = math.nan
if math.isnan(maybeNan) do
print("This line will be hit!")
end
```
While adding `NaN`, we also have an opportunity to modernize the set of constants available through the `math` library.
Some history:
- Lua 5.0 (2003) introduced `math.pi`, representing the mathematical constant $\pi$.
- Lua 5.1 (2006) introduced `math.huge`, mapping directly to infinity as defined by the [IEEE 754 standard](https://en.wikipedia.org/wiki/IEEE_754).
- Lua 5.3 (2015) introduced integers and the `math.maxinteger` and `math.mininteger` constants alongside them.
Luau currently only supports `math.pi` and `math.huge`.
Integer types are not supported in Luau, so `math.maxinteger` and `math.mininteger` wouldn't make much sense, but adding other commonly used constants would be an easy value-add for users of the language.
## Design
In addition to `math.pi` and `math.huge`, we propose adding the following constants.
- `math.nan`: some `NaN` value as defined by IEEE 754.
- The exact representation (payload and sign) is unspecified.
- Comparing to `math.nan` will always return `false`; lint rules recommend using `math.isnan`.
- `math.e`: the mathematical constant $e = 2.71828182845904523536$.
- `math.phi`: the golden ratio, defined as $\phi = 1.61803398874989484820$.
- `math.sqrt2`: the square root of `2`, defined as equal to `math.sqrt(2)`.
- `math.tau`: the mathematical constant $\tau$, defined as equal to `2 * math.pi`.
## Drawbacks
- The semantics of `math.nan` are unique and somewhat unexpected for users not already familiar with IEEE 754's floating-point arithmetic rules.
- Adding new constants increases the API surface of the `math` library and diverges further from Lua 5.1.
## Alternatives
- Do nothing: continue requiring users to define their own constants (e.g., `local nan = 0/0`, `local e = math.exp(1)`).
- Only add `math.nan` without the additional constants, deferring modernization for a future RFC.
- Rely on these constants being provided through external libraries.

0
.cursor/rules/luau/math-isnan-isfinite-isinf.md → .cursor/rules/luau-rfcs/math-isnan-isfinite-isinf.md
View file

0
.cursor/rules/luau/metatable-type-functions.md → .cursor/rules/luau-rfcs/metatable-type-functions.md
View file

0
.cursor/rules/luau/method-type-issubtypeof.md → .cursor/rules/luau-rfcs/method-type-issubtypeof.md
View file

0
.cursor/rules/luau/negation-types.md → .cursor/rules/luau-rfcs/negation-types.md
View file

0
.cursor/rules/luau/never-and-unknown-types.md → .cursor/rules/luau-rfcs/never-and-unknown-types.md
View file

0
.cursor/rules/luau/new-nonstrict.md → .cursor/rules/luau-rfcs/new-nonstrict.md
View file

0
.cursor/rules/luau/new-require-by-string-semantics.md → .cursor/rules/luau-rfcs/new-require-by-string-semantics.md
View file

0
.cursor/rules/luau/property-readonly.md → .cursor/rules/luau-rfcs/property-readonly.md
View file

0
.cursor/rules/luau/property-writeonly.md → .cursor/rules/luau-rfcs/property-writeonly.md
View file

0
.cursor/rules/luau/rawget-type-operator.md → .cursor/rules/luau-rfcs/rawget-type-operator.md
View file

0
.cursor/rules/luau/recursive-type-restriction.md → .cursor/rules/luau-rfcs/recursive-type-restriction.md
View file

0
.cursor/rules/luau/relax-recursive-type-restriction.md → .cursor/rules/luau-rfcs/relax-recursive-type-restriction.md
View file

0
.cursor/rules/luau/require-by-string-aliases.md → .cursor/rules/luau-rfcs/require-by-string-aliases.md
View file

0
.cursor/rules/luau/reserve-dollar-sign.md → .cursor/rules/luau-rfcs/reserve-dollar-sign.md
View file

0
.cursor/rules/luau/sealed-table-subtyping.md → .cursor/rules/luau-rfcs/sealed-table-subtyping.md
View file

Some files were not shown because too many files have changed in this diff Show more