--- globs: src/syncback/**,src/cli/syncback.rs alwaysApply: false --- # Atlas Syncback Architecture > Atlas is a fork of Rojo. Many internal identifiers, API endpoints, and struct names still reference the original "Rojo" naming. ## Quality Standard This is a production-grade synchronization system. The filesystem must represent a 1:1 copy of the Studio place. The core invariant: **Round-trip identity:** Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a bit-identical instance tree -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug. Every audit item must be evaluated against this standard. A "works most of the time" finding is not acceptable. If there is ANY code path where an instance name, property, or hierarchy relationship can be lost, mangled, or silently altered through a syncback→rebuild cycle, flag it as critical. ## Code Quality Standard We are building a maintainable codebase. During audits, watch for duplicated logic -- the same slugify/dedup/meta-update pattern copy-pasted across `change_processor.rs`, `api.rs`, `dir.rs`, `project.rs`, etc. If you find duplicated code that could be consolidated into a shared helper: - **Small refactor** (extracting a helper function, consolidating 2-3 call sites): flag it AND fix it as part of the audit. - **Major system rewrite** (restructuring data flow, changing function signatures across 10+ call sites, redesigning the snapshot pipeline): flag it clearly in a "Deferred Refactors" section at the bottom of the report. We will evaluate whether it's worth doing after the audit is complete. The goal is clean, DRY code that is easy to reason about and hard to get wrong. If the same "compute meta path, check if name needs slugify, upsert/remove name field" sequence appears in 3+ places, that's a refactor candidate. **Syncback** is the reverse operation of building: it takes a Roblox binary file (rbxl/rbxm) and writes instances back to the filesystem according to project structure. **Entry point:** `atlas syncback --input ` ## Core Concept ``` Roblox File (rbxl/rbxm) ↓ Parse with rbx_binary WeakDom (in-memory instances) ↓ Match to project structure Filesystem (Lua files, models, directories) ``` **Key challenge:** Determining which instances map to which files, and preserving existing file structure when possible. ## Architecture Overview (`src/syncback/`) ### Module Structure ``` syncback/ mod.rs - Main loop, orchestration, ref path post-processing snapshot.rs - Convert instances → snapshots, record_ref_path() fs_snapshot.rs - Filesystem representation, fix_ref_paths() file_names.rs - slugify_name(), deduplicate_name_with_ext(), name_for_inst() matching.rs - Recursive change-count matching algorithm dedup_suffix.rs - Dedup cleanup rules (gap-tolerant, base-name promotion, group-to-1) meta.rs - Meta file read/write helpers (upsert/remove name, update ref paths) property_filter.rs - Filter properties per syncback rules ref_properties.rs - Ref property path linking (Rojo_Ref_*, tentative_fs_path) hash/ - Content hashing for change detection mod.rs - Hash tree computation variant.rs - Variant-specific hashing stats.rs - Statistics tracking ``` ### Main Loop (`syncback_loop`) **Flow:** 1. **Parse input file** - Read rbxl/rbxm with `rbx_binary` 2. **Collect pre-prune paths** - `collect_all_paths()` maps all Refs to filesystem-name paths before pruning (for external refs) 3. **Prune tree** - Remove instances not in original project 4. **Filter hidden services** - Remove internal Roblox services (AdService, Analytics, etc.) 5. **Collect referents** - `collect_referents()` finds Ref properties and writes `Rojo_Ref_*` attributes using tentative paths 6. **Filter properties** - Apply syncback rules (ignore classes, properties, etc.) 7. **Hash comparison** - Compare hashes of old tree vs new tree 8. **Recursive syncback walk** - Match instances to files via matching algorithm, assign filenames with dedup 9. **Record ref path map** - During the walk, `record_ref_path()` maps each Ref to its final dedup'd filesystem path 10. **Post-process ref paths** - `fix_ref_paths()` compares tentative paths vs final paths, substitutes in written meta/model files 11. **Write filesystem** - Apply FsSnapshot to disk (sequential or parallel) **Signature:** ```rust pub fn syncback_loop( vfs: &Vfs, old_tree: &mut RojoTree, // Existing project tree new_tree: WeakDom, // Roblox file contents project: &Project, // Project configuration incremental: bool, // Preserve structure vs rebuild ) -> anyhow::Result ``` ## Syncback Modes ### Incremental Mode (`incremental: true`) **Default behavior.** Preserves existing file structure: - If `src/MyScript.luau` exists, update it - If `src/MyFolder/init.luau` exists (directory structure), preserve it - Only creates new files when instances don't match anything - Compares hashes to detect actual changes (avoids rewriting unchanged files) **Use case:** Normal workflow where you sync changes from Studio back to disk. ### Clean Mode (`incremental: false`) **Rebuild from scratch.** Treats all children as new: - Ignores existing file structure - Generates new filenames for everything - Chooses best middleware for each instance - Useful when project structure changed significantly **Use case:** Major refactoring, or when incremental mode produces wrong results. **Implementation detail:** `old_ref` is ignored in `SyncbackSnapshot::with_joined_path` when `incremental=false`. ## Matching Algorithm (`matching.rs`) Pairs new instances (from Roblox file) with old instances (from filesystem) to determine which files to update vs create. **Entry point:** `match_children(new_children, old_children, new_dom, old_dom, hashes)` **Algorithm:** 1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring. 2. **Ambiguous groups:** For groups with multiple candidates on either side, compute pairwise `compute_change_count()` -- a recursive scoring function that counts property diffs + recursive children cost. Greedy assignment sorted by cost ascending, child-order tiebreaker for stability. 3. **Hash fast-path:** If `new_hash == old_hash`, return 0 cost immediately (skips property comparison and recursion). **Constants:** - `UNMATCHED_PENALTY = 10,000` - Cost per unmatched child (create/delete is categorically more expensive than property tweaks) - `MAX_SCORING_DEPTH = 3` - Depth limit for recursive scoring to prevent O(n^k) explosion **`compute_change_count()` and `match_children()` are mutually recursive.** Base case: leaf nodes (no children). **Output:** `MatchResult { matched: Vec<(new_ref, old_ref)>, unmatched_new: Vec, unmatched_old: Vec }` **Known limitation:** Greedy assignment is not globally optimal. The Hungarian algorithm (O(N^3)) would give optimal results, but ambiguous groups are typically small (2-10 instances). ## Dedup Suffix Management (`dedup_suffix.rs`) Manages `~N` suffix cleanup after instance deletions during two-way sync. **Key functions:** - `parse_dedup_suffix(stem) -> Option<(base, N)>` - Parse `~N` from filename stem - `build_dedup_name(base, suffix, ext)` - Build filename with optional `~N` suffix - `compute_cleanup_action(removed_stem, remaining_stems) -> DedupCleanupAction` - Determine what to do after a deletion **Cleanup rules:** - **Gap-tolerant:** Deleting `~1` from `{Foo, Foo~1, Foo~2}` leaves `{Foo, Foo~2}`. No renumbering. - **Base-name promotion:** Deleting `Foo` (base) from `{Foo, Foo~1, Foo~2}` promotes `Foo~1` to `Foo`. `Foo~2` stays. - **Group-to-1 cleanup:** If group shrinks to 1 member, remove its suffix entirely. ## Syncback Rules Configured in `syncbackRules` section of project file: ```rust pub struct SyncbackRules { /// Roblox tree paths to ignore (e.g., "ServerStorage/Secrets") pub ignore_trees: Vec, /// Filesystem glob patterns to ignore (e.g., "src/client/*") pub ignore_paths: Vec, /// Properties to not sync back per class /// Example: {"BasePart": ["Color"], "Script": ["Disabled"]} pub ignore_properties: IndexMap>, /// Instance classes to skip entirely pub ignore_classes: Vec, /// Whether to sync Workspace.CurrentCamera (default: false) pub sync_current_camera: Option, /// Whether to sync unscriptable properties (default: true) pub sync_unscriptable: Option, /// Whether to skip Ref properties (default: false) pub ignore_referents: Option, /// Whether /** in ignore paths also matches the directory itself pub create_ignore_dir_paths: Option, /// Override root-level ignoreHiddenServices for syncback pub ignore_hidden_services: Option, /// Warn on duplicate child names during syncback pub warn_duplicate_names: Option, } ``` **Property filtering** (`property_filter.rs`): - Filters out `Name`, `Parent` (always) - Filters out properties in `ignore_properties` - Filters out unscriptable properties if `sync_unscriptable: false` - Filters out middleware-specific properties (e.g., `Script.Source` when writing to `.luau` file) ### Hidden Services **Default behavior (`ignoreHiddenServices: true`):** Only sync "visible" services: ```rust const VISIBLE_SERVICES: &[&str] = &[ "Lighting", "MaterialService", "ReplicatedFirst", "ReplicatedStorage", "ServerScriptService", "ServerStorage", "SoundService", "StarterGui", "StarterPack", "StarterPlayer", "Teams", "TextChatService", "VoiceChatService", "Workspace", ]; ``` **Rationale:** Prevents plugin from trying to delete services like `HttpService`, `AnalyticsService`, `Chat` that were never synced to disk. ## Hash-Based Change Detection (`hash/`) ### Why Hashing? **Problem:** Writing every file on every syncback is slow and creates noise in git diffs. **Solution:** Hash both old and new instance trees. Only write files where hashes differ. ### Hash Algorithm Hashing uses **blake3** directly - there's no `HashVariant` struct. ```rust /// Returns HashMap mapping each instance to its content hash pub fn hash_tree(project: &Project, dom: &WeakDom, root_ref: Ref) -> HashMap ``` **Hashing strategy:** 1. Process instances bottom-up (children first) 2. Hash class name, then filtered properties 3. Sort and hash children's hashes 4. Finalize with blake3 **Special cases:** - Ref properties are hashed as paths (since Refs change between trees) - Some properties are normalized (e.g., line endings in scripts) - Unscriptable properties can be excluded from hash ### Hash Comparison ```rust if old_hash == new_hash { // No changes - skip writing file return Ok(None); } else { // Changed - write file write_instance_to_file(snapshot)?; } ``` ## SyncbackSnapshot (`snapshot.rs`) **Core type representing instance to sync back:** ```rust pub struct SyncbackSnapshot<'sync> { pub data: SyncbackData<'sync>, // Context (VFS, trees, project) pub old: Option, // Old tree instance (if exists) pub new: Ref, // New tree instance (Roblox file) pub path: PathBuf, // Target filesystem path pub middleware: Option, // How to write (Lua, Rbxm, etc.) pub needs_meta_name: bool, // Filesystem name differs from instance name } ``` ### Snapshot Creation **Methods:** ```rust // Join child to current path snapshot.with_joined_path(new_ref, old_ref)? // Result: parent_path/ChildName.luau // Use base path snapshot.with_base_path(&base, new_ref, old_ref)? // Result: base/ChildName.luau // Explicit path snapshot.with_new_path(path, new_ref, old_ref) // Result: path (exactly as provided) ``` **Filename generation** (`file_names.rs`): - `slugify_name()`: forbidden chars (`<>:"/\|?*~`) → `_`. Also strips leading/trailing spaces/dots, neutralizes dangerous suffixes (`.server`, `.meta`, etc.), handles Windows reserved names (`CON`, `PRN`, etc.) - `deduplicate_name_with_ext()`: Dedup key = full filesystem name (slug + extension), case-insensitive. `Folder "Foo"` (key `"foo"`) and `Script "Foo"` (key `"foo.server.luau"`) do NOT collide. Appends `~1`, `~2`, etc. on collision, with suffix inserted before compound extension: `Foo~1.server.luau` - `name_for_inst()` returns `(filename, needs_meta_name, dedup_key)`. `needs_meta_name` is true when the filesystem name differs from the real instance name. - `~` is in `SLUGIFY_CHARS` so natural tilde names (e.g., `"Foo~1"`) get slugified to `"Foo_1"`, avoiding confusion with dedup suffixes. - Example: `Script` named "My Script" → `My Script.server.luau` - Example: `Script` named "Hey/Bro" → `Hey_Bro.server.luau` + meta with `name: "Hey/Bro"` - Example: Two `Folder "Data"` → `Data/` + `Data~1/` (meta: `name: "Data"`) ### Property Filtering ```rust snapshot.get_path_filtered_properties(new_ref) ``` **Filters out:** - `Source` property for scripts (content goes in file body) - `Value` property for StringValue (content goes in `.txt` file) - Properties specified in `ignoreProperties` rules - Name/Parent (structural, not data) ## Middleware Selection **Decision tree:** 1. **Old instance exists?** → Use same middleware as before (preserve structure) 2. **Script type?** → Choose appropriate Lua middleware (`.server.luau`, `.client.luau`, etc.) 3. **Model/container?** → Choose directory or rbxm based on complexity 4. **Special types?** → CSV for LocalizationTable, TXT for StringValue 5. **Fallback** → Rbxm or JsonModel **Debug override:** `ROJO_SYNCBACK_DEBUG` environment variable - `1` = Use Rbxmx instead of Rbxm - `2` = Use JsonModel ## Filesystem Snapshot (`fs_snapshot.rs`) **Represents filesystem operations to perform:** ```rust pub struct FsSnapshot { added_files: HashMap>, // New files to create added_dirs: HashSet, // New directories removed_files: HashSet, // Files to delete removed_dirs: HashSet, // Directories to delete } ``` **Key methods:** ```rust // Create snapshot with operations fs_snapshot.with_added_file(path, contents) fs_snapshot.with_added_dir(path) fs_snapshot.merge(other) // Combine snapshots // Apply to filesystem fs_snapshot.write_to_vfs(base_path, vfs)?; fs_snapshot.write_to_vfs_parallel(base_path, vfs)?; // Parallel writes with retry logic // Post-process ref paths fs_snapshot.fix_ref_paths(&substitutions); // Replace tentative paths with final dedup'd paths // Query fs_snapshot.is_empty() fs_snapshot.added_paths() fs_snapshot.removed_paths() ``` ## Ref Properties (`ref_properties.rs`) **Challenge:** Ref properties (like `ObjectValue.Value`, `Model.PrimaryPart`) point to other instances by Ref, but Refs are different between old and new trees. **Solution: Path-based `Rojo_Ref_*` attributes (preferred)** Syncback writes filesystem-name-based paths as `Rojo_Ref_PropertyName` attributes: ```json5 { "properties": {}, "attributes": { "Rojo_Ref_PrimaryPart": "Workspace/Handle.model.json5" } } ``` Forward sync reads `Rojo_Ref_*` attributes and resolves them to tree instances via `get_instance_by_path()`. **Legacy: `Rojo_Target_*` / `$id` system (still supported for reading)** ```json5 { "id": "some-unique-id", "attributes": { "Rojo_Target_Value": "some-unique-id" } } ``` Forward sync also reads `Rojo_Target_*` attributes and resolves via `get_specified_id()`. Priority: `Rojo_Ref_*` takes precedence over `Rojo_Target_*`. **Key functions:** - `collect_all_paths(dom)` - Maps all Refs to filesystem-name-compatible paths (called pre-prune for external refs) - `collect_referents(dom, paths, final_paths)` - Finds all Ref properties, stores as (source, target_path) pairs - `tentative_fs_path(dom, ref)` - Computes pre-dedup path: slugified name + extension based on class/children - `tentative_fs_path_public()` - Public wrapper used in post-processing - `link_referents(dom, referents)` - Writes `Rojo_Ref_*` attributes to instances **Ref path post-processing (in `mod.rs`):** During the syncback walk, `record_ref_path()` records each instance's final dedup'd filesystem path in `ref_path_map`. After the walk, `fix_ref_paths()` compares tentative paths to final paths and substitutes in all written meta/model files. Substitutions are sorted by length descending to prevent partial prefix matches. ## Meta File Helpers (`meta.rs`) Shared helpers for reading/writing `.meta.json5` and `.model.json5` files: - `upsert_meta_name(path, name)` - Upsert `name` field in `.meta.json5` - `remove_meta_name(path)` - Remove `name` field; returns `RemoveNameOutcome` (NoOp, FieldRemoved, FileDeleted) - `upsert_model_name(path, name)` - Upsert `name` field in `.model.json5` - `update_ref_paths_in_file(path, old_prefix, new_prefix)` - Update `Rojo_Ref_*` attribute paths in a meta/model file ## Statistics Tracking (`stats.rs`) ```rust pub struct SyncbackStats { duplicate_name_count: AtomicUsize, // Instances skipped (name collision) rbxm_fallback_count: AtomicUsize, // Instances using rbxm fallback unknown_class_count: AtomicUsize, // Unknown classes encountered unknown_property_count: AtomicUsize, // Unknown properties encountered unknown_classes: Mutex>, unknown_properties: Mutex>, } ``` **Key methods:** ```rust stats.record_duplicate_name(path, name) stats.record_rbxm_fallback(path, reason) stats.record_unknown_class(class_name) stats.record_unknown_property(class, property) stats.has_issues() // Any warnings? stats.has_unknown_types() // Unknown classes/properties? stats.log_summary() // Log final report ``` **Logged at end of syncback:** ``` Syncback complete. Warnings: 2 duplicate names, 1 unknown class ``` ## Common Patterns ### Syncing a Script **Roblox file has:** ``` Script "MyScript" Source = "print('hello')" Disabled = false ``` **Syncback produces:** ```luau -- src/MyScript.server.luau print('hello') ``` **Meta file (if needed):** ```json5 // src/MyScript.server.meta.json5 (for MyScript.server.luau) { "properties": { "Disabled": false } } ``` ### Syncing a Model **Simple model (few children):** → `.rbxm` or `.rbxmx` file **Complex model (many children):** → Directory with `init.meta.json5` or `init.luau` ### Syncing Nested Folders ``` Roblox: Folder "Modules" ModuleScript "Utils" Folder "Math" ModuleScript "Add" Filesystem: src/Modules/ Utils.luau Math/ Add.luau ``` ## Error Handling **Common errors:** 1. **Middleware mismatch** - Instance type changed but file structure can't accommodate - Solution: Use clean mode 2. **Invalid filename** - Instance name contains invalid chars - Solution: Handled automatically by `slugify_name()` + metadata `name` field 3. **Ref to pruned instance** - Property points to instance not in project - Logged as warning, property not synced 4. **Write permission** - Can't write to file - Error reported, syncback continues for other files **Error propagation:** Errors are propagated via `anyhow::bail!`, not collected in stats. Fatal errors abort the syncback; non-fatal issues are logged as warnings. ## Integration with Plugin **Plugin does NOT handle syncback.** It only handles: 1. Downloading place file from Studio 2. Sending file to server 3. Server runs syncback command **Two-way sync** is different - see `atlas-api.mdc`. ## Performance Considerations **Hashing overhead:** Hashing is fast (~50µs per instance), negligible compared to disk I/O. **Incremental benefit:** On large projects, incremental mode with hashing can be 10-100x faster than rewriting everything. **Parallelization:** File writes can be parallelized via `write_to_vfs_parallel()` (includes Windows retry logic). ## Debugging **Enable verbose logging:** ```bash RUST_LOG=rojo::syncback=debug atlas syncback project/ --input place.rbxl ``` **Shows:** - Which instances are being processed - Hash comparisons - File write decisions - Middleware selection **Test syncback:** ```bash # Syncback once atlas syncback project/ --input place.rbxl # Build back to verify roundtrip atlas build project/ -o rebuilt.rbxl # Compare diff place.rbxl rebuilt.rbxl ``` ## Future Improvements **From code comments:** - Better detection of when root is directory vs file - More sophisticated middleware selection - Better handling of Roblox format changes - Support for partial syncback (only changed instances) - Globally optimal matching (Hungarian algorithm) for large ambiguous groups - Matching cache (parent_ref -> match_assignments) for avoiding redundant scoring ## Key Files to Reference - `src/syncback/mod.rs` - Main orchestration loop, ref path post-processing - `src/syncback/snapshot.rs` - SyncbackSnapshot implementation - `src/syncback/matching.rs` - Recursive change-count matching algorithm - `src/syncback/file_names.rs` - slugify_name(), deduplicate_name_with_ext() - `src/syncback/dedup_suffix.rs` - Dedup cleanup rules - `src/syncback/meta.rs` - Meta file read/write helpers - `src/syncback/ref_properties.rs` - Ref property path linking - `src/syncback/hash/` - Hashing algorithm - `src/syncback/property_filter.rs` - Property filtering logic - `src/cli/syncback.rs` - CLI command implementation