rojo/.cursor/rules/atlas-syncback.mdc

---
globs: src/syncback/**,src/cli/syncback.rs
alwaysApply: false
---

# Atlas Syncback Architecture

> Atlas is a fork of Rojo. Many internal identifiers, API endpoints, and struct names still reference the original "Rojo" naming.

## Quality Standard

This is a production-grade synchronization system. The filesystem must represent a 1:1 copy of the Studio place. The core invariant:

**Round-trip identity:** Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a bit-identical instance tree -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.

Every audit item must be evaluated against this standard. A "works most of the time" finding is not acceptable. If there is ANY code path where an instance name, property, or hierarchy relationship can be lost, mangled, or silently altered through a syncback→rebuild cycle, flag it as critical.

## Code Quality Standard

We are building a maintainable codebase. During audits, watch for duplicated logic -- the same slugify/dedup/meta-update pattern copy-pasted across `change_processor.rs`, `api.rs`, `dir.rs`, `project.rs`, etc. If you find duplicated code that could be consolidated into a shared helper:

- **Small refactor** (extracting a helper function, consolidating 2-3 call sites): flag it AND fix it as part of the audit.
- **Major system rewrite** (restructuring data flow, changing function signatures across 10+ call sites, redesigning the snapshot pipeline): flag it clearly in a "Deferred Refactors" section at the bottom of the report. We will evaluate whether it's worth doing after the audit is complete.

The goal is clean, DRY code that is easy to reason about and hard to get wrong. If the same "compute meta path, check if name needs slugify, upsert/remove name field" sequence appears in 3+ places, that's a refactor candidate.

**Syncback** is the reverse operation of building: it takes a Roblox binary file (rbxl/rbxm) and writes instances back to the filesystem according to project structure.

**Entry point:** `atlas syncback <project> --input <file>`

## Core Concept

```
Roblox File (rbxl/rbxm)
  ↓ Parse with rbx_binary
WeakDom (in-memory instances)
  ↓ Match to project structure
Filesystem (Lua files, models, directories)
```

**Key challenge:** Determining which instances map to which files, and preserving existing file structure when possible.

## Architecture Overview (`src/syncback/`)

### Module Structure

```
syncback/
  mod.rs              - Main loop, orchestration, ref path post-processing
  snapshot.rs         - Convert instances → snapshots, record_ref_path()
  fs_snapshot.rs      - Filesystem representation, fix_ref_paths()
  file_names.rs       - slugify_name(), deduplicate_name_with_ext(), name_for_inst()
  matching.rs         - Recursive change-count matching algorithm
  dedup_suffix.rs     - Dedup cleanup rules (gap-tolerant, base-name promotion, group-to-1)
  meta.rs             - Meta file read/write helpers (upsert/remove name, update ref paths)
  property_filter.rs  - Filter properties per syncback rules
  ref_properties.rs   - Ref property path linking (Rojo_Ref_*, tentative_fs_path)
  hash/               - Content hashing for change detection
    mod.rs            - Hash tree computation
    variant.rs        - Variant-specific hashing
  stats.rs            - Statistics tracking
```

### Main Loop (`syncback_loop`)

**Flow:**

1. **Parse input file** - Read rbxl/rbxm with `rbx_binary`
2. **Collect pre-prune paths** - `collect_all_paths()` maps all Refs to filesystem-name paths before pruning (for external refs)
3. **Prune tree** - Remove instances not in original project
4. **Filter hidden services** - Remove internal Roblox services (AdService, Analytics, etc.)
5. **Collect referents** - `collect_referents()` finds Ref properties and writes `Rojo_Ref_*` attributes using tentative paths
6. **Filter properties** - Apply syncback rules (ignore classes, properties, etc.)
7. **Hash comparison** - Compare hashes of old tree vs new tree
8. **Recursive syncback walk** - Match instances to files via matching algorithm, assign filenames with dedup
9. **Record ref path map** - During the walk, `record_ref_path()` maps each Ref to its final dedup'd filesystem path
10. **Post-process ref paths** - `fix_ref_paths()` compares tentative paths vs final paths, substitutes in written meta/model files
11. **Write filesystem** - Apply FsSnapshot to disk (sequential or parallel)

**Signature:**
```rust
pub fn syncback_loop(
    vfs: &Vfs,
    old_tree: &mut RojoTree,      // Existing project tree
    new_tree: WeakDom,             // Roblox file contents
    project: &Project,             // Project configuration
    incremental: bool,             // Preserve structure vs rebuild
) -> anyhow::Result<FsSnapshot>
```

## Syncback Modes

### Incremental Mode (`incremental: true`)

**Default behavior.** Preserves existing file structure:

- If `src/MyScript.luau` exists, update it
- If `src/MyFolder/init.luau` exists (directory structure), preserve it
- Only creates new files when instances don't match anything
- Compares hashes to detect actual changes (avoids rewriting unchanged files)

**Use case:** Normal workflow where you sync changes from Studio back to disk.

### Clean Mode (`incremental: false`)

**Rebuild from scratch.** Treats all children as new:

- Ignores existing file structure
- Generates new filenames for everything
- Chooses best middleware for each instance
- Useful when project structure changed significantly

**Use case:** Major refactoring, or when incremental mode produces wrong results.

**Implementation detail:** `old_ref` is ignored in `SyncbackSnapshot::with_joined_path` when `incremental=false`.

## Matching Algorithm (`matching.rs`)

Pairs new instances (from Roblox file) with old instances (from filesystem) to determine which files to update vs create.

**Entry point:** `match_children(new_children, old_children, new_dom, old_dom, hashes)`

**Algorithm:**
1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring.
2. **Ambiguous groups:** For groups with multiple candidates on either side, compute pairwise `compute_change_count()` -- a recursive scoring function that counts property diffs + recursive children cost. Greedy assignment sorted by cost ascending, child-order tiebreaker for stability.
3. **Hash fast-path:** If `new_hash == old_hash`, return 0 cost immediately (skips property comparison and recursion).

**Constants:**
- `UNMATCHED_PENALTY = 10,000` - Cost per unmatched child (create/delete is categorically more expensive than property tweaks)
- `MAX_SCORING_DEPTH = 3` - Depth limit for recursive scoring to prevent O(n^k) explosion

**`compute_change_count()` and `match_children()` are mutually recursive.** Base case: leaf nodes (no children).

**Output:** `MatchResult { matched: Vec<(new_ref, old_ref)>, unmatched_new: Vec<new_ref>, unmatched_old: Vec<old_ref> }`

**Known limitation:** Greedy assignment is not globally optimal. The Hungarian algorithm (O(N^3)) would give optimal results, but ambiguous groups are typically small (2-10 instances).

## Dedup Suffix Management (`dedup_suffix.rs`)

Manages `~N` suffix cleanup after instance deletions during two-way sync.

**Key functions:**
- `parse_dedup_suffix(stem) -> Option<(base, N)>` - Parse `~N` from filename stem
- `build_dedup_name(base, suffix, ext)` - Build filename with optional `~N` suffix
- `compute_cleanup_action(removed_stem, remaining_stems) -> DedupCleanupAction` - Determine what to do after a deletion

**Cleanup rules:**
- **Gap-tolerant:** Deleting `~2` from `{Foo, Foo~2, Foo~3}` leaves `{Foo, Foo~3}`. No renumbering.
- **Base-name promotion:** Deleting `Foo` (base) from `{Foo, Foo~2, Foo~3}` promotes `Foo~2` to `Foo`. `Foo~3` stays.
- **Group-to-1 cleanup:** If group shrinks to 1 member, remove its suffix entirely.

## Syncback Rules

Configured in `syncbackRules` section of project file:

```rust
pub struct SyncbackRules {
    /// Roblox tree paths to ignore (e.g., "ServerStorage/Secrets")
    pub ignore_trees: Vec<String>,
    
    /// Filesystem glob patterns to ignore (e.g., "src/client/*")
    pub ignore_paths: Vec<String>,
    
    /// Properties to not sync back per class
    /// Example: {"BasePart": ["Color"], "Script": ["Disabled"]}
    pub ignore_properties: IndexMap<Ustr, Vec<Ustr>>,
    
    /// Instance classes to skip entirely
    pub ignore_classes: Vec<String>,
    
    /// Whether to sync Workspace.CurrentCamera (default: false)
    pub sync_current_camera: Option<bool>,
    
    /// Whether to sync unscriptable properties (default: true)
    pub sync_unscriptable: Option<bool>,
    
    /// Whether to skip Ref properties (default: false)
    pub ignore_referents: Option<bool>,
    
    /// Whether /** in ignore paths also matches the directory itself
    pub create_ignore_dir_paths: Option<bool>,
    
    /// Override root-level ignoreHiddenServices for syncback
    pub ignore_hidden_services: Option<bool>,
    
    /// Warn on duplicate child names during syncback
    pub warn_duplicate_names: Option<bool>,
}
```

**Property filtering** (`property_filter.rs`):
- Filters out `Name`, `Parent` (always)
- Filters out properties in `ignore_properties`
- Filters out unscriptable properties if `sync_unscriptable: false`
- Filters out middleware-specific properties (e.g., `Script.Source` when writing to `.luau` file)

### Hidden Services

**Default behavior (`ignoreHiddenServices: true`):**

Only sync "visible" services:
```rust
const VISIBLE_SERVICES: &[&str] = &[
    "Lighting", "MaterialService", "ReplicatedFirst",
    "ReplicatedStorage", "ServerScriptService", "ServerStorage",
    "SoundService", "StarterGui", "StarterPack", "StarterPlayer",
    "Teams", "TextChatService", "VoiceChatService", "Workspace",
];
```

**Rationale:** Prevents plugin from trying to delete services like `HttpService`, `AnalyticsService`, `Chat` that were never synced to disk.

## Hash-Based Change Detection (`hash/`)

### Why Hashing?

**Problem:** Writing every file on every syncback is slow and creates noise in git diffs.

**Solution:** Hash both old and new instance trees. Only write files where hashes differ.

### Hash Algorithm

Hashing uses **blake3** directly - there's no `HashVariant` struct.

```rust
/// Returns HashMap<Ref, Hash> mapping each instance to its content hash
pub fn hash_tree(project: &Project, dom: &WeakDom, root_ref: Ref) -> HashMap<Ref, Hash>
```

**Hashing strategy:**
1. Process instances bottom-up (children first)
2. Hash class name, then filtered properties
3. Sort and hash children's hashes
4. Finalize with blake3

**Special cases:**
- Ref properties are hashed as paths (since Refs change between trees)
- Some properties are normalized (e.g., line endings in scripts)
- Unscriptable properties can be excluded from hash

### Hash Comparison

```rust
if old_hash == new_hash {
    // No changes - skip writing file
    return Ok(None);
} else {
    // Changed - write file
    write_instance_to_file(snapshot)?;
}
```

## SyncbackSnapshot (`snapshot.rs`)

**Core type representing instance to sync back:**

```rust
pub struct SyncbackSnapshot<'sync> {
    pub data: SyncbackData<'sync>,  // Context (VFS, trees, project)
    pub old: Option<Ref>,            // Old tree instance (if exists)
    pub new: Ref,                    // New tree instance (Roblox file)
    pub path: PathBuf,               // Target filesystem path
    pub middleware: Option<Middleware>, // How to write (Lua, Rbxm, etc.)
    pub needs_meta_name: bool,       // Filesystem name differs from instance name
}
```

### Snapshot Creation

**Methods:**

```rust
// Join child to current path
snapshot.with_joined_path(new_ref, old_ref)?
// Result: parent_path/ChildName.luau

// Use base path
snapshot.with_base_path(&base, new_ref, old_ref)?
// Result: base/ChildName.luau

// Explicit path
snapshot.with_new_path(path, new_ref, old_ref)
// Result: path (exactly as provided)
```

**Filename generation** (`file_names.rs`):
- `slugify_name()`: forbidden chars (`<>:"/\|?*~`) → `_`. Also strips leading/trailing spaces/dots, neutralizes dangerous suffixes (`.server`, `.meta`, etc.), handles Windows reserved names (`CON`, `PRN`, etc.)
- `deduplicate_name_with_ext()`: Dedup key = full filesystem name (slug + extension), case-insensitive. `Folder "Foo"` (key `"foo"`) and `Script "Foo"` (key `"foo.server.luau"`) do NOT collide. Appends `~2`, `~3`, etc. on collision, with suffix inserted before compound extension: `Foo~2.server.luau`
- `name_for_inst()` returns `(filename, needs_meta_name, dedup_key)`. `needs_meta_name` is true when the filesystem name differs from the real instance name.
- `~` is in `SLUGIFY_CHARS` so natural tilde names (e.g., `"Foo~1"`) get slugified to `"Foo_1"`, avoiding confusion with dedup suffixes.
- Example: `Script` named "My Script" → `My Script.server.luau`
- Example: `Script` named "Hey/Bro" → `Hey_Bro.server.luau` + meta with `name: "Hey/Bro"`
- Example: Two `Folder "Data"` → `Data/` + `Data~2/` (meta: `name: "Data"`)

### Property Filtering

```rust
snapshot.get_path_filtered_properties(new_ref)
```

**Filters out:**
- `Source` property for scripts (content goes in file body)
- `Value` property for StringValue (content goes in `.txt` file)
- Properties specified in `ignoreProperties` rules
- Name/Parent (structural, not data)

## Middleware Selection

**Decision tree:**

1. **Old instance exists?** → Use same middleware as before (preserve structure)
2. **Script type?** → Choose appropriate Lua middleware (`.server.luau`, `.client.luau`, etc.)
3. **Model/container?** → Choose directory or rbxm based on complexity
4. **Special types?** → CSV for LocalizationTable, TXT for StringValue
5. **Fallback** → Rbxm or JsonModel

**Debug override:** `ROJO_SYNCBACK_DEBUG` environment variable
- `1` = Use Rbxmx instead of Rbxm
- `2` = Use JsonModel

## Filesystem Snapshot (`fs_snapshot.rs`)

**Represents filesystem operations to perform:**

```rust
pub struct FsSnapshot {
    added_files: HashMap<PathBuf, Vec<u8>>,   // New files to create
    added_dirs: HashSet<PathBuf>,             // New directories
    removed_files: HashSet<PathBuf>,          // Files to delete
    removed_dirs: HashSet<PathBuf>,           // Directories to delete
}
```

**Key methods:**
```rust
// Create snapshot with operations
fs_snapshot.with_added_file(path, contents)
fs_snapshot.with_added_dir(path)
fs_snapshot.merge(other)  // Combine snapshots

// Apply to filesystem
fs_snapshot.write_to_vfs(base_path, vfs)?;
fs_snapshot.write_to_vfs_parallel(base_path, vfs)?;  // Parallel writes with retry logic

// Post-process ref paths
fs_snapshot.fix_ref_paths(&substitutions);  // Replace tentative paths with final dedup'd paths

// Query
fs_snapshot.is_empty()
fs_snapshot.added_paths()
fs_snapshot.removed_paths()
```

## Ref Properties (`ref_properties.rs`)

**Challenge:** Ref properties (like `ObjectValue.Value`, `Model.PrimaryPart`) point to other instances by Ref, but Refs are different between old and new trees.

**Solution: Path-based `Rojo_Ref_*` attributes (preferred)**

Syncback writes filesystem-name-based paths as `Rojo_Ref_PropertyName` attributes:

```json5
{
    "properties": {},
    "attributes": {
        "Rojo_Ref_PrimaryPart": "Workspace/Handle.model.json5"
    }
}
```

Forward sync reads `Rojo_Ref_*` attributes and resolves them to tree instances via `get_instance_by_path()`.

**Legacy: `Rojo_Target_*` / `$id` system (still supported for reading)**

```json5
{
    "id": "some-unique-id",
    "attributes": { "Rojo_Target_Value": "some-unique-id" }
}
```

Forward sync also reads `Rojo_Target_*` attributes and resolves via `get_specified_id()`. Priority: `Rojo_Ref_*` takes precedence over `Rojo_Target_*`.

**Key functions:**
- `collect_all_paths(dom)` - Maps all Refs to filesystem-name-compatible paths (called pre-prune for external refs)
- `collect_referents(dom, paths, final_paths)` - Finds all Ref properties, stores as (source, target_path) pairs
- `tentative_fs_path(dom, ref)` - Computes pre-dedup path: slugified name + extension based on class/children
- `tentative_fs_path_public()` - Public wrapper used in post-processing
- `link_referents(dom, referents)` - Writes `Rojo_Ref_*` attributes to instances

**Ref path post-processing (in `mod.rs`):**

During the syncback walk, `record_ref_path()` records each instance's final dedup'd filesystem path in `ref_path_map`. After the walk, `fix_ref_paths()` compares tentative paths to final paths and substitutes in all written meta/model files. Substitutions are sorted by length descending to prevent partial prefix matches.

## Meta File Helpers (`meta.rs`)

Shared helpers for reading/writing `.meta.json5` and `.model.json5` files:

- `upsert_meta_name(path, name)` - Upsert `name` field in `.meta.json5`
- `remove_meta_name(path)` - Remove `name` field; returns `RemoveNameOutcome` (NoOp, FieldRemoved, FileDeleted)
- `upsert_model_name(path, name)` - Upsert `name` field in `.model.json5`
- `update_ref_paths_in_file(path, old_prefix, new_prefix)` - Update `Rojo_Ref_*` attribute paths in a meta/model file

## Statistics Tracking (`stats.rs`)

```rust
pub struct SyncbackStats {
    duplicate_name_count: AtomicUsize,     // Instances skipped (name collision)
    rbxm_fallback_count: AtomicUsize,      // Instances using rbxm fallback
    unknown_class_count: AtomicUsize,      // Unknown classes encountered
    unknown_property_count: AtomicUsize,   // Unknown properties encountered
    unknown_classes: Mutex<HashSet<String>>,
    unknown_properties: Mutex<HashSet<String>>,
}
```

**Key methods:**
```rust
stats.record_duplicate_name(path, name)
stats.record_rbxm_fallback(path, reason)
stats.record_unknown_class(class_name)
stats.record_unknown_property(class, property)
stats.has_issues()         // Any warnings?
stats.has_unknown_types()  // Unknown classes/properties?
stats.log_summary()        // Log final report
```

**Logged at end of syncback:**
```
Syncback complete. Warnings: 2 duplicate names, 1 unknown class
```

## Common Patterns

### Syncing a Script

**Roblox file has:**
```
Script "MyScript"
  Source = "print('hello')"
  Disabled = false
```

**Syncback produces:**
```luau
-- src/MyScript.server.luau
print('hello')
```

**Meta file (if needed):**
```json5
// src/MyScript.server.meta.json5 (for MyScript.server.luau)
{
    "properties": {
        "Disabled": false
    }
}
```

### Syncing a Model

**Simple model (few children):** → `.rbxm` or `.rbxmx` file
**Complex model (many children):** → Directory with `init.meta.json5` or `init.luau`

### Syncing Nested Folders

```
Roblox:
  Folder "Modules"
    ModuleScript "Utils"
    Folder "Math"
      ModuleScript "Add"

Filesystem:
  src/Modules/
    Utils.luau
    Math/
      Add.luau
```

## Error Handling

**Common errors:**

1. **Middleware mismatch** - Instance type changed but file structure can't accommodate
   - Solution: Use clean mode
   
2. **Invalid filename** - Instance name contains invalid chars
   - Solution: Handled automatically by `slugify_name()` + metadata `name` field
   
3. **Ref to pruned instance** - Property points to instance not in project
   - Logged as warning, property not synced
   
4. **Write permission** - Can't write to file
   - Error reported, syncback continues for other files

**Error propagation:** Errors are propagated via `anyhow::bail!`, not collected in stats. Fatal errors abort the syncback; non-fatal issues are logged as warnings.

## Integration with Plugin

**Plugin does NOT handle syncback.** It only handles:
1. Downloading place file from Studio
2. Sending file to server
3. Server runs syncback command

**Two-way sync** is different - see `atlas-api.mdc`.

## Performance Considerations

**Hashing overhead:** Hashing is fast (~50µs per instance), negligible compared to disk I/O.

**Incremental benefit:** On large projects, incremental mode with hashing can be 10-100x faster than rewriting everything.

**Parallelization:** File writes can be parallelized via `write_to_vfs_parallel()` (includes Windows retry logic).

## Debugging

**Enable verbose logging:**
```bash
RUST_LOG=rojo::syncback=debug atlas syncback project/ --input place.rbxl
```

**Shows:**
- Which instances are being processed
- Hash comparisons
- File write decisions
- Middleware selection

**Test syncback:**
```bash
# Syncback once
atlas syncback project/ --input place.rbxl

# Build back to verify roundtrip
atlas build project/ -o rebuilt.rbxl

# Compare
diff place.rbxl rebuilt.rbxl
```

## Future Improvements

**From code comments:**
- Better detection of when root is directory vs file
- More sophisticated middleware selection
- Better handling of Roblox format changes
- Support for partial syncback (only changed instances)
- Globally optimal matching (Hungarian algorithm) for large ambiguous groups
- Matching cache (parent_ref -> match_assignments) for avoiding redundant scoring

## Key Files to Reference

- `src/syncback/mod.rs` - Main orchestration loop, ref path post-processing
- `src/syncback/snapshot.rs` - SyncbackSnapshot implementation
- `src/syncback/matching.rs` - Recursive change-count matching algorithm
- `src/syncback/file_names.rs` - slugify_name(), deduplicate_name_with_ext()
- `src/syncback/dedup_suffix.rs` - Dedup cleanup rules
- `src/syncback/meta.rs` - Meta file read/write helpers
- `src/syncback/ref_properties.rs` - Ref property path linking
- `src/syncback/hash/` - Hashing algorithm
- `src/syncback/property_filter.rs` - Property filtering logic
- `src/cli/syncback.rs` - CLI command implementation