rojo/.cursor/plans/parallel_io_audit_fixes_35f49f83.plan.md

name	overview	todos	isProject
Parallel IO audit fixes	Fix three issues found during audit of the parallelization/optimization commits: a data loss bug in the orphan scanner, a duplicate-children bug in the prefetch cache, and a nested-detection correctness issue, plus a small DRY refactor.	id content status fix1-hidden-dirs Fix orphan scanner walkdir to prune hidden directories using filter_entry() completed id content status fix2-dedup-roots Deduplicate overlapping $path roots in prefetch_project_files before walking completed id content status fix3-nested-detection Replace broken binary search nested detection with O(n log n) ancestor-tracking approach completed id content status fix4-unc-refactor Extract shared strip_unc_prefix helper from 3 inline copies completed id content status test-hidden-dir Add integration test for hidden directory survival in clean syncback completed id content status test-overlap Add test for overlapping $path roots with prefetch cache completed id content status test-nested Add unit test for nested path detection correctness completed id content status run-ci Run full CI pipeline (cargo test, clippy, fmt) completed	false

Parallel I/O Audit Fixes

This plan was generated by /audit from a Plan-mode session. Refer to .cursor/rules/atlas.mdc for full project standards.

Standards

These standards govern every fix in this plan. An implementer MUST read .cursor/rules/atlas.mdc before starting.

Round-Trip Identity (from atlas.mdc - Quality Standard)

Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a bit-identical instance tree -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.

CLI Syncback Parity (from atlas.mdc - Two-Way Sync Strategy)

Plugin-based sync must produce exactly the same filesystem output that atlas syncback would give for the same input. Byte-for-byte identical files, identical directory structures, identical naming. Any divergence is a bug until proven otherwise.

Code Quality (from atlas.mdc - Code Quality Standard)

DRY code that is easy to reason about and hard to get wrong. The same slugify/dedup/meta-update pattern should not be copy-pasted across files. Small refactors (2-3 call sites) are included as fixes. Major rewrites (10+ call sites) are deferred.

Context

Audited the last 7 commits on the current branch (from 1ffed121 through 56e1ab4a), which add:

• VFS prefetch cache for parallel file reading at startup
• Parallel snapshot building via rayon in dir.rs
• Parallel RefPathIndex population via rayon + walkdir
• Parallel tree hashing in syncback
• Orphan scanner rewrite using walkdir
• Placeholder-based ref path substitution optimization
• ServeCommand refactor and dead code cleanup

Fixes

Fix 1: Prune hidden directories from walkdir orphan scanner

• Status: Approved
• Finding: Orphan scanner walkdir replacement doesn't prune hidden directories
• Files: [src/syncback/mod.rs](src/syncback/mod.rs)
• Problem: Lines ~406-424. The walkdir::WalkDir iterator yields ALL entries including children of hidden directories. The hidden-name filter only checks file_name().starts_with('.'), so settings.json inside .vscode/ passes the filter and gets added to existing_paths. In clean mode, these files would be deleted. The old recursive scan_directory function skipped hidden directories AND never recursed into them. The walkdir replacement must replicate this behavior.
• Solution: Replace the flat .flatten() + post-hoc hidden check with filter_entry() to prune the walk at hidden directories:

  for entry in walkdir::WalkDir::new(&canonical_dir)
      .follow_links(true)
      .into_iter()
      .filter_entry(|e| {
          if e.depth() == 0 {
              return true;
          }
          e.file_name()
              .to_str()
              .map_or(true, |n| !n.starts_with('.') || n == ".gitkeep")
      })
      .flatten()
  {
      if entry.depth() == 0 {
          continue;
      }
      let path = entry.path().to_path_buf();
      if !is_valid_path(&ignore_patterns, project_path, &path) {
          continue;
      }
      // Hidden check no longer needed here -- filter_entry handles it
      paths.insert(path);
  }
  

filter_entry controls which directories walkdir descends into. When it returns false for a directory, walkdir skips the directory AND all its descendants. This exactly matches the old recursive behavior.

• Risk: Low -- only touches the orphan scanner loop. filter_entry is walkdir's standard pruning mechanism.
• Verify round-trip: Clean mode syncback with a .vscode/ directory inside a $path source dir should NOT delete .vscode/ contents.
• Verify syncback parity: N/A -- this only affects clean mode orphan removal, not the sync pipeline.
• Tests required:
  • Integration test: create a temp project with src/.hidden/keep.txt, run clean-mode syncback, verify .hidden/keep.txt survives.

Fix 2: Deduplicate overlapping roots in prefetch cache

• Status: Approved
• Finding: Overlapping $path roots cause duplicate children in prefetch cache
• Files: [src/serve_session.rs](src/serve_session.rs)
• Problem: Lines ~136-156 in prefetch_project_files. collect_path_roots collects ALL $path entries including nested ones. For a project with $path: "src" and $path: "src/shared", walkdir walks src/shared twice -- once as part of src and once as its own root. The children_map accumulates duplicate entries for src/shared's children. When consumed by VFS read_dir, these duplicates cause duplicate instance snapshots.
• Solution: After collecting roots in prefetch_project_files, deduplicate by removing any root that is a descendant of another root:

  let mut roots: Vec<std::path::PathBuf> = Vec::new();
  collect_path_roots(&project.tree, folder, &mut roots);

  // Remove roots that are descendants of other roots to avoid
  // walking the same directory twice and producing duplicate children.
  roots.sort();
  let mut deduped: Vec<PathBuf> = Vec::new();
  for root in roots {
      if !deduped.iter().any(|existing| root.starts_with(existing)) {
          deduped.push(root);
      }
  }
  let roots = deduped;
  

After sorting, a parent path always sorts before its descendants. By iterating in order and only keeping roots that are NOT descendants of already-kept roots, we eliminate all overlaps in O(n^2) worst case but typically O(n) since most projects have few $path entries.

• Risk: Low -- only modifies which directories are walked during prefetch. The dedup logic is simple and well-understood.
• Verify round-trip: Build a project with overlapping $path entries. Verify snapshot tree has no duplicate instances. Run build 5 times, verify deterministic output.
• Verify syncback parity: N/A -- prefetch cache only affects forward sync initial load.
• Tests required:
  • Unit test in tests/tests/build.rs: create a project with $path: "src" and a child with $path: "src/shared". Run snapshot_from_vfs with prefetch cache. Verify no duplicate children under the shared directory instance.

Fix 3: Fix nested path detection in orphan removal

• Status: Approved (O(n log n) approach)
• Finding: Binary search nested detection misses sibling paths
• Files: [src/syncback/mod.rs](src/syncback/mod.rs)
• Problem: Lines ~869-888. The optimization uses binary_search to find the current path's position, then checks only sorted_removals[idx - 1]. This misses cases where a sibling path sits between the parent and the current path in sorted order. Example: sorted = ["a/b", "a/b/c", "a/b/d"]. For a/b/d at idx 2, the code checks sorted[1] = "a/b/c" (a sibling, not an ancestor). The actual ancestor "a/b" at idx 0 is not checked.
• Solution: Replace binary search with a sorted single-pass using an current_ancestor tracker:

  let mut sorted_removals: Vec<_> = paths_to_remove.iter().cloned().collect();
  sorted_removals.sort();

  let mut current_ancestor: Option<&PathBuf> = None;
  for old_path in &sorted_removals {
      if let Some(ancestor) = current_ancestor {
          if old_path.starts_with(ancestor) && old_path != ancestor {
              continue; // nested under current ancestor, skip
          }
      }
      current_ancestor = Some(old_path);

      let relative_path = old_path
          .strip_prefix(&canonical_project_path)
          .unwrap_or(old_path);

      log::debug!("Removing orphaned path: {}", relative_path.display());
      if old_path.is_dir() {
          fs_snapshot.remove_dir(relative_path);
      } else {
          fs_snapshot.remove_file(relative_path);
      }
  }
  

In sorted path order, all descendants of a path appear consecutively after it (because Path comparison is component-based). So a single variable tracking the current top-level ancestor is sufficient. When we encounter a non-descendant, it becomes the new ancestor. This is O(n log n) for the sort + O(n) for the walk.

This also removes the intermediate paths_vec and the binary_search code, simplifying the function.

• Risk: Low -- the logic is simpler than what it replaces. The correctness argument is straightforward: sorted paths guarantee all descendants appear consecutively.
• Verify round-trip: Clean mode syncback with nested directories. Verify parent-only removal (no redundant child removals).
• Verify syncback parity: N/A -- orphan removal is internal to clean mode.
• Tests required:
  • Unit test: create paths ["a/b", "a/b/c", "a/b/d", "x/y", "x/y/z"], verify only "a/b" and "x/y" are top-level removals.

Fix 4: Extract shared strip_unc_prefix helper

• Status: Approved
• Finding: Duplicated UNC prefix stripping logic in 3 locations
• Files: [src/syncback/mod.rs](src/syncback/mod.rs)
• Problem: The same UNC prefix stripping pattern appears at:
  1. Lines ~396-404 (orphan scanner: canonicalize dirs_to_scan entries)
  2. Lines ~649-656 (strip_unc_prefix helper definition inside orphan removal block)
  3. Lines ~703-711 (inline canonicalize in collect_all_path_refs_and_mappings) Location 2 defines a proper fn strip_unc_prefix, but locations 1 and 3 inline the same logic.
• Solution: Move strip_unc_prefix to module level (or just above the clean-mode block), and replace all 3 inline instances with calls to it. The function is already defined at location 2:

  fn strip_unc_prefix(p: PathBuf) -> PathBuf {
      let path_str = p.to_string_lossy();
      if path_str.starts_with(r"\\?\") || path_str.starts_with("//?/") {
          PathBuf::from(&path_str[4..])
      } else {
          p
      }
  }
  

Move this definition before the existing_paths block (around line ~258), then replace the inline code at lines ~396-404 with strip_unc_prefix(std::fs::canonicalize(dir).unwrap_or_else(|_| dir.clone())) and at lines ~703-711 with strip_unc_prefix(p).

• Risk: Zero -- pure refactor, no behavior change.
• Verify round-trip: N/A -- refactor only.
• Verify syncback parity: N/A -- refactor only.
• Tests required: None -- existing tests cover this code path.

Skipped Fixes

Skipped: Serve command loads project twice

• Finding: src/cli/serve.rs loads the project file via Project::load_initial_project for port/address, then ServeSession::new loads it again in init_tree.
• Problem: Minor performance -- project file read twice from disk at startup.
• Reason skipped: Acceptable overhead. Project files are small and this only happens once.

Skipped: Commented-out git config in init.rs

• Finding: Four commented-out git config lines in src/cli/init.rs lines 134-137.
• Reason skipped: User indicated these are planned for future enablement.

Accepted Limitations

• Limitation: Symlinks within project $path directories may cause path mismatches
• Scenario: If a symlink inside a $path source directory points elsewhere, walkdir-collected paths won't match added_paths during clean-mode orphan removal. Files behind the symlink could be incorrectly removed.
• User decision: Accepted for this release
• Mitigation: Symlinks within source directories are not a supported configuration in Roblox development. Root-level symlinks (project folder itself) still work because canonicalize is called on the root.

Deferred Refactors

None identified beyond the small strip_unc_prefix extraction (included as Fix 4).

Test Plan

Rust Integration Tests (tests/tests/)

• Hidden directory survival in clean syncback (Fix 1): Create temp project with src/.hidden/keep.txt, run clean-mode syncback, assert .hidden/keep.txt still exists after syncback.
• Overlapping $path roots don't create duplicates (Fix 2): Create project with overlapping $path entries ("src" and "src/shared"), run snapshot_from_vfs with prefetch cache, verify no duplicate children.

Rust Unit Tests

• Nested path detection correctness (Fix 3): Test the ancestor-tracking algorithm with paths ["a/b", "a/b/c", "a/b/d", "x/y", "x/y/z"], verify only "a/b" and "x/y" are emitted as top-level.

Test Rules

• Every bug fix MUST have a test that would have FAILED before the fix and PASSES after.
• Fix 4 (refactor) does not need new tests -- existing test coverage is sufficient.

Final Step: Run CI

After ALL fixes and tests are implemented, run the full CI pipeline (cargo test --locked, cargo clippy, cargo fmt --check). Every fix must pass CI before the plan is considered complete.