hakowork/rojo
3
0
Fork 0
mirror of https://github.com/rojo-rbx/rojo.git synced 2026-10-22 21:41:37 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update submodule and implement parallel I/O audit fixes

Browse source 
- Updated the `vscode-rojo` submodule to the latest commit.
- Added a new plan for addressing issues found during the audit of parallelization and optimization commits, including fixes for a data loss bug in the orphan scanner, a duplicate-children bug in the prefetch cache, and a nested-detection correctness issue.
- Implemented a small DRY refactor by extracting a shared helper function for UNC path stripping.
- Enhanced the `syncback` logic to ensure hidden directories are preserved during clean mode operations and added tests to validate these changes.

These updates aim to improve the reliability and correctness of the parallel I/O functionality while maintaining code clarity and performance.
This commit is contained in:
Joe 2026-02-19 04:36:44 -06:00
parent 1c25b928a8
commit 43834f357e
6 changed files with 536 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

267
.cursor/plans/parallel_io_audit_fixes_35f49f83.plan.md Normal file
View file

@ -0,0 +1,267 @@
---
name: Parallel IO audit fixes
overview: "Fix three issues found during audit of the parallelization/optimization commits: a data loss bug in the orphan scanner, a duplicate-children bug in the prefetch cache, and a nested-detection correctness issue, plus a small DRY refactor."
todos:
- id: fix1-hidden-dirs
content: Fix orphan scanner walkdir to prune hidden directories using filter_entry()
status: completed
- id: fix2-dedup-roots
content: Deduplicate overlapping $path roots in prefetch_project_files before walking
status: completed
- id: fix3-nested-detection
content: Replace broken binary search nested detection with O(n log n) ancestor-tracking approach
status: completed
- id: fix4-unc-refactor
content: Extract shared strip_unc_prefix helper from 3 inline copies
status: completed
- id: test-hidden-dir
content: Add integration test for hidden directory survival in clean syncback
status: completed
- id: test-overlap
content: Add test for overlapping $path roots with prefetch cache
status: completed
- id: test-nested
content: Add unit test for nested path detection correctness
status: completed
- id: run-ci
content: Run full CI pipeline (cargo test, clippy, fmt)
status: completed
isProject: false
---
# Parallel I/O Audit Fixes
> This plan was generated by `/audit` from a Plan-mode session.
> Refer to `.cursor/rules/atlas.mdc` for full project standards.
## Standards
These standards govern every fix in this plan. An implementer MUST read `.cursor/rules/atlas.mdc` before starting.
### Round-Trip Identity (from `atlas.mdc` - Quality Standard)
Syncback (or two-way sync) writes a directory tree. Building an rbxl from that directory tree and forward-syncing it back must produce a **bit-identical instance tree** -- same names, same classes, same properties, same hierarchy, same ref targets. Any deviation is a bug.
### CLI Syncback Parity (from `atlas.mdc` - Two-Way Sync Strategy)
Plugin-based sync must produce **exactly the same filesystem output** that `atlas syncback` would give for the same input. Byte-for-byte identical files, identical directory structures, identical naming. Any divergence is a bug until proven otherwise.
### Code Quality (from `atlas.mdc` - Code Quality Standard)
DRY code that is easy to reason about and hard to get wrong. The same slugify/dedup/meta-update pattern should not be copy-pasted across files. Small refactors (2-3 call sites) are included as fixes. Major rewrites (10+ call sites) are deferred.
## Context
Audited the last 7 commits on the current branch (from `1ffed121` through `56e1ab4a`), which add:
- VFS prefetch cache for parallel file reading at startup
- Parallel snapshot building via rayon in `dir.rs`
- Parallel RefPathIndex population via rayon + walkdir
- Parallel tree hashing in syncback
- Orphan scanner rewrite using walkdir
- Placeholder-based ref path substitution optimization
- ServeCommand refactor and dead code cleanup
## Fixes
### Fix 1: Prune hidden directories from walkdir orphan scanner
- **Status:** Approved
- **Finding:** Orphan scanner walkdir replacement doesn't prune hidden directories
- **Files:** `[src/syncback/mod.rs](src/syncback/mod.rs)`
- **Problem:** Lines ~406-424. The `walkdir::WalkDir` iterator yields ALL entries including children of hidden directories. The hidden-name filter only checks `file_name().starts_with('.')`, so `settings.json` inside `.vscode/` passes the filter and gets added to `existing_paths`. In clean mode, these files would be deleted.
The old recursive `scan_directory` function skipped hidden directories AND never recursed into them. The walkdir replacement must replicate this behavior.
- **Solution:** Replace the flat `.flatten()` + post-hoc hidden check with `filter_entry()` to prune the walk at hidden directories:
```rust
for entry in walkdir::WalkDir::new(&canonical_dir)
.follow_links(true)
.into_iter()
.filter_entry(|e| {
if e.depth() == 0 {
return true;
}
e.file_name()
.to_str()
.map_or(true, |n| !n.starts_with('.') || n == ".gitkeep")
})
.flatten()
{
if entry.depth() == 0 {
continue;
}
let path = entry.path().to_path_buf();
if !is_valid_path(&ignore_patterns, project_path, &path) {
continue;
}
// Hidden check no longer needed here -- filter_entry handles it
paths.insert(path);
}
```
`filter_entry` controls which directories walkdir descends into. When it returns `false` for a directory, walkdir skips the directory AND all its descendants. This exactly matches the old recursive behavior.
- **Risk:** Low -- only touches the orphan scanner loop. `filter_entry` is walkdir's standard pruning mechanism.
- **Verify round-trip:** Clean mode syncback with a `.vscode/` directory inside a `$path` source dir should NOT delete `.vscode/` contents.
- **Verify syncback parity:** N/A -- this only affects clean mode orphan removal, not the sync pipeline.
- **Tests required:**
- Integration test: create a temp project with `src/.hidden/keep.txt`, run clean-mode syncback, verify `.hidden/keep.txt` survives.
### Fix 2: Deduplicate overlapping roots in prefetch cache
- **Status:** Approved
- **Finding:** Overlapping `$path` roots cause duplicate children in prefetch cache
- **Files:** `[src/serve_session.rs](src/serve_session.rs)`
- **Problem:** Lines ~136-156 in `prefetch_project_files`. `collect_path_roots` collects ALL `$path` entries including nested ones. For a project with `$path: "src"` and `$path: "src/shared"`, walkdir walks `src/shared` twice -- once as part of `src` and once as its own root. The `children_map` accumulates duplicate entries for `src/shared`'s children. When consumed by VFS `read_dir`, these duplicates cause duplicate instance snapshots.
- **Solution:** After collecting roots in `prefetch_project_files`, deduplicate by removing any root that is a descendant of another root:
```rust
let mut roots: Vec<std::path::PathBuf> = Vec::new();
collect_path_roots(&project.tree, folder, &mut roots);
// Remove roots that are descendants of other roots to avoid
// walking the same directory twice and producing duplicate children.
roots.sort();
let mut deduped: Vec<PathBuf> = Vec::new();
for root in roots {
if !deduped.iter().any(|existing| root.starts_with(existing)) {
deduped.push(root);
}
}
let roots = deduped;
```
After sorting, a parent path always sorts before its descendants. By iterating in order and only keeping roots that are NOT descendants of already-kept roots, we eliminate all overlaps in O(n^2) worst case but typically O(n) since most projects have few `$path` entries.
- **Risk:** Low -- only modifies which directories are walked during prefetch. The dedup logic is simple and well-understood.
- **Verify round-trip:** Build a project with overlapping `$path` entries. Verify snapshot tree has no duplicate instances. Run build 5 times, verify deterministic output.
- **Verify syncback parity:** N/A -- prefetch cache only affects forward sync initial load.
- **Tests required:**
- Unit test in `tests/tests/build.rs`: create a project with `$path: "src"` and a child with `$path: "src/shared"`. Run `snapshot_from_vfs` with prefetch cache. Verify no duplicate children under the `shared` directory instance.
### Fix 3: Fix nested path detection in orphan removal
- **Status:** Approved (O(n log n) approach)
- **Finding:** Binary search nested detection misses sibling paths
- **Files:** `[src/syncback/mod.rs](src/syncback/mod.rs)`
- **Problem:** Lines ~869-888. The optimization uses `binary_search` to find the current path's position, then checks only `sorted_removals[idx - 1]`. This misses cases where a sibling path sits between the parent and the current path in sorted order.
Example: `sorted = ["a/b", "a/b/c", "a/b/d"]`. For `a/b/d` at idx 2, the code checks `sorted[1] = "a/b/c"` (a sibling, not an ancestor). The actual ancestor `"a/b"` at idx 0 is not checked.
- **Solution:** Replace binary search with a sorted single-pass using an `current_ancestor` tracker:
```rust
let mut sorted_removals: Vec<_> = paths_to_remove.iter().cloned().collect();
sorted_removals.sort();
let mut current_ancestor: Option<&PathBuf> = None;
for old_path in &sorted_removals {
if let Some(ancestor) = current_ancestor {
if old_path.starts_with(ancestor) && old_path != ancestor {
continue; // nested under current ancestor, skip
}
}
current_ancestor = Some(old_path);
let relative_path = old_path
.strip_prefix(&canonical_project_path)
.unwrap_or(old_path);
log::debug!("Removing orphaned path: {}", relative_path.display());
if old_path.is_dir() {
fs_snapshot.remove_dir(relative_path);
} else {
fs_snapshot.remove_file(relative_path);
}
}
```
In sorted path order, all descendants of a path appear consecutively after it (because `Path` comparison is component-based). So a single variable tracking the current top-level ancestor is sufficient. When we encounter a non-descendant, it becomes the new ancestor. This is O(n log n) for the sort + O(n) for the walk.
This also removes the intermediate `paths_vec` and the `binary_search` code, simplifying the function.
- **Risk:** Low -- the logic is simpler than what it replaces. The correctness argument is straightforward: sorted paths guarantee all descendants appear consecutively.
- **Verify round-trip:** Clean mode syncback with nested directories. Verify parent-only removal (no redundant child removals).
- **Verify syncback parity:** N/A -- orphan removal is internal to clean mode.
- **Tests required:**
- Unit test: create paths `["a/b", "a/b/c", "a/b/d", "x/y", "x/y/z"]`, verify only `"a/b"` and `"x/y"` are top-level removals.
### Fix 4: Extract shared `strip_unc_prefix` helper
- **Status:** Approved
- **Finding:** Duplicated UNC prefix stripping logic in 3 locations
- **Files:** `[src/syncback/mod.rs](src/syncback/mod.rs)`
- **Problem:** The same UNC prefix stripping pattern appears at:
1. Lines ~396-404 (orphan scanner: canonicalize `dirs_to_scan` entries)
2. Lines ~649-656 (`strip_unc_prefix` helper definition inside orphan removal block)
3. Lines ~703-711 (inline canonicalize in `collect_all_path_refs_and_mappings`)
Location 2 defines a proper `fn strip_unc_prefix`, but locations 1 and 3 inline the same logic.
- **Solution:** Move `strip_unc_prefix` to module level (or just above the clean-mode block), and replace all 3 inline instances with calls to it. The function is already defined at location 2:
```rust
fn strip_unc_prefix(p: PathBuf) -> PathBuf {
let path_str = p.to_string_lossy();
if path_str.starts_with(r"\\?\") || path_str.starts_with("//?/") {
PathBuf::from(&path_str[4..])
} else {
p
}
}
```
Move this definition before the `existing_paths` block (around line ~258), then replace the inline code at lines ~396-404 with `strip_unc_prefix(std::fs::canonicalize(dir).unwrap_or_else(|_| dir.clone()))` and at lines ~703-711 with `strip_unc_prefix(p)`.
- **Risk:** Zero -- pure refactor, no behavior change.
- **Verify round-trip:** N/A -- refactor only.
- **Verify syncback parity:** N/A -- refactor only.
- **Tests required:** None -- existing tests cover this code path.
## Skipped Fixes
### Skipped: Serve command loads project twice
- **Finding:** `src/cli/serve.rs` loads the project file via `Project::load_initial_project` for port/address, then `ServeSession::new` loads it again in `init_tree`.
- **Problem:** Minor performance -- project file read twice from disk at startup.
- **Reason skipped:** Acceptable overhead. Project files are small and this only happens once.
### Skipped: Commented-out git config in init.rs
- **Finding:** Four commented-out git config lines in `src/cli/init.rs` lines 134-137.
- **Reason skipped:** User indicated these are planned for future enablement.
## Accepted Limitations
- **Limitation:** Symlinks within project `$path` directories may cause path mismatches
- **Scenario:** If a symlink inside a `$path` source directory points elsewhere, walkdir-collected paths won't match `added_paths` during clean-mode orphan removal. Files behind the symlink could be incorrectly removed.
- **User decision:** Accepted for this release
- **Mitigation:** Symlinks within source directories are not a supported configuration in Roblox development. Root-level symlinks (project folder itself) still work because `canonicalize` is called on the root.
## Deferred Refactors
None identified beyond the small `strip_unc_prefix` extraction (included as Fix 4).
## Test Plan
### Rust Integration Tests (`tests/tests/`)
- **Hidden directory survival in clean syncback** (Fix 1): Create temp project with `src/.hidden/keep.txt`, run clean-mode syncback, assert `.hidden/keep.txt` still exists after syncback.
- **Overlapping `$path` roots don't create duplicates** (Fix 2): Create project with overlapping `$path` entries (`"src"` and `"src/shared"`), run `snapshot_from_vfs` with prefetch cache, verify no duplicate children.
### Rust Unit Tests
- **Nested path detection correctness** (Fix 3): Test the ancestor-tracking algorithm with paths `["a/b", "a/b/c", "a/b/d", "x/y", "x/y/z"]`, verify only `"a/b"` and `"x/y"` are emitted as top-level.
## Test Rules
- Every bug fix MUST have a test that would have FAILED before the fix and PASSES after.
- Fix 4 (refactor) does not need new tests -- existing test coverage is sufficient.
## Final Step: Run CI
After ALL fixes and tests are implemented, run the full CI pipeline (`cargo test --locked`, `cargo clippy`, `cargo fmt --check`). Every fix must pass CI before the plan is considered complete.