hakowork/rojo
3
0
Fork 0
mirror of https://github.com/rojo-rbx/rojo.git synced 2026-10-23 05:51:39 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Implement auto-download feature for syncback command

Browse source 
- Enhanced the `syncback` command to automatically download Roblox place files from the asset delivery API, allowing for zero-argument usage.
- Updated `Cargo.toml` to include compression features for `reqwest` and moved `tempfile` to dependencies.
- Introduced a new `download` argument in `SyncbackCommand` to specify place IDs for downloading.
- Implemented logic to resolve input file paths, prioritizing downloads when necessary.
- Added sourcemap generation after successful syncback and refreshed the git index if in a repository.
- Updated tests to validate the new download functionality and sourcemap creation.
This commit is contained in:
Joe 2026-02-01 09:46:34 -06:00
parent 8fe908740f
commit 2b2ada219e
32 changed files with 1180 additions and 125 deletions
Download patch file Download diff file Expand all files Collapse all files

180
.cursor/plans/destructive_syncback_mode_acb8cb54.plan.md
View file

@ -5,7 +5,7 @@ todos: []
isProject: false
---
# Destructive Syncback Mode
# Destructive Syncback Mode (Clean by Default)
## Root Cause Analysis
@ -50,41 +50,46 @@ if let Some(old_child) = old_child_map.remove(new_child.name.as_str()) {
}
```
## Solution: Add `--clean` Flag
## Solution: Clean Mode by Default with `--incremental` Flag
Add a `--clean` flag that makes syncback treat all instances as "new" by setting `old` references to `None`, ensuring consistent output regardless of existing filesystem state.
Make clean (destructive) mode the default behavior, with an `--incremental` flag to opt into the legacy structure-preserving behavior.
### Ensuring No Residue Files
The key challenge: if we set `old = None`, the `removed_children` logic won't work (nothing to compare against). We need a two-phase approach:
1. **Phase 1**: Collect ALL existing paths from the old tree before syncback
2. **Phase 2**: After computing new paths, remove any old paths that aren't in the new set
This guarantees no orphaned files remain.
### Changes Required
#### 1. Add CLI Flag (`[src/cli/syncback.rs](src/cli/syncback.rs)`)
Add `--clean` flag to the command definition:
Add `--incremental` flag (clean mode is now default):
```rust
/// If provided, syncback will ignore existing file structure and create
/// a fresh project layout. This ensures consistent output regardless of
/// existing filesystem state.
#[clap(long)]
pub clean: bool,
/// If provided, syncback will preserve existing file structure and middleware
/// formats when possible. Without this flag, syncback creates a fresh project
/// layout that exactly matches the input file.
#[clap(long, short = 'n')]
pub incremental: bool,
```
#### 2. Pass Flag to `syncback_loop` (`[src/cli/syncback.rs](src/cli/syncback.rs)`)
Modify the `syncback_loop` call to pass the clean flag:
```rust
let snapshot = syncback_loop(
session_old.vfs(),
&mut dom_old,
dom_new,
session_old.root_project(),
self.clean, // New parameter
self.incremental, // false = clean mode (default)
)?;
```
#### 3. Modify `syncback_loop` (`[src/syncback/mod.rs](src/syncback/mod.rs)`)
Update function signature and initial snapshot:
#### 3. Modify `syncback_loop` Signature (`[src/syncback/mod.rs](src/syncback/mod.rs)`)
```rust
pub fn syncback_loop(
@ -92,67 +97,120 @@ pub fn syncback_loop(
old_tree: &mut RojoTree,
mut new_tree: WeakDom,
project: &Project,
clean_mode: bool, // New parameter
incremental: bool, // false = clean mode (default)
) -> anyhow::Result<FsSnapshot>
```
When `clean_mode` is true, set the initial `old` reference to `None`:
#### 4. Collect Existing Paths Before Syncback (`[src/syncback/mod.rs](src/syncback/mod.rs)`)
Before the main loop, collect all existing filesystem paths (respecting ignore patterns):
```rust
// Collect all existing paths from old tree for cleanup in clean mode
let existing_paths: HashSet<PathBuf> = if !incremental {
let mut paths = HashSet::new();
for ref_id in descendants(old_tree.inner(), old_tree.get_root_id()) {
if let Some(inst) = old_tree.get_instance(ref_id) {
for path in inst.metadata().relevant_paths.iter() {
// Skip paths matching ignore patterns
if is_valid_path(&ignore_patterns, project_path, path) {
paths.insert(path.clone());
}
}
}
}
paths
} else {
HashSet::new()
};
```
#### 5. Set `old = None` in Clean Mode (`[src/syncback/mod.rs](src/syncback/mod.rs)`)
```rust
let mut snapshots = vec![SyncbackSnapshot {
data: syncback_data,
old: if clean_mode { None } else { Some(old_tree.get_root_id()) },
old: if incremental { Some(old_tree.get_root_id()) } else { None },
new: new_tree.root_ref(),
path: project.file_location.clone(),
middleware: Some(Middleware::Project),
}];
```
#### 4. Track Removed Files in Clean Mode (`[src/syncback/mod.rs](src/syncback/mod.rs)`)
#### 6. Remove Orphaned Files After Syncback (`[src/syncback/mod.rs](src/syncback/mod.rs)`)
In clean mode, we need to remove all existing files in the target paths. Add logic to collect existing paths for removal:
After the main loop, remove any existing paths that weren't written to:
```rust
if clean_mode {
// Collect all existing paths under the project tree for removal
for ref_id in descendants(old_tree.inner(), old_tree.get_root_id()) {
if let Some(inst) = old_tree.get_instance(ref_id) {
for path in inst.metadata().relevant_paths.iter() {
if path.is_dir() {
fs_snapshot.remove_dir(path);
} else {
fs_snapshot.remove_file(path);
}
}
// In clean mode, remove any existing paths that weren't overwritten
if !incremental {
let added_paths: HashSet<&Path> = fs_snapshot.added_paths().iter().map(|p| p.as_path()).collect();
for old_path in &existing_paths {
// Skip if this path (or a parent) is being written to
if added_paths.contains(old_path.as_path()) {
continue;
}
// Skip if a parent directory is being added (the new structure will replace it)
if added_paths.iter().any(|p| old_path.starts_with(p)) {
continue;
}
if old_path.is_dir() {
fs_snapshot.remove_dir(old_path);
} else {
fs_snapshot.remove_file(old_path);
}
}
}
```
#### 5. Update Child Snapshot Creation (`[src/snapshot_middleware/dir.rs](src/snapshot_middleware/dir.rs)`)
#### 7. Skip Hash Comparison in Clean Mode (`[src/syncback/mod.rs](src/syncback/mod.rs)`)
The `SyncbackSnapshot::with_joined_path` method already handles `old = None` correctly - it will create a fresh child snapshot without old references.
The hash-based skipping should be disabled in clean mode since we want to rewrite everything:
```rust
// Skip hash comparison in clean mode - we want to rewrite everything
if incremental {
if let Some(old_ref) = snapshot.old {
match (old_hashes.get(&old_ref), new_hashes.get(&snapshot.new)) {
(Some(old), Some(new)) => {
if old == new {
log::trace!(
"Skipping {inst_path} due to it being identically hashed as {old:?}"
);
continue;
}
}
_ => unreachable!("All Instances in both DOMs should have hashes"),
}
}
}
```
### Data Flow Diagram
```mermaid
flowchart TD
subgraph currentBehavior [Current Behavior]
A1[Syncback on Existing Files] --> B1{Old Instance Exists?}
B1 -->|Yes| C1[Preserve Filename]
B1 -->|Yes| D1[Preserve Middleware]
B1 -->|Yes| E1[Match Children by Name]
C1 --> F1[Inconsistent Structure]
D1 --> F1
E1 --> F1
subgraph cleanMode [Default: Clean Mode]
A1[rojo syncback -i game.rbxl] --> B1[Collect existing paths]
B1 --> C1[Set old = None for all snapshots]
C1 --> D1[Generate fresh filenames]
C1 --> E1[Choose best middleware]
C1 --> F1[All children treated as new]
D1 --> G1[Compute new paths]
E1 --> G1
F1 --> G1
G1 --> H1[Remove orphaned files]
H1 --> I1[Consistent 1:1 Structure]
end
subgraph cleanMode [Clean Mode]
A2[Syncback with --clean] --> B2[Set old = None]
B2 --> C2[Generate Fresh Filename]
B2 --> D2[Choose Best Middleware]
B2 --> E2[All Children Are New]
C2 --> F2[Consistent 1:1 Structure]
subgraph incrementalMode [With --incremental Flag]
A2[rojo syncback -i game.rbxl --incremental] --> B2{Old Instance Exists?}
B2 -->|Yes| C2[Preserve Filename]
B2 -->|Yes| D2[Preserve Middleware]
B2 -->|Yes| E2[Match Children by Name]
C2 --> F2[Legacy Behavior]
D2 --> F2
E2 --> F2
end
@ -163,16 +221,30 @@ flowchart TD
### Usage
```bash
# Current behavior (preserves existing structure)
# Default: Clean mode (creates fresh, consistent layout)
rojo syncback -i game.rbxl
# Destructive mode (ignores existing structure, creates fresh layout)
rojo syncback -i game.rbxl --clean
# Preview what clean mode would do
rojo syncback -i game.rbxl --list --dry-run
# Combine with other flags
rojo syncback -i game.rbxl --clean --list --dry-run
# Legacy behavior (preserves existing structure)
rojo syncback -i game.rbxl --incremental
rojo syncback -i game.rbxl -n
# Combine flags
rojo syncback -i game.rbxl --incremental --list
```
### Alternative Consideration
### Project-Level Configuration (Optional Enhancement)
If you also want to support project-level configuration (so you don't need to pass `--clean` every time), we could add a `syncbackRules.cleanMode: true` option in the project file. Let me know if you'd prefer that approach.
If you want to make incremental mode the default for a specific project without passing the flag, we could also add:
```json5
{
"syncbackRules": {
"incremental": true // Opt into legacy behavior per-project
}
}
```
This would be read in `syncback_loop` and combined with the CLI flag (CLI takes precedence).

276
.cursor/plans/syncback_auto-downloader_0508b4bb.plan.md Normal file
View file

@ -0,0 +1,276 @@
---
name: Syncback Auto-Downloader
overview: Add an auto-download feature to the syncback command that fetches the .rbxl file from Roblox's asset delivery API, with sensible defaults to make `rojo syncback` work with zero arguments.
todos:
- id: update-cargo
content: Add gzip/brotli/deflate features to reqwest, move tempfile to dependencies
status: completed
- id: download-function
content: Implement download_place() function with no-cache headers and cookie auth
status: completed
- id: update-args
content: "Update SyncbackCommand args: --download, new defaults, --interactive flip"
status: completed
- id: input-resolution
content: Implement input file resolution logic (download vs existing file)
status: completed
- id: sourcemap-gen
content: Add automatic sourcemap.json generation after syncback
status: completed
- id: git-refresh
content: Add git update-index --refresh for git repos
status: completed
- id: testing
content: Test the complete flow with various argument combinations
status: completed
isProject: false
---
# Syncback Auto-Downloader Feature
## Overview
Enhance the `rojo syncback` command to automatically download place files from Roblox's asset delivery API, generate sourcemaps after completion, and refresh the git index. The goal is to make `rojo syncback` work with zero arguments.
## Key Files to Modify
- `[src/cli/syncback.rs](src/cli/syncback.rs)` - Main syncback command implementation
- `[Cargo.toml](Cargo.toml)` - Add compression features to reqwest, move tempfile to dependencies
## Implementation Details
### 1. Update Dependencies in `Cargo.toml`
```toml
# Add compression features to reqwest
reqwest = { version = "0.11.27", default-features = false, features = [
"blocking",
"json",
"rustls-tls",
"gzip",
"brotli",
"deflate",
] }
# Move tempfile from dev-dependencies to dependencies
tempfile = "3.24.0"
```
### 2. Add Download Arguments to `SyncbackCommand`
In `src/cli/syncback.rs`, modify the struct:
```rust
pub struct SyncbackCommand {
/// Path to the project to sync back to.
#[clap(default_value = "default.project.json5")]
pub project: PathBuf,
/// Path to the Roblox file to pull Instances from.
#[clap(long, short, default_value = "Project.rbxl")]
pub input: PathBuf,
/// Download the place file from Roblox. If no place ID is provided,
/// uses the first ID from servePlaceIds in the project file.
#[clap(long, short = 'd')]
pub download: Option<Option<u64>>,
/// Interactive mode - prompts before writing. Disabled by default.
#[clap(long, short = 'i')]
pub interactive: bool,
// ... rest of existing fields
}
```
Note: Flip `--non-interactive` to `--interactive` (new default is non-interactive).
### 3. Download Function Implementation
Create the download logic with these characteristics:
- **URL**: `https://assetdelivery.roblox.com/v1/asset/?id={PLACEID}`
- **Auth**: `ROBLOSECURITY` cookie from environment variable
- **No-cache headers**:
- `Cache-Control: no-cache, no-store, must-revalidate`
- `Pragma: no-cache`
- `Expires: 0`
- **Compression**: Request `Accept-Encoding: gzip, deflate, br` (reqwest handles decompression)
- **Temp file**: Use `tempfile::NamedTempFile` with `.rbxl` extension, keep it alive during processing
```rust
fn download_place(place_id: u64) -> anyhow::Result<tempfile::NamedTempFile> {
let cookie = std::env::var("ROBLOSECURITY")
.context("ROBLOSECURITY environment variable not set")?;
let url = format!("https://assetdelivery.roblox.com/v1/asset/?id={}", place_id);
let client = reqwest::blocking::Client::builder()
.gzip(true)
.brotli(true)
.deflate(true)
.no_proxy() // Avoid proxy caching
.build()?;
let response = client
.get(&url)
.header(COOKIE, format!(".ROBLOSECURITY={}", cookie))
.header(CACHE_CONTROL, "no-cache, no-store, must-revalidate")
.header(PRAGMA, "no-cache")
.header("Expires", "0")
.header(USER_AGENT, "Rojo")
.send()?;
// Create temp file with .rbxl extension
let mut temp_file = tempfile::Builder::new()
.suffix(".rbxl")
.tempfile()?;
// Stream response to file
let bytes = response.bytes()?;
std::io::copy(&mut bytes.as_ref(), &mut temp_file)?;
Ok(temp_file)
}
```
### 4. Resolve Place ID Logic
When `--download` is provided without a value:
1. Load the project file first
2. Extract `serve_place_ids` from the project
3. Take the first place ID from the set (convert HashSet to Vec, sort for determinism, take first)
4. Error if `servePlaceIds` is empty or not set
### 5. Input File Resolution Logic
In the `run()` method:
```rust
// Determine the actual input path
let input_path = if let Some(download_opt) = &self.download {
// --download was specified
if !self.input_exists() || download_opt.is_some() {
// Download needed: either file doesn't exist or explicit place ID
let place_id = match download_opt {
Some(id) => *id,
None => {
// Load project to get servePlaceIds
let project = Project::load_fuzzy(&path_old)?;
project.serve_place_ids
.as_ref()
.and_then(|ids| ids.iter().min().copied())
.context("No servePlaceIds in project and no place ID provided")?
}
};
eprintln!("Downloading place {}...", place_id);
let temp_file = download_place(place_id)?;
// Keep temp_file alive during processing
DownloadedInput::Temp(temp_file)
} else {
DownloadedInput::Existing(resolve_path(&self.input).into_owned())
}
} else if self.input.exists() {
DownloadedInput::Existing(resolve_path(&self.input).into_owned())
} else {
anyhow::bail!("Input file {} not found. Use --download to fetch from Roblox.", self.input.display())
};
```
### 6. Sourcemap Generation After Syncback
After successful syncback, generate `sourcemap.json` in the project root:
```rust
// After snapshot.write_to_vfs()
let sourcemap_path = session_old.root_project().folder_location().join("sourcemap.json");
write_sourcemap(&session_old, Some(&sourcemap_path), filter_nothing, false)?;
eprintln!("Generated sourcemap at {}", sourcemap_path.display());
```
This requires extracting or sharing the `write_sourcemap` function from `sourcemap.rs` (likely needs to be made `pub(crate)`).
### 7. Git Index Refresh
After syncback completes, check if in a git repo and refresh the index:
```rust
fn refresh_git_index(project_dir: &Path) {
// Check if .git exists in project dir or any parent
let mut check_dir = Some(project_dir);
let mut is_git_repo = false;
while let Some(dir) = check_dir {
if dir.join(".git").exists() {
is_git_repo = true;
break;
}
check_dir = dir.parent();
}
if is_git_repo {
let result = std::process::Command::new("git")
.arg("update-index")
.arg("--refresh")
.current_dir(project_dir)
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status();
if let Err(e) = result {
log::warn!("Failed to refresh git index: {}", e);
}
}
}
```
### 8. Summary of Default Changes
| Argument | Old Default | New Default |
| ------------------- | ------------------ | ----------------------------------------------------------------- |
| `project` | `""` (current dir) | `default.project.json5` |
| `--input` | Required | `Project.rbxl` |
| `--non-interactive` | `false` | Removed, replaced with `--interactive` (default: non-interactive) |
## Flow Diagram
```mermaid
flowchart TD
Start[rojo syncback] --> CheckDownload{--download flag?}
CheckDownload -->|No| CheckInput{Input file exists?}
CheckDownload -->|Yes| CheckPlaceId{Place ID provided?}
CheckPlaceId -->|Yes| Download[Download from Roblox]
CheckPlaceId -->|No| LoadProject[Load project.json5]
LoadProject --> GetPlaceId[Get first servePlaceIds]
GetPlaceId --> Download
Download --> TempFile[Save to temp file]
TempFile --> ProcessDom[Process DOM]
CheckInput -->|Yes| UseExisting[Use existing file]
CheckInput -->|No| Error[Error: file not found]
UseExisting --> ProcessDom
ProcessDom --> Syncback[Run syncback]
Syncback --> WriteFiles[Write to filesystem]
WriteFiles --> GenSourcemap[Generate sourcemap.json]
GenSourcemap --> GitRefresh[git update-index --refresh]
GitRefresh --> Cleanup[Cleanup temp file]
Cleanup --> End[Done]
```
## Edge Cases
- If `--input` file exists and `--download` not specified: use existing file (no download)
- If `--input` file exists but `--download <placeid>` explicitly given: download anyway (override)
- If `--download` with no arg and no `servePlaceIds` in project: error with helpful message
- If `ROBLOSECURITY` not in environment: error with helpful message
- If download fails: error with status code and helpful message
- If not a git repo: silently skip git refresh