Commit graph

71 commits

Author SHA1 Message Date
Joe
dd3466b225 Implement unsuppress_path method in JobThreadContext for better VFS event handling
- Added a new `unsuppress_path` method to the `JobThreadContext` to remove suppressions on paths when filesystem operations fail, ensuring that future events for those paths are processed correctly.
- Updated various sections of `change_processor.rs` to call `unsuppress_path` in relevant error handling scenarios during rename and write operations, improving the accuracy of event processing.
- Enhanced the logic for managing suppressed paths, contributing to a more robust change processing mechanism in the VFS.
2026-02-06 11:23:31 -06:00
Joe
6fd9037cd1 Add stress tests for VFS and Change Processor under maximum load
- Introduced a comprehensive suite of stress tests targeting various combinations of rename, ClassName changes, and source updates to evaluate the robustness of the VFS and Change Processor.
- Implemented multiple test cases that simulate rapid, concurrent operations with zero wait times, ensuring the system can handle high-pressure scenarios without race conditions.
- Enhanced the existing test framework to support fire-and-forget operations, validating the correct handling of file existence and metadata updates after intensive changes.
- Documented the purpose and structure of the new tests to clarify their role in stress testing the two-way sync functionality.
2026-02-06 10:41:46 -06:00
Joe
9bed6dfaa2 Complete todos for stress tests and add syncback_stress fixture
- Marked all todos in the stress test plan as completed, including the implementation of timing helpers, a PRNG, and various fuzz tests for two-way sync operations.
- Introduced a new `syncback_stress` fixture with a project configuration and multiple Luau modules (Alpha, Bravo, Charlie, Delta, Echo) to support stress testing.
- Enhanced the `api.rs` file with improved path handling logic for better integration with the stress testing framework.
2026-02-05 22:36:19 -06:00
Joe
206ffd3720 Refactor path handling in tests for encoded names consistency
- Consolidated multiple lines of code into single statements for improved readability in test cases related to encoded names.
- Updated the logic in various tests to ensure consistent handling of file paths, particularly for encoded module and script names.
- Enhanced clarity in test assertions by streamlining path construction, contributing to better maintainability and understanding of the test suite.
2026-02-05 21:26:18 -06:00
Joe
5b2ea2a23f Refactor file path handling in JobThreadContext to support combined rename and ClassName changes
- Updated the logic to use the effective file path when handling script-to-script transitions, ensuring that the correct path is used even after a rename operation.
- Enhanced the test suite with a new test case for combined rename and ClassName changes, verifying that the extension change is applied correctly after a file rename.
- Improved comments for clarity on the renaming process and its implications for file handling during updates.
2026-02-05 21:19:18 -06:00
Joe
9c92204956 Add syncback_encoded_names fixture and related tests for encoded name handling
- Introduced a new fixture `syncback_encoded_names` with various scripts and models demonstrating encoded characters in names, including `What%QUESTION%Module` and `Key%COLON%Script`.
- Added tests to validate the correct handling of encoded names during directory conversions, ensuring that the filesystem names match the expected encoded formats.
- Updated `change_processor.rs` and `api.rs` to improve file handling and metadata management related to encoded names, enhancing overall robustness and consistency in operations.
2026-02-05 21:12:06 -06:00
Joe
1edca38c34 Add directory name derivation functions in ApiService for improved file handling
- Introduced `dir_name_from_script_path` and `dir_name_from_instance_path` functions to derive directory names from script and instance paths, respectively, enhancing consistency in file operations.
- Updated the `convert_standalone_script_to_directory` and related methods to utilize the new functions for accurate directory naming, ensuring proper handling of various script suffixes.
- Added comprehensive unit tests for the new functions to validate their behavior across different file types and edge cases, improving overall test coverage and reliability.
2026-02-05 21:02:39 -06:00
Joe
6ce3717663 Complete integration tests for two-way sync operations
- Marked all planned integration tests for the two-way sync functionality as completed, ensuring coverage for critical operations such as source property writes, property persistence, instance renames, and lifecycle management.
- Added a new test module specifically for two-way sync, enhancing overall test coverage and reliability of the sync system.
- Updated the change processor to handle file path overrides during instance updates, improving the accuracy of source writes and file management.
2026-02-05 19:51:14 -06:00
Joe
f4f07744c7 Refactor suppression mechanism for file watcher events
- Updated the suppression logic for file watcher events to use a HashMap instead of a HashSet, allowing for count-based suppression. Each API write increments the count, while each suppressed VFS event decrements it.
- Introduced a new method to register paths in the suppression map, improving the handling of redundant re-snapshots and WebSocket messages.
- Enhanced logging for suppressed events to provide better context during debugging.
- Adjusted related structures and methods across ChangeProcessor, ServeSession, and ApiService to accommodate the new suppression mechanism.
2026-02-05 18:46:36 -06:00
Joe
4a03d4bab5 Refactor serve tests to use recv_socket_packet for file operations
- Replaced direct file operations with the new recv_socket_packet method in multiple test functions to ensure WebSocket listeners are active before changes are made.
- This change helps avoid race conditions and improves the reliability of test outcomes by ensuring that actions triggering file changes are executed after the WebSocket connection is established.
2026-02-05 17:12:07 -06:00
Joe
b1aa2a384f Improve error handling in TestServeSession for Rojo process
- Enhanced the error handling logic when the Rojo process exits unexpectedly by capturing and displaying stderr output. This provides better context for debugging failures during test execution.
- Updated the process spawning to pipe stderr, allowing for more informative panic messages when the process fails.
2026-02-05 16:13:09 -06:00
Joe
df06fd0852 Add sleep duration in serve tests for stability
- Introduced a 100ms sleep in multiple test functions to ensure stability during file operations and socket packet retrieval.
- This change aims to mitigate timing issues that may arise from rapid file changes and improve the reliability of test outcomes.
2026-02-04 15:59:19 -06:00
Joe
fe2bf358e7 Enhance syncback logic for orphan file handling and improve test coverage
- Updated the syncback process to prevent scanning the project root folder, ensuring that unrelated files like .gitignore and README.md are not deleted.
- Introduced functionality to collect alternate file representations for paths, allowing orphan files to be identified and removed correctly.
- Enhanced the logic for handling ignored paths, particularly for directories, to improve accuracy in clean mode operations.
- Updated tests to reflect changes in orphan file handling, ensuring that only orphans within $path directories are removed, and clarified documentation regarding clean mode behavior.
2026-02-01 20:18:45 -06:00
Joe
411f79ee81 Enhance syncback functionality and improve code readability
- Refactored the syncback process to streamline directory management and improve handling of class transitions in clean mode.
- Updated the `clone_tree_filtered` function to accept a tuple of compiled Globs and original pattern strings, enhancing flexibility in ignore pattern handling.
- Simplified logic for checking if paths are inside removed directories, improving code clarity.
- Added detailed comments to clarify the purpose of new logic for handling script directory transitions, ensuring better maintainability.
- Improved logging for better traceability during syncback operations.
2026-02-01 20:10:11 -06:00
Joe
1534a54808 Refactor syncback process to improve clean mode handling and directory management
- Removed the ensure_project_paths_exist function, streamlining the syncback process by directly managing directory creation and cleanup within the syncback command.
- Enhanced error handling for sourcemap generation, logging warnings when clean mode removes directories that only contain meta files.
- Updated the logic for middleware selection in clean mode to better handle directory and script transitions based on the filesystem structure.
- Added utility functions to ensure base project directories exist before syncback, addressing the need for directory structure in clean mode.
- Improved test coverage for clean mode scenarios, ensuring that directory existence checks are performed and documented.
2026-02-01 19:55:54 -06:00
Joe
fe1ac1dd95 Fix bugs in syncback process and enhance error handling
- Resolved issues with nested orphan file removal, ensuring that both top-level and nested orphan files are correctly handled.
- Improved directory removal logic to gracefully handle cases where directories may have already been removed, preventing crashes.
- Enhanced the path scanning algorithm to accurately collect all source directories defined in the project, addressing missing paths in the syncback process.
- Updated logging to provide clearer debug information during directory scanning and error handling in the syncback operations.
2026-02-01 19:14:12 -06:00
Joe
4a66eff938 Add comprehensive syncback roundtrip and clean mode stress tests
- Completed the implementation of roundtrip tests to validate the syncback process, ensuring that building a project, syncing back, and rebuilding produces semantically identical outputs.
- Created a new utility module for roundtrip testing, including functions for running builds and asserting directory equality.
- Developed clean mode stress tests to verify that syncing back to a dirty directory yields identical results to syncing back to a fresh directory, covering various mutation scenarios.
- Updated existing test modules to incorporate new test cases and ensure consistency in handling standalone scripts transitioning to directory format when children are added.
2026-02-01 19:01:42 -06:00
Joe
b95b28179f Update LocalScript references in syncback format transition tests
- Changed references from `init.client.luau` to `init.local.luau` in the test for directory LocalScript to accurately reflect the script type being tested.
- Updated assertions to match the new file naming convention, ensuring consistency in the test's expectations for standalone files.
2026-02-01 18:35:22 -06:00
Joe
c08f701f0e Add syncback roundtrip and clean mode stress tests
- Introduced a new testing plan for comprehensive build-syncback roundtrip tests and mutation stress tests to ensure clean mode produces consistent results.
- Created utility functions for running builds and syncbacks, and for asserting directory equality.
- Updated existing tests to reflect changes in behavior when standalone scripts receive children, ensuring they convert to directory format.
- Added new test cases to validate the conversion of standalone scripts and modules when children are added, enhancing coverage of syncback functionality.
2026-02-01 18:34:02 -06:00
Joe
0eb135ee15 Enhance service visibility configuration in snapshot tests
- Added `ignoreHiddenServices: true` and a comprehensive list of `visibleServices` to multiple snapshot test files to improve service visibility management during testing.
- Refactored logging in the `diff.lua` file to streamline the trace output for hidden services, enhancing clarity in the sync process.
2026-02-01 18:28:31 -06:00
Joe
b368eb2401 Add CI pipeline documentation and enhance syncback architecture
- Introduced a new CI pipeline documentation file outlining steps for auto-fixing formatting, running linters, building, and testing.
- Documented the Rojo syncback architecture, detailing improvements to prevent duplicate file creation and format mismatches.
- Updated syncback logic to ensure existing instances are updated instead of creating duplicates, enhancing project file management.
- Added tests for encoding instances and handling nested children to improve reliability in syncback operations.
2026-02-01 18:18:31 -06:00
Joe
2b2ada219e Implement auto-download feature for syncback command
- Enhanced the `syncback` command to automatically download Roblox place files from the asset delivery API, allowing for zero-argument usage.
- Updated `Cargo.toml` to include compression features for `reqwest` and moved `tempfile` to dependencies.
- Introduced a new `download` argument in `SyncbackCommand` to specify place IDs for downloading.
- Implemented logic to resolve input file paths, prioritizing downloads when necessary.
- Added sourcemap generation after successful syncback and refreshed the git index if in a repository.
- Updated tests to validate the new download functionality and sourcemap creation.
2026-02-01 09:46:34 -06:00
Joe
8fe908740f Add statistics tracking for syncback operations
- Introduced a new `SyncbackStats` module to track and report issues during syncback, including duplicate names, unknown classes, and properties.
- Updated the `syncback_loop` and related functions to utilize the stats tracker, logging summaries of issues encountered.
- Enhanced property filtering and instance syncing to record statistics on unknown classes and properties.
- Modified tests to ensure consistent output by normalizing temporary paths in snapshots.
- Added `regex` dependency for path normalization in tests.
2026-02-01 08:29:00 -06:00
Joe
58399bb863 Implement incremental syncback mode to preserve existing file structure
- Added an `incremental` flag to the `SyncbackCommand` to allow for preserving existing file structure and middleware formats during syncback operations.
- Enhanced the `syncback_loop` function to differentiate between clean mode and incremental mode, ensuring orphaned files are removed only in clean mode.
- Updated logging to reflect the mode of operation during syncback.
- Introduced new tests to validate the behavior of clean mode versus incremental mode, ensuring proper handling of orphaned files and preservation of existing structures.
2026-02-01 08:14:20 -06:00
Joe
5f91a0777a Implement new oneshot Vfs for CLI commands and enhance JSON5 serialization
- Added `new_oneshot` method to the Vfs struct, providing a backend suitable for one-time operations without file watching.
- Updated CLI commands (`fmt_project` and `syncback`) to utilize the new oneshot Vfs, improving error handling during file operations.
- Enhanced JSON5 serialization by introducing a custom serializer that directly builds a tree structure, improving performance and maintaining sorted keys.
- Improved error logging in JSON serialization to provide more context on failures.
2026-01-31 13:06:37 -06:00
Joe
8c6aa36027 Enhance error handling in TestServeSession for interrupted reads
- Updated the error handling logic in the TestServeSession implementation to account for `std::io::ErrorKind::Interrupted`, ensuring that the session can properly retry on interrupted read operations, in addition to handling timeouts and would-block scenarios.
2026-01-30 11:40:01 -06:00
Joe
f08a4d21e7 Update file extensions from .lua to .luau and adjust README for glob ignore tests
- Changed return statements in `init.client.lua` and `init.lua` files to reflect the new .luau extension.
- Updated README in `ignore_glob_spec` to specify that glob ignores now work with .spec.luau files instead of .spec.lua.
- Modified various test files to transition from .json to .json5 format, ensuring consistency across project configurations.
2026-01-30 09:19:35 -06:00
Joe
84f76d48fc Add project configuration and test files in JSON5 format
- Introduced new configuration files for testing, including `default.project.json5`, `client_in_folder`, and others.
- Updated existing project files to use JSON5 format, ensuring consistency across configurations.
- Added new Lua scripts and model files to support various test cases.
- Adjusted snapshot files to align with the new project structure and properties.
2026-01-30 09:16:46 -06:00
Joe
d3cc11659a RunContext refactor + sync updates 2026-01-30 08:12:04 -06:00
Joe
bda2af3dfb Use msgpack for API 2026-01-27 07:13:02 -06:00
Micah
a4e80eb114 Fix tests running infinitely 2026-01-27 06:46:00 -06:00
Joe
fdd1d145cf json5 work 2026-01-22 22:19:00 -06:00
Joe
cd63b5c9c0 json5 2026-01-22 21:27:34 -06:00
Joe
1af3a18bcd Update snapshot files to remove unnecessary attributes and improve consistency in property handling
- Cleared out redundant attributes from various snapshot files, specifically removing `Rojo_Target_Value` and `Rojo_Target_PrimaryPart` where applicable.
- Ensured that attributes are consistently represented as empty objects when not needed, enhancing clarity in the snapshot structure.
- Adjusted related test cases to reflect these changes, ensuring alignment with the updated snapshot expectations.
2026-01-20 23:12:43 -06:00
Ken Loeffler
02b41133f8
Use post for ref patch and serialize (#1192) 2026-01-19 22:44:42 +00:00
Micah
d08780fc14
Ensure that pruned Instances aren't treated as existing in syncback (#1179)
Closes #1178.
2025-11-29 21:21:48 -08:00
boatbomber
87f58e0a55
Use WebSocket instead of Long Polling (#1142) 2025-11-26 19:57:01 -08:00
Micah
a61a1bef55
Roundtrip schemas in syncback (#1173) 2025-11-26 16:11:39 -08:00
Micah
9b5a07191b
Implement Syncback to support converting Roblox files to a Rojo project (#937)
This is a very large commit.
Consider checking the linked PR for more information.
2025-11-19 09:21:33 -08:00
boatbomber
d0b029f995
Add JSONC Support for Project, Meta, and Model JSON files (#1144)
Replaces `serde_json` parsing with `jsonc-parser` throughout the
codebase, enabling support for **comments** and **trailing commas** in
all JSON files including `.project.json`, `.model.json`, and
`.meta.json` files.
MSRV bumps from `1.83.0` to `1.88.0` in order to
use the jsonc_parser dependency.
2025-10-28 17:29:57 -07:00
Micah
f3c423d77d
Fix the various lints (#1124) 2025-10-10 13:00:56 -07:00
Micah
80a381dbb1
Use SerializationService as a fallback for when patch application fails (#1030) 2025-09-21 15:09:20 -07:00
Micah
3bac38ee34
Re-add the hack to write NeedsPivotMigration as false for models (#1027) 2025-04-16 15:03:09 -07:00
Jack T
55a207a275
Fix clippy lint warnings (#1004) 2025-01-13 10:07:53 -08:00
Micah
5e1cab2e75
Actually include attribute-defined properties in patch computation (#944) 2024-08-19 15:41:02 -07:00
Kenneth Loeffler
5f7dd45361
Sleep between file copies and serve for macOS serve tests (#945) 2024-07-20 09:52:05 -07:00
Micah
3ca975d81d
Correct issue with default.project.json files with no name being named default after change (#917)
Co-authored-by: Kenneth Loeffler <kenloef@gmail.com>
2024-07-15 09:24:51 -07:00
Micah
7e2bab921a
Support setting referent properties via attributes (#843)
Co-authored-by: Kenneth Loeffler <kenloef@gmail.com>
2024-06-20 23:48:52 +01:00
Micah
48bb760739
Make the name field in projects optional (#870)
Closes #858.

If a project is named `default.project.json`, it acts as an `init` file
and gains the name of the folder it's inside of. If it is named
something other than `default.project.json`, it gains the name of the
file with `.project.json` trimmed off. So e.g. `foo.project.json`
becomes `foo`.
2024-02-20 17:25:57 -08:00
Micah
73828af715
Add a new syncRules field project files to allow users to specify middleware to use for files (#813) 2024-01-19 22:18:17 -08:00