Add live syncback functionality and optimize service encoding

- Introduced a new endpoint `/api/syncback` for one-click full sync from Roblox Studio to the filesystem, eliminating the need for an active WebSocket connection.
- Implemented `SyncbackRequest` and `ServiceChunk` structures to handle service-level properties and child instances during syncback.
- Enhanced `encodeService` to include ObjectValue carriers for accurate reference handling and prevent data loss.
- Updated the syncback process to utilize Luau-style relative paths for reference attributes, improving path management and consistency.
- Integrated session caching in the matching algorithm to optimize performance during syncback operations.

These changes significantly improve the live syncback experience, ensuring reliable data synchronization and enhanced performance during development.
This commit is contained in:
Joe 2026-02-18 22:34:52 -06:00
parent 7f4b5c4941
commit b47eef1173
165 changed files with 5205 additions and 206 deletions

View file

@ -0,0 +1,563 @@
# /luau-optimize - Roblox Luau Performance Optimization
Optimize Roblox Luau files for maximum performance, type safety, and native codegen quality using `luau-compile`, `luau-analyze`, `luau-ast`, and the `luau` CLI.
All scripts are **Roblox Luau**. Roblox APIs are always available. The vector type is `Vector3` (Roblox userdata). We are NOT using the new type solver. Studio testing uses `--!optimize 1` by default; live experiences use `--!optimize 2`. All `luau-compile` invocations must include `--vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3`. Prefer `.luau` extension over `.lua` everywhere.
**Workspace hygiene:** CLI tools generate residual files (`stats.json`, `profile.out`, `coverage.out`, `trace.json`). At the start of execution, create a temp working directory and use it for all tool output:
```
# Windows
$LUAU_TMP = New-Item -ItemType Directory -Path "$env:TEMP\luau-optimize-$(Get-Random)"
# Linux/Mac
LUAU_TMP=$(mktemp -d)
```
Route all output there: `--stats-file=$LUAU_TMP\stats.json`, write benchmark files to `$LUAU_TMP\bench.luau`, run profiling/coverage tools with `$LUAU_TMP` as working directory. The OS handles cleanup -- no manual deletion needed.
---
## Phase 0 -- Scope Resolution and Configuration
### Resolve target
- If user specifies a file: optimize that file.
- If user specifies a directory: discover all `.luau`/`.lua` files, enter Phase 6 (multi-file mode).
- If no argument: optimize the currently open file.
### Infer script context
Check in this order:
1. **Filename suffix** (highest priority):
- `*.server.luau` / `*.server.lua` -> **server** (`--!native` is high-value)
- `*.client.luau` / `*.client.lua` -> **client** (`--!native` less beneficial due to device diversity)
- `*.legacy.luau` / `*.legacy.lua` -> **ambiguous** (could be cloned/moved at runtime). Ask the user.
- `*.luau` / `*.lua` (no suffix) -> ModuleScript, fall through to path check.
2. **Path-based inference** (for ModuleScripts and ambiguous cases):
- `ServerScriptService`/`ServerStorage` in path -> server-side module
- `StarterPlayer`/`StarterGui`/`StarterCharacterScripts`/`ReplicatedFirst` in path -> client-side module
- `ReplicatedStorage`/`shared` in path -> shared module (conservative `--!native`)
- Ambiguous or outside Roblox project -> ask the user
### Quiz the user
Use the AskQuestion tool with these three questions:
**Question 1: "Compilation optimization intensity"**
- `minimal` -- Headers (`--!strict`/`--!native`/`--!optimize 2`), type annotations on all function signatures, deprecated pattern replacements only
- `moderate` -- + function restructuring for inlining, import hoisting, compound operators, fastcall enablement, allocation reduction
- `insane` -- + every micro-optimization from the pattern catalog, full bytecode analysis, register pressure optimization, closure caching analysis
**Question 2: "Algorithm optimization"**
- `none` -- Don't touch logic or data flow
- `low hanging fruit` -- Obvious O(n^2)->dict, redundant iterations, missing caches
- `moderate` -- + data structure changes, caching strategies, event-driven refactors, loop fusion
- `insane` -- + full algorithmic redesign where beneficial, dynamic programming, architectural restructuring
**Question 3: "Benchmarking"**
- `yes` -- Benchmark all non-trivial changes with `luau` CLI (when function is standalone)
- `no` -- Trust bytecode/codegen metrics only
- `+EV only` -- Benchmark only when algorithmic changes or ambiguous bytecode results warrant it
### Derived behavior
| Setting | minimal | moderate | insane |
|---|---|---|---|
| Headers | Always | Always | Always |
| Type annotations | Function signatures | + hot path locals | + all locals |
| Pattern replacements | Deprecated only | Full Priority 2 | + Priority 4 |
| Function restructuring | No | Yes (Priority 3) | Aggressive splitting |
| Bytecode verification | After all changes | After each priority | After each change |
| Algorithmic changes | Per algo quiz | Per algo quiz | Per algo quiz |
| `@native` vs `--!native` | `--!native` unless big file | Selective `@native` on hot functions | Full per-function analysis |
**Restructuring scope** scales with compilation intensity:
- **minimal**: No restructuring. Only additive changes (headers, annotations, pattern swaps).
- **moderate**: Break up monoliths, reorder locals for inlining, hoist imports. Preserve exports/API.
- **insane**: Aggressive restructuring permitted. May change internal function boundaries, rewrite data flow.
---
## Phase 1 -- Deep Code Read and Algorithmic Analysis
This phase comes FIRST. Algorithmic improvements produce order-of-magnitude speedups that dwarf any bytecode optimization. This is also the largest code change, so it must happen before any bytecode-level work.
Read the entire source file thoroughly before touching any CLI tools.
### Step 1 -- Structural decomposition
Monolithic functions hide their algorithmic structure behind interleaved concerns. Before analyzing complexity, understand what the code is *actually doing*:
- **Identify responsibilities**: A 200-line function usually contains 3-5 logical steps that should be separate functions.
- **Trace data flow**: What goes in, what comes out, what is mutated along the way.
- **Separate concerns**: Validation, transformation, I/O, state management, error handling are often tangled. Separating them reveals the core algorithm.
- **Name the steps**: If you can't name what a code block does in 3-5 words, it's doing too much.
- **Look for hidden abstractions**: Repeated "iterate, filter, transform, collect" patterns that could be a single helper.
When the quiz allows restructuring (moderate/insane):
- Physically decompose monoliths into focused local functions. Each does one thing.
- This decomposition often *reveals* optimization opportunities invisible in the monolith: step 2 recomputes what step 1 already knew, or an inner loop could be a lookup table built in the outer loop.
- Smaller functions are easier for the compiler to inline at `-O2` -- clearer code AND better bytecode.
- Preserve the original function as a thin orchestrator calling the decomposed pieces.
Even at "minimal" intensity, still perform this mental decomposition to inform the analysis below.
### Step 2 -- Algorithmic analysis
With the structure understood (or cleaned up), analyze:
**Complexity analysis** -- Identify time complexity of every significant function:
- O(n^2) or worse: nested loops, repeated linear searches, quadratic string building
- O(n) where O(1) is possible: linear search replaceable with dictionary/set lookup
- Repeated work: same computation done multiple times, cacheable/memoizable
- Unnecessary copies: cloning data that could be referenced or mutated in place
- Hidden quadratics: O(n) function that calls another O(n) function inside a loop
**Data structure fitness** -- Is the right data structure being used?
- Array for membership testing -> set (dictionary with `true` values)
- Linear scan for lookup by key -> dictionary
- Repeated sort of mostly-sorted data -> maintain sorted order on insert
- Flat list with frequent removal from middle -> swap-and-pop
- Large string built incrementally -> `table.concat` pattern or `buffer`
- Multiple parallel arrays -> single array of structs (table of tables)
- Unbounded growth without cleanup -> size limits, LRU eviction, periodic pruning
**Caching and memoization:**
- Pure functions called repeatedly with same args -> memoize
- Expensive property reads in loops -> cache in local before loop
- Computed values unchanged within a scope -> hoist out
- Derived state recomputed from scratch on every access -> maintain incrementally
**Redundant work elimination:**
- Multiple passes over same data -> fuse into one
- Recomputing derived state -> maintain incrementally
- Sort/filter that could be done once and reused
- Deep clone where shallow clone or reference suffices
- Intermediate result immediately discarded after extracting one field
**Architectural patterns:**
- Per-frame polling where event-driven would work (RunService events are expensive)
- Synchronous work that could be chunked across frames with `task.wait()`
- Undisconnected event connections (memory leak + wasted computation)
- Rebuilding entire state on small change -> incremental update
- God-object owning too much state -> split responsibilities
**Hot path identification:**
- Event handlers on RunService (PreAnimation, PreRender, PreSimulation, PostSimulation, Heartbeat)
- Functions called inside loops
- Recursive functions
- Functions called from other hot functions
### Step 3 -- Present findings and quiz the user
Before applying any changes, present a summary of all findings organized by impact. For each finding:
- **What**: One-line description of the issue
- **Where**: Function name / line range
- **Why**: What makes it slow (e.g., "O(n^2) nested scan on every frame")
- **Fix**: Proposed change in plain language
- **Impact**: Estimated improvement (order-of-magnitude, constant-factor, or clarity-only)
Then quiz the user using the AskQuestion tool:
- For fewer than 5 findings: one question per finding with options `apply` / `skip` / `modify`
- For 5+ findings: group by category with options `apply all` / `cherry-pick` / `skip all`:
- Structural decomposition
- Data structure changes
- Algorithm changes
- Architectural changes
If the algorithm quiz from Phase 0 was "none", present findings as informational only (no apply options). The user still sees what was found.
If the user selects "modify", wait for their input before proceeding with that specific change.
### Step 4 -- Apply approved changes
Apply only user-approved changes, in this order (dependencies flow downward):
1. **Structural decomposition** -- break monoliths into focused functions (foundation for everything else)
2. **Data structure changes** -- swap arrays for dicts, add indexes, etc.
3. **Algorithm changes** -- replace O(n^2) with O(n), add caching/memoization
4. **Architectural changes** -- event-driven refactors, incremental updates
Benchmark algorithmic changes if the benchmarking quiz warrants it (see Phase 4.5).
---
## Phase 2 -- Baseline Capture
Run all diagnostic tools on the code (post-algorithmic changes if any were applied in Phase 1). These metrics are the baseline for bytecode optimization phases.
### Tools to run
Detect OS: use `--target=x64_ms` on Windows, `--target=x64` on Linux/Mac. Route stats output to temp directory.
```
luau-analyze --mode=strict <file>
luau-analyze --annotate <file>
luau-compile --remarks -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>
luau-compile --text -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>
luau-compile --codegen --target=x64_ms --record-stats=function --stats-file=$LUAU_TMP\stats.json -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>
```
### Metrics to capture
- Total bytecode instruction count (from `--text`)
- Allocation count: NEWTABLE, NEWCLOSURE occurrences (from `--remarks`)
- Inlining success/failure count (from `--remarks`: "inlining succeeded"/"inlining failed")
- Type coverage: count of `any` types in `--annotate` output
- Register spills and skipped functions (from `--record-stats` JSON `lowerStats`)
- Lint warning count (from `--mode=strict`)
---
## Phase 3 -- Compilation-Level Code Analysis
Using both the source (already read in Phase 1) and tool output from Phase 2, identify bytecode-level optimization opportunities:
- **Missing types**: `any` inferences hurting native codegen (especially `Vector3`, `CFrame`, `buffer` params). JIT uses annotations directly -- no runtime type analysis. Unannotated params assumed to be tables.
- **Import patterns**: `math.max` resolved at load time via GETIMPORT. Broken by `getfenv`/`setfenv`/`loadstring` (marks env "impure").
- **Allocation in loops**: NEWTABLE, NEWCLOSURE in hot loops. High allocation rate = more GC assist work.
- **Closure caching**: Repeated function expressions cached when: no upvalues, or all upvalues immutable and module-scope. Mutable captures prevent caching.
- **Upvalue mutability**: ~90% immutable in typical code. Immutable = no allocation, no closing, faster access. Mutable = extra object.
- **Method call patterns**: `obj:Method()` uses fast method call instruction. Avoid `obj.Method(obj)`. `__index` should point at a table directly (not function or deep chain).
- **Inline caching**: Best when field name known at compile time, no metatables, uniform shapes.
- **pcall in hot paths**: Prevents native codegen optimization.
- **String concatenation**: `..` in loops -> `table.concat` or string interpolation.
- **Deoptimizing APIs**: `getfenv`/`setfenv` (even read-only!), `loadstring` disable ALL import optimization and fastcalls.
- **Builtin global writes**: `math = ...` disables fastcall. Lint `BuiltinGlobalWrite` catches this.
- **Metamethod cost**: `__eq` always called on `==`/`~=` even for rawequal values.
- **GC pressure**: Incremental GC with "GC assists" -- allocating code pays proportional GC work.
---
## Phase 4 -- Compilation Optimization
Apply changes in priority order. Re-run `luau-compile --remarks -O2 --vector-lib=Vector3 --vector-ctor=new --vector-type=Vector3 <file>` after structural changes to verify the compiler benefits.
### Priority 1 -- Headers and type safety (always apply)
- Add `--!strict` / `--!native` / `--!optimize 2` header. (`--!optimize 2` is default in live but not Studio testing.)
- Fix type errors from `luau-analyze`.
- Add missing type annotations where `--annotate` shows `any`. Focus on function signatures first, then locals in hot paths per intensity level.
- Annotate `Vector3`, `CFrame`, `buffer` params explicitly -- native codegen generates specialized vector code.
- Replace `getfenv`/`setfenv` -- disables builtins, imports, and optimizations globally.
- Replace `table.getn` -> `#t`, `table.foreach`/`table.foreachi` -> `for..in`.
### Priority 2 -- Low-hanging structural wins
- Hoist library functions: `local floor = math.floor` enables GETIMPORT fastcall. Fastcall builtins: `assert`, `type`, `typeof`, `rawget`/`rawset`/`rawequal`, `getmetatable`/`setmetatable`, `tonumber`/`tostring`, most `math.*` (not `noise`, `random`/`randomseed`), `bit32.*`, some `string.*`/`table.*`. Partial specializations: `assert` (unused return + truthy), `bit32.extract` (constant field/width), `select(n, ...)` O(1).
- Replace `pairs(t)`/`ipairs(t)` -> `for k, v in t do`. Generalized iteration skips the `pairs()` call. `for i=1,#t` is slightly slower.
- `math.floor(a / b)` -> `a // b` (dedicated VM opcode, `//=` compound form).
- Use compound assignment (`+=`, `-=`, `*=`, `//=`, `..=`) -- LHS evaluated once.
- String concat in loops -> `table.concat` or backtick interpolation (lowers to optimized `string.format`).
- `table.create(n)` for known-size arrays. Sequential fill: `local t = table.create(N); for i=1,N do t[i] = ... end`.
- `table.insert(t, v)` for unknown-size append -- `#t` is O(1) cached, worst case O(log N).
- `a + (b - a) * t` -> `math.lerp(a, b, t)` (exact at endpoints, monotonic).
- Manual byte swap -> `bit32.byteswap(n)` (CPU bswap).
- Manual log2 -> `bit32.countlz(n)` (CPU instruction, ~8x faster).
- Manual linear search -> `table.find(t, v)`.
- Manual `pairs` clone -> `table.clone(t)`.
- Manual `string.byte`/`string.char` -> `string.pack`/`string.unpack`.
- `rawlen(t)` when metamethods not needed.
- Explicit `./`/`../`/`@` prefixes in `require()`.
### Priority 3 -- Function structure for inlining
- Break monoliths into small local functions (compiler inlines at `-O2`).
- Inlining requirements: local, non-mutated, non-recursive, not OOP (`:` syntax), not metamethod. Disabled by `getfenv`/`setfenv`.
- Inlining + constant folding cascade: `local function double(x) return x*2 end; local y = double(5)` folds to `y = 10`.
- Create local function wrappers for frequently-called imported module methods.
- Move `pcall` out of hot loops.
- Use `obj:Method()` not `obj.Method(obj)` -- fast method call instruction.
- `__index` should point at a table directly for inline caching.
- Reduce mutable upvalue captures in loop closures. Immutable upvalues = no allocation, no closing.
- Set all table fields at once (compiler infers hash capacity from subsequent assignments).
- Loop unrolling: constant bounds only, simple body.
- Native codegen size limits: 64K instructions/block, 32K blocks/function, 1M instructions/module. Split if exceeded.
### Priority 4 -- Micro-optimizations (hot paths only)
- Table shape consistency (same fields, same order) -- inline caching predicts hash slots.
- `buffer` for binary data (fixed-size, offset-based, efficient native lowering).
- `buffer.readu32` + `bit32` over `buffer.readbits` when schema known.
- Strength reduction: `* 2^n` -> `bit32.lshift`, `/ 2^n` -> `bit32.rshift`.
- Minimize `tostring`/`tonumber` in tight loops.
- `table.freeze` for readonly config (avoids proxy `__index` overhead).
- Keep `__eq` cheap -- fires on every `==`/`~=` and `table.find`.
- `math.isfinite`/`math.isnan`/`math.isinf` over `x ~= x`.
- Annotate `Vector3` params for native specialization.
- `if expr then A else B` over `cond and A or B` (one branch, falsy-safe).
---
## Phase 4.5 -- Benchmark with `luau` CLI
When a non-trivial optimization is applied to an isolated, pure-logic function (no Roblox API dependencies), write a small benchmark harness:
```luau
local function original(...)
-- paste original implementation
end
local function optimized(...)
-- paste optimized implementation
end
local ITERATIONS = 1_000_000
local clock = os.clock
for _ = 1, 1000 do original(...) end
for _ = 1, 1000 do optimized(...) end
local t0 = clock()
for _ = 1, ITERATIONS do original(...) end
local t1 = clock()
for _ = 1, ITERATIONS do optimized(...) end
local t2 = clock()
print(`Original: {t1 - t0:.4f}s`)
print(`Optimized: {t2 - t1:.4f}s`)
print(`Speedup: {(t1 - t0) / (t2 - t1):.2f}x`)
```
Run with both paths:
```
luau -O2 bench.luau
luau -O2 --codegen bench.luau
```
**When to benchmark:** Self-contained function, algorithmic/data structure change, ambiguous bytecode diff, tiebreak between approaches.
**When NOT to benchmark:** Roblox API dependencies (use Studio Script Profiler instead), purely additive changes, cold code.
Write the benchmark file to `$LUAU_TMP\bench.luau`, not the project directory. Run `luau` from `$LUAU_TMP` as working directory so `profile.out` and other residuals land there too.
---
## Phase 5 -- Verification
Re-run all Phase 2 tools and present a before/after comparison:
- Bytecode instruction count delta
- Allocation count delta
- Inlining success rate delta
- Type coverage improvement (fewer `any` types)
- Register spill delta
- Lint warning delta
- Benchmark results (if Phase 4.5 was used)
If any metric regressed, investigate and explain why (or revert).
---
## Phase 6 -- Multi-file Mode
When optimizing a directory:
1. Run Phase 1 (code read) on all files to identify algorithmic improvement potential.
2. Run Phase 2 (baseline) on all files to identify bytecode optimization potential.
3. Prioritize by: worst algorithmic complexity, most lint warnings, most `any` types, most allocations, most failed inlines.
4. Process files one at a time through Phases 1-5.
5. Present aggregate metrics at the end.
---
## Key Principles
These principles govern every optimization decision:
- **Same implementation, better performance** -- behavioral equivalence is non-negotiable. Never change what the code does, only how fast it does it.
- **Algorithms first, bytecode second** -- an O(n) algorithm with unoptimized bytecode beats an O(n^2) algorithm with perfect bytecode every time.
- **Low-hanging fruit first** -- headers and types before restructuring, restructuring before micro-optimizations.
- **Compiler feedback loop** -- verify each structural change with `luau-compile --remarks`. If the compiler didn't benefit, the change wasn't worth it.
- **Small functions over monoliths** -- easier for the compiler to inline, easier for humans to maintain, easier to reason about algorithmically.
- **Local functions for inlining** -- even at the cost of duplicating imported behavior in high-traffic code.
- **Quantify everything** -- before/after metrics for every optimization pass. No "I think this is faster" -- prove it.
- **Type annotations drive native codegen quality** -- every `any` type is a missed specialization opportunity. The JIT uses annotations directly with no runtime analysis.
- **Event-driven over polling** -- avoid per-frame calculations when events suffice.
---
## Reference: Type System (Old Typechecker Only)
**Available NOW:**
- `--!strict` / `--!nonstrict` / `--!nocheck`
- Basic annotations: `local x: number`, `function f(a: string): boolean`
- Optional: `string?`, Union: `number | string`, Intersection: `T1 & T2`
- Generics: `function id<T>(x: T): T`, explicit instantiation: `f<<number>>()`
- Cast: `x :: string`
- Array: `{number}`, Dict: `{[string]: number}`, Table: `type T = { field: type }`
- `typeof()`, `export type`, typed variadics, alias defaults, singletons
- `never`, `unknown`, `read`/`write` modifiers
- Type refinements: `type()`, `typeof()`, `:IsA()`, truthiness, equality, `assert()`
**DO NOT recommend (requires new typechecker):**
- `keyof<T>`, `index<T,K>`, `rawkeyof<T>`, `rawget<T,K>`
- `getmetatable<T>`, `setmetatable<T,M>`
- User-defined type functions, negation types (`~T`)
- Relaxed recursive type restrictions
## Reference: Type Refinement Patterns
The compiler narrows types after certain checks, improving both type safety and native codegen. Use refinements to provide type information without explicit annotations where the type is known from a runtime check:
- **Truthiness:** `if x then` narrows `x` from falsy (`nil`/`false`)
- **Type guards:** `if type(x) == "number" then` narrows `x` to `number`
- **Typeof guards (Roblox):** `if typeof(x) == "Vector3" then` narrows to `Vector3`; `x:IsA("TextLabel")` narrows `Instance` to subclass
- **Equality:** `if x == "hello" then` narrows `x` to singleton `"hello"`
- **Assert:** `assert(type(x) == "string")` narrows `x` to `string` after the call
- **Composition:** Supports `and`, `or`, `not` for compound refinements
## Reference: Deprecated Patterns
| Deprecated | Replacement | Impact |
|---|---|---|
| `getfenv()` / `setfenv()` | `debug.info(i, "snl")` | Disables ALL optimizations. Even read-only `getfenv()` deoptimizes. |
| `loadstring()` | Restructure with `require` | Marks env "impure", disables imports |
| `table.getn(t)` | `#t` | Slower |
| `table.foreach(t, f)` | `for k, v in t do` | Slower |
| `table.foreachi(t, f)` | `for i, v in t do` | Slower |
| `wait()` | `task.wait()` | Deprecated Roblox API |
| `obj.Method(obj, ...)` | `obj:Method(...)` | Misses fast method call instruction |
| `string:method()` | `string.method(s)` | Fastcall: `string.byte(s)` faster than `s:byte()` |
## Reference: Lint Rules
| Lint | Name | Optimization impact |
|---|---|---|
| 10 | BuiltinGlobalWrite | Overwriting builtins disables fastcall |
| 22 | DeprecatedApi | Deprecated APIs: perf/correctness issues |
| 23 | TableOperations | Wrong index, redundant insert, `#` on non-arrays |
| 3 | GlobalUsedAsLocal | Global in one function -> should be local |
| 7 | LocalUnused | Dead locals |
| 12 | UnreachableCode | Dead code |
| 25 | MisleadingAndOr | `a and false or c` bugs |
## Reference: Bytecode Pattern Catalog
| Slow | Fast | Why |
|---|---|---|
| `math.floor(a / b)` | `a // b` | Dedicated VM opcode |
| `data[i].x = data[i].x + 1` | `data[i].x += 1` | LHS evaluated once |
| `a + (b - a) * t` | `math.lerp(a, b, t)` | Exact at endpoints, monotonic |
| `"pre" .. v .. "suf"` | `` `pre{v}suf` `` | Optimized `string.format` |
| Manual byte swap | `bit32.byteswap(n)` | CPU bswap |
| Manual log2 loop | `bit32.countlz(n)` | CPU instruction, ~8x faster |
| `cond and A or B` | `if cond then A else B` | Safe for falsy, one branch |
## Reference: Allocation Reduction
| Slow | Fast | Why |
|---|---|---|
| `local t = {}` in loop | `table.clear(t)` + reuse | Avoids GC pressure |
| Repeated `table.insert` | `table.create(n)` + indexed writes | Preallocated |
| Manual `pairs` clone | `table.clone(t)` | Faster, copies layout |
| `string.byte`/`char` loops | `string.pack`/`unpack` | Native implementation |
| String binary data | `buffer` type | Fixed-size, efficient |
## Reference: Inlining Rules
| Blocks inlining | Enables inlining | Why |
|---|---|---|
| `function Module.foo()` | `local function foo()` | Mutable table prevents |
| `function Obj:method()` | `local function method(self)` | OOP syntax not inlineable |
| Recursive function | Split base/recursive | Recursion prevents |
| Deep upvalue captures | Pass as parameters | Reduces closure cost |
## Reference: Native Codegen
**`@native` vs `--!native`:**
- `--!native`: entire script. Good for math-heavy utility modules.
- `@native`: per-function. Better for selective hot functions or near the 1M instruction limit.
- Inner functions do NOT inherit `@native`.
- Top-level code runs once; `@native` has minimal benefit there.
**Hurts native:** `getfenv`/`setfenv`, wrong types to typed functions, non-numeric math args, breakpoints, size limits (64K instructions/block, 32K blocks/function, 1M/module).
**Helps native:** Type annotations (especially `Vector3`), small functions, consistent table shapes, `buffer` ops, `bit32` ops.
## Reference: CLI Tools
### `luau-compile` -- Static bytecode/codegen analysis (primary tool)
Modes (mutually exclusive): `binary` (raw bytecode), `text` (human-readable opcodes with source annotations), `remarks` (source with inlining/allocation comments -- **most useful**), `codegen` (native assembly, requires `--target`).
Options: `-O<n>` (0-2, default 1; `-O2` enables inlining), `-g<n>` (debug level 0-2), `--target=<arch>` (`x64`, `x64_ms`, `a64`, `a64_nf`; use `x64_ms` on Windows), `--record-stats=<total|file|function>` (JSON stats: bytecodeInstructionCount, lowerStats with spills/skipped/blocks/errors), `--bytecode-summary` (opcode distribution, requires `--record-stats=function`), `--stats-file=<name>` (default `stats.json`), `--timetrace` (trace.json), `--vector-lib`/`--vector-ctor`/`--vector-type` (always set for Roblox).
### `luau-analyze` -- Type checking and linting
Modes: omitted (typecheck + lint), `--annotate` (source with all inferred types inline -- **critical for finding `any`**).
Options: `--mode=strict` (force strict even without directive), `--formatter=plain` (machine-parseable), `--formatter=gnu` (grep-friendly), `--timetrace`.
### `luau` -- Runtime execution, profiling, and coverage
**Limited for Roblox scripts** (most depend on engine APIs). Only usable for pure-logic modules. For Roblox scripts, use Studio Script Profiler and `debug.dumpcodesize()`.
Options: `-O<n>`, `-g<n>`, `--codegen` (native execution), `--profile[=N]` (sampling at N Hz, default 10000, outputs `profile.out`), `--coverage` (outputs `coverage.out`), `--timetrace`, `-a` (program args).
### `luau-ast` -- AST dump
No flags. Outputs full JSON AST with node types, source locations, variable scopes, type annotations. Useful for programmatic analysis: function nesting depth, function body size (monolith detection), closure captures (upvalue analysis), table construction patterns, loop-invariant expression detection.
## Reference: VM Architecture
**Fastcall builtins:** `assert`, `type`, `typeof`, `rawget`/`rawset`/`rawequal`/`rawlen`, `getmetatable`/`setmetatable`, `tonumber`/`tostring`, most `math.*` (not `noise`/`random`/`randomseed`), `bit32.*`, some `string.*`/`table.*`. Partial: `assert` (unused return + truthy), `bit32.extract` (constant field/width), `select(n, ...)` O(1). With `-O2`: constant-arg builtins folded; `math.pi`, `math.huge` folded.
**Inline caching:** HREF-style for table fields. Compiler predicts hash slot; VM corrects. Best: compile-time field name, no metatables, uniform shapes.
**Import resolution:** `math.max` resolved at load time. Invalidated by `getfenv`/`setfenv`/`loadstring`.
**GC:** Incremental mark-sweep with "GC assists" (allocation pays proportional GC). Paged sweeper (16 KB pages). No `__gc` metamethod.
**Compiler:** Multi-pass. Constant folding across functions/locals. Upvalue optimization. `-O2`: inlining, loop unrolling (constant bounds), aggressive constant folding. Interprocedural limited to single modules.
**Table length:** `#t` is O(1) cached, worst case O(log N) branch-free binary search. `table.insert`/`table.remove` update the cache.
**Upvalues:** ~90% immutable. Immutable = no allocation, no closing, faster access. Mutable = extra object.
**Closures:** Cached when no upvalues, or all upvalues immutable and module-scope.
## Reference: Documentation to Consult
Read these docs during execution if needed for specific detail. Two doc trees:
- `.cursor/rules/luau/` -- Official Luau site docs (guides, types, reference, getting-started)
- `.cursor/rules/luau-rfcs/` -- Luau RFCs (language proposals, library additions)
**Luau site docs (primary):**
- `.cursor/rules/luau/guides/performance.md` -- **The official Luau performance guide.** Covers interpreter, compiler, fastcalls, inline caching, imports, table creation, GC, inlining, loop unrolling, upvalues, closures.
- `.cursor/rules/luau/guides/profile.md` -- Profiling guide (flame graphs, sampling)
- `.cursor/rules/luau/types/` -- Full type system docs (basic-types, tables, generics, refinements, unions-and-intersections, object-oriented-programs, roblox-types, considerations)
- `.cursor/rules/luau/getting-started/lint.md` -- All 28 lint rules
- `.cursor/rules/luau/reference/library.md` -- Complete library reference
- `.cursor/rules/luau/getting-started/compatibility.md` -- Luau vs Lua differences
**Luau RFCs (specific features):**
- `.cursor/rules/luau-rfcs/function-inlining.md` -- Inlining RFC (no user `@inline`, automatic at `-O2`)
- `.cursor/rules/luau-rfcs/syntax-attribute-functions-native.md` -- `@native` per-function attribute
- `.cursor/rules/luau-rfcs/function-table-create-find.md`, `function-table-clear.md`, `function-table-clone.md`, `function-table-freeze.md` -- Table operations
- `.cursor/rules/luau-rfcs/function-math-lerp.md` -- `math.lerp`
- `.cursor/rules/luau-rfcs/function-bit32-byteswap.md`, `function-bit32-countlz-countrz.md` -- CPU-level bit ops
- `.cursor/rules/luau-rfcs/function-buffer-bits.md`, `type-byte-buffer.md` -- Buffer type and ops
- `.cursor/rules/luau-rfcs/vector-library.md` -- Vector library and fastcalls
- `.cursor/rules/luau-rfcs/function-string-pack-unpack.md` -- Binary string operations
- `.cursor/rules/luau-rfcs/syntax-floor-division-operator.md` -- `//` dedicated opcode
- `.cursor/rules/luau-rfcs/syntax-compound-assignment.md` -- `+=` single LHS evaluation
- `.cursor/rules/luau-rfcs/deprecate-getfenv-setfenv.md` -- Why fenv disables optimizations
- `.cursor/rules/luau-rfcs/deprecate-table-getn-foreach.md` -- Deprecated table functions
- `.cursor/rules/luau-rfcs/generalized-iteration.md` -- Modern iteration patterns
- `.cursor/rules/luau-rfcs/generic-functions.md`, `never-and-unknown-types.md`, `syntax-type-ascription.md` -- Type system RFCs
**Cursor rules:**
- `.cursor/rules/luau.mdc` -- Luau language rules and optimization tips
**Roblox docs:**
- `.cursor/rules/roblox/en-us/luau/native-code-gen.md` -- Native codegen (size limits, Vector3 annotation impact)
- `.cursor/rules/roblox/en-us/luau/type-checking.md` -- Type annotation syntax and modes
- `.cursor/rules/roblox/en-us/performance-optimization/improve.md` -- Script computation, memory, physics, rendering
- `.cursor/rules/roblox/en-us/performance-optimization/design.md` -- Event-driven patterns, frame budgets
- `.cursor/rules/roblox/en-us/luau/variables.md` -- Local vs global performance
- `.cursor/rules/roblox/en-us/luau/scope.md` -- Scope performance implications

View file

@ -45,6 +45,7 @@ Studio → ChangeBatcher → Plugin → WebSocket → Server → Filesystem
| `/api/write` | POST | Two-way sync: plugin sends changes | | `/api/write` | POST | Two-way sync: plugin sends changes |
| `/api/open/:id` | POST | Open script in editor (`{"cursor": [line, col]}`) | | `/api/open/:id` | POST | Open script in editor (`{"cursor": [line, col]}`) |
| `/api/ref-patch/:id` | GET | Fix Ref properties after syncback | | `/api/ref-patch/:id` | GET | Fix Ref properties after syncback |
| `/api/syncback` | POST | Live syncback: full Roblox → filesystem sync |
| `/api/socket/:cursor` | GET | WebSocket upgrade for live patch streaming | | `/api/socket/:cursor` | GET | WebSocket upgrade for live patch streaming |
### Server Info Response (`GET /api/rojo`) ### Server Info Response (`GET /api/rojo`)
@ -95,6 +96,99 @@ Recomputed fresh on each `GET /api/rojo` connect. Uses a two-phase tree lock: sn
**Format detection:** `Name/init.luau` → directory | `Name.luau` → file | neither → use `has_children` (`.lua` also supported) **Format detection:** `Name/init.luau` → directory | `Name.luau` → file | neither → use `has_children` (`.lua` also supported)
### Live Syncback Endpoint (`POST /api/syncback`)
One-click full sync from Studio to filesystem. Does **not** require an active WebSocket connection -- the plugin POSTs directly from the NotConnected page using the host/port fields.
**Request (`SyncbackRequest`):**
```rust
pub struct SyncbackRequest {
pub protocol_version: f64, // f64 because Luau has no integer type
pub server_version: String,
pub place_id: Option<f64>,
pub data: Vec<u8>, // Single rbxm blob containing ALL children from ALL services
pub services: Vec<ServiceChunk>,
}
```
**`ServiceChunk`:**
```rust
pub struct ServiceChunk {
pub class_name: String,
pub child_count: u32,
pub ref_target_count: u32,
pub properties: HashMap<String, Variant>, // Service-level properties (omitted if empty)
pub refs: HashMap<String, u32>, // Ref property name → 1-based carrier index (omitted if empty)
}
```
**Data format:** The `data` field is a single rbxm binary blob containing all children from all services concatenated. Each service's range in the blob is `child_count + ref_target_count` consecutive entries. First `child_count` entries are real children (parented to service). Next `ref_target_count` entries are temporary ObjectValue carriers encoding Ref properties (destroyed after resolving). The `refs` map connects property names to carrier indices.
**Validation:**
- Protocol version must match `PROTOCOL_VERSION` (6) → 400 Bad Request
- Server version MAJOR.MINOR must match → 400 Bad Request
- Place ID must pass `servePlaceIds`/`blockedPlaceIds` checks if configured → 403 Forbidden
**Response:** `{"status": "syncback_initiated"}`
**Server-side types:**
```rust
pub struct SyncbackPayload {
pub data: Vec<u8>,
pub services: Vec<ServiceChunk>,
}
pub enum ServerExitReason {
SyncbackRequested(SyncbackPayload),
}
```
**`SyncbackSignal` (`src/web/mod.rs`):** Thread-safe signal for API handler → serve loop communication. Uses `Mutex<Option<SyncbackPayload>>` + `Notify`. `fire()` deposits payload and notifies; `take_payload()` retrieves it. `LiveServer::start()` uses `tokio::select!` to wait for either new connections OR the syncback signal, exiting the accept loop when fired.
### Live Syncback Data Flow
```
Plugin (NotConnected page)
│ User clicks "Syncback" button → SyncbackConfirm dialog
│ User confirms → performSyncback()
│ Encodes services via encodeService() + serializes children via SerializationService
▼
HTTP POST /api/syncback (MessagePack body)
│
▼
handle_api_syncback() [src/web/api.rs]
│ Validates protocol, version, place ID
│ Fires SyncbackSignal
▼
LiveServer::start() receives notification [src/web/mod.rs]
│ Exits accept loop
│ Returns ServerExitReason::SyncbackRequested(payload)
▼
run_live_syncback() [src/cli/serve.rs]
│ build_dom_from_chunks(payload) → reconstructs DOM from rbxm + service chunks
│ syncback_loop(incremental = false) → clean mode, full filesystem rewrite
│ write_to_vfs_parallel() → writes files
│ refresh_git_index() → updates git index
▼
Serve loop restarts (new session, ready for connections)
```
### Live Syncback vs Two-Way Sync
| Aspect | Live Syncback (`/api/syncback`) | Two-Way Sync (`/api/write`) |
|--------|--------------------------------|---------------------------|
| **Purpose** | Full Roblox → filesystem sync | Incremental property/instance changes |
| **Scope** | Entire instance tree (all services) | Individual instance changes |
| **Mode** | Clean mode (replaces everything) | Incremental (add/update/remove) |
| **Server restart** | Yes (restarts serve loop) | No (continues serving) |
| **Data format** | Single rbxm blob + service chunks | Patch-based (added/updated/removed) |
| **Connection** | No WebSocket needed (HTTP POST) | Requires active WebSocket session |
| **When used** | User-initiated one-click syncback | Real-time bidirectional sync |
| **Session** | Creates new session after restart | Uses existing live session |
## WebSocket Protocol (v6) ## WebSocket Protocol (v6)
**Connection:** `ws://host:port/api/socket/:cursor` **Connection:** `ws://host:port/api/socket/:cursor`
@ -652,14 +746,16 @@ Only **Updated** items get defaults. Added/Removed items remain `nil`.
The plugin uses a recursive change-count scoring algorithm (implemented in `matching.lua`) to pair incoming virtual instances with existing Studio instances during hydration. The plugin uses a recursive change-count scoring algorithm (implemented in `matching.lua`) to pair incoming virtual instances with existing Studio instances during hydration.
**Entry point:** `Matching.matchChildren(virtualChildren, studioChildren, parentVirtual, parentStudio)` **Entry point:** `Matching.matchChildren(virtualChildren, studioChildren, virtualInstances)`
**Algorithm:** **Algorithm:**
1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring. 1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring.
2. **Ambiguous groups:** For groups with multiple candidates, compute pairwise `_computeChangeCount()` (recursive, scores entire subtree). Greedy assignment sorted by cost ascending, child-order tiebreaker. 2. **Ambiguous groups:** For groups with multiple candidates, compute pairwise `computePairCost()` (recursive, scores entire subtree). Greedy assignment sorted by cost ascending, child-order tiebreaker.
3. **Leftovers:** Unmatched virtual = new instances, unmatched Studio = deleted. 3. **Leftovers:** Unmatched virtual = new instances, unmatched Studio = deleted.
**Value equality:** Shared `trueEquals.lua` module used by both `matching.lua` and `diff.lua` (fuzzy equality for floats, Color3, CFrame, Vector3; NaN handling; nil/null-ref equivalence). **Performance:** File-level `--!native` and `--!optimize 2` directives for native codegen. `MatchingSession` caches scoring results across recursive calls (`matchCache` for match results, `costCache` for pair costs), avoiding redundant computation when hydrate re-calls `matchChildren` on the same children. `cacheVirtual()` filters children to only those existing in `virtualInstances`, preventing crashes on incomplete virtual instance trees.
**Value equality:** Shared `trueEquals.lua` module used by both `matching.lua` and `diff.lua` (fuzzy equality for floats with epsilon 0.0001, Color3, CFrame, Vector3; NaN handling; nil/null-ref equivalence).
## Protocol Enforcement ## Protocol Enforcement
@ -701,8 +797,10 @@ Browser: `http://localhost:34872` → Network tab → WebSocket messages
| Server | Plugin | | Server | Plugin |
|--------|--------| |--------|--------|
| `src/web/api.rs` - Endpoints | `plugin/src/ServeSession.lua` - Connection | | `src/web/api.rs` - Endpoints | `plugin/src/ServeSession.lua` - Connection |
| `src/web/interface.rs` - Packets | `plugin/src/ApiContext.lua` - HTTP/WS client | | `src/web/interface.rs` - Packets, syncback types | `plugin/src/ApiContext.lua` - HTTP/WS client |
| `src/serve_session.rs` - Session | `plugin/src/ChangeBatcher/` - Two-way sync | | `src/web/mod.rs` - SyncbackSignal, LiveServer | `plugin/src/ChangeBatcher/` - Two-way sync |
| `src/change_processor.rs` - FS writer | `plugin/src/Reconciler/` - Patch application | | `src/serve_session.rs` - Session | `plugin/src/ChangeBatcher/encodeService.lua` - Live syncback encoding |
| `src/rojo_ref.rs` - Ref paths | `plugin/src/Reconciler/matching.lua` - Matching | | `src/cli/serve.rs` - Live syncback handler | `plugin/src/Reconciler/` - Patch application |
| `src/git.rs` - Git integration | `plugin/src/PatchTree.lua` - Default selections | | `src/change_processor.rs` - FS writer | `plugin/src/Reconciler/matching.lua` - Matching |
| `src/rojo_ref.rs` - Ref paths | `plugin/src/PatchTree.lua` - Default selections |
| `src/git.rs` - Git integration | `plugin/src/App/init.lua` - Live syncback UI |

View file

@ -36,6 +36,10 @@ The matching algorithm exists in 3 parallel implementations that must produce id
All three use the same recursive change-count scoring + greedy assignment algorithm. Changes to the algorithm must be applied to all three. All three use the same recursive change-count scoring + greedy assignment algorithm. Changes to the algorithm must be applied to all three.
All three now have **`MatchingSession` caching** to avoid redundant recursive scoring:
- **Rust:** `cost_cache: RefCell<HashMap<(Ref, Ref), u32>>` -- caches `compute_change_count` results
- **Lua:** `matchCache` (parent virtual ID → Studio instance → MatchResult), `costCache` (virtual ID → Studio instance → cost)
### Key Files Map ### Key Files Map
| File | Subsystem | Purpose | | File | Subsystem | Purpose |
@ -45,7 +49,7 @@ All three use the same recursive change-count scoring + greedy assignment algori
| `src/syncback/matching.rs` | Matching | Syncback matching: `match_children` (WeakDom) | | `src/syncback/matching.rs` | Matching | Syncback matching: `match_children` (WeakDom) |
| `src/snapshot/matching.rs` | Matching | Forward sync matching: `match_forward` (Snapshot vs Tree) | | `src/snapshot/matching.rs` | Matching | Forward sync matching: `match_forward` (Snapshot vs Tree) |
| `plugin/src/Reconciler/matching.lua` | Matching | Lua matching: `Matching.matchChildren` (Virtual vs Studio) | | `plugin/src/Reconciler/matching.lua` | Matching | Lua matching: `Matching.matchChildren` (Virtual vs Studio) |
| `src/rojo_ref.rs` | Ref Paths | `ref_target_path_from_tree`, `RefPathIndex`, `Rojo_Ref_*` constants | | `src/rojo_ref.rs` | Ref Paths | `ref_target_path_from_tree`, `compute_relative_ref_path`, `resolve_ref_path_to_absolute`, `RefPathIndex`, `Rojo_Ref_*` constants |
| `src/snapshot/tree.rs` | Ref Paths | `get_instance_by_path`, `filesystem_name_for` | | `src/snapshot/tree.rs` | Ref Paths | `get_instance_by_path`, `filesystem_name_for` |
| `src/syncback/ref_properties.rs` | Ref Paths | `collect_referents`, `tentative_fs_path`, `RefLinks` | | `src/syncback/ref_properties.rs` | Ref Paths | `collect_referents`, `tentative_fs_path`, `RefLinks` |
| `src/variant_eq.rs` | Matching | `variant_eq` property comparison (Rust) | | `src/variant_eq.rs` | Matching | `variant_eq` property comparison (Rust) |
@ -213,8 +217,6 @@ function Matching.matchChildren(
**Fast-path: (Name, ClassName) Grouping.** Group children by `(Name, ClassName)`. 1:1 groups get instant-matched with zero scoring. **Fast-path: (Name, ClassName) Grouping.** Group children by `(Name, ClassName)`. 1:1 groups get instant-matched with zero scoring.
> **Note on Ref Pin:** The Lua matching module has a commented-out Ref pin fast-path (scanning parent Ref properties to pin child matches). In practice this is a no-op because during initial hydration the instanceMap is empty. The Rust implementations do not have Ref pin logic. Ref property correctness is maintained by `computeChangeCount` counting Ref property diffs.
**Ambiguous Groups: Recursive Scoring + Greedy Assignment.** For groups with multiple candidates on at least one side: **Ambiguous Groups: Recursive Scoring + Greedy Assignment.** For groups with multiple candidates on at least one side:
``` ```
@ -271,6 +273,14 @@ Syncback matching has precomputed blake3 subtree hashes. If `new_hash == old_has
**Lua (`trueEquals`):** Recursive deep equality. Treats `nil` and null ref (`{Ref = "000...0"}`) as equal. Floats use epsilon 0.0001 (absolute + relative). CFrame/Vector3 component-wise. Color3 via RGB ints. EnumItem via `.Value`. **Lua (`trueEquals`):** Recursive deep equality. Treats `nil` and null ref (`{Ref = "000...0"}`) as equal. Floats use epsilon 0.0001 (absolute + relative). CFrame/Vector3 component-wise. Color3 via RGB ints. EnumItem via `.Value`.
### Session Caching
All three implementations wrap scoring in a `MatchingSession` that caches results:
- **Rust:** `compute_change_count` checks `cost_cache` before scoring. If `(ref_a, ref_b)` was scored before, returns cached value.
- **Lua:** `computePairCost` checks `costCache`; `matchChildren` checks `matchCache`. Avoids redundant work when hydrate re-calls matching on the same children.
The Lua matching file uses `--!native` and `--!optimize 2` file-level directives for native codegen performance. `cacheVirtual()` filters children to only those existing in `virtualInstances`, preventing crashes on incomplete virtual instance trees.
### Known Limitation: Greedy vs Optimal ### Known Limitation: Greedy vs Optimal
The greedy algorithm is NOT globally optimal. Edge cases exist where a locally-best pick forces a globally-worse assignment. The Hungarian algorithm (O(N^3)) would give optimal results, but ambiguous groups are typically small (2-10 instances). Accepted for current release. The greedy algorithm is NOT globally optimal. Edge cases exist where a locally-best pick forces a globally-worse assignment. The Hungarian algorithm (O(N^3)) would give optimal results, but ambiguous groups are typically small (2-10 instances). Accepted for current release.
@ -291,7 +301,21 @@ Dedup'd ModuleScript → segment "Foo~2.luau"
Init-style Script → segment "Foo" (directory) Init-style Script → segment "Foo" (directory)
``` ```
Paths are slash-separated from DataModel's children downward: `"Workspace/TestModel/Part1.model.json5"`. Absolute paths are slash-separated from DataModel's children downward: `"Workspace/TestModel/Part1.model.json5"`.
### Luau-Style Relative Ref Paths
Syncback now writes **relative** ref paths using Luau require-by-string semantics via `compute_relative_ref_path()`:
- `@self` -- target is the source instance itself
- `@self/Child.luau` -- target is a descendant of source
- `./Sibling.luau` -- target is a same-parent sibling
- `../Uncle.luau` -- target is a parent's sibling
- `@game/Service/Path` -- cross-service reference
Forward sync resolves both absolute and relative formats via `resolve_ref_path_to_absolute()` + `get_instance_by_path()`.
**Placeholder system:** During syncback, before dedup suffixes are finalized, ref paths are written as `__ROJO_REF_<source>_TO_<target>__` placeholders. After the syncback walk assigns final filesystem names, `fix_ref_paths()` substitutes placeholders with actual relative paths.
### Key Functions ### Key Functions
@ -492,9 +516,9 @@ Duplicate-named children under ProjectNode are currently skipped during syncback
The greedy assignment may produce suboptimal pairing in rare ambiguous groups. A globally optimal algorithm (Hungarian, O(N^3)) would be better but is deferred since ambiguous groups are typically small. The greedy assignment may produce suboptimal pairing in rare ambiguous groups. A globally optimal algorithm (Hungarian, O(N^3)) would be better but is deferred since ambiguous groups are typically small.
### Matching Cache Not Implemented ### ~~Matching Cache Not Implemented~~ (Resolved)
Phase 2e from the original plan: optional cache mapping `parent_ref → (children_list_hash, match_assignments)` to avoid re-running matching when children haven't changed. Optimization only, not needed for correctness. `MatchingSession` caching is now implemented in all three matching implementations. See "Session Caching" in Subsystem 3. The per-parent invalidation cache (mapping `parent_ref → (children_list_hash, match_assignments)`) remains deferred.
### ~~Stale Ref Paths After Rename~~ (Resolved) ### ~~Stale Ref Paths After Rename~~ (Resolved)
@ -506,7 +530,7 @@ Both dedup-triggered renames and user-initiated renames now update `Rojo_Ref_*`
- **Shared filesystem-segment derivation:** Consolidate tentative filesystem name computation into one shared helper used by `rojo_ref.rs`, `tree.rs`, `api.rs`, and `ref_properties.rs`. Currently duplicated across 4+ call sites. - **Shared filesystem-segment derivation:** Consolidate tentative filesystem name computation into one shared helper used by `rojo_ref.rs`, `tree.rs`, `api.rs`, and `ref_properties.rs`. Currently duplicated across 4+ call sites.
- **Optimal assignment algorithm:** Replace greedy with Hungarian for large ambiguous groups. - **Optimal assignment algorithm:** Replace greedy with Hungarian for large ambiguous groups.
- **Matching cache:** Add per-parent cache with invalidation for performance. - **Per-parent matching invalidation cache:** `MatchingSession` cost/match caching is implemented, but the per-parent cache with `children_list_hash` invalidation (avoids re-running matching when children haven't changed) remains deferred.
- **Duplicated name→index grouping:** All three matching implementations repeat the same HashMap pattern. Could be factored but types differ. - **Duplicated name→index grouping:** All three matching implementations repeat the same HashMap pattern. Could be factored but types differ.
--- ---
@ -549,3 +573,6 @@ The dedup system was built across these plans (chronological):
8. **ambiguous_paths_audit_fixes** -- Second audit: recursive matching, `added_paths` format, dedup-aware ref collection, full-FS-name cleanup grouping, shared `trueEquals` 8. **ambiguous_paths_audit_fixes** -- Second audit: recursive matching, `added_paths` format, dedup-aware ref collection, full-FS-name cleanup grouping, shared `trueEquals`
9. **audit_ambiguous_paths_followup** -- Third audit: metadata drift after cleanup rename, co-removed sibling exclusion, `added_paths` extension/suffix, attribute-scoped final paths 9. **audit_ambiguous_paths_followup** -- Third audit: metadata drift after cleanup rename, co-removed sibling exclusion, `added_paths` extension/suffix, attribute-scoped final paths
10. **audit_ref_two-way_sync** -- Ref audit: round-trip fidelity, path computation correctness, meta lifecycle, VFS suppression, wire format 10. **audit_ref_two-way_sync** -- Ref audit: round-trip fidelity, path computation correctness, meta lifecycle, VFS suppression, wire format
11. **matching_session_caching** -- Session-scoped caching for all three matching implementations, `--!native`/`--!optimize 2` for Lua, valid children tracking
12. **luau-style_ref_paths** -- Relative ref paths (`@self`, `./`, `../`, `@game/`), source-aware placeholder system
13. **live_syncback** -- One-click full sync from Studio via `POST /api/syncback`, service property encoding, server restart

View file

@ -25,9 +25,10 @@ Built with Roact (React-like), handles WebSocket sync with server, patch applica
- `Studio/` - Studio API wrappers (`StudioPluginAction`, `StudioPluginContext`, `StudioPluginGui`, `StudioToggleButton`, `StudioToolbarContext`, `StudioToolbar`) - `Studio/` - Studio API wrappers (`StudioPluginAction`, `StudioPluginContext`, `StudioPluginGui`, `StudioToggleButton`, `StudioToolbarContext`, `StudioToolbar`)
- General: `Header`, `TextInput`, `TextButton`, `Checkbox`, `Dropdown`, `Tag`, `IconButton`, `Tooltip`, `TouchRipple`, `VirtualScroller`, `ScrollingFrame`, `Spinner`, `BorderedContainer`, `EditableImage`, `SlicedImage`, `ClassIcon` - General: `Header`, `TextInput`, `TextButton`, `Checkbox`, `Dropdown`, `Tag`, `IconButton`, `Tooltip`, `TouchRipple`, `VirtualScroller`, `ScrollingFrame`, `Spinner`, `BorderedContainer`, `EditableImage`, `SlicedImage`, `ClassIcon`
- **StatusPages/** - Main UI pages - **StatusPages/** - Main UI pages
- `NotConnected.lua` - Connection form (host/port input) - `NotConnected.lua` - Connection form (host/port input), "Syncback" button (Danger style)
- `Connecting.lua` - Loading screen - `Connecting.lua` - Loading screen
- `Confirming.lua` - Patch confirmation dialog with push/pull/ignore per-node selections - `Confirming.lua` - Patch confirmation dialog with push/pull/ignore per-node selections
- `SyncbackConfirm` - Floating dialog for live syncback confirmation (rendered by `App/init.lua`)
- `Connected.lua` - Status display with patch history - `Connected.lua` - Status display with patch history
- `Error.lua` - Error display - `Error.lua` - Error display
- `Settings/` - Settings UI (`init.lua`, `Setting.lua`) - `Settings/` - Settings UI (`init.lua`, `Setting.lua`)
@ -56,7 +57,8 @@ Built with Roact (React-like), handles WebSocket sync with server, patch applica
- **init.lua** - Main batcher: accumulates changes every 200ms, pause/resume, deferred Ref resolution - **init.lua** - Main batcher: accumulates changes every 200ms, pause/resume, deferred Ref resolution
- **createPatchSet.lua** - Converts pending changes to PatchSet - **createPatchSet.lua** - Converts pending changes to PatchSet
- **encodePatchUpdate.lua** - Encodes property updates including Ref properties via InstanceMap - **encodePatchUpdate.lua** - Encodes property updates including Ref properties via InstanceMap
- **encodeInstance.lua** - Full instance encoding (for additions/syncback) - **encodeInstance.lua** - Full instance encoding (for additions/syncback); includes duplicate detection helpers (`findDuplicateNames`, `hasDuplicateSiblings`, `isPathUnique`)
- **encodeService.lua** - Encodes service-level properties/attributes/refs for live syncback; uses ObjectValue carriers for Ref properties
- **encodeProperty.lua** - Single property value encoding via RbxDom - **encodeProperty.lua** - Single property value encoding via RbxDom
- **propertyFilter.lua** - Property type filtering (only `UniqueId` blocked; `Ref` is allowed) - **propertyFilter.lua** - Property type filtering (only `UniqueId` blocked; `Ref` is allowed)
@ -242,12 +244,22 @@ Pairs incoming virtual instances with existing Studio instances during hydration
**Algorithm:** **Algorithm:**
1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring. 1. **Fast path:** Group by (Name, ClassName). 1:1 groups get instant-matched with zero scoring.
2. **Ambiguous groups:** For groups with multiple candidates, compute pairwise `_computeChangeCount()` (recursive, scores entire subtree). Greedy assignment sorted by cost ascending, child-order tiebreaker for stability. 2. **Ambiguous groups:** For groups with multiple candidates, compute pairwise `computePairCost()` (recursive, scores entire subtree). Greedy assignment sorted by cost ascending, child-order tiebreaker for stability.
3. **Leftovers:** Unmatched virtual = new instances, unmatched studio = deleted. 3. **Leftovers:** Unmatched virtual = new instances, unmatched studio = deleted.
**Performance:** File-level `--!native` and `--!optimize 2` directives for native codegen. `MatchingSession` caches results across recursive calls:
```lua
type MatchingSession = {
matchCache: { [string]: { [Instance]: MatchResult } },
costCache: { [string]: { [Instance]: number } },
}
```
**Valid children tracking:** `cacheVirtual()` filters children to only those existing in `virtualInstances`, preventing crashes on incomplete virtual instance trees.
**Sort stability note:** `table.sort` is NOT stable in Luau. Comparator breaks ties by insertion index to ensure deterministic ordering. **Sort stability note:** `table.sort` is NOT stable in Luau. Comparator breaks ties by insertion index to ensure deterministic ordering.
**Value equality:** Shared `trueEquals.lua` module used by both `matching.lua` and `diff.lua` -- fuzzy equality for floats (epsilon 0.0001), Color3 via RGB ints, CFrame/Vector3 component-wise, NaN handling, nil/null-ref equivalence. **Value equality:** Shared `trueEquals.lua` module used by both `matching.lua` and `diff.lua` -- fuzzy equality for floats (epsilon 0.0001, absolute + relative), Color3 via RGB ints, CFrame/Vector3 component-wise, NaN handling, nil/null-ref equivalence, EnumItem/number interop.
## InstanceMap (`InstanceMap.lua`) ## InstanceMap (`InstanceMap.lua`)
@ -308,6 +320,29 @@ Watches Studio changes and sends them to the server in real-time. Enabled by def
- `__paused` flag blocks flushes during confirmation dialog - `__paused` flag blocks flushes during confirmation dialog
- `RunService:IsRunning()` blocks during Play mode - `RunService:IsRunning()` blocks during Play mode
## Live Syncback
One-click full sync from Studio to filesystem. Available from the **NotConnected page** -- does not require an active WebSocket session. Sends directly to the server via HTTP POST using the host/port fields from the connection form.
**`SYNCBACK_SERVICES` constant (14 services):** Lighting, MaterialService, ReplicatedFirst, ReplicatedStorage, ServerScriptService, ServerStorage, SoundService, StarterGui, StarterPack, StarterPlayer, Teams, TextChatService, VoiceChatService, Workspace.
**Flow:**
1. User clicks "Syncback" button on NotConnected page → `SyncbackConfirm` floating dialog appears
2. Dialog warns: "This will overwrite your project files with the current Studio state. This cannot be undone."
3. User confirms → `performSyncback()`:
a. Iterates `SYNCBACK_SERVICES`, calls `encodeService(service)` for each
b. Collects all children instances and ObjectValue Ref carriers
c. Serializes all children via `SerializationService:SerializeInstancesAsync()` into a single buffer
d. Destroys temporary ObjectValue carriers
e. POSTs to `http://host:port/api/syncback` (MessagePack): `{protocolVersion, serverVersion, placeId, data, services}`
4. Shows success/error notification
**`encodeService(service)`** (`ChangeBatcher/encodeService.lua`):
- Encodes service-level properties via `RbxDom.findClassDescriptor()`
- Skips: `Parent`, `Name`, `Archivable`, `Attributes`, `Tags` (handled separately), class-specific excludes (`Lighting.ClockTime`)
- Ref properties: creates temporary `ObjectValue` carriers (Name = property name, Value = target instance), stores 1-based index in `refs` map
- Returns `(chunk, refTargets)` where `chunk = {className, childCount, refTargetCount, properties?, refs?}`
## Dependencies ## Dependencies
### Vendored Packages (`plugin/Packages/`) ### Vendored Packages (`plugin/Packages/`)

View file

@ -191,7 +191,7 @@ Configure `syncbackRules` object for Roblox → filesystem sync:
| `ignoreProperties` | object | `{}` | Properties to skip per class: `{"ClassName": ["Prop1", "Prop2"]}` | | `ignoreProperties` | object | `{}` | Properties to skip per class: `{"ClassName": ["Prop1", "Prop2"]}` |
| `ignoreClasses` | string[] | `[]` | Instance classes to skip entirely | | `ignoreClasses` | string[] | `[]` | Instance classes to skip entirely |
| `syncCurrentCamera` | bool | `false` | Whether to sync `Workspace.CurrentCamera` | | `syncCurrentCamera` | bool | `false` | Whether to sync `Workspace.CurrentCamera` |
| `syncUnscriptable` | bool | `true` | Whether to sync unscriptable properties | | `syncUnscriptable` | bool | `false` | Whether to sync unscriptable properties |
| `ignoreReferents` | bool | `false` | Whether to skip Ref properties (e.g., `Model.PrimaryPart`) | | `ignoreReferents` | bool | `false` | Whether to skip Ref properties (e.g., `Model.PrimaryPart`) |
| `createIgnoreDirPaths` | bool | `false` | Whether `/**` in ignore paths also matches the directory itself | | `createIgnoreDirPaths` | bool | `false` | Whether `/**` in ignore paths also matches the directory itself |
| `ignoreHiddenServices` | bool | — | Override root-level `ignoreHiddenServices` for syncback | | `ignoreHiddenServices` | bool | — | Override root-level `ignoreHiddenServices` for syncback |
@ -208,7 +208,7 @@ Configure `syncbackRules` object for Roblox → filesystem sync:
}, },
"ignoreClasses": ["Terrain", "Camera"], "ignoreClasses": ["Terrain", "Camera"],
"syncCurrentCamera": false, "syncCurrentCamera": false,
"syncUnscriptable": true, "syncUnscriptable": false,
"ignoreReferents": false, "ignoreReferents": false,
"createIgnoreDirPaths": false, "createIgnoreDirPaths": false,
"warnDuplicateNames": false "warnDuplicateNames": false
@ -341,9 +341,15 @@ Uses filesystem-name paths (slugified, dedup'd) to reference target instances:
} }
``` ```
**Attribute format:** `Rojo_Ref_<PropertyName> = "<filesystem-name path>"` **Attribute format:** `Rojo_Ref_<PropertyName> = "<path>"`
Path segments use full filesystem names (slug + extension + dedup suffix). Resolved during forward sync via `get_instance_by_path()` (case-insensitive filesystem name matching with instance name fallback). Syncback writes **Luau-style relative paths**:
- `@self` / `@self/Child.luau` -- target is same instance or a descendant
- `./Sibling.luau` -- target is a same-parent sibling
- `../Uncle.luau` -- target is a parent's sibling
- `@game/Service/Path` -- cross-service reference
Forward sync resolves both relative and absolute formats via `resolve_ref_path_to_absolute()` + `get_instance_by_path()`. Path segments use full filesystem names (slug + extension + dedup suffix), matched case-insensitive with instance name fallback.
### ID-Based: `Rojo_Target_*` + `$id` (legacy, still read by forward sync) ### ID-Based: `Rojo_Target_*` + `$id` (legacy, still read by forward sync)

View file

@ -19,7 +19,7 @@ Pure functions: filesystem → instance tree → minimal patches → apply chang
- `mod.rs` - System overview with detailed explanation - `mod.rs` - System overview with detailed explanation
- `patch_compute.rs` - Generate diffs (uses matching algorithm) - `patch_compute.rs` - Generate diffs (uses matching algorithm)
- `patch_apply.rs` - Apply changes (resolves `Rojo_Ref_*` and `Rojo_Target_*` refs) - `patch_apply.rs` - Apply changes (resolves `Rojo_Ref_*` and `Rojo_Target_*` refs)
- `matching.rs` - Forward sync matching algorithm (recursive change-count scoring) - `matching.rs` - Forward sync matching algorithm (recursive change-count scoring, `MatchingSession` caching)
- `instance_snapshot.rs` - Core snapshot type - `instance_snapshot.rs` - Core snapshot type
```rust ```rust
@ -79,7 +79,7 @@ Instance names with forbidden filesystem chars (`<>:"/\|?*~`) are handled by:
### 3. Commands (`src/cli/`) ### 3. Commands (`src/cli/`)
- `serve.rs` - Live sync with WebSocket - `serve.rs` - Live sync with WebSocket; live syncback via `run_live_syncback()` + `build_dom_from_chunks()` (server restarts after syncback)
- `build.rs` - Generate rbxl/rbxm/rbxmx/rbxlx - `build.rs` - Generate rbxl/rbxm/rbxmx/rbxlx
- `upload.rs` - Upload to Roblox.com - `upload.rs` - Upload to Roblox.com
- `sourcemap.rs` - Generate sourcemaps - `sourcemap.rs` - Generate sourcemaps
@ -92,17 +92,19 @@ Reverse: Roblox file → filesystem
- `snapshot.rs` - Convert instances to snapshots - `snapshot.rs` - Convert instances to snapshots
- `fs_snapshot.rs` - Filesystem representation - `fs_snapshot.rs` - Filesystem representation
- `matching.rs` - Syncback matching algorithm (recursive change-count scoring) - `matching.rs` - Syncback matching algorithm (recursive change-count scoring, `MatchingSession` caching)
- `dedup_suffix.rs` - Dedup cleanup rules (gap-tolerant, base-name promotion, group-to-1) - `dedup_suffix.rs` - Dedup cleanup rules (gap-tolerant, base-name promotion, group-to-1)
- `meta.rs` - Meta file read/write helpers (upsert/remove name, update ref paths) - `meta.rs` - Meta file read/write helpers (upsert/remove name, update ref paths)
- `ref_properties.rs` - Ref property path linking (`Rojo_Ref_*`, tentative_fs_path) - `ref_properties.rs` - Ref property path linking (`Rojo_Ref_*`, tentative_fs_path, placeholder system)
- `hash/` - Content hashing for change detection - `hash/` - Content hashing for change detection
- `property_filter.rs` - Apply syncback rules - `fs_snapshot.rs` - Filesystem representation, parallel writes (`write_to_vfs_parallel`), ref path post-processing (`fix_ref_paths`)
- `property_filter.rs` - Apply syncback rules (default: `sync_unscriptable = false`)
### 5. Web Server (`src/web/`) ### 5. Web Server (`src/web/`)
- `api.rs` - HTTP/WebSocket endpoints - `api.rs` - HTTP/WebSocket endpoints, including `POST /api/syncback` handler (`handle_api_syncback`)
- `interface.rs` - Browser UI - `interface.rs` - Wire types: patches, `SyncbackRequest`, `ServiceChunk`, `SyncbackPayload`, `ServerExitReason`
- `mod.rs` - `SyncbackSignal` (Mutex + Notify pattern), `LiveServer` (uses `tokio::select!` to wait for connections or syncback signal)
- Protocol version enforcement - Protocol version enforcement
### 6. Project System (`src/project.rs`) ### 6. Project System (`src/project.rs`)
@ -123,6 +125,8 @@ Parse/validate `.project.json5` (or legacy `.project.json`), handle nested proje
**Key:** `src/serve_session.rs` manages active sessions **Key:** `src/serve_session.rs` manages active sessions
**Live syncback integration:** The serve loop checks for `ServerExitReason::SyncbackRequested(payload)`. When received, calls `run_live_syncback()` which reconstructs the DOM from the plugin's rbxm blob + service chunks via `build_dom_from_chunks()`, runs `syncback_loop(incremental=false)` (clean mode), writes files via `write_to_vfs_parallel()`, refreshes git index, then restarts the serve loop.
### `rojo build` ### `rojo build`
1. Load project 1. Load project
@ -149,7 +153,7 @@ pub struct SyncbackRules {
ignore_properties: IndexMap<Ustr, Vec<Ustr>>, // Class → properties ignore_properties: IndexMap<Ustr, Vec<Ustr>>, // Class → properties
ignore_classes: Vec<String>, // Classes to skip entirely ignore_classes: Vec<String>, // Classes to skip entirely
sync_current_camera: Option<bool>, // Default: false sync_current_camera: Option<bool>, // Default: false
sync_unscriptable: Option<bool>, // Default: true sync_unscriptable: Option<bool>, // Default: false
ignore_referents: Option<bool>, // Skip Ref properties, default: false ignore_referents: Option<bool>, // Skip Ref properties, default: false
} }
``` ```
@ -220,6 +224,7 @@ Review: `cargo insta review`
**HTTP:** **HTTP:**
- `GET /` - Browser UI - `GET /` - Browser UI
- `GET /api/rojo` - Server info (version, project name, protocol) - `GET /api/rojo` - Server info (version, project name, protocol)
- `POST /api/syncback` - Live syncback: full Roblox → filesystem sync (validates protocol, version, place ID)
**WebSocket:** `ws://host:port/api/rojo` **WebSocket:** `ws://host:port/api/rojo`
@ -312,6 +317,7 @@ to_writer(&mut buffer, &dom, dom.root())?;
**Core:** rbx_binary, rbx_xml, rbx_dom_weak, memofs (local crate) **Core:** rbx_binary, rbx_xml, rbx_dom_weak, memofs (local crate)
**Async:** tokio, hyper, hyper-tungstenite **Async:** tokio, hyper, hyper-tungstenite
**Serialization:** serde, json5, toml, yaml-rust2, csv **Serialization:** serde, json5, toml, yaml-rust2, csv
**Float precision:** lexical-write-float (for roundtrip-safe f32/f64 formatting in JSON)
**Utilities:** clap, anyhow, thiserror, crossbeam-channel, rayon, globset **Utilities:** clap, anyhow, thiserror, crossbeam-channel, rayon, globset
## Module Organization ## Module Organization
@ -378,10 +384,11 @@ src/
matching.rs - Syncback matching algorithm matching.rs - Syncback matching algorithm
dedup_suffix.rs - Dedup cleanup rules dedup_suffix.rs - Dedup cleanup rules
meta.rs - Meta file helpers meta.rs - Meta file helpers
web/ - HTTP/WebSocket web/ - HTTP/WebSocket (incl. SyncbackSignal, LiveServer)
interface.rs - Wire types (patches, syncback requests)
change_processor.rs - Two-way sync filesystem writer change_processor.rs - Two-way sync filesystem writer
git.rs - Git integration (changed files, SHA1 hashing, auto-staging) git.rs - Git integration (changed files, SHA1 hashing, auto-staging)
rojo_ref.rs - Ref path system (Rojo_Ref_*, RefPathIndex) rojo_ref.rs - Ref path system (Rojo_Ref_*, RefPathIndex, Luau-style relative paths)
variant_eq.rs - Property value comparison variant_eq.rs - Property value comparison
lib.rs - Public API lib.rs - Public API
main.rs - CLI entry main.rs - CLI entry

View file

@ -114,6 +114,30 @@ pub fn syncback_loop(
**Implementation detail:** `old_ref` is ignored in `SyncbackSnapshot::with_joined_path` when `incremental=false`. **Implementation detail:** `old_ref` is ignored in `SyncbackSnapshot::with_joined_path` when `incremental=false`.
### Live Syncback Mode (via `atlas serve`)
Triggered via `POST /api/syncback` during an active `atlas serve` session. The plugin encodes the Studio state (service properties + children as rbxm blob) and POSTs directly -- no WebSocket connection required.
**Server-side flow:**
1. `handle_api_syncback()` validates protocol, version, place ID
2. Fires `SyncbackSignal` → `LiveServer` exits accept loop
3. `run_live_syncback()` in `src/cli/serve.rs`:
a. `build_dom_from_chunks()` reconstructs DOM from rbxm blob + `ServiceChunk` metadata
b. Runs `syncback_loop(incremental=false)` -- always clean mode
c. `write_to_vfs_parallel()` writes files with Windows retry logic
d. `refresh_git_index()` updates git
4. Serve loop restarts (new session)
**Key differences from CLI syncback:**
- Runs during an active serve session (not standalone)
- Receives data via HTTP (not file-based)
- Always clean mode (replaces everything)
- Server restarts after completion
**Plugin-side encoding:** `encodeService()` encodes service properties/attributes/refs (ObjectValue carriers for Ref), `SerializationService:SerializeInstancesAsync()` serializes all children into a single rbxm buffer.
See `atlas-api.mdc` for wire format details (`SyncbackRequest`, `ServiceChunk`).
## Matching Algorithm (`matching.rs`) ## Matching Algorithm (`matching.rs`)
Pairs new instances (from Roblox file) with old instances (from filesystem) to determine which files to update vs create. Pairs new instances (from Roblox file) with old instances (from filesystem) to determine which files to update vs create.
@ -133,6 +157,8 @@ Pairs new instances (from Roblox file) with old instances (from filesystem) to d
**Output:** `MatchResult { matched: Vec<(new_ref, old_ref)>, unmatched_new: Vec<new_ref>, unmatched_old: Vec<old_ref> }` **Output:** `MatchResult { matched: Vec<(new_ref, old_ref)>, unmatched_new: Vec<new_ref>, unmatched_old: Vec<old_ref> }`
**Session caching:** `MatchingSession` wraps a `cost_cache: RefCell<HashMap<(Ref, Ref), u32>>` to cache `compute_change_count` results, avoiding redundant recursive scoring when the same pairs are encountered at different tree levels.
**Known limitation:** Greedy assignment is not globally optimal. The Hungarian algorithm (O(N^3)) would give optimal results, but ambiguous groups are typically small (2-10 instances). **Known limitation:** Greedy assignment is not globally optimal. The Hungarian algorithm (O(N^3)) would give optimal results, but ambiguous groups are typically small (2-10 instances).
## Dedup Suffix Management (`dedup_suffix.rs`) ## Dedup Suffix Management (`dedup_suffix.rs`)
@ -171,7 +197,7 @@ pub struct SyncbackRules {
/// Whether to sync Workspace.CurrentCamera (default: false) /// Whether to sync Workspace.CurrentCamera (default: false)
pub sync_current_camera: Option<bool>, pub sync_current_camera: Option<bool>,
/// Whether to sync unscriptable properties (default: true) /// Whether to sync unscriptable properties (default: false)
pub sync_unscriptable: Option<bool>, pub sync_unscriptable: Option<bool>,
/// Whether to skip Ref properties (default: false) /// Whether to skip Ref properties (default: false)
@ -340,10 +366,10 @@ fs_snapshot.merge(other) // Combine snapshots
// Apply to filesystem // Apply to filesystem
fs_snapshot.write_to_vfs(base_path, vfs)?; fs_snapshot.write_to_vfs(base_path, vfs)?;
fs_snapshot.write_to_vfs_parallel(base_path, vfs)?; // Parallel writes with retry logic fs_snapshot.write_to_vfs_parallel(base_path, vfs)?; // Parallel writes (see below)
// Post-process ref paths // Post-process ref paths
fs_snapshot.fix_ref_paths(&substitutions); // Replace tentative paths with final dedup'd paths fs_snapshot.fix_ref_paths(&substitutions); // Replace placeholder paths with final relative paths
// Query // Query
fs_snapshot.is_empty() fs_snapshot.is_empty()
@ -357,18 +383,22 @@ fs_snapshot.removed_paths()
**Solution: Path-based `Rojo_Ref_*` attributes (preferred)** **Solution: Path-based `Rojo_Ref_*` attributes (preferred)**
Syncback writes filesystem-name-based paths as `Rojo_Ref_PropertyName` attributes: Syncback writes **Luau-style relative paths** as `Rojo_Ref_PropertyName` attributes via `compute_relative_ref_path()`:
```json5 ```json5
{ {
"properties": {}, "properties": {},
"attributes": { "attributes": {
"Rojo_Ref_PrimaryPart": "Workspace/Handle.model.json5" "Rojo_Ref_PrimaryPart": "./Handle.model.json5"
} }
} }
``` ```
Forward sync reads `Rojo_Ref_*` attributes and resolves them to tree instances via `get_instance_by_path()`. Relative path prefixes: `@self` (same/descendant), `./` (sibling), `../` (parent's sibling), `@game/` (cross-service).
Forward sync resolves both relative and absolute formats via `resolve_ref_path_to_absolute()` + `get_instance_by_path()`.
**Placeholder system:** During the syncback walk, before dedup suffixes are finalized, ref paths are written as `__ROJO_REF_<source>_TO_<target>__` placeholders (encoding both source and target refs). After the walk assigns final filesystem names, `fix_ref_paths()` substitutes placeholders with actual relative paths. Substitutions are sorted by length descending to prevent partial prefix matches.
**Legacy: `Rojo_Target_*` / `$id` system (still supported for reading)** **Legacy: `Rojo_Target_*` / `$id` system (still supported for reading)**
@ -498,12 +528,9 @@ Filesystem:
## Integration with Plugin ## Integration with Plugin
**Plugin does NOT handle syncback.** It only handles: **Live syncback (plugin-initiated):** The plugin encodes the Studio state via `encodeService()` + `SerializationService` and POSTs to `/api/syncback`. The server reconstructs the DOM, runs `syncback_loop(incremental=false)`, writes files, and restarts. Available from the NotConnected page -- no active WebSocket session required. See `atlas-api.mdc` for wire format.
1. Downloading place file from Studio
2. Sending file to server
3. Server runs syncback command
**Two-way sync** is different - see `atlas-api.mdc`. **Two-way sync** is a separate system for incremental bidirectional changes -- see `atlas-api.mdc`.
## Performance Considerations ## Performance Considerations
@ -511,7 +538,9 @@ Filesystem:
**Incremental benefit:** On large projects, incremental mode with hashing can be 10-100x faster than rewriting everything. **Incremental benefit:** On large projects, incremental mode with hashing can be 10-100x faster than rewriting everything.
**Parallelization:** File writes can be parallelized via `write_to_vfs_parallel()` (includes Windows retry logic). **Parallelization:** File writes can be parallelized via `write_to_vfs_parallel()`:
- 4-phase approach: create dirs (sequential) → write files (parallel via rayon) → remove files (parallel) → remove dirs (sequential via VFS)
- Windows retry logic: exponential backoff (10ms initial, doubles each retry, max 3 retries) for OS errors 5 (Access denied) and 32 (Sharing violation)
## Debugging ## Debugging
@ -546,7 +575,7 @@ diff place.rbxl rebuilt.rbxl
- Better handling of Roblox format changes - Better handling of Roblox format changes
- Support for partial syncback (only changed instances) - Support for partial syncback (only changed instances)
- Globally optimal matching (Hungarian algorithm) for large ambiguous groups - Globally optimal matching (Hungarian algorithm) for large ambiguous groups
- Matching cache (parent_ref -> match_assignments) for avoiding redundant scoring - ~~Matching cache (parent_ref -> match_assignments) for avoiding redundant scoring~~ (session-level caching now implemented via `MatchingSession`; per-parent invalidation cache still deferred)
## Key Files to Reference ## Key Files to Reference
@ -558,5 +587,9 @@ diff place.rbxl rebuilt.rbxl
- `src/syncback/meta.rs` - Meta file read/write helpers - `src/syncback/meta.rs` - Meta file read/write helpers
- `src/syncback/ref_properties.rs` - Ref property path linking - `src/syncback/ref_properties.rs` - Ref property path linking
- `src/syncback/hash/` - Hashing algorithm - `src/syncback/hash/` - Hashing algorithm
- `src/syncback/fs_snapshot.rs` - Filesystem operations, parallel writes, ref path post-processing
- `src/syncback/property_filter.rs` - Property filtering logic - `src/syncback/property_filter.rs` - Property filtering logic
- `src/cli/syncback.rs` - CLI command implementation - `src/cli/syncback.rs` - CLI command implementation
- `src/cli/serve.rs` - Live syncback handler (`run_live_syncback`, `build_dom_from_chunks`)
- `src/web/interface.rs` - `SyncbackRequest`, `ServiceChunk`, `SyncbackPayload` types
- `src/rojo_ref.rs` - Luau-style relative ref paths (`compute_relative_ref_path`, `resolve_ref_path_to_absolute`)

View file

@ -32,6 +32,7 @@ Atlas bridges filesystem and Roblox Studio, enabling professional development to
- Build/upload rbxl/rbxm files from command line - Build/upload rbxl/rbxm files from command line
- Syncback: pull instances from Roblox → filesystem (7.7.0+) - Syncback: pull instances from Roblox → filesystem (7.7.0+)
- Two-way sync (plugin-based): live bidirectional sync during `atlas serve` sessions - Two-way sync (plugin-based): live bidirectional sync during `atlas serve` sessions
- Live syncback: one-click full sync from Studio to filesystem during `atlas serve` (no WebSocket connection required)
### Two-Way Sync Strategy ### Two-Way Sync Strategy
@ -206,7 +207,7 @@ pub struct InstanceContext {
} }
``` ```
**Filename handling:** Instance names with forbidden filesystem chars are slugified (forbidden chars → `_`), deduplicated (`~1`, `~2`, etc. on collision), and the real name is stored in `.meta.json5`/`.model.json5` `name` fields. See `src/syncback/file_names.rs`. **Filename handling:** Instance names with forbidden filesystem chars are slugified (forbidden chars → `_`), deduplicated (`~2`, `~3`, etc. on collision), and the real name is stored in `.meta.json5`/`.model.json5` `name` fields. See `src/syncback/file_names.rs`.
**VFS passed separately to snapshot functions:** **VFS passed separately to snapshot functions:**
```rust ```rust
@ -322,6 +323,7 @@ src/ - Rust server/CLI
dedup_suffix.rs - Dedup cleanup rules dedup_suffix.rs - Dedup cleanup rules
meta.rs - Meta file read/write helpers meta.rs - Meta file read/write helpers
web/ - HTTP/WebSocket server web/ - HTTP/WebSocket server
interface.rs - Wire types (patches, syncback requests)
change_processor.rs - Two-way sync filesystem writer change_processor.rs - Two-way sync filesystem writer
git.rs - Git integration (changed files, SHA1 hashing, auto-staging) git.rs - Git integration (changed files, SHA1 hashing, auto-staging)
rojo_ref.rs - Ref path system (Rojo_Ref_*, RefPathIndex) rojo_ref.rs - Ref path system (Rojo_Ref_*, RefPathIndex)
@ -333,6 +335,7 @@ plugin/src/ - Lua Studio plugin
matching.lua - Lua matching algorithm matching.lua - Lua matching algorithm
trueEquals.lua - Shared value equality module trueEquals.lua - Shared value equality module
ChangeBatcher/ - Two-way sync patch creation ChangeBatcher/ - Two-way sync patch creation
encodeService.lua - Service-level property encoding for live syncback
Config.lua - Version and protocol config Config.lua - Version and protocol config
PatchTree.lua - Patch visualization tree (git-based default selections) PatchTree.lua - Patch visualization tree (git-based default selections)
SHA1.luau - SHA1 hash for git blob format comparison SHA1.luau - SHA1 hash for git blob format comparison

View file

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

Before After
Before After

View file

@ -6,11 +6,11 @@ A builtin 'integer' type to represent 64 bit integer numbers.
## Motivation ## Motivation
Current methods of representing 64-bit integers in Luau with default environment are done with high-low pairs or by splitting them among vectors which leads to poor ergonomics and may not be sufficient for cases where performance is important. Current methods of representing 64-bit integers in Luau with the default environment are done with high-low pairs or by splitting them among vectors which leads to poor ergonomics and may not be sufficient for cases where performance is important.
In the case of a high-low pair implementation, each individual number takes 16 bytes (or 24) and needs to be be handled together which can be confusing. In the case of a high-low pair implementation, each individual number takes 16 bytes (or 24) and needs to be handled together which can be confusing.
In the case of vector implementations, ergonomics are improved by storing the entire integer in one value however you have the same issues of implementation complexity and lack of integration with the type system. In the case of vector implementations, ergonomics are improved by storing the entire integer in one value. However you have the same issues of implementation complexity and lack of integration with the type system.
In both cases, performance is lacking compared to what could be provided by a native implementation. In both cases, performance is lacking compared to what could be provided by a native implementation.
@ -21,11 +21,11 @@ Besides the above solutions in userland, runtimes are also able to define their
Further, representing 64-bit numbers as heap-allocated userdata is sufficient for correctness, but also adds a layer of indirection for something that could fit within the existing value size. In domains where the use of these numbers is more common, the performance characteristics of these varying implementation techniques could be problematic or even prohibitive. Further, representing 64-bit numbers as heap-allocated userdata is sufficient for correctness, but also adds a layer of indirection for something that could fit within the existing value size. In domains where the use of these numbers is more common, the performance characteristics of these varying implementation techniques could be problematic or even prohibitive.
Userdata remain the right mechanism for complex or embedder-specific numeric types, but 64-bit integers have a sufficiently large problem domain to justify their inclusion as a core value type, and bring significant benefits for performance and interoperability in the overall Luau ecosystem. Userdata remains the right mechanism for complex or embedder-specific numeric types, but 64-bit integers have a sufficiently large problem domain to justify their inclusion as a core value type, and bring significant benefits for performance and interoperability in the overall Luau ecosystem.
## Design ## Design
This will be implemented a with new type called `integer`. This will be implemented with a new type called `integer`.
An additional character may be specified at the end of numeric literals `i` which will signify an 64 bit integer literal. An additional character may be specified at the end of numeric literals `i` which will signify an 64 bit integer literal.
64 bit integer literals will support separators, hex, and binary values: 64 bit integer literals will support separators, hex, and binary values:
@ -43,10 +43,18 @@ Operations performing a string formatting of an integer should format the number
Integer values will have a built-in equality comparison, but will not have any other operators or metamethods defined. Integer values will have a built-in equality comparison, but will not have any other operators or metamethods defined.
Integers are never automatically converted to numbers or strings, and vice-versa.
Passing an integer to a function expecting a number (or string) will result in a type error.
While there are no arithmentic operators defined for integers in this RFC, it is still worth noting that operations between `number` and `integer` are also not supported.
Functions for creating and manipulating this type will exist in a new library called 'integer`. Functions for creating and manipulating this type will exist in a new library called 'integer`.
### Library ### Library
Unless otherwise specified, all operations interpret integer values as signed two's complement 64 bit numbers.
Operations that treat the value as unsigned are explicitly prefixed with 'u'.
`function integer.create(n: number): integer?` `function integer.create(n: number): integer?`
Converts a number to an integer. Converts a number to an integer.
@ -64,7 +72,7 @@ String is allowed to have leading and trailing spaces and number can be preceded
If base is not specified, number can have a `0x` or `0X` prefix to be converted in base 16, otherwise base 10 is used. If base is not specified, number can have a `0x` or `0X` prefix to be converted in base 16, otherwise base 10 is used.
When base is specified, it has to be in range between 2 and 36 inclusive. When base is specified, it has to be in range between 2 and 36 inclusive.
In base 10 and base 16, number is allowed to have a `0x` or `0X` prefix. In base 10 and base 16, numbers are allowed to have a `0x` or `0X` prefix.
Returns `nil` if the string doesn't contain a number. Returns `nil` if the string doesn't contain a number.
@ -102,7 +110,7 @@ Overflow wraps around according to rules of two-complement arithmetic.
`function integer.div(a: integer, b: integer): integer` `function integer.div(a: integer, b: integer): integer`
Performes signed truncated division of `a` by `b`. Performs signed truncated division of `a` by `b`.
If `b` is 0, throws a division by zero error. If `b` is 0, throws a division by zero error.
@ -118,7 +126,7 @@ If `a` is -2^63 and `b` is -1, result is 0.
`function integer.idiv(a: integer, b: integer): integer` `function integer.idiv(a: integer, b: integer): integer`
Performes signed floored division of `a` by `b`. Performs signed floored division of `a` by `b`.
If `b` is 0, throws a division by zero error. If `b` is 0, throws a division by zero error.
@ -126,7 +134,7 @@ If `a` is -2^63 and `b` is -1, throws an overflow error.
`function integer.mod(a: integer, b: integer): integer` `function integer.mod(a: integer, b: integer): integer`
Performes signed floored modulus division of `a` by `b`. Performs signed floored modulus division of `a` by `b`.
If `b` is 0, throws a division by zero error. If `b` is 0, throws a division by zero error.
@ -293,9 +301,9 @@ Integer value representing -2^63 (`-9_223_372_036_854_775_808i`)
`string.format` function is updated to support integer arguments. `string.format` function is updated to support integer arguments.
'd', 'i' and '*' format specifiers will format integer as a signed 64 bit integer number. 'd', 'i' and '*' format specifiers will format an integer as a signed 64 bit integer number.
'o', 'u', 'x' and 'X' format specifiers will format integer as an unsigned 64 bit integer number. 'o', 'u', 'x' and 'X' format specifiers will format an integer as an unsigned 64 bit integer number.
### C API ### C API
@ -307,7 +315,7 @@ If `isinteger` is not a null pointer, writes 1 if the value at `idx` was an inte
`void lua_pushinteger64(lua_State *L, int64_t n)` `void lua_pushinteger64(lua_State *L, int64_t n)`
Pushes a integer value on top of the stack. Pushes an integer value on top of the stack.
`int lua_isinteger64(lua_State *L, int idx)` `int lua_isinteger64(lua_State *L, int idx)`
@ -332,8 +340,54 @@ Included in `luaL_openlibs`.
## Drawbacks ## Drawbacks
This increases the complexity of the VM as it may need to implement separate paths for mathematical operations. This introduces a library with a large set of functions which will likely take a large number of fastcall slots.
The lack of operator overloads causes code with integers to be more verbose, decreasing readability in complex expressions.
Introduction of a new type will require updates to libraries (serialization, input/output) which try to handle all potential Luau values.
The suffix 'i' is standard mathematical notation for imaginary numbers.
Adopting it for integers will likely block proposals for a built-in `complex` number type in the future.
## Alternatives ## Alternatives
Do nothing, the workarounds exist and are in use but this will restrict the areas Luau can be used without implementation burden or reliance on an externally maintained implementation of 64-bit integers. Do nothing, the workarounds exist and are in use but this will restrict the areas Luau can be used without implementation burden or reliance on an externally maintained implementation of 64-bit integers.
### Operator overloads
This proposal does not introduce overloaded operators aside from equality checks.
This omission is deliberate.
While the lack of operators makes code more verbose, Luau's primary numeric type remains `number`.
The `integer` type is intended to cover specialized use cases, many of which will specifically prefer explicit library functions over operators for performance.
Polymorphic overloaded operators present many challenges and can cause unexpected drops of performance.
This is already true when working with `vector` types without type annotations and given the focus of this proposal, we do not want to provide additional pitfalls.
Monomorphic library functions allow compiler and runtime to optimize more aggressively.
Heavy uses of 64-bit integers often rely on bitwise operations.
Since Luau does not have bitwise operators in its syntax, even with a subset of overloaded operators in place, code will still end up using library functions.
Integer division has multiple conflicting definitions (truncated vs. floored).
Operators like `/` and `%` would force a specific default behavior that might not match the specific rounding that the user intended.
Default behavior might also be less performant, causing developers to still rely on library methods.
Having different library methods allows the developers to choose the specific operation semantics they require.
### Separate signed and unsigned types
It has been suggested to create two separate types for signed and unsigned integers or have it be a property of the value.
Having distinct types would provide the typechecker with the ability to highlight mixed operations and a runtime check could throw an error.
As mentioned above, integers are already not expected to be the default value type to be chosen by developers to work with.
Unsigned integer use is even rarer and other programming languages followed the similar path of only supporting auxiliary operations on them.
Other languages with unsigned integer support like C++ recommend limiting their use in their guidelines.
Given this, we have rejected the idea of distinct signed and unsigned integer types.
Having two distinct integer types increases the API and library surface area and implementation complexity significantly for little gain.
In two's complement representation, a single data type with default signed interpretation covers many signed and unsigned operations in an identical way.
Only a limited subset of operations interpreting data as unsigned is added in this proposal to cover most needs for unsigned integer numbers.
Having library methods instead of operators allows the developer to make an explicit choice of the interpretation required (e.g. `lt` vs `ult`).
Reinterpretation of a negative value as a positive unsigned value can be made deliberately by the choice of function, and is not as error-prone as using an operator.

View file

@ -0,0 +1,133 @@
---
slug: compatibility
title: Compatibility with Lua
toc: true
---
Luau is based on Lua 5.1, and as such incorporates all features of 5.1, except for ones that had to be taken out due to sandboxing limitations. Because of backwards compatibility constraints, we don't remove features deprecated by later versions (e.g. we still support `getfenv`/`setfenv`). Later Lua versions introduce new features into the language and new libraries/functions.
Our overall goal is to incorporate features from the later versions of Lua when it makes sense for us to do so - the motivations behind some newer features are unclear or don't apply to the domain Luau is used in, and many features carry costs that don't always make sense to pay. The rest of this document describes the status of all features of Lua 5.2 and beyond, with the following classification:
- ✔️ - the feature is available in Luau
- ❌ - the feature is not available in Luau because we don't believe it makes sense to include it
- 😞 - the feature is not available in Luau because of compatibility/sandboxing concerns
- 🔜 - the feature is not available in Luau yet but we'd like to include it and are possibly working on it
- 🤷‍♀️ - the feature is not available in Luau yet; we don't have strong opinions on it so it might make it at some point
Please note that all of these decisions are not final, they are just our current stance. In some cases evolution of our VM may make a feature that was previously impractical to support due to performance complications feasible. In some cases a feature that didn't have a strong use case gains one, so we can implement it.
## Implementation limits
Luau has certain limitations around the number of local variables, registers, upvalues, constants and instructions. These limits are often different from the limits imposed by various versions of Lua, and are documented here without promising that future versions will adhere to these. Note that writing code that is close to any of these limits is dangerous because this code may become invalid as our codegen evolves.
- Local variables: 200 per function (same as all versions of Lua, this includes function arguments)
- Upvalues: 200 per function (up from 60 in Lua 5.1)
- Registers: 255 per function (same as all versions of Lua, this includes local variables and function arguments)
- Constants: 2^23 per function (up from 2^18 in Lua 5.1)
- Instructions: 2^23 per function (up from 2^17 in Lua 5.1, although in both cases the limit only applies to control flow)
- Nested functions: 2^15 per function (down from 2^18 in Lua 5.1)
- Stack depth: 20000 Lua calls per Lua thread, 200 C calls per C thread (e.g. `coroutine.resume`/`pcall` nesting is limited to 200)
Note that Lua 5.3 has a larger upvalue limit (255) and a larger constant limit (2^26); existing Luau limits are likely sufficient for reasonable use cases.
## Lua 5.1
Since several features were removed from Lua 5.1 for sandboxing reasons, this table lists them for completeness.
| feature | notes |
|---------|------|
| `io`, `os`, `package` and `debug` library | note that some functions in `os`/`debug` are still present |
| `loadfile`, `dofile` | removed for sandboxing, no direct file access |
| `loadstring` bytecode and `string.dump` | exposing bytecode is dangerous for sandboxing reasons |
| `newproxy` can only be called with nil or boolean | extra flexibility removed for sandboxing |
Sandboxing challenges are [covered in the dedicated section](../sandbox).
## Lua 5.2
| feature | status | notes |
|---------|--------|------|
| yieldable pcall/xpcall | ✔️ | |
| yieldable xpcall error handler | ❌ | no strong use cases, VM complexity and performance implications |
| yieldable metamethods | ❌ | significant performance implications |
| ephemeron tables | ❌ | this complicates and slows down the garbage collector esp. for large weak tables |
| emergency garbage collector | 🤷‍ | Luau runs in environments where handling memory exhaustion in emergency situations is not tenable |
| goto statement | ❌ | this complicates the compiler, makes control flow unstructured and doesn't address a significant need |
| finalizers for tables | ❌ | no `__gc` support due to sandboxing and performance/complexity |
| no more fenv for threads or functions | 😞 | we love this, but it breaks compatibility |
| tables honor the `__len` metamethod | ✔️ | |
| hex and `\z` escapes in strings | ✔️ | |
| support for hexadecimal floats | 🤷‍♀️ | no strong use cases |
| order metamethods (`__lt`/`__le`) are called for unrelated metatables | ❌ | no strong use cases and more complicated semantics, compatibility and performance implications |
| empty statement | 🤷‍♀️ | less useful in Lua than in JS/C#/C/C++ |
| `break` statement may appear in the middle of a block | 🤷‍♀️ | we'd like to do it consistently for `break`/`return`/`continue` but there be dragons |
| arguments for function called through `xpcall` | ✔️ | |
| optional base in `math.log` | ✔️ | |
| optional separator in `string.rep` | 🤷‍♀️ | no strong use cases |
| new metamethods `__pairs` and `__ipairs` | ❌ | superseded by `__iter` |
| frontier patterns | ✔️ | |
| `%g` in patterns | ✔️ | |
| `\0` in patterns | ✔️ | |
| `bit32` library | ✔️ | |
| `string.gsub` is stricter about using `%` on special characters only | ✔️ | |
| light C functions | 😞 | this changes semantics of fenv on C functions and has complex implications wrt runtime performance |
| NaN keys are supported for tables with `__newindex` | ✔️ | |
Two things that are important to call out here are various new metamethods for tables and yielding in metamethods. In both cases, there are performance implications to supporting this - our implementation is *very* highly tuned for performance, so any changes that affect the core fundamentals of how Lua works have a price. To support yielding in metamethods we'd need to make the core of the VM more involved, since almost every single "interesting" opcode would need to learn how to be resumable - which also complicates future JIT/AOT story. Metamethods in general are important for extensibility, but very challenging to deal with in implementation, so we err on the side of not supporting any new metamethods unless a strong need arises.
For `__pairs`/`__ipairs`, we felt that extending library functions to enable custom containers wasn't the right choice. Instead we revisited iteration design to allow for self-iterating objects via `__iter` metamethod, which results in a cleaner iteration design that also makes it easier to iterate over tables. As such, we have no plans to support `__pairs`/`__ipairs` as all use cases for it can now be solved by `__iter`.
Ephemeron tables may be implemented at some point since they do have valid uses and they make weak tables semantically cleaner, however the cleanup mechanism for these is expensive and complicated, and as such this can only be considered after the pending GC rework is complete.
## Lua 5.3
| feature | status | notes |
|---------|--------|------|
| `\u` escapes in strings | ✔️ | |
| integers (64-bit by default) | ❌ | backwards compatibility and performance implications |
| bitwise operators | ❌ | `bit32` library covers this in absence of 64-bit integers |
| basic utf-8 support | ✔️ | we include `utf8` library and other UTF8 features |
| functions for packing and unpacking values (string.pack/unpack/packsize) | ✔️ | |
| floor division | ✔️ | |
| `ipairs` and the `table` library respect metamethods | ❌ | no strong use cases, performance implications |
| new function `table.move` | ✔️ | |
| `collectgarbage("count")` now returns only one result | ✔️ | |
| `coroutine.isyieldable` | ✔️ | |
| stricter error checking for `table.insert`/`table.remove` | 😞 | we love this, but it breaks compatibility |
| `__eq` metamethod is called for unrelated metatables | ❌ | backwards compatibility and typechecking implications |
It's important to highlight integer support and bitwise operators. For Luau, it's rare that a full 64-bit integer type is necessary - double-precision types support integers up to 2^53 (in Lua which is used in embedded space, integers may be more appealing in environments without a native 64-bit FPU). However, there's a *lot* of value in having a single number type, both from performance perspective and for consistency. Notably, Lua doesn't handle integer overflow properly, so using integers also carries compatibility implications.
If integers are taken out of the equation, bitwise operators make less sense, as integers aren't a first class feature; additionally, `bit32` library is more fully featured (includes commonly used operations such as rotates and arithmetic shift; bit extraction/replacement is also more readable). Adding operators along with metamethods for all of them increases complexity, which means this feature isn't worth it on the balance. Common arguments for this include a more familiar syntax, which, while true, gets more nuanced as `^` isn't available as a xor operator, and arithmetic right shift isn't expressible without yet another operator, and performance, which in Luau is substantially better than in Lua because `bit32` library uses VM builtins instead of expensive function calls.
## Lua 5.4
| feature | status | notes |
|--|--|--|
| new generational mode for garbage collection | 🔜 | we're working on gc optimizations and generational mode is on our radar
| to-be-closed variables | ❌ | the syntax is inconsistent with how we'd like to do attributes long-term; no strong use cases in our domain |
| const variables | ❌ | while there's some demand for const variables, we'd never adopt this syntax |
| new implementation for math.random | ✔️ | our RNG is based on PCG, unlike Lua 5.4 which uses Xoroshiro |
| optional `init` argument to `string.gmatch` | 🤷‍♀️ | no strong use cases |
| new functions `lua_resetthread` and `coroutine.close` | ✔️ ||
| coercions string-to-number moved to the string library | 😞 | we love this, but it breaks compatibility |
| new format `%p` in `string.format` | 🤷‍♀️ | no strong use cases |
| `utf8` library accepts codepoints up to 2^31 | 🤷‍♀️ | no strong use cases |
| The use of the `__lt` metamethod to emulate `__le` has been removed | ❌ | breaks compatibility and complicates comparison overloading story |
| When finalizing objects, Lua will call `__gc` metamethods that are not functions | ❌ | no `__gc` support due to sandboxing and performance/complexity |
| function `print` calls `__tostring` instead of `tostring` to format its arguments | ✔️ | |
| decoding functions in the utf8 library do not accept surrogates | ✔️ | |
Taking syntax aside (which doesn't feel idiomatic or beautiful), `<close>` isn't very useful in Luau - its dominant use case is for code that works with external resources like files or sockets, but we don't provide such APIs - and has a very large complexity cost, evidences by a lot of bug fixes since the initial implementation in 5.4 work versions. `<const>` in Luau doesn't matter for performance - our multi-pass compiler is already able to analyze the usage of the variable to know if it's modified or not and extract all performance gains from it - so the only use here is for code readability, where the `<const>` syntax is... suboptimal.
If we do end up introducing const variables, it would be through a `const var = value` syntax, which is backwards compatible through a context-sensitive keyword similar to `type`. That said, there's ambiguity wrt whether `const` should simply behave like a read-only variable, ala JavaScript, or if it should represent a stronger contract, for example by limiting the expressions on the right hand side to ones compiler can evaluate ahead of time, or by freezing table values and thus guaranteeing immutability.
## Differences from Lua
We have a few behavior deviations from Lua 5.x that come from either a different implementation, or our desire to clean up small inconsistencies in the language/libraries:
* Tail calls are not supported to simplify implementation, make debugging/stack traces more predictable and allow deep validation of caller identity for security
* Order of table assignment in table literals follows program order in mixed tables (Lua 5.x assigns array elements first in some cases)
* Equality comparisons call `__eq` metamethod even when objects are rawequal (which matches other metamethods like `<=` and facilitates NaN checking)
* `function()` expressions may reuse a previously created closure in certain scenarios (when all upvalues captured are the same) for efficiency, which changes object identity but doesn't change call semantics -- this is different from Lua 5.1 but similar to Lua 5.2/5.3
* `os.time` returns UTC timestamp when called with a table for consistency

View file

@ -0,0 +1,147 @@
---
slug: getting-started
title: An introduction to Luau
sidebar:
order: 1
---
Luau is a fast, small, safe, gradually typed embeddable scripting language derived from Lua 5.1. Luau ships as a command line tool for running, analyzing, and linting your Luau scripts, and is also integrated with Roblox Studio. Roblox developers should also visit our [Creator Docs Luau Section](https://create.roblox.com/docs/luau).
To get started with Luau, you can use the `luau` command line binary to run your code and `luau-analyze` to run static analysis (including type checking and linting). You can download these from [a recent release](https://github.com/luau-lang/luau/releases).
## Creating a script
To create your own testing script, create a new file with `.luau` as the extension:
```luau
--!hidden mode=nonstrict
function ispositive(x)
return x > 0
end
print(ispositive(1))
print(ispositive("2"))
function isfoo(a)
return a == "foo"
end
print(isfoo("bar"))
print(isfoo(1))
```
You can now run the file using `luau test.luau` and analyze it using `luau-analyze test.luau`.
Note that there are no warnings about calling ``ispositive()`` with a string, or calling ``isfoo()`` with a number. This is because Luau's type checking uses non-strict mode by default, which only reports errors if it's certain a program will error at runtime.
## Type checking
Now modify the script to include ``--!strict`` at the top:
```luau
--!strict
function ispositive(x)
return x > 0
end
print(ispositive(1))
print(ispositive("2"))
```
We just mentioned ``nonstrict`` mode, where we only report errors if we can prove that a program will error at runtime. We also have ``strict`` mode, which will report errors if a program might error at runtime.
In this case, Luau will use the ``return x > 0`` statement to infer that ``ispositive()`` is a function taking a number and returning a boolean. Note that in this case, we were able to determine the type of `ispositive` without the presence of any explicit type annotations.
Based on Luau's type inference, the analysis tool will now flag the incorrect call to ``ispositive()``:
```
$ luau-analyze test.luau
test.luau(7,18): TypeError: Type 'string' could not be converted into 'number'
```
## Annotations
You can add annotations to locals, arguments, and function return types. Among other things, annotations can help enforce that you don't accidentally do something silly. Here's how we would add annotations to ``ispositive()``:
```luau
--!strict
function ispositive(x : number) : boolean
return x > 0
end
local result : boolean
result = ispositive(1)
```
Now we've explicitly told Luau that ``ispositive()`` accepts a number and returns a boolean. This wasn't strictly (pun intended) necessary in this case, because Luau's inference was able to deduce this already. But even in this case, there are advantages to explicit annotations. Imagine that later we decide to change ``ispositive()`` to return a string value:
```luau
--!strict
function ispositive(x : number) : boolean
if x > 0 then
return "yes"
else
return "no"
end
end
local result : boolean
result = ispositive(1)
```
Oops -- we're returning string values, but we forgot to update the function return type. Since we've told Luau that ``ispositive()`` returns a boolean (and that's how we're using it), the call site isn't flagged as an error. But because the annotation doesn't match our code, we get a warning in the function body itself:
```
$ luau-analyze test.luau
test.luau(5,9): TypeError: Type 'string' could not be converted into 'boolean'
test.luau(7,9): TypeError: Type 'string' could not be converted into 'boolean'
```
The fix is simple; just change the annotation to declare the return type as a string:
```luau
--!strict
function ispositive(x : number) : string
if x > 0 then
return "yes"
else
return "no"
end
end
local result : boolean
result = ispositive(1)
```
Well, almost - since we declared ``result`` as a boolean, the call site is now flagged:
```
$ luau-analyze test.luau
test.luau(12,10): TypeError: Type 'string' could not be converted into 'boolean'
```
If we update the type of the local variable, everything is good. Note that we could also just let Luau infer the type of ``result`` by changing it to the single line version ``local result = ispositive(1)``.
```luau
--!strict
function ispositive(x : number) : string
if x > 0 then
return "yes"
else
return "no"
end
end
local result : string
result = ispositive(1)
```
## Conclusions
This has been a brief tour of the basic functionality of Luau, but there's lots more to explore. If you're interested in reading more, check out our main reference pages for [syntax](/syntax) and [typechecking](/types).

View file

@ -0,0 +1,387 @@
---
slug: lint
title: Luau's linter
sidebar:
order: 4
---
Luau comes with a set of linting passes, that help make sure that the code is correct and consistent. Unlike the type checker, that models the behavior of the code thoroughly and points toward type mismatches that are likely to result in runtime errors, the linter is more opinionated and produces warnings that can often be safely ignored, although it's recommended to keep the code clean of the warnings.
Linter produces many different types of warnings; many of these are enabled by default, and can be suppressed by declaring `--!nolint NAME` at the top of the file. In dire situations `--!nolint` at the top of the file can be used to completely disable all warnings (note that the type checker is still active, and requires a separate `--!nocheck` declaration).
The rest of this page documents all warnings produced by the linter; each warning has a name and a numeric code, the latter is used when displaying warnings. Note that the in-browser version of Luau on this website does not currently use the linter.
## UnknownGlobal (1)
By default, variables in Luau are global (this is inherited from Lua 5.x and can't be changed because of backwards compatibility). This means that typos in identifiers are invisible to the parser, and often break at runtime. For this reason, the linter considers all globals that aren't part of the builtin global table and aren't explicitly defined in the script "unknown":
```luau
--!hidden mode=nocheck
local displayName = "Roblox"
-- Unknown global 'displaName'
print(displaName)
```
Note that injecting globals via `setfenv` can produce this warning in correct code; global injection is incompatible with type checking and has performance implications so we recommend against it and in favor of using `require` with correctly scoped identifiers.
## DeprecatedGlobal (2)
Some global names exist for compatibility but their use is discouraged. This mostly affects globals introduced by Roblox, and since they can have problematic behavior or can break in the future, this warning highlights their uses:
```luau
--!hidden mode=nocheck
-- Global 'ypcall' is deprecated, use 'pcall' instead
ypcall(function()
print("hello")
end)
```
## GlobalUsedAsLocal (3)
The UnknownGlobal lint can catch typos in globals that are read, but can't catch them in globals that are assigned to. Because of this, and to discourage the use of globals in general, linter detects cases when a global is only used in one function and can be safely converted to a local variable. Note that in some cases this requires declaring the local variable in the beginning of the function instead of where it's being assigned to.
```luau
--!hidden mode=nocheck
local function testFunc(a)
if a < 5 then
-- Global 'b' is only used in the enclosing function; consider changing it to local
b = 1
else
b = 2
end
print(b)
end
```
## LocalShadow (4)
In Luau, it is valid to shadow locals and globals with a local variable, including doing it in the same function. This can result in subtle bugs, since the shadowing may not be obvious to the reader. This warning detects cases where local variables shadow other local variables in the same function, or global variables used in the script.
```luau
--!hidden mode=nocheck
local function foo()
for i=1,10 do
-- Variable 'i' shadows previous declaration at line 2
for i=1,10 do
print(i)
end
end
end
```
## SameLineStatement (5)
Luau doesn't require the use of semicolons and doesn't automatically insert them at line breaks. When used wisely this results in code that is easy to read and understand, however it can cause subtle issues and hard to understand code when abused by using many different statements on the same line. This warning highlights cases where code should either be broken into multiple lines, or use `;` as a visual guide.
```luau
--!hidden mode=nocheck
-- A new statement is on the same line; add semi-colon on previous statement to silence
if b < 0 then local a = b + 1 print(a, b) end
```
## MultiLineStatement (6)
An opposite problem is having statements that span multiple lines. This is good for readability when the code is indented properly, but when it's not it results in code that's hard to understand, as its easy to confuse the next line for a separate statement.
```luau
--!hidden mode=nocheck
-- Statement spans multiple lines; use indentation to silence
print(math.max(1,
math.min(2, 3)))
```
## LocalUnused (7)
This warning is one of the few warnings that highlight unused variables. Local variable declarations that aren't used may indicate a bug in the code (for example, there could be a typo in the code that uses the wrong variable) or redundant code that is no longer necessary (for example, calling a function to get its result and never using this result). This warning warns about locals that aren't used; if the locals are not used intentionally they can be prefixed with `_` to silence the warning:
```luau
--!hidden mode=nocheck
local x = 1
-- Variable 'y' is never used; prefix with '_' to silence
local y = 2
print(x, x)
```
## FunctionUnused (8)
While unused local variables could be useful for debugging, unused functions usually contain dead code that was meant to be removed. Unused functions clutter code, can be a result of typos similar to local variables, and can mislead the reader into thinking that some functionality is supported.
```luau
--!hidden mode=nocheck
-- Function 'bar' is never used; prefix with '_' to silence
local function bar()
end
local function foo()
end
return foo()
```
## ImportUnused (9)
In Luau, there's no first-class module system that's part of the syntax, but `require` function acts as an import statement. When a local is initialized with a `require` result, and the local is unused, this is classified as "unused import". Removing unused imports improves code quality because it makes it obvious what the dependencies of the code are:
```luau
--!hidden mode=nocheck
-- Import 'Roact' is never used; prefix with '_' to silence
local Roact = require("@game/Packages/Roact")
```
## BuiltinGlobalWrite (10)
While the sandboxing model of Luau prevents overwriting built-in globals such as `table` for the entire program, it's still valid to reassign these globals - this results in "global shadowing", where the script's global table contains a custom version of `table` after writing to it. This is problematic because it disables some optimizations, and can result in misleading code. When shadowing built-in globals, use locals instead.
```luau
-- Built-in global 'math' is overwritten here; consider using a local or changing the name
math = {}
```
## PlaceholderRead (11)
`_` variable name is commonly used as a placeholder to discard function results. The linter follows this convention and doesn't warn about the use of `_` in various cases where a different name would cause a warning otherwise. To make sure the placeholder is only used to write values to it, this warning detects the cases where it's read instead:
```luau
local _ = 5
-- Placeholder value '_' is read here; consider using a named variable
return _
```
## UnreachableCode (12)
In some cases the linter can detect code that is never executed, because all execution paths through the function exit the function or the loop before reaching it. Such code is misleading because it's not always obvious to the reader that it never runs, and as such it should be removed.
```luau
function cbrt(v)
if v >= 0 then
return v ^ (1/3)
else
error('cbrt expects a non-negative argument')
end
-- Unreachable code (previous statement always returns)
return 0
end
```
## UnknownType (13)
Luau provides several functions to get the value type as a string (`type`, `typeof`), and some Roblox APIs expose class names through string arguments (`Instance.new`). This warning detects incorrect use of the type names by checking the string literals used in type comparisons and function calls.
```luau
--!hidden mode=nocheck
-- Unknown type 'String' (expected primitive type)
if type(v) == "String" then
print("v is a string")
end
```
## ForRange (14)
When using a numeric for, it's possible to make various mistakes when specifying the for bounds. For example, to iterate through the table backwards, it's important to specify the negative step size. This warning detects several cases where the numeric for only runs for 0 or 1 iterations, or when the step doesn't divide the size of the range evenly.
```luau
--!hidden mode=nocheck
-- For loop should iterate backwards; did you forget to specify -1 as step?
for i=#t,1 do
end
```
## UnbalancedAssignment (15)
Assignment statements and local variable declarations in Luau support multiple variables on the left side and multiple values on the right side. The number of values doesn't need to match; when the right side has more values, the extra values are discarded, and then the left side has more variables the extra variables are set to `nil`. However, this can result in subtle bugs where a value is omitted mistakenly. This warning warns about cases like this; if the last expression on the right hand side returns multiple values, the warning is not emitted.
```luau
-- Assigning 2 values to 3 variables initializes extra variables with nil; add 'nil' to value list to silence
local x, y, z = 1, 2
```
## ImplicitReturn (16)
In Luau, there's a subtle difference between returning no values from a function and returning `nil`. In many contexts these are equivalent, but when the results are passed to a variadic function (perhaps implicitly), the difference can be observed - for example, `print(foo())` prints nothing if `foo` returns no values, and `nil` if it returns `nil`.
To help write code that has consistent behavior, linter warns about cases when a function implicitly returns no values, if there are cases when it explicitly returns a result. For code like this it's recommended to use explicit `return` or `return nil` at the end of the function (these have different semantics, so the correct version to use depends on the function):
```luau
--!hidden mode=nocheck
local function find(t, expected)
for k,v in pairs(t) do
if k == expected then
return v
end
end
-- Function 'find' can implicitly return no values even though there's an explicit return at line 4; add explicit return to silence
end
```
## DuplicateLocal (17)
Luau syntax allows to use the same name for different parameters of a function as well as different local variables declared in the same statement. This is error prone, even if it's occasionally useful, so a warning is emitted in cases like this, unless the duplicate name is the placeholder `_`:
```luau
--!hidden mode=nocheck
function foo(a, b, a) -- Function parameter 'a' already defined on column 14
end
function obj:method(self) -- Function parameter 'self' already defined implicitly
end
local x, y, x = v:GetComponents() -- Variable 'x' already defined on column 7
```
## FormatString (18)
Luau has several library functions that expect a format string that specifies the behavior for the function. These format strings follow a specific syntax that depends on the question; mistakes in these strings can lead to runtime errors or unexpected behavior of the code.
To help make sure that the strings used for these functions are correct, linter checks calls to `string.format`, `string.pack`, `string.packsize`, `string.unpack`, `string.match`, `string.gmatch`, `string.find`, `string.gsub` and `os.date` and issues warnings when the call uses a literal string with an incorrect format:
```luau
--!hidden mode=nocheck
-- Invalid match pattern: invalid capture reference, must refer to a closed capture
local cap = string.match(s, "(%d)%2")
-- Invalid format string: unfinished format specifier
local str = ("%d %"):format(1, 2)
```
Note that with the exception of `string.format` this only works when the function is called via the library, not via the method call (so prefer `string.match(s, "pattern")` to `s:match("pattern")`).
## TableLiteral (19)
Table literals are often used in Luau to create objects or specify data. The syntax for table literals is rich but invites mistakes, for example it allows duplicate keys or redundant index specification for values already present in the list form. This warning is emitted for cases where some entries in the table literal are ignored at runtime as they're duplicate:
```luau
print({
first = 1,
second = 2,
first = 3, -- Table field 'first' is a duplicate; previously defined at line 2
})
```
## UninitializedLocal (20)
It's easy to forget to initialize a local variable and then proceed to use it regardless. This can happen due to a typo, or sometimes it can happen because the original variable definition is shadowed. When a local variable doesn't have an initial value and is used without ever being assigned to, this warning is emitted:
```luau
local foo
if foo then -- Variable 'foo' defined at line 1 is never initialized or assigned; initialize with 'nil' to silence
print(foo)
end
```
## DuplicateFunction (21)
This warning is emitted when a script defines two functions with the same name in the same scope.
The warning is not produced when the functions are defined in different scopes because this is much more likely to be intentional.
```luau
--!hidden mode=nocheck
function foo() end
function foo() end -- Duplicate function definition: 'foo' also defined on line 1
-- OK: the functions are not defined in the same scope.
if x then
function bar() end
else
function bar() end
end
```
## DeprecatedApi (22)
This warning is emitted when a script accesses a method or field that is marked as deprecated. Use of deprecated APIs is discouraged since they may have performance or correctness issues, may result in unexpected behavior, and could be removed in the future.
```luau
--!hidden mode=nocheck
function getSize(i: Instance)
return i.DataCost -- Member 'Instance.DataCost' is deprecated
end
```
## TableOperations (23)
To manipulate array-like tables, Luau provides a set of functions as part of the standard `table` library. To help use these functions correctly, this warning is emitted when one of these functions is used incorrectly or can be adjusted to be more performant.
```luau
local t = {}
table.insert(t, 0, 42) -- table.insert uses index 0 but arrays are 1-based; did you mean 1 instead?
table.insert(t, #t+1, 42) -- table.insert will append the value to the table; consider removing the second argument for efficiency
```
In addition, when type information is present, this warning will be emitted when `#` or `ipairs` is used on a table that has no numeric keys or indexers. This helps avoid common bugs like using `#t == 0` to check if a dictionary is empty.
```luau
local message = { data = { 1, 2, 3 } }
if #message == 0 then -- Using '#' on a table without an array part is likely a bug
end
```
## DuplicateCondition (24)
When checking multiple conditions via `and/or` or `if/elseif`, a copy & paste error may result in checking the same condition redundantly. This almost always indicates a bug, so a warning is emitted when use of a duplicate condition is detected.
```luau
--!hidden mode=nocheck
assert(self._adorns[normID1] and self._adorns[normID1]) -- Condition has already been checked on column 8
```
## MisleadingAndOr (25)
In Lua, there is no first-class ternary operator but it can be emulated via `a and b or c` pattern. However, due to how boolean evaluation works, if `b` is `false` or `nil`, the resulting expression evaluates to `c` regardless of the value of `a`. Luau solves this problem with the `if a then b else c` expression; a warning is emitted for and-or expressions where the first alternative is `false` or `nil` because it's almost always a bug.
```luau
--!hidden mode=nocheck
-- The and-or expression always evaluates to the second alternative because the first alternative is false; consider using if-then-else expression instead
local x = flag and false or true
```
The code above can be rewritten as follows to avoid the warning and the associated bug:
```luau
--!hidden mode=nocheck
local x = if flag then false else true
```
## CommentDirective (26)
Luau uses comments that start from `!` to control certain aspects of analysis, for example setting type checking mode via `--!strict` or disabling individual lints with `--!nolint`. Unknown directives are ignored, for example `--!nostrict` doesn't have any effect on the type checking process as the correct spelling is `--!nonstrict`. This warning flags comment directives that are ignored during processing:
```luau
--!nostrict
-- Unknown comment directive 'nostrict'; did you mean 'nonstrict'?"
```
## IntegerParsing (27)
Luau parses hexadecimal and binary literals as 64-bit integers before converting them to Luau numbers. As a result, numbers that exceed 2^64 are silently truncated to 2^64, which can result in unexpected program behavior. This warning flags literals that are truncated:
```luau
-- Hexadecimal number literal exceeded available precision and was truncated to 2^64
local x = 0x1111111111111111111111111111111111111
```
Luau numbers are represented as double precision IEEE754 floating point numbers; they can represent integers up to 2^53 exactly, but larger integers may lose precision. This warning also flags literals that are parsed with a precision loss:
```luau
-- Number literal exceeded available precision and was truncated to closest representable number
local x = 9007199254740993
```
## ComparisonPrecedence (28)
Because of operator precedence rules, not X == Y parses as (not X) == Y; however, often the intent was to invert the result of the comparison. This warning flags erroneous conditions like that, as well as flagging cases where two comparisons happen in a row without any parentheses:
```luau
--!hidden mode=nocheck
-- not X == Y is equivalent to (not X) == Y; consider using X ~= Y, or wrap one of the expressions in parentheses to silence
if not x == 5 then
end
-- X <= Y <= Z is equivalent to (X <= Y) <= Z; wrap one of the expressions in parentheses to silence
if 1 <= x <= 3 then
end
```

Some files were not shown because too many files have changed in this diff Show more