feat: add serializeArrays option

README.md

@@ -24,13 +24,13 @@ See an example of Charm's features in [this example repository](https://github.c
 
 ## 🍀 Features
 
-- ⚛️ **Manage state with _atoms_.** Decompose state into tiny, composable containers called _atoms_, as opposed to combining them into a single store.
+-   ⚛️ **Manage state with _atoms_.** Decompose state into tiny, composable containers called _atoms_, as opposed to combining them into a single store.
 
-- 💪 **Minimal, yet powerful.** Less boilerplate — write simple functions to read from and write to state.
+-   💪 **Minimal, yet powerful.** Less boilerplate — write simple functions to read from and write to state.
 
-- 🔬 **Immediate updates.** Listeners run asynchronously by default, avoiding the cascading effects of deferred updates and improving responsiveness.
+-   🔬 **Immediate updates.** Listeners run asynchronously by default, avoiding the cascading effects of deferred updates and improving responsiveness.
 
-- 🦄 **Like magic.** Selector functions can be subscribed to as-is — with implicit dependency tracking, atoms are captured and memoized for you.
+-   🦄 **Like magic.** Selector functions can be subscribed to as-is — with implicit dependency tracking, atoms are captured and memoized for you.
 
 ---
 
@@ -59,12 +59,12 @@ Charm provides a debug mode to help you identify potential bugs in your project.
 
 Enabling `__DEV__` adds a few helpful features:
 
-- Better error handling for selectors, subscriptions, and batched functions:
+-   Better error handling for selectors, subscriptions, and batched functions:
 
-  - Errors provide the function's name and line number.
-  - Yielding in certain functions will throw an error.
+    -   Errors provide the function's name and line number.
+    -   Yielding in certain functions will throw an error.
 
-- Server state is validated for [remote event limitations](https://create.roblox.com/docs/scripting/events/remote#argument-limitations) before being passed to the client.
+-   Server state is validated for [remote event limitations](https://create.roblox.com/docs/scripting/events/remote#argument-limitations) before being passed to the client.
 
 Enabling debug mode in unit tests, storybooks, and other development environments can help you catch potential issues early. However, remember to turn off debug mode in production to avoid the performance overhead.
 
@@ -85,11 +85,11 @@ local todosAtom: Atom<{ string }> = atom({})
 
 #### Parameters
 
-- `state`: The value to assign to the atom initially.
+-   `state`: The value to assign to the atom initially.
 
-- **optional** `options`: An object that configures the behavior of this atom.
+-   **optional** `options`: An object that configures the behavior of this atom.
 
-  - **optional** `equals`: An equality function to determine whether the state has changed. By default, strict equality (`==`) is used.
+    -   **optional** `equals`: An equality function to determine whether the state has changed. By default, strict equality (`==`) is used.
 
 #### Returns
 
@@ -144,9 +144,9 @@ nameAtom("Jane") --> "JANE"
 
 #### Parameters
 
-- `callback`: The function to subscribe to. This may be an atom or a selector function that depends on an atom.
+-   `callback`: The function to subscribe to. This may be an atom or a selector function that depends on an atom.
 
-- `listener`: The listener is called when the result of the callback changes. It receives the new state and the previous state as arguments.
+-   `listener`: The listener is called when the result of the callback changes. It receives the new state and the previous state as arguments.
 
 #### Returns
 
@@ -173,7 +173,7 @@ Because `effect` implicitly tracks all atoms read within the callback, it might
 
 #### Parameters
 
-- `callback`: The function to track for state changes. The callback will run once to retrieve its dependencies, and then again whenever they change.
+-   `callback`: The function to track for state changes. The callback will run once to retrieve its dependencies, and then again whenever they change.
 
 #### Returns
 
@@ -202,9 +202,9 @@ This function is also useful for optimizing `effect` calls that depend on multip
 
 #### Parameters
 
-- `callback`: A function that returns a new value depending on one or more atoms.
+-   `callback`: A function that returns a new value depending on one or more atoms.
 
-- **optional** [`options`](#parameters): An object that configures the behavior of this atom.
+-   **optional** [`options`](#parameters): An object that configures the behavior of this atom.
 
 #### Returns
 
@@ -232,9 +232,9 @@ end)
 
 #### Parameters
 
-- `callback`: An atom or selector function that returns a dictionary or an array of values. When a key is added to the state, the factory will be called with the new key and its initial value.
+-   `callback`: An atom or selector function that returns a dictionary or an array of values. When a key is added to the state, the factory will be called with the new key and its initial value.
 
-- `factory`: A function that will be called whenever a key is added or removed from the atom's state. The callback will receive the key and the entry's initial value as arguments, and may return a cleanup function.
+-   `factory`: A function that will be called whenever a key is added or removed from the atom's state. The callback will receive the key and the entry's initial value as arguments, and may return a cleanup function.
 
 #### Returns
 
@@ -255,13 +255,13 @@ end)
 
 #### Parameters
 
-- `callback`: The function whose result you want to map over. This can be an atom or a selector function that reads from atoms.
+-   `callback`: The function whose result you want to map over. This can be an atom or a selector function that reads from atoms.
 
-- `mapper`: The mapper is called for each key in your state. Given the current value and key, it should return a new corresponding value and key:
+-   `mapper`: The mapper is called for each key in your state. Given the current value and key, it should return a new corresponding value and key:
 
-  1. Return a single value to map the table's original key to a new value.
-  2. Return two values, the first being the value and the second being the key, to update both keys and values.
-  3. Return `nil` for the value to remove the key from the resulting table.
+    1. Return a single value to map the table's original key to a new value.
+    2. Return two values, the first being the value and the second being the key, to update both keys and values.
+    3. Return `nil` for the value to remove the key from the resulting table.
 
 #### Returns
 
@@ -285,9 +285,9 @@ end)
 
 #### Parameters
 
-- `value`: Any value. If the value is a function, `peek` will call it without tracking dependencies and return the result.
+-   `value`: Any value. If the value is a function, `peek` will call it without tracking dependencies and return the result.
 
-- **optional** `...args`: Additional arguments to pass to the value if it is a function.
+-   **optional** `...args`: Additional arguments to pass to the value if it is a function.
 
 #### Returns
 
@@ -311,7 +311,7 @@ end)
 
 #### Parameters
 
-- `callback`: A function that updates atoms. Listeners will only be notified once all changes have been applied.
+-   `callback`: A function that updates atoms. Listeners will only be notified once all changes have been applied.
 
 #### Returns
 
@@ -361,9 +361,9 @@ end, { props.filter })
 
 #### Parameters
 
-- `callback`: An atom or selector function that depends on an atom.
+-   `callback`: An atom or selector function that depends on an atom.
 
-- **optional** `dependencies`: An array of outside values that the selector depends on. If the dependencies change, the subscription is re-created and the component re-renders with the new state.
+-   **optional** `dependencies`: An array of outside values that the selector depends on. If the dependencies change, the subscription is re-created and the component re-renders with the new state.
 
 #### Returns
 
@@ -405,7 +405,7 @@ end
 
 #### Parameters
 
-- `callback`: An atom or selector function that depends on an atom.
+-   `callback`: An atom or selector function that depends on an atom.
 
 #### Returns
 
@@ -441,7 +441,7 @@ local syncer = CharmSync.server({
 	atoms = atomsToSync,     -- A dictionary of the atoms to sync, matching the client's
 	interval = 0,            -- The minimum interval between state updates
 	preserveHistory = false, -- Whether to send a full history of changes made to the atoms (slower)
-	convertArrays = true,    -- Safety measures for arrays, should be false when using ByteNet or Zap
+	serializeArrays = true,  -- Safety measures for arrays, should be false when using ByteNet or Zap
 })
 
 -- Sends state updates to clients when a synced atom changes.
@@ -459,31 +459,31 @@ end)
 
 #### Parameters
 
-- `options`: An object to configure sync behavior.
+-   `options`: An object to configure sync behavior.
 
-  - `atoms`: A dictionary of the atoms to sync. The keys should match the keys on the client.
+    -   `atoms`: A dictionary of the atoms to sync. The keys should match the keys on the client.
 
-  - **optional** `interval`: The interval at which to batch state updates to clients. Defaults to `0`, meaning updates are batched every frame.
+    -   **optional** `interval`: The interval at which to batch state updates to clients. Defaults to `0`, meaning updates are batched every frame.
 
-  - **optional** `preserveHistory`: Whether to sync an exhaustive history of changes made to the atoms since the last sync event. If `true`, the server sends multiple payloads instead of one. Defaults to `false` for performance.
+    -   **optional** `preserveHistory`: Whether to sync an exhaustive history of changes made to the atoms since the last sync event. If `true`, the server sends multiple payloads instead of one. Defaults to `false` for performance.
 
-  - **optional** `convertArrays`: Whether to convert sparse arrays to dictionaries before firing remote events. Because arrays in state patches can have holes, extra work is required to send them over a remote event intact. Defaults to `true`, but should be `false` if payloads are serialized (i.e. if you use [ByteNet](https://github.com/ffrostfall/ByteNet) or [Zap](https://github.com/red-blox/zap)).
+    -   **optional** `serializeArrays`: Whether to convert problematic arrays to dictionaries before firing remote events. Because arrays in state patches can have holes, extra work is required to send them over a remote event intact. Defaults to `true`, but should be `false` if payloads are serialized (i.e. if you use [ByteNet](https://github.com/ffrostfall/ByteNet) or [Zap](https://github.com/red-blox/zap)).
 
 #### Returns
 
 `server` returns an object with the following methods:
 
-- `syncer:connect(callback)`: Registers a callback to send state updates to clients. The callback will receive the player and the payload(s) to send, and should fire a remote event. The payload is read-only, so any changes should be applied to a copy of the payload.
+-   `syncer:connect(callback)`: Registers a callback to send state updates to clients. The callback will receive the player and the payload(s) to send, and should fire a remote event. The payload is read-only, so any changes should be applied to a copy of the payload.
 
-- `syncer:hydrate(player)`: Sends the player a full state update for all synced atoms.
+-   `syncer:hydrate(player)`: Sends the player a full state update for all synced atoms.
 
 #### Caveats
 
-- **Do not use values that cannot be sent over remotes** in your shared atoms. This includes functions, threads, and non-string keys in dictionaries.
+-   **Do not use values that cannot be sent over remotes** in your shared atoms. This includes functions, threads, and non-string keys in dictionaries.
 
-- **By default, Charm omits the individual changes made to atoms** between sync events (i.e. a `counterAtom` set to `1` and then `2` will only send the final state of `2`). If you need to preserve a history of changes, set `preserveHistory` to `true`.
+-   **By default, Charm omits the individual changes made to atoms** between sync events (i.e. a `counterAtom` set to `1` and then `2` will only send the final state of `2`). If you need to preserve a history of changes, set `preserveHistory` to `true`.
 
-- **Charm does not handle network communication.** You must use remote events to send sync payloads and set `convertArrays` accordingly. This is implemented via the `remotes` namespace in the example above.
+-   **Charm does not handle network communication.** Use remote events or a network library to send sync payloads - and remember to set `serializeArrays` accordingly!
 
 ---
 
@@ -510,21 +510,21 @@ remotes.requestState:fire()
 
 #### Parameters
 
-- `options`: An object to configure sync behavior.
+-   `options`: An object to configure sync behavior.
 
-  - `atoms`: A dictionary of the atoms to sync. The keys should match the keys on the server.
+    -   `atoms`: A dictionary of the atoms to sync. The keys should match the keys on the server.
 
-  - **optional** `ignoreUnhydrated`: Whether to ignore state updates before setting the initial state. Defaults to `true`.
+    -   **optional** `ignoreUnhydrated`: Whether to ignore state updates before setting the initial state. Defaults to `true`.
 
 #### Returns
 
 `client` returns an object with the following methods:
 
-- `syncer:sync(...payloads)` applies a state update from the server.
+-   `syncer:sync(...payloads)` applies a state update from the server.
 
 #### Caveats
 
-- The client sync object does not handle network communication. You must implement your own network layer to send and receive state updates. This includes requesting the initial state, which is implemented via `requestState` in the example above.
+-   **Charm does not handle network communication.** Use remote events or a network library to receive sync payloads. This includes requesting the initial state, which is implemented via `requestState` in the example above.
 
 ---
 
@@ -551,7 +551,7 @@ end)
 
 #### Parameters
 
-- `value`: Any value. If the value is `None`, `isNone` will return `true`.
+-   `value`: Any value. If the value is `None`, `isNone` will return `true`.
 
 #### Returns
 

src/charm-sync/.luaurc

@@ -1,3 +0,0 @@
-{
-	"languageMode": "strict"
-}

src/charm-sync/src/client.luau

@@ -1,6 +1,5 @@
 local Charm = require(script.Parent.Parent.Charm)
-local packets = require(script.Parent.packets)
-type Packet = packets.Packet
+local patch = require(script.Parent.patch)
 local types = require(script.Parent.types)
 type SyncPayload = types.SyncPayload
 
@@ -49,14 +48,14 @@ local function client(options: ClientOptions): ClientSyncer
 		end
 	end
 
-	local function patch(data: { Packet })
+	local function apply(data: { [string | number]: any })
 		local target = {}
 
 		for key, atom in next, atoms do
 			target[key] = atom()
 		end
 
-		target = packets.apply(target, data)
+		target = patch.apply(target, data)
 
 		for key, atom in next, atoms do
 			atom(target[key])
@@ -71,7 +70,7 @@ local function client(options: ClientOptions): ClientSyncer
 				if payload.type == "init" then
 					hydrate(payload.data)
 				elseif not ignoreUnhydrated or hydrated then
-					patch(payload.data)
+					apply(payload.data)
 				end
 			end)
 		end

src/charm-sync/src/index.d.ts

@@ -22,6 +22,15 @@ declare namespace CharmSync {
 		readonly [P in keyof T]: T[P] extends Selector<infer State> ? State : never;
 	};
 
+	/**
+	 * Represents the removal of a value from the state.
+	 */
+	interface None {
+		readonly __none: "__none";
+	}
+
+	type MaybeNone<T> = undefined extends T ? None : never;
+
 	/**
 	 * Creates a `ClientSyncer` object that receives patches from the server and
 	 * applies them to the local state.
@@ -43,19 +52,43 @@ declare namespace CharmSync {
 	function server<Selectors extends SelectorMap>(options: ServerOptions<Selectors>): ServerSyncer<Selectors>;
 
 	/**
-	 * A type that should not be made partial in patches.
+	 * Checks whether a value is `None`. If `true`, the value is scheduled to be
+	 * removed from the state when the patch is applied.
+	 *
+	 * @param value The value to check.
+	 * @returns `true` if the value is `None`, otherwise `false`.
 	 */
-	type DataType = CheckableTypes[keyof Omit<CheckableTypes, "table" | "userdata" | "nil">];
+	function isNone(value: unknown): value is None;
 
-	/**
-	 * A packet representing a modified value in the state.
-	 */
-	type SyncPacket<Selectors extends SelectorMap> = {
-		readonly path: SyncPacketPath<StateOfMap<Selectors>>;
-		readonly value: unknown;
+	type DataTypes = {
+		[P in keyof CheckableTypes as P extends keyof CheckablePrimitives ? never : P]: CheckableTypes[P];
 	};
 
-	type SyncPacketPath<T> = T extends DataType ? readonly [] : readonly [keyof T, ...SyncPacketPath<T[keyof T]>];
+	/**
+	 * A type that should not be made partial in patches.
+	 */
+	type DataType = DataTypes[keyof DataTypes];
+
+	/**
+	 * A partial patch that can be applied to the state to update it. Represents
+	 * the difference between the current state and the next state.
+	 *
+	 * If a value was removed, it is replaced with `None`. This can be checked
+	 * using the `sync.isNone` function.
+	 */
+	type SyncPatch<State> =
+		| MaybeNone<State>
+		| (State extends ReadonlyMap<infer K, infer V> | Map<infer K, infer V>
+				? ReadonlyMap<K, SyncPatch<V> | None>
+				: State extends Set<infer T> | ReadonlySet<infer T>
+					? ReadonlyMap<T, true | None>
+					: State extends readonly (infer T)[]
+						? readonly (SyncPatch<T> | None | undefined)[]
+						: State extends DataType
+							? State
+							: State extends object
+								? { readonly [P in keyof State]?: SyncPatch<State[P]> }
+								: State);
 
 	/**
 	 * A payload that can be sent from the server to the client to synchronize
@@ -63,7 +96,7 @@ declare namespace CharmSync {
 	 */
 	type SyncPayload<Selectors extends SelectorMap> =
 		| { readonly type: "init"; readonly data: StateOfMap<Selectors> }
-		| { readonly type: "patch"; readonly data: readonly SyncPacket<Selectors>[] };
+		| { readonly type: "patch"; readonly data: SyncPatch<StateOfMap<Selectors>> };
 
 	interface ClientOptions<Atoms extends AtomMap> {
 		/**
@@ -98,6 +131,15 @@ declare namespace CharmSync {
 		 * to reconstruct the state's changes over time.
 		 */
 		preserveHistory?: boolean;
+		/**
+		 * When `true`, convert problematic sparse arrays into dictionaries
+		 * with string keys. Defaults to `true`.
+		 *
+		 * While this is mandatory for safely syncing arrays over vanilla Roblox
+		 * remotes, it can be disabled if your network library uses a custom
+		 * serialization method (i.e. Zap, ByteNet).
+		 */
+		serializeArrays?: boolean;
 	}
 
 	interface ClientSyncer<Atoms extends AtomMap> {

src/charm-sync/src/init.luau

@@ -1,10 +1,9 @@
 local client = require(script.client)
-local packets = require(script.packets)
+local patch = require(script.patch)
 local server = require(script.server)
 local types = require(script.types)
 
 export type SyncPayload = types.SyncPayload
-export type Packet = packets.Packet
 
 --[=[
 	Synchronizes state between the client and server. The server sends patches
@@ -13,4 +12,5 @@ export type Packet = packets.Packet
 return {
 	client = client,
 	server = server,
+	isNone = patch.isNone,
 }

src/charm-sync/src/packets.luau

@@ -1,100 +0,0 @@
-export type Packet = {
-	path: { string | number },
-	data: any,
-}
-
-local packets: { Packet }
-local counter: number
-
-local function push(path: { string | number }, key: string | number)
-	local result = table.clone(path)
-	table.insert(result, key)
-	return result
-end
-
-local function append(path: { string | number }, data: unknown)
-	packets[counter] = { path = path, data = data }
-	counter += 1
-end
-
-local function visit(
-	previousState: { [string | number]: any },
-	currentState: { [string | number]: any },
-	path: { string | number }
-)
-	local empty = next(currentState) == nil
-
-	if empty then
-		-- Overwrite with an empty table instead of deleting keys
-		return append(path, currentState)
-	end
-
-	-- Search keys for removals and changes, but not additions
-	for key, previous in next, previousState do
-		local current = currentState[key]
-
-		if current == previous then
-			continue
-		elseif current == nil then
-			append(push(path, key), nil)
-		elseif type(current) == "table" and type(previous) == "table" then
-			visit(previous, current, push(path, key))
-		else
-			append(push(path, key), current)
-		end
-	end
-
-	-- Search for additions not present in the previous state
-	for key, current in next, currentState do
-		if previousState[key] == nil then
-			append(push(path, key), current)
-		end
-	end
-end
-
-local function diff(previous: { [string | number]: any }, current: { [string | number]: any })
-	packets = {}
-	counter = 1
-	visit(previous, current, {})
-	return packets
-end
-
-local function apply(state: { [string | number]: any }, packets: { Packet })
-	local result = table.clone(state)
-	local copied = {}
-
-	local function resolve(path: { string | number })
-		local object = result
-		local length = #path
-		local id = ""
-
-		for index = 1, length - 1 do
-			local key = path[index]
-
-			id ..= "\\" .. tostring(key)
-
-			-- Prepare a deep copy of the object if it hasn't been accessed yet
-			if not copied[id] then
-				copied[id] = true
-				object[key] = table.clone(object[key])
-			end
-
-			object = object[key]
-		end
-
-		return object, path[length]
-	end
-
-	for _, packet in next, packets do
-		local object, key = resolve(packet.path)
-
-		object[key] = packet.data
-	end
-
-	return result
-end
-
-return {
-	diff = diff,
-	apply = apply,
-}

src/charm-sync/src/patch.luau

@@ -0,0 +1,121 @@
+local validate = require(script.Parent.validate)
+
+local __DEV__ = _G.__DEV__
+
+--[=[
+	A special symbol that denotes the absence of a value. Used to represent
+	deleted values in patches.
+]=]
+local None = { __none = "__none" }
+
+local function isNone(value: any): boolean
+	return type(value) == "table" and value.__none == "__none"
+end
+
+--[=[
+	JSON serialization can drop the last values of a sparse array. For example,
+	`{ [1] = 1, [3] = 3 }` becomes `[1]` in some cases. Ambiguity in the
+	serialization of sparse arrays can be resolved by converting them to
+	dictionaries with string keys.
+
+	The `apply` function will convert these dictionaries back into arrays as
+	long as the target array is not sparse. This function ensures that the diff
+	function produces a patch that can be correctly applied.
+]=]
+local function stringifySparseArray(object: { any }): { [string | number]: any }
+	local maxn = table.maxn(object)
+
+	if maxn == 0 or maxn == #object then
+		-- The object is either not an array or a dense array
+		return object
+	end
+
+	local dictionary = {}
+
+	for index, value in next, object do
+		dictionary[tostring(index)] = value
+	end
+
+	return dictionary
+end
+
+--[=[
+	Compares two states and returns a patch that can be applied to the previous
+	state to produce the next state.
+
+	@param prevState The previous state.
+	@param nextState The next state.
+	@param serializeArrays Whether to convert sparse arrays into dictionaries. Defaults to `true`.
+	@returns A patch that can be applied to the previous state.
+]=]
+local function diff(prevState: { [any]: any }, nextState: { [any]: any }, serializeArrays: boolean?)
+	local patches = table.clone(nextState)
+
+	for key, previous in next, prevState do
+		local current = nextState[key]
+
+		if previous == current then
+			patches[key] = nil
+		elseif current == nil then
+			patches[key] = None
+		elseif type(previous) == "table" and type(current) == "table" then
+			patches[key] = diff(previous, current, serializeArrays)
+		end
+	end
+
+	-- Coerce sparse array patches into dictionaries
+	if serializeArrays ~= false and (prevState[1] ~= nil or nextState[1] ~= nil) then
+		patches = stringifySparseArray(patches)
+	end
+
+	if __DEV__ then
+		for key, value in next, prevState do
+			validate(value, key)
+		end
+
+		for key, value in nextState do
+			if prevState[key] ~= value then
+				validate(value, key)
+			end
+		end
+	end
+
+	return patches
+end
+
+--[=[
+	Applies a patch to a state and returns the next state.
+
+	@param state The current state.
+	@param patches The patches to apply.
+	@returns The next state.
+]=]
+local function apply(state: any, patches: any): any
+	if type(patches) == "table" and patches.__none == "__none" then
+		return nil
+	elseif type(state) ~= "table" or type(patches) ~= "table" then
+		return patches
+	end
+
+	local nextState = table.clone(state)
+	local stateIsArray = state[1] ~= nil
+
+	for key, patch in next, patches do
+		-- Diff-checking an array produces a sparse array, which will not be
+		-- preserved when converted to JSON. To prevent this, we turn string
+		-- keys back into numeric keys.
+		if stateIsArray and type(key) == "string" then
+			key = tonumber(key) or key
+		end
+
+		nextState[key] = apply(nextState[key], patch)
+	end
+
+	return nextState
+end
+
+return {
+	isNone = isNone,
+	diff = diff,
+	apply = apply,
+}

src/charm-sync/src/server.luau

@@ -2,8 +2,7 @@ local Players = game:GetService("Players")
 
 local Charm = require(script.Parent.Parent.Charm)
 local interval = require(script.Parent.interval)
-local packets = require(script.Parent.packets)
-type Packet = packets.Packet
+local patch = require(script.Parent.patch)
 local types = require(script.Parent.types)
 type SyncPayload = types.SyncPayload
 
@@ -24,6 +23,15 @@ type ServerOptions = {
 		per frame, where each individual change is important. Defaults to `false`.
 	]=]
 	preserveHistory: boolean?,
+	--[=[
+		Whether to convert problematic sparse arrays into dictionaries with
+		string keys. Defaults to `true`.
+
+		While this is mandatory for safely syncing arrays over vanilla Roblox
+		remotes, it can be disabled if your network library uses a custom
+		serialization method (i.e. Zap, ByteNet).
+	]=]
+	serializeArrays: boolean?,
 }
 
 type ServerSyncer = {
@@ -136,7 +144,7 @@ local function server(options: ServerOptions): ServerSyncer
 
 					table.insert(payloads, {
 						type = "patch",
-						data = packets.diff(snapshots[index - 1], snapshot),
+						data = patch.diff(snapshots[index - 1], snapshot, options.serializeArrays),
 					})
 				end
 
@@ -173,7 +181,7 @@ local function server(options: ServerOptions): ServerSyncer
 
 				table.insert(payloads, {
 					type = "patch",
-					data = packets.diff(snapshots[index - 1], snapshot),
+					data = patch.diff(snapshots[index - 1], snapshot, options.serializeArrays),
 				})
 			end
 
@@ -210,7 +218,7 @@ local function server(options: ServerOptions): ServerSyncer
 
 			local payload: SyncPayload = {
 				type = "patch",
-				data = packets.diff(previousSnapshot, currentSnapshot),
+				data = patch.diff(previousSnapshot, currentSnapshot, options.serializeArrays),
 			}
 
 			previousSnapshot = table.clone(currentSnapshot)
@@ -239,7 +247,7 @@ local function server(options: ServerOptions): ServerSyncer
 
 		local payload: SyncPayload = {
 			type = "patch",
-			data = packets.diff(previousSnapshot, currentSnapshot),
+			data = patch.diff(previousSnapshot, currentSnapshot, options.serializeArrays),
 		}
 
 		previousSnapshot = table.clone(currentSnapshot)

src/charm-sync/src/types.luau

@@ -1,6 +1,3 @@
-local packets = require(script.Parent.packets)
-type Packet = packets.Packet
-
 --[=[
 	A payload that can be sent from the server to the client to synchronize
 	state between the two.
@@ -10,7 +7,7 @@ export type SyncPayload = {
 	data: { [string | number]: any },
 } | {
 	type: "patch",
-	data: { Packet },
+	data: { [string | number]: any },
 }
 
 return nil

src/charm-sync/src/validate.luau

@@ -1,7 +1,5 @@
-local Charm = require(script.Parent.Parent.Charm)
-
 local SAFE_KEYS = { string = true, number = true }
-local SAFE_VALUES = { string = true, number = true, boolean = true, table = true, userdata = true }
+local SAFE_VALUES = { string = true, number = true, boolean = true, table = true, userdata = true, vector = true }
 
 local function isTableUnsafe(object: { [any]: any })
 	local keyType = nil
@@ -47,8 +45,6 @@ local function validate(value: any, key: any)
 	elseif typeOfValue == "table" then
 		if getmetatable(value) ~= nil then
 			error(`Cannot sync tables with metatables! Got {value} at key '{key}'`)
-		elseif Charm.isAtom(value) then
-			error(`Cannot sync nested atoms! Got atom at key '{key}'`)
 		elseif isTableUnsafe(value) then
 			error(
 				`Cannot sync tables unsupported by remote events! The value has the key '{key}'.\n\n`
@@ -69,7 +65,7 @@ local function validate(value: any, key: any)
 	elseif typeOfKey == "number" then
 		if key == math.huge or key == -math.huge then
 			error("Cannot sync infinity as key")
-		elseif math.floor(key) ~= key then
+		elseif key ~= math.floor(key) then
 			error("Cannot sync non-integer number as key")
 		end
 	end

src/charm/.luaurc

@@ -1,3 +0,0 @@
-{
-	"languageMode": "strict"
-}

src/charm/src/store.luau

@@ -3,6 +3,8 @@ type Atom<T> = types.Atom<T>
 type Set<T> = types.Set<T>
 type WeakMap<K, V> = types.WeakMap<K, V>
 
+local __DEV__ = _G.__DEV__
+
 local listeners: WeakMap<Atom<any>, WeakMap<() -> (), unknown>> = setmetatable({}, { __mode = "k" })
 
 local batched: Set<() -> ()> = {}
@@ -24,7 +26,7 @@ local capturing = {
 	@return The result of the callback.
 ]=]
 local function try<T, U...>(callback: (U...) -> T | any, finally: (() -> ())?, ...: U...): T
-	if _G.__DEV__ then
+	if __DEV__ then
 		local thread = coroutine.create(callback)
 		local success, result = coroutine.resume(thread, ...)
 

src/react/.luaurc

@@ -1,3 +0,0 @@
-{
-	"languageMode": "strict"
-}

src/vide/.luaurc

@@ -1,3 +0,0 @@
-{
-	"languageMode": "strict"
-}

src/vide/src/init.luau

@@ -9,10 +9,12 @@ local Vide = if script.Parent:FindFirstChild("vide")
 	@param callback The atom or selector to subscribe to.
 	@return The reactive source.
 ]=]
-local function useAtom<State>(callback: () -> State): State
+local function useAtom<State>(callback: () -> State): () -> State
 	local state = Vide.source(callback())
 	local unsubscribe = Charm.subscribe(callback, function(value)
-		task.spawn(state, value)
+		task.spawn(function()
+			state(value)
+		end)
 	end)
 
 	Vide.cleanup(unsubscribe)

tests/charm-sync/client.spec.luau

@@ -19,30 +19,6 @@ return function()
 		expect(a()).to.equal(2)
 	end)
 
-	it("requires init by default", function()
-		local a = atom(1)
-		local client = client({ atoms = { a = a } })
-
-		client:sync({
-			type = "patch",
-			data = { { path = { "a" }, data = 2 } },
-		})
-
-		expect(a()).to.equal(1)
-	end)
-
-	it("can ignore init requirement", function()
-		local a = atom(1)
-		local client = client({ atoms = { a = a }, ignoreUnhydrated = false })
-
-		client:sync({
-			type = "patch",
-			data = { { path = { "a" }, data = 2 } },
-		})
-
-		expect(a()).to.equal(2)
-	end)
-
 	it("syncs state patch", function()
 		local a = atom({ b = 1, c = 2 })
 		local previous = a()
@@ -57,7 +33,7 @@ return function()
 
 		client:sync({
 			type = "patch",
-			data = { { path = { "a", "b" }, data = 3 } },
+			data = { a = { b = 3 } },
 		})
 
 		expect(a()).to.never.equal(previous)
@@ -65,6 +41,27 @@ return function()
 		expect(a().c).to.equal(2)
 	end)
 
+	it("fixes sparse array patches", function()
+		local a = atom({ 1 })
+		local client = client({ atoms = { a = a } })
+
+		client:sync({
+			type = "init",
+			data = { a = a() },
+		})
+
+		-- This patch would be the result of a diff between { 1 } and { 1, 2 } that
+		-- was JSON-serialized and deserialized. The client should be able to catch
+		-- this and convert the string key back to a number.
+		client:sync({
+			type = "patch",
+			data = { a = { ["2"] = 2 } },
+		})
+
+		expect(a()["2" :: any]).to.never.be.ok()
+		expect(a()[2]).to.equal(2)
+	end)
+
 	it("applies multiple patches", function()
 		local a = atom(1)
 		local client = client({ atoms = { a = a } })
@@ -81,10 +78,10 @@ return function()
 
 		client:sync({
 			type = "patch",
-			data = { { path = { "a" }, data = 2 } },
+			data = { a = 2 },
 		}, {
 			type = "patch",
-			data = { { path = { "a" }, data = 3 } },
+			data = { a = 3 },
 		})
 
 		expect(a()).to.equal(3)
@@ -113,10 +110,7 @@ return function()
 
 		client:sync({
 			type = "patch",
-			data = {
-				{ path = { "a" }, data = 1 },
-				{ path = { "b" }, data = 2 },
-			},
+			data = { a = 1, b = 2 },
 		})
 
 		expect(a()).to.equal(1)

tests/charm-sync/packets.spec.luau

@@ -1,68 +0,0 @@
-local ReplicatedStorage = game:GetService("ReplicatedStorage")
-
-local packets = require(ReplicatedStorage.DevPackages.CharmSync.packets)
-
-return function()
-	it("creates a packet", function()
-		local prevState = { foo = "bar", baz = "qux" }
-		local nextState = { foo = "bar", baz = "quux" }
-		local results = packets.diff(prevState, nextState)
-
-		expect(#results).to.equal(1)
-		expect(table.concat(results[1].path, "/")).to.equal("baz")
-		expect(results[1].data).to.equal("quux")
-	end)
-
-	it("applies a packet", function()
-		local prevState = { foo = "bar", baz = "qux" }
-		local packet: packets.Packet = { path = { "baz" }, data = "quux" }
-		local nextState = packets.apply(prevState, { packet })
-
-		expect(nextState.baz).to.equal("quux")
-		expect(nextState.foo).to.equal("bar")
-	end)
-
-	it("applies nested packets", function()
-		local prevState = { foo = { bar = "baz" } }
-		local nextState = { foo = { bar = "qux" } }
-
-		local results = packets.diff(prevState, nextState)
-		local applied = packets.apply(prevState, results)
-
-		expect(prevState.foo.bar).to.equal("baz")
-		expect(applied.foo.bar).to.equal("qux")
-	end)
-
-	it("only copies changed tables", function()
-		local prevState = { foo = { bar = "baz" }, baz = { qux = "quux" } }
-		local nextState = { foo = { bar = "qux" }, baz = table.clone(prevState.baz) }
-
-		local results = packets.diff(prevState, nextState)
-		local applied = packets.apply(prevState, results)
-
-		expect(applied).to.never.equal(prevState)
-		expect(applied.foo).to.never.equal(prevState.foo)
-		expect(applied.foo.bar).to.equal("qux")
-		expect(applied.baz).to.equal(prevState.baz)
-	end)
-
-	it("excludes unchanged tables", function()
-		local prevState = { foo = { bar = "baz" }, baz = { qux = "quux" } }
-		local nextState = { foo = { bar = "qux" }, baz = table.clone(prevState.baz) }
-		local results = packets.diff(prevState, nextState)
-
-		expect(#results).to.equal(1)
-		expect(table.concat(results[1].path, "/")).to.equal("foo/bar")
-		expect(results[1].data).to.equal("qux")
-	end)
-
-	it("overwrites cleared tables", function()
-		local prevState = { foo = { 1, 2, 3 } }
-		local nextState = { foo = {} }
-		local results = packets.diff(prevState, nextState)
-
-		expect(#results).to.equal(1)
-		expect(table.concat(results[1].path, "/")).to.equal("foo")
-		expect(#results[1].data).to.equal(0)
-	end)
-end

tests/charm-sync/patch.spec.luau

@@ -0,0 +1,111 @@
+local HttpService = game:GetService("HttpService")
+local ReplicatedStorage = game:GetService("ReplicatedStorage")
+
+local patch = require(ReplicatedStorage.DevPackages.CharmSync.patch)
+
+return function()
+	local function serdes(value: { [any]: any })
+		return HttpService:JSONDecode(HttpService:JSONEncode(value))
+	end
+
+	it("returns a patch", function()
+		local prevState = { foo = "bar", baz = "qux" }
+		local nextState = { foo = "bar", baz = "quux" }
+		local patches = patch.diff(prevState, nextState)
+
+		expect(patches).to.be.ok()
+		expect(patches.baz).to.equal("quux")
+		expect(patches.foo).to.never.be.ok()
+	end)
+
+	it("applies a patch", function()
+		local prevState = { foo = "bar", baz = "qux" }
+		local patches = { baz = "quux", qaz = "qux" }
+		local applied = patch.apply(prevState, patches)
+
+		expect(applied).to.be.ok()
+		expect(applied.baz).to.equal("quux")
+		expect(applied.foo).to.equal("bar")
+		expect(applied.qaz).to.equal("qux")
+	end)
+
+	it("allows nested patches", function()
+		local prevState = { foo = { bar = "baz" } }
+		local nextState = { foo = { bar = "qux" } }
+		local patches = patch.diff(prevState, nextState)
+		local applied = patch.apply(prevState, patches)
+
+		expect(applied.foo.bar).to.equal("qux")
+	end)
+
+	it("preserves array density when applying", function()
+		local prevState = { "a", "b", "c", "d" }
+		local nextState = { ["2"] = "B", ["4"] = "D" }
+		local applied = patch.apply(prevState, nextState)
+
+		expect(applied[1]).to.equal("a")
+		expect(applied[2]).to.equal("B")
+		expect(applied[3]).to.equal("c")
+		expect(applied[4]).to.equal("D")
+
+		expect(applied["2"]).to.never.be.ok()
+		expect(applied["4"]).to.never.be.ok()
+	end)
+
+	it("preserves array density after serdes", function()
+		-- This particular change results in a patch that cannot be serialized
+		-- without losing information. Charm resolves this issue internally.
+		local prevState = { "a", "b", "c", "d", "e" }
+		local nextState = { "A", "b", "c", "D" }
+
+		local patches = serdes(patch.diff(prevState, nextState))
+		local applied = patch.apply(prevState, patches)
+
+		for index in next, prevState do
+			expect(applied[index]).to.equal(nextState[index])
+		end
+
+		for index in next, applied do
+			expect(applied[index]).to.equal(nextState[index])
+		end
+	end)
+
+	it("preserves large gaps between keys", function()
+		local prevState = table.create(10000, "foo")
+		local nextState = table.move(prevState, 2, 9998, 2, { [1] = "bar", [9999] = "bar" })
+
+		local start = os.clock()
+
+		local patches = serdes(patch.diff(prevState, nextState))
+		local applied = patch.apply(prevState, patches)
+
+		assert(os.clock() - start < 10e-3, `Performance is not acceptable ({(os.clock() - start) * 1e3}ms)`)
+
+		for index in next, prevState do
+			expect(applied[index]).to.equal(nextState[index])
+		end
+	end)
+
+	it("cannot preserve sparse arrays", function()
+		local prevState = {}
+		local nextState = { [2] = "foo" }
+
+		local patches = serdes(patch.diff(prevState, nextState))
+		local applied = patch.apply(prevState, patches)
+
+		-- Not enough information is available to determine whether 'patches'
+		-- should keep the string keys added during remote event serialization.
+		expect(applied["2"]).to.be.ok()
+		expect(applied[2]).to.never.be.ok()
+	end)
+
+	it("optionally fixes sparse arrays", function()
+		for index in next, patch.diff({ 1, 1, 1, 1, 1 }, { 2, 1, 1, 2 }) do
+			expect(index).to.be.a("string")
+		end
+
+		for index in next, patch.diff({ 1, 1, 1, 1, 1 }, { 2, 1, 1, 2 }, false) do
+			expect(index).to.be.a("number")
+		end
+	end)
+end

tests/charm-sync/server.spec.luau

@@ -5,6 +5,7 @@ local atom = Charm.atom
 local CharmSync = require(ReplicatedStorage.DevPackages.CharmSync)
 type SyncPayload = CharmSync.SyncPayload
 local server = CharmSync.server
+local patch = require(ReplicatedStorage.DevPackages.CharmSync.patch)
 
 return function()
 	local target: Player = {} :: any
@@ -21,14 +22,14 @@ return function()
 		server:hydrate(target)
 
 		expect(player).to.equal(target)
-		assert(payload.type == "init", "payload.type is not 'init'")
+		expect(payload.type).to.equal("init")
 		expect(payload.data.a).to.equal(1)
 
 		disconnect()
 	end)
 
 	it("sends state patches", function()
-		local a = atom({ 1, 2 })
+		local a = atom({ b = 1, c = 2 } :: { [string]: number? })
 		local server = server({ atoms = { a = a }, interval = -1 })
 		local player: Player, payload: SyncPayload
 
@@ -36,27 +37,20 @@ return function()
 			player, payload = ...
 		end)
 
-		a({ 1, 5 })
+		a({ b = 1, c = 5 })
 		server:_sendPatch(target)
 
 		expect(player).to.equal(target)
-		assert(payload.type == "patch", "payload.type is not 'patch'")
-		expect(#payload.data).to.equal(1)
+		expect(payload.type).to.equal("patch")
+		expect(payload.data.a.b).to.never.be.ok()
+		expect(payload.data.a.c).to.equal(5)
 
-		expect(table.concat(payload.data[1].path, "/")).to.equal("a/2")
-		expect(payload.data[1].data).to.equal(5)
-
-		a({ 1, nil :: any, 1 })
+		a({ b = 1, d = 1 })
 		server:_sendPatch(target)
 
-		assert(payload.type == "patch", "payload.type is not 'patch'")
-		expect(#payload.data).to.equal(2)
-
-		expect(table.concat(payload.data[1].path, "/")).to.equal("a/2")
-		expect(payload.data[1].data).to.never.be.ok()
-
-		expect(table.concat(payload.data[2].path, "/")).to.equal("a/3")
-		expect(payload.data[2].data).to.equal(1)
+		expect(payload.data.a.b).to.never.be.ok()
+		expect(patch.isNone(payload.data.a.c)).to.equal(true)
+		expect(payload.data.a.d).to.equal(1)
 
 		disconnect()
 	end)
@@ -81,16 +75,9 @@ return function()
 		server:_sendPatch(target)
 
 		expect(#payloads).to.equal(1)
-		assert(payloads[1].type == "patch", "payload 1 type is not 'patch'")
-		expect(#payloads[1].data).to.equal(2)
-
-		-- a(2)
-		expect(table.concat(payloads[1].data[1].path, "/")).to.equal("a")
-		expect(payloads[1].data[1].data).to.equal(2)
-
-		-- b(2)
-		expect(table.concat(payloads[1].data[2].path, "/")).to.equal("b")
-		expect(payloads[1].data[2].data).to.equal(2)
+		expect(payloads[1].type).to.equal("patch")
+		expect(payloads[1].data.a).to.equal(2)
+		expect(payloads[1].data.b).to.equal(2)
 
 		a(3)
 		a(4)
@@ -98,24 +85,12 @@ return function()
 		server:_sendPatch(target)
 
 		expect(#payloads).to.equal(2)
-		assert(payloads[1].type == "patch", "payload 1 type is not 'patch'")
-		expect(#payloads[1].data).to.equal(1)
-
-		-- a(3)
-		expect(table.concat(payloads[1].data[1].path, "/")).to.equal("a")
-		expect(payloads[1].data[1].data).to.equal(3)
-
-		-- second a() call cannot be collapsed, so it is added to a new payload
-		assert(payloads[2].type == "patch", "payload 2 type is not 'patch'")
-		expect(#payloads[2].data).to.equal(2)
-
-		-- a(4)
-		expect(table.concat(payloads[2].data[1].path, "/")).to.equal("a")
-		expect(payloads[2].data[1].data).to.equal(4)
-
-		-- b(3)
-		expect(table.concat(payloads[2].data[2].path, "/")).to.equal("b")
-		expect(payloads[2].data[2].data).to.equal(3)
+		expect(payloads[1].type).to.equal("patch")
+		expect(payloads[1].data.a).to.equal(3)
+		expect(payloads[1].data.b).to.never.be.ok()
+		expect(payloads[2].type).to.equal("patch")
+		expect(payloads[2].data.a).to.equal(4)
+		expect(payloads[2].data.b).to.equal(3)
 
 		disconnect()
 	end)

tests/charm/init.spec.luau

@@ -1,19 +0,0 @@
-local ReplicatedStorage = game:GetService("ReplicatedStorage")
-
-local store = require(ReplicatedStorage.DevPackages.Charm.store)
-
-return function()
-	beforeAll(function()
-		_G.__DEV__ = true
-	end)
-
-	afterAll(function()
-		_G.__DEV__ = nil
-	end)
-
-	afterEach(function()
-		table.clear(store.listeners :: any)
-		table.clear(store.capturing.stack)
-		store.capturing.index = 0
-	end)
-end

tests/init.server.luau

@@ -1,5 +1,12 @@
+--!nocheck
+
 local ReplicatedStorage = game:GetService("ReplicatedStorage")
 
 local TestEZ = require(ReplicatedStorage.DevPackages.TestEZ)
 
+-- Start Charm in debug mode
+_G.__DEV__ = true
+require(ReplicatedStorage.DevPackages.Charm)
+_G.__DEV__ = false
+
 TestEZ.TestBootstrap:run({ script.Parent }, TestEZ.Reporters.TextReporter)

tests/init.spec.luau

@@ -3,14 +3,6 @@ local ReplicatedStorage = game:GetService("ReplicatedStorage")
 local store = require(ReplicatedStorage.DevPackages.Charm.store)
 
 return function()
-	beforeAll(function()
-		_G.__DEV__ = true
-	end)
-
-	afterAll(function()
-		_G.__DEV__ = nil
-	end)
-
 	afterEach(function()
 		table.clear(store.listeners :: any)
 		table.clear(store.capturing.stack)