Full KDBX Semantic Synchronization Plan

Status

Planned breaking pre-production change.

This document is an implementation specification for making every shared, decrypted KeePass value part of Lattice’s synchronized model. The project is still pre-production, so this plan deliberately permits incompatible record, protocol, revision, and persisted-state changes. It does not require migration or rolling compatibility with existing Lattice state.

Motivation

The physical two-machine relay-only E2E test proved that the current synchronized V2 record projection converges. An exhaustive follow-up comparison also proved that the complete decrypted KeePass XML does not converge.

The synchronized record manifests matched across both machines:

1 database record
4 group records
5 entry records
1 attachment resource
Identical record IDs and content hashes

The complete KeePass XML still differed in:

Entry history
Group EnableAutoType and EnableSearching tri-state values
Database metadata and metadata change timestamps
Database custom data
Entry and group access/location/expiry timestamps
Other values currently preserved from each destination KDBX rather than represented by synchronized records

The current test harness and audit utilities are:

xtask/two-machine-relay-e2e.sh
xtask/redacted-kdbx-xml-diff.py
src/bin/lattice-smoke.rs (manifest mode)

Target invariant

After all authorized replicas have synchronized and no local edits are pending:

Every shared, decrypted KeePass semantic value is identical on every replica.

Raw KDBX bytes are not expected to match. KDBX encryption uses file-local headers, nonces, keys, and serialization details.

The exhaustive E2E audit may ignore only an explicit, reviewed allowlist of file-local or derived values:

KDBX cipher, KDF, compression, and protected-stream configuration
Master-key material and key-file paths
Master-key timestamps that describe a machine-local credential
Random encryption headers and nonces
Writer Generator
Numeric attachment IDs
Attachment and custom-icon back-reference indexes
Other values proven to be derived from synchronized records

No user-visible entry, group, history, attachment, icon, metadata, ordering, expiration, access, or view state may be omitted merely because it is difficult to merge.

Non-goals

Backward compatibility with current pre-production Lattice records
Reading current encrypted sync-state files after the breaking release
Mixed old/new peer operation
Network protocol negotiation or fallback to protocol 9
Preserving current signed revision graphs
Automatic migration of current memberships or object bundles
Byte-for-byte KDBX equality

Breaking-upgrade policy

The implementation may directly edit the existing Rust record types. It must still bump every affected format version so stale binaries and state fail clearly instead of being misdecoded.

Format	Current	Required new value
Network sync protocol	9	11
Signed revision format	6	8
Encrypted sync state	7	9
Object bundle	2	4
Materialization journal	3	5

No legacy decoder is required. Old state must produce an actionable pre-production reset error.

Data-model design

Canonical hashes

Do not continue defining record hashes as postcard serialization of a serde-derived Record.

Implement an explicit, versioned canonical encoder for each record variant:

Use a domain separator per record kind and format generation.
Encode every option with an explicit presence tag.
Encode integer widths and endianness explicitly.
Encode maps and sets in canonical key order.
Encode history and child ordering deterministically.
Exclude content_hash itself.
Include state and version data only where the protocol specification says they are part of content identity.

The canonical encoder must have committed golden vectors. Future Rust struct layout changes must not silently alter hashes.

KeePass timestamps

Replace the lossy timestamp projection used by the new synchronized records. KeePass timestamps have second precision and can represent dates before the Unix epoch. Preserve absence separately from epoch and sentinel values.

#[derive(Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd)]
pub struct KeePassTime(i64); // signed seconds from Unix epoch

#[derive(Clone, Debug, Eq, PartialEq)]
pub struct KeePassTimes {
    pub creation: Option<KeePassTime>,
    pub last_modification: Option<KeePassTime>,
    pub last_access: Option<KeePassTime>,
    pub expiry: Option<KeePassTime>,
    pub location_changed: Option<KeePassTime>,
    pub expires: Option<bool>,
    pub usage_count: Option<u64>,
}

Required distinctions:

Missing timestamp versus Unix epoch
Missing Expires versus explicit false
Explicit false with an inactive expiry timestamp
Explicit true with an active expiry timestamp
KeePass sentinel values versus absence

Do not collapse the expiration pair into Option<i64>.

High-churn activity metadata

Synchronize LastAccessTime and UsageCount, but do not place them in the same conflict unit as passwords or other entry content. Otherwise opening an entry on one machine can create password-history conflicts.

Add a separate record kind, conceptually:

pub struct ItemActivityRecord {
    pub item_id: ItemId,
    pub last_access: Option<KeePassTime>,
    pub usage_count: Option<u64>,
    pub state: RecordState,
    pub version: Version,
    pub content_hash: [u8; 32],
}

Merge rules:

last_access: maximum timestamp
usage_count: maximum observed value, not sum
Reapplying the same record is idempotent
Activity-only changes never create entry-history items

If actual KeePass behavior shows that usage counters are safely additive across independent replicas, that can be proposed separately with dedicated CRDT semantics. Do not assume additivity in the first implementation.

Tri-state booleans

Represent KeePass inherited booleans as an enum rather than collapsing Option<bool>:

pub enum InheritedBool {
    Inherit,
    Enabled,
    Disabled,
}

Use it for at least:

Group EnableAutoType
Group EnableSearching

The XML values null, true, and false must round-trip distinctly.

Typed custom data

Current custom-data conversion loses timestamps, skips binary values, and maps missing values to empty strings. Replace it with a typed model:

pub struct CustomDataItem {
    pub value: CustomDataValue,
    pub last_modification: Option<KeePassTime>,
}

pub enum CustomDataValue {
    Absent,
    Text(SecretString),
    Binary(ResourceHash),
}

Requirements:

Preserve absent versus empty text.
Preserve text versus binary.
Preserve item modification time.
Store large binary values as normal encrypted resources.
Redact text values in every Debug, log, error, and audit output.
Merge per key using modification time and deterministic version/hash tie-breaking when timestamps are missing or equal.

Use this model for database, group, entry, and historical-entry custom data.

Complete auto-type configuration

Expand auto-type synchronization to include:

Enabled state
Data-transfer obfuscation
Default sequence
Ordered window/sequence associations

Do not preserve omitted auto-type fields independently on each destination KDBX after this change.

Database record

Expand DatabaseRecord to synchronize all shared metadata, including:

Name and DatabaseNameChanged
Description and DatabaseDescriptionChanged
Default username and DefaultUserNameChanged
Maintenance history days
Database color
Master-key recommendation/force policy
Recycle-bin enabled, group UUID, and change time
Entry-template group UUID and change time
History maximum items and size
Settings change time
Memory-protection policy
Typed custom data
Custom-icon membership and metadata

Keep file-local credential and encryption-envelope values outside the shared record. In particular, do not copy a remote KDF or key-file configuration onto a local KDBX.

Group record

Expand GroupRecord to include:

Tags
Default auto-type sequence
EnableAutoType tri-state
EnableSearching tri-state
Previous parent group
Complete synchronized timestamps
Typed custom data
Icon reference
View fields such as IsExpanded and LastTopVisibleEntry
Explicit child ordering

View fields are synchronized because the product requirement is full decrypted semantic convergence. Keep them in a separate view-state record if necessary to avoid content conflicts and excessive group revisions.

Entry record

Expand EntryRecord to include:

All fields and protection state
Ordered tags if KeePass ordering is user-visible
Foreground/background colors
Override URL
Complete auto-type configuration
Icon reference
Typed custom data
Quality-check state
Previous parent group
Complete synchronized timestamps except activity values split into the activity record
Attachment name, content hash, and protection state

Parent group and LocationChanged must be one atomic merge unit. A merge must never select a parent from one revision and location timestamp from another.

Child ordering

The vendored keepass model currently stores group and entry children in sets, which cannot preserve mixed XML ordering reliably.

Change the vendored model or add an explicit Lattice ordering representation. The first implementation may use a per-group ordered child list with these merge rules:

Select the winning ordered list by normal version ordering.
Append children missing from the winning list in deterministic ItemId order.
Remove children whose synchronized parent is no longer the group.
Reject duplicate child IDs.

A future sequence CRDT may replace this if concurrent reorder quality becomes a requirement. The first implementation must at least converge without dropping children.

Entry-history architecture

Do not embed recursive entries

Do not add Vec<EntryRecord> to EntryRecord. A historical entry must not contain nested history, sync envelope state, or an independent live parent.

Add a dedicated record variant for each immutable history item:

pub struct HistoryItemId([u8; 32]);

pub struct HistoricalEntry {
    pub fields: BTreeMap<String, FieldValue>,
    pub tags: Vec<String>,
    pub foreground_color: Option<String>,
    pub background_color: Option<String>,
    pub override_url: Option<String>,
    pub auto_type: Option<AutoTypeConfig>,
    pub icon: Option<IconRef>,
    pub custom_data: BTreeMap<String, CustomDataItem>,
    pub times: KeePassTimes,
    pub attachments: Vec<AttachmentRef>,
    pub quality_check: bool,
    pub previous_parent_group: Option<GroupUuid>,
}

pub struct EntryHistoryItemRecord {
    pub owner_entry_uuid: EntryUuid,
    pub history_item_id: HistoryItemId,
    pub snapshot: HistoricalEntry,
    pub state: RecordState,
    pub version: Version,
    pub content_hash: [u8; 32],
}

Add corresponding Record and RecordId variants.

History identity

Derive HistoryItemId from a domain-separated canonical hash of:

Owner entry UUID
Complete non-recursive historical snapshot

Exact duplicates deduplicate naturally. Two revisions with the same LastModificationTime but different content remain distinct.

History ordering

Materialize KeePass history in this deterministic order:

LastModificationTime, newest first
HistoryItemId as tie-breaker

Missing modification timestamps sort after present timestamps and then by ID. Document the ordering in tests.

Conflict behavior

For concurrent live-entry edits:

Select the live winner using the existing revision/version rules.
Union existing history-item records.
Convert the losing live snapshot into a history-item record.
Commit the winner and history addition in the same revision.
Reapplying or receiving the same branch must not duplicate the history item.

Delete versus live edit must not resurrect a deleted entry merely to hold history. Preserve the existing encrypted conflict path where necessary.

Resolving a synchronization conflict must not delete a legitimate synchronized KeePass history item. Explicit history pruning is a separate user action.

History pruning

Deleting a history item creates a normal tombstone for its history record. Tombstones prevent stale peers from resurrecting old passwords.

Apply configured history count/size limits deterministically. Pruning must account for attachment sizes and produce the same tombstones on every peer.

History resources

Resource reachability and garbage collection must include:

Live entry attachments
Historical-entry attachments
Live custom icons
Custom icons referenced only by history
Binary custom data
Retained tombstone/conflict windows where required

No payload may be deleted while referenced by any live history item.

Vendored `keepass` history API

Add an atomic API to third_party/keepass that:

Removes old live/history attachment and icon back-references.
Replaces the complete history list.
Resolves or creates attachment IDs.
Rebuilds (EntryId, history_index) attachment references.
Rebuilds custom-icon history references.
Garbage-collects only truly unreferenced payloads.
Validates that historical entries have no nested history.

Do not implement synchronized history by repeatedly calling the current History::add_entry; inserting at index zero can invalidate existing resource indexes.

Merge semantics

Define and test merge behavior per field category rather than using one blanket record-level policy.

Category	Rule
Entry/group content	Existing deterministic version ordering
Parent + location time	Atomic deterministic winner
Expiration enabled + time	Atomic deterministic winner
Last access	Maximum
Usage count	Maximum
History additions	Set union by `HistoryItemId`
History deletions	Tombstone/version ordering
Custom data	Per-key timestamp, then version/hash tie-break
Metadata field + changed time	Atomic deterministic winner
Tri-state settings	Normal version ordering, preserving all three states
Ordered children	Winning list plus deterministic missing-child repair
View state	Normal version ordering in a separate record

Property-test every merge intended to be a join operation for idempotence, commutativity, and associativity.

KDBX conversion and materialization

Read conversion

Primary files:

src/kdbx/convert.rs
src/kdbx/metadata.rs
src/kdbx/resources.rs
src/kdbx/unsupported.rs

Required changes:

Convert every modeled database/group/entry field exactly.
Convert full non-recursive history entries.
Include history-only resources.
Convert custom-icon name and modification time.
Convert typed custom data without dropping binary values.
Preserve exact timestamp option states and sentinel values.
Replace no-op V2 unknown-data rejection hooks with explicit validation.
Fail closed on an unmodeled XML field rather than silently preserving it locally.

Write conversion

Primary files:

src/kdbx/write.rs
src/kdbx/metadata.rs
third_party/keepass/src/db/types/
third_party/keepass/src/format/xml_db/

Required changes:

Materialize every shared field from synchronized records.
Stop preserving newly synchronized fields from the destination KDBX.
Combine content and activity records into exact KeePass Times.
Apply parent and location time atomically.
Materialize full history in one atomic pass.
Rebuild resource and icon references after history materialization.
Apply custom-icon metadata after payload upsert so local now() values do not leak into output.
Reopen every prepared KDBX and compare the complete canonical model before committing.

Direct breaking protocol changes

Primary files:

src/core/record.rs
src/core/v2_hash.rs
src/core/v2_manifest.rs
src/core/v2_merge.rs
src/core/v2_protocol.rs
src/core/v2_sync_state.rs
src/core/revision.rs
src/core/resource.rs
src/net/codec.rs
src/membership/session.rs
src/daemon/sync_state_store.rs
src/daemon/vault_task.rs
src/offline/object_bundle.rs

Requirements:

Directly update the existing record types and call sites.
Bump all format constants listed above.
Reject protocols 9 and 10 without attempting to decode new record bodies.
Bind session authentication and message-auth transcripts to protocol 11.
Reject state formats through 8, revision formats through 7, bundle formats through 3, and journal formats through 4 with an actionable pre-production reset error.
Do not add fallback decoders or migration code.
Update tombstone compaction so every new sensitive field is cleared from compacted deleted records.
Ensure descriptor, manifest, revision, and content hashes include the intended new semantic state.

Pre-production reset workflow

Old record hashes and signed revisions are invalid after this change. Existing Lattice state must be discarded and vaults paired again.

Implement a guarded reset command or script that:

Requires the daemon to be stopped.
Discovers every managed KDBX.
Copies each KDBX to an explicit backup directory.
Verifies backup size and SHA-256.
Reopens each backup with supplied credentials before deleting metadata.
Prints every path it will remove.
Requires explicit confirmation unless running in an isolated test root.
Deletes membership, sync state, revisions, journals, peer caches, and object caches.
Never deletes the only verified copy of a KDBX.
Prints fresh vault init / invite / join instructions.

Operational reset sequence:

Upgrade every binary together.
Back up all KDBXs.
Choose or manually prepare one authoritative KDBX if pre-production replicas contain divergent data worth keeping.
Reset old Lattice metadata.
Register the authoritative vault.
Initialize fresh membership.
Pair the other devices again.
Allow the new complete model to synchronize.

The physical E2E harness already uses isolated roots and should simply create fresh state.

Implementation sequence

Phase 1: field inventory and fixtures

Deliverables:

A table classifying every supported KeePass XML field as synchronized, file-local, or derived.
Real KeePassXC fixtures covering every newly modeled field.
A fixture with full entry history, a history-only attachment, a history-only custom icon, binary custom data, all timestamp option states, all tri-state values, and child ordering.
Updated raw XML gate coverage.

Gate:

No accepted XML field remains unclassified.
Unknown fields fail closed.

Phase 2: canonical model and hashes

Deliverables:

Exact timestamp, tri-state, custom-data, full auto-type, activity, view-state, ordering, and history types.
Directly expanded current record enum and IDs.
Explicit canonical hash encoder.
Golden canonical bytes and hashes.

Gate:

Every semantic field affects the intended record hash.
Map/set input order cannot change canonical output.
Debug output exposes no protected values.

Phase 3: full-fidelity KDBX read/write

Deliverables:

Complete read projection.
Complete materialization.
Vendored atomic history/reference API.
History-aware resource retention.

Gate:

KDBX -> records -> KDBX -> records is exactly equal.
Decrypted XML differs only on the explicit file-local/derived allowlist.
Historical attachment/icon payload hashes survive round-trip.

Phase 4: merge semantics

Deliverables:

History union and tombstones.
Activity merge.
Per-key custom-data merge.
Atomic metadata, parent/location, and expiration merges.
Deterministic ordering repair.

Gate:

Repeated and out-of-order synchronization is idempotent.
Concurrent history additions converge without loss.
Activity-only edits create no password history.
Delete/edit races do not resurrect entries.

Phase 5: breaking format bump

Deliverables:

Protocol 11.
Revision 8.
Sync state 9.
Object bundle 4.
Materialization journal 5.
Actionable stale-state failures.
Guarded reset workflow.

Gate:

Old binaries cannot communicate with new binaries.
Old state fails before any KDBX write.
Fresh state initializes and synchronizes normally.

Phase 6: exhaustive E2E

Deliverables:

Expanded in-process two- and three-node tests.
Expanded physical relay-only harness.
Redacted exhaustive audit evidence.

Gate:

All authorized replicas produce identical complete semantic manifests.
Full decrypted XML has zero unexpected differences.
Repeating sync and restarting all nodes produces no new differences.

Test plan

Core and hash tests

Primary files:

tests/core_v2_records.rs
tests/core_model.rs
tests/revision.rs
tests/tombstone_compaction.rs

Cases:

Canonical encoding/hash golden vectors for every record variant.
Distinct hashes for all tri-state values.
Distinct hashes for absent, false, sentinel, and active expiry states.
Protected fields remain redacted.
History IDs are deterministic.
History union is idempotent, commutative, and associative.
Compacted tombstones contain no sensitive new fields.

KDBX fidelity tests

Primary files:

tests/kdbx_gate.rs
tests/kdbx_v2_compat.rs

Cases:

Exact metadata timestamp round-trip.
Typed text/binary/absent custom-data round-trip.
All group tri-state combinations.
Complete auto-type associations.
Exact timestamp option and sentinel preservation.
Entry history with protected fields.
History-only attachment and icon.
Child ordering.
Deleted-object timestamps.
Full semantic equality after save and reopen.

Replace tests that assert affected values are “locally preserved but not synced” with convergence assertions.

Protocol and persistence tests

Primary files:

tests/v2_protocol.rs
tests/net_codec.rs
tests/membership_session.rs
tests/sync_state.rs
tests/object_bundle.rs
tests/daemon.rs

Cases:

Protocol 11 encode/decode and transcript binding.
Protocols 9 and 10 rejected before record decoding.
State formats through 8, revision formats through 7, bundle formats through 3, and journal formats through 4 produce actionable rejection.
Fresh state 9 initialization.
Journal 5 crash recovery at every durable boundary.
Object bundle 4 includes history-only resources.
Tampering with any new semantic field fails hash/signature validation.

Synchronization tests

Primary files:

tests/e2e_sync.rs
tests/daemon_v2_sync.rs
tests/network_sync_operation.rs
tests/local_two_node_e2e.rs
tests/local_three_node_e2e.rs

Cases:

Existing history on A appears identically on B and C.
Concurrent history additions union on all nodes.
Concurrent live edits produce one deterministic winner and one synchronized losing history item.
Repeating sync adds no duplicate history.
History deletion tombstones prevent resurrection.
History-only attachment/icon survives transfer, restart, and GC.
Binary custom data converges.
Activity metadata converges without content conflicts.
Group tri-state and view metadata converge.
Parent moves preserve matching LocationChanged.
Active and inactive expiration representations converge exactly.
Child ordering converges after concurrent additions.

Physical relay-only E2E

Update xtask/two-machine-relay-e2e.sh to make complete semantic equality a mandatory gate. The test must still prove:

Fresh signed pairing
Rendezvous-discovered /p2p-circuit addresses
Peer connections with relayed=true
Zero successful hole punches
Bidirectional edits
Attachment transfer
Cache-free restart and rediscovery
Deletion propagation
Infrastructure stability

Additional final assertions:

record_manifest_result=identical
history_manifest_result=identical
full_semantic_xml_result=identical
unexpected_xml_differences=0

The XML allowlist must contain only reviewed crypto-envelope and derived fields. A new unexpected path fails the test.

Validation commands

Run targeted checks during implementation, then the full acceptance suite:

cargo test --locked --test core_v2_records
cargo test --locked --test kdbx_gate --test kdbx_v2_compat
cargo test --locked --test v2_protocol --test net_codec
cargo test --locked --test sync_state --test object_bundle
cargo test --locked --test e2e_sync --test daemon_v2_sync
cargo test --locked --test local_two_node_e2e --test local_three_node_e2e
cargo test --locked --test net_runtime
cargo check --locked
cargo fmt --check
git diff --check

Physical acceptance:

LATTICE_E2E_REQUIRE_INFRA_JOURNAL=1 \
LATTICE_E2E_REQUIRE_FULL_XML_MATCH=1 \
  bash xtask/two-machine-relay-e2e.sh

Definition of done

This work is complete only when all of the following are true:

Every supported shared KeePass XML field is represented in the canonical synchronized model.
Every excluded field is documented as crypto-envelope-local or derived.
No accepted field is silently preserved only from the destination KDBX.
Entry history is identical across replicas.
History attachments and custom icons survive transfer and garbage collection.
Tri-state settings preserve inheritance exactly.
Custom data preserves type, absence, timestamp, and bytes.
Activity-only changes do not generate content conflicts.
Parent/location and expiration pairs are atomic.
Child ordering converges without dropping items.
Old pre-production state fails safely and instructs the user to reset.
Freshly paired two- and three-node vaults converge after repeated restarts.
The physical relay-only E2E produces zero unexpected decrypted XML differences.

Guidance for the implementation agent

Treat this as a coordinated breaking release; do not add legacy migration or dual-stack compatibility unless requirements change.
Work in the phases above and keep each phase independently testable.
Do not weaken or delete existing security, anti-rollback, atomic-write, or resource-integrity checks to make new tests pass.
Do not normalize away a KeePass distinction merely because it complicates the model.
Keep secrets out of logs, diagnostics, manifests, and test failure output.
Use content hashes and redacted XML paths for exhaustive evidence.
Preserve KDBX files before any reset or incompatible-state cleanup.
Stop and update this design if the vendored keepass model cannot represent a required shared value without loss.

Full KDBX Semantic Synchronization Plan

Status

Motivation

Target invariant

Non-goals

Breaking-upgrade policy

Data-model design

Canonical hashes

KeePass timestamps

High-churn activity metadata

Tri-state booleans

Typed custom data

Complete auto-type configuration

Database record

Group record

Entry record

Child ordering

Entry-history architecture

Do not embed recursive entries

History identity

History ordering

Conflict behavior

History pruning

History resources

Vendored keepass history API

Merge semantics

KDBX conversion and materialization

Read conversion

Write conversion

Direct breaking protocol changes

Pre-production reset workflow

Implementation sequence

Phase 1: field inventory and fixtures

Phase 2: canonical model and hashes

Phase 3: full-fidelity KDBX read/write

Phase 4: merge semantics

Phase 5: breaking format bump

Phase 6: exhaustive E2E

Test plan

Core and hash tests

KDBX fidelity tests

Protocol and persistence tests

Synchronization tests

Physical relay-only E2E

Validation commands

Definition of done

Guidance for the implementation agent

Vendored `keepass` history API