diff --git a/README.md b/README.md index 584af51..42c227f 100644 --- a/README.md +++ b/README.md @@ -83,10 +83,11 @@ npx cdb-converter --version | `.cdb` | CDB → SQLite | `.sqlite` | | `.sqlite` / `.db` | SQLite → CDB | `.cdb` | -| Option | Effect | -| ------------------- | --------------------------------------------------------------------------------------------------------------------------- | -| `-n`, `--normalize` | (CDB → SQLite only) reconstruct PK/FK constraints from PCM naming conventions. See [Normalized schema](#normalized-schema). | -| `--index-fk` | Implies `--normalize`; also indexes every FK column for faster JOINs (roughly doubles output size). | +| Option | Effect | +| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `-n`, `--normalize` | (CDB → SQLite only) reconstruct PK/FK constraints from PCM naming conventions. See [Normalized schema](#normalized-schema). | +| `--index-fk` | Implies `--normalize`; also indexes every FK column for faster JOINs (roughly doubles output size). | +| `--precise-types` | (CDB → SQLite only) preserve the exact CDB type (BOOLEAN, INTEGER_BYTE, INTEGER_SHORT) instead of collapsing it to plain INTEGER. See [Compatibility](#compatibility). | ## Library usage @@ -203,6 +204,7 @@ Convert CDB binary data into a SQLite database instance. - **`SQL`** — `SqlJsStatic`, the module returned by `initSqlJs()`. - **`options.normalize`** — `boolean` (default `false`). Reconstruct PK/FK constraints from PCM naming conventions. See [Normalized schema](#normalized-schema). - **`options.indexForeignKeys`** — `boolean` (default `false`). When normalizing, also index every FK column for faster JOINs (roughly doubles the output size). +- **`options.preciseTypes`** — `boolean` (default `false`). Preserve the exact CDB type (BOOLEAN, INTEGER_BYTE, INTEGER_SHORT) and each table's flags in the `.sqlite` file instead of the official-tool-compatible defaults. See [How metadata is preserved](#how-metadata-is-preserved). - **returns** — a `sql.js` `Database` with the CDB tables loaded. ### `sqlToCdb(db): ArrayBuffer` @@ -242,18 +244,20 @@ Every CDB data type is preserved during conversion: The library uses a special `DB_STRUCTURE` table to round-trip CDB metadata that has no native SQLite equivalent: ```sql -CREATE TABLE DB_STRUCTURE ( - TableName TEXT '274', - ID INTEGER, - Flags INTEGER -) +-- default (compatible with the official PCM SQLiteExporter tool) +CREATE TABLE DB_STRUCTURE (TableName '274', ID '0') + +-- with { preciseTypes: true } +CREATE TABLE DB_STRUCTURE (TableName TEXT '274', ID INTEGER, Flags INTEGER) ``` -Each table's flags (their exact meaning is unknown but must be preserved) are stored in the `Flags` column, so they are written into the `.sqlite` file itself and survive an `export()`/reopen cycle. Column indices and data types are encoded into each column's declared type annotation. Together this makes `cdb → sqlite → cdb` lossless even when the SQLite database is saved to disk and reopened in a separate process. +Column indices and data types are encoded into each column's declared type annotation, so `cdb → sqlite → cdb` preserves every row value even when the SQLite database is saved to disk and reopened in a separate process. How much of the _schema_ survives depends on the mode: `preciseTypes: true` round-trips the CDB types and table flags exactly, while the default trades some of that fidelity for interop. By default, CDB's narrower integer types (`BOOLEAN`, `INTEGER_BYTE`, `INTEGER_SHORT`) are encoded as plain `INTEGER`, and each table's flags (their exact meaning is unknown but must be preserved) are **not** written to the `.sqlite` file — `sqlToCdb` falls back to a static table of flags extracted from official PCM saves (`TABLE_FLAGS_BY_ID`) instead. Pass `{ preciseTypes: true }` (`--precise-types` on the CLI) to encode the exact CDB type and store each table's real flags in the `Flags` column instead of relying on that fallback. + +This default exists specifically for interop: the official PCM `SQLiteExporter` tool only recognizes `FLOAT`, `STRING` and the two list types in this metadata and has no `Flags` column — a `.sqlite` written with `preciseTypes: true` crashes it on import. Leave `preciseTypes` off if you need the output to be re-importable by that tool; turn it on if `cdb-converter` (via `sqlToCdb`) is the only tool that will ever read the file back and you want the extra fidelity. ## Compatibility -The CDB parser is **format-driven, not version-specific**, so it is not tied to a single Pro Cycling Manager release. Lossless round-trip conversion (`cdb → sqlite → cdb`) is tested against the official databases of: +The CDB parser is **format-driven, not version-specific**, so it is not tied to a single Pro Cycling Manager release. Round-trip conversion (`cdb → sqlite → cdb`) is tested against the official databases of — losslessly, including types and flags, with `preciseTypes: true`, and preserving all row data in the default mode: | Version | Status | | ------------------------ | --------- | @@ -263,6 +267,8 @@ The CDB parser is **format-driven, not version-specific**, so it is not tied to | Pro Cycling Manager 2021 | ✅ tested | | Pro Cycling Manager 2025 | ✅ tested | +The default (non-`preciseTypes`) `.sqlite` output is also verified importable by the official PCM `SQLiteExporter` tool (`-import`) on Pro Cycling Manager 2025 saves, round-tripping back through `cdb-converter` with identical data. `SQLiteExporter` itself cannot export the 2014 fixture (it crashes on that file directly, independent of anything produced by this library), so that combination isn't claimed. + ## Performance & size A full `cdb → sqlite → cdb` round-trip on a real ~60k-row database stays well under half a second, and the library's own code adds only **~28 kB** — the SQLite WASM runtime is the real weight, and you would pay for it with any SQLite-in-JS approach. diff --git a/src/cdbToSql.ts b/src/cdbToSql.ts index 7ff3443..7c9a081 100644 --- a/src/cdbToSql.ts +++ b/src/cdbToSql.ts @@ -132,10 +132,18 @@ export function cdbToSql( // DB_STRUCTURE mirrors the PCM convention used by sqlToCdb: TableName keeps the // literal type annotation '274' so the schema matches the metadata table shape // expected by round-trip consumers, while only the table rows are read back. - // Flags persists each table's TABLE_FLAGS into the SQLite file so it survives an - // export()/reopen round-trip (its meaning is unknown but must be preserved). + // + // The official SQLiteExporter tool declares this table with no SQL type + // keyword at all (`TableName '274',ID '0'`) and no Flags column; matching that + // exactly by default keeps our output importable there (a mismatch here isn't + // just cosmetic — SQLiteExporter crashes on the extra/typed columns). Flags + // persists each table's TABLE_FLAGS so it survives an export()/reopen + // round-trip; it's only added under preciseTypes since sqlToCdb already falls + // back to TABLE_FLAGS_BY_ID when the column is absent. db.run( - `CREATE TABLE DB_STRUCTURE (TableName TEXT '274', ID INTEGER, Flags INTEGER)`, + options?.preciseTypes + ? `CREATE TABLE DB_STRUCTURE (TableName TEXT '274', ID INTEGER, Flags INTEGER)` + : `CREATE TABLE DB_STRUCTURE (TableName '274',ID '0')`, ); const keyMap = options?.normalize ? inferKeys(tables) : null; @@ -159,11 +167,18 @@ export function cdbToSql( db.run("BEGIN TRANSACTION"); tables.forEach((table) => { - db.run(`INSERT INTO DB_STRUCTURE VALUES (?, ?, ?)`, [ - table.name, - table.tableId, - table.tableFlags, - ]); + if (options?.preciseTypes) { + db.run(`INSERT INTO DB_STRUCTURE VALUES (?, ?, ?)`, [ + table.name, + table.tableId, + table.tableFlags, + ]); + } else { + db.run(`INSERT INTO DB_STRUCTURE VALUES (?, ?)`, [ + table.name, + table.tableId, + ]); + } const escapedTableName = escapeSqlIdentifier(table.name); // Keep columns in original file order (do NOT sort) @@ -171,25 +186,40 @@ export function cdbToSql( .map((col) => { const escapedColumnName = escapeSqlIdentifier(col.name); let baseType: string; + let encodedType: number; switch (col.type) { case DATA_TYPE.FLOAT: baseType = "REAL"; + encodedType = col.type; break; case DATA_TYPE.STRING: case DATA_TYPE.INTEGER_LIST: case DATA_TYPE.FLOAT_LIST: baseType = "TEXT"; + encodedType = col.type; break; case DATA_TYPE.BOOLEAN: - baseType = "NUMERIC"; + // `SQLiteExporter` (the official PCM tool) has no case for BOOLEAN, + // INTEGER_BYTE or INTEGER_SHORT: they all fall into its default + // branch and get encoded as plain INTEGER. Match that by default so + // our output stays importable there; preciseTypes opts back into + // preserving the exact CDB type for our own round-trip. + baseType = options?.preciseTypes ? "NUMERIC" : "INTEGER"; + encodedType = options?.preciseTypes ? col.type : DATA_TYPE.INTEGER; + break; + case DATA_TYPE.INTEGER_BYTE: + case DATA_TYPE.INTEGER_SHORT: + baseType = "INTEGER"; + encodedType = options?.preciseTypes ? col.type : DATA_TYPE.INTEGER; break; default: baseType = "INTEGER"; + encodedType = DATA_TYPE.INTEGER; break; } const encodedValue = - (table.tableId * 256 + col.columnIndex) * 16 + (col.type & 0xf); + (table.tableId * 256 + col.columnIndex) * 16 + (encodedType & 0xf); return `"${escapedColumnName}" '${baseType} ${encodedValue}'`; }) .join(", "); diff --git a/src/cli.ts b/src/cli.ts index e592750..d3ad6d0 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -21,6 +21,7 @@ export interface ParsedArgs { output?: string; normalize?: boolean; indexForeignKeys?: boolean; + preciseTypes?: boolean; } const HELP_TEXT = `cdb-converter — convert Pro Cycling Manager CDB files to/from SQLite @@ -46,12 +47,19 @@ Options: schema. Ignored when converting sqlite -> cdb. --index-fk (implies --normalize) also index every foreign-key column for faster JOINs. Roughly doubles the output size. + --precise-types + (cdb -> sqlite only) preserve the exact CDB type (BOOLEAN, + INTEGER_BYTE, INTEGER_SHORT) in the SQLite schema instead + of collapsing them to plain INTEGER. Off by default so the + output stays importable by the official PCM SQLiteExporter + tool, which does not recognize those types. Examples: cdb-converter save.cdb cdb-converter save.cdb save.sqlite cdb-converter save.cdb save.sqlite --normalize cdb-converter save.cdb save.sqlite --normalize --index-fk + cdb-converter save.cdb save.sqlite --precise-types cdb-converter -- --data.cdb (use -- to treat a leading-dash path as a positional argument) cdb-converter save.sqlite save.cdb`; @@ -59,6 +67,7 @@ export function parseArgs(argv: string[]): ParsedArgs { const positionals: string[] = []; let normalize = false; let indexForeignKeys = false; + let preciseTypes = false; let optionsEnded = false; @@ -87,6 +96,10 @@ export function parseArgs(argv: string[]): ParsedArgs { indexForeignKeys = true; continue; } + if (arg === "--precise-types") { + preciseTypes = true; + continue; + } if (arg.startsWith("-") && arg !== "-") { throw new Error( `Unknown option "${arg}". Run "cdb-converter --help" for usage.`, @@ -101,6 +114,7 @@ export function parseArgs(argv: string[]): ParsedArgs { output: positionals[1], normalize, indexForeignKeys, + preciseTypes, }; } @@ -148,6 +162,7 @@ async function convert( output: string | undefined, normalize: boolean, indexForeignKeys: boolean, + preciseTypes: boolean, ): Promise { const direction = detectDirection(input); const inputPath = resolve(process.cwd(), input); @@ -163,7 +178,11 @@ async function convert( let summary: string[] = []; if (direction === "cdb-to-sql") { - const db = cdbToSql(inputBytes, SQL, { normalize, indexForeignKeys }); + const db = cdbToSql(inputBytes, SQL, { + normalize, + indexForeignKeys, + preciseTypes, + }); try { const tables = db.exec( @@ -177,6 +196,9 @@ async function convert( if (normalize && indexForeignKeys) { summary.push("FK indexes : yes"); } + if (preciseTypes) { + summary.push("Precise types : yes"); + } outputBytes = db.export(); } finally { @@ -235,6 +257,7 @@ export async function run(argv: string[]): Promise { parsed.output, parsed.normalize ?? false, parsed.indexForeignKeys ?? false, + parsed.preciseTypes ?? false, ); } catch (error) { console.error(`Error: ${error instanceof Error ? error.message : error}`); diff --git a/src/types.ts b/src/types.ts index 280518b..5dcc1b3 100644 --- a/src/types.ts +++ b/src/types.ts @@ -96,4 +96,19 @@ export interface CdbToSqlOptions { * intend to run frequent filtered JOINs on the output. */ indexForeignKeys?: boolean; + + /** + * Encode each column's exact CDB data type (BOOLEAN, INTEGER_BYTE, + * INTEGER_SHORT) into the SQLite schema instead of collapsing them to plain + * INTEGER. Off by default. + * + * The official PCM `SQLiteExporter` tool only recognizes FLOAT, STRING and + * the two list types in this metadata; anything else (including BOOLEAN, + * INTEGER_BYTE and INTEGER_SHORT) is written as plain INTEGER. A `.sqlite` + * produced with `preciseTypes: true` preserves the exact CDB type through + * `sqlToCdb` round-trips, but its schema is not understood by that + * third-party tool and re-importing it there will crash. Leave this off if + * you need the output to be interchangeable with `SQLiteExporter`. + */ + preciseTypes?: boolean; } diff --git a/test/cdbToSql.test.ts b/test/cdbToSql.test.ts index 3e2f794..194af05 100644 --- a/test/cdbToSql.test.ts +++ b/test/cdbToSql.test.ts @@ -151,4 +151,88 @@ describe("cdb/sql conversion surface", () => { mockReadChunk.mockReset(); }); + + it("collapses BOOLEAN/INTEGER_BYTE/INTEGER_SHORT to plain INTEGER by default", () => { + const sql = createMockSqlJs(); + + // DataType: INTEGER=0, FLOAT=1, BOOLEAN=3, INTEGER_BYTE=4, INTEGER_SHORT=5 + const tableColumns = [ + { name: "id", columnIndex: 0, type: 0, data: [] }, + { name: "flag", columnIndex: 1, type: 3, data: [] }, + { name: "small", columnIndex: 2, type: 4, data: [] }, + { name: "medium", columnIndex: 3, type: 5, data: [] }, + { name: "ratio", columnIndex: 4, type: 1, data: [] }, + ]; + + mockReadChunk.mockReturnValueOnce({ + children: { + 1: [ + { + name: "Narrow", + tableId: 2, + tableFlags: 0, + rowCount: 0, + columns: tableColumns, + }, + ], + }, + }); + + cdbToSql(new Uint8Array([1, 2, 3]), sql); + const [db] = sql.createdDatabases; + + const createStatement = db.sqlOperations.find((op) => + op.sql.startsWith('CREATE TABLE "Narrow"'), + ); + + // tableId=2 -> base 8192 (2*4096); +columnIndex*16; nibble collapsed to 0 + // for id/flag/small/medium, kept as 1 (FLOAT) for ratio. + expect(createStatement?.sql).toBe( + 'CREATE TABLE "Narrow" ("id" \'INTEGER 8192\', "flag" \'INTEGER 8208\', ' + + "\"small\" 'INTEGER 8224', \"medium\" 'INTEGER 8240', \"ratio\" 'REAL 8257')", + ); + + mockReadChunk.mockReset(); + }); + + it("preserves the exact CDB type with preciseTypes: true", () => { + const sql = createMockSqlJs(); + + const tableColumns = [ + { name: "id", columnIndex: 0, type: 0, data: [] }, + { name: "flag", columnIndex: 1, type: 3, data: [] }, + { name: "small", columnIndex: 2, type: 4, data: [] }, + { name: "medium", columnIndex: 3, type: 5, data: [] }, + { name: "ratio", columnIndex: 4, type: 1, data: [] }, + ]; + + mockReadChunk.mockReturnValueOnce({ + children: { + 1: [ + { + name: "Narrow", + tableId: 2, + tableFlags: 0, + rowCount: 0, + columns: tableColumns, + }, + ], + }, + }); + + cdbToSql(new Uint8Array([1, 2, 3]), sql, { preciseTypes: true }); + const [db] = sql.createdDatabases; + + const createStatement = db.sqlOperations.find((op) => + op.sql.startsWith('CREATE TABLE "Narrow"'), + ); + + // Same base offsets, but the true nibble (3/4/5) is kept instead of 0. + expect(createStatement?.sql).toBe( + 'CREATE TABLE "Narrow" ("id" \'INTEGER 8192\', "flag" \'NUMERIC 8211\', ' + + "\"small\" 'INTEGER 8228', \"medium\" 'INTEGER 8245', \"ratio\" 'REAL 8257')", + ); + + mockReadChunk.mockReset(); + }); }); diff --git a/test/cli.test.ts b/test/cli.test.ts index e0ecea4..b51d3da 100644 --- a/test/cli.test.ts +++ b/test/cli.test.ts @@ -23,6 +23,7 @@ describe("parseArgs", () => { output: undefined, normalize: false, indexForeignKeys: false, + preciseTypes: false, }); expect(parseArgs(["save.cdb", "out.sqlite"])).toEqual({ command: "convert", @@ -30,6 +31,7 @@ describe("parseArgs", () => { output: "out.sqlite", normalize: false, indexForeignKeys: false, + preciseTypes: false, }); }); @@ -40,6 +42,7 @@ describe("parseArgs", () => { output: "out.sqlite", normalize: true, indexForeignKeys: false, + preciseTypes: false, }); expect(parseArgs(["-n", "save.cdb"])).toEqual({ command: "convert", @@ -47,6 +50,7 @@ describe("parseArgs", () => { output: undefined, normalize: true, indexForeignKeys: false, + preciseTypes: false, }); }); @@ -57,6 +61,18 @@ describe("parseArgs", () => { output: "out.sqlite", normalize: true, indexForeignKeys: true, + preciseTypes: false, + }); + }); + + it("parses the --precise-types flag", () => { + expect(parseArgs(["save.cdb", "out.sqlite", "--precise-types"])).toEqual({ + command: "convert", + input: "save.cdb", + output: "out.sqlite", + normalize: false, + indexForeignKeys: false, + preciseTypes: true, }); }); @@ -67,6 +83,7 @@ describe("parseArgs", () => { output: undefined, normalize: false, indexForeignKeys: false, + preciseTypes: false, }); }); @@ -84,6 +101,7 @@ describe("parseArgs", () => { output: undefined, normalize: false, indexForeignKeys: false, + preciseTypes: false, }); expect(parseArgs(["--", "--data.cdb", "-out.sqlite"])).toEqual({ command: "convert", @@ -91,6 +109,7 @@ describe("parseArgs", () => { output: "-out.sqlite", normalize: false, indexForeignKeys: false, + preciseTypes: false, }); expect( parseArgs(["save.cdb", "out.sqlite", "--normalize", "--", "-x"]), @@ -100,6 +119,7 @@ describe("parseArgs", () => { output: "out.sqlite", normalize: true, indexForeignKeys: false, + preciseTypes: false, }); }); }); diff --git a/test/roundtrip.test.ts b/test/roundtrip.test.ts index 2dc30bb..a7e8d24 100644 --- a/test/roundtrip.test.ts +++ b/test/roundtrip.test.ts @@ -40,15 +40,22 @@ interface TableSnapshot { * so two conversions can be compared without false negatives from table ordering. */ function snapshot(db: SqlDatabase): TableSnapshot[] { + const structureInfo = db.exec(`PRAGMA table_info("DB_STRUCTURE")`); + const hasFlagsColumn = + structureInfo.length > 0 && + structureInfo[0].values.some((row) => row[1] === "Flags"); + const structure = db.exec( - `SELECT TableName, ID, Flags FROM DB_STRUCTURE ORDER BY ID`, + hasFlagsColumn + ? `SELECT TableName, ID, Flags FROM DB_STRUCTURE ORDER BY ID` + : `SELECT TableName, ID FROM DB_STRUCTURE ORDER BY ID`, ); if (structure.length === 0) return []; return structure[0].values.map((row) => { const name = row[0] as string; const id = row[1] as number; - const flags = (row[2] ?? null) as number | null; + const flags = (hasFlagsColumn ? (row[2] ?? null) : null) as number | null; const schema = db.exec(`PRAGMA table_info("${name}")`); const columns = schema[0].values.map( @@ -68,8 +75,14 @@ describe("cdb <-> sql round-trip (no data loss)", () => { (_label, fixturePath) => { const original = readFileSync(fixturePath); + // preciseTypes: true exercises full fidelity (exact CDB type + table + // flags stored in the .sqlite itself, not the TABLE_FLAGS_BY_ID + // fallback). The default (compatible with the official SQLiteExporter + // tool) is covered separately below. + const options = { preciseTypes: true }; + // 1. cdb -> sql - const db1 = cdbToSql(original, SQL); + const db1 = cdbToSql(original, SQL, options); const before = snapshot(db1); // 2. Serialize to SQLite bytes and reopen — mirrors the CLI writing a .sqlite @@ -77,7 +90,7 @@ describe("cdb <-> sql round-trip (no data loss)", () => { const db2 = new SQL.Database(db1.export()) as SqlDatabase; // 3. sql -> cdb -> sql - const db3 = cdbToSql(sqlToCdb(db2), SQL); + const db3 = cdbToSql(sqlToCdb(db2), SQL, options); const after = snapshot(db3); try { @@ -101,5 +114,46 @@ describe("cdb <-> sql round-trip (no data loss)", () => { db3.close(); } }, + 15000, + ); + + it.each(saveFixtures)( + "preserves row data by default (official-compatible schema, no preciseTypes)", + (_label, fixturePath) => { + const original = readFileSync(fixturePath); + + // Default options: DB_STRUCTURE has no Flags column and narrow CDB + // types (BOOLEAN/INTEGER_BYTE/INTEGER_SHORT) collapse to plain INTEGER, + // matching the official SQLiteExporter tool. Table flags round-trip via + // the TABLE_FLAGS_BY_ID fallback instead of the .sqlite file itself, so + // they're intentionally not asserted here (see the preciseTypes test + // above for exact flag fidelity). + const db1 = cdbToSql(original, SQL); + const before = snapshot(db1); + + const db2 = new SQL.Database(db1.export()) as SqlDatabase; + const db3 = cdbToSql(sqlToCdb(db2), SQL); + const after = snapshot(db3); + + try { + expect(after.map((t) => `${t.id}:${t.name}`)).toEqual( + before.map((t) => `${t.id}:${t.name}`), + ); + + for (let i = 0; i < before.length; i++) { + expect(after[i].columns, `columns of ${before[i].name}`).toEqual( + before[i].columns, + ); + expect(after[i].rows, `rows of ${before[i].name}`).toEqual( + before[i].rows, + ); + } + } finally { + db1.close(); + db2.close(); + db3.close(); + } + }, + 15000, ); });