Added the pgvector extension

[Hydrocharged]

This emulates a majority of the pgvector extension, with the major missing pieces being the special index types which we cannot support at this moment (HNSW and IVFFlat). This also includes a port of most of the tests from the actual pgvector repository. Implementing and testing this uncovered additional bugs which were fixed as well.

[github-actions]

	Main	PR
Total	42090	42090
Successful	19192	19206
Failures	22898	22884
Partial Successes1	5472	5469

	Main	PR
Successful	45.5975%	45.6308%
Failures	54.4025%	54.3692%

${\color{lightgreen}Progressions (15)}$

arrays

QUERY: SELECT NOT ARRAY[1.1,1.2,1.3] = ARRAY[1.1,1.2,1.3] AS "FALSE";

QUERY: SELECT * FROM array_op_test WHERE i = '{}' ORDER BY seqno;

QUERY: SELECT * FROM array_op_test WHERE i = '{NULL}' ORDER BY seqno;

QUERY: SELECT * FROM array_op_test WHERE t = '{}' ORDER BY seqno;

QUERY: select * from arr_tbl where f1 > '{1,2,3}' and f1 <= '{1,5,3}';

create_index

QUERY: SELECT * FROM array_index_op_test WHERE i = '{NULL}' ORDER BY seqno;

QUERY: SELECT * FROM array_index_op_test WHERE i = '{47,77}' ORDER BY seqno;

QUERY: SELECT * FROM array_index_op_test WHERE i = '{}' ORDER BY seqno;

QUERY: SELECT * FROM array_index_op_test WHERE t = '{AAAAAAAAAA646,A87088}' ORDER BY seqno;

QUERY: SELECT * FROM array_index_op_test WHERE t = '{}' ORDER BY seqno;

QUERY: SELECT * FROM array_index_op_test WHERE t = '{}' ORDER BY seqno;

create_view

QUERY: create type nestedcomposite as (x int8_tbl);

opr_sanity

QUERY: SELECT p1.oid, p1.proname, p1.proargtypes, p1.proallargtypes, p1.proargmodes
FROM pg_proc as p1
WHERE proallargtypes IS NOT NULL AND
  ARRAY(SELECT unnest(proargtypes)) <>
  ARRAY(SELECT proallargtypes[i]
        FROM generate_series(1, array_length(proallargtypes, 1)) g(i)
        WHERE proargmodes IS NULL OR proargmodes[i] IN ('i', 'b', 'v'));

subselect

QUERY: select count(*) from tenk1 t
where (exists(select 1 from tenk1 k where k.unique1 = t.unique2) or ten < 0);

triggers

QUERY: insert into self_ref values (1, null), (2, 1), (3, 2), (4, 3);

Footnotes

1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct. ↩

[itoqa]

Commit: 9cef722: 19 test cases ran, 4 failed ❌, 14 passed ✅, 1 additional finding ⚠️.

Summary

Coverage spans core vector, half-vector, sparse-vector, and UUID operations, including arithmetic, distances, aggregates, conversions, reconnect behavior, and schema resolution. It also exercises boundary values, malformed and non-finite inputs, dimension mismatches, constrained columns and arrays, and database metadata behavior.

Not safe to merge yet — PR-attributable high-severity failures allow dimension-constrained columns to store incorrectly sized values and lose their declared metadata, creating a risk of silently invalid data; a separate PR-attributable resolution issue also breaks unqualified vector distance calls in a common schema configuration. The missing foreign-key metadata is unrelated to this PR and is a flag for later rather than a merge decision driver.

Tests run by Ito

Result	Severity	Type	Description
❌		General	The dimension limit written in a table column declaration disappeared after the table was created. The database accepted a vector with two dimensions in a column declared as dimension three, and the system catalog could not report the declared modifier.
❌		Types	Creating columns with vector(3), halfvec(3), sparsevec(3), and the corresponding arrays did not preserve the declared dimension. Metadata showed atttypmod = -1 and format_type = ??? for all six columns, and inserting a two-dimensional value into the constrained table succeeded.
❌		General	A two-dimensional value was accepted by columns declared for three dimensions. The same columns reported atttypmod=-1 and format_type=??? instead of retaining their declared dimension.
❌		Search	The expected vector distance calls failed with a function-does-not-exist error even though the extension provides matching routines. A same-named routine with a different number of arguments was allowed to hide the extension routines.
✅	—	General	Boundary values were accepted, kept three dimensions, and produced consistent results across normalization, addition, and type conversion. Values with a fourth dimension were rejected with clear dimension errors.
✅	—	General	The tested dimension 1,000,001 is accepted because it is below the supported sparse-vector limit of 1,000,000,000. The session stayed usable after the boundary check, and the correct one-past value is 1,000,000,001.
✅	—	General	Vector, halfvec, and sparsevec data continued to work after reconnecting. Distance checks, aggregate results, type metadata, and the installed extension version all stayed consistent.
✅	—	General	Invalid sparse vector text was rejected, and the same session continued to return valid sparse vector and scalar results. The binary-parameter check could not run because the local SQL engine and installed clients did not provide a way to send binary parameters.
✅	—	General	Direct distance calls and their matching operators returned the same values for valid bit strings, and strict calls returned NULL when either input was NULL. The prepared-statement step was unavailable because Doltgres does not yet support PREPARE, but this is a platform limitation outside the vector routines.
✅	—	General	Existing and newly created local databases kept the same UUID routines after a fresh connection. Deterministic results matched, and random UUID generation returned valid non-null values in both databases.
✅	—	General	Changing the schema search order selected the routine in the new first schema, and recreating a routine took effect immediately. The vector distance routine still worked after returning to the public schema.
✅	—	Extension	The vector extension installed successfully, and vector, halfvec, and sparsevec values worked in SQL. The expected types, functions, aggregates, operators, and casts were present.
✅	—	Extension	The average of two three-value vectors returned [2,3,4], and their total returned [4,6,8]. Both results kept dimension 3.
✅	—	Operators	The distance operators and direct functions return matching values. The all-zero Jaccard case returns 1, and calls with NULL return NULL as expected.
✅	—	Sparse	The sparse value was reordered into canonical form, its distance and normalization results were correct, and conversion to both dense formats kept dimension 5.
✅	—	Uuid	The UUID extension installed successfully, and its namespace, nil, deterministic, and random UUID routines returned the expected results.
✅	—	Vector	Dense vectors and half-precision vectors accepted valid values and returned the expected dimensions, distances, math results, normalized values, conversions, averages, and sums.
✅	—	Vector	Empty, malformed, non-finite, and wrong-size vector values were rejected with clear validation errors. No invalid vector or halfvec value was created.
⚠️		Foreign	The foreign key rejects a missing parent as expected, but the metadata query shows no referenced column for the constraint.

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Foreign-key metadata is missing from schema queries

• Severity: Medium
• Description: The foreign key rejects a missing parent as expected, but the metadata query shows no referenced column for the constraint.
• Impact: Tools that inspect database relationships through the standard metadata views cannot see foreign-key links. Foreign-key checks still protect data, but schema inspection and tools that depend on this metadata may give incomplete results.
• Steps to Reproduce:
  1. Create schema myschema and a table myschema.t with a primary key and a parent_id column referencing myschema.t(id).
  2. Insert a parent row, a row with a NULL parent_id, and a valid child row; confirm that an invalid child pointing to id 999 is rejected.
  3. Query information_schema.table_constraints and information_schema.constraint_column_usage for myschema.
  4. Observe that table_constraints lists t_parent_id_fkey as a FOREIGN KEY, while constraint_column_usage returns zero rows instead of the referenced column metadata.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: constraintColumnUsageRowIter in server/tables/information_schema/constraint_column_usage_view.go:48-71 builds the view by calling functions.IterateCurrentDatabase with a callbacks object that contains only Check at lines 51-65. That callback appends rows with column_name, constraint_catalog, and constraint_schema set to nil at lines 59-61, and there is no ForeignKey callback at all. Consequently, the iterator cannot produce any row for any foreign key, including myschema.t. The foreign key does exist in the internal catalog path: server/tables/pgcatalog/pg_constraint.go:438-484 registers a ForeignKey callback and materializes a pgConstraint with conType "f", the table OID, referenced table OID, key columns, and referenced key columns. The creation-side qualification change in server/analyzer/generate_fk_name.go:56-62 only normalizes self-referential table and parent schema names; it does not populate constraint_column_usage and is not the source of the missing metadata. The smallest practical fix is to add a ForeignKey callback to constraintColumnUsageRowIter that emits one row per referenced parent column with the table and constraint schema/name fields populated, while retaining the existing check-constraint handling.

Evidence Package

• text/FOREIGN-1/FOREIGN-1-sql-and-code-evidence.txt — 2 KB

Tip

Reply with @itoqa to send us feedback on this test run.

[zachmu]

This seems fine overall except for a couple weird interface things and behaviors I've called out.

My main comment here is that there doesn't seem to be any thought to how this is going to integrate with dolt's existing vector index support, which we absolutely need to figure out before we lock in the storage format. Dolt's current implementation operates on JSON objects, I think more out of convenience than any real principled engineering decision. But before we release this we need to decide how these two halves of the product are going to work together. This means agreeing on a serialization format / encoding, either the current JSON one or a new set of ones supported natively on the Dolt side as well that map to []float and potentially a sparse encoding. You and @nicktobey should sort this out and come up with a plan. I don't think this makes sense to release until we figure out those elements of the plan that are difficult to change later, like encoding.

[coffeegoddd]

@Hydrocharged DOLT

read_tests	from_latency	to_latency	percent_change
covering_index_scan_postgres	2.43	2.43	0.0
groupby_scan_postgres	75.82	74.46	-1.79
index_join_postgres	2.22	2.22	0.0
index_join_scan_postgres	1.58	1.58	0.0
index_scan_postgres	493.24	493.24	0.0
oltp_point_select	0.37	0.37	0.0
oltp_read_only	6.32	6.55	3.64
select_random_points	0.7	0.72	2.86
select_random_ranges	1.03	1.03	0.0
table_scan_postgres	484.44	493.24	1.82
types_table_scan_postgres	1235.62	1235.62	0.0

write_tests	from_latency	to_latency	percent_change
oltp_delete_insert_postgres	6.67	6.67	0.0
oltp_insert	3.36	3.36	0.0
oltp_read_write	13.46	13.46	0.0
oltp_update_index	3.55	3.55	0.0
oltp_update_non_index	3.25	3.25	0.0
oltp_write_only	7.04	7.04	0.0
types_delete_insert_postgres	7.17	7.17	0.0

[itoqa]

History reset (rebase or force-push detected). Starting test narrative over.

Commit: 9eda870: 18 test cases ran, 1 failed ❌, 16 passed ✅, 1 additional finding ⚠️.

Summary

The run covers end-to-end vector functionality, including installation, storage, conversions, arithmetic, distance calculations, sparse and dense values, indexing, nearest-neighbor queries, transaction recovery, and persistence. It also exercises malformed and boundary inputs, invalid definitions, cleanup behavior, and schema/name-resolution edge cases, with broad happy-path and adversarial coverage.

Merge with caution — an attributable medium-severity regression affects how unqualified types are selected when multiple schemas define the same name, causing valid queries to fail despite search-path ordering. An unrelated medium-severity extension-cleanup limitation remains a flag for later rather than a PR merge blocker.

Tests run by Ito

Result	Severity	Type	Description
❌		Search	A cast to the unqualified type label failed with an error saying that label is ambiguous between search_d.label and search_c.label, even though search_path had a defined order.
✅	—	General	Indexes at the supported vector and half-vector dimensions were created successfully. The first oversized definitions were rejected with clear errors, left no invalid index records, and valid nearest-neighbor queries still returned the closest row.
✅	—	General	Vector indexes were created successfully with both supported methods, returned the expected nearest rows, and showed the correct method and operator class in the index catalog.
✅	—	General	Verified acceptable by independent adversarial review: the reported expectation does not match what the code actually promises. Review notes: The source contradicts the finding's core selection premise: unqualified length is an engine built-in, and even the external provider checks pg_catalog before considering application schemas. The cited root-object resolver is a persistence/name-resolution facility rather than the SQL function-dispatch path. The PR-causation claim of False is supported because the PR diff shows pg_catalog-first sel…
✅	—	Dense	The extension accepts vector and halfvec values, stores and converts them, and returns the expected results for arithmetic, distances, normalization, subvectors, averages, and nearest-neighbor ordering.
✅	—	Dense	Empty, malformed, non-finite, overflowing, wrong-size, and out-of-range vector values were rejected. The probe table stayed empty, so no invalid value was stored.
✅	—	Extension	The vector extension installed at version 0.8.6. The expected vector types, catalog entries, and callable values were available.
✅	—	Foreignkey	A schema-qualified table accepted a root row and a valid child row, rejected a child with a missing parent, and kept the valid rows unchanged.
✅	—	Index	Verified acceptable by independent adversarial review: the observed behavior is intended and documented in this codebase. Review notes: The SQL is reachable and the reported error is statically confirmed, but the finding incorrectly elevates an explicit, repo-wide unsupported SQL capability into a defect of the pgvector work. The implementation deliberately rejects index renames, and the PR's own vector-index contract covers creation, querying, and DML maintenance without claiming rename support. The row's introduced_by_this_pr=Fa…
✅	—	Resolution	A qualified procedure name removes only the matching procedure, while a same-name procedure found through the search path remains. Two equally exact procedures still produce an ambiguity error.
✅	—	Rev	Distance operators returned NULL for NULL inputs, rejected mismatched dimensions cleanly, preserved the documented zero-vector cosine result, and returned the same nearest-neighbor order with and without the index.
✅	—	Rev	Invalid vector and halfvec array casts returned clear errors, and the same connection continued to handle valid casts, saving data, reading it back, and running a later query.
✅	—	Rev	An invalid HNSW operator class was rejected without leaving a partial index behind. A valid index was then created successfully, and nearest-neighbor results returned in the expected order.
✅	—	Rev	Verified acceptable by independent adversarial review: the reported expectation does not match what the code actually promises. Review notes: The finding's source-level explanation is false: setting transactionState in handleQueryOutsideEngine does not mean the statement is handled there. For idle BEGIN and ordinary ROLLBACK the switch falls through with handled=false, after which the caller invokes h.query and ComQuery on the original transaction-control statement. Thus the cited code does establish engine-side BEGIN and ROLLBACK, and …
✅	—	Routine	Stored vector values can be read back, compared, sorted, and used with distance and normalization functions.
✅	—	Sparse	Sparse vector values stayed sorted and unchanged while moving through text, binary, dense, and half-precision formats. Distance, inner-product, cosine, norm, and normalization results matched the expected values.
✅	—	Sparse	The database rejected malformed sparse values, invalid numbers, bad dimensions, out-of-range indexes, duplicate indexes, and more than 16,000 non-zero entries. A valid value was stored and read back, while invalid rows were not saved.
⚠️		Rev	The required cleanup command returns an error instead of removing the extension and its dependent objects. Because cleanup stops at this step, the extension cannot be recreated and checked in the same lifecycle.

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Extension cannot be dropped for a clean reinstall

• Severity: Medium
• Description: The required cleanup command returns an error instead of removing the extension and its dependent objects. Because cleanup stops at this step, the extension cannot be recreated and checked in the same lifecycle.
• Impact: Database operators cannot remove the vector extension or clean up objects that use it. This can block extension upgrades or rollback and leave the database in an unwanted state.
• Steps to Reproduce:
  1. Connect to a local Doltgres database.
  2. Run CREATE EXTENSION vector.
  3. Create a table with a vector column, insert vector values, create a vector function, and create an HNSW index.
  4. Run DROP EXTENSION vector CASCADE.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: The SQL parser and AST create a DropExtension node with Names, IfExists, and Cascade fields, and server/node/drop_extension.go declares that node as an executable source relation. However, DropExtension.RowIter at lines 60-64 contains only a TODO and immediately returns errors.Errorf("DROP EXTENSION is not yet implemented"). It never resolves the requested extension, removes extension-owned types, routines, operators, casts, aggregates, access methods, or catalog entries, and it ignores both CASCADE and IF EXISTS. The observed local run created vector version 0.8.6, a dependent table, function, and HNSW index, then received the exact hard-coded error at DROP EXTENSION vector CASCADE. The smallest practical fix is to implement RowIter's drop path for the requested extension, including the dependency behavior required by CASCADE, and update the relevant catalog and registry state; add focused tests for a dependent vector schema, post-drop catalog cleanup, IF EXISTS, and reinstall. Since server/node/drop_extension.go is not present in the supplied PR file diff, this report marks PR introduction as False.

Evidence Package

• text/REV-4/REV-4-extension-drop-lifecycle.txt — 2 KB

Tip

Reply with @itoqa to send us feedback on this test run.

[itoqa]

History reset (rebase or force-push detected). Starting test narrative over.

Commit: cb73f26: 14 test cases ran, 13 passed ✅, 1 additional finding ⚠️.

Summary

Coverage spans core vector storage and arithmetic, type conversions, indexing and nearest-neighbor ordering, catalog and schema behavior, compatibility with shared SQL features, and safe handling of malformed or adversarial values. Overall, the exercised product behavior is healthy across normal workflows, edge cases, invalid input, and database integration paths.

Safe to merge — the sole finding is an unrelated medium-severity metadata issue affecting declared dimensions for certain vector types, with no regression or new failure attributable to this PR. It is a flag for later rather than a merge blocker.

Tests run by Ito

Result	Severity	Type	Description
✅	—	General	An invalid index request was rejected cleanly, and creating the corrected index with the same name succeeded. The resulting index returned the expected nearest rows in order.
✅	—	General	The database listed both vector index methods, accepted an index using each method, and returned the same nearest-neighbor order as the exact scan.
✅	—	Compare	Vector values can be compared, sorted, updated, stored, and read back successfully after the extension is installed.
✅	—	Compatibility	The database created and used a table that points back to itself. Array comparisons returned the right true and false results, and floating-point values displayed normal and infinite values correctly.
✅	—	Extension	The vector extension installed successfully. Its types, functions, operators, casts, aggregates, index methods, and operator classes were available and usable in SQL.
✅	—	Index	The vector extension was installed, a vector table was filled, and the index was created successfully. The nearest-neighbor query returned rows in ascending distance order.
✅	—	Modifier	Tables accepted vector(3), halfvec(3), sparsevec(3), and arrays of each type. Matching values kept their three dimensions and values when read back, while a two-dimensional vector was rejected as expected.
✅	—	Resolution	The directly named table was selected even though another table with the same name appeared first in the search path.
✅	—	Resolution	The vector extension was found by its name alone. Adding a schema produced no match, and the qualified drop command was rejected without removing the loaded extension.
✅	—	Sparsevec	Halfvec values came back with the expected half-precision rounding, and sparsevec values kept dimension 3 and their non-zero indexes. Conversion, equality, distance, and normalization checks also returned the expected results.
✅	—	Sparsevec	Invalid sparse vector dimensions, indexes, duplicates, and non-finite values were rejected. A valid value was stored in canonical order, and the SQL session remained usable afterward.
✅	—	Vector	The database stored both three-dimensional vectors, returned their original values, and calculated the expected distance and sum.
✅	—	Vector	Empty, malformed, non-finite, and out-of-range vector values are rejected, while a valid vector can still be inserted afterward.
⚠️		General	The round-trip values, equality checks, distances, and normalization worked. The metadata check failed because pg_catalog.pg_attribute returned atttypmod -1 for both declared halfvec(3) and sparsevec(3) columns.

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Declared vector dimensions are lost

• Severity: Medium
• Description: The round-trip values, equality checks, distances, and normalization worked. The metadata check failed because pg_catalog.pg_attribute returned atttypmod -1 for both declared halfvec(3) and sparsevec(3) columns.
• Impact: Clients that inspect database metadata may see declared halfvec and sparsevec dimensions as missing. Vector values still round-trip correctly, but schema-aware tools and clients may not reconstruct these types correctly.
• Steps to Reproduce:
  1. Create a table with columns declared as halfvec(3) and sparsevec(3).
  2. Query pg_catalog.pg_attribute for those columns and read atttypmod.
  3. Check the result: both columns return -1 instead of metadata for dimension 3.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: The PR's new type-resolution path is capable of producing the required modifier: server/analyzer/resolve_type.go calls TypeCollection.GetTypeWithTypmod with typ.UnresolvedTypmods, and core/typecollection/typecollection.go passes those modifiers through typeWithTypmod. DoltgresType stores the result in attTypMod, exposes it through GetAttTypMod in server/types/type.go:726-729, and server/doltgres_handler.go:526-555 uses that value as the PostgreSQL wire field TypeModifier. However, server/tables/pgcatalog/pg_attribute.go:120-130 constructs each pgAttribute without an attTypMod field. The struct at lines 409-419 has no place to store it, and pgAttributeToRow at lines 461-471 unconditionally emits int32(-1) for atttypmod. Therefore the declared modifier is lost at the catalog boundary even though type resolution and the wire-field code support it. The smallest practical fix is to add the resolved column typmod to pgAttribute while caching table and view attributes, then emit that stored value from pgAttributeToRow instead of the unconditional -1; the default should remain -1 only when a column has no modifier.

Evidence Package

• text/BF-INTEGRATION-1/BF-INTEGRATION-1-typmod-evidence.txt — 5 KB

Tip

Reply with @itoqa to send us feedback on this test run.