x/platform
Watch
1
0
Fork
You've already forked platform
 0
Code Issues Pull requests Projects Releases 24 Packages Wiki Activity Actions
24 releases 24 tags
  • v0.9.0 c89afbd3b8

    v0.9.0 Stable

    x released this 2026-10-17 17:27:28 +00:00 | 7 commits to main since this release

    VSKI v0.9.0

    Embeddings: model tags, vector fetch, and a fixed vec0 layout

    The embeddings API grows the primitives downstream ML consumers need:
    provenance (which embedding model produced these vectors) and export
    (the actual vectors, not just distances).

    Model tags

    Upsert accepts an optional model form field. The first tagged write pins
    it for the collection in the new _vec_metadata system table (tenant db,
    next to the <collection>_vec table — deliberately not collection
    options, which dashboard PATCHes can rewrite). Later upserts declaring a
    different model are rejected with 409; search responses and the new
    info endpoint echo the pinned tag, so a consumer can refuse embeddings from
    a model it wasn't built for:

    curl -F recordId=$RID -F model=llama-backbone-q8 -F vector=@v.f32 \
  localhost:3000/api/collections/notes/embeddings

    New endpoints

    • GET /api/collections/:name/embeddings — {collection, model, dimensions, metric, count}; a collection with no vectors yet reports
      nulls and count: 0.
    • GET /api/collections/:name/embeddings/:recordId — the exact stored
      vector as a JSON float array (float32 values, lossless round-trip).
    • POST /api/collections/:name/embeddings/fetch — {"ids": [...]} →
      {"vectors": [{recordId, vector}], "missing": [...]}, indexed point
      lookups (max 1000 ids).

    Search additionally validates the query dimension against the pinned one
    (clear 400 instead of a deep vec0 error) and can be combined with
    vec_to_json/hex through the admin SQL console as before.

    The vec0 layout fix (migration required)

    The old virtual-table layout (rowid INTEGER PRIMARY KEY with record_id
    as an unindexed metadata column) had two problems: point lookups by
    record_id were full scans, and INSERT OR REPLACE never conflicted —
    every re-upsert of the same record appended a duplicate row that then
    duplicated search hits.

    The canonical layout is now record_id TEXT PRIMARY KEY, which indexes
    record_id in the shadow rowids table (true point queries) and makes
    re-upserts replace. vec0 tables cannot be ALTERed or renamed, so
    BootstrapDB rebuilds legacy tables transactionally on first boot after
    the upgrade — copying vectors through a temp table, deduplicating with
    last-write-wins — and seeds _vec_metadata (model: null, dimensions
    parsed from the old DDL, metric: "l2").

    Migration notes

    • The rebuild copies every vector once; on multi-GB collections expect a
      one-time boot proportional to table size. It runs inside a single
      transaction per table and is idempotent.
    • Duplicate rows accumulated by old re-upserts are collapsed to the
      latest vector per record_id.
    • Distance semantics are unchanged and now documented: sqlite-vec's
      default euclidean L2 over unnormalized float32 vectors, ascending (the vendored sqlite-vec l2 kernel returns the sqrt — plain distance, not the square).
    • make test-vec runs the vec0-dependent Go tests (plain go test
      compiles without the sqlite_vec extension).

    Compatibility

    API is additive (one new optional upsert field, three new endpoints, two
    new response fields). The TS client and the Python SDK gain matching
    methods. Existing clients see one behavior change: re-upserting a record
    no longer duplicates it in search results — that was the bug.

    The vendored sqlite driver fork also gained an unconditional
    #cgo LDFLAGS: -lm in sqlite3.go: cgo compiles every .c file in the
    package directory, so sqlite-vec.c (and its sqrt calls) links in every
    build, not just sqlite_vec-tagged ones — untagged go test binaries
    failed to link on toolchains where libm is not implicit.

    Downloads