Searchable json query api #257

tobyhede · 2026-01-16T02:41:19Z

No description provided.

…eries

Add new query encryption API for searchable encryption: - encryptQuery(): Single value query encryption with index type control - createQuerySearchTerms(): Bulk query encryption with mixed index types - createJsonSearchTerms(): JSON path and containment query encryption Features: - Support for all index types: ore, match, unique, ste_vec - Lock context support for all query operations - SEM-only payloads (no ciphertext) optimized for database queries - Path queries (dot notation and array format) - Containment queries (contains/contained_by) Test coverage includes: - Lock context integration tests - Boundary conditions (empty strings, Unicode, emoji, large numbers) - Deep JSON nesting (5+ levels) - Bulk operation edge cases - Error handling scenarios

changeset-bot · 2026-01-16T02:41:22Z

🦋 Changeset detected

Latest commit: e1ea208

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 10 packages

Name	Type
@cipherstash/protect	Major
@cipherstash/schema	Major
@cipherstash/drizzle	Major
@cipherstash/protect-dynamodb	Major
@cipherstash/basic-example	Patch
@cipherstash/dynamo-example	Patch
nest	Patch
next-drizzle-mysql	Patch
@cipherstash/nextjs-clerk-example	Patch
@cipherstash/typeorm-example	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Remove public API additions that diverged from requirements: - Requirements specified using existing createSearchTerms function - Requirements specified NOT changing the existing protectjs public API Removed: - encryptQuery(), createQuerySearchTerms(), createJsonSearchTerms() methods - Public type exports for query-specific types - Test files for removed public API Internal operation files remain for potential future use.

- Revert package.json from local link to published 0.19.0 - Define IndexTypeName and QueryOpName locally in types.ts - These types will be available from FFI once 0.20.0 is released

Add 32 tests covering JsonSearchTermsOperation including: - Path queries (string/array paths, deep paths, path-only) - Containment queries (simple/nested objects, multiple keys) - Bulk operations (mixed queries, multiple columns) - Lock context integration - Edge cases (unicode, deep nesting, special chars) - Error handling (missing ste_vec index) - Selector generation verification

Add missing public methods to ProtectClient: - encryptQuery: encrypt single value with explicit index type - createQuerySearchTerms: bulk query term encryption - createJsonSearchTerms: JSON path/containment query encryption Update tests to use public API instead of unsafe internal access. Export new operation types and search term types.

Updates README.md, schema reference, and searchable encryption guides to include details on the new JSON search capabilities (path and containment queries).

…operations Covers encryptQuery and createQuerySearchTerms with unique, ORE, and match indexes, as well as composite-literal return types and lock context integration.

SearchTerm is now a union of SimpleSearchTerm, JsonPathSearchTerm, and JsonContainmentSearchTerm, enabling createSearchTerms to accept all search term types in a single call. - Add SimpleSearchTerm type alias for original behavior - Update SearchTerm to union type - Export SimpleSearchTerm from public API

SearchTermsOperation.execute() now handles JSON search terms: - Partitions terms by type (simple, JSON path, JSON containment) - Encrypts simple terms with encryptBulk (original behavior) - Encrypts JSON terms with encryptQueryBulk (ste_vec index) - Reassembles results in original order - Supports mixed batches of simple and JSON terms Also includes: - Type guards for SearchTerm variants - Helper functions (pathToSelector, buildNestedObject, flattenJson) - withLockContext support for JSON terms - Extracted shared logic into encryptSearchTermsHelper to reduce duplication

Tests for: - JSON path search term via createSearchTerms - JSON containment search term via createSearchTerms - Mixed simple and JSON search terms in single call

@deprecated

Add @deprecated JSDoc tag to guide users toward createSearchTerms. Implementation unchanged to avoid breaking existing code.

Remove the deprecated createJsonSearchTerms function and supporting code, consolidating JSON search functionality into the unified createSearchTerms API. - Remove createJsonSearchTerms method from ProtectClient - Delete json-search-terms.ts operation file - Remove JsonSearchTermsOperation export from index - Migrate comprehensive tests to search-terms.test.ts - Update documentation examples to use createSearchTerms

Add missing lock context integration tests for JSON search terms and refactor test file to use shared beforeAll client for efficiency.

Remove __RESOLVE_AT_BUILD__ placeholder in favor of inferring the ste_vec prefix from table/column context when not explicitly set. Changes: - searchableJson() now sets empty ste_vec object - ProtectTable.build() and buildEncryptConfig() infer prefix when missing - Simplified error checks in search-terms.ts - Enabled previously commented test for ste_vec index

Add tests to prevent regressions based on code review feedback: - Selector prefix resolution test verifying table/column prefix - encryptQuery(null) null handling verification - escaped-composite-literal return type for createQuerySearchTerms - ste_vec index with default queryOp for JSON object encryption

Set temporary column name prefix in searchableJson() to satisfy type requirements, then always overwrite with full table/column prefix during build. Update search-terms.ts to always derive prefix from table/column names rather than relying on column.build() which may have incomplete prefix. This fixes the DTS build error where prefix was required by the type but not set until table build time.

packages/protect/src/types.ts

calvinbrewer

This looks great @tobyhede good stuff

packages/protect/src/types.ts

calvinbrewer · 2026-01-20T22:06:09Z

Note: you don't have a changeset in here - might be worth bumping the major version just to be safe to account for all the different typescript targets

docs/reference/schema.md

docs/reference/searchable-encryption-postgres.md

packages/protect/src/types.ts

…ching values - Rename IndexTypeName to QueryTypeName - Change values: ore → orderAndRange, match → freeTextSearch, unique → equality, ste_vec → searchableJson - Add queryTypes constant for convenient import - Update JSDoc examples to use new API - Add work files to .gitignore

coderdan · 2026-01-22T04:42:55Z

docs/reference/model-operations.md

 ```typescript
-const users = [
+const usersList = [
  {
    id: "1",
    email: "user1@example.com",
    address: "123 Main St",
  },
  {
    id: "2",
    email: "user2@example.com",
    address: "456 Oak Ave",
  },
 ];

-const encryptedResult = await protectClient.bulkEncryptModels(users, users);
+const encryptedResult = await protectClient.bulkEncryptModels(usersList, usersSchema);


Not a blocker for this PR but these reference docs won't end up on the main docs site. They will need to be in the typedoc.

coderdan · 2026-01-22T04:43:32Z

docs/reference/schema.md

+> [!WARNING]
+> `searchableJson()` is mutually exclusive with other index types (`equality()`, `freeTextSearch()`, `orderAndRange()`) on the same column. Combining them will result in runtime errors. This is enforced by the encryption backend, not at the TypeScript type level.


Is the plan to make this typescript enforceable in the future?

Yeah, it needs some thought.

docs/reference/searchable-encryption-postgres.md

packages/protect/__tests__/batch-encrypt-query.test.ts

packages/protect/__tests__/query-search-terms.test.ts

coderdan · 2026-01-22T05:52:30Z

packages/protect/src/types.ts

+
+/**
+ * Query operation type for ste_vec index.
+ * - 'default': Standard JSON query using column's cast_type


I suspect this might be a quirk of how the cipherstash-client is implemented leaking through. The default queryop doesn't mean anything for an ste_vec.

coderdan · 2026-01-22T06:04:13Z

packages/protect/src/ffi/operations/search-terms.ts

+            // Add lock context if provided
+            if (lockContextData) {
+              return { ...plaintext, lockContext: lockContextData.context }
+            }


LockContext isn't relevant to generation search terms.

coderdan · 2026-01-22T06:05:00Z

packages/protect/src/ffi/operations/search-terms.ts

+  client: Client,
+  terms: SearchTerm[],
+  metadata: Record<string, unknown> | undefined,
+  lockContextData: { context: Context; ctsToken: CtsToken } | undefined,


lockContext isn't used for search.

coderdan · 2026-01-22T06:05:41Z

packages/protect/src/ffi/operations/query-search-terms.ts

+  public withLockContext(
+    lockContext: LockContext,
+  ): QuerySearchTermsOperationWithLockContext {
+    return new QuerySearchTermsOperationWithLockContext(this, lockContext)
+  }


This can be removed. LockContext isn't applicable to query generation.

coderdan · 2026-01-22T06:06:23Z

packages/protect/src/ffi/operations/search-terms.ts

+  public withLockContext(
+    lockContext: LockContext,
+  ): SearchTermsOperationWithLockContext {
+    return new SearchTermsOperationWithLockContext(this, lockContext)
+  }


coderdan · 2026-01-22T06:06:47Z

packages/protect/src/ffi/operations/query-search-terms.ts

+  }
+}
+
+export class QuerySearchTermsOperationWithLockContext extends ProtectOperation<


Again, this can be removed.

coderdan · 2026-01-22T06:09:52Z

packages/protect/src/ffi/operations/encrypt-query.ts

+  public withLockContext(
+    lockContext: LockContext,
+  ): EncryptQueryOperationWithLockContext {
+    return new EncryptQueryOperationWithLockContext(this, lockContext)
+  }


Not applicable to queries.

coderdan · 2026-01-22T06:10:08Z

packages/protect/src/ffi/operations/encrypt-query.ts

+  }
+}
+
+export class EncryptQueryOperationWithLockContext extends ProtectOperation<Encrypted> {


Not applicable to queries.

coderdan · 2026-01-22T06:10:29Z

packages/protect/src/ffi/operations/batch-encrypt-query.ts

+  public withLockContext(
+    lockContext: LockContext,
+  ): BatchEncryptQueryOperationWithLockContext {
+    return new BatchEncryptQueryOperationWithLockContext(this, lockContext)
+  }


Not applicable to queries.

coderdan · 2026-01-22T06:21:05Z

packages/protect/src/ffi/operations/batch-encrypt-query.ts

+      }
+
+      const prefix = `${term.table.tableName}/${term.column.getName()}`
+      const pairs = flattenJson(term.contains, prefix)


I might be missing something but I don't understand why we are flattening the JSON here. The json should be treated as opaque at this level and all the processing should be done by the rust json indexer in cipherstash-client.

coderdan · 2026-01-22T06:21:20Z

packages/protect/src/ffi/operations/batch-encrypt-query.ts

+      }
+
+      const prefix = `${term.table.tableName}/${term.column.getName()}`
+      const pairs = flattenJson(term.containedBy, prefix)


coderdan · 2026-01-22T06:22:48Z

packages/protect/src/ffi/operations/batch-encrypt-query.ts

+          : term.path.split('.')
+        const wrappedValue = buildNestedObject(pathArray, term.value)
+        jsonItemsWithIndex.push({
+          selector: pathToSelector(term.path, prefix),


Same comments as above. Just leave the heavy lifting to cs client.

- Encrypt ste_vec selectors using the 'ste_vec_selector' operation. - Normalize JSON paths to standard '$' prefixed JSONPath strings for FFI compatibility. - Remove 'withLockContext' from all search-related operations (SearchTermsOperation, QuerySearchTermsOperation, EncryptQueryOperation, BatchEncryptQueryOperation) as it is not applicable to queries. - Update test suite to expect hex string tokens for selectors and remove now-unsupported LockContext tests. - Address review feedback regarding naming conventions and result construction.

- Move schema documentation to packages/schema/src/index.ts. - Move configuration and initialization documentation to packages/protect/src/index.ts. - Move model operation documentation to packages/protect/src/ffi/index.ts. - Move searchable encryption and PostgreSQL integration documentation to packages/protect/src/types.ts. - Move Supabase and composite type helper documentation to packages/protect/src/helpers/index.ts. - Add integration tips such as the ::jsonb cast requirement for Supabase/PostgreSQL. - Wired in shared test helpers into batch-encrypt-query.test.ts and search-terms.test.ts.

tobyhede added 3 commits January 15, 2026 16:02

feat(schema): enable searchableJson() method for SteVec indexing

e941688

feat(protect): add JSON search term types for containment and path qu…

6a9208c

…eries

tobyhede added 14 commits January 16, 2026 13:57

chore: update protect-ffi to 0.20.0

a112a44

fix: use local type definitions until protect-ffi 0.20.0 release

bedda29

- Revert package.json from local link to published 0.19.0 - Define IndexTypeName and QueryOpName locally in types.ts - These types will be available from FFI once 0.20.0 is released

chore: update protect-ffi to 0.20.0

b0c00d2

docs: add documentation for searchable encrypted JSON

b660611

Updates README.md, schema reference, and searchable encryption guides to include details on the new JSON search capabilities (path and containment queries).

test(protect): add comprehensive tests for explicit query encryption …

f5e4793

…operations Covers encryptQuery and createQuerySearchTerms with unique, ORE, and match indexes, as well as composite-literal return types and lock context integration.

test(protect): add JSON support tests for createSearchTerms

6be18b4

Tests for: - JSON path search term via createSearchTerms - JSON containment search term via createSearchTerms - Mixed simple and JSON search terms in single call

deprecate(protect): mark createJsonSearchTerms as deprecated

2d7a40d

Add @deprecated JSDoc tag to guide users toward createSearchTerms. Implementation unchanged to avoid breaking existing code.

test(protect): add lock context tests and optimize client initialization

3f9aed2

Add missing lock context integration tests for JSON search terms and refactor test file to use shared beforeAll client for efficiency.

tobyhede force-pushed the searchable-json-query-api branch from 66227cd to c4f5d8c Compare January 20, 2026 04:29

tobyhede added 3 commits January 20, 2026 15:46

tobyhede commented Jan 20, 2026

View reviewed changes

packages/protect/src/types.ts Show resolved Hide resolved

calvinbrewer approved these changes Jan 20, 2026

View reviewed changes

packages/protect/src/types.ts Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/schema.md Outdated Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/schema.md Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/searchable-encryption-postgres.md Outdated Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/searchable-encryption-postgres.md Show resolved Hide resolved

coderdan reviewed Jan 20, 2026

View reviewed changes

docs/reference/searchable-encryption-postgres.md Outdated Show resolved Hide resolved

chore: merge

c6bc59b

calvinbrewer reviewed Jan 21, 2026

View reviewed changes

packages/protect/src/types.ts Outdated Show resolved Hide resolved

packages/protect/src/types.ts Outdated Show resolved Hide resolved

tobyhede force-pushed the searchable-json-query-api branch from b79c96d to 50d5f27 Compare January 22, 2026 00:18

tobyhede requested a review from calvinbrewer January 22, 2026 00:20