Skip to content

lore: fix date filtering being applied after FTS limit#20

Open
theihor wants to merge 1 commit intofacebookexperimental:mainfrom
theihor:limit-filter-fix
Open

lore: fix date filtering being applied after FTS limit#20
theihor wants to merge 1 commit intofacebookexperimental:mainfrom
theihor:limit-filter-fix

Conversation

@theihor
Copy link

@theihor theihor commented Feb 18, 2026

The lore search with --since/--until and --limit produced incorrect results: e.g. --limit 20 returned 0 results while --limit 30 returned 30 results. The root cause was that date filtering happened in Rust after the FTS query already applied its limit, so the FTS returned top-N candidates by text relevance (regardless of date), then date filtering removed most or all of them.

Add a date_timestamp (Unix epoch) field to LoreEmailInfo and the lore table schema, populated at indexing time from the RFC 2822 date header. Use this field to push date filtering into the database via .only_if() on FTS queries and SQL WHERE clauses on non-FTS queries, so the limit is applied to already-date-filtered results.

Specific changes:

  • types.rs: add date_timestamp: i64 to LoreEmailInfo
  • schema.rs: add date_timestamp Int64 column to lore table schema
  • indexer.rs: add parse_rfc2822_to_timestamp() helper, populate date_timestamp when parsing emails
  • connection.rs: rewrite search_lore_emails() to combine FTS with .only_if(date_timestamp filter) so limit applies post-date-filter; fix adaptive expansion loop early-exit when first iteration yields zero results; rewrite search_lore_emails_multi_field() to use new fetch_lore_emails_by_message_ids_with_filter() with SQL date filter; rewrite search_lore_emails_by_subject() to include date_timestamp in WHERE clause; update all lore read sites for the new column
  • search.rs: fix column index offsets in update_lore_vectors() and fetch_emails_by_ids() to account for the new date_timestamp column at index 3

Assisted-by: Claude Code (claude-opus-4-6)

The lore search with --since/--until and --limit produced incorrect
results: e.g. --limit 20 returned 0 results while --limit 30 returned
30 results. The root cause was that date filtering happened in Rust
after the FTS query already applied its limit, so the FTS returned
top-N candidates by text relevance (regardless of date), then date
filtering removed most or all of them.

Add a date_timestamp (Unix epoch) field to LoreEmailInfo and the lore
table schema, populated at indexing time from the RFC 2822 date header.
Use this field to push date filtering into the database via .only_if()
on FTS queries and SQL WHERE clauses on non-FTS queries, so the limit
is applied to already-date-filtered results.

Specific changes:
- types.rs: add date_timestamp: i64 to LoreEmailInfo
- schema.rs: add date_timestamp Int64 column to lore table schema
- indexer.rs: add parse_rfc2822_to_timestamp() helper, populate
  date_timestamp when parsing emails
- connection.rs: rewrite search_lore_emails() to combine FTS with
  .only_if(date_timestamp filter) so limit applies post-date-filter;
  fix adaptive expansion loop early-exit when first iteration yields
  zero results; rewrite search_lore_emails_multi_field() to use new
  fetch_lore_emails_by_message_ids_with_filter() with SQL date filter;
  rewrite search_lore_emails_by_subject() to include date_timestamp in
  WHERE clause; update all lore read sites for the new column
- search.rs: fix column index offsets in update_lore_vectors() and
  fetch_emails_by_ids() to account for the new date_timestamp column
  at index 3

Assisted-by: Claude Code (claude-opus-4-6)
Signed-off-by: Ihor Solodrai <ihor.solodrai@linux.dev>
@meta-cla meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Feb 18, 2026
@theihor
Copy link
Author

theihor commented Feb 18, 2026

@masoncl the change was vibe-coded (I don't really know Rust, but I have some idea how one should query a database, lol).

I verified locally that the bug is fixed: the filter+limit queries now work as expected, and they got pretty snappy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments