Skip to content

Conversation

@bencap
Copy link
Collaborator

@bencap bencap commented Mar 14, 2025

This change adds various utilities that make it easier to create and manage materialized views using our existing SQLAlchemy models.

Define a view like:

    class MyView(Base):
        __table__ = view(
            "my_view",
            select(
                MyModel.id.label("id"),
                MyModel.name.label("name"),
            ),
            materialized=False,
        )

    id = __table__.c.id
    name = __table__.c.name

and use the view as you would any other SQLAlchemy model object. We should take care to manage database migrations with views, as they are more brittle than other Alembic migrations.

As a proof of concept, this change adds a published_variants_materialized_view that significantly improves the performance of variant statistics endpoints.

@bencap bencap marked this pull request as ready for review March 14, 2025 18:20
@bencap bencap requested review from jstone-dev and sallybg March 14, 2025 18:20
@bencap bencap linked an issue Mar 14, 2025 that may be closed by this pull request
@bencap bencap requested a review from nickzoic March 14, 2025 18:27
@bencap bencap force-pushed the feature/bencap/sa-mat-views branch from b4721d2 to eb0299b Compare March 18, 2025 03:10
Copy link
Collaborator

@jstone-dev jstone-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all made sense to me. My only question is about how we'll trigger materialized view refreshes. I see the refresh utilities, but do you envision adding scripts for refreshing MVs and triggering them using some scheduler?

@bencap
Copy link
Collaborator Author

bencap commented Mar 18, 2025

@jstone-dev Yeah Nick had some examples in #343 of this sort of thing running on ARQ. We could either have a refresh job running on a schedule or we could call the refresh job as the last step of (for an MV like this one) publishing a score set. Maybe we'd even like to do both, just to make certain things are refreshed at least once per day.

I'll add it now, it should be straightforward.

@bencap bencap force-pushed the feature/bencap/sa-mat-views branch from eb0299b to 455de3d Compare March 18, 2025 22:12
bencap added 5 commits March 18, 2025 15:19
Adds utilities for managing materialized and non-materialized views with SQLAlchemy. These ORM methods aren't perfect and will still require knowledge of the underlying view, but
make accessing them easier and make their creation automated for test cases. Might also be useful for #89.
@bencap bencap force-pushed the feature/bencap/sa-mat-views branch 2 times, most recently from 4b9a1b2 to 91fb506 Compare March 18, 2025 23:10
bencap added 2 commits March 18, 2025 16:30
Adds refresh jobs for existing materialized views and invokes them:
- Refreshes the published variants materialized view whenever a new score set
is published. (We cannot delete a published score set, nor edit it's variants,
so publication time is the only moment this view becomes outdated).
- Refreshes all materialized views at 0300.

Adds tests for newly placed publication view refresh.

Clarifies connection -> inspector flow during mat view refresh.

Opens #405.
@bencap bencap force-pushed the feature/bencap/sa-mat-views branch from 91fb506 to d140d50 Compare March 18, 2025 23:30
@bencap bencap mentioned this pull request Mar 18, 2025
@bencap bencap merged commit 5bbd315 into release-2025.1.1 Mar 18, 2025
5 checks passed
@bencap bencap deleted the feature/bencap/sa-mat-views branch March 18, 2025 23:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Materialized Views in SQLAlchemy

3 participants