-
Notifications
You must be signed in to change notification settings - Fork 2
Materialized View Utilities for SQLAlchemy Models #402
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
b4721d2 to
eb0299b
Compare
jstone-dev
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This all made sense to me. My only question is about how we'll trigger materialized view refreshes. I see the refresh utilities, but do you envision adding scripts for refreshing MVs and triggering them using some scheduler?
|
@jstone-dev Yeah Nick had some examples in #343 of this sort of thing running on ARQ. We could either have a refresh job running on a schedule or we could call the refresh job as the last step of (for an MV like this one) publishing a score set. Maybe we'd even like to do both, just to make certain things are refreshed at least once per day. I'll add it now, it should be straightforward. |
eb0299b to
455de3d
Compare
Adds utilities for managing materialized and non-materialized views with SQLAlchemy. These ORM methods aren't perfect and will still require knowledge of the underlying view, but make accessing them easier and make their creation automated for test cases. Might also be useful for #89.
4b9a1b2 to
91fb506
Compare
Adds refresh jobs for existing materialized views and invokes them: - Refreshes the published variants materialized view whenever a new score set is published. (We cannot delete a published score set, nor edit it's variants, so publication time is the only moment this view becomes outdated). - Refreshes all materialized views at 0300. Adds tests for newly placed publication view refresh. Clarifies connection -> inspector flow during mat view refresh. Opens #405.
91fb506 to
d140d50
Compare
This change adds various utilities that make it easier to create and manage materialized views using our existing SQLAlchemy models.
Define a view like:
and use the view as you would any other SQLAlchemy model object. We should take care to manage database migrations with views, as they are more brittle than other Alembic migrations.
As a proof of concept, this change adds a
published_variants_materialized_viewthat significantly improves the performance of variant statistics endpoints.