NRL-1884 seed script generates perf test input files #1119

anjalitrace2-nhs · 2026-01-13T16:18:02Z

seed script will now generate the needed input files for perf tests on the fly and upload these to S3 (along with some metadata about when the file was generated e.g. commit hash). This enables the seed script to be run from within our new bastion & saves us the previous computationally expensive generation which scanned the entire pointer table 3 times
repurpose generate_pointer_table_extract function to be useful for refreshing these input file based on current table state. Ran this for baseline table.
perf tests prepare step will grab the input files from s3 and regenerate distribution info before a test run (cheap to do)
get rid of unused input files & steps
fix a bug in the seed script where the final batch of writes were never written (patient_counter <= patients_with_pointers rather than <)
renamed px > patient(s) for my brain
updated readme instructions to reflect updated make commands

Tested seed stuff works using a tiny test table anjali-test-2 by seeding ~20 pointers and checking files generated. Ran prepare step and ran (most - all successes 30 mins in) perf tests and it was all able to use the newly generated input files & download them from s3

…ady in git

…st input file

…ady in git

…st input file

…om:NHSDigital/NRLF into NRL-1884-seed-script-generates-input-files

… extract was taken. Zip & save to s3 per table name

…es for refreshing input files. Will not be used much but helpful in case something goes wrong producing these files

…y generated from prep steps

…make tests read newly downloaded file

… run & prepare

github-actions · 2026-01-13T16:23:57Z

🚀 PR environment successfully deployed.
Commit Hash: 16bda8d3603f0d7adc2374b8e36147c753182cfd
URL: https://nrl1884-01ba47.api.record-locator.dev.national.nhs.uk/

mattdean3-nhs

Love this 😍 it works a treat.

Only issue I hit while running was that the seed-pointers-extract.csv file wasn't there (see comment for client_perftest.js).

Makefile

mattdean3-nhs · 2026-01-14T07:47:14Z

tests/performance/producer/client_perftest.js

 const csvPath = __ENV.DIST_PATH
-  ? `../../../${__ENV.DIST_PATH}/producer_reference_data.csv`
-  : "../producer_reference_data.csv";
+  ? `../../../${__ENV.DIST_PATH}/seed-pointers-extract.csv`


should this be the .csv file in the ./dist/nft directory? the one that comes out of the .zip file? (same in consumer/client_perftest.js too)

this is that same file - it's copied out of the unzipped directory in the prepare step (mostly to remove the table name from the file name).

This is a bit confusing though! I'll skip the copy step and use the file inside the ./dist/nft instead 👍

github-actions · 2026-01-14T11:04:34Z

🚀 PR environment successfully deployed.
Commit Hash: 23cb0216ca2f990cbd8c42079cfc9966c97a9d5e
URL: https://nrl1884-01ba47.api.record-locator.dev.national.nhs.uk/

…n relying on prepare step to copy around. Include table name in info file rather than input file name

sonarqubecloud · 2026-01-14T12:14:02Z

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-01-14T12:20:47Z

🚀 PR environment successfully deployed.
Commit Hash: 5cd19394e3f5d2a5f358e343f7718018c1ad933f
URL: https://nrl1884-01ba47.api.record-locator.dev.national.nhs.uk/

anjalitrace2-nhs added 15 commits January 9, 2026 15:21

NRL-1875 Rename px -> patient(s)

a609e7e

NRL-1875 Create bucket and write perftest input files

9501e0e

NRL-1875 Consumer reference data unused. Producer reference json alre…

be5c890

…ady in git

NRL-1884 Remove new S3 bucket and use metadata bucket to store perfte…

dc3fe05

…st input file

NRL-1884 Rename px -> patient(s)

0486111

NRL-1884 Create bucket and write perftest input files

2fa2f9b

NRL-1884 Consumer reference data unused. Producer reference json alre…

e053ddd

…ady in git

NRL-1884 Remove new S3 bucket and use metadata bucket to store perfte…

717f652

…st input file

Merge branch 'NRL-1884-seed-script-generates-input-files' of github.c…

213dd0f

…om:NHSDigital/NRLF into NRL-1884-seed-script-generates-input-files

NRL-1884 Create info.json file containing metadata about when pointer…

cb7ddb2

… extract was taken. Zip & save to s3 per table name

NRL-1884 Fixes from testing the seed script with upload on my test table

349e6a9

NRL-1884 Repurpose existing prepare command which generates input fil…

113cdf1

…es for refreshing input files. Will not be used much but helpful in case something goes wrong producing these files

NRL-1884 Align file generated from seed table script to one previousl…

58fc594

…y generated from prep steps

NRL-1884 create prepare command to arrange perf test input files and …

7c3f5cd

…make tests read newly downloaded file

NRL-1884 fix import issues with scripts in make commands. Tested test…

16bda8d

… run & prepare

anjalitrace2-nhs temporarily deployed to pull-request January 13, 2026 16:18 — with GitHub Actions Inactive

anjalitrace2-nhs temporarily deployed to pull-request January 13, 2026 16:21 — with GitHub Actions Inactive

anjalitrace2-nhs temporarily deployed to pull-request January 13, 2026 16:24 — with GitHub Actions Inactive

anjalitrace2-nhs changed the title ~~NRL-1884 seed script generates input files~~ NRL-1884 seed script generates perf test input files Jan 13, 2026

anjalitrace2-nhs temporarily deployed to pull-request January 13, 2026 16:26 — with GitHub Actions Inactive

anjalitrace2-nhs marked this pull request as ready for review January 13, 2026 16:26

anjalitrace2-nhs requested a review from a team as a code owner January 13, 2026 16:26

mattdean3-nhs reviewed Jan 14, 2026

View reviewed changes

anjalitrace2-nhs temporarily deployed to pull-request January 14, 2026 10:57 — with GitHub Actions Inactive

anjalitrace2-nhs temporarily deployed to pull-request January 14, 2026 11:01 — with GitHub Actions Inactive

anjalitrace2-nhs temporarily deployed to pull-request January 14, 2026 11:04 — with GitHub Actions Inactive

anjalitrace2-nhs temporarily deployed to pull-request January 14, 2026 11:06 — with GitHub Actions Inactive

NRL-1884 Fix tests to use default location for input files rather tha…

5cd1939

…n relying on prepare step to copy around. Include table name in info file rather than input file name

anjalitrace2-nhs force-pushed the NRL-1884-seed-script-generates-input-files branch from 23cb021 to 5cd1939 Compare January 14, 2026 12:13

anjalitrace2-nhs temporarily deployed to pull-request January 14, 2026 12:13 — with GitHub Actions Inactive

anjalitrace2-nhs temporarily deployed to pull-request January 14, 2026 12:17 — with GitHub Actions Inactive

anjalitrace2-nhs temporarily deployed to pull-request January 14, 2026 12:20 — with GitHub Actions Inactive

anjalitrace2-nhs temporarily deployed to pull-request January 14, 2026 12:22 — with GitHub Actions Inactive

anjalitrace2-nhs requested a review from mattdean3-nhs January 14, 2026 13:52

mattdean3-nhs approved these changes Jan 14, 2026

View reviewed changes

anjalitrace2-nhs merged commit 6d08175 into develop Jan 14, 2026
9 checks passed

anjalitrace2-nhs deleted the NRL-1884-seed-script-generates-input-files branch January 14, 2026 13:56

anjalitrace2-nhs deployed to pull-request January 14, 2026 13:57 — with GitHub Actions Active

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

NRL-1884 seed script generates perf test input files #1119

NRL-1884 seed script generates perf test input files #1119

Uh oh!

anjalitrace2-nhs commented Jan 13, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

mattdean3-nhs left a comment

Uh oh!

Uh oh!

mattdean3-nhs Jan 14, 2026 •

edited

Loading

Uh oh!

anjalitrace2-nhs Jan 14, 2026 •

edited

Loading

Uh oh!

anjalitrace2-nhs Jan 14, 2026

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

sonarqubecloud bot commented Jan 14, 2026

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NRL-1884 seed script generates perf test input files #1119

NRL-1884 seed script generates perf test input files #1119

Uh oh!

Conversation

anjalitrace2-nhs commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

mattdean3-nhs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattdean3-nhs Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anjalitrace2-nhs Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anjalitrace2-nhs Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

sonarqubecloud bot commented Jan 14, 2026

Quality Gate passed

Uh oh!

github-actions bot commented Jan 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

anjalitrace2-nhs commented Jan 13, 2026 •

edited

Loading

mattdean3-nhs Jan 14, 2026 •

edited

Loading

anjalitrace2-nhs Jan 14, 2026 •

edited

Loading