Run One Surveillance Detection Cycle

This is the function you schedule to run regularly (e.g. daily, via cron): it pulls in new laboratory data, checks every monitored stream for statistical aberrations with the configured detectors, reconciles the results into cluster dossiers for the board to assess, and records everything in the database. A run either completes in full or leaves no trace at all - it runs inside a single database transaction, so a failed run is always safe to simply retry.

Usage

episodic_run_cron(
  cases,
  denominators = NULL,
  institution_activity = NULL,
  episodic_config_path = Sys.getenv("EPISODIC_CONFIG", unset = NA),
  pathogen_config_path = Sys.getenv("EPISODIC_PATHOGEN_CONFIG", unset = NA),
  db_path = Sys.getenv("EPISODIC_DB"),
  host = Sys.info()[["nodename"]],
  account = Sys.info()[["user"]],
  run_date = Sys.Date(),
  backfill = NULL,
  debug = FALSE
)

Arguments

cases: Your laboratory data: a data frame or tibble in the shape episodic_case_data describes, or a zero-argument function that returns one. Defaults to the bundled synthetic generator, useful for demos and testing but not real surveillance.
denominators: Optional: your testing-volume data, in the same form as cases - normally a data set, a function if it has to be produced at run time (see episodic_synthetic_denominators() for the expected shape). Leave as NULL (the default) if you have none to supply - positivity panels simply stay blank.
institution_activity: Optional: your hospital patient-days data (see episodic_synthetic_institution_activity() for the expected shape), normally as a data set, or as a function taking the current institutions table. Its institution_key is the same identifier your case data uses - EpiSODIC hashes both on load, so a key taken from the institutions table this passes a function is already hashed and will match nothing. Leave as NULL (the default) if you have none - detection falls back to raw case counts.
episodic_config_path: The config path.
pathogen_config_path: Path to an operator's own per-pathogen CSV, overlaid row-by-row on the shipped inst/config/episodic_default_pathogen_config.csv. Defaults to the EPISODIC_PATHOGEN_CONFIG environment variable; if unset, only the shipped defaults are used. Set to a path that does not exist, this stops with an error. See vignette("tuning-detection") for the column reference.
db_path: Path to the EpiSODIC database: a SQLite file (created automatically if it does not exist yet) or a MariaDB/MySQL DSN (see episodic_db_dsn_mariadb()).
host, account: Recorded with the run for audit purposes; default to the current machine and account.
run_date: The date to treat as "today". Defaults to the system date. Every detector anchors on it: same_place and rare_trigger report only hits inside their lookback_days of it, and Farrington tests only weeks that are complete on or before it. Detection against a historical extract is therefore one run per date with a run_date inside the extract's own window, not one run dated today holding the whole archive - a run dated today reports what is current, and the archive is not. episodic_validate_detection() replays exactly that way.
backfill: Whether this run reports the whole case history it is given rather than only what falls inside the detectors' lookback windows. NULL, the default, decides from the database: a run with no completed run behind it is a backfill, so a first import of years of history arrives as years of clusters (the settled ones closed in that same run by reconciliation.stale_open_days, and so straight into the Archive) rather than as an empty dashboard. Every run after it is bounded again, so nothing is ever reported twice. Pass FALSE to bound the first run too, which is what a prospective replay wants: episodic_validate_detection() measures detection delay, and a first replay run that reported its whole baseline at once would measure the import instead. TRUE forces it on, for a database whose history was imported by something other than a run.
debug: If TRUE, print a good deal more than the phase-by-phase progress this function always writes: sessionInfo(), the versions of every package whose own behaviour a run depends on, memory snapshots at the start and end, and per-stream detail inside the detection loop - which stream, how many cases, whether it cleared the eligibility gate, what each detector did with it. Useful when a run's results are not what an operator expects and the question is which stream, or which phase, they diverged at. Noisy; leave off for routine scheduled runs.

Value

Invisibly, the run_id of the completed run. The run's row in episodic_detection_run holds its status, the per-feed load counts, and error_text if it failed. Case data that does not satisfy the requirements throws instead of returning - the run row is still written, with status = "failed" and the same message in error_text.

Details

EpiSODIC never connects to your laboratory system directly. You extract and transform your own data beforehand, and hand it over as a plain data frame or tibble: cases for the laboratory results themselves (see episodic_case_data for the required columns and their allowed values), and optionally denominators and institution_activity for testing volume and hospital activity. A data set is the normal case; if producing the data only makes sense at run time (a live database query, for instance), a zero-argument function returning one is accepted just as well.

The exact detection settings used are recorded with the run, so any past result can always be traced back to the configuration that produced it.

A database whose schema is behind the installed package is brought up to date before the run starts, the way episodic_db_migrate() would, so upgrading an instance is upgrading the package. Set database.auto_migrate to false in your configuration where schema changes go through a DBA instead; the run then refuses such a database and records why. See vignette("deployment"), "Upgrading EpiSODIC".

So is what each feed delivered. Before the run writes anything, your case data goes through episodic_check_cases(). Structural problems - a missing column, a value outside the allowed set, a date that does not read as a date - stop the run with an error naming every offending column, its values and the rows they are in, and are recorded on the run as well, so the reason is visible both where the run was started and in the dashboard's activity screen. Advisory findings are mentioned once and the run proceeds. Run episodic_check_cases() on your extract yourself to see all of it without starting a run at all. A run that fails later, for any other reason, records the reason and warns rather than returning quietly. Rows that are merely unmatched are counted rather than dropped in silence: institution activity whose institution_key matches no known institution is skipped with a warning, its count recorded, and the run finishes "partial" instead of "success". Both are complete runs the dashboard reads from; "partial" says go and look at why rows were skipped. episodic_detection_run carries the counts (n_cases_supplied, n_cases_inserted, n_activity_skipped, and the rest).

Check your data before you run anything

Do not find out from an empty dashboard. Run episodic_check_cases() on your extract - it needs no database, changes nothing, and reports every problem it finds at once, with the rows and values involved and what to do about each:

cases <- my_extract_and_transform_function()
episodic_check_cases(cases)

It also reports what is merely worth a look: one pathogen spelled two ways (two streams instead of one), no ward on any hospital row (no ward-level detection), a patient_key that never repeats (no deduplication), postcodes the map cannot place, sample dates in the future. episodic_check_cases(..., stop_on_problem = TRUE) runs the same checks but throws an error, for use in a script. episodic_run_cron() runs that before every run, and refuses to start on data it cannot use, naming what to fix.

Examples

# \donttest{
db_path <- tempfile(fileext = ".sqlite")
cases <- episodic_synthetic_cases(
  start_date = as.Date("2025-01-01"), end_date = as.Date("2025-03-31")
)
run_id <- episodic_run_cron(db_path = db_path, cases = cases)
#> 2026-09-30 10:43:09.943 | episodic_run_cron() starting (host=runnervmtr4k5, account=runner)
#> 2026-09-30 10:43:09.944 | Resolving configuration
#> 2026-09-30 10:43:09.950 | Configuration resolved (hash 7fef366a6991)
#> 2026-09-30 10:43:09.950 | Connecting to database
#> 2026-09-30 10:43:09.950 | No existing database found - creating one
#> 2026-09-30 10:43:09.982 | Database connected (SQLite 3.53.3)
#> 2026-09-30 10:43:09.983 | Run 1 started
#> 2026-09-30 10:43:09.983 | Resolving and checking case data
#> 2026-09-30 10:43:09.996 | Case data checked: 421 rows, 0 problems, 0 advisory finding(s)
#> 2026-09-30 10:43:09.997 | Beginning transaction
#> 2026-09-30 10:43:09.997 | Loading pathogen configuration
#> 2026-09-30 10:43:10.001 | Pathogen configuration loaded (39 pathogen(s))
#> 2026-09-30 10:43:10.001 | Loading case data into the database
#> 2026-09-30 10:43:10.111 | Case data loaded: supplied=421, deduplicated=403, inserted=403
#> 2026-09-30 10:43:10.112 | Fetching all known cases and institutions
#> 2026-09-30 10:43:10.114 | First run on this database: reporting the whole case history rather than only what falls inside the detectors' lookback windows. Clusters whose last case is more than 60 day(s) before this run's date close in this same run and go straight to the Archive; the rest open for assessment. Every run after this one is bounded again, so nothing here is reported twice.
#> 2026-09-30 10:43:10.116 | Case history on file spans 2025-01-01 to 2025-03-31, ending 548 day(s) before this run's date (2026-09-30)
#> 2026-09-30 10:43:10.117 | Enumerating lattice streams
#> ! 2026-09-30 10:43:10.144 | no province could be resolved for any of the 105 postcode values in this run - province-level (L4) detection has nothing to run on. EPISODIC_PC_PROVINCE_MAP is unset, and there is no built-in rule to fall back on - deriving a province from a postcode is country-specific. Point it at your own pc/province_code CSV, or leave it unset and the province level stays empty.
#> 2026-09-30 10:43:10.151 | Running same-place detector
#> 2026-09-30 10:43:10.228 | Same-place detector found 5 detection(s)
#> 2026-09-30 10:43:10.228 | Running rare-trigger detector
#> 2026-09-30 10:43:10.234 | Rare-trigger detector found 1 detection(s)
#> 2026-09-30 10:43:10.234 | Farrington tests every week its streams can carry this run, rather than the 8-week catch-up cap: there is no earlier run to catch up to
#> 2026-09-30 10:43:10.236 | Reconciling 320 stream(s) (Farrington/MEM detection, triangle update, cluster reconciliation)
#> ! 2026-09-30 10:43:10.720 | MEM: declined stream Bordetella pertussis/pathogen_region/REGION, disabled by mem_mode
#> ! 2026-09-30 10:43:10.726 | MEM: declined stream Campylobacter/pathogen_region/REGION, insufficient history for climatology
#> ! 2026-09-30 10:43:10.732 | MEM: declined stream Clostridioides difficile/pathogen_region/REGION, disabled by mem_mode
#> ! 2026-09-30 10:43:10.737 | MEM: declined stream Giardia lamblia/pathogen_region/REGION, insufficient history for climatology
#> ! 2026-09-30 10:43:10.744 | MEM: declined stream Influenza A/pathogen_region/REGION, insufficient history for climatology
#> ! 2026-09-30 10:43:10.750 | MEM: declined stream MRSA/pathogen_region/REGION, disabled by mem_mode
#> ! 2026-09-30 10:43:10.755 | MEM: declined stream Neisseria meningitidis/pathogen_region/REGION, disabled by mem_mode
#> ! 2026-09-30 10:43:10.760 | MEM: declined stream Norovirus/pathogen_region/REGION, insufficient history for climatology
#> ! 2026-09-30 10:43:10.766 | MEM: declined stream RSV/pathogen_region/REGION, insufficient history for climatology
#> ! 2026-09-30 10:43:10.779 | MEM: declined stream Salmonella/pathogen_region/REGION, insufficient history for climatology
#> 2026-09-30 10:43:10.815 | Stream loop time by stage: cases 0.0s, mem 0.0s, eligibility 0.2s, farrington 0.0s, trend 0.0s, detections 0.0s, reconciliation 0.0s, total 0.4s (320 stream(s))
#> 2026-09-30 10:43:10.815 | Stream reconciliation done: 6 detection(s), 6 new signal(s), 0 updated signal(s)
#> 2026-09-30 10:43:10.816 | Backfill: 6 cluster(s) opened from the case history, 6 of them already closed by the system and in the Archive, 0 left open for assessment
#> 2026-09-30 10:43:10.818 | Suppressing lattice: weighing 6 cluster(s) across 5 pathogen(s)
#> 2026-09-30 10:43:10.819 | Suppressing lattice: 42 case link(s) read for 6 cluster(s), 0 assessed cluster(s) exempt
#> 2026-09-30 10:43:10.826 | Suppressing lattice done: 0 cluster(s) suppressed (0 parent(s) behind a dominant child, 0 child(ren) behind a diffuse parent)
#> 2026-09-30 10:43:10.827 | Nowcast: no stream carries an open cluster
#> 2026-09-30 10:43:10.828 | Epidemic direction: no open epidemic
#> 2026-09-30 10:43:10.829 | Epidemic outlook: no open epidemic
#> 2026-09-30 10:43:10.831 | Outbreak-end forecast: 0 of 0 open outbreak(s) of a transmissible pathogen computed, 0 point source, 0 without the parameters or reporting history, 0 failed
#> 2026-09-30 10:43:10.831 | Committing transaction
#> 2026-09-30 10:43:10.833 | Finishing run 1 (status: success)
#> 2026-09-30 10:43:10.836 | episodic_run_cron() finished in 0.9s (status: success)
file.remove(db_path)
#> [1] TRUE
# }