Detection and reconciliation

This vignette walks through how a laboratory result becomes an outbreak or epidemic dossier an epidemiologist can act on: detection, scale routing, reconciliation, and suppression across runs.

The lattice: streams, not one big feed

Every case belongs to several streams at once - a ward-level stream, an institution-level stream, an area, a province, the whole catchment - each watched independently. Streams are enumerated automatically from the data every run, so a newly appearing pathogen creates its own streams without any configuration step:

Level	Unit
L1	Ward or specialism
L2	Hospital or care home
L3	A contiguous geographic grouping
L4	Province
L5	Full catchment

Each level is a strict partition of the one above it - a ward sits inside exactly one hospital, a hospital inside exactly one area, and so on - which is what makes suppression (below) meaningful.

Four detectors, not one algorithm

• farrington (surveillance::farringtonFlexible()) - the standard statistical aberration detector for streams with enough baseline history, run at L1/L2 with a patient-day population offset where one exists, so a busy month and a quiet one at identical transmission-per-patient-day read as the same signal strength.
• same_place - a rule-based detector needing no baseline at all: n or more cases of the same organism at the same place within k days. This is what catches long-term care institutions, which are individually too sparse for a statistical baseline, and it also runs alongside Farrington at hospitals, where it can fire first on a fast cluster before a weekly aggregation has produced a point to test.
• rare_trigger - a curated list of organisms (default is the shipped inst/config/episodic_default_config.yaml) where a single occurrence anywhere is itself the signal - certain notifiable or exceptionally rare pathogens do not need a second case to warrant a look.
• mem (the mem package’s Moving Epidemic Method) - for organisms whose own data show a seasonal pattern (derived automatically, or overridden per pathogen via mem_mode), answering “has the epidemic started” rather than “are counts elevated”, with pre-/post-epidemic thresholds derived from historical seasons. The season anchor is derived from a 52-week climatology, so it works in either hemisphere with no configuration. MEM runs at province (L4) and region (L5) by default, configurable via mem.levels.

A stream’s own history feeds back into its baseline: a period an epidemiologist confirmed as a real epidemic is excluded from what farringtonFlexible() sees on subsequent runs for that stream, so last winter’s outbreak does not silently raise this winter’s threshold.

Scale routing: outbreaks and epidemics

A configurable boundary (scale.epidemic_levels) splits the lattice into two scales. Streams below the boundary produce outbreaks (localised, L1 to L3), shown on the Outbreaks screen. Streams at or above it produce epidemics (regional, L4 and L5), shown on the Epidemics screen. Each cluster receives its scale at creation.

Outbreaks are identified as O-{id} and epidemics as E-{id} in English. Both draw on one cluster_id sequence, so a given number is either an outbreak or an epidemic, never both. The prefix is not stored: it is the first letter of the screen’s own name in the reader’s language (U-12 and E-12 in Dutch, A-12 and E-12 in German), and a language whose two names begin with the same letter carries a distinct pair instead.

Reconciliation: turning detections into outbreaks and epidemics

Detection alone produces a burst of raised signals every run, the same underlying outbreak would otherwise appear as a new row every single time it is still elevated. Reconciliation, run per stream immediately after detection inside the same transaction, is what turns that into a small number of persistent outbreaks and epidemics an epidemiologist can actually track:

1. Detections from different detectors whose intervals overlap merge into one candidate episode.
2. Each candidate is matched against currently open clusters in the same stream by interval overlap (or proximity within case_free_days): no match creates a new cluster; one match extends the existing one and recomputes its score; multiple matches merge, with the oldest surviving and the others’ assessment history preserved, never discarded.
3. An outbreak nobody has seen a new candidate for gets its runs_since_detected counter incremented; past a threshold, if it has never been assessed at all, it closes itself automatically, a plain system closure, no verdict attached. An outbreak an epidemiologist has actually classified, whether as noise or as worth watching, is never auto-closed by any verdict of its own, closing it always takes a person’s deliberate decision from the assessment form’s own closure checkbox.

Epidemic closure

Seasonal epidemics close automatically when the weekly case count drops below the post-epidemic threshold (ended_reason = 'post_epidemic_threshold'). As a backstop for noisy series that never cross cleanly, closure also fires when the evaluated week’s count is back at or below the pre-epidemic threshold in a week inside the pathogen’s derived trough (ended_reason = 'trough'). The trough on its own is a calendar rather than a measurement: an epidemic running out of season, with its count still above the pre-epidemic threshold, stays open until the post-epidemic threshold or an epidemiologist’s declaration ends it. Non-seasonal epidemics close on the same case_free_days and stale_open_days rules as outbreaks.

An epidemic’s identity is preserved across the season anchor rollover: a rise that has not been closed keeps its cluster ID when the season label changes, and no spurious second epidemic is opened.

The “during” link

After reconciliation and before suppression, each open outbreak is checked against open epidemics for the same pathogen with overlapping dates and geographically nested geography. An outbreak that qualifies is linked as occurring during that epidemic, never part of it, because claiming a ward outbreak is part of the regional epidemic is a transmission claim that cannot be supported without typing data. An outbreak can be during more than one epidemic (a province and a region epidemic at once).

Because the anchor is the sample date, a late-arriving case can legitimately change an already-detected signal retrospectively - reconciliation deliberately re-runs over a rolling window, not only “today”, sized from the empirical reporting-delay curve.

Suppression: one lattice, not five parallel views

Without suppression, one real outbreak would appear as five separate signals, one per level, burying the actual signal in restatements of itself. After reconciliation:

• A child cluster suppresses its parent when it accounts for most of the parent’s excess (the rise is local; the wider view adds nothing).
• A parent suppresses its children when the rise is spread across several of them with no single dominant one (the rise is diffuse; five separate area-level outbreaks would be noise).
• The suppressed clusters are never discarded, they stay attached as context, visible on the surviving cluster’s detail screen.

Suppression is continuous across the outbreak/epidemic scale boundary. The pathogen_area (outbreak) to pathogen_province (epidemic) pair crosses it, so an area-level outbreak can suppress or be suppressed by a province-level epidemic. The geographic levels group on the patient’s postcode, so a hospital’s cases sit in no single area; and letting a diffuse regional signal suppress a ward outbreak built into it would hide the more actionable of the two behind the vaguer one.

That leaves clusters which share cases and both stand: a regional norovirus rise driven by a ward outbreak and a nursing home is one set of cases in three dossiers. Rather than collapse them, the dossier says so, each names the others in its header and lists them under Related clusters. Two views of one rise, and the epidemiologist is told which is which.

Derived state, not a stored status

A cluster’s displayed state (new, assessing, monitoring, closed, needs reassessment) is computed, every time, from its classification history, the case-free clock, and whether it changed since it was last assessed, never stored and hand-edited. See episodic_derive_state() for the exact rule and state_derive.R’s own tests for every transition it covers.

Declarations: a human act

MEM’s threshold crossing is a measurement. Declaring the season started is a decision with an author, a timestamp and policy consequences across multiple hospitals. The system raises the signal; a person declares. An epidemiologist records season_started, season_not_yet or season_ended through the assessment event stream on the Epidemics screen. Nothing in the system writes a declaration automatically.