pepe/ALLSHIFT
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ALLSHIFT/docs/03-energy-management/bidmc-datasets-manual.md
pepe 72dd781dbc Organize documentation into docs/ and superseded/ 
Audit every document in the repository, convert the non-markdown ones into
markdown reports, and split current documentation from outdated material.

docs/ — 31 markdown documents in seven numbered sections. Twenty are new
reports generated from .docx / .pdf / .xlsx / .mlx / .m sources that were
previously unreadable in the browser and undiffable in git. Each report
carries a provenance block (source path, format, MD5) and links back to its
original; all 13 recorded checksums verify against the files on disk.
Machine-extraction losses (PDF table column interleaving, Word OMML
equations, embedded figures) are called out explicitly rather than silently
smoothed over.

superseded/ — outdated material with a documented reason per entry:
two byte-identical ClickUp re-exports, an older revision of the BIDMC/UCSD
energy-flow doc (the retained copy adds the SoC Violation Rate KPI), a
duplicate of Shift input data.docx, the May 2026 simulation plan, the
root PV+Battery.md now covered by a fuller report, GitHub's stock
demo-repository template, and a zero-byte placeholder. Its README also
records what was deliberately NOT retired and why — the "Old Frameworks"
and "Old Simulations" folders hold unique Simulink revisions, and
"Big Ugly Folder" holds the only copy of framework revision 1.3.

Findings worth flagging, all documented in the reports:
- Simulink lineage recovered from each .slx's internal coreProperties.xml
  revision counter. The current model is
  Current Framework/Bobert0206_Initial_Simulation_Framework.slx (rev 2.7);
  the top-level copy is rev 1.3, five revisions behind.
- Simulations/Constants.m is a truncated byte-prefix of the Current
  Framework copy, silently missing H2_leak, H2_cap and E_H2_vol_h.
- The PEM electrolyser and fuel cell are unmodified MathWorks Simscape
  examples still at vendor defaults; the "10x bigger" sizing TODO recorded
  in Constants.m was never carried out.
- controller-claude.m does not compile — undefined P_Electro_max, outputs
  unassigned on several paths.
- The specification set uses two incompatible variable naming conventions
  and disagrees on action-space size (5 vs 16).
- MA_hourly_load.csv (13.7 MB) is the same 35,040 rows as 89993-0.parquet
  (2.4 MB).
- Clinical data is the MIMIC-IV *demo* (ODbL, 100 patients), not full
  MIMIC-IV — redistributable, but the licence and citation are unrecorded.

Housekeeping: untrack 21 Simulink build artefacts (slprj/, *.slxc) and add
ignore rules for them. Root README rewritten around the new layout.

Recruitment notes naming individual candidates are excluded from version
control via .gitignore rather than committed; the generic question template
is kept in docs/07-team-and-operations/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-25 21:20:33 -07:00

221 lines
17 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# BIDMC Datasets Manual (v2)
> **Markdown report of a non-markdown source document.**
>
> | | |
> |---|---|
> | **Source** | `Shift Matlab Drive/Shift Matlab Drive/Energy_Managment/Beth Israel Deaconess Medical Center/BIDMC_Datasets_Manual_v2.docx` |
> | **Format** | Microsoft Word (.docx), 29.3 kB |
> | **MD5** | `78426a0944272a1549be13772a83e9c1` |
> | **Owner** | Energy Management cluster |
> | **Status** | Current (**v2**) — no v1 exists in this repository |
> | **Report generated** | 2026-07-25 |
## What this document is
Manual for the data related to the BIDMC building structure. It documents four datasets — three
clinical and one synthetic building-energy file — and explains how each maps onto the
reinforcement-learning formulation (state space, action space, reward).
> [!IMPORTANT]
> **Provenance of the clinical files.** The column names, `itemid` codes, and de-identification
> scheme are those of **MIMIC-IV**. The
> [Digital Twin Readiness Checklist](bidmc-digital-twin-readiness-checklist.md) in the same
> folder links to `physionet.org/content/mimic-iv-demo/2.2/icu/inputevents.csv.gz`, and this
> manual records **100 patients** — both consistent with the **MIMIC-IV Clinical Database
> Demo**, which is openly published under the Open Data Commons ODbL v1.0 and does *not*
> require credentialed access.
>
> That makes redistribution here defensible, but the **licence and citation are not recorded
> anywhere in the repository**. ODbL requires attribution and share-alike. Adding a
> `LICENSE`/`SOURCE` note next to these files is a small fix worth making — see
> [Dataset Inventory](../06-data/dataset-inventory.md).
>
> These files are *not* BIDMC's own clinical records. Despite the folder name, no real BIDMC
> patient data is present.
## How the datasets link together
All three clinical files share the join key **`subject_id`**. It connects patients
(`patients.csv`) to ICU monitoring (`chartevents.csv`) and treatments (`inputevents.csv`). The
energy file links via **`Patient ID`** (format `P0001`).
Within the clinical files:
- `hadm_id` = hospital admission
- `stay_id` = specific ICU stay within that admission
- One patient can have multiple admissions and multiple ICU stays
> **All dates in the clinical files are shifted forward by around 100 years for
> de-identification.** Only the *length of stay* can be understood from the dates.
---
## 1. `patients.csv`
- Unique patient data giving an understanding of patient behaviours and needs.
**100 rows, 6 columns**, no repeated rows per patient.
- **Static** dataset, recorded once.
- Can be used as a *"virtual patient avatar"* — by modelling different ages the system can
predict energy used per person.
- For RL, this data is in the **state space**.
| Column | Type | Example | Description |
|---|---|---|---|
| `subject_id` | Integer | 10014729 | Unique patient identifier. Primary join key across all clinical files. Filter `chartevents` and `inputevents` by this to get one patient's records |
| `gender` | String | F or M | Biological sex at ICU admission |
| `anchor_age` | Integer (years) | 21–91, mean 62 | Patient age at their first ICU admission |
| `anchor_year` | Integer (shifted) | 2110–2201 | De-identified year of first ICU admission. **Cannot be used as a real calendar date** |
| `anchor_year_group` | String | 2011 – 2013 | The real calendar period of the patient's admission. Useful for understanding what medical equipment was in use |
| `dod` | Date or NULL | 2137-09-02 or empty | De-identified date of death. NULL for 69 of 100 patients (alive or unknown) |
---
## 2. `chartevents.csv`
*(5.29 MB gzipped)*
- Acts as a chart, tracking how the patient's body is performing while in ICU: heart rate,
blood pressure, oxygen levels, body temperature, ventilator settings.
- **Units:** Heart Rate bpm | Oxygen/SpO2 % | Respiratory Rate breaths/min | Temperature °C or
°F (check `valueuom` per row) | Blood Pressure mmHg | Weight kg
- Data intervals are **irregular** — monitored whenever a nurse takes measurements. Always use
`charttime` (not `storetime`) for time-series work.
- Gives a real-time trajectory of a patient's health in the digital twin.
- Represents the **dynamic state space**. The AI can judge whether a patient is getting better
or worse and thus predict whether more equipment usage will be required.
- Can also feed the **reward**: when the AI adjusts environment flows and an unstable heart rate
returns to a safe range, it can receive a major positive reward.
| Column | Type | Example | Description |
|---|---|---|---|
| `subject_id` / `hadm_id` / `stay_id` | Integer | 10005817 / 20626031 / 32604416 | `subject_id` links to `patients.csv`. `hadm_id` = hospital admission. `stay_id` = specific ICU stay. One patient can have multiple `stay_id`s. **Always filter by `stay_id` when analysing one ICU episode** |
| `caregiver_id` | Float or NULL | 6770.0 | ID of the nurse or clinician who recorded this reading. NULL means an automated monitor recording. Can be used to model nurse visit frequency, which correlates with lighting and device power usage per room |
| `charttime` | Datetime (shifted) | 2132-12-16 00:00:00 | When the measurement was taken. Use this for all time-series work. Dates are de-identified so use relative time differences only |
| `storetime` | Datetime (shifted) | 2132-12-15 23:45:00 | When the value was entered into the system. Can be ignored; `charttime` is better |
| `itemid` | Integer (code) | 220045, 220050, 220210 | Numeric code identifying what was measured. **Key codes:** 220045 = Heart Rate (bpm), 220050 = Arterial BP Systolic (mmHg), 220051 = BP Diastolic, 220052 = BP Mean, 220210 = Respiratory Rate, 220277 = SpO2 (%), 220048 = Heart Rhythm. **483 unique codes** in this dataset |
| `value` | String | "100" or "On" | Raw recorded value, always stored as text. For numeric readings also use `valuenum`. For categorical readings (e.g. ventilator On/Off) `valuenum` is NULL and you must read `value` |
| `valuenum` | Float or NULL | 100.0 | Numeric version of `value`. Used for all calculations. NULL for categorical readings |
| `valueuom` | String (unit) | %, mmHg, bpm, /min, °C | Unit of measurement for this row. **Temperature appears in both °C and °F** depending on the monitor |
| `warning` | Integer (0/1) | 0 or 1 | Clinical alarm flag. 1 = value was outside normal thresholds at the time of recording. A flagged vital after an AI action should trigger a negative reward |
---
## 3. `inputevents.csv`
*(0.75 MB gzipped)*
- Tracks everything actively pumped or injected into the patient's body: IV fluids, active
medications, nutrition.
- Records the type of medication or fluid (`ordercategoryname`), the exact amount, how fast it
was given (`rate`), and exact start and end times.
- **Units:** mL or mg for dosage (`amountuom`); mL/hour or mg/hour for infusion rate
(`rateuom`); weight in kg (`patientweight`).
- Logs continuous periods with exact start and end times, showing how long an infusion lasted.
- If a patient is actively receiving high-risk continuous IV medications, their body is under
high stress. The digital twin can use this timeline to flag that the room environment must
remain perfectly steady — the AI manager reads this status and knows it cannot alter the HVAC
settings in that room right now.
| Column | Type | Example | Description |
|---|---|---|---|
| `starttime` / `endtime` | Datetime (shifted) | 2132-12-16 19:50:00 | Start and end of the infusion. Any `chartevents.charttime` within this window occurred during active treatment |
| `ordercategoryname` | String | 01-Drips, 08-Antibiotics (IV) | Clinical type of what is being given. **`01-Drips`** = continuous vasopressor or sedative infusions (highly unstable patient — do not alter room). Full list in this dataset: 01-Drips, 02-Fluids (Crystalloids), 05-Med Bolus, 06-Insulin, 07-Blood Products, 08-Antibiotics (IV), 10-Prophylaxis (IV), 13-Enteral Nutrition, 14-Oral/Gastric Intake |
| `amount` / `amountuom` | Float / String | 1.0 / "dose" or 500 / "ml" | Quantity given and its unit. Units seen: ml, mg, mcg, grams, units, mEq, dose. Always read both together. "dose" means a standard clinical dose with no specific volume recorded |
| `rate` / `rateuom` | Float or NULL / String | 5.0 / "mL/hour" | Speed of infusion. NULL = given all at once. Non-NULL = continuous drip, patient is actively on a pump. **Active pumps consume measurable medical equipment power** |
| `statusdescription` | String | FinishedRunning, Stopped, ChangeDose/Rate, Paused, Bolus | Outcome of this infusion record. `FinishedRunning` = completed normally. `Stopped` = discontinued early (possible adverse event). `ChangeDose/Rate` = dose adjusted mid-infusion (a new row follows) |
| `patientweight` | Float (kg) | 91.0 | Patient body weight at the time of this event. Weight-based dosing is common in ICU. Also feeds the virtual avatar: heavier patients have higher metabolic output, affecting room temperature modelling |
| `ordercategorydescription` | String | Drug Push, Continuous Med | How the order was administered. `Drug Push` = fast IV push. `Continuous Med` = ongoing infusion. Useful for classifying clinical intensity |
| `isopenbag` / `continueinnextdept` | Integer (0/1) | 0 or 1 | `isopenbag` = 1 if the bag was already open when this event started. `continueinnextdept` = 1 if the infusion continued after the patient moved to another unit. Both useful for tracking infusion continuity across ICU transfers |
---
## 4. `hospital_communication_energy_system.csv`
*(3.16 MB)*
- Merges real-time patient environment data with the hospital's power grid, climate systems,
and hardware performance. **10,000 rows, 25 columns.** 5-minute intervals from
**2025-01-01 to 2025-02-04**. No missing values.
- Contains room and outdoor temperatures/humidity, grid and renewable energy sources, energy
consumed by lighting/HVAC/medical gear, and system health checks.
- **Units:** Power & Energy Consumption kWh | Renewable Energy Usage % | Vitals bpm, mmHg, % |
Temperatures °C | Humidity %
- Synchronised and recorded at exactly 5-minute intervals.
- Creates the digital twin infrastructure of the building, linking patients to environment. Can
be used to predict how a heatwave or sudden grid blackout affects indoor room temperatures and
the power stability of life-support gear.
- Contains the exact switches of AI controls: Energy Saving Mode on/off, HVAC Mode
(Heating/Cooling), power distribution adjustments.
- At every 5-minute interval the AI can look at *Total Power Usage* and *Renewable Energy
Usage*. If it reduces grid reliance or lowers kWh usage while keeping all patient rooms
comfortable, it can receive a positive reward.
- By comparing *Outdoor Temperature* with *Internal Room Temperature*, the AI can predict
upcoming thermal loads (e.g. afternoon heat spikes) and pre-cool the building using cheap,
clean renewable energy before peak pricing hits the main grid.
> [!WARNING]
> **This is synthetic data.** The manual states the timestamps are *"synthetic operational data
> that is not shifted"*, energy sources are distributed *"~33% each"* across Grid/Solar/Wind,
> and HVAC Mode is a *"~50/50 split"* — signatures of generated rather than measured data. It
> is useful as a schema and an RL-environment mock-up, **not** as evidence about BIDMC's real
> operation. Note also that each row is a *single patient room snapshot*, not a whole-hospital
> total.
| Column | Type | Example | Description |
|---|---|---|---|
| `Timestamp` | Datetime | 2025-01-01 00:00:00 | The RL time-step index. Each row = one decision point (every 5 min). Sort by this before any time-series work |
| `Patient ID` | String | P0001 | Room/patient identifier. Links to clinical files via `subject_id` mapping. Each row is a different patient's room snapshot |
| `Temperature (°C)` | Float | 35.96 | **Patient body** temperature. Normal ICU range 36.1–37.2 °C. Above 38 °C = fever, below 36 °C = hypothermia. AI uses this to decide if HVAC needs to compensate |
| `Humidity (%)` | Float | 53.1 | Relative humidity near the patient. ICU target 30–60%. High humidity raises infection risk; low humidity causes respiratory discomfort for ventilated patients. Use as an HVAC reward constraint |
| `Oxygen Level (%)` | Float | 98.9 | Blood oxygen saturation (SpO2). Normal 95–100%. Below 90% = clinical emergency |
| `Heart Rate (bpm)` | Integer | 76 | Beats per minute. Normal ICU resting range 60–100 bpm |
| `Blood Pressure (mmHg)` | String | (111, 84) | Stored as a **tuple string** (systolic, diastolic). Normal ICU systolic 90–140 mmHg. *Requires parsing* |
| `Energy Consumption (kWh)` | Float | 16.75 | Total energy consumed in this 5-min window. Primary cost signal for the RL agent. **Distinct from `Total Power Usage`** — this is measured consumption; that is the sum of subsystem loads |
| `Energy Source` | String | Grid, Solar, Wind | Dominant power source at this timestamp (~33% each). RL agent should prefer Solar and Wind over Grid |
| `Renewable Energy Usage (%)` | Float | 52.8 | Percentage of total power drawn from renewable sources. Direct reward component: higher = better |
| `HVAC Power Usage (kWh)` | Float | 5.12 | Energy used by HVAC in this interval. **Controllable by the AI action space.** Reducing this while maintaining Room Temperature = positive reward |
| `Lighting Power Usage (kWh)` | Float | 1.87 | Energy used by lighting. Also controllable. Can be dimmed during low-activity periods without affecting patient safety |
| `Medical Equipment Power Usage (kWh)` | Float | 2.30 | Energy used by medical devices. Treated as a **fixed load** |
| `Total Power Usage (kWh)` | Float | 9.29 | `= HVAC + Lighting + Medical Equipment`. Minimising this while keeping Room Temperature and Humidity in safe ranges is an AI reward |
| `Room Temperature (°C)` | Float | 21.7 | Ambient temperature inside the patient's room — what the HVAC controls. **AI target: 21–23 °C** for ICU comfort |
| `Room Humidity (%)` | Float | 33.7 | Ambient humidity inside the patient room. AI target 30–60% |
| `HVAC Mode` | String | Heating or Cooling | Current HVAC operating mode (~50/50 split). Both an observable state and the AI's last action. Heating = 1, Cooling = 0 |
| `HVAC Efficiency (%)` | Float | 92.9 | How efficiently HVAC converts energy into useful heating/cooling. Low efficiency = system working hard achieving little. Reward penalty: high power + low efficiency = bad |
| `Outdoor Temperature (°C)` | Float | 18.9 | External ambient temperature. Gap between outdoor and room temperature = the thermal load HVAC must overcome. Used for predictive pre-conditioning |
| `Outdoor Humidity (%)` | Float | 58.3 | External ambient humidity. Used alongside outdoor temperature to predict HVAC load and pre-plan energy sourcing |
| `Day of the Week` | Integer (0–6) | 2 | 0 = Monday, 6 = Sunday. Used for time-of-week patterns in energy demand |
| `Season` | String | Fall, Winter, Spring, Summer | Calendar season. Affects expected outdoor temperature range and baseline HVAC load. Context variable in the RL state space |
| `Energy Saving Mode` | Integer (0/1) | 0 or 1 | **AI action flag.** 1 = energy saving mode active (reduced HVAC and lighting intensity). A direct action the RL agent can toggle each step |
| `System Health Check` | Integer (0/1) | 0 or 1 | Hardware status. 0 = all systems healthy, 1 = fault or maintenance alert active. When 1, the AI should avoid aggressive energy optimisation and prioritise stability |
| `AI Predicted Health Status` | String | Healthy, Unhealthy | AI's prediction of overall patient/system health at this timestep. When Unhealthy, the RL agent should weight patient safety constraints more heavily |
---
## How this fits the wider project
This manual describes a **building-and-patient digital twin** — HVAC control, room comfort,
patient vitals as reward signals. That is a materially different system from the one the
[Simulator I/O Interface](../02-specifications/simulator-io-interface.md) specifies, which is a
**power-dispatch** environment: PV, battery, electrolyser, fuel cell, H₂ tank, grid.
The two are complementary rather than contradictory — HVAC is the largest shiftable load in the
[tier model](bidmc-ucsd-energy-flow-and-balances.md) — but nothing yet connects them. In
particular:
- No hydrogen, electrolyser, fuel cell, or battery SoC appears in any of these datasets.
- `Season` covering all four values within a 2025-01-01 → 2025-02-04 window is internally
inconsistent, reinforcing that the file is generated.
- Clinical reward signals (heart rate returning to safe range) are not present in the RL
[reward specification](../02-specifications/rl-reward-function.md), which is purely
energy/resilience-oriented.
Deciding whether the clinical layer is in scope is an open project question, not a documentation
gap.
## Related
- [BIDMC ComStock Load Profile](bidmc-parquet-load-data-manual.md) — the whole-building load proxy that complements these files
- [BIDMC Digital Twin Readiness Checklist](bidmc-digital-twin-readiness-checklist.md) — the readiness assessment over this data
- [BIDMC/UCSD Energy Flow and Balances](bidmc-ucsd-energy-flow-and-balances.md) — where the tier power figures come from
- [Dataset Inventory](../06-data/dataset-inventory.md) — every data file in the repo
Reference in a new issue View git blame Copy permalink