# Dataset Inventory

> **Markdown report of the repository's data files.**
>
> | | |
> |---|---|
> | **Scope** | Every `.csv`, `.csv.gz`, `.parquet` and `.xlsx` data file in the repository |
> | **Total data volume** | **≈ 194 MB** — the great majority of the repository's size |
> | **Status** | Current inventory |
> | **Report generated** | 2026-07-25 |

Data files carry no README of their own, so this page records what each one contains, where it
came from, and how it relates to the others.

## Summary

| File | Size | Rows | Source | Role |
|---|---:|---:|---|---|
| `2019-1338340-one_axis.csv` | **167.9 MB** | 8,762 | NREL NSRDB | Satellite meteorology, Boston 2019 |
| `MA_hourly_load.csv` | 13.7 MB | 35,040 | NREL ComStock | ⚠️ **CSV twin of the parquet below** |
| `hospital_communication_energy_system.csv` | 3.2 MB | 10,000 | Synthetic | Building + patient environment mock-up |
| `89993-0.parquet` | 2.4 MB | 35,040 | NREL ComStock | Hospital load profile (BIDMC proxy) |
| `chartevents.csv.gz` | 5.3 MB | 668,862 | MIMIC-IV Demo | ICU vital signs |
| `inputevents.csv.gz` | 0.75 MB | 20,404 | MIMIC-IV Demo | ICU infusions |
| `patients.csv.gz` | small | 100 | MIMIC-IV Demo | Patient demographics |
| `pvwatts_hourly.csv` | 0.58 MB | 8,760 | NREL PVWatts | PV performance, Boston |
| `Profiles SHIFT(...).xlsx` ×2 | 16 kB each | 24 | SHIFT | Generation profiles (identical copies) |
| `Profiles SHIFT(...).csv` | 9 kB | 24 | SHIFT | CSV export of the above |
| `Team03_demand.csv` ×2 | 0.65 MB each | — | EST model (vendored) | Demand example (identical copies) |
| `Team03_supply.csv` ×2 | 0.65 MB each | — | EST model (vendored) | Supply example (identical copies) |

All the Boston-area datasets converge on the same coordinates — **latitude 42.33, longitude
−71.10** — which is BIDMC's location at 330 Brookline Avenue, Boston MA 02215.

---

## Hospital load

### `89993-0.parquet` — 2.4 MB

NREL ComStock simulated hospital load profile, building ID 89993, Massachusetts Climate Zone 5A.
15-minute resolution, 35,040 rows covering 2018. Fully documented in
[BIDMC ComStock Load Profile](../03-energy-management/bidmc-parquet-load-data-manual.md).

### `MA_hourly_load.csv` — 13.7 MB ⚠️

> [!WARNING]
> **This is the same dataset as `89993-0.parquet`, stored as CSV.** Verified:
>
> | Property | `MA_hourly_load.csv` | `89993-0.parquet` |
> |---|---|---|
> | `bldg_id` | 89993 | 89993 |
> | Columns | 64 | 64 |
> | Data rows | 35,040 | 35,040 |
> | First timestamp | `01/01/2018 00:15` | 2018-01-01 00:15 |
> | Last timestamp | `01/01/2019 00:00` | 2019-01-01 00:00 |
>
> The CSV is **5.8× larger** than the parquet for identical content. The filename
> (`MA_hourly_load`) is also misleading — the data is at **15-minute**, not hourly, resolution.

Keeping the parquet and dropping the CSV would save **13.7 MB** with no information loss.
Left in place for now because some team tooling may read the CSV directly — but nothing in the
repository does, and this is the single easiest size win available.

### `hospital_communication_energy_system.csv` — 3.2 MB

10,000 rows, 25 columns, 5-minute intervals from 2025-01-01 to 2025-02-04. **Synthetic.** Each
row is a single patient-room snapshot combining vitals, room climate, and subsystem power.
Schema documented in [BIDMC Datasets Manual §4](../03-energy-management/bidmc-datasets-manual.md).

---

## Clinical data — MIMIC-IV Demo

| File | Rows | Columns |
|---|---:|---:|
| `patients.csv.gz` | 100 | 6 |
| `chartevents.csv.gz` | 668,862 | 11 |
| `inputevents.csv.gz` | 20,404 | 26 |

> [!IMPORTANT]
> **Provenance and licence.** These are the **MIMIC-IV Clinical Database Demo**, not full
> MIMIC-IV and not BIDMC's own records. Three lines of evidence:
>
> 1. `patients.csv.gz` contains exactly **100 patients** — the demo's size.
> 2. The [Digital Twin Readiness Checklist](../03-energy-management/bidmc-digital-twin-readiness-checklist.md)
>    links directly to `physionet.org/content/mimic-iv-demo/2.2/icu/inputevents.csv.gz`.
> 3. Column names, `itemid` codes, and the ~100-year date shift are MIMIC-IV's de-identification
>    scheme.
>
> The demo is published under the **Open Data Commons Open Database License (ODbL) v1.0** and,
> unlike full MIMIC-IV, does **not** require credentialed PhysioNet access. Redistribution here
> is therefore permissible — but ODbL requires **attribution and share-alike**, and neither the
> licence nor the citation is recorded anywhere in this repository.
>
> **Recommended:** add a `SOURCE.md` beside these three files with the PhysioNet DOI, version
> (2.2), and the ODbL notice. Small fix, closes a real obligation.

All dates are shifted forward ~100 years for de-identification; only lengths of stay are
meaningful. See [BIDMC Datasets Manual](../03-energy-management/bidmc-datasets-manual.md) for
full column references.

---

## Solar and meteorological data

### `pvwatts_hourly.csv` — 0.58 MB

NREL **PVWatts** hourly PV performance simulation. 8,760 rows (one year, hourly).

| Header field | Value |
|---|---|
| Requested location | 330 Brookline Avenue, Boston, Massachusetts 02215, United States |
| Solar resource data | NSRDB Lat/Lng 42.33, −71.10 |
| **DC system size** | **4 kW** |
| Module type | Standard |
| Array type | Fixed (open rack) |
| Array tilt | 20° |
| Array azimuth | 180° |
| System losses | 14.08% |
| DC-to-AC size ratio | 1.2 |
| Inverter efficiency | 96% |
| Albedo | From weather file |
| Bifacial | No |

> [!WARNING]
> **The simulated array is 4 kW — a residential system.** The project specifies **2,300 m²**
> of panels (≈ 460 kW peak at 20% efficiency), and the
> [forecasting spec](../02-specifications/forecasting-requirements.md#11-inputs) assumes tilt
> **35°** and system size **~100 kWp**. Three different array configurations are in circulation:
>
> | Source | System size | Tilt |
> |---|---|---|
> | This dataset | 4 kW | 20° |
> | Forecasting Requirements | ~100 kWp | 35° |
> | Simulator I/O Interface | 2,300 m² ≈ 460 kW | — |
>
> The **shape** of the generation profile is reusable — it scales linearly with array size at
> fixed tilt — but the tilt difference (20° vs 35°) changes the seasonal profile, and the
> absolute values are 100× too small. Rerunning PVWatts at the specified configuration would
> take minutes and remove the ambiguity.

Documented in [PV Hourly Dataset Manual](../03-energy-management/pv-hourly-dataset-manual.md).

### `2019-1338340-one_axis.csv` — 167.9 MB

NREL **NSRDB** satellite meteorological data, location ID 1338340, latitude 42.33, longitude
−71.1, elevation 14 m, time zone −5. Version 3.2.2. 8,762 rows (2019, hourly).

Column groups: clear-sky DHI/DNI/GHI, measured DHI/DNI/GHI, dew point, temperature, pressure,
relative humidity, precipitable water, wind direction and speed, solar zenith and azimuth,
surface albedo, panel tilt and azimuth, spectral data, plus cloud-type and fill-flag
categoricals and per-technology performance columns (Si, GaAs, InGaP, CdTe).

**This single file is 87% of the repository's data volume and roughly 12× the next largest
file.** It covers one year, one location, hourly — 8,762 rows. The bulk is column count, not
row count, and most of those columns (the per-technology performance series) are unused: SHIFT
models a single array with a scalar efficiency `mu_PV`.

Extracting the ~15 columns the forecasting stack actually needs — GHI, DNI, DHI, their clear-sky
counterparts, temperature, humidity, wind speed, pressure, precipitable water, solar zenith and
azimuth — would reduce it by well over an order of magnitude while losing nothing the project
uses.

Documented in [Satellite PV Dataset Manual](../03-energy-management/satellite-pv-dataset-manual.md).

---

## SHIFT-generated profiles

| File | Location | Note |
|---|---|---|
| `Profiles SHIFT(Generators Factors + Battery).xlsx` | `Simulations/Current Framework/` | MD5 `cb529138` |
| `Profiles SHIFT(Generators Factors + Battery).xlsx` | `Simulations/PV+Battery Simulink/` | MD5 `cb529138` — **identical** |
| `Profiles SHIFT(Generators Factors + Battery).csv` | `Simulations/PowerWorld/New Powerworld/` | Semicolon-delimited, comma-decimal export of sheet 1 |

Both `.xlsx` copies are retained: `PVBatteryRead.mlx` opens the workbook by **bare filename**,
so MATLAB resolves it from the working directory and the copy beside the script is load-bearing.

Contents documented in [Generation Profiles Workbook](../04-simulations/generation-profiles-workbook.md).

---

## Vendored data (EST model)

| File | Size | Copies |
|---|---:|---|
| `Team03_demand.csv` | 0.65 MB | `EST-model-main/` and `EST-model-main/data/` — byte-identical |
| `Team03_supply.csv` | 0.65 MB | `EST-model-main/` and `EST-model-main/data/` — byte-identical |

~1.3 MB duplicated, but **inside the vendored MIT-licensed subtree**. Left untouched — see
[Existing EST Model](../04-simulations/existing-est-model.md).

---

## Size analysis

| Category | Size | Share |
|---|---:|---:|
| Satellite meteorology (1 file) | 167.9 MB | 87% |
| Hospital load — CSV twin | 13.7 MB | 7% |
| Clinical (MIMIC-IV demo, 3 files) | 6.1 MB | 3% |
| Synthetic building energy | 3.2 MB | 2% |
| Hospital load — parquet | 2.4 MB | 1% |
| PV performance | 0.6 MB | <1% |
| Vendored EST data | 2.6 MB | 1% |
| SHIFT profiles | 41 kB | <1% |

**Two changes would remove ~150 MB** without losing anything the project uses:

1. Column-subset `2019-1338340-one_axis.csv` to the ~15 forecasting features (~−160 MB).
2. Drop `MA_hourly_load.csv`, keeping the identical `89993-0.parquet` (−13.7 MB).

Neither is applied here — pruning datasets is a data-team decision, not a documentation one.
Both are recorded so the choice is available.

## Gaps

| Gap | Detail |
|---|---|
| **No measured hospital load** | Every load figure traces to NREL ComStock simulation or synthetic generation. The [forecasting spec](../02-specifications/forecasting-requirements.md#21-inputs) flags this as the project's biggest data risk, and it is unresolved |
| **No electricity price series** | `price(t)` is a required simulator input ([§2.1](../02-specifications/simulator-io-interface.md#21-exogenous-time-series)) and the price forecaster needs ENTSO-E data. Nothing here |
| **No carbon intensity series** | `CO2_int(t)` likewise |
| **No hydrogen system data** | No electrolyser, fuel cell, or H₂ tank measurements anywhere |
| **Geographic split** | All datasets are **Boston** (BIDMC). The price forecaster targets the **Dutch** market, and the placeholder irradiance in the profiles workbook is **Eindhoven**. The two case studies have not been reconciled |
| **No licence/citation files** | MIMIC-IV demo (ODbL), NREL ComStock, NSRDB and PVWatts all have attribution expectations; none are recorded |

## Related

- [BIDMC Datasets Manual](../03-energy-management/bidmc-datasets-manual.md)
- [BIDMC ComStock Load Profile](../03-energy-management/bidmc-parquet-load-data-manual.md)
- [PV Hourly Dataset Manual](../03-energy-management/pv-hourly-dataset-manual.md) · [Satellite PV Dataset Manual](../03-energy-management/satellite-pv-dataset-manual.md)
- [Generation Profiles Workbook](../04-simulations/generation-profiles-workbook.md)
- [Forecasting Requirements](../02-specifications/forecasting-requirements.md)