ALLSHIFT/docs/06-data/dataset-inventory.md

Dataset Inventory

Markdown report of the repository's data files.

Scope	Every .csv, .csv.gz, .parquet and .xlsx data file in the repository
Total data volume	≈ 194 MB — the great majority of the repository's size
Status	Current inventory
Report generated	2026-07-25

Data files carry no README of their own, so this page records what each one contains, where it came from, and how it relates to the others.

Summary

File	Size	Rows	Source	Role
2019-1338340-one_axis.csv	167.9 MB	8,762	NREL NSRDB	Satellite meteorology, Boston 2019
MA_hourly_load.csv	13.7 MB	35,040	NREL ComStock	⚠️ CSV twin of the parquet below
hospital_communication_energy_system.csv	3.2 MB	10,000	Synthetic	Building + patient environment mock-up
89993-0.parquet	2.4 MB	35,040	NREL ComStock	Hospital load profile (BIDMC proxy)
chartevents.csv.gz	5.3 MB	668,862	MIMIC-IV Demo	ICU vital signs
inputevents.csv.gz	0.75 MB	20,404	MIMIC-IV Demo	ICU infusions
patients.csv.gz	small	100	MIMIC-IV Demo	Patient demographics
pvwatts_hourly.csv	0.58 MB	8,760	NREL PVWatts	PV performance, Boston
Profiles SHIFT(...).xlsx ×2	16 kB each	24	SHIFT	Generation profiles (identical copies)
Profiles SHIFT(...).csv	9 kB	24	SHIFT	CSV export of the above
Team03_demand.csv ×2	0.65 MB each	—	EST model (vendored)	Demand example (identical copies)
Team03_supply.csv ×2	0.65 MB each	—	EST model (vendored)	Supply example (identical copies)

All the Boston-area datasets converge on the same coordinates — latitude 42.33, longitude −71.10 — which is BIDMC's location at 330 Brookline Avenue, Boston MA 02215.

---

Hospital load

89993-0.parquet — 2.4 MB

NREL ComStock simulated hospital load profile, building ID 89993, Massachusetts Climate Zone 5A. 15-minute resolution, 35,040 rows covering 2018. Fully documented in BIDMC ComStock Load Profile.

MA_hourly_load.csv — 13.7 MB ⚠️

Warning

This is the same dataset as 89993-0.parquet, stored as CSV. Verified:

Property	MA_hourly_load.csv	89993-0.parquet
bldg_id	89993	89993
Columns	64	64
Data rows	35,040	35,040
First timestamp	01/01/2018 00:15	2018-01-01 00:15
Last timestamp	01/01/2019 00:00	2019-01-01 00:00

The CSV is 5.8× larger than the parquet for identical content. The filename (MA_hourly_load) is also misleading — the data is at 15-minute, not hourly, resolution.

Keeping the parquet and dropping the CSV would save 13.7 MB with no information loss. Left in place for now because some team tooling may read the CSV directly — but nothing in the repository does, and this is the single easiest size win available.

hospital_communication_energy_system.csv — 3.2 MB

10,000 rows, 25 columns, 5-minute intervals from 2025-01-01 to 2025-02-04. Synthetic. Each row is a single patient-room snapshot combining vitals, room climate, and subsystem power. Schema documented in BIDMC Datasets Manual §4.

---

Clinical data — MIMIC-IV Demo

File	Rows	Columns
patients.csv.gz	100	6
chartevents.csv.gz	668,862	11
inputevents.csv.gz	20,404	26

Important

Provenance and licence. These are the MIMIC-IV Clinical Database Demo, not full MIMIC-IV and not BIDMC's own records. Three lines of evidence:

1. patients.csv.gz contains exactly 100 patients — the demo's size.
2. The Digital Twin Readiness Checklist links directly to physionet.org/content/mimic-iv-demo/2.2/icu/inputevents.csv.gz.
3. Column names, itemid codes, and the ~100-year date shift are MIMIC-IV's de-identification scheme.

The demo is published under the Open Data Commons Open Database License (ODbL) v1.0 and, unlike full MIMIC-IV, does not require credentialed PhysioNet access. Redistribution here is therefore permissible — but ODbL requires attribution and share-alike, and neither the licence nor the citation is recorded anywhere in this repository.

Recommended: add a SOURCE.md beside these three files with the PhysioNet DOI, version (2.2), and the ODbL notice. Small fix, closes a real obligation.

All dates are shifted forward ~100 years for de-identification; only lengths of stay are meaningful. See BIDMC Datasets Manual for full column references.

---

Solar and meteorological data

pvwatts_hourly.csv — 0.58 MB

NREL PVWatts hourly PV performance simulation. 8,760 rows (one year, hourly).

Header field	Value
Requested location	330 Brookline Avenue, Boston, Massachusetts 02215, United States
Solar resource data	NSRDB Lat/Lng 42.33, −71.10
DC system size	4 kW
Module type	Standard
Array type	Fixed (open rack)
Array tilt	20°
Array azimuth	180°
System losses	14.08%
DC-to-AC size ratio	1.2
Inverter efficiency	96%
Albedo	From weather file
Bifacial	No

Warning

The simulated array is 4 kW — a residential system. The project specifies 2,300 m² of panels (≈ 460 kW peak at 20% efficiency), and the forecasting spec assumes tilt 35° and system size ~100 kWp. Three different array configurations are in circulation:

Source	System size	Tilt
This dataset	4 kW	20°
Forecasting Requirements	~100 kWp	35°
Simulator I/O Interface	2,300 m² ≈ 460 kW	—

The shape of the generation profile is reusable — it scales linearly with array size at fixed tilt — but the tilt difference (20° vs 35°) changes the seasonal profile, and the absolute values are 100× too small. Rerunning PVWatts at the specified configuration would take minutes and remove the ambiguity.

Documented in PV Hourly Dataset Manual.

2019-1338340-one_axis.csv — 167.9 MB

NREL NSRDB satellite meteorological data, location ID 1338340, latitude 42.33, longitude −71.1, elevation 14 m, time zone −5. Version 3.2.2. 8,762 rows (2019, hourly).

Column groups: clear-sky DHI/DNI/GHI, measured DHI/DNI/GHI, dew point, temperature, pressure, relative humidity, precipitable water, wind direction and speed, solar zenith and azimuth, surface albedo, panel tilt and azimuth, spectral data, plus cloud-type and fill-flag categoricals and per-technology performance columns (Si, GaAs, InGaP, CdTe).

This single file is 87% of the repository's data volume and roughly 12× the next largest file. It covers one year, one location, hourly — 8,762 rows. The bulk is column count, not row count, and most of those columns (the per-technology performance series) are unused: SHIFT models a single array with a scalar efficiency mu_PV.

Extracting the ~15 columns the forecasting stack actually needs — GHI, DNI, DHI, their clear-sky counterparts, temperature, humidity, wind speed, pressure, precipitable water, solar zenith and azimuth — would reduce it by well over an order of magnitude while losing nothing the project uses.

Documented in Satellite PV Dataset Manual.

---

SHIFT-generated profiles

File	Location	Note
Profiles SHIFT(Generators Factors + Battery).xlsx	Simulations/Current Framework/	MD5 cb529138
Profiles SHIFT(Generators Factors + Battery).xlsx	Simulations/PV+Battery Simulink/	MD5 cb529138 — identical
Profiles SHIFT(Generators Factors + Battery).csv	Simulations/PowerWorld/New Powerworld/	Semicolon-delimited, comma-decimal export of sheet 1

Both .xlsx copies are retained: PVBatteryRead.mlx opens the workbook by bare filename, so MATLAB resolves it from the working directory and the copy beside the script is load-bearing.

Contents documented in Generation Profiles Workbook.

---

Vendored data (EST model)

File	Size	Copies
Team03_demand.csv	0.65 MB	EST-model-main/ and EST-model-main/data/ — byte-identical
Team03_supply.csv	0.65 MB	EST-model-main/ and EST-model-main/data/ — byte-identical

~1.3 MB duplicated, but inside the vendored MIT-licensed subtree. Left untouched — see Existing EST Model.

---

Size analysis

Category	Size	Share
Satellite meteorology (1 file)	167.9 MB	87%
Hospital load — CSV twin	13.7 MB	7%
Clinical (MIMIC-IV demo, 3 files)	6.1 MB	3%
Synthetic building energy	3.2 MB	2%
Hospital load — parquet	2.4 MB	1%
PV performance	0.6 MB	<1%
Vendored EST data	2.6 MB	1%
SHIFT profiles	41 kB	<1%

Two changes would remove ~150 MB without losing anything the project uses:

1. Column-subset 2019-1338340-one_axis.csv to the 15 forecasting features (−160 MB).
2. Drop MA_hourly_load.csv, keeping the identical 89993-0.parquet (−13.7 MB).

Neither is applied here — pruning datasets is a data-team decision, not a documentation one. Both are recorded so the choice is available.

Gaps

Gap	Detail
No measured hospital load	Every load figure traces to NREL ComStock simulation or synthetic generation. The forecasting spec flags this as the project's biggest data risk, and it is unresolved
No electricity price series	price(t) is a required simulator input (§2.1) and the price forecaster needs ENTSO-E data. Nothing here
No carbon intensity series	CO2_int(t) likewise
No hydrogen system data	No electrolyser, fuel cell, or H₂ tank measurements anywhere
Geographic split	All datasets are Boston (BIDMC). The price forecaster targets the Dutch market, and the placeholder irradiance in the profiles workbook is Eindhoven. The two case studies have not been reconciled
No licence/citation files	MIMIC-IV demo (ODbL), NREL ComStock, NSRDB and PVWatts all have attribution expectations; none are recorded

Related

• BIDMC Datasets Manual
• BIDMC ComStock Load Profile
• PV Hourly Dataset Manual · Satellite PV Dataset Manual
• Generation Profiles Workbook
• Forecasting Requirements