pepe/ALLSHIFT
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
ALLSHIFT/docs/06-data/dataset-inventory.md
pepe 72dd781dbc Organize documentation into docs/ and superseded/ 
Audit every document in the repository, convert the non-markdown ones into
markdown reports, and split current documentation from outdated material.

docs/ — 31 markdown documents in seven numbered sections. Twenty are new
reports generated from .docx / .pdf / .xlsx / .mlx / .m sources that were
previously unreadable in the browser and undiffable in git. Each report
carries a provenance block (source path, format, MD5) and links back to its
original; all 13 recorded checksums verify against the files on disk.
Machine-extraction losses (PDF table column interleaving, Word OMML
equations, embedded figures) are called out explicitly rather than silently
smoothed over.

superseded/ — outdated material with a documented reason per entry:
two byte-identical ClickUp re-exports, an older revision of the BIDMC/UCSD
energy-flow doc (the retained copy adds the SoC Violation Rate KPI), a
duplicate of Shift input data.docx, the May 2026 simulation plan, the
root PV+Battery.md now covered by a fuller report, GitHub's stock
demo-repository template, and a zero-byte placeholder. Its README also
records what was deliberately NOT retired and why — the "Old Frameworks"
and "Old Simulations" folders hold unique Simulink revisions, and
"Big Ugly Folder" holds the only copy of framework revision 1.3.

Findings worth flagging, all documented in the reports:
- Simulink lineage recovered from each .slx's internal coreProperties.xml
  revision counter. The current model is
  Current Framework/Bobert0206_Initial_Simulation_Framework.slx (rev 2.7);
  the top-level copy is rev 1.3, five revisions behind.
- Simulations/Constants.m is a truncated byte-prefix of the Current
  Framework copy, silently missing H2_leak, H2_cap and E_H2_vol_h.
- The PEM electrolyser and fuel cell are unmodified MathWorks Simscape
  examples still at vendor defaults; the "10x bigger" sizing TODO recorded
  in Constants.m was never carried out.
- controller-claude.m does not compile — undefined P_Electro_max, outputs
  unassigned on several paths.
- The specification set uses two incompatible variable naming conventions
  and disagrees on action-space size (5 vs 16).
- MA_hourly_load.csv (13.7 MB) is the same 35,040 rows as 89993-0.parquet
  (2.4 MB).
- Clinical data is the MIMIC-IV *demo* (ODbL, 100 patients), not full
  MIMIC-IV — redistributable, but the licence and citation are unrecorded.

Housekeeping: untrack 21 Simulink build artefacts (slprj/, *.slxc) and add
ignore rules for them. Root README rewritten around the new layout.

Recruitment notes naming individual candidates are excluded from version
control via .gitignore rather than committed; the generic question template
is kept in docs/07-team-and-operations/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-25 21:20:33 -07:00

234 lines
11 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Dataset Inventory
> **Markdown report of the repository's data files.**
>
> | | |
> |---|---|
> | **Scope** | Every `.csv`, `.csv.gz`, `.parquet` and `.xlsx` data file in the repository |
> | **Total data volume** | **≈ 194 MB** — the great majority of the repository's size |
> | **Status** | Current inventory |
> | **Report generated** | 2026-07-25 |
Data files carry no README of their own, so this page records what each one contains, where it
came from, and how it relates to the others.
## Summary
| File | Size | Rows | Source | Role |
|---|---:|---:|---|---|
| `2019-1338340-one_axis.csv` | **167.9 MB** | 8,762 | NREL NSRDB | Satellite meteorology, Boston 2019 |
| `MA_hourly_load.csv` | 13.7 MB | 35,040 | NREL ComStock | ⚠️ **CSV twin of the parquet below** |
| `hospital_communication_energy_system.csv` | 3.2 MB | 10,000 | Synthetic | Building + patient environment mock-up |
| `89993-0.parquet` | 2.4 MB | 35,040 | NREL ComStock | Hospital load profile (BIDMC proxy) |
| `chartevents.csv.gz` | 5.3 MB | 668,862 | MIMIC-IV Demo | ICU vital signs |
| `inputevents.csv.gz` | 0.75 MB | 20,404 | MIMIC-IV Demo | ICU infusions |
| `patients.csv.gz` | small | 100 | MIMIC-IV Demo | Patient demographics |
| `pvwatts_hourly.csv` | 0.58 MB | 8,760 | NREL PVWatts | PV performance, Boston |
| `Profiles SHIFT(...).xlsx` ×2 | 16 kB each | 24 | SHIFT | Generation profiles (identical copies) |
| `Profiles SHIFT(...).csv` | 9 kB | 24 | SHIFT | CSV export of the above |
| `Team03_demand.csv` ×2 | 0.65 MB each | — | EST model (vendored) | Demand example (identical copies) |
| `Team03_supply.csv` ×2 | 0.65 MB each | — | EST model (vendored) | Supply example (identical copies) |
All the Boston-area datasets converge on the same coordinates — **latitude 42.33, longitude
−71.10** — which is BIDMC's location at 330 Brookline Avenue, Boston MA 02215.
---
## Hospital load
### `89993-0.parquet` — 2.4 MB
NREL ComStock simulated hospital load profile, building ID 89993, Massachusetts Climate Zone 5A.
15-minute resolution, 35,040 rows covering 2018. Fully documented in
[BIDMC ComStock Load Profile](../03-energy-management/bidmc-parquet-load-data-manual.md).
### `MA_hourly_load.csv` — 13.7 MB ⚠️
> [!WARNING]
> **This is the same dataset as `89993-0.parquet`, stored as CSV.** Verified:
>
> | Property | `MA_hourly_load.csv` | `89993-0.parquet` |
> |---|---|---|
> | `bldg_id` | 89993 | 89993 |
> | Columns | 64 | 64 |
> | Data rows | 35,040 | 35,040 |
> | First timestamp | `01/01/2018 00:15` | 2018-01-01 00:15 |
> | Last timestamp | `01/01/2019 00:00` | 2019-01-01 00:00 |
>
> The CSV is **5.8× larger** than the parquet for identical content. The filename
> (`MA_hourly_load`) is also misleading — the data is at **15-minute**, not hourly, resolution.
Keeping the parquet and dropping the CSV would save **13.7 MB** with no information loss.
Left in place for now because some team tooling may read the CSV directly — but nothing in the
repository does, and this is the single easiest size win available.
### `hospital_communication_energy_system.csv` — 3.2 MB
10,000 rows, 25 columns, 5-minute intervals from 2025-01-01 to 2025-02-04. **Synthetic.** Each
row is a single patient-room snapshot combining vitals, room climate, and subsystem power.
Schema documented in [BIDMC Datasets Manual §4](../03-energy-management/bidmc-datasets-manual.md).
---
## Clinical data — MIMIC-IV Demo
| File | Rows | Columns |
|---|---:|---:|
| `patients.csv.gz` | 100 | 6 |
| `chartevents.csv.gz` | 668,862 | 11 |
| `inputevents.csv.gz` | 20,404 | 26 |
> [!IMPORTANT]
> **Provenance and licence.** These are the **MIMIC-IV Clinical Database Demo**, not full
> MIMIC-IV and not BIDMC's own records. Three lines of evidence:
>
> 1. `patients.csv.gz` contains exactly **100 patients** — the demo's size.
> 2. The [Digital Twin Readiness Checklist](../03-energy-management/bidmc-digital-twin-readiness-checklist.md)
> links directly to `physionet.org/content/mimic-iv-demo/2.2/icu/inputevents.csv.gz`.
> 3. Column names, `itemid` codes, and the ~100-year date shift are MIMIC-IV's de-identification
> scheme.
>
> The demo is published under the **Open Data Commons Open Database License (ODbL) v1.0** and,
> unlike full MIMIC-IV, does **not** require credentialed PhysioNet access. Redistribution here
> is therefore permissible — but ODbL requires **attribution and share-alike**, and neither the
> licence nor the citation is recorded anywhere in this repository.
>
> **Recommended:** add a `SOURCE.md` beside these three files with the PhysioNet DOI, version
> (2.2), and the ODbL notice. Small fix, closes a real obligation.
All dates are shifted forward ~100 years for de-identification; only lengths of stay are
meaningful. See [BIDMC Datasets Manual](../03-energy-management/bidmc-datasets-manual.md) for
full column references.
---
## Solar and meteorological data
### `pvwatts_hourly.csv` — 0.58 MB
NREL **PVWatts** hourly PV performance simulation. 8,760 rows (one year, hourly).
| Header field | Value |
|---|---|
| Requested location | 330 Brookline Avenue, Boston, Massachusetts 02215, United States |
| Solar resource data | NSRDB Lat/Lng 42.33, −71.10 |
| **DC system size** | **4 kW** |
| Module type | Standard |
| Array type | Fixed (open rack) |
| Array tilt | 20° |
| Array azimuth | 180° |
| System losses | 14.08% |
| DC-to-AC size ratio | 1.2 |
| Inverter efficiency | 96% |
| Albedo | From weather file |
| Bifacial | No |
> [!WARNING]
> **The simulated array is 4 kW — a residential system.** The project specifies **2,300 m²**
> of panels (≈ 460 kW peak at 20% efficiency), and the
> [forecasting spec](../02-specifications/forecasting-requirements.md#11-inputs) assumes tilt
> **35°** and system size **~100 kWp**. Three different array configurations are in circulation:
>
> | Source | System size | Tilt |
> |---|---|---|
> | This dataset | 4 kW | 20° |
> | Forecasting Requirements | ~100 kWp | 35° |
> | Simulator I/O Interface | 2,300 m² ≈ 460 kW | — |
>
> The **shape** of the generation profile is reusable — it scales linearly with array size at
> fixed tilt — but the tilt difference (20° vs 35°) changes the seasonal profile, and the
> absolute values are 100× too small. Rerunning PVWatts at the specified configuration would
> take minutes and remove the ambiguity.
Documented in [PV Hourly Dataset Manual](../03-energy-management/pv-hourly-dataset-manual.md).
### `2019-1338340-one_axis.csv` — 167.9 MB
NREL **NSRDB** satellite meteorological data, location ID 1338340, latitude 42.33, longitude
−71.1, elevation 14 m, time zone −5. Version 3.2.2. 8,762 rows (2019, hourly).
Column groups: clear-sky DHI/DNI/GHI, measured DHI/DNI/GHI, dew point, temperature, pressure,
relative humidity, precipitable water, wind direction and speed, solar zenith and azimuth,
surface albedo, panel tilt and azimuth, spectral data, plus cloud-type and fill-flag
categoricals and per-technology performance columns (Si, GaAs, InGaP, CdTe).
**This single file is 87% of the repository's data volume and roughly 12× the next largest
file.** It covers one year, one location, hourly — 8,762 rows. The bulk is column count, not
row count, and most of those columns (the per-technology performance series) are unused: SHIFT
models a single array with a scalar efficiency `mu_PV`.
Extracting the ~15 columns the forecasting stack actually needs — GHI, DNI, DHI, their clear-sky
counterparts, temperature, humidity, wind speed, pressure, precipitable water, solar zenith and
azimuth — would reduce it by well over an order of magnitude while losing nothing the project
uses.
Documented in [Satellite PV Dataset Manual](../03-energy-management/satellite-pv-dataset-manual.md).
---
## SHIFT-generated profiles
| File | Location | Note |
|---|---|---|
| `Profiles SHIFT(Generators Factors + Battery).xlsx` | `Simulations/Current Framework/` | MD5 `cb529138` |
| `Profiles SHIFT(Generators Factors + Battery).xlsx` | `Simulations/PV+Battery Simulink/` | MD5 `cb529138` — **identical** |
| `Profiles SHIFT(Generators Factors + Battery).csv` | `Simulations/PowerWorld/New Powerworld/` | Semicolon-delimited, comma-decimal export of sheet 1 |
Both `.xlsx` copies are retained: `PVBatteryRead.mlx` opens the workbook by **bare filename**,
so MATLAB resolves it from the working directory and the copy beside the script is load-bearing.
Contents documented in [Generation Profiles Workbook](../04-simulations/generation-profiles-workbook.md).
---
## Vendored data (EST model)
| File | Size | Copies |
|---|---:|---|
| `Team03_demand.csv` | 0.65 MB | `EST-model-main/` and `EST-model-main/data/` — byte-identical |
| `Team03_supply.csv` | 0.65 MB | `EST-model-main/` and `EST-model-main/data/` — byte-identical |
~1.3 MB duplicated, but **inside the vendored MIT-licensed subtree**. Left untouched — see
[Existing EST Model](../04-simulations/existing-est-model.md).
---
## Size analysis
| Category | Size | Share |
|---|---:|---:|
| Satellite meteorology (1 file) | 167.9 MB | 87% |
| Hospital load — CSV twin | 13.7 MB | 7% |
| Clinical (MIMIC-IV demo, 3 files) | 6.1 MB | 3% |
| Synthetic building energy | 3.2 MB | 2% |
| Hospital load — parquet | 2.4 MB | 1% |
| PV performance | 0.6 MB | <1% |
| Vendored EST data | 2.6 MB | 1% |
| SHIFT profiles | 41 kB | <1% |
**Two changes would remove ~150 MB** without losing anything the project uses:
1. Column-subset `2019-1338340-one_axis.csv` to the ~15 forecasting features (~−160 MB).
2. Drop `MA_hourly_load.csv`, keeping the identical `89993-0.parquet` (−13.7 MB).
Neither is applied here — pruning datasets is a data-team decision, not a documentation one.
Both are recorded so the choice is available.
## Gaps
| Gap | Detail |
|---|---|
| **No measured hospital load** | Every load figure traces to NREL ComStock simulation or synthetic generation. The [forecasting spec](../02-specifications/forecasting-requirements.md#21-inputs) flags this as the project's biggest data risk, and it is unresolved |
| **No electricity price series** | `price(t)` is a required simulator input ([§2.1](../02-specifications/simulator-io-interface.md#21-exogenous-time-series)) and the price forecaster needs ENTSO-E data. Nothing here |
| **No carbon intensity series** | `CO2_int(t)` likewise |
| **No hydrogen system data** | No electrolyser, fuel cell, or H₂ tank measurements anywhere |
| **Geographic split** | All datasets are **Boston** (BIDMC). The price forecaster targets the **Dutch** market, and the placeholder irradiance in the profiles workbook is **Eindhoven**. The two case studies have not been reconciled |
| **No licence/citation files** | MIMIC-IV demo (ODbL), NREL ComStock, NSRDB and PVWatts all have attribution expectations; none are recorded |
## Related
- [BIDMC Datasets Manual](../03-energy-management/bidmc-datasets-manual.md)
- [BIDMC ComStock Load Profile](../03-energy-management/bidmc-parquet-load-data-manual.md)
- [PV Hourly Dataset Manual](../03-energy-management/pv-hourly-dataset-manual.md) · [Satellite PV Dataset Manual](../03-energy-management/satellite-pv-dataset-manual.md)
- [Generation Profiles Workbook](../04-simulations/generation-profiles-workbook.md)
- [Forecasting Requirements](../02-specifications/forecasting-requirements.md)
Reference in a new issue View git blame Copy permalink