pepe
72dd781dbc
Audit every document in the repository, convert the non-markdown ones into markdown reports, and split current documentation from outdated material. docs/ — 31 markdown documents in seven numbered sections. Twenty are new reports generated from .docx / .pdf / .xlsx / .mlx / .m sources that were previously unreadable in the browser and undiffable in git. Each report carries a provenance block (source path, format, MD5) and links back to its original; all 13 recorded checksums verify against the files on disk. Machine-extraction losses (PDF table column interleaving, Word OMML equations, embedded figures) are called out explicitly rather than silently smoothed over. superseded/ — outdated material with a documented reason per entry: two byte-identical ClickUp re-exports, an older revision of the BIDMC/UCSD energy-flow doc (the retained copy adds the SoC Violation Rate KPI), a duplicate of Shift input data.docx, the May 2026 simulation plan, the root PV+Battery.md now covered by a fuller report, GitHub's stock demo-repository template, and a zero-byte placeholder. Its README also records what was deliberately NOT retired and why — the "Old Frameworks" and "Old Simulations" folders hold unique Simulink revisions, and "Big Ugly Folder" holds the only copy of framework revision 1.3. Findings worth flagging, all documented in the reports: - Simulink lineage recovered from each .slx's internal coreProperties.xml revision counter. The current model is Current Framework/Bobert0206_Initial_Simulation_Framework.slx (rev 2.7); the top-level copy is rev 1.3, five revisions behind. - Simulations/Constants.m is a truncated byte-prefix of the Current Framework copy, silently missing H2_leak, H2_cap and E_H2_vol_h. - The PEM electrolyser and fuel cell are unmodified MathWorks Simscape examples still at vendor defaults; the "10x bigger" sizing TODO recorded in Constants.m was never carried out. - controller-claude.m does not compile — undefined P_Electro_max, outputs unassigned on several paths. - The specification set uses two incompatible variable naming conventions and disagrees on action-space size (5 vs 16). - MA_hourly_load.csv (13.7 MB) is the same 35,040 rows as 89993-0.parquet (2.4 MB). - Clinical data is the MIMIC-IV *demo* (ODbL, 100 patients), not full MIMIC-IV — redistributable, but the licence and citation are unrecorded. Housekeeping: untrack 21 Simulink build artefacts (slprj/, *.slxc) and add ignore rules for them. Root README rewritten around the new layout. Recruitment notes naming individual candidates are excluded from version control via .gitignore rather than committed; the generic question template is kept in docs/07-team-and-operations/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
110 lines
6.3 KiB
Markdown
110 lines
6.3 KiB
Markdown
# Satellite PV / Meteorological Dataset Manual (2019)
|
||
|
||
> **Markdown report of a non-markdown source document.**
|
||
>
|
||
> | | |
|
||
> |---|---|
|
||
> | **Source** | `Shift Matlab Drive/Shift Matlab Drive/Energy_Managment/Sattelite PV data/Data-set2/readme.docx` |
|
||
> | **Format** | Microsoft Word (.docx) |
|
||
> | **MD5** | `1843622a8681b1814ce6777a6d5360b7` |
|
||
> | **Describes** | `2019-1338340-one_axis.csv` — **167.9 MB**, the largest file in the repository |
|
||
> | **Owner** | Energy Management cluster |
|
||
> | **Status** | Current — but **not recommended** for PV rate extraction |
|
||
> | **Report generated** | 2026-07-25 |
|
||
|
||
## Summary
|
||
|
||
Very detailed satellite meteorological data from **Boston, Massachusetts**, for **2019**.
|
||
Hourly performance rates for solar panels could be derived from this data, but that would
|
||
require thorough analysis and effort.
|
||
|
||
> *"This dataset is not recommended for extracting PV rates, but it is perfect for other more
|
||
> detailed meteorological variables that could be useful down the line."*
|
||
|
||
For PV output, use the [PV hourly dataset](pv-hourly-dataset-manual.md) instead.
|
||
|
||
The `one_axis` in the filename indicates a **single-axis tracking** array configuration. The
|
||
folder name `Sattelite` is a typo for *Satellite* preserved from the original.
|
||
|
||
## What the dataset covers
|
||
|
||
The dataset tracks the physical environment of the solar array across four distinct categories:
|
||
|
||
- **Atmospheric / weather data** (pressure, humidity, dew point, precipitable water) — measure
|
||
the state of the air around the project, which directly affects how much sunlight can pass
|
||
through the atmosphere.
|
||
- **Solar geometry / angles** (zenith, azimuth, panel tilt) — track the geometry of the system:
|
||
where the sun is in the sky versus which way the panels face.
|
||
- **Irradiance data** (GHI, DNI, DHI, clear-sky variants) — the raw energy resource, measuring
|
||
the actual power of sunlight hitting the earth under both real-world and perfectly clear skies.
|
||
- **Technology performance** (Si, GaAs, CdTe, cell temperature) — the hardware's physical
|
||
response, showing how different solar panel chemistries perform and heat up under those
|
||
environmental conditions.
|
||
|
||
## Column reference
|
||
|
||
### Atmospheric and location
|
||
|
||
| Column | Definition | Units |
|
||
|---|---|---|
|
||
| **Solar Zenith Angle** | The angle between the sun's position in the sky and the point directly overhead (the zenith). If the sun is straight up the angle is 0; if it is on the horizon, 90 | Degrees |
|
||
| **Surface Albedo** | The fraction of sunlight reflected by the ground (grass, snow, asphalt) back up into the atmosphere or onto the panels | — |
|
||
| **Precipitable Water** | The total amount of water vapour in a vertical column of the atmosphere if it were all to condense and fall as rain. Higher values mean more moisture in the air, which absorbs specific wavelengths of sunlight before they reach the panels | cm |
|
||
| **Solar Azimuth Angle** | The compass direction of the sun in the sky. Usually measured from due North (0°) moving clockwise — East 90°, South 180°, West 270° | Degrees |
|
||
|
||
### Solar panel technologies
|
||
|
||
| Technology | Description |
|
||
|---|---|
|
||
| **Si** (BPR, Wacker, Eurosil) | Different types or manufacturing variations of standard silicon solar cells |
|
||
| **GaAs** (Gallium Arsenide) | A highly advanced, very efficient, but very expensive solar cell material. Because of the high cost, GaAs panels are primarily used in space satellites and aerospace applications |
|
||
| **InGaP** (Indium Gallium Phosphide) | Another high-efficiency semiconductor material, often layered with GaAs in *multijunction* solar cells to capture different colours of the solar spectrum |
|
||
| **CdTe** (Cadmium Telluride) | A common thin-film solar panel technology. These panels are flexible and perform well in low-light conditions, often used in large industrial-scale solar farms |
|
||
|
||
### Panel setup
|
||
|
||
| Column | Definition | Units |
|
||
|---|---|---|
|
||
| **Panel Tilt** | The angle at which the panels are tilted up from flat ground. A tilt of 0 means completely flat; 90 means standing straight up like a wall | Degrees |
|
||
| **Panel Azimuth** | The compass direction the face of the panels points. In the Northern Hemisphere panels are typically faced South (azimuth 180°) to capture maximum daily sunlight | Degrees |
|
||
|
||
### Clear-sky vs standard irradiance
|
||
|
||
| Column group | Meaning |
|
||
|---|---|
|
||
| **Standard** (DHI / DNI / GHI) | The actual or predicted solar radiation hitting the ground, taking real-world weather, clouds, and smog into account |
|
||
| **Clearsky** (Clearsky DHI / DNI / GHI) | Theoretical calculation of how much solar radiation would hit the panels if the sky were 100% perfectly clear with zero clouds or pollution |
|
||
|
||
## Why this dataset still matters
|
||
|
||
Despite the "not recommended" verdict for PV rates, it is the **only** source in the repository
|
||
for several features the
|
||
[forecasting spec](../02-specifications/forecasting-requirements.md#11-inputs) asks for:
|
||
|
||
| Forecasting feature | Only available here |
|
||
|---|---|
|
||
| Clear-sky index (`actual GHI / clear-sky GHI`) | ✅ — needs both GHI and Clearsky GHI, and only this dataset has clear-sky |
|
||
| Relative humidity | ✅ |
|
||
| Solar zenith and azimuth angles | ✅ (otherwise compute via `pvlib`) |
|
||
| Precipitable water, dew point, pressure | ✅ |
|
||
|
||
The spec notes that adding solar-position features **reduced RMSE by 13.1%** in the reference
|
||
study, and that the clear-sky index *"normalises seasonal/diurnal trends; stabilises model
|
||
variance"*. Both come from here.
|
||
|
||
## Size caveat
|
||
|
||
At **167.9 MB**, `2019-1338340-one_axis.csv` is by a wide margin the largest file in the
|
||
repository — more than 12× the next largest. It dominates clone time and repository size for
|
||
everyone on the team.
|
||
|
||
Because it covers a single year at hourly resolution for one location, most of that bulk is the
|
||
per-technology performance columns (Si/GaAs/InGaP/CdTe variants), which SHIFT does not use — the
|
||
project models a single PV array with a scalar efficiency `mu_PV`. Extracting the ~15 columns
|
||
the forecasting stack actually needs would shrink it by well over an order of magnitude.
|
||
|
||
## Related
|
||
|
||
- [PV Hourly Dataset Manual](pv-hourly-dataset-manual.md) — the recommended PV source
|
||
- [Forecasting Requirements](../02-specifications/forecasting-requirements.md) — §1, PV generation forecasting
|
||
- [Dataset Inventory](../06-data/dataset-inventory.md) — every data file in the repo, with sizes
|