pepe
72dd781dbc
Audit every document in the repository, convert the non-markdown ones into markdown reports, and split current documentation from outdated material. docs/ — 31 markdown documents in seven numbered sections. Twenty are new reports generated from .docx / .pdf / .xlsx / .mlx / .m sources that were previously unreadable in the browser and undiffable in git. Each report carries a provenance block (source path, format, MD5) and links back to its original; all 13 recorded checksums verify against the files on disk. Machine-extraction losses (PDF table column interleaving, Word OMML equations, embedded figures) are called out explicitly rather than silently smoothed over. superseded/ — outdated material with a documented reason per entry: two byte-identical ClickUp re-exports, an older revision of the BIDMC/UCSD energy-flow doc (the retained copy adds the SoC Violation Rate KPI), a duplicate of Shift input data.docx, the May 2026 simulation plan, the root PV+Battery.md now covered by a fuller report, GitHub's stock demo-repository template, and a zero-byte placeholder. Its README also records what was deliberately NOT retired and why — the "Old Frameworks" and "Old Simulations" folders hold unique Simulink revisions, and "Big Ugly Folder" holds the only copy of framework revision 1.3. Findings worth flagging, all documented in the reports: - Simulink lineage recovered from each .slx's internal coreProperties.xml revision counter. The current model is Current Framework/Bobert0206_Initial_Simulation_Framework.slx (rev 2.7); the top-level copy is rev 1.3, five revisions behind. - Simulations/Constants.m is a truncated byte-prefix of the Current Framework copy, silently missing H2_leak, H2_cap and E_H2_vol_h. - The PEM electrolyser and fuel cell are unmodified MathWorks Simscape examples still at vendor defaults; the "10x bigger" sizing TODO recorded in Constants.m was never carried out. - controller-claude.m does not compile — undefined P_Electro_max, outputs unassigned on several paths. - The specification set uses two incompatible variable naming conventions and disagrees on action-space size (5 vs 16). - MA_hourly_load.csv (13.7 MB) is the same 35,040 rows as 89993-0.parquet (2.4 MB). - Clinical data is the MIMIC-IV *demo* (ODbL, 100 patients), not full MIMIC-IV — redistributable, but the licence and citation are unrecorded. Housekeeping: untrack 21 Simulink build artefacts (slprj/, *.slxc) and add ignore rules for them. Root README rewritten around the new layout. Recruitment notes naming individual candidates are excluded from version control via .gitignore rather than committed; the generic question template is kept in docs/07-team-and-operations/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
128 lines
8.2 KiB
Markdown
128 lines
8.2 KiB
Markdown
# RL Reward Function Components
|
||
|
||
> **Markdown report of a non-markdown source document.**
|
||
>
|
||
> | | |
|
||
> |---|---|
|
||
> | **Source** | [`docs/_originals/REWARD FUNCTION COMPONENTS AND DESCRIPTIONS.docx`](../_originals/REWARD%20FUNCTION%20COMPONENTS%20AND%20DESCRIPTIONS.docx) |
|
||
> | **Format** | Microsoft Word (.docx), 17.0 kB |
|
||
> | **MD5** | `5d2828ea38008cbeaccd35b4672a9191` |
|
||
> | **Owner** | AI & Control Systems cluster |
|
||
> | **Status** | Current — component list complete, **weights undefined** |
|
||
> | **Report generated** | 2026-07-25 |
|
||
|
||
## What this document is
|
||
|
||
Twenty-six candidate terms for the reinforcement-learning reward function, each with a sign
|
||
(reward or penalty) and a rationale. It enumerates *what* should be rewarded; it does **not**
|
||
assign relative weights, and weighting is the entire difficulty of a multi-objective reward
|
||
like this one.
|
||
|
||
The one ordering constraint the document does state is decisive:
|
||
`Penalty_UnservedCriticalLoad` **must dominate** every other term. This is the reward-function
|
||
expression of the project-wide rule that *critical load is sacred*.
|
||
|
||
## Reward components
|
||
|
||
### Economics
|
||
|
||
| Term | Sign | Description |
|
||
|---|:---:|---|
|
||
| `Cost_EnergyImport` | − | Penalizes buying electricity from the grid; higher grid import cost per kWh leads to more negative reward. |
|
||
| `Revenue_EnergyExport` | + | Rewards selling excess electricity back to the grid; more export at high price increases positive reward. |
|
||
| `Cost_DemandPeak` | − | Penalizes high instantaneous grid power import peaks that could increase demand charges on the electricity bill. |
|
||
| `Cost_FuelCellHydrogenUse` | − | Penalizes hydrogen consumption by the fuel cell to reflect the value of stored hydrogen for future emergencies. |
|
||
| `Cost_ElectrolyzerEnergyUse` | − | Penalizes energy consumed by the electrolyzer so it does not waste electricity when hydrogen is not needed. |
|
||
|
||
### Service continuity — the dominant group
|
||
|
||
| Term | Sign | Description |
|
||
|---|:---:|---|
|
||
| `Penalty_UnservedCriticalLoad` | −− | **Very strong penalty** when critical hospital loads are not fully supplied; represents risk to patient safety and **must dominate the reward**. |
|
||
| `Penalty_UnservedNonCriticalLoad` | − | Smaller penalty when non-critical loads are curtailed; this allows controlled load shedding but still discourages unnecessary discomfort. |
|
||
| `Penalty_GridOutageViolation` | − | Penalty applied when the microgrid fails to maintain supply during a grid outage despite having theoretical local resources to do so. |
|
||
| `Reward_ResilienceScore` | + | Composite reward for meeting resilience objectives during disturbances (grid outage, sudden load spikes, PV drops), e.g. no loss of critical load over an event window. |
|
||
|
||
### Battery health
|
||
|
||
| Term | Sign | Description |
|
||
|---|:---:|---|
|
||
| `Reward_BatterySOC_Stability` | + | Rewards keeping the battery state of charge within a desirable mid-range band, avoiding both deep discharge and overcharge regions. |
|
||
| `Penalty_BatteryDeepDischarge` | − | Extra penalty when the battery SOC falls below a critical threshold (e.g. 20%), representing accelerated degradation and risk of blackout. |
|
||
| `Penalty_BatteryHighSOC` | − | Penalty when battery SOC stays too close to 100% for long periods, representing stress and lack of usable headroom. |
|
||
| `Penalty_BatteryC_Rate` | − | Penalizes very high charge or discharge power relative to capacity (high C-rates), which accelerates battery wear. |
|
||
|
||
### Hydrogen chain health
|
||
|
||
| Term | Sign | Description |
|
||
|---|:---:|---|
|
||
| `Penalty_FuelCellRamping` | − | Penalizes rapid changes in fuel cell power setpoint to avoid thermal and mechanical stress on the stack. |
|
||
| `Penalty_ElectrolyzerRamping` | − | Penalizes large power swings in the electrolyzer to prevent efficiency loss and extend stack lifetime. |
|
||
| `Penalty_HydrogenOverfill` | − | Strong penalty when hydrogen tank level approaches or exceeds safe storage capacity. |
|
||
| `Reward_HydrogenStrategicReserve` | + | Rewards keeping hydrogen level above a minimum strategic reserve threshold for blackout resilience. |
|
||
|
||
### Sustainability
|
||
|
||
| Term | Sign | Description |
|
||
|---|:---:|---|
|
||
| `Reward_RenewableUtilization` | + | Rewards using locally generated solar power instead of drawing from the grid, reflecting decarbonization and self-consumption goals. |
|
||
| `Penalty_CurtailmentPV` | − | Penalizes curtailing PV generation (turning down solar output) when there is still unserved load or available storage capacity. |
|
||
| `Reward_EmissionReduction` | + | Rewards operating modes that correspond to lower CO₂ emissions (e.g. less grid power during high-carbon periods, more use of PV and stored hydrogen). |
|
||
|
||
### Safety, stability, and comfort
|
||
|
||
| Term | Sign | Description |
|
||
|---|:---:|---|
|
||
| `Penalty_TemperatureLimits` | − | Penalty if any component (battery, fuel cell, electrolyzer, inverter) operates outside its safe temperature band. |
|
||
| `Penalty_SwitchingFrequency` | − | Penalizes excessively frequent switching of modes (battery charge/discharge, fuel cell on/off, electrolyzer on/off) to avoid wear and control instability. |
|
||
| `Penalty_VoltageFrequencyDeviation` | − | Penalty when microgrid bus voltage or frequency deviates outside specified tolerance, representing power quality issues. |
|
||
| `Penalty_ConstraintViolation` | − | Generic penalty term that triggers when any hard operational constraint is violated (safety limits, reserve margins, equipment operating envelopes). |
|
||
| `Reward_ComfortHVAC` | + | Rewards keeping indoor climate (temperature or inferred comfort) within a comfortable range while still being energy efficient. |
|
||
|
||
### Forecast coupling
|
||
|
||
| Term | Sign | Description |
|
||
|---|:---:|---|
|
||
| `Reward_PredictionTracking` | + | Rewards the controller when actual system evolution matches its internal forecasts (for load or PV), encouraging actions that reduce forecast error impact. |
|
||
|
||
## Open issues with this reward design
|
||
|
||
Four problems are worth resolving before any agent is trained against these terms.
|
||
|
||
**1. No weights.** Twenty-six terms with no relative magnitudes is not yet a reward function.
|
||
Only one ordering constraint exists (`Penalty_UnservedCriticalLoad` dominates). Everything else
|
||
is unranked, and a badly scaled multi-objective reward is the most common cause of degenerate
|
||
RL policies.
|
||
|
||
**2. Terms the simulator cannot currently compute.** Several components require quantities the
|
||
[Simulator I/O Interface](simulator-io-interface.md) does not return:
|
||
|
||
| Term | Missing simulator output |
|
||
|---|---|
|
||
| `Penalty_TemperatureLimits` | No component temperatures are returned (isothermal assumption) |
|
||
| `Penalty_VoltageFrequencyDeviation` | No bus voltage or frequency is modelled |
|
||
| `Reward_ComfortHVAC` | No thermal/HVAC model exists; HVAC is part of non-critical load |
|
||
| `Penalty_BatteryC_Rate` | Requires `Q_rated`, defined as static config but not returned per step |
|
||
| `Reward_PredictionTracking` | Requires the policy's internal forecast, which the simulator does not see |
|
||
|
||
Either the simulator grows these outputs or these terms are dropped.
|
||
|
||
**3. `Penalty_ConstraintViolation` may be unreachable.** The simulator sheet says state limits
|
||
are *hard*: setpoints breaching `SoC` or `H2_level` bounds are **clipped**, not permitted. If
|
||
violations are clipped before they happen, a violation penalty never fires. The useful signal
|
||
is the *clipping magnitude* (`P_batt(t)` vs `u_batt(t)`) — i.e. how far the policy tried to
|
||
overreach — not a binary violation flag.
|
||
|
||
**4. Double counting.** `Reward_ResilienceScore` is explicitly a composite over disturbance
|
||
events, but `Penalty_UnservedCriticalLoad` and `Penalty_GridOutageViolation` already price the
|
||
same failures. Likewise `Reward_RenewableUtilization` and `Penalty_CurtailmentPV` reward
|
||
opposite sides of one quantity, and `Cost_EnergyImport` overlaps `Reward_EmissionReduction`
|
||
whenever carbon intensity tracks price. Overlapping terms are not fatal but they distort
|
||
effective weighting in ways that are hard to reason about.
|
||
|
||
## Related
|
||
|
||
- [RL State Variables](rl-state-variables.md) — the observation space
|
||
- [RL Action Variables](rl-action-variables.md) — the action space
|
||
- [Simulator I/O Interface](simulator-io-interface.md) — what the simulator can actually return
|
||
- [AI Efficiency Improvements](../01-project/ai-efficiency-improvements.md) — the loss targets the reward should attack
|