# RL Reward Function Components

> **Markdown report of a non-markdown source document.**
>
> | | |
> |---|---|
> | **Source** | [`docs/_originals/REWARD FUNCTION COMPONENTS AND DESCRIPTIONS.docx`](../_originals/REWARD%20FUNCTION%20COMPONENTS%20AND%20DESCRIPTIONS.docx) |
> | **Format** | Microsoft Word (.docx), 17.0 kB |
> | **MD5** | `5d2828ea38008cbeaccd35b4672a9191` |
> | **Owner** | AI & Control Systems cluster |
> | **Status** | Current — component list complete, **weights undefined** |
> | **Report generated** | 2026-07-25 |

## What this document is

Twenty-six candidate terms for the reinforcement-learning reward function, each with a sign
(reward or penalty) and a rationale. It enumerates *what* should be rewarded; it does **not**
assign relative weights, and weighting is the entire difficulty of a multi-objective reward
like this one.

The one ordering constraint the document does state is decisive:
`Penalty_UnservedCriticalLoad` **must dominate** every other term. This is the reward-function
expression of the project-wide rule that *critical load is sacred*.

## Reward components

### Economics

| Term | Sign | Description |
|---|:---:|---|
| `Cost_EnergyImport` | − | Penalizes buying electricity from the grid; higher grid import cost per kWh leads to more negative reward. |
| `Revenue_EnergyExport` | + | Rewards selling excess electricity back to the grid; more export at high price increases positive reward. |
| `Cost_DemandPeak` | − | Penalizes high instantaneous grid power import peaks that could increase demand charges on the electricity bill. |
| `Cost_FuelCellHydrogenUse` | − | Penalizes hydrogen consumption by the fuel cell to reflect the value of stored hydrogen for future emergencies. |
| `Cost_ElectrolyzerEnergyUse` | − | Penalizes energy consumed by the electrolyzer so it does not waste electricity when hydrogen is not needed. |

### Service continuity — the dominant group

| Term | Sign | Description |
|---|:---:|---|
| `Penalty_UnservedCriticalLoad` | −− | **Very strong penalty** when critical hospital loads are not fully supplied; represents risk to patient safety and **must dominate the reward**. |
| `Penalty_UnservedNonCriticalLoad` | − | Smaller penalty when non-critical loads are curtailed; this allows controlled load shedding but still discourages unnecessary discomfort. |
| `Penalty_GridOutageViolation` | − | Penalty applied when the microgrid fails to maintain supply during a grid outage despite having theoretical local resources to do so. |
| `Reward_ResilienceScore` | + | Composite reward for meeting resilience objectives during disturbances (grid outage, sudden load spikes, PV drops), e.g. no loss of critical load over an event window. |

### Battery health

| Term | Sign | Description |
|---|:---:|---|
| `Reward_BatterySOC_Stability` | + | Rewards keeping the battery state of charge within a desirable mid-range band, avoiding both deep discharge and overcharge regions. |
| `Penalty_BatteryDeepDischarge` | − | Extra penalty when the battery SOC falls below a critical threshold (e.g. 20%), representing accelerated degradation and risk of blackout. |
| `Penalty_BatteryHighSOC` | − | Penalty when battery SOC stays too close to 100% for long periods, representing stress and lack of usable headroom. |
| `Penalty_BatteryC_Rate` | − | Penalizes very high charge or discharge power relative to capacity (high C-rates), which accelerates battery wear. |

### Hydrogen chain health

| Term | Sign | Description |
|---|:---:|---|
| `Penalty_FuelCellRamping` | − | Penalizes rapid changes in fuel cell power setpoint to avoid thermal and mechanical stress on the stack. |
| `Penalty_ElectrolyzerRamping` | − | Penalizes large power swings in the electrolyzer to prevent efficiency loss and extend stack lifetime. |
| `Penalty_HydrogenOverfill` | − | Strong penalty when hydrogen tank level approaches or exceeds safe storage capacity. |
| `Reward_HydrogenStrategicReserve` | + | Rewards keeping hydrogen level above a minimum strategic reserve threshold for blackout resilience. |

### Sustainability

| Term | Sign | Description |
|---|:---:|---|
| `Reward_RenewableUtilization` | + | Rewards using locally generated solar power instead of drawing from the grid, reflecting decarbonization and self-consumption goals. |
| `Penalty_CurtailmentPV` | − | Penalizes curtailing PV generation (turning down solar output) when there is still unserved load or available storage capacity. |
| `Reward_EmissionReduction` | + | Rewards operating modes that correspond to lower CO₂ emissions (e.g. less grid power during high-carbon periods, more use of PV and stored hydrogen). |

### Safety, stability, and comfort

| Term | Sign | Description |
|---|:---:|---|
| `Penalty_TemperatureLimits` | − | Penalty if any component (battery, fuel cell, electrolyzer, inverter) operates outside its safe temperature band. |
| `Penalty_SwitchingFrequency` | − | Penalizes excessively frequent switching of modes (battery charge/discharge, fuel cell on/off, electrolyzer on/off) to avoid wear and control instability. |
| `Penalty_VoltageFrequencyDeviation` | − | Penalty when microgrid bus voltage or frequency deviates outside specified tolerance, representing power quality issues. |
| `Penalty_ConstraintViolation` | − | Generic penalty term that triggers when any hard operational constraint is violated (safety limits, reserve margins, equipment operating envelopes). |
| `Reward_ComfortHVAC` | + | Rewards keeping indoor climate (temperature or inferred comfort) within a comfortable range while still being energy efficient. |

### Forecast coupling

| Term | Sign | Description |
|---|:---:|---|
| `Reward_PredictionTracking` | + | Rewards the controller when actual system evolution matches its internal forecasts (for load or PV), encouraging actions that reduce forecast error impact. |

## Open issues with this reward design

Four problems are worth resolving before any agent is trained against these terms.

**1. No weights.** Twenty-six terms with no relative magnitudes is not yet a reward function.
Only one ordering constraint exists (`Penalty_UnservedCriticalLoad` dominates). Everything else
is unranked, and a badly scaled multi-objective reward is the most common cause of degenerate
RL policies.

**2. Terms the simulator cannot currently compute.** Several components require quantities the
[Simulator I/O Interface](simulator-io-interface.md) does not return:

| Term | Missing simulator output |
|---|---|
| `Penalty_TemperatureLimits` | No component temperatures are returned (isothermal assumption) |
| `Penalty_VoltageFrequencyDeviation` | No bus voltage or frequency is modelled |
| `Reward_ComfortHVAC` | No thermal/HVAC model exists; HVAC is part of non-critical load |
| `Penalty_BatteryC_Rate` | Requires `Q_rated`, defined as static config but not returned per step |
| `Reward_PredictionTracking` | Requires the policy's internal forecast, which the simulator does not see |

Either the simulator grows these outputs or these terms are dropped.

**3. `Penalty_ConstraintViolation` may be unreachable.** The simulator sheet says state limits
are *hard*: setpoints breaching `SoC` or `H2_level` bounds are **clipped**, not permitted. If
violations are clipped before they happen, a violation penalty never fires. The useful signal
is the *clipping magnitude* (`P_batt(t)` vs `u_batt(t)`) — i.e. how far the policy tried to
overreach — not a binary violation flag.

**4. Double counting.** `Reward_ResilienceScore` is explicitly a composite over disturbance
events, but `Penalty_UnservedCriticalLoad` and `Penalty_GridOutageViolation` already price the
same failures. Likewise `Reward_RenewableUtilization` and `Penalty_CurtailmentPV` reward
opposite sides of one quantity, and `Cost_EnergyImport` overlaps `Reward_EmissionReduction`
whenever carbon intensity tracks price. Overlapping terms are not fatal but they distort
effective weighting in ways that are hard to reason about.

## Related

- [RL State Variables](rl-state-variables.md) — the observation space
- [RL Action Variables](rl-action-variables.md) — the action space
- [Simulator I/O Interface](simulator-io-interface.md) — what the simulator can actually return
- [AI Efficiency Improvements](../01-project/ai-efficiency-improvements.md) — the loss targets the reward should attack