ALLSHIFT/docs/02-specifications/rl-reward-function.md

RL Reward Function Components

Markdown report of a non-markdown source document.

Source	docs/_originals/REWARD FUNCTION COMPONENTS AND DESCRIPTIONS.docx
Format	Microsoft Word (.docx), 17.0 kB
MD5	5d2828ea38008cbeaccd35b4672a9191
Owner	AI & Control Systems cluster
Status	Current — component list complete, weights undefined
Report generated	2026-07-25

What this document is

Twenty-six candidate terms for the reinforcement-learning reward function, each with a sign (reward or penalty) and a rationale. It enumerates what should be rewarded; it does not assign relative weights, and weighting is the entire difficulty of a multi-objective reward like this one.

The one ordering constraint the document does state is decisive: Penalty_UnservedCriticalLoad must dominate every other term. This is the reward-function expression of the project-wide rule that critical load is sacred.

Reward components

Economics

Term	Sign	Description
Cost_EnergyImport	−	Penalizes buying electricity from the grid; higher grid import cost per kWh leads to more negative reward.
Revenue_EnergyExport	+	Rewards selling excess electricity back to the grid; more export at high price increases positive reward.
Cost_DemandPeak	−	Penalizes high instantaneous grid power import peaks that could increase demand charges on the electricity bill.
Cost_FuelCellHydrogenUse	−	Penalizes hydrogen consumption by the fuel cell to reflect the value of stored hydrogen for future emergencies.
Cost_ElectrolyzerEnergyUse	−	Penalizes energy consumed by the electrolyzer so it does not waste electricity when hydrogen is not needed.

Service continuity — the dominant group

Term	Sign	Description
Penalty_UnservedCriticalLoad	−−	Very strong penalty when critical hospital loads are not fully supplied; represents risk to patient safety and must dominate the reward.
Penalty_UnservedNonCriticalLoad	−	Smaller penalty when non-critical loads are curtailed; this allows controlled load shedding but still discourages unnecessary discomfort.
Penalty_GridOutageViolation	−	Penalty applied when the microgrid fails to maintain supply during a grid outage despite having theoretical local resources to do so.
Reward_ResilienceScore	+	Composite reward for meeting resilience objectives during disturbances (grid outage, sudden load spikes, PV drops), e.g. no loss of critical load over an event window.

Battery health

Term	Sign	Description
Reward_BatterySOC_Stability	+	Rewards keeping the battery state of charge within a desirable mid-range band, avoiding both deep discharge and overcharge regions.
Penalty_BatteryDeepDischarge	−	Extra penalty when the battery SOC falls below a critical threshold (e.g. 20%), representing accelerated degradation and risk of blackout.
Penalty_BatteryHighSOC	−	Penalty when battery SOC stays too close to 100% for long periods, representing stress and lack of usable headroom.
Penalty_BatteryC_Rate	−	Penalizes very high charge or discharge power relative to capacity (high C-rates), which accelerates battery wear.

Hydrogen chain health

Term	Sign	Description
Penalty_FuelCellRamping	−	Penalizes rapid changes in fuel cell power setpoint to avoid thermal and mechanical stress on the stack.
Penalty_ElectrolyzerRamping	−	Penalizes large power swings in the electrolyzer to prevent efficiency loss and extend stack lifetime.
Penalty_HydrogenOverfill	−	Strong penalty when hydrogen tank level approaches or exceeds safe storage capacity.
Reward_HydrogenStrategicReserve	+	Rewards keeping hydrogen level above a minimum strategic reserve threshold for blackout resilience.

Sustainability

Term	Sign	Description
Reward_RenewableUtilization	+	Rewards using locally generated solar power instead of drawing from the grid, reflecting decarbonization and self-consumption goals.
Penalty_CurtailmentPV	−	Penalizes curtailing PV generation (turning down solar output) when there is still unserved load or available storage capacity.
Reward_EmissionReduction	+	Rewards operating modes that correspond to lower CO₂ emissions (e.g. less grid power during high-carbon periods, more use of PV and stored hydrogen).

Safety, stability, and comfort

Term	Sign	Description
Penalty_TemperatureLimits	−	Penalty if any component (battery, fuel cell, electrolyzer, inverter) operates outside its safe temperature band.
Penalty_SwitchingFrequency	−	Penalizes excessively frequent switching of modes (battery charge/discharge, fuel cell on/off, electrolyzer on/off) to avoid wear and control instability.
Penalty_VoltageFrequencyDeviation	−	Penalty when microgrid bus voltage or frequency deviates outside specified tolerance, representing power quality issues.
Penalty_ConstraintViolation	−	Generic penalty term that triggers when any hard operational constraint is violated (safety limits, reserve margins, equipment operating envelopes).
Reward_ComfortHVAC	+	Rewards keeping indoor climate (temperature or inferred comfort) within a comfortable range while still being energy efficient.

Forecast coupling

Term	Sign	Description
Reward_PredictionTracking	+	Rewards the controller when actual system evolution matches its internal forecasts (for load or PV), encouraging actions that reduce forecast error impact.

Open issues with this reward design

Four problems are worth resolving before any agent is trained against these terms.

1. No weights. Twenty-six terms with no relative magnitudes is not yet a reward function. Only one ordering constraint exists (Penalty_UnservedCriticalLoad dominates). Everything else is unranked, and a badly scaled multi-objective reward is the most common cause of degenerate RL policies.

2. Terms the simulator cannot currently compute. Several components require quantities the Simulator I/O Interface does not return:

Term	Missing simulator output
Penalty_TemperatureLimits	No component temperatures are returned (isothermal assumption)
Penalty_VoltageFrequencyDeviation	No bus voltage or frequency is modelled
Reward_ComfortHVAC	No thermal/HVAC model exists; HVAC is part of non-critical load
Penalty_BatteryC_Rate	Requires Q_rated, defined as static config but not returned per step
Reward_PredictionTracking	Requires the policy's internal forecast, which the simulator does not see

Either the simulator grows these outputs or these terms are dropped.

3. Penalty_ConstraintViolation may be unreachable. The simulator sheet says state limits are hard: setpoints breaching SoC or H2_level bounds are clipped, not permitted. If violations are clipped before they happen, a violation penalty never fires. The useful signal is the clipping magnitude (P_batt(t) vs u_batt(t)) — i.e. how far the policy tried to overreach — not a binary violation flag.

4. Double counting. Reward_ResilienceScore is explicitly a composite over disturbance events, but Penalty_UnservedCriticalLoad and Penalty_GridOutageViolation already price the same failures. Likewise Reward_RenewableUtilization and Penalty_CurtailmentPV reward opposite sides of one quantity, and Cost_EnergyImport overlaps Reward_EmissionReduction whenever carbon intensity tracks price. Overlapping terms are not fatal but they distort effective weighting in ways that are hard to reason about.

Related

• RL State Variables — the observation space
• RL Action Variables — the action space
• Simulator I/O Interface — what the simulator can actually return
• AI Efficiency Improvements — the loss targets the reward should attack