ALLSHIFT/docs/02-specifications/rl-action-variables.md

RL Action Variables

Markdown report of a non-markdown source document.

Source	docs/_originals/ACTION VARIABLES AND DESCRIPTIONS.docx
Format	Microsoft Word (.docx), 16.2 kB
MD5	9939b9c2f76fbb5bf54fbfeff25a31f4
Owner	AI & Control Systems cluster
Status	Current — but broader than the simulator currently accepts
Report generated	2026-07-25

What this document is

The action space for the reinforcement-learning controller: every lever the agent is allowed to pull at each timestep. Sixteen actions, spanning direct power setpoints, discrete mode selections, and demand-side commands.

Action variables

Generation and storage setpoints

Action	Description
PV_Setpoint	Command that limits or caps the PV inverter output to avoid overgeneration, match load, or reduce battery stress.
Battery_ChargePower	Amount of power (kW) the controller instructs the battery to absorb; used when excess solar or cheap grid power is available.
Battery_DischargePower	Amount of power (kW) the controller instructs the battery to release; used to support critical loads, reduce grid cost, or stabilize voltage.
Battery_Mode	Direct mode selection (charge, discharge, idle) that constrains what the battery is allowed to do in the next control interval.

Hydrogen chain

Action	Description
Electrolyzer_PowerSetpoint	Electrical power sent to the electrolyzer to control hydrogen production; higher power increases H₂ output but consumes more energy and heats the stack.
Electrolyzer_Mode	Operating mode such as standby, active production, cooldown, or shutdown; controls whether the electrolyzer is allowed to start producing hydrogen.
FuelCell_PowerSetpoint	Power command that determines how much hydrogen is converted back into electricity; used to supply critical loads during low-PV or grid failure conditions.
FuelCell_Mode	Operational mode (idle, ramp-up, nominal, emergency) that governs stack usage and prevents rapid degradation.
HydrogenTank_ValveCommand	Opening/closing command controlling the flow of hydrogen between the tank and fuel cell or between tank and buffer systems.

Grid exchange

Action	Description
Grid_ImportPower	Amount of electrical power drawn from the grid; controlled to minimize cost, follow load, or maintain battery SOC.
Grid_ExportPower	Amount of power sent back to the grid; used when PV or fuel cell production exceeds local demand and export is economically or operationally beneficial.

Demand side

Action	Description
Load_SheddingCommand	Decision to temporarily turn off certain non-critical loads to preserve battery SOC, maintain resilience, or avoid overloading.
Load_ShiftingCommand	Reschedules flexible loads (e.g., HVAC pre-cooling, non-critical equipment) to times of high solar availability or low grid pricing.
ThermalManagement_Command	Controls cooling or heating for PV inverters, electrolyzers, battery packs, and fuel cells to maintain safe temperatures.

Supervisory

Action	Description
Microgrid_Mode	High-level system mode (normal operation, island mode, emergency mode, cost-optimization mode, resilience mode) determining overall behavior and constraints.
Forecast_UpdateTrigger	Forces the agent to update internal predictions (PV forecast, load forecast) earlier than scheduled when sudden weather/load changes are detected.

Scope gap against the simulator contract

The Simulator I/O Interface defines a much narrower control input set — five signals, of which two are optional:

Simulator control input	Corresponding action here
u_ele(t) — electrolyser power setpoint, W	Electrolyzer_PowerSetpoint
u_fc(t) — fuel cell power setpoint, W	FuelCell_PowerSetpoint
u_batt(t) — signed battery setpoint, W	Battery_ChargePower + Battery_DischargePower collapsed into one signed value
sw_ele(t) — electrolyser ON/OFF (optional)	Electrolyzer_Mode (coarser)
sw_fc(t) — fuel cell ON/OFF (optional)	FuelCell_Mode (coarser)

Nine of the sixteen actions listed here have no simulator counterpart: PV_Setpoint, HydrogenTank_ValveCommand, Grid_ImportPower, Grid_ExportPower, Load_SheddingCommand, Load_ShiftingCommand, ThermalManagement_Command, Microgrid_Mode, and Forecast_UpdateTrigger.

Two of those gaps are deliberate design decisions recorded in the simulator sheet:

• Grid import/export is not an action. The simulator treats the grid as the slack bus — it absorbs whatever residual remains after every other component acts, bounded by P_grid_max and grid_on(t). Exposing Grid_ImportPower/Grid_ExportPower as independent actions would over-determine the power balance.
• Load shedding is an outcome, not an action. The simulator reports unmet_crit(t) and unmet_noncrit(t) as consequences of the setpoints, and overrides the policy outright if critical load would go unserved.

The rest — PV curtailment control, thermal management, mode supervision, forecast triggering — are genuine capability gaps between the two documents that need a decision.

Design questions still open

The simulator sheet raises two action-space questions this document does not answer:

1. Continuous or discrete? Continuous setpoints (u_ele ∈ [0, P_ele_max]) versus discrete buckets. This document implies continuous power setpoints alongside discrete *_Mode selections — i.e. a hybrid action space — but never says so explicitly.
2. Should simultaneous electrolyser and fuel cell operation be structurally forbidden (via a mode-switch action space) or merely penalised in the reward? This document's Electrolyzer_Mode and FuelCell_Mode are independent, which permits the physically wasteful both-on state.

Related

• RL State Variables — the observation space
• RL Reward Function — the reward terms
• Simulator I/O Interface — the typed simulator contract
• Rule-Based Controller — the baseline policy