reward design and evaluation strategy. This study compares three actor–critic RL algorithms: Soft Actor-Critic

(SAC), Advantage Actor-Critic (A2C), and Proximal Policy Optimization (PPO) for personalized insulin dosing

using real-world continuous glucose monitoring, insulin delivery, basal insulin, and meal intake data from the

OhioT1DM dataset. A custom Gymnasium-based environment was developed, and all algorithms were trained

under identical conditions for 100,000 timesteps. Performance was evaluated using cumulative reward together

with clinically relevant measures, including Time in Range (TIR) and insulin dosing behaviour. Although A2C

and PPO achieved higher cumulative rewards than SAC, both converged to near-zero insulin dosing policies that

exploited the reward formulation rather than learning clinically meaningful glucose regulation. In contrast, SAC

maintained adaptive dosing behaviour, achieving a TIR of 72.71% with an average insulin dose of 1.769 U/step.

These findings show that higher cumulative reward does not necessarily correspond to better clinical decision-

Type 1 diabetes mellitus (T1DM) requires continuous monitoring of blood glucose levels and timely insulin

administration to maintain glucose within a safe therapeutic range. Despite advances in continuous glucose

monitoring (CGM) and automated insulin delivery technologies, achieving optimal glycaemic control remains

difficult because glucose responses vary considerably among individuals and are influenced by factors such as

meals, physical activity, stress, and daily physiological changes. These challenges highlight the need for

decision-support systems capable of adapting insulin therapy to individual patient needs.

Reinforcement learning (RL) has emerged as a promising approach for personalized healthcare decision-making

simulators and real-world datasets. However, most existing work focuses primarily on improving glucose control

and cumulative reward, with limited attention to whether the learned policies represent clinically meaningful

behaviour. In particular, open-loop environments based on historical patient data introduce an important

challenge because an agent's actions do not influence subsequent physiological states. Under these conditions, a

policy may maximize reward by exploiting weaknesses in the reward function rather than learning an effective

insulin dosing strategy.

To investigate this challenge, this study compares three widely used actor-critic reinforcement-learning

algorithms: Soft Actor-Critic (SAC), Advantage Actor-Critic (A2C), and Proximal Policy Optimization (PPO)

for personalized insulin dosing using real-world data from the OhioT1DM dataset. A reproducible Gymnasium-

ii. An empirical investigation of reward hacking in an open-loop clinical reinforcement learning environment,

iii. A behaviour-focused evaluation framework that combines reward-based performance with clinical metrics

iv. Practical insights into reward design and evaluation strategies for developing safer reinforcement learning

By demonstrating how reward optimization can diverge from clinically meaningful decision-making, this study

contributes to the development of more reliable reinforcement learning methods for healthcare decision support.

dependent effects. Similarly, personalized glucose prediction models have demonstrated that individualized

approaches can improve glycaemic management compared with generalized models (Parveen, 2021).

Several actor-critic algorithms have been developed for continuous control problems. Mnih et al. (2016)

introduced the Advantage Actor-Critic (A2C) framework, while Schulman et al. (2017) proposed Proximal

Policy Optimization (PPO), which improves training stability through a clipped policy update mechanism.

Haarnoja et al. (2018) later introduced Soft Actor-Critic (SAC), an off-policy algorithm that incorporates entropy

regularization to improve exploration and policy robustness. Beyond these foundational methods, Zhao et al.

(2019) demonstrated that asynchronous deep reinforcement learning with dynamic weight updating can improve

learning efficiency in complex environments, while Zheng et al. (2025) proposed a reward-based prioritization

incorporated temporal dependencies to support personalized insulin management, while Dénes-Fazakas et al.

Despite these advances, important limitations remain. Many existing studies rely on physiological simulators

that simplify glucose-insulin dynamics and may not fully capture the variability observed in real-world clinical

data. Furthermore, evaluation is often centred on cumulative reward or glucose outcome metrics, providing

limited insight into whether the learned policies reflect clinically meaningful insulin dosing behaviour. As a

result, policies that achieve favourable numerical performance may still exploit weaknesses in the reward

formulation rather than learning appropriate treatment strategies. This study addresses these limitations by

comparing SAC, A2C, and PPO using real-world OhioT1DM data within a reproducible open-loop

reinforcement learning environment. Rather than evaluating algorithms solely on cumulative reward or glucose

outcomes, the study examines policy behaviour alongside clinical performance to determine whether the learned

dosing strategies are clinically meaningful. By explicitly investigating reward hacking in an open-loop clinical

This study utilized the OhioT1DM dataset, a publicly available real-world dataset containing continuous glucose

monitoring (CGM), insulin delivery, physiological measurements, and lifestyle information collected from

individuals with Type 1 diabetes mellitus (T1DM). The dataset has been widely adopted in diabetes research

because it provides high-resolution longitudinal data that capture real patient glucose management patterns.

To ensure a controlled and reproducible experimental setting, data from a single patient over a continuous one-

Missing CGM measurements were handled using forward filling based on the most recent observation, while

glucose values were constrained to a physiologically plausible range of 40–400 mg/dL to minimize the influence

of sensor artefacts and outliers. After preprocessing, the dataset comprised 8,928 sequential time steps with a

mean glucose level of 146.48 mg/dL (SD = 57.20), a mean basal insulin rate of 0.36 U/h, and a mean meal intake

of 2.72 kcal per five-minute interval. Because the dataset consists of historical patient observations, the resulting

A custom Gymnasium environment was developed to model personalized insulin dosing as a sequential decision-

making problem. At each decision step, the agent observes the patient's current physiological state and selects

an insulin dose with the objective of maintaining blood glucose within the recommended clinical range.

The observation space consists of five normalized features: three consecutive continuous glucose monitoring

(CGM) readings, the current basal insulin rate, and meal intake. Including multiple glucose measurements

enables the agent to capture short-term glucose trends rather than relying on a single observation.

The action space is continuous, allowing the agent to generate values between -1 and 1, which are linearly

dose) discourages unnecessary insulin administration. Because the environment is open-loop, insulin actions do

not alter subsequent glucose measurements. Consequently, minimizing insulin administration can increase

cumulative reward without improving glucose regulation, creating the conditions under which reward hacking

may occur.

Each training episode begins from a randomly selected point within the dataset to expose the agent to diverse

glucose patterns. Episodes continue for a maximum of 288 steps, representing 24 hours of patient data, and

terminate earlier if the end of the dataset is reached or glucose values fall outside predefined safety limits.

hyperparameters for its respective algorithm. No algorithm-specific hyperparameter tuning was performed so

that differences in performance could be attributed primarily to the learning algorithms rather than parameter

optimization. We acknowledge this as a limitation of the current study because default hyperparameters were

used uniformly. It remains possible that tuned variants of A2C or PPO could partially mitigate the zero-dosing

convergence reported in section 5, and this possibility is addressed further in section 6. During training, episode

reward, average glucose level, and average insulin dose were recorded to monitor learning behaviour and policy

convergence. Using identical training conditions for all three algorithms enabled a consistent comparison of both

reward optimization and learned insulin dosing behaviour.

mean glucose deviation, and average insulin dose were calculated. Combining reward-based and clinically

meaningful metrics enabled a comprehensive assessment of policy performance and helped identify cases where

high cumulative reward did not correspond to clinically appropriate insulin dosing.

Table 4 summarizes the average evaluation performance of the three reinforcement learning algorithms across

ten independent evaluation episodes. Although A2C and PPO achieved the highest mean cumulative reward

(53.60), both converged to near-zero insulin dosing policies. In contrast, SAC obtained a lower mean reward

(43.03) while maintaining an average insulin dose of 1.769 U per step.

Table 4: Mean evaluation performance of SAC, A2C, and PPO across ten evaluation episodes.

reward, whereas average glucose remained almost identical across all three algorithms. The principal difference

and PPO generated almost no insulin throughout the evaluation period.

These results indicate that reward-based performance differed substantially from the learned insulin dosing

behaviour.

patient data were replayed irrespective of the agent's actions. In contrast, the insulin dosing trajectories revealed

clear behavioural differences.

SAC generated adaptive insulin doses that varied in response to the observed glucose profile, indicating that the

learned policy actively responded to changes in the patient's physiological state. Conversely, both A2C and PPO

produced near-zero insulin doses throughout the evaluation period, regardless of changes in glucose levels. The

similarity of these two policies suggests that both on-policy algorithms converged to the same low-dosing

strategy despite differences in their optimization procedures.

Examining the evaluation trajectories provides additional insight into the behavioural differences among the

Table 5 presents the clinical performance of SAC, A2C, and PPO based on Time in Range (TIR), hypoglycaemia

rate, hyperglycaemia rate, mean glucose deviation, and average insulin dose. All three algorithms achieved

identical glucose-related metrics, including a TIR of 72.7%, because these measures were computed from the

same historical glucose trajectory replayed by the open-loop environment.

Although the glucose outcome metrics were identical, average insulin dose differed markedly across the

algorithms. SAC maintained an average insulin dose of 1.769 U per step, whereas both A2C and PPO

administered virtually no insulin during evaluation. This contrast indicates that similar clinical outcome metrics

did not necessarily reflect similar treatment policies.

Figure 6 further illustrates these findings. While SAC, A2C, and PPO exhibited identical values for Time in

Range, hypoglycaemia rate, hyperglycaemia rate, and mean glucose deviation, only the insulin dosing metric

distinguished the learned policies. The clinical metrics, therefore, should be interpreted alongside policy

behaviour.

Collectively, the results demonstrate that cumulative reward alone is insufficient for evaluating reinforcement

learning policies in open-loop clinical environments. Although A2C and PPO achieved higher rewards, both

converged to near-zero insulin dosing policies, whereas SAC maintained adaptive dosing behaviour despite

receiving a lower cumulative reward. These findings highlight the importance of combining reward-based,