Assimilating Human Feedback from Autonomous Vehicle Interaction in Reinforcement Learning Models

doi:10.21203/rs.3.rs-3405901/v1

Download PDF

Research Article

Assimilating Human Feedback from Autonomous Vehicle Interaction in Reinforcement Learning Models

https://doi.org/10.21203/rs.3.rs-3405901/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

A significant challenge for real-world automated vehicles (AVs) is their interaction with human pedestrians. This paper develops a methodology to directly elicit the AV behaviour pedestrians find suitable by collecting quantitative data that can be used to measure and improve an algorithm's performance. Starting with a Deep Q Network (DQN) trained on a simple Pygame/Python-based pedestrian crossing environment, the reward structure was adapted to allow adjustment by human feedback. Feedback was collected by eliciting behavioural judgements collected from people in a controlled environment. The reward was shaped by the inter-action vector, decomposed into feature aspects for relevant behaviours, thereby facilitating both implicit preference selection and explicit task discovery in tandem. Using computational RL and behavioural-science techniques, we harness a formal iterative feedback loop where the rewards are repeatedly adapted based on human behavioural judgments. Experiments were conducted with 124 participants that showed strong initial improvement in the judgement of AV behaviours with the adaptive reward structure. The results indicate that the primary avenue for enhancing vehicle behaviour lies in the predictability of its movements when introduced. More broadly, recognising AV behaviours that receive favourable human judgments can pave the way for enhanced performance.

Preference Learning

Autonomous Driving

Human Factors

A primary concern of automation is its ability to integrate with existing systems, which, when considering road usage, have a strong social element [1]. On top of optimising aspects of driving that naturally lend themselves to reward metrics, i.e., time to destination, fuel efficiency and lane discipline [2], vehicle automation must learn valid and useful behaviours when interacting with pedestrians and other road users [3], [4]. A key challenge researchers face in building such an agent is designing a suitable reward function to elicit such behaviours [5]. What are these behaviours, and how does one translate that to an RL algorithm? This paper addresses this question by developing a new methodology for embedding human judgements in an iterative-feedback loop for adjusting the reward function for an RL agent.

Most approaches to embedding human judgments into autonomous RL agents focus on the efficiency of an algorithm to minimise utilitarian measures, such as time to collision, time to destination and fuel efficiency. For example, Knox et al. [5] provide an overview of shaped reward functions used in work on autonomous vehicle optimisation, defining several categories of reward, one of which is human preferences. The system being analysed often has many complex inputs [6], [7], as an autonomous vehicle would have. Still, an equally complex social environment exists underneath [3], [4], which the agent can only access through the medium of human feedback.

A second related approach is preference learning, which allows the agent to learn from an expert’s ranking over a set of actions. The basic approach is to elicit feedback to learn a policy, using a pair-wise trajectory preference as a justification of policy quality. This approach, however, often requires experts in narrow and highly skilled domains. To alleviate the limitation of pair-wise preferences to inform on policy quality, there has been work that allows for the agent to use the expert’s rankings to extrapolate new behaviours [8]. By seeking explanatory patterns for the preferences, the best policy even exceeded the best performance that was demonstrated. In other work, Jain et al. [9] allowed their model to learn context-dependent behaviours much more efficiently, using a reward-based utility function without an explicit cost function to be optimised. Instead, their model used the preference feedback to compute a gradient of the utility parameters directly, permitting the policy to be defined by the RL agent. Creating such a utility function can be especially useful for the current project because feedback from pedestrians is not deterministic nor even necessarily stationary.

When directly considering how pedestrians interact with AVs, Jayaraman et al. showed that pedestrians’ acceptance of AVs depends on their trust in the AVs [10]. They conducted an objective-based evaluation of behaviours [11], in terms of safety, performance, and comfort with 30 human participants in a virtual-reality environment. They found that pedestrians’ trust in AVs was influenced by AV driving behaviour as well as the presence of a signal light. In [11], the authors propose a model representing the three objectives–safety, efficiency, and comfort–by the weights of a linear regression of observable variables, including latent trust variables and other experimental considerations. The intuition behind these objectives is that driving behaviours can be characterised by weighting values, especially as there is a competing optimal weighting between AV passengers’ preference and the pedestrian’s preference, both of which should be optimised.

This paper develops a methodology to directly ask what behaviour pedestrians find suitable by collecting quantitative data that can be used to measure an algorithm's performance and assumes that the performance of an AV aligns with its passengers’ preferences. The paper then shows improvements, in the judgement of the pedestrians, in “human-like” behaviour when such systematic feedback is collected from human agents and embedded into the reward function. The technique harnesses a formal iterative feedback loop using computational RL and behavioural science techniques, where the reward structure is adapted by eliciting behavioural judgements collected from people in a controlled environment. The policy is updated by training with the updated reward and then shown, without explicit participant knowledge, back to the participant to assess the updated behaviour. The reward function is based on three terms gleaned from pilot experiment data [12], in which participants were asked to judge aspects of the vehicle’s behaviour. Based on the current human feedback, an agent that was trained with each one of these three terms in the reward function altered to match that feedback is selected and the behaviour of the resultant policy is then presented for judgement in another round of feedback.

Agents were trained with Deep Q Network (DQN) [13] models as we wish to use the simplest methods to control complications around inference of what is altered by changing the reward function and on what participants are giving judgements. Given that the state space of our custom Pygame [14] simulator1 is large, based on pixel coordinates, we still require a deep approach as the simpler tabular methods would be computationally intractable. The DQN was trained on a simple Python-based Pygame pedestrian crossing environment, and the reward structure was adapted by eliciting behavioural judgements collected from people in a controlled environment. We elicit participant judgements, which are subjective ratings on vehicle driving quality, rather than eliciting a preference between multiple given vehicle trajectories. Participants, thus, are not considered experts; we regard them as normal pedestrians and guide the reward function by their judgements.

In the present work, not only are judgments elicited, but the participants are also fully in control of the pedestrian in the simulator using the keyboard. Allowing control of the pedestrian not only allows for an interactive experience and implicit understanding of the environment dynamics and vehicle behaviour, but also allows for the contrast of judgements reported with the behaviour of the participant. Such a comparison enables further analysis of the reported judgements; if a participant rates the vehicle as safe, do they act more confidently, or is there a contradiction between ratings and behaviour? The ultimate goal is to craft autonomous driving policies that resonate with human expectations and instincts—intertwining the realms of behavioural science and reinforcement learning, with a view to ensuring that the future of autonomous driving is not just technologically advanced, but also intrinsically human-centric.

[1] https://github.com/Rik-Fox/pygame_ped_env

In developing and testing autonomous vehicles, simulators play an essential role in providing controlled environments for experimentation. To address specific research requirements, a custom driving simulator was developed using Pygame, a cross-platform set of Python modules designed for writing video games. Leveraging the capabilities of the Pygame library was instrumental in defining the basic entities within our simulation environment, which operates on Pygames grid system at its core. This simulated driving environment was structured to conform to OpenAI Gym's specifications to enhance the simulator's compatibility with a range of reinforcement-learning algorithms. The custom simulator¹ offers a blend of simplicity and functionality meeting the specific needs of our research.

The simulator was designed with two primary modes of operation: a render mode and a headless mode. Figure 1 depicts the rendered display as shown to participants in behavioural experiments. The render mode was developed for visualising the interactions between the autonomous vehicle and the pedestrian. This visual representation provides a user-friendly interface where participants can visually discern the road, the car, and their avatar as a pedestrian. Open-source assets¹ were employed in this mode to ensure an aesthetically pleasing environment, facilitating participant immersion in the simulation. Conversely, the headless mode operates without any graphical display, offering a significant advantage during the training phase of the autonomous vehicle. In this mode, the environment was essentially reduced to four primary rectangles representing the pedestrian, the car, and the vertical and horizontal stretches of the road.

The vehicle's observations within this environment were straightforward yet effective. The vehicle perceived bounding rectangles, or 'Rects', for both itself and the pedestrian. These Rects were defined within Pygame's internal grid system, ensuring precision and consistency in the vehicle's perception. Additionally, the vehicle was provided with a clear goal, indicating the coordinates it should aim for from its starting position. The simulated actions were defined as one of the four cardinal and four intercardinal directions at a speed of 1,2 or 3 grid-squares/pixels per computational time step or a no-move action resulting in 25 total actions.

The other important modelling features, in the context of the simulator, were if a vehicle collides with a pedestrian, the episode concludes immediately, and a large negative reward is allocated to that state. In addition, agents were granted an augmented positive reward when they intersected with their designated goal coordinates. The DQN loss function was computed over actions with a lower cost for actions considered best. As the action space is discrete with negative rewards, there was the possibility of an infinite valued loss-term arising, in the event of a pedestrian collision. The model avoided such infinite-valued loss-term issues because there was always an action which avoids a collision by having pedestrians limited to a move speed of 2 while the vehicle can move at a speed of 3. Combined with either the shaped or constant reward, there was always an action better than colliding with a pedestrian to which the policy could converge. When the task was successfully accomplished, the reward given was positive and matched the absolute value of the smallest possible reward the agent could obtain while still reaching the goal, meaning that a spatially optimal trajectory received a reward of 0.

Additionally, the simulator considered only two sets of starting points and goals, representing driving directions from right to left and vice versa. The maximum episode duration was set to the distance to the goal plus one time step. This constraint ensured that the agent-controlled vehicle can reach its destination by moving at the slowest pace in a direct path. By implementing a negative reward system and concluding the task upon successful completion, this approach ensured that agents were motivated to identify the most efficient routes. This approach aims to terminate episodes swiftly, thereby reducing the accrual of negative rewards.

During the preliminary stages of model training and testing, the DQN models struggled to develop an effective driving policy. The models exhibited a tendency to achieve only slight improvement in rewards without demonstrating any genuine driving behaviour. A common tactic adopted by the models was to drive off-screen, thereby avoiding the accumulation of penalties immediately. Rather than further refining the reward structure to address this challenge, which could potentially obfuscate the desired behavioural outcomes, a technique known as action-masking was implemented [15]. Action-masking restricts an agent's choices by allowing it to execute only valid actions in a given state. To facilitate this constraint, the simulator was enhanced with built-in action masking functionality. This addition ensured that for every state presented to the agent, a corresponding action mask was provided. Actions that would result in the vehicle exiting the screen were deemed invalid and were consequently masked out, preventing the agent from selecting them. This approach aimed to guide the DQN models towards more realistic and effective driving behaviours without overly complicating the reward structure.

Reward Design and RL Training

The agent’s goal is to find the best set of actions $A$ given the states $S$ visited, referred to as the optimal policy ${\pi }^{\text{*}}\left(s\right),$ where policy function $\pi$ returns an action $a\in A$ to be taken in the state $s\in S$ of the environment of interest. A trajectory${\tau }^{\pi }=\left\{{\left(s,a=\pi (s\right))}_{t}\right\} \forall t\in \left\{\text{1,2},\cdots ,T\right\}$is the set of state-action pairs returned by policy $\pi$ for a given terminated instance of environment interaction over time $T$. The total expected return from a trajectory is:

$${R}_{{\tau }^{\pi }}= {\sum }_{t=0}^{T}r\left({s}_{t}\right) {s}_{t}\in {\tau }^{\pi }$$

( 1 )

which is general to the theory of RL. For this work, however, we also consider a shaped reward, based on the inter-action vector, decomposed into reward for relevant behaviours. The reward function is determined by three terms, providing rewards for speed ${r}_{\nu }$, proximity to destination ${r}_{\delta }$, and rate of heading change ${r}_{{\Delta }}$. These are summed and scaled to give us the full reward for a given state $r\left(s\right)$:

$$r\left({s}_{t}\right)={r}_{\nu }+{r}_{\delta }+{r}_{{\Delta }}-3=\frac{{\nu }_{t}}{{\nu }_{max}}+\left(1-\frac{{\delta }_{t}}{{\delta }_{max}}\right)+\left(1-\frac{{{\Delta }}_{t}}{2\pi }\right)-3,$$

( 2 )

where ${\nu }_{t}$ is the speed of the vehicle at time $t$, and ${{\Delta }}_{t}$ is the amount of heading change or the amount of steering at time $t$; both are determined by the previous action ${a}_{t-1}$ which is a dependence on the previous state-action pair as Markovian dynamics allows. The term ${\delta }_{t}$ is the distance from the state that the vehicle agent is currently in ${s}_{t}$, to the agents given destination. Each of the three terms are in the interval $\left(\text{0,1}\right]$ with $1$ being the optimal value, and we apply a negative $3$ to our reward to ensure that the reward is always negative, making an expected reward of $0$ the theoretically optimal solution.

All three of these quantities are required to be within the range $\left[\text{0,1}\right]$, to avoid violating constraints for optimisation that we discuss further below and we restrict this further to $\left(\text{0,1}\right]$ to avoid numerical errors in computation with variables that are exactly $0$. As such, they are normalised by their maximums, ${\nu }_{max},{\delta }_{max}$ and $2\pi$ in order of appearance. The term ${{\Delta }}_{t}$ is a radian angle so ${{\Delta }}_{t}\in \left[\text{0,2}\pi \right)$. The other 2 terms are set by environment variables which are effectively hyperparameters that could be explored, but for the purposes of this study, they are set to fixed values as the best environment hyperparameters are not of particular interest. Also, to accurately reflect the task, close distances from the goal and smooth/slight steering should be rewarded more so both ${\delta }_{t}$ and ${{\Delta }}_{t}$ are negated, as the task requires minimising these quantities whilst maximising reward.

The reward structure, as described in Eq. 2, was designed to capture three critical aspects of the vehicle's behaviour: its speed, its proximity to the goal, and the smoothness of its steering. Each of these aspects plays a distinct role in determining the overall performance and safety of the vehicle. In essence, while distance serves as a foundational metric, speed and steering encapsulate both basic and advanced driving behaviours. The dual nature of the speed term, where it can belong to either category, highlights its versatility in capturing different facets of driving. The reward structure, therefore, provides a comprehensive evaluation of the vehicle's performance, balancing both essential and subtle driving behaviours.

In our approach, utility functions are transforms of the reward function, which directly influence the system, serving as an alternative to traditional reward models. Instead of altering the inherent task reward, we adjust the reward to align with individual preferences. Utility takes up a role of direct influence in the place of a reward model, but is not a reward model; we are shifting the reward function to match the individual, not changing what the task reward is. By leveraging utility, we aim to discern implicit preferences derived from judgments, thereby facilitating both implicit preference selection and explicit task discovery in tandem.

We apply a utility transform ${U}_{\theta }$ to each term, which takes the form of a parameterised exponent:

$${U}_{\theta }\left(r\left({s}_{t}\right)\right)={r}_{\nu }^{{\theta }_{\nu }}+{r}_{\delta }^{{\theta }_{\delta }}+{r}_{{\Delta }}^{{\theta }_{{\Delta }}}-3={\left(\frac{{\nu }_{t}}{{\nu }_{max}}\right)}^{{\theta }_{\nu }}-1+{\left(1-\frac{{\delta }_{t}}{{\delta }_{max}}\right)}^{{\theta }_{\delta }}-1+{\left(1-\frac{{{\Delta }}_{t}}{2\pi }\right)}^{{\theta }_{{\Delta }}}-1.$$

( 3 )

The advantage of parameterising each term is that it enables feedback to be elicited for each term individually, allowing each term to be non-linear over a trajectory, but decompose into a linear sum at a policy execution level. As we do not want to yield a complex valued reward, the negative domain shifting is also decomposed and applied to each term individually, importantly after applying the parametrised exponent. Figure 2 shows the variation the utility parameters provide and informs as to why the construction of the reward requires such care to remain within the range $\left[\text{0,1}\right]$, ensuring a symmetric and bounded variation.

At policy execution, we can still return a sum of 3 scalar terms only depending on the previous state, therefore not violating either of the assumptions of Markovian and sub-game optimal dynamics for learning. Hence the nonlinearity is applied to expressions in the bounded positive interval $\left(\text{0,1}\right]$ (Fig. 2), before other scaling is applied. As a result, the utility function mathematically cannot violate this assumption as it remains within this bound, respecting the linear optimality by preserving monotonicity and only altering the asymptotic nature of the function. This utility function therefore acts as a filter over the reward and can be optimised directly but statically, only learning a policy for one fixed set of parameters. When the utility is higher than the base linear reward, rewards increase across all states with the effect compounding as the linear reward increases, effectively reinforcing the current policy’s behaviours. The converse is true for low ratings, which encourage exploration of different behaviours as the policy reconverges. The linear form of the utility, equivalent to the standard shaped reward, is used as a pre-trained model from which to train each utility variant model for a further fixed number of episodes.

For the training of agents, the framework of choice was Stable Baselines3 (SB3) [16], a popular library that offers a variety of reinforcement-learning algorithms. A notable limitation of SB3, however, is its lack of support for action-masked DQN, a crucial feature for our simulator. Thus, adaptations to the standard SB3 DQN implementation were needed¹. We used a simulation grid with dimensions of 640x480 pixels. The training process was initiated with two distinct DQN models. The primary model was trained using our specially designed reward-shaping mechanism, while another model was trained with a consistent reward for each time step. When the vehicle deviated from the road in the simulation, the fixed reward was marginally decreased. The magnitude of these fixed rewards was deliberately chosen to align with the rewards from our shaped function, facilitating a more straightforward comparison between the two models. The primary purpose of the baseline shaped model, where all parameters were set to zero, was to serve as a foundational model. From this baseline, variant models were trained, each for a predetermined number of episodes. These variant models were derived by multiplying the utility-parameterised three-term reward function with a five-point Likert scale, resulting in a range of $125$ parameter sets for the experiments.

The first objective an agent must learn is the path to the goal, and Fig. 2 shows how this takes a much longer time when the reward is not shaped to guide the agent's progress in that sparse reward regime. To learn that heading to the goal improves the return, the agent must avoid a pedestrian with noisy movement heading perpendicular to them to get there. Again, the shaped reward helps with this learning; however, a straight-line path has a high probability to intersect with pedestrian in both cases. There is then a trade-off between an efficient route with high reward and the possibility of hitting the pedestrian. Honing in on early successes too strongly can lead to failures and can cause the agent to change its policy to find safer behaviours.

Behavioural Experimental Design

The feedback collection experiment was conducted in a controlled environment, within the behavioural science lab in the Psychology Department at the University of Warwick. There was a total of 124 participants in the experiment. Experiments took up to 30 minutes, and participants received course credit for participating. All participants provided informed consent, and the research was approved by the Department of Psychology Research Ethics committee.

Participants were invited to the lab where they were seated in front of a PC and presented with an instruction sheet. Each session accommodated approximately 12 participants. Upon arrival, participants were given a brief overview of the study's objectives and what was expected of them during the session; they also were asked to sign for their consent for their data to be collected and anonymised before commencing. Following this introduction, the trained models were presented. Participants provided responses judging different aspects of the AV behaviour. These responses were obtained with three different initialization seeds, ensuring variability in the scenarios and interactions presented to the participants. Aggregating the feedback, each participant contributed a total of 200 feedback data points. Of these, 150 were direct evaluations related to the utility parameters, while the remaining 50 were judgments about the likelihood of the agent being an AV.

Each participant went through a structured process comprising five blocks of trials, with each block containing 10 individual trials before a reset to the linear model configuration for the next block. During each trial, participants were prompted to provide feedback through the three safety ratings. Additionally, after every trial, participants were asked to make a judgment about whether they believed the agent's behaviour was more characteristic of an AV or a human driver. To elicit feedback from human participants, an interactive experiment2 was set up using PsychoPy [17]. In this experiment, participants took on the role of a pedestrian within a simulated crossing environment, as depicted in Fig. 1. Their task was to navigate this environment while interacting with the various models. Integrating the Pygame environment into a PsychoPy experiment enabled participants to experience the agent trained with a given reward function. Participants then provided safety ratings on each of the 3 aspects considered in the reward design in a seamless loop. This setup allowed the parameters for subsequent models to be adjusted based on feedback from the previous trial.

To ensure consistency and estimate inherent uncertainty, participants were reintroduced to the linear model after every 10 trials, restarting the process of selecting subsequent parameters through safety ratings. This approach created a consistent baseline for comparison, providing a consistent reference point for participants throughout the experiment and aided in pinpointing any potentially insincere feedback. The 7th trial in every set of 10 featured a random non-shaped reward trajectory. These trajectories were either from researcher-controlled vehicles or derived from the fixed-reward model. Additionally, after each trial, participants were asked to judge whether the vehicle was human-controlled or operated by an AV. By collecting judgments on human-like behaviour, we aimed to discern if trajectories deemed "expert" were indeed superior based on participant evaluations. To further refine our study and account for another potential bias, we introduced variability in vehicle colours. With a palette of eight colours, the vehicle's hue was randomly selected each time the simulator was run, and this colour data was stored alongside other trial information. This measure was implemented to identify and account for any colour-based biases in judgments, such as the perception that red cars, often associated with sportiness, might be inherently riskier.

Feedback was gathered from the human participants by asking them to evaluate specific interactive trajectories. Their judgments were recorded on a 7-point Likert scale ranging from − 3 (very unsafe) to + 3 (very safe). A 5-point scale would not have had enough intervals to include an interval for 0 as the linear utility parameter, so to ensure the scale remained symmetric around 0, a 7-point scale was used. After gathering this feedback, participants were then presented with a new interactive trajectory. This trajectory was produced by a utility-adjusted reward function that matched their initial judgment, as described in Eq. 3. This relationship allowed a mapping from the integer Likert-scale points (LP) to the parameterised exponents in Eq. 3 directly, according to:

$${\theta }_{i}=\left\{\begin{array}{c}sgn\left(L{P}_{i}\right)\cdot 2 \left|L{P}_{i}\right|\ge 1.8\\ sgn\left(L{P}_{i}\right)\cdot 1 \left|L{P}_{i}\right|\ge 0.6\\ 0 Otherwise\end{array}\right.$$

( 4 )

At ${\theta }_{i}=0$, this equation reverts to the non-parameterized reward function, as shown in Eq. 2. The structure of Eq. 4 ensures that higher ratings correspond to increased utility, especially when the utility mirrors the form of ${U}_{\theta }$. When we consider a three-term reward/utility function combined with the reduced seven-point Likert scale (see below), we obtain $125$ (or ${5}^{3}$) distinct parameter sets. These five sets are derived from the Likert scale's range of $-3$ to $3$, which was divided into five distinct intervals: $[-3,-1.8]$, $(-1.8,-0.6]$, $(-\text{0.6,0.6})$, $\left[\text{0.6,1.8}\right)$, and $\left[\text{1.8,3}\right].$ Each of these intervals corresponds to an integer parameter ranging from $-2$ to $2$ in consecutive order. The resulting function retains its symmetry and exhibits convexity for positive LP values and concavity for negative LP values, as depicted in Fig. 2. This approach ensures a systematic and mathematically sound method of correlating human judgments with specific utility parameters.

The participant thus effectively picked the subsequent model from the pre-trained set with which they then interacted. By analysing these selection data, we aim to identify the utility that aligns most closely with their perception of a superior-performing agent. While individual preferences will naturally vary, often significantly, any consistent trends observed in aggregate data can validate our parameter choices. We have pre-emptively trained models for all potential parameter sets. This enables us to present participants with agents that appear to adjust in real-time based on their feedback, without the need for extensive re-training in real time. During the behavioural experiments, participants were exposed to policies that reflect adjustments in parameters, guided by their LP judgments.

Participants' judgments were not based on passive observation of a scenario. Instead, the participant actively controlled the pedestrian in the environment, and so they based their evaluations on direct interactions with the agent rather than just observing a predetermined trajectory. This active engagement can lead to unique instances, revealing a broader spectrum of the policies' learned behaviours, even when utility parameters are similar or identical.

[2] www.github.com/Rik-Fox/paavi

Figure 4 shows how, for some measures, there was a positive trajectory throughout the entirety of the initial round, suggesting that the model was quickly adapting to the feedback and making noticeable changes in its behaviour. The speed of the agent was judged to be moderately appropriate from the outset, and this assessment remained consistent throughout the trials with $\left(Mdn=0, {{\sigma }}_{n-1}=1.14\right)$. This suggests that the initial model's speed parameterization was already in line with participants' expectations. In contrast, the other two behavioural aspects started with relatively poor ratings $\left(Mdn=-1, {{\sigma }}_{n-1}=1.3\right)$ and $\left(Mdn=-1, {{\sigma }}_{n-1}=1.27\right)$ respectively. Notably, there was a discernible improvement in the ratings for position and steering right after the very first interaction. Over the course of the first 10 rounds, these ratings improved more strongly on average, transitioning from poor $\left(Mdn=-1.33, {\sigma }_{n-1}=0.77\right)$ to moderate $\left(Mdn=-0.66, {{\sigma }}_{n-1}=0.923\right)$, with this improvement being statistically significant under a two-sided Z test with a null hypothesis of equality, $Z\left(123\right)=-2.85,p=4.65\times {10}^{-3}$. This result indicates that where there was room for improvement, the model was indeed responsive to feedback and made significant strides in these areas.

The utility model thus demonstrated enhancement when matching utility parameters with judgments, particularly during the initial trials. Interestingly, there is the possibility that as the model tried to diverge from this state in search of further improvement, it might have been perceived negatively, leading to poorer ratings. By incorporating the utility into this paradigm, each piece of feedback used contributes to finding the best behaviour without prescribing what that behaviour should look like. This approach selects for higher performing behaviours that assimilates better into the social climate. Therefore, judgements are not an alternative reward signal, but the reward signal that fully represents the underlying problem and is the task we want to achieve success in. While it is theoretically possible to embed any judgements using this approach, non-passive judgements are recommended, where the participants are in control of an entity in the environment and judging based on their interactions with the agent, instead of simply observing trajectory. Participant control leads to each instance being unique and showing a broader range of the policies’ learned behaviours, even with the same or similar utility parameters, than the observation of a fixed representative trajectory.

What stands out prominently from the data is that the majority of the improvement, from the inception of the trials to their conclusion, was realized during the initial block. Figure 5A highlights this effect by showing the model rating at the first and last trial of each block, and that the final rating within a block significantly improved for the first 2 blocks but not for the final 3. Several factors could have caused these improvements to plateau. The model may have achieved the maximum improvement possible within the constraints of the given models. Alternatively, participants might be adjusting their expectations downward as they become more familiar with the system. Another consideration is the transient dynamics that emerge as the collective data settles into a state of equilibrium. Notably, the results indicate that the primary avenue for enhancing vehicle behaviour remains in the predictability of its movements. Erratic movement consistently emerged as a common feature among trajectories that were judged as lacking in safety aspects.

This diminishing return on improvement does, however, suggest that the model may have been approaching an optimal state, at least within the confines of the feedback provided, and even plausibly being rated poorly as it tried to move away from that state. The rapid initial improvement underscores the efficacy of the feedback mechanism in quickly refining the agent's behaviour, especially in the early stages of interaction. Such rapid initial refinement is advantageous and is the ultimate goal of this methodology. This ability to adjust its remit based on feedback is instrumental, putting the vehicle’s behaviour within the scope of human-accepted behaviour. This method is ideal to use before further refinement with more direct methods, such as relative preference ranking on shown trajectories, especially when considering that nuanced preferences might be pivotal for further honing the agent's behaviour.

For the fixed models presented during the 7th trial of each block, several observations were made. Firstly, the simple fixed model's performance remained consistent throughout the trials. On an aggregate level, its ratings did not exhibit any significant fluctuations from $\left(Mdn=-0.333 , {\sigma }_{n-1}=0.9 \right)$, essentially aligning with the ratings of the shaped model as it is a consistent but significant difference $Z\left(123\right)=-4.27,p=1.94\times {10}^{-5}$. This results suggests that, in terms of overall performance, there was not a discernible difference between the two models in the eyes of the participants, i.e. a poor shaped model instance and a fixed reward instance were not considered as different by participants.

A contrasting trend was observed when human trajectories were introduced. Figure 5B displays the spread of judgments regarding the likelihood of a trajectory being controlled by an AV against their average rating across all 3 reward terms; it also indicates multi-modal characteristics for the highly rated human-controlled trajectories. This multi-modality contrasts with findings from a previous experiment [12], where participants predominantly associated higher-rated judgements of trajectories with a greater likelihood of being autonomously driven. These trajectories consistently received superior ratings across all three evaluated aspects, $\left(Mdn=1 , {\sigma }_{n-1}=1.32\right)$, $\left(Mdn=2 , {\sigma }_{n-1}=0.789\right)$, $\left(Mdn=2 , {\sigma }_{n-1}=1.37\right)$, when compared to the AI-driven models ($~H)$, with an overall significantly increased median $Md{n}_{H}=1.67, Md{n}_{~H}=-0.67,$ $Z\left(123\right)=-4.27, p=1.94\times {10}^{-5}$, and when testing for the one-sided null hypothesis $H>~H$, $Z\left(123\right)=20,p=1.41\times {10}^{-86}$. These results suggest a perceptual bias or preference by people towards human-like behaviour or decision-making in the driving context. Yet, when participants were tasked with determining whether a particular trajectory was human-driven or AI-driven, the results were far from consistent. There is a notable variance in their judgments of these trajectories on average, as visible in the reported standard deviation values,covering the full spectrum of available ratings. This implies evidence for a new, nuanced interpretation: while a large cohort of participants did still equate superior performance with autonomous driving capabilities, there appears to be a distinct sub-group that held the opposite belief. Specifically, this latter group appeared to believe an exceptionally well-executed trajectory was more indicative of human control.

The variability of judgments regarding the likelihood of a trajectory being from an AV could be associated to a multitude of influences, but further research in that area would be required. Yet, not all participants in this study shared the same viewpoint, especially for human-controlled trajectories as evidenced by their multi-modal characteristics. A distinct group appeared to lean towards the belief that human drivers would outperform AI in driving tasks. This split in perspectives further amplified the variability in judgments, underscoring the subjective nature of human perceptions when comparing AI and human driving behaviours. The repeated interactions, sub-group classifications, and aggregate statistics go part of the way to remedying the inherent fallibility of human feedback. Human judgements (and indeed expressed preferences) may not align with their true interests or quality criteria. The current method incorporates the subjective feedback in a direct numerical way and can optimise from feedback in an out-of-the-loop manner across multiple iterations. As a result, the method finds the reward shape that matches judgements/preferences the closest, not the judgements/preferences themselves, which allows for effects such as sample bias to propagate.

Regarding potential biases based on vehicle colour, the data did not reveal significant influence, all having $p=1$ for the null hypothesis of equality; the distribution of judgments remained uniform across all vehicle colours presented. When the data was segmented into overly specific categories, some degree of variability does emerge; however, these variations were exclusively observed in categories that were poorly represented, having fewer than 500 trial instances. Given the limited sample size in these categories, the observed variability might be attributed to sample bias rather than any genuine effect related to vehicle colour.

Utility variants drove most of the vehicle trajectories observed, and a surprising 75% of these trajectories moved from left to right across the three seeds. It remains unclear why the `numpy.random.choice` function, given only two directional options, exhibited this bias. Of the 50 trials, only 10 could deviate from this pattern, and for each of these 10 trials, there were four potential scenarios to select from: human-controlled trajectories moving left or right and fixed reward trajectories also moving left or right. By design, the policies were not fully converged. Despite this, a significant portion of the trajectories managed to approach their intended destinations closely. However, it was observed that many of these trajectories, although they were close to their goals, did not complete their tasks directly. Instead, they often meandered away from the destination, displaying a pronounced bias of moving upward and then returning to the centre. Interestingly, for the fixed model, the trajectories moving from right to left performed better. Yet, for the utility model moving right to left, they typically only managed to traverse about three-quarters of the intended path before their actions appeared to become random.

The ratings related to speed were predominantly centred around the mid-range of the scale. Speed ratings on the extreme were typically associated with trajectories that either did not head towards the goal (resulting in poor ratings) or those that deviated minimally from the path (resulting in favourable ratings), not clearly rating on speed alone. Steering ratings, on the other hand, were generally lower, and there was a distinct correlation whereby erratic trajectories, that is trajectories with frequent large changes in heading, received poorer ratings. The position term's ratings were in line with expectations; trajectories that strayed from an efficient path, especially those that deviated early on, were judged as having low safety. This demonstrates that some possible qualitative metrics of the trajectories are consistent with the broader findings regarding overall ratings. Regarding the pedestrian trajectories, their characteristics were uniformly distributed across all five rating bins and for all three rewards aspects. No discernible exploitative behaviour was detected in the pedestrian's movements. However, numerous trajectories displayed instances where the pedestrian seemed to halt or linger, suggesting that the participant might have been observing the vehicle's behaviour during these moments, rather than interacting naturally.

Drawing insight similar to [18], [19], it becomes clear that accurately capturing the nuances of human preferences or judgments, especially as agents evolve towards more human-centric behaviours, is a formidable challenge. This complexity is evident in our results and collected data. For instance, while certain anticipated biases, such as those related to vehicle colour, were absent, unexpected biases emerged, such as the assessments regarding the likelihood of a trajectory being from an AV displayed a multi-modal distribution. These findings underscore the intricate nature of human perceptions and expectations. The existing literature corroborates that this domain offers a fertile ground for further investigation. The collected data still provides valuable insights, contributing to a deeper understanding of how AI models can be evaluated and improved through human feedback. It paved the way for future research and development that can leverage human insights to create more effective and human-centred AI models. It highlighted the need for careful consideration in designing AI systems that not only perform well but also resonate with human intuition and values.

This approach delves deeper into the intricate dance between autonomous vehicles and human pedestrians, aiming to bridge the gap between computational advancements and human perceptions. By placing participants at the helm of the pedestrian's controls, we not only provide them with a more immersive experience but also capture a richer dataset, containing both explicit judgements and implicit behavioural nuances. The dual-layered approach offers a unique lens to scrutinise the alignment (or lack thereof) between a participant's actions and their stated perceptions.

This work attempts to shift the paradigm of policy evaluations to be more human-centric. It leaves the efficient processing of observation data that accurately represents the vehicle’s environmental state to other work. Instead, the work focuses on including the subjective human stakeholder’s opinions to improve automated driving performance in the social aspect of road use. This focus helps to address the simulation-to-real gap felt by subjective stakeholders in the real world, embedding automation into the social climate with minimal friction.

We have shown that incorporating the judgements of other road users can improve the performance in their subjective view. We do not push temporal credit assignment onto judgement makers, by only eliciting feedback on full trajectories of converged policies, instead transforming the already shaped reward. The results show that an out-of-the-loop approach with iterative trials across many individuals helps give an overview picture of behaviours judged more favourably and allow for adaptive behaviours to better target a particular individual. For the latter, on-the-fly training may be possible with sufficient computation, which would alleviate the need for mass training of variants for the participant to effectively grid-search though the space of utility parameters.

Moreover, by leveraging repeated interactions, subgroup classifications, and aggregate data, we can address some of the inherent fallibility of human feedback. Participants' judgments (and derived preferences) might not always reflect their genuine interests or quality criteria. Our aim is not to claim a precise modelling of their choices. Instead, we have crafted a method that numerically integrates subjective feedback. By optimizing this feedback in an out-of-the-loop manner, we strive to align the reward shape with judgments/preferences as closely as possible, rather than replicating or modelling the generation of judgments/preferences themselves.

Future avenues could aim to delve deeper into the potential of generalizing and automating environmental reward shaping based on the judgments gathered, and preferences inferred. Testing the value of interacting with the AV and its effects on the subjects’ responses would also add to our understanding of the feedback interactions, possibly by running a comparative study with some no-interaction control groups. Furthermore, harnessing online policy optimization, i.e., ad hoc on-the-fly model training with adaptable feedback parameters, to iteratively refine judgment parameter estimates is thought to be enable further refinement. In summary, the in-depth questioning employed in the survey provided a holistic insight into how humans perceive AI-driven and human-driven behaviours in a simulated vehicular context and recognising behaviours that receive favourable judgments can pave the way for enhanced performance. Human demonstrations, preferences, or judgements are not only worthwhile to improve agent behaviour in complex environments, but also to assist in integrating automation into our lives.

Bonnefon, J. F., Shariff, A., & Rahwan, I. (Jun. 2016). The social dilemma of autonomous vehicles. Science, 352(6293), 1573–1576. 10.1126/science.aaf2654.
Pal, A., Philion, J., Liao, Y. H., & Fidler, S. (2020). ‘Emergent Road Rules In Multi-Agent Driving Environments’, ArXiv201110753 Cs, Nov. Accessed: Feb. 23, 2021. [Online]. Available: http://arxiv.org/abs/2011.10753.
Chater, N., Misyak, J., Watson, D., Griffiths, N., & Mouzakitis, A. (Feb. 2018). Negotiating the Traffic: Can Cognitive Science Help Make Autonomous Vehicles a Reality? Trends In Cognitive Sciences, 22(2), 93–95. 10.1016/j.tics.2017.11.008.
Ritchie, O. T. (Oct. 2019). ‘How should autonomous vehicles overtake other drivers?’, Transp. Res. Part F Traffic Psychol. Behav., vol. 66, pp. 406–418, 10.1016/j.trf.2019.09.016.
Knox, W. B., Allievi, A., Banzhaf, H., Schmitt, F., & Stone, P. (2021). ‘Reward (Mis)design for Autonomous Driving’, ArXiv210413906 Cs, Apr. Accessed: Jul. 26, 2021. [Online]. Available: http://arxiv.org/abs/2104.13906.
Kiran, B. R., et al. (2021). Deep Reinforcement Learning for Autonomous Driving: A Survey. Ieee Transactions On Intelligent Transportation Systems, 1–18. 10.1109/TITS.2021.3054625.
Zhou, M., Liu, Z., Sui, P., Li, Y., & Chung, Y. Y. (2020). ‘Learning Implicit Credit Assignment for Cooperative Multi-Agent Reinforcement Learning’, ArXiv200702529 Cs Stat, Oct. Accessed: Feb. 23, 2021. [Online]. Available: http://arxiv.org/abs/2007.02529.
Brown, D., Goo, W., Nagarajan, P., & Niekum, S. (2019). ‘Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations’, in Proceedings of the 36th International Conference on Machine Learning, PMLR, May pp. 783–792. Accessed: Feb. 22, 2023. [Online]. Available: https://proceedings.mlr.press/v97/brown19a.html.
Jain, A., Sharma, S., Joachims, T., & Saxena, A. (2015). ‘Learning preferences for manipulation tasks from online coactive feedback’, Int. J. Robot. Res., vol. 34, no. 10, pp. 1296–1313, Sep. 10.1177/0278364915581193.
Suresh (Oct. 2019). ‘Pedestrian Trust in Automated Vehicles: Role of Traffic Signal and Av Driving Behavior’, Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3478133, Accessed: Feb. 03, 2022. [Online]. Available: https://papers.ssrn.com/abstract=3478133.
Suresh, L., Robert, J., Yang, & Tilbury, D. (2021). ‘Automated Vehicle Behavior Design for Pedestrian Interactions at Unsignalized Crosswalks’. Rochester, NY, Jun. 03, 10.2139/ssrn.3859366.
Fox, R., & Ludvig, E. A., ‘Using Human Behaviour to Guide Reward Functions for Autonomous Vehicles’, presented at the RLDM 2022–2.141 (Poster), Providence, Rhode Island, Oct. 06, 2022. [Online]. Available: https://rldm.org/.
Mnih, V. (Dec. 2013). ‘Playing Atari with Deep Reinforcement Learning’, ArXiv13125602 Cs, Accessed: Mar. 16, 2021. [Online]. Available: http://arxiv.org/abs/1312.5602.
Shinners, P. (2011). ‘PyGame - Python Game Development’. [Online]. Available: http://www.pygame.org.
Huang, S., & Ontañón, S. (2022). ‘A Closer Look at Invalid Action Masking in Policy Gradient Algorithms’, Int. FLAIRS Conf. Proc., vol. 35, May 10.32473/flairs.v35i.130584.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., & Dormann, N., ‘Stable-Baselines3: Reliable Reinforcement Learning Implementations’.
Peirce, J. W. (May 2007). PsychoPy—Psychophysics software in Python. Journal Of Neuroscience Methods, 162, 1–2. 10.1016/j.jneumeth.2006.11.017.
Devlin, S. (May 2021). ‘Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation’, ArXiv210509637 Cs, Accessed: Jul. 26, 2021. [Online]. Available: http://arxiv.org/abs/2105.09637.
Yazdanpanah, V. (Aug. 2023). ‘Reasoning about responsibility in autonomous systems: challenges and opportunities’, AI Soc., vol. 38, no. 4, pp. 1453–1464, 10.1007/s00146-022-01607-8.

No competing interests reported.

Download PDF

Editorial decision: Revision requested
12 Jan, 2024
Reviews received at journal
08 Jan, 2024
Reviewers agreed at journal
18 Oct, 2023
Reviewers invited by journal
18 Oct, 2023
Editor assigned by journal
10 Oct, 2023
Submission checks completed at journal
10 Oct, 2023
First submitted to journal
02 Oct, 2023

You are reading this latest preprint version

Assimilating Human Feedback from Autonomous Vehicle Interaction in Reinforcement Learning Models

Status:

Version 1

Abstract

Figures

Introduction

Simulator Design

Reward Design and RL Training

Behavioural Experimental Design

Results and Discussion

Conclusion

References

Additional Declarations

Status:

Version 1