Context Model for Pedestrian Intention Prediction using Factored Latent-Dynamic Conditional Random Fields
Satyajit Neogi Michael Hoy Kang Dang Hang Yu Justin Dauwels
cs.CV cs.RO eess.IV stat.ML















































2019 IEEE. Accepted by IEEE Transactions on Intelligent Transportation Systems
Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals























Context Model for Pedestrian Intention Prediction using Factored Latent-Dynamic Conditional Random Fields
    Satyajit Neogi, Student Member, IEEE,
Michael Hoy, Member, IEEE,
Kang Dang, Member, IEEE,
Hang Yu, Member, IEEE,
and Justin Dauwels, Senior Member, IEEE 
    December 30, 2023
=============================================================================================================================================================






Smooth handling of pedestrian interactions is a key requirement for Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS). Such systems call for early and accurate prediction of a pedestrian's crossing/not-crossing behaviour in front of the vehicle. Existing approaches to pedestrian behaviour prediction make use of pedestrian motion, his/her location in a scene and static context variables such as traffic lights, zebra crossings etc. We stress on the necessity of early prediction for smooth operation of such systems. We introduce the influence of vehicle interactions on pedestrian intention for this purpose. In this paper, we show a discernible advance in prediction time aided by the inclusion of such vehicle interaction context. We apply our methods to two different datasets, one in-house collected - NTU dataset and another public real-life benchmark - JAAD dataset. We also propose a generic graphical model Factored Latent-Dynamic Conditional Random Fields (FLDCRF) for single and multi-label sequence prediction as well as joint interaction modeling tasks. FLDCRF outperforms Long Short-Term Memory (LSTM) networks across the datasets (∼100 sequences per dataset) over identical time-series features. While the existing best system predicts pedestrian stopping behaviour with 70% accuracy 0.38 seconds before the actual events, our system achieves such accuracy at least 0.9 seconds on an average before the actual events across datasets. 




Autonomous Vehicles, Intention Prediction, Context Models, Probabilistic Graphical Models, Conditional Random Fields.





















§ INTRODUCTION
   















As[S. Neogi, J. Dauwels, "PedAI: Smart Pedestrian Interactions for AV and ADAS", Singapore patent application number: 10202008970S. Filed 14th Sep, 2020.] we enter the era of autonomous driving with the first ever self-driving taxi launched in December 2018, smooth handling of pedestrian interactions still remains a challenge. The trade-off is between on-road pedestrian safety and smoothness of the ride. Recent user experiences and available online footage suggest conservative autonomous rides resulting from the emphasis on on-road pedestrian safety. To achieve rapid user adoption, the AVs must be able to simulate a smooth human driver-like experience without unnecessary interruptions, in addition to ensuring 100% pedestrian safety. 

Automated braking systems in an ADAS tackle the emergency pedestrian interactions. These brakes get activated on detecting pedestrians' crossing behaviours within the vehicle safety range. Such a system may need to brake strongly on late prediction of pedestrian crossing behaviour, which can occur frequently in crowded areas. A future ADAS must be able of offer a smoother experience on such interactions.

The key to a safe and smooth autonomous pedestrian interaction lies in early and accurate prediction of a pedestrian's crossing/not-crossing behaviour in front of the vehicle. Accurate and timely prediction of pedestrian behaviour ensures on-road pedestrian safety, while early anticipation of the crossing/not-crossing behaviour offers more path planning time and consequently a smoother control over the vehicle dynamics. 

Recent works on on-road pedestrian behaviour prediction (<cit.> - <cit.>) rely on a pedestrian's motion, skeletal pose, his/her location in scene (on road, at curb etc.) and certain static context variables (e.g., presence of zebra crossings, traffic lights etc.). While pedestrian motion, skeletal pose and location are reliable indicators of the current pedestrian action, they hardly reflect on the long-term (>1 sec ahead) pedestrian behaviour. Static context variables (e.g., zebra crossings, traffic lights) certainly are important factors to predict future behaviour, but are seldom present in a scene. More often than not, an Autonomous Vehicle is likely to encounter pedestrian(s) without these variables. 

 

As pedestrians, our crossing/not-crossing intentions are largely influenced by the distances and speeds of the approaching vehicles. We refer to this causal information as dynamic context or vehicle interaction context (see Fig. <ref>) and utilize it together with pedestrian motion and location information for early prediction of future behaviour. To the best of our knowledge, we are the first <cit.> to consider influence of such dynamic context on pedestrian intention, which proves to be the strongest factor for early prediction, as we demonstrate in Section <ref>. Adding such context improves prediction time of the pedestrian stopping (at the curb) behaviour by ≥0.5 seconds on average across considered datasets (see Section <ref>), and can potentially reduce unnecessary halts in future autonomous rides. This dynamic context also improves prediction time and accuracy of other behaviour types across the datasets.

Since each pedestrian interaction with the ego-vehicle is continuous in time for a certain duration (say, T_p) and we want to classify crossing/not-crossing intentions of a pedestrian at each instant within this duration, we formulate this problem as a time-series binary classification task, P(y_t | x_1:t), t ∈ 1  T_p, where y_t is the intention variable to be predicted and x_t are the input features to the system at t. Conditional Random Fields (CRF) model such conditional task directly from the data. Each of the pedestrian crossing and not-crossing intentions has underlying latent intrinsic dynamics (see Section <ref> for examples), while there also exists an extrinsic dynamics between the intention labels. A Latent-Dynamic CRF (LDCRF, see Fig. <ref>b) <cit.> can capture the intrinsic dynamics within class labels y_t by embedding latent states in a labeled CRF structure. In contrast to similar models (that capture the intrinsic dynamics) like Hidden Markov Models (HMM) <cit.> and Hidden Conditional Random Fields (HCRF) <cit.>, a LDCRF does not require prior segmentation of the training sequences according to the class labels, thus preserving the extrinsic dynamics between labels as well in the model. LDCRF has been shown to outperform HMM and HCRF on similar tasks like gesture recognition <cit.>. Considering its theoretical and experimental superiority over similar models, we apply LDCRF as baseline in our work.

The only way to improve learning performance with a LDCRF is by varying the number of hidden states (say, N_h) associated to each class label. Such a variation not only restricts the model capabilities, but also results in a rapidly growing state transition matrix with size (N_l · N_h) × (N_l · N_h), where N_l is the number of class labels. Such increment results in greater model complexity and requires larger training data. Moreover, it is possible that there exists multiple interacting latent dynamics (e.g., contextual and motion-related, see Section <ref>) within the class labels. Considering these limitations of LDCRF and inspired by Factorial HMM <cit.>, we propose a generalization of LDCRF, called Factored LDCRF (FLDCRF) <cit.>, in order to 



  * generate new models by varying the number of hidden layers,

  * generate factorized models <cit.> with fewer parameters to fit better small/medium datasets, and

  * capture relationship between multiple interacting latent dynamics (context and motion) within class labels. Each hidden layer models a different latent dynamics and the connections among different hidden layers model their interactions.


We show such generalization of LDCRF to improve performance on our validation sets (see Section <ref>). Such interacting hidden layers also allow to capture sequential multi-label/multi-agent interactions (see Fig. <ref>). See Section <ref> for more details on FLDCRF. 

LSTMs are the most popular kind of Recurrent Neural Networks (RNN), largely due to their ability to capture longer range dependencies among sequence values. They are frequently applied to similar sequence modeling tasks, e.g., path prediction <cit.>, action recognition <cit.> etc. Deep learning systems like LSTMs have been known to involve tedious hyperparameter tuning and their performance are largely big-data-driven. We compare LSTM and FLDCRF (general LDCRF) on our datasets over identical input features and labels. FLDCRF not only outperforms LSTM across our datasets, but also provides easier model selection and requires significantly lesser training time (see Section <ref>).

We highlight the main contributions of this paper below:


  * We propose a graphical model Factored Latent-Dynamic Conditional Random Fields (FLDCRF[https://github.com/satyajitneogiju/FLDCRF-for-sequence-labeling.]) for joint sequence prediction/tagging tasks. 

  * We introduce the influence of vehicle interactions (dynamic context) on pedestrian intention for early and accurate prediction. 

  * We have created a dataset[Dataset link to be provided.] specifically designed to capture vehicle influences on pedestrian intention, collected inside Nanyang Technological University campus (NTU dataset). We also conduct an evaluation set on a public real-life dataset, known as JAAD (Joint Attention for Autonomous Driving <cit.>), devised for early pedestrian intention prediction. 

  * We compare FLDCRF and LSTM over identical features and labels on both datasets.


In addition to early and accurate pedestrian intention prediction, we put stress on the necessity to simulate real-life prediction environment for our experiments. We apply state-of-the-art object detectors <cit.> and semantic segmentation systems <cit.> to detect pedestrians/vehicles and to determine pedestrian location respectively. Fig. <ref> shows the different modules and their inter-connections in our intention prediction system.

The outline of this paper is as follows. In Section <ref>, we describe state-of-the-art systems on pedestrian behaviour prediction and CRFs. We briefly define a standard nomenclature on considered pedestrian behaviour types in Section <ref>. We propose FLDCRF in Section <ref> and introduce our context and motion feature extraction techniques in Section <ref>. Next, we discuss characteristics of the considered datasets in Section <ref>. We describe our experiments in Section <ref> and present our results in Section <ref>. We offer concluding remarks and ideas for future research in Section <ref>.




§ LITERATURE REVIEW
 

From an Autonomous Vehicle perspective, there are two popular problems concerning pedestrian behaviour predictions, a) continuous path prediction <cit.>, <cit.> - <cit.>, <cit.>, <cit.>, where future trajectory of a pedestrian is estimated and b) discrete intention prediction <cit.> - <cit.>, where the goal is to predict discretized pedestrian behaviours. Given a detected pedestrian, his/her position w.r.t. the AV and a marked AV safety range, solving any problem (path or intent prediction) can ensure a safe autonomous ride. However, a smooth autonomous ride of future (avoiding repeated halts and hard brakes) would require early predictions of pedestrian behaviour for optimal path and dynamics planning. Although pedestrian interactions and static traffic scene elements have been successfully utilized <cit.> - <cit.>, <cit.>, <cit.> to improve path prediction accuracy, long-term path prediction (>1 sec ahead) is still challenging and error prone, prediction error growing exponentially with prediction horizon <cit.>. Additionally, path prediction given fewer (≤10) observed frames, distant objects and moving cameras invoke additional uncertainties. Existing discrete intention prediction approaches (described shortly) primarily focus on accurate prediction rather than early prediction. In this paper, we analyze the feasibility of accurate long-term (up to 2 sec ahead) discrete intent prediction in a variety of scenarios.

We briefly discuss existing literature on discrete intent prediction and sequential CRFs below.



 §.§ Pedestrian Intention Prediction


Existing intent prediction approaches can be broadly classified into two categories, a) Pedestrian-only, which only take features from the pedestrian of interest and b) Context based, which consider context variables (location, traffic elements etc.) together with pedestrian features.

Pedestrian-only approaches <cit.>, <cit.>, <cit.>, <cit.>,<cit.>, <cit.>, in general, learn machine learning models over pedestrian features (motion and/or appearance) and n-ary intention labels. Pedestrian features include dense optical flow <cit.>, concatenated position, velocity and head pose <cit.>, Motion Contour image based Histogram of Oriented Gradients descriptor (MCHOG) <cit.>, skeletal features <cit.>, <cit.> and deep learning appearance features <cit.>. Applied machine learning have been either time-series models (GPDM, PHTM <cit.>, LDCRF <cit.>, HMM <cit.>, LSTM <cit.>) or SVM <cit.>, <cit.>. 

Context based intent prediction approaches <cit.>, <cit.>, <cit.>, <cit.> consider information like scene spatial layout, i.e. location of curb <cit.>, <cit.> and traffic elements <cit.>, <cit.>, <cit.>, <cit.>, such as cross walks, bus stops, traffic lights, zebra crossings etc. along with pedestrian features.

The majority of these approaches <cit.>, <cit.>, <cit.>, <cit.> were evaluated on the Daimler dataset <cit.>, a public benchmark consisting of pedestrian crossing/not-crossing sequences. Stopping probability vs time <cit.>, <cit.>, <cit.>, <cit.> and classification accuracy <cit.>, <cit.>, <cit.>, <cit.> are the most common metrics. Since the Daimler data does not capture spontaneous vehicular interactions with pedestrians, we present our results on this data to Appendix <ref>. 

Recently a group of researchers have proposed a relatively large, labeled pedestrian dataset called JAAD <cit.>, captured from moving vehicles. This dataset provides real-life examples of pedestrian interactions with a vehicle in complex scenes. In <cit.>, the same group proposes a context based system to classify pedestrian crossing/not-crossing behaviour, where they combine an action recognition classifier and a learned attention model for relevant static context variables (traffic light, zebra crossing, stop sign etc.) to classify pedestrian intention. In <cit.>, Gujjar et al. showed improvements in performance on the same dataset by considering a future generation model. However, we find their evaluation dataset to be weakly relevant to early prediction of intention (evaluation does not involve any temporal information) and hence select our own evaluation sequences (described in Section <ref>) from the JAAD dataset.

All context models discussed above for pedestrian behaviour prediction rely on static scene context variables and do not consider dynamic vehicle interactions. To our best knowledge, we are the first to consider effects of such vehicle interactions on pedestrian behaviour <cit.>. While <cit.> presents a 70% prediction accuracy of pedestrian stopping actions 0.06 seconds before the event on <cit.>, we achieve such accuracy at least 0.9 seconds before the event across our datasets in presence of vehicle interactions. Current best approach <cit.> on the Daimler dataset gives such an accuracy 0.38 seconds before event, although it is tested on a small test dataset (with 13 pedestrian sequences) with limited variations in pedestrian behaviour.

We apply proposed FLDCRF over input features (context + motion) and intention labels. We briefly describe related existing sequential CRF models below.



 §.§ Conditional Random Fields (CRFs)


CRFs were first introduced in <cit.> to accomplish the task of segmenting and labeling sequence data. CRFs are discriminative classifiers and aim at directly modeling the conditional distribution P(y|x) over input features x and classification labels y. A simple sequential type of CRFs is the Linear Chain CRF (LCCRF, see Fig. <ref>a), which is frequently applied to sequence labeling tasks like word recognition, NP chunking, POS tagging etc. 



Over the years, multiple variants like semi-markov CRFs <cit.> and Dynamic CRFs <cit.> have been proposed. Semi-markov CRFs allow long range dependencies among labels y = {y_1, y_2, ..., y_T} at the expense of higher computational cost. Dynamic CRFs <cit.> allow multiple interacting labels (y_1,t, y_2,t etc.) at each time step, making it suitable for multi-label classification tasks.

LDCRF was proposed in <cit.> (see Fig. <ref>b), where hidden states are embedded in a labeled CRF structure by means of an additional layer of hidden variables h = {h_1, h_2, ..., h_T}. Each possible classification label is associated with a given set of hidden states. The hidden states model the intrinsic dynamics within each label.

FLDCRF utilizes primary LDCRF concept and constraints. We will briefly discuss the concept behind LDCRF and its applicability on the intent prediction problem in Section <ref>.




§ NOMENCLATURE
 

In general, an on-road or near-road pedestrian's crossing/not-crossing behaviour is characterized by the real world lateral component of his/her motion w.r.t. the ego-vehicle moving direction, i.e.,



    laterally    moving      ⟷   crossing 
    
    laterally    static      ⟷   not-crossing.




At each instant t, we try to predict whether a detected pedestrian will be laterally moving or laterally static[Laterally static instances include pedestrians walking along the curb or sidewalk.] in the near future. Associated critical measures and necessary vehicle dynamics can be derived by further analyzing pedestrian's position w.r.t. the vehicle.

We evaluate pedestrian sequences where early prediction is relevant from an AV perspective. We categorize each such encountered pedestrian sequence into one of the following (see Fig. <ref>): 



  * Continuous crossing: The pedestrian initially has a lateral movement before the curb, continues to move laterally and crosses in front of the vehicle.

  * Stopping at the curb: The pedestrian initially has a lateral movement before the curb and stops at/near the curb. We call this behaviour `stopping' in the paper for simplicity.

  * Standing at the curb: The pedestrian continues to be laterally static at/near the curb. This behaviour is referred as `standing' in the paper.

  * Starting to cross: The pedestrian is initially static laterally at/near the curb and starts to cross in front of the vehicle. This behaviour is called `starting' in the paper.


Stopping and starting sequences carry a change in state of motion (laterally moving to laterally static and vice versa) of the pedestrian. As our goal is to predict pedestrian intention early, our primary aim is to detect these state changes early (preferably before any discernible change in motion/appearance) and accurately. In addition, we also want stable and accurate predictions of `crossing' and `not-crossing' intentions for sequences with no change in state of motion (continuous crossing and standing respectively). 

Each sequence type is associated with an event instant (e.g., crossing, stopping or starting instant), illustrated in Fig. <ref>, which provides the ground truth classification labels and also serves as reference during evaluation. Since a standing sequence cannot be characterized with such an instant, we define a critical point based on pedestrian distance and ego-vehicle velocity. To aid the labeling process during training (for early prediction of the transitions in stopping and starting sequences), we define a pred_ahead parameter for each dataset, in number of frames before the respective event instants. We explain how we labeled the pedestrian sequences during training in Section <ref>.





§ MODELS
 

In this section, we propose Factored Latent-Dynamic Conditional Random Fields (FLDCRF). We briefly discuss the idea behind applying LDCRF to our problem in Section <ref> and introduce FLDCRF in Section <ref>. Since, FLDCRF subsumes LDCRF (structure and model constraints), we move the mathematical details about LDCRF to Appendix <ref>. 



 §.§ LDCRF
 

As mentioned before, we employ CRF type classifiers in our problem as they directly model the discriminative task P(y|x). Each of the pedestrian `crossing' and `not-crossing' intentions has underlying intrinsic dynamics. Such intrinsic dynamics can be contextual or motion-related. For example, stopping behaviour of a pedestrian (see Fig. <ref>) can be characterized by a transition from pedestrian `moving' to `rest' state (motion) as well as an interacting vehicle going from `far' to `near' state (context). A standing behaviour may be associated with continuation of the `rest' and `near' states respectively. Since such motion-related or contextual states are hidden (latent), overall intrinsic dynamics of the `not-crossing' intention can be captured by associating hidden states to the intention label and allowing their transitions. These states can be learned from training data by connecting them to the observed context and motion features x_t. A LCCRF (see Fig. <ref>a) only allows transition between class labels and hence fails to capture such intrinsic dynamics. LDCRF captures such dynamics by means of an additional layer of hidden variables {h_t}_t = 1:T (see Fig. <ref>b). h_t can assume values from a predefined set of hidden states. For example, if the class labels `cross' or `not-cross' are associated with sets (of hidden states) ℋ_c and ℋ_nc respectively, and y_t is `cross', then h_t ∈ℋ_c. To keep training and inference tractable, LDCRF restricts sets ℋ_c and ℋ_nc to be disjoint.



 §.§ FLDCRF
 

LDCRF captures the latent dynamics within class labels by means of a layer of hidden variables. We propose multiple interacting hidden layers in a LDCRF structure (see Fig. <ref>) to generate new models, reduce model parameters and to capture interaction among different latent dynamics within class labels (see Introduction).



Such interacting hidden layers can also aid multi-label sequence prediction (see Fig. <ref>) and social interactions in a multi-agent environment (see <cit.>). We defer possible applications of such models to our future work. However, as the single-label variant of FLDCRF (see Fig. <ref>) is graphically a special case of the multi-label variant (see Fig. <ref>), we describe the multi-label mathematical model for better generalization.

















  §.§.§ Model
 

Fig. <ref> shows the graph structure for FLDCRF in multi-label classification problems. Although, we depict the model for only two hidden layers, we mathematically define the model for L layers. 

Let, x = {x_1, x_2, ... , x_T} denote the sequence of observations. y_i = {y_i,1, y_i,2, ... , y_i,T} are the observed labels along layer i, i = 1:L. In the case of a single-label prediction task, all layers take the same labels, i.e. y_i,t is same ∀ i. 

Let, y_i,t ∈ Υ_i, where Υ_i is the alphabet for all possible label categories in layer i. h_i = {h_i,1, h_i.2, ... , h_i,T} constitutes the i-th hidden layer. Each label ℓ_i ∈ Υ_i is associated with a set of hidden states ℋ_i,ℓ_i. ℋ_i is the set of all possible hidden states for layer i written as ℋ_i = ⋃_ℓ_iℋ_i,ℓ_i. 



Model constraints: 



  * ℋ_i,ℓ_i are disjoint ∀ℓ_i ∈Υ_i, ∀ i = 1:L. 


  * h_i,t can only assume values from the set of hidden states assigned to the label y_i,t, i.e., h_i,t ∈ ℋ_i,y_i,t, ∀ i = 1:L and ∀ t = 1:T.
 


The joint conditional model is defined as:



    P( {y_i}_1:L|x, θ) = ∑_{h_i}_1:L P({y_i}_1:L|{h_i}_1:L, x, θ) 
    ·  P({h_i}_1:L|x, θ).


Using the graph structure in Fig. <ref>a, we obtain:



    P(   {y_i}_1:L|x, θ) = ∑_{h_i}_1:L( ∏_i = 1^L P(y_i|h_i)) P({h_i}_1:L|x, θ) 
       = ∑_{h_i}_1:L : ∀ h_i,t∈ℋ_i,y_i,t( ∏_i = 1^L P(y_i|h_i)) P({h_i}_1:L|x, θ) 
       + ∑_{h_i}_1:L : ∃ h_i,t∉ℋ_i,y_i,t( ∏_i = 1^L P(y_i|h_i)) P({h_i}_1:L|x, θ).


P(y_i|h_i) can be further simplified by the graph as,



    P(y_i|h_i) = ∏_t = 1^T P(y_i,t|h_i,t).


Applying model constraints, we can write,



    P(y_i,t=ℓ_i |h_i,t) = 
    
    1,     h_i,t∈ℋ_i,y_i,t=ℓ_i
    
    0,     h_i,t∉ℋ_i,y_i,t=ℓ_i.



Finally, the FLDCRF model in equation (<ref>) can be simplified by (<ref>) and (<ref>) as:



    P( {y_i}_1:L|x, θ) = ∑_{h_i}_1:L : ∀ h_i,t∈ℋ_i,y_i,t P({h_i}_1:L|x, θ).


P({h_i}_1:L|x, θ) is defined from the CRF formulation,



    P({h_i}_1:L|x, θ) = 1/Z(x,θ)exp(∑_k θ_k . F_k({h_i}_1:L,x)),


where index k ranges over all parameters θ = {θ_k} and Z(x,θ) is the partition function defined as:



    Z(x,θ) = ∑_{h_i}_1:Lexp(∑_k θ_k . F_k({h_i}_1:L,x)).


In this paper, we only assume Markov connections (as depicted in Figures <ref> and <ref>) along different hidden layers. Therefore, the feature functions F_k's are defined as: 



    F_k({h_i}_1:L,x) = ∑_t=1^T f_k({h_i,t-1}_1:L, {h_i,t}_1:L, x, t).


Also, we only allow different hidden layers to influence each other at the same time instant t. Thus, each component function f_k({h_i,t-1}_1:L, {h_i,t}_1:L, x, t) can be a state function s_k(h_i,t, x, t), a transition function t_k(h_i,t-1, h_i,t, x, t) or an influence function i_k(h_i,t, h_j,t, x, t), i,j ∈{1:L}. We define state and transition functions by the following indicator functions,



    s_k(h_i, t, x, t)    = 1_{(h_i, t,x_t)= k}, 
    
    t_k(h_i, t-1, h_i, t, x, t)    = 1_{(h_i, t,h_i, t-1)= k},


for discrete observations x. For continuous observations x, state functions are defined by:



    s_k(h_i, t, x, t) = 1_{(h_i, t)= k}. x_t.



We define influence functions i_k(h_i,t, h_j,t, x, t) over two or more (depending on the number of inter-related layers of hidden states) hidden variables between layers as:



    i_k(h_i,t, h_j,t, x, t)= 1_{(h_i,t, h_j,t)= k},    i,j ∈{1:L}.


We avoid longer range influences (e.g., h_i,t-1 to h_j,t, i,j ∈{1:L}) to reduce parameters. For the interaction model depicted in Fig. <ref>b, the mathematical details remain same only with a minor change in the state function, being s_k(h_i,t, x_i,t) instead of s_k(h_i,t, x_t).

A FLDCRF single-label variant with 1 hidden layer and >1 hidden states per label corresponds to a LDCRF, while 1 layer and 1 hidden state per label constitutes a LCCRF.



  §.§.§ Training model parameters
 

We estimate the model parameters by maximizing the conditional log-likelihood of the training data given by:



    L(θ) = ∑_n=1^N log P({y_i}_1:L^(n)|x^(n), θ) - ∥θ∥^2/2σ^2,


where N is the total number of available labeled sequences. The second term in equation (<ref>) is the log of a Gaussian prior with variance σ^2. 




  §.§.§ Inference
 

Multiple label sequences y_i, i = 1:L, can be inferred from the same graph structure by marginalizing over other labels:



    ŷ_i = argmax_y_i  ∑_{{y_i}_1:L -  y_i} P({y_i}_1:L|x, θ̂),


where P({y_i}_1:L|x, θ̂) is obtained by (<ref>) and estimated parameters θ̂. At each instant t, the marginals P({h_i,t}_1:L|x_1:t, θ̂) are computed and summed according to the disjoint sets of hidden states to obtain joint estimates of desired labels ŷ_i,t, t = 1,2, ..., ∀ i = 1:L, as follows: 



    P({y_i,t}_1:L|x) = ∑_{h_i,t}_1:L:h_i,t∈ℋ_i,y_i,t P({h_i,t}_1:L|x_1:t, θ̂).


After marginalizing according to (<ref>), the label ŷ_i,t corresponding to the maximum probability is inferred. As the intention prediction problem must be solved online, we compute P({h_i,t}_1:L|x_1:t, θ̂) by the forward algorithm. Forward-backward algorithm <cit.> and Viterbi algorithm <cit.> can also be applied for problems where online inference is not required.




§ FEATURE EXTRACTION
 

We describe the process to prepare our feature vector x_t in this section. x_t comprises of two components: context features, x_c,t, and motion features, x_m,t.



 §.§ Context Features
 

Context features employed in our work are of two types:







  * Pedestrian location context: Also called spatial context in the paper, this context variable encodes the location (e.g. on road, at curb, away) of the pedestrian in a scene.



  * Vehicle interaction context: We call it vehicle context in the paper. This context variable contains approximate longitudinal distance (depth) and velocity measures of the nearest vehicles to the pedestrian of interest.





  §.§.§ Pedestrian location context


Fig. <ref>(a-b) show a scene and its semantic segmentation. We utilize two segmentation categories, road pixels and non-road pixels. Road pixels are directly obtained from the pre-trained semantic segmentation software <cit.>, and all other pixels are marked as non-road pixels. We highlight 3 different regions around the pedestrian bounding box and their considered dimensions in Fig. <ref>c. We compute the mode of the refined semantic values (road or non-road) in each of the three regions, given by {m_1, m_2, m_3}, which constitute the location context feature vector x_sp,t.



  §.§.§ Vehicle interaction context


Longitudinal distances (or depths) and velocities of approaching vehicles play a significant role in determining crossing intention of a pedestrian.




Depth estimation: As existing techniques for monocular depth estimation are not much reliable, we employ a pre-calibrated approach.

To estimate pedestrian and vehicle depths from camera, we pre-calibrate a scene by 10m, 20m and 30m depth lines (see Fig. <ref>d). We assume that once a camera is steadily mounted on a vehicle, these depth lines will approximately remain same during driving time, unless the road ahead has high slope[One of the reviewers kindly pointed out that the depth lines may vary with tyre pressure. We will look to replace with a better monocular depth estimation technique in future.]. The depth lines can be written as:



    y = mx +c,


where (x,y) is a point co-ordinate on the image (in pixels). We assume that the slope m and the intercept c (in pixels) of the straight line have the following approximate exponential relations with depth d (in meters) from camera:



    m(d) = k_m - p_m · e^- dl_m,


    c(d) = k_c - p_c · e^- dl_c,



where m(d) and c(d) are known for d = 10, 20 and 30. k_m, p_m, l_m, k_c, p_c and l_c can be determined by solving equations (<ref>) and (<ref>) using these 3 pairs of values of {m(d), c(d), d}. Once these parameters are obtained, depth of any given point (x,y) in the image can be approximately determined from (<ref>), (<ref>) and (<ref>). We consider the bottom center pixels of the concerned pedestrian and vehicle bounding boxes to estimate their depths.




Velocity estimation: To obtain velocity measures of the nearest vehicles to a pedestrian in the NTU dataset, we compute mean optical flow <cit.> magnitude inside the corresponding vehicle bounding boxes. We utilize the provided ego-vehicle velocity labels in the JAAD dataset. At each instant, the ego-vehicle is assigned one of these velocity labels: `moving slow', `moving fast' or `standing'. Some instants only have the ego-acceleration labels (`slowing down' and `speeding up') and do not contain velocity labels. We associate velocity labels to these instants by imputation/observation. We ignore the ego-acceleration labels in this paper.




The complete (depth and velocity) vehicle context for the NTU dataset is given by, x_vNTU,t = {ncl, ncr, nclf, ncrf}, where ncl and ncr are the relative depths (in meters) of the nearest vehicles to a pedestrian on left and right lane respectively and nclf, ncrf are the mean optical flow of the non-pedestrian pixels within the bounding boxes of the respective vehicles. At instants with no influencing vehicle, we set ncl and ncr to -1 and nclf and ncrf to 0.

The complete vehicle context for the JAAD dataset is given by, x_vJAAD,t = {ego_dep, ego_vel}, where ego_dep is the estimated pedestrian depth from the ego-vehicle and ego_vel is the ego-velocity attribute (`moving slow', `moving fast' etc.). As the other vehicles do not influence pedestrian behaviours in most cases within JAAD dataset, we keep things simpler by only considering the ego-vehicle context. We defer more complete interactions including other vehicles in the scene to our future work.




 §.§ Motion Features
 

We propose a computationally inexpensive heuristic method to capture relevant pedestrian motion dynamics. The features obtained by this method perform comparably to one of the better benchmarks <cit.> on the Daimler dataset <cit.> (see Appendix <ref>). First, we apply a pre-trained object detector <cit.> to detect and track pedestrians in the videos[For considered pedestrians in the NTU data. We utilized provided bounding box annotations for pedestrians in the JAAD data.]. For extracting pedestrian motion features at t, we consider a sliding window of length τ, i.e., frames t-τ+1 to t. The feature extraction process involves three steps: 


∙ Pre-processing: We run a Kalman filter on tracked pedestrian bounding boxes to remove noise/occlusions. Further occlusions (e.g., by tree) are linearly interpolated.

∙ Real world lateral displacement estimate: A pedestrian's crossing/not-crossing behaviour is primarily characterized by the real world lateral motion. Lateral motion captured in an image sequence contains components from both lateral and longitudinal pedestrian movements (w.r.t. the ego-moving direction) in real world. In addition, the real world longitudinal movement is mixed with the ego-vehicle motion. Assuming negligible pedestrian movement along the vertical direction in real world, we obtain approximate estimates of the real world lateral motion in image frame using the camera matrix information (see Appendix <ref>).

∙ Fitting non-linear dynamics: We extract real world lateral displacement estimates for 9 symmetrical points inside the pedestrian bounding box, given by {v_p}, p = 1:9 (see Fig. <ref>e). The choice of the points is arbitrary, as long as they are well distributed within the bounding box. Each of these points is tracked and processed over the sliding window to generate τ - 1 lateral displacement values v_p,t_1, t-τ + 2 ≤ t_1 ≤ t. Finally, we fit a degree 2 curve to the values {v_p,t-τ+2:t}, p = 1:9, by least-squares fitting:



    v_p,t_1 = a_0_p,t + a_1_p,tt_1 + a_2_p,tt_1^2 ,


resulting in a 27 dimensional feature set, x_m,t = {{a_0_p,t}_p = 1:9, {a_1_p,t}_p = 1:9, {a_2_p,t}_p = 1:9}. τ must be large enough (≥4) for a good fitting but not too large to include unnecessary past frames in the current dynamics computation. We set τ = 10, but for test sequences with less than 10 frames, we adjust τ accordingly to generate the motion features. We also tested optical flow features and visual representations as in another work from our group <cit.>. However, including such features resulted in poorer performance on our datasets. Thus we do not include such features in this paper. We plan to test such visual representations using end-to-end models in future.




§ DATASETS
 

We evaluate on an in-house collected NTU dataset and the public real-life JAAD dataset <cit.>.

The NTU dataset is captured inside Nanyang Technological University Campus by a pair of static cameras at 30 fps framerate and 1920 × 1080 resolution. Since our aim is to capture natural vehicle influences on pedestrian intention, we place two synchronized cameras to capture the whole scene of interest, so that approaching vehicles on both sides of the pedestrian can be captured, as depicted in Fig. <ref>. In addition, Camera1 is placed in a way to simulate a camera mounted on a stationery ego-vehicle.



In the videos, we had actors and actual pedestrians strolling in a natural road scene with vehicles. Actors were not given any specific instructions and were asked to move around the scene naturally. We extract 35 continuous crossing and 35 stopping sequences from the videos for our evaluation.

The JAAD dataset <cit.> is a recent pedestrian benchmark which captures real-world pedestrian behaviours in complex scenes from cameras mounted on moving vehicles. It contains 346 video clips at 1920 × 1080 resolution (few at 1280 × 720) and 15 fps framerate, each containing multiple pedestrian sequences. All pedestrians of interest are annotated by bounding boxes and are assigned certain movement labels (moving, crossing, stopping, standing, etc.) and personal details (age, sex, movement direction, etc.). Each frame in the videos is also associated with certain traffic attributes, such as zebra crossing, parking lot, etc.

To our best knowledge, there is no JAAD data subset proposed for early pedestrian intention prediction. We extract a total of 120 pedestrian sequences distributed across continuous crossing, stopping, starting, and standing scenarios (see Appendix <ref> for sequence details).




§ EXPERIMENTAL SETUP
 

We describe here our experimental data, extracted features and labels and the model specifications.



Data for evaluation: The distribution of test sequences in the two datasets is presented in Table <ref>.

We downsampled the NTU dataset sequences to 15 fps. We detect and track <cit.> pedestrians in Camera1 images. We also define pedestrian spatial context only in Camera1 images. Vehicles are detected and tracked in both Camera1 and Camera2 images. We eliminate ambiguity within the common regions of Camera1 and Camera2 by a pre-defined separator (marked by black lines, see Fig. <ref>). The object detector could efficiently detect pedestrians and vehicles up to 80 meters from the camera. For pedestrian sequences in the JAAD data, we utilize provided bounding box annotations. 






Features and labels: In the NTU data, we try to predict the stopping event 30 frames (2 sec) before the stopping instant, i.e., pred_ahead = 30. Therefore, we label data from 30 frames before the stopping instant in stopping sequences as `not-crossing' during training. Remaining frames in stopping sequences and all frames in continuous crossing sequences are labeled `crossing'. The annotations are depicted in Fig. <ref>. We do not have standing and starting sequences for the NTU data. The complete feature set (see Section <ref>) on the NTU data is given by, x_t,NTU = {x_m,t, x_sp,t, x_vNTU,t}. 

For JAAD dataset, we set pred_ahead = 20 (1.33 sec). The labeling is illustrated in Fig. <ref>. Considered feature set on the JAAD data is given by, x_t,JAAD = {x_m,t, x_sp,t, x_vJAAD,t}.





Models compared: We utilized FLDCRF single-label variant and LSTM as our learning models. We show results for three different systems for both the NTU and JAAD data: 


1) Only pedestrian motion based, referred as mtn (or m) in the Results section. This system only utilizes pedestrian motion as input features, i.e., x_t = {x_m,t}.

2) Pedestrian motion and location based, referred as mtn+spa (or ms). This system takes pedestrian's semantic location (see Section <ref>) along with pedestrian motion as inputs, i.e., x_t = {x_m,t, x_sp,t}.

3) Pedestrian motion, location and vehicle context based, referred as mtn+spa+veh (or msv). This system considers vehicle context features (see Section <ref>) together with pedestrian motion and location, i.e., x_t = {x_m,t, x_sp,t, x_v,t}.

We employ a nested cross-validation (with 5 outer folds and 4 inner folds) for selecting models and generating results. For selecting LSTM models, we tested the following number of hidden units (HU): 2, 5, 10, 20, 50, 150, 300, 500. We trained each model for 1000 epochs, saving model performances at every 100 epochs during validation. We fed all training sequences at each epoch at the rate of 1 sequence per batch.

An FLDCRF single-label has two hyper-parameters: number of layers (num_layer) and number of hidden states per label ({num_state_i}) along layers i= 1:L. For simplicity, we keep same num_state along all layers, i.e. num_state_i = num_state, ∀ i = 1:L. We denote such model by FLDCRF-<num_layer>/<num_state>. We selected our models from the following <num_layer>/<num_state> combinations: 1/1, 1/2, 1/3, 1/4, 1/5, 1/6, 2/1, 2/2, 2/3. 

We trained FLDCRF model parameters with Stan's <cit.> in-built BFGS optimizer. We implemented the LSTM models defined in Keras, a deep learning library for Python running Tensorflow in the backend. LSTM parameters were trained by a default Adam optimizer in Keras. We plotted the results in MATLAB 2015b <cit.>. LSTM models are trained with a Nvidia Tesla K80 GPU, while the FLDCRF models are trained with a Intel Xeon E5-1630 v3 CPU.




§ RESULTS
 

We compare different systems on the NTU and JAAD datasets using existing state-of-the-art (LSTM) and proposed (FLDCRF) sequential models. We evaluate the models by two metrics: stopping probability vs time and classification accuracy vs time. Stopping probability corresponds to the probability of the `not-crossing' (laterally static) label.

We ideally want high classification accuracy of appropriate class labels (e.g., `not-crossing' in stopping sequences) over the prediction interval. At the same time, it is desired to have high and stable average probability values of appropriate class labels over the prediction interval. It is also preferred that the sequences of same type have minor standard deviation in predicted probability values at each instant. So, our model hyper-parameters are chosen based on the following metric (mt), 



    mt = 1/T_p∑_t=T_l.f^T_u.f (acc_t + 1/N_v∑_i=1^N_v prob_i,t - 1/N_s∑_j=1^N_s std_j,t),


where acc_t denotes overall classification accuracy, prob_i,t denotes predicted probability of the appropriate class (e.g., `not-crossing' in stopping sequences) for sequence i and std_j,t is standard deviation in probability values among sequences of type j at t. f is the dataset framerate (15 fps for both datasets). The prediction interval is given by [T_l   T_u]s and T_p = (|T_u.f - T_l.f| + 1). N_v is the number of considered validation sequences and N_s is the number of considered sequence types (2 for NTU and 4 for JAAD data).




 §.§ NTU dataset


We compare three different systems - mtn (m), mtn+spa (ms) and mtn+spa+veh (msv) with FLDCRF and LSTM on the NTU dataset. `0' along x-axis (see Fig. <ref> - <ref> and Fig. <ref>) represents the event (crossing, stopping etc.) instants. Positive and negative values along x-axis indicate gain and loss in prediction time (in seconds) respectively. Since we try to predict 2 sec (30 frames) ahead of the event, we will limit our analysis within [2 -0.5] s prediction interval. 

Model hyper-parameters were chosen by considering the value of the metric in (<ref>) within [2 -0.5] s interval.  Table <ref> presents a comparison between LDCRF (FLDCRF with 1 hidden layer), FLDCRF and LSTM over nested validation sets on mtn+spa+veh (msv) systems. Two layered FLDCRF settings (2/1, 2/2, 2/3) performed better than the 1 layered settings (1/1, 1/2, 1/3, 1/4, 1/5, 1/6) within most of the inner validation sets. The new models (2/1, 2/2, 2/3) also help to enhance the overall FLDCRF performance. LSTM models perform worse than the FLDCRF models on the validation sets in general. We select best performing FLDCRF and LSTM settings for each of the three systems on the inner validation sets and average their performance on the outer folds to obtain our results.




Fig. <ref> displays the performances of different systems on 35 stopping sequences of the NTU dataset. Motion-only FLDCRF model (FLDCRF-m) has consistently low average stopping probability (see Fig. <ref>a) and classification accuracy (see Fig. <ref>b) values than models with context in the earlier regions of the curves (before `0'). Adding location context to FLDCRF (FLDCRF-ms) slightly improves time, probability and accuracy of predicting the stopping intention in earlier regions. The results improve significantly with higher average stopping probability and accuracy on adding vehicle context features to the FLDCRF model (FLDCRF-msv). On average, all LSTM models perform worse than the corresponding FLDCRF models in terms of prediction time, accuracy and probability values. 






Fig. <ref> compares different models on stopping scenarios aided by positive vehicle context (ncl≥0 and/or ncr≥0). At each instant of the plots, only frames with such vehicle context values are considered. While we find models without vehicle context (mtn and mtn+spa) to perform similarly to the earlier results, models with vehicle context (mtn+spa+veh) display an enhanced prediction performance with better average probability and accuracy in earlier regions. An average stopping probability of 0.6 and a classification accuracy of 0.7 was achieved and consistently improved from ∼0.9 seconds before the actual stopping events by FLDCRF-msv system. The vehicle context aids a gain of ∼0.5 seconds in prediction time over FLDCRF-ms system. LSTM-msv performance also improves in these scenarios, but the accuracy values still fail to approach 1 (like FLDCRF models) in the vicinity of the stopping event.

Different systems are compared in Fig. <ref> on 35 continuous crossing sequences of the NTU dataset. Accuracy-wise, all mtn and mtn+spa+veh type models display quite consistent and accurate (close to 1) performance across time. Although, joint models (mtn+spa+veh) output smaller average probability values indicating better reliability of the system. LSTM and FLDCRF joint (mtn+spa+veh) models perform comparably across time, with FLDCRF type model exhibiting marginally better probability and accuracy values at certain time points. 






Overall accuracy of considered systems at different prediction horizons on NTU dataset is displayed in Fig. <ref>. `0' in the figure represents event (stopping and crossing) instants of respective sequences. FLDCRF-msv system proves to be the most accurate at majority of early prediction instants, while performing comparable to certain systems (FLDCRF-ms and FLDCRF-m) after the `0' mark, bolstering itself to be selected as the primary choice for an early and accurate intention prediction system on the NTU data. 



Failed case analysis:

Fig. <ref> shows individual sequence outputs by the FLDCRF-msv system. We highlight sequences (in bold red) where the system fails to make early prediction (i.e., before `0') of the events. 

All crossing events were predicted correctly before respective crossing instants (see Fig. <ref>a). An unwanted spike can be observed on sequence crossing_17 near the `0' mark, caused by an approaching vehicle. However, the system was able to correct the prediction before the event instant to avoid any critical failure. 

Individual probability outputs of 35 stopping sequences by FLDCRF-msv system are displayed in Fig. <ref>b. The system fails to make early prediction (before `0') of the stopping event in the two highlighted sequences (by bold red), stopping_9 and stopping_32. 



FLDCRF-msv system performs the earliest and most accurate predictions of the stopping behaviour among considered systems and provides better reliability on crossing sequences at the same time, indicating the power of including vehicle context and better performance of FLDCRF over LSTM on the NTU dataset. Next, we investigate our approach on real-life pedestrian behaviour prediction.




 §.§ JAAD dataset


On the JAAD dataset, we evaluate on all four types of sequences: 45 continuous crossing, 20 stopping, 40 standing and 15 starting (see Section <ref>). 



We compare results from FLDCRF and LSTM on three different systems: mtn (m), mtn+spa (ms) and mtn+veh (mv). Since the JAAD data contains variety of scenarios (some during night-time), semantic segmentation <cit.> performance is not consistent. This results in mtn+spa models performing slightly worse than only mtn models. Therefore, we omit pedestrian location information from our final model, which takes pedestrian motion (x_m,t) and vehicle context (x_vJAAD,t) features.

Similar to the NTU data, we obtain our test results from a 5-fold nested CV. We make use of the same metric in (<ref>), within [1.33 -1] s interval, to select hyper-parameter settings on inner validation sets. FLDCRF (1.3640) improves over the LDCRF (1.3627) average performance on the validation sets.

Due to space limitation, we only present classification accuracy vs time by different systems on JAAD sequences. We refer to Appendix <ref> for stopping probability vs time curves. 

Fig. <ref>a compares different systems on JAAD continuous crossing sequences. FLDCRF systems produce comparable and stable performance with more than 95% average accuracy in the important region of [1 -0.5] s. LSTM-mv consistently performs worse than FLDCRF-mv in early prediction regions and lacks stability in accuracy across time. LSTM systems without vehicle context exhibit significantly poorer performance compared to other systems.

On JAAD stopping scenarios, systems with vehicle context (FLDCRF-mv and LSTM-mv) output more stable and early prediction of the stopping behaviour (see Fig. <ref>b) than mtn and mtn+spa models. These two systems also produce comparable accuracy performances across time, LSTM-mv being marginally better around the stopping instant (`0' mark). LSTM-m and LSTM-ms perform considerably better than FLDCRF-m and FLDCRF-ms respectively, but lacks stability in accuracy across time compared to the systems with vehicle context.




All considered systems produce similar prediction patterns on the JAAD starting scenarios (see Fig. <ref>c), with accuracy performances improving over time, indicating successful predictions of the change of state in pedestrian motion. FLDCRF systems perform considerably better than their LSTM counterparts. FLDCRF-mv performs best among all systems in the early prediction region [1 0] s. However, FLDCRF-m and FLDCRF-ms outperform FLDCRF-mv after the `0' mark.

We compare different systems on 15 standing sequences in Fig. <ref>d. LSTM-m and LSTM-mv perform comparably and better than other systems, reaching a stable 100% prediction accuracy around/before the critical point. FLDCRF-mv produces a stable 93% prediction accuracy around and after the critical point, failing to predict the not-crossing behaviour in time in one of the sequences. We will analyze such failed cases shortly. 

The systems are compared on the task of predicting early transitions (standing-starting and crossing-stopping) in Tables <ref> and <ref>. We consider accuracy of the systems over stipulated prediction windows. As expected, systems with vehicle context FLDCRF-mv and LSTM-mv perform significantly better than systems without vehicle context. FLDCRF-mv outperforms LSTM-mv in all early prediction windows. However, LSTM-mv performs marginally better than FLDCRF-mv after the `0' mark on crossing and stopping scenarios considered together (see Table <ref>).







FLDCRF-mv and LSTM-mv consistently performed well across all sequence types. Other systems have been better at times but lacked consistency. We can observe dominant performance of systems with vehicle context (FLDCRF-mv and LSTM-mv) in Fig. <ref>, producing superior overall accuracy compared to systems without vehicle context at all prediction instants. Moreover, FLDCRF-mv performs similar/better than LSTM-mv at all considered prediction instants, proving to be the best performing model on the JAAD dataset. FLDCRF-mv produces an average accuracy of ∼78% within the 1-0 s window and ∼82% within the 0.5-0 s window, while LSTM-mv outputs ∼75% and ∼78% in respective windows.

Figure <ref> compares the required training and inference time by the considered FLDCRF and LSTM settings on mtn+veh system. While both models needed similar time for inference, FLDCRF models required significantly lesser training time compared to the LSTM models. FLDCRF training and inference times can be further reduced by GPU implementation.





We analyze individual sequence outputs and failed cases by the FLDCRF-mv system below.



Failed case analysis:

Fig. <ref> depicts failures in bold red from FLDCRF-mv system. We denote a pedestrian sequence by its video_id and pedestrian_id in JAAD dataset, i.e. <video_id>_<pedestrian_id>.

Crossing sequences 0161_1 and 0177_2, where the system fails to establish stable and accurate (stopping probability < 0.5) outputs after -0.25 s, are highlighted in Fig. <ref>a. In sequence  0161_1, we find a momentary prediction glitch within [-0.5 -1] s window caused by a temporary hesitance from the pedestrian to continue crossing. The vehicle happened to be quite far from the pedestrian during the glitch avoiding a critical failure by the system. The failure in sequence 0177_2 is caused by inappropriate vehicle context as the ego-vehicle is moving fast quite near (<15 m) the pedestrian when the crossing event commenced in a different lane. However, the vehicle decelerated within a short period and stable, accurate output was achieved shortly after the -1.5 s mark. Such errors can be corrected by adding the lane information as context.




All stopping scenarios were predicted correctly by the FLDCRF-mv system, latest by 0.75 s after the stopping instant. The system fails to make early prediction (before `0') of the event on the highlighted sequences (0334_2 and 0336_1). A few sequences fall below probability 0.5 after the -1 s mark, but all such cases correspond to a starting event followed by the stopping event.

We have found early prediction of the `starting' event quite challenging. Fig. <ref>c highlights three starting scenarios that fail to be stable and accurate by -1.5 s on the curves. However, all of them become stable and accurate shortly after the -2 s mark. 

Standing sequence 0208_2 is wrongly predicted as `crossing' by FLDCRF-mv before and well after (≈1 s) the defined critical point. A possible reason behind this is incomplete and inaccurate vehicle context. In this case, the other vehicles before the ego-vehicle are primarily responsible for the pedestrian to remain stationery (see Fig. <ref>). The ego-vehicle is moving slowly during the event in a busy scene and is quite far from the pedestrian, causing the inaccurate context. We defined the critical point randomly due to limited number of images in the sequence. In such cases, we need to consider more complete pedestrian-vehicle interactions, which include other vehicles in the scene. 



JAAD dataset is not equipped with precise camera matrix information for the video sequences. Consequently, we utilized approximate values for these parameters, and observe our motion features to perform weaker than NTU dataset. However, combined with context features we obtain relatively early and accurate intention prediction for all four types of sequences from the FLDCRF-mv and LSTM-mv systems.




§ CONCLUSION
 

We presented a context model for pedestrian intention prediction for Autonomous Vehicles. We introduced vehicle interaction context in the problem for earlier and more accurate prediction. We also proposed Factored Latent-Dynamic Conditional Random Fields (FLDCRF) for single and multi-label sequence prediction/interaction tasks. FLDCRF led to more accurate and stable performance than LSTM over identical input features across our datasets. We plan to compare FLDCRF and LSTM on standard single and multi-label sequential machine learning datasets. We are also specifically interested in utilizing the interaction model variant of FLDCRF to capture complex pedestrian-vehicle interactions and apply the model on joint scene modeling tasks. In the system proposed in the paper, we will look to replace our pre-calibrated depth estimation with fully automated techniques, train with more data and build a real-time system on our approach. Moreover, in future work we will look to augment our intention prediction system with static scene context variables (e.g., zebra crossing, bus stop, lanes etc.) by attention models and propose more complete end-to-end models involving general static and dynamic interactions. 



















































































































§ ACKNOWLEDGMENT


We are very thankful to all the reviewers for their valuable feedback.






























1

Daimler
C. G. Keller, D.M. Gavrila, “Will the Pedestrian Cross? A Study on Pedestrian Path Prediction", IEEE Transactions On Intelligent Transportation Systems, VOL. 15, NO. 2, APRIL 2014.

Ped-crossing-intent

F. Schneemann, P. Heinemann, “Context-based Detection of Pedestrian Crossing Intention for Autonomous Driving in Urban Environments", International Conference on Intelligent Robots and Systems (IROS), 2016.

LDCRF-intent
A. Schulz and R. Stiefelhagen, “Pedestrian Intention Recognition using Latent-dynamic Conditional Random Fields", 2015 IEEE Intelligent Vehicles Symposium (IV).

Intent-intersections
S. Köhler, M. Goldhammer, S. Bauer, S. Zecha, K. Doll, U. Brunsmann, K. Dietmayer, “Stationary Detection of the Pedestrian’s Intention at Intersections", IEEE Intelligent Transportation Systems Magazine (Volume: 5, Issue: 4, winter 2013).


Motion-classification
J. Hariyono and K. H. Jo, “Pedestrian Action Recognition using Motion Type Classification", 2015 IEEE 2nd International Conference on Cybernetics (CYBCONF).

Particle-intent

Y. Hashimoto, Y.Gu, L. Hsu, M. Iryo-Asano, S. Kamijo, “A Probabilistic Model of Pedestrian Crossing Behavior at Signalized Intersections for Connected Vehicles”, Transportation Research Part C: Emerging Technologies, Volume 71, October 2016, Pages 164-181.

HMM-intent

R. Quintero, I. Parra, J. Lorenzo, D. Fernandez-Llorca and M. A. Sotelo, “Pedestrian Intention Recognition by Means of a Hidden Markov Model and Body Language", 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).

JAAD-application

A. Rasouli, I. Kotseruba, J. K. Tsotsos. “Are They Going to Cross? A Benchmark Dataset and Baseline for Pedestrian Crosswalk Behavior", ICCVW (2017). 

JAAD-application2

P. Gujjar, R. Vaughn, “Classifying Pedestrian Actions In Advance Using Predicted Video of Urban Driving Scenes", ICRA 2019.

Openpose

Z. Fang, D. Vázquez, A. M. López, “On-Board Detection of Pedestrian Intentions", Sensors, 2017, doi: 10.3390/s17102193.

Michael

M. Hoy, Z. Tu, D. Kang, and J. Dauwels (2018), “Learning to Predict Pedestrian Intention via Variational Tracking Networks", Proceedings of the IEEE 21st International Conference on Intelligent Transportation Systems, Maui, USA [accepted].

Context-based
J. F. P. Kooij, N. Schneider, F. Flohr, D. M. Gavrila, “Context-Based Pedestrian Path Prediction", ECCV 2014.

LDCRF-path

A. T. Schulz, R. B. GmbH, “A Controlled Interactive Multiple Model Filter for combined Pedestrian Intention Recognition and Path Prediction", 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

Bayesian-filters
N. Schneider, D. M. Gavrila, “Pedestrian Path Prediction with Recursive Bayesian Filters”, GCPR 2013.

traj-est2

V. Karasev, A. Ayvaci, B. Heisele, S. Soatto, “Intent-Aware Long-Term Prediction of Pedestrian Motion", ICRA 2016.

LSTM

A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, S. Savarese, “Social LSTM: Human Trajectory Prediction in Crowded Spaces", CVPR 2016.

traj-est1

N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. S. Torr, M. Chandraker, “DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents", CVPR 2017.

mypaper
S. Neogi, M. Hoy, W. Chaoqun, J. Dauwels, “Context Based Pedestrian Intention Prediction Using Factored Latent Dynamic Conditional Random Fields", IEEE SSCI-2017.

JAAD

I. Kotseruba, A. Rasouli, J. K. Tsotsos, “Joint Attention in Autonomous Driving (JAAD)." arXiv preprint arXiv:1609.04741 (2016). 

CRF-first

J. Lafferty, A. McCallum, F. C. N. Pereira, “Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data", ACM, June 2001.

DCRF
C. Sutton, K. Rohanimanesh, A. McCallum, “Dynamic Conditional Random Fields: Factorized Probabilistic Models for Labeling and Segmenting Sequence Data", The Journal of Machine Learning Research, Volume 8, 5/1/2007, Pages 693-723.

semi-CRF

S. Sarawagi, W. W. Cohen, . “Semi-Markov Conditional Random Fields for Information Extraction". In Lawrence K. Saul, Yair Weiss, Léon Bottou (eds.). (2005). Advances in Neural Information Processing Systems 17. Cambridge, MA: MIT Press. pp. 1185–1192.

LDCRF
L. P. Morency, A. Quattoni, T. Darrell, “Latent-Dynamic Discriminative Models for Continuous Gesture Recognition", CVPR, IEEE Computer Society, (2007).

HMM
L. R. Rabiner. “A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition". Proceedings of the IEEE, 77(2):257–286, 1989.

Factorial-HMM

Z. Ghahramani, M. I. Jordan, “Factorial Hidden Markov Models", Machine Learning (1997) 29: 245. https://doi.org/10.1023/A:1007425814087.

traj-est3

D. Varshneya, G. Srinivasaraghavan, “Human Trajectory Prediction using Spatially aware Deep Attention Models", NIPS 2017.

traj-est5

W. C. Ma, D. Huang, N. Lee, K. M. Kitani, “Forecasting Interactive Dynamics of Pedestrians with Fictitious Play", CVPR 2017.

action-lstm

A. Ullah, J. Ahmad, K. Muhammad, M. Sajjad, S. W. Baik, “Action Recognition in Video Sequences using Deep Bi-directional LSTM with CNN features", IEEE Access, 2017.

Viterbi
G. D. Forney Jr. “The Viterbi algorithm". Proceedings of the IEEE. 61 (3): 268–278. doi:10.1109/PROC.1973.9030, (March 1973).

HCRF

S. B. Wang, A. Quattoni, L. P. Morency, D. Demirdjian, T. Darrell, “Hidden Conditional Random Fields for Gesture Recognition", IEEE Transactions on Pattern Analysis and Machine Intelligence (Volume: 29, Issue: 10, Oct. 2007).

Stan
B. Carpenter, D.Lee, M. A. Brubaker, A. Riddell, A. Gelman, B. Goodrich, J. Guo, M. Hoffman, M. Betancourt, P. Li, “Stan: A Probabilistic Programming Language", Journal of Statistical Software, Volume VV, Issue II, Nov. 2017.

Optical Flow
C. Liu. “Beyond Pixels: Exploring New Representations and Applications for Motion Analysis". Doctoral Thesis. Massachusetts Institute of Technology. May 2009.

obj-det

S. Ren, K. He, R. Girshick, J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", NIPS 2015.

semantic-seg

A. Kendall, V. Badrinarayanan, R. Cipolla, “Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding", BMVC 2017.

MATLAB
MATLAB 8.0 and Statistics Toolbox 8.1, The MathWorks, Inc., Natick, Massachusetts, United States.

Daimler Dataset
N. Schneider and D. M. Gavrila, “Pedestrian Path Prediction with Recursive Bayesian Filters: A Comparative Study", In Lecture Notes in Computer Science: Proc. of the German Conference on Pattern Recognition (GCPR), vol. 8142, Springer, 2013.

CMU-data

CMU, “Cmu graphics lab motion capture database", http://mocap.cs.cmu.edu/subjects.php.

FLDCRF-arxiv S. Neogi, J. Dauwels, “Factored Latent-Dynamic Conditional Random Fields for Single and Multi-label Sequence Modeling", URL: https://arxiv.org/abs/1911.03667, 2019.
















[
    < g r a p h i c s >
]Satyajit Neogi
received his Bachelor of Engineering (B.E.) degree in Electronics and Telecommunications Engineering from Jadavpur University, Kolkata, India. He is currently pursuing his Ph.D. in Electrical Engineering at Nanyang Technological University, Singapore. His research interests include Probabilistic Graphical Models, Image and Video Processing and Scene Understanding.

0pt plus -1fil

[
    < g r a p h i c s >
]Michael Hoy
received a Ph.D. degree (Electrical Engineering) from the University of New South Wales, Australia in 2013. He is currently a poctdoctoral researcher at Nanyang Technological University, Singapore. His current research interests include Computer Vision, Statistical Signal Processing and perception for Autonomous systems (in particular, methods for applying deep learning to problems from these areas).




0pt plus -1fil
[
    < g r a p h i c s >
]Kang Dang
received a Ph.D. degree from Nanyang Technological University in 2018. He is currently a research intern at Tencent Youtu Lab in Shanghai, China. His research interests include Computer Vision and Medical Image Analysis.

0pt plus -1fil
[
    < g r a p h i c s >
]Hang Yu
Hang Yu (S'12-M'15) received the B.E. degree in Electronic and Information Engineering from University of Science and Technology Beijing (USTB), China in 2010, and the Ph.D. degree in Electrical and Electronics Engineering from Nanyang Technological University (NTU), Singapore in 2015. He is now a Postdoctoral Research Fellow in Centre for System Intelligence and Efficiency (EXQUISITUS), NTU, under the guidance of Prof. Justin Dauwels. His research interests include Statistical Signal Processing, Machine Learning, Graphical Models, Copulas, and Extreme-events modeling.

0pt plus -1fil
[
    < g r a p h i c s >
]Justin Dauwels
is an Associate Professor with the School of Electrical and Electronic Engineering at Nanyang Technological University (NTU), Singapore. He is also the Deputy Director of the ST Engineering-NTU corporate lab. His research interests lie in Probabilistic Graphical Models, Intelligent Transportation and Human behaviour analysis. He received a PhD degree in Electrical Engineering from the Swiss Polytechnical Institute of Technology (ETH), Zurich in 2005. He was a postdoctoral fellow at the RIKEN Brain Science Institute (2006-2007) and a research scientist at the Massachusetts Institute of Technology (2008-2010). He has been a JSPS postdoctoral fellow (2007), a BAEF fellow (2008), a Henri-Benedictus fellow of the King Baudouin Foundation (2008), and a JSPS invited fellow (2010, 2011). His research on intelligent transportation systems has been featured by the BBC, Straits Times and Channel 5. His research on diagnosis of Alzheimers disease is featured at a 5-year exposition at the Science Center in Singapore. His research team has won several best paper awards at international conferences. He has filed 10+ US patents related to data analytics.




















§ LATENT DYNAMIC CONDITIONAL RANDOM FIELDS (LDCRF)
 

In this section, we will describe the LDCRF <cit.> mathematical model.



 §.§ Model


The task is to learn a probabilistic mapping between a time-series sequence of observed input features x = {x_1, x_2, ... , x_T} and a sequence of observed classifcation labels y = {y_1, y_2, ... , y_T}. y_t ∈ Υ, ∀ j = 1,2, ... , T, where Υ is the set of classification labels. h = {h_1, h_2, ... , h_T} denotes the hidden layer for capturing intrinsic dynamics within each label. Each label ℓ ∈ Υ is associated with a set of hidden states ℋ_ℓ. ℋ is the set of all possible hidden states written as ℋ = ⋃_ℓℋ_ℓ. ℋ_ℓ are disjoint ∀ℓ ∈ Υ. Each h_t is restricted to belong to the set ℋ_y_t, i.e., h_t ∈ ℋ_y_t, ∀ t = 1,2, ..., T . The conditional model is defined as:


    P(y|x, θ) = ∑_h P(y|x, h, θ) P(h|x, θ).


Equation (<ref>) can be re-written using the graph structure in Fig. <ref>c as:


    P(y|x, θ) = ∑_h:∀ h_t ∈ℋ_y_t P(y|h, θ) P(h|x, θ) 
     +
    ∑_h:∃ h_t ∉ℋ_y_t P(y|h, θ) P(h|x, θ).


Applying model constraints, we can write the following:



    P(y_t=ℓ|h_t) = 
    
    1,     h_t ∈ℋ_y_t=ℓ
    
    0,     h_t ∉ℋ_y_t=ℓ.


The model in equation (<ref>) can be simplified using equation (<ref>) as:


    P(y|x, θ) = ∑_h:∀ h_t ∈ℋ_y_t P(h|x, θ).


P(h|x, θ) is described using Conditional Random Field formulation given by:


    P(h|x, θ) = 1/Z(x,θ)exp(∑_k θ_k . F_k(h,x)),


where index k ranges over all parameters θ = {θ_k} and Z(x,θ) is the partition function defined as:


    Z(x,θ) = ∑_hexp(∑_k θ_k . F_k(h,x)).


The feature functions F_k's are defined as: 
    F_k(h,x) = ∑_t=1^T f_k(h_t-1, h_t, x, t),
 

Feature functions f_k(h_t-1, h_t, x, t) can be either an observation (also called state) function s_k(h_t, x, t) or a transition function t_k(h_t-1, h_t, x, t).



 §.§ Training Model Parameters
 

Parameters of the model can be estimated by maximizing the conditional log-likelihood of the training data given by equation (<ref>):


    L(θ) = ∑_n=1^N log P(y^(n)|x^(n), θ) - ∥θ∥^2/2σ^2,


where N is the total number of available labeled sequences. The second term in equation (<ref>) is the log of a Gaussian prior with variance σ^2.



 §.§ Inference


Given a new test sequence x, the inference task is given by:


    ŷ = argmax_y   P(y|x, θ̂),


Using model constraints, equation (<ref>) can be re-written as:


    ŷ = argmax_y  ∑_h:∀ h_t ∈ℋ_y_t P(h|x, θ̂),


We apply forward recursions of belief propagation for our inference as the problem of intention prediction must be solved online. In other words, at each time instant t, we compute the marginals P(h_t |x_1:t, θ) and sum them according to the disjoint sets of hidden states to obtain P(y_t |x_1:t, θ) = ∑_h_t ∈ℋ_y_t P(h_t |x_1:t, θ), t = 1,2, .... Then, we infer the label y_t corresponding to the maximum probability. Forward-backward algorithm <cit.> and Viterbi algorithm <cit.> can also be applied for problems where online inference is not required. 






§ REAL WORLD LATERAL MOTION ESTIMATION FROM MONOCULAR IMAGE SEQUENCE
 

As we find a pedestrian's intent of crossing/not-crossing is characterized by the real world lateral component of his/her motion (w.r.t. the ego-vehicle moving direction), we attempt to extract an approximate equivalent of such motion component in the image plane of a monocular image sequence using the camera matrix information. 



Fig. <ref> depicts the scenario. O is the camera center and I_p is the principle point on image plane. x_R and y_R are the lateral and longitudinal (movement direction of the ego-vehicle) directions in real world respectively. P⃗Q⃗ = S⃗ represents relative displacement of a pedestrian point w.r.t. the ego-vehicle in real world between two consecutive time instants. We assume negligible pedestrian displacement along the vertical real world direction z_R. As we can see, corresponding displacement along x_i image axis is S_ix, which contains components from both real world lateral (S_x) and longitudinal displacements (S_y), projected as S_xx and S_yx respectively onto the image plane, i.e., S_ix = S_xx + S_yx. Similarly, considering displacement along z_i image axis to be S_iz, we can write the following two equations:


    S_ix   = ± r_x ·θ_x ± r_x ·θ_y, 
    
    S_iz   = ± r_z ·θ_y


Since, we assume negligible pedestrian motion along vertical real world direction, S_iz only carries traces of pedestrian's real world longitudinal motion. ± indicates the components can be additive or subtractive depending on the image quadrant of movement. Considering + θ_x and + θ_y to reflect real world +ve x and +ve y direction respectively, different possible scenarios are illustrated on the image plane x_iz_i in Fig. <ref>. r_x and r_z are the known respective distances from camera center O to the center of the displacements (situated at (p_x, p_z) on image plane) along axes x_i and z_i, given by,


    r_x = √(f^2 + p_x^2),   
    r_z = √(f^2 + p_z^2)


The center of displacement is considered to minimize error in approximating s = r ·θ (instead of s⃗ = r ·θ·θ̂+ dr ·r̂), to reduce number of unknown variables. The goal is to solve for θ_x and θ_y using the equations illustrated in Fig. <ref>.



S_xx gives approximate pedestrian's lateral motion equivalent on image plane and is obtained using S_xx = ± r_x ·θ_x.



§ DAIMLER DATASET
 

In this section, we present results of our motion-only approach on the Daimler dataset <cit.>.

Daimler Dataset is a public pedestrian benchmark dataset introduced in <cit.>. It contains several single pedestrian movement sequences captured at 16 fps using a stereo camera setup mounted on a moving vehicle. The movements primarily include three distinct scenarios - a) A laterally moving pedestrian `crossing' or `stopping' before a vehicle and b) A pedestrian moving along the sidewalk `bending in' to the road and c) A pedestrian `starting' to cross the road in front of a vehicle from a static position. Ground truth labels are provided by means of bounding boxes. For all scenarios, important events like `stopping' , `crossing', `bending' and `starting' instants are marked as `0'-th instant. The whole dataset contains 36 training and 32 testing scenarios. Each video sequence has a resolution 1176 × 640.

We evaluate on the same train and test splits as in <cit.>. We also utilize same training labels as in <cit.>. For training crossing scenarios, intent label `crossing' has been marked throughout. For training stopping sequences, data from 20 frames before the stopping instant (specified in the dataset for each stopping sequence) has been labeled as `stopping' intent (or `not-crossing', as we call it). For Daimler dataset, our feature vector x_t,Daim only contains pedestrian motion features, i.e. x_t,Daim = {x_m,t}.



 §.§ Results
 

Fig. <ref> presents a comparison of our approach with the best results presented in <cit.>. `0' along x-axis represent the event (`crossing' or `stopping') instants and are labeled for all test sequences in the dataset. As stated earlier, we find vehicle context irrelevant within the Daimler dataset and hence only test our motion based approach for intention prediction, i.e. here x_t,Daim = {x_m,t}. We apply a FLDCRF with 1 layer and 4 states per label over our motion features. 

The best results in <cit.> are obtained from a Gaussian Process Dynamic Model (GPDM) applied on dense optical flow derived features from stereo images. As we can see from Fig. <ref>, our motion features with FLDCRF perform comparably to the results in <cit.>. While our approach performs marginally better for the stopping scenarios, <cit.> is more consistent through time over the crossing sequences. Our methods although enjoys the liberty of using only single camera images.







§ JAAD ADDITIONAL MATERIAL
 



 §.§ Sequence details
 

Considered JAAD sequences are presented in Fig. <ref>. A pedestrian sequence is denoted by its video_id and pedestrian_id in JAAD dataset, in the following format: <video_id>_<pedestrian_id>.





 §.§ Stopping Probability vs Time curves
 

Figures <ref>(a)-(d) compare performances of different systems on stopping, continuous crossing, starting and standing sequences of JAAD dataset by the stopping probability vs time performances.



While FLDCRF-(mtn+veh) and LSTM-(mtn+veh) produce comparable probability performances on continuous crossing and standing sequences, one outperforms the other in terms of stopping probability on each of the remaining two sequence types (stopping and starting).



 §.§ Early prediction tables
 

The systems are compared on the task of predicting early transitions (standing-starting and crossing-stopping) in Tables <ref> and <ref>. We consider accuracy of the systems over stipulated prediction windows. As expected, systems with vehicle context FLDCRF-mv and LSTM-mv perform significantly better than systems without vehicle context. FLDCRF-mv outperforms LSTM-mv in all early prediction windows. However, LSTM-mv performs marginally better than FLDCRF-mv after the `0' mark on crossing and stopping scenarios considered together (see Table <ref>).








 §.§ Failed cases
 

Also described in Main text with the figures. See pages 12-13.

Fig. <ref> depicts failures in bold red from FLDCRF-mv system. We denote a pedestrian sequence by its video_id and pedestrian_id in JAAD dataset, i.e. <video_id>_<pedestrian_id>.

Crossing sequences 0161_1 and 0177_2, where the system fails to establish stable and accurate (stopping probability < 0.5) outputs after -0.25 s, are highlighted in Fig. <ref>a. In sequence  0161_1, we find a momentary prediction glitch within [-0.5 -1] s window caused by a temporary hesitance from the pedestrian to continue crossing. The vehicle happened to be quite far from the pedestrian during the glitch avoiding a critical failure by the system. The failure in sequence 0177_2 is caused by inappropriate vehicle context as the ego-vehicle is moving fast quite near (<15 m) the pedestrian when the crossing event commenced in a different lane. However, the vehicle decelerated within a short period and stable, accurate output was achieved shortly after the -1.5 s mark. Such errors can be corrected by adding the lane information as context.















All stopping scenarios were predicted correctly by the FLDCRF-mv system, latest by 0.75 s after the stopping instant. The system fails to make early prediction (before `0') of the event on the highlighted sequences (0334_2 and 0336_1). A few sequences fall below probability 0.5 after the -1 s mark, but all such cases correspond to a starting event followed by the stopping event.

We have found early prediction of the `starting' event quite challenging. Fig. <ref>c highlights three starting scenarios that fail to be stable and accurate by -1.5 s on the curves. However, all of them become stable and accurate shortly after the -2 s mark. 

Standing sequence 0208_2 is wrongly predicted as `crossing' by FLDCRF-mv before and well after (≈1 s) the defined critical point. A possible reason behind this is incomplete and inaccurate vehicle context. In this case, the other vehicles before the ego-vehicle are primarily responsible for the pedestrian to remain stationery (see Fig. <ref>). The ego-vehicle is moving slowly during the event in a busy scene and is quite far from the pedestrian, causing the inaccurate context. We defined the critical point randomly due to limited number of images in the sequence. In such cases, we need to consider more complete pedestrian-vehicle interactions, which include other vehicles in the scene. 




§ NTU DATA FAILED CASES
 

We analyze the stability of sequence predictions and failed cases by the FLDCRF-msv system on NTU data, as depicted in Fig. <ref>. Material with the figure is also available in main text, see pages 10-11.

Fig. <ref> shows individual sequence outputs by the FLDCRF-msv system. We highlight sequences (in bold red) where the system fails to make early prediction (i.e., before `0') of the events. 

All crossing events were predicted correctly before respective crossing instants (see Fig. <ref>a). An unwanted spike can be observed on sequence crossing_17 near the `0' mark, caused by an approaching vehicle. However, the system was able to correct the prediction before the event instant to avoid any critical failure. 

Individual probability outputs of 35 stopping sequences by FLDCRF-msv system are displayed in Fig. <ref>b. The system fails to make early prediction (before `0') of the stopping event in the two highlighted sequences (by bold red), stopping_9 and stopping_32. 











