vehicletrajectorypredictionbyknowledge-drivenlstm ... · researcharticle...

Research ArticleVehicle Trajectory Prediction by Knowledge-Driven LSTMNetwork in Urban Environments

Shaobo Wang ,1,2,3,4 Pan Zhao ,1,3,4 Biao Yu ,1,3,4 Weixin Huang ,1,2,3,4

and Huawei Liang 1,3,4

1Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei 230031, China2University of Science and Technology of China, Hefei 230026, China3Anhui Engineering Laboratory for Intelligent Driving Technology and Application, Hefei, Anhui, China4Innovation Research Institute of Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, Hefei, Anhui, China

Correspondence should be addressed to Huawei Liang; [email protected]

Received 9 September 2020; Revised 21 September 2020; Accepted 21 October 2020; Published 7 November 2020

Academic Editor: Kun Wang

Copyright © 2020 ShaoboWang et al.(is is an open access article distributed under the Creative Commons Attribution License,which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

An accurate prediction of future trajectories of surrounding vehicles can ensure safe and reasonable interaction between in-telligent vehicles and other types of vehicles. Vehicle trajectories are not only constrained by a priori knowledge about roadstructure, traffic signs, and traffic rules but also affected by posterior knowledge about different driving styles of drivers. (eexisting predictionmodels cannot fully combine the prior and posterior knowledge in the driving scene and performwell only in aspecific traffic scenario. (is paper presents a long short-term memory (LSTM) neural network driven by knowledge. First, adriving knowledge base is constructed to describe the prior knowledge about a driving scenario. (en, the prediction referencebaseline (PRB) based on driving knowledge base is determined by using the rule-based online reasoning system. Finally, the futuretrajectory of the target vehicle is predicted by an LSTM neural network based on the prediction reference baseline, while thepredicted trajectory considers both posterior and prior knowledge without increasing the computation complexity. (e ex-perimental results show that the proposed trajectory prediction model can adapt to different driving scenarios and predicttrajectories with high accuracy due to the unique combination of the prior and posterior knowledge in the driving scene.

1. Introduction

Since the 1980s, autonomous vehicles have been regarded aseffective solutions to the problems of road safety, trafficcongestion, and energy crisis. However, autonomous vehi-cles still face many driving difficulties in the real urban trafficenvironment. A major problem is how to interact safely andreasonably with other types of vehicles in a driving scene.Experienced human drivers can predict the future trajectoryof other vehicles in a driving scene, thereby making safe,reasonable, and efficient decisions. Accurately predicting thefuture trajectory of a vehicle not only can reduce or eliminatethe collision risk when autonomous vehicles performcomplex drivingmaneuvers, such as merge, lane change, andovertaking, but also can improve the driving efficiency andcomfort of autonomous vehicles [1]. In a real urban trafficscenario, the vehicle’s driving trajectory is not only

constrained by prior knowledge, such as that about the roadstructure, traffic signs, and traffic rules, but also by uncertainposterior knowledge, including subjective driving intentionsof the driver. (e influence of driving knowledge on vehicletrajectory is shown in Figure 1, where it can be seen thatwhen the road structure constraints are not considered, thepredicted future trajectory, denoted as the red curve, isincorrect. As shown in Figure 1(b), there is a large slow-moving truck in front of the target vehicle. In such a case,based on human driving experience, the target vehicle islikely to adopt a lane change strategy.(erefore, how to fullycombine the prior and posterior knowledge in a drivingscene in the prediction process is crucial for accuracy im-provement of the long-term trajectory prediction and safeinteraction with other vehicles.

According to the specific prediction process, the existingprediction models can be roughly divided into three

HindawiJournal of Advanced TransportationVolume 2020, Article ID 8894060, 20 pageshttps://doi.org/10.1155/2020/8894060

mailto:[email protected]://orcid.org/0000-0001-7775-6760https://orcid.org/0000-0002-2579-1254https://orcid.org/0000-0003-0472-157Xhttps://orcid.org/0000-0002-1053-0641https://orcid.org/0000-0003-1581-304Xhttps://creativecommons.org/licenses/by/4.0/https://doi.org/10.1155/2020/8894060

categories: physics-based models, maneuver-based models,and learning-based models [2]. (e physics-based modelsuse vehicle kinematics and dynamics model to predict thefuture position of a target vehicle, and they include theconstant turn rate and acceleration model [3], switchingKalman filters [4], and Monte Carlo simulation [5]. How-ever, these models ignore the prior and posterior knowledgeabout a driving scenario, such as road structure, traffic rules,and driver’s subjective intentions, which limits these modelsto short-term prediction (less than 1 s) [6].

Maneuver-based models divide the prediction processinto two parts. First, driving intention is estimated accordingto the physical state of a vehicle, information about the roadnetwork, and driver behavior, and then the predicted tra-jectory is fitted based on the driving intention. For maneuverclassification in more complex scenarios, discriminativelearning algorithms, including the multilayer perceptions(MLPs) [7], logistic regression [8], relevance vector ma-chines (RVMs) [9], and support vector machines (SVMs)[10], have been very popular. Complex vehicle motion isdecomposed into predefined driving action sequences,which makes driving intention easier to identify and classify,and the prediction result is more stable and accurate thanthat of the physics-based models, and the prediction horizonis longer. However, in complex traffic scenarios, the tradi-tional algorithms, such as finite vector machines and con-ditional random fields, have the problem of low sceneadaptability, while Bayesian network andMarkov model cansolve the problem of driving maneuver classification inuncertain environments. In addition, the state space of theabove models is extremely large, and these models are proneto “curse of dimensionality,” and unable to real-time pre-diction. (e approach of [11] predicts the fictive collisionprobabilities stemming from the execution of each intentionby updating a prior intention distribution based on pos-tulation that drivers do not perform maneuvers with highcollision risks, but this assumption prevents the detection ofspecific dangerous maneuvers that are conducive to drivingefficiency. Recently, artificial neural networks have beenused to classify vehicle driving actions, but the existing high-quality calibrated datasets are limited and cannot cover allpossible driving scenarios (data sparsity) [12], which makesthe network training difficult and challenging, and sceneadaptability is low.

Learning-based models skip the step of maneuver rec-ognition and perform trajectory prediction directly based onthe historical observation of a target vehicle, so the posteriorknowledge in driving scenarios can be effectively learned,and incorrect driving motion recognition can be avoided.Recently, artificial neural networks have been used to predictfuture trajectories of vehicles, bicycles, and pedestrians[13–15]. As a type of recurrent neural network (RNN), thelong short-term memory (LSTM) neural network has beenproven to be very effective in solving the time seriesproblems, and thus has been widely used in pedestriantrajectory prediction, intersection vehicle destination pre-diction, and highway vehicle trajectory prediction. However,in previous works, specific-scenario models, such as lanechange models for nonintersection sections and left-/right-turn models for intersection areas, have been proposed[10, 16, 17], and the training data needed manual annota-tion, which increased the training difficulty of the model. In[18], an encoder-decoder LSTM model is proposed forpredicting vehicle trajectory by using an occupancy gridmap, and the maximum prediction horizon of this model istwo seconds, which is not sufficient for applications.

When an intelligent vehicle is driving in a real urbanenvironment, the driving scene changes dynamically overtime, which means that the prediction model should au-tomatically adapt to a driving scene. In order to solve theproblem of vehicle adaptability to the driving scene, manystudies incorporated the maneuver-based and learning-based models. In [19], two LSTMs were used to identifyhigh-level driver intentions and analyze low-level complexvehicle motion dynamics. (is method is better geography-adaptive than the traditional LSTM networks. An LSTMmodel for interaction aware motion prediction of sur-rounding vehicles on freeways was presented in [20]. (ismodel assigns confidence values to maneuvers being per-formed by vehicles and outputs a multimodal distributionover future motion based on these values. (e mentionedmethods predict the multimodal trajectory based on ma-neuver classes, which improves the road adaptability, but theprior knowledge in driving scenarios is not used. In [21], along short-term memory (LSTM) network was employed toanticipate the driving policy of a vehicle (such as forward,yield, turn left, and turn right) using its sequential historyobservations. (e policy was then used to guide a low-level

(a)

V1

V2

V3 V0

(b)

Figure 1: Influence of driving knowledge on trajectory prediction: (a) influence of road structure on trajectory prediction; (b) influence ofdriving experience on trajectory prediction.

2 Journal of Advanced Transportation

optimization-based context reasoning process. (is methodcombines the prior knowledge in the driving scene andconstructs the cost map to perform the second optimizationof the previously obtained driving intention to generate thefinal predicted trajectory, but the driving intention esti-mation of the upper-level does not utilize the priorknowledge of the driving scene, and the weight of thefunction cannot be adjusted adaptively to a driving scenario.Deo and Trivedi [22] adopt a convolutional social poolingLSTM-based model. (is approach predicts a distribution offuture vehicle trajectory dependent on maneuver, but thisapproach ignores the impact of the interaction of the roadusers. Dai et al. [23] proposed a spatiotemporal LSTM-basedmodel, which considers the spatial interactions of the sur-rounding vehicles, but the constraints of other priorknowledge such as road structure, traffic rules, and drivingexperience are not considered. (e dual learning model(DLM) which takes information from two different inputs topredict vehicle trajectory was presented in [24]. (is modelembeds the occupancy map and risk map into the trajectorymodel to consider a comprehensive definition of risk in thetraffic scene, but the computational complexity usuallygrows exponentially if the dimensionality of the featurespace increases. (us, it becomes difficult to meet the onlinerequirement.

In this article, an integrated trajectory prediction model,which combines knowledge reasoning and LSTM neuralnetworks, is proposed. (e contribution of this study can besummarized as follows:

(1) In order to consider the constraints of the priorknowledge. (e prediction reference baseline ob-tained by knowledge reasoning is introduced into theLSTM network, where the proposed model can ef-fectively combine the prior knowledge without in-creasing the computation complexity.

(2) In order to learn the spatial interactions of thesurrounding vehicles and solve combinatorial ex-plosion problem caused by a large number of con-dition attributes. A method of deterministic sceneevaluation is employed to classify and analyze themain conditions that affect the future trajectory of avehicle from the perspectives of safety, legitimacy,and reasonableness, which simplifies modeling of thespatial interactions.

(3) In order to improve the adaptability of the proposedmodel. (e Frenet coordinates based on the PRB areused to train the LSTM network, and it is notnecessary to annotate the training data set manuallyaccording to the specific driving scenario.(e resultsof the field test prove the adaptive performance of theproposed model.

(4) (e performance of the proposed model is evaluatedwith state-of-the-art methods on a naturalistichighway driving dataset (NGSIM), and the resultsshow that our proposed model outperforms thestate-of-the-art methods.

(e rest of the paper is organized as follows. (e pre-diction reference baseline determination method and theproposed LSTM network are presented in Section 2. (eproposed prediction model is evaluated by both simulationsand real-traffic urban roadway experiments, and the ob-tained results are presented and discussed in Section 3.Finally, the main conclusions, limitations, and future workare presented in Section 4.

2. Materials and Methods

2.1. Problem Formulation and Method Overview

2.1.1. Problem Formulation. (e proposed trajectory pre-diction model is divided into two layers. (e first layerdetermines the PRB of a target vehicle, and the second layerpredicts the future trajectory based on the PRB. PRB is atrajectory that indicates the driving intention of the targetvehicle based on prior driving knowledge, which connectsthe online reasoning system and the LSTM network.

(e process of driving intention prediction for a targetvehicle Vi at time t is presented in Figure 2, where it can beseen that it is necessary to understand and evaluate drivingscene Stvi , and generate the scene evaluation parameters E

tviof

the target vehicle, including the safety assessment SAFEtvi ,legitimacy assessment LEGALtvi , and reasonable assessmentREASONABLEtvi , which is expressed as

Etvi

� SAFEtvi , LEGALtvi

, REASONABLEtvi . (1)

According to the prior knowledge of driving scenarios,such as traffic rules and driving experience, the drivingintention btvi of the target vehicle is inferred based on theprolog online reasoning system. A maneuver btvi is classifiedby the lateral movement of the vehicle, which is expressed bya finite set B:

btvi∈B ∶ � LK, LCL, LCR,TR,TL,GS, SS, . . . ,{ }. (2)

Finite set B includes the following maneuvers: lanekeeping (LK), lane change to left (LCL), lane change to right(LCR), turn right (TR), turn left (TL), go straight at inter-section (GS), and stop before the stop line (SS).

Finally, driving intention btvi is fitted to the PRB PRBtviby

the cubic Bezier curves.(e second layer predicts the future vehicle trajectory. First,

the coordinate transformation is performed on the historicaltrajectory of the target vehicle based on the PRB PRBtvi , asshown in Figure 3. In Figure 3, stvi denotes the distance thetarget vehicle has traveled along the PRB, and ltvi is thetransverse distance between the target vehicle and the PRBtvi.(e absolute position denoted as (lattvi , lng

tvi

) is transformed tothe Frenet coordinates that are denoted as (stvi , l

tvi

). (e set ofobservation vectors denoted as Ot(n)vi is used for trajectoryprediction of the target vehicle, and it is given by

Ot(n)vi

� stvi

, ltvi

, κtvi , θtvi

, vtvi

, atvi

, n � t − M + 1, . . . , t,

(3)

Journal of Advanced Transportation 3

where a set (stvi , ltvi

) denotes the Frenet coordinates, κtvi denotesthe curvature, θtvi represents vehicle heading, v

tvidenotes the

vehicle speed, atvi denotes the vehicle acceleration, andM is theinput step of the network.

(e network output Pt(n)vi is expressed as

Pt(n)vi

� stvi

, ltvi

, vtvi

, , � t + 1, t + 2, . . . , t + K, (4)

where K denotes the output step of the network.Finally, the predicted trajectory denoted as Trat(n)vi is

obtained by transforming the reference coordinates to theabsolute coordinates represented by the latitude and lon-gitude, which is expressed as

Trat(n)vi � lattvi

, lngtvi , vtvi

, n � t + 1, t + 2, . . . , t + K.

(5)

2.1.2. Overview of the Proposed Approach. (is paper pro-poses a trajectory prediction model based on knowledgereasoning and LSTM neural network.(e architecture of theproposed model is shown in Figure 4, where it can be seenthat the proposed model consists of two phases: PRB de-termination phase and trajectory prediction phase. Duringthe PRB determination phase, by analyzing the relationshipbetween “human-vehicle-road” in the driving scene andextracting the knowledge of road network, traffic partici-pants, and road traffic facilities, the conceptual ontologymodel of the driving scene is established. (e main con-ditional attributes that affect the behavior decision-makingprocess are classified and analyzed from the perspectives of

safety, legitimacy, and reasonableness using the proposeddeterministic situation assessment method, and situationparameters in the horizontal and vertical directions areobtained. (e behavior prediction rule base is constructedusing the situation parameters, traffic rules, and drivingexperience. Based on the prolog online reasoning system, thebehavioral prediction rules are matched with the factualknowledge obtained by the conceptual ontology model, andthe driving intentions are inferred. Finally, a third-orderBezier curve is used to fit the driving intention to a PRB.(etrajectory prediction phase uses the LSTM network to learnthe continuous features of the historical trajectory of a targetvehicle on the basis of the PRB and generates the finalpredicted trajectory.

2.2. Prediction Reference Baseline Determination. (e ar-chitecture of the proposed PRB determination method ispresented in Figure 5, where it can be seen that this methodconsists of online and offline phases. (e offline phase es-tablishes the conceptual ontology model (Tbox) of a drivingscene and extracts the behavioral prediction rules based ontraffic rules and driving experience. According to the con-ceptual ontology model, the road network and real-timeenvironment perception information are used to instantiatethe entities and related relationships in the driving scene(Abox). (e entities and entity relationships in the drivingscene are classified by the deterministic scene assessmentmethod and analyzed from the perspectives of safety, le-gitimacy, and reasonableness. (e scene evaluation pa-rameters in both horizontal and vertical directions are

Scenario assessments

Safety assessmentLane keepingLane change to leftLane change to rightTurn left...

Attribute of obstacleAttribute of traffic signAttribute of traffic ruleAttribute of road structure...

Legitimacy assessment

Reasonable assessment

Rule-based reasoning

Driving intentionInput condition attribute

Figure 2: Driving intention prediction process.

Referencepath

Start point

Projectionpoint

S

LV

(a)

Projection point (xp , yp)

SReference path

Start point (xs , ys)

Vehicle point (xv , yv)

(b)

Figure 3: Frenet coordinate of a vehicle: (a) driving scenario; (b) description of Frenet coordinate.


generated, and the behavioral prediction rules are matchedwith the scene evaluation parameters by using the prologonline reasoning system.(e driving intentions are inferred,and finally, the third-order Bezier curve is used to fit thedriving intention to the PRB.

2.2.1. Semantic Modeling of Driving Scene. In a drivingscenario, there are various road element entities, such astraffic participants, road networks, and road traffic facilitiesin urban driving scenarios. (e environment perceptionsystem can provide only the spatial location of each entity,but it cannot describe the correlation between entities, andmake full use of prior information, such as traffic rules anddriving experience, which is crucial for improving theprediction model adaptability to the driving scene. Ontol-ogy, as a form of knowledge expression, is used to model theconcepts of specific domains and relationships between

concepts, which can be used to model driving scenarioseffectively [25–27].

(e conceptual ontology model is divided into twomodule types: entities and attributes. (is study takes thetarget vehicle as a perspective and summarizes five entitytypes on the basis of [28]:

(1) Target vehicle(e target vehicle entity describes the vehicle to bepredicted.

(2) Behavior(e behavior entity is a collection of driving ma-neuvers of a vehicle. (ree behavior types aredesigned: LongtiBehavior, LatiBehavior, andAdvancedBehavior. (e LongtiBehavior representsbasic vertical driving behavior and includes fourbehaviors: accelerate, decelerate, keep, and stop. (eLatiBehavior represents basic horizontal driving

LSTM network

Vehicle statetransformation

Vehicle V1(1) (1)

OT–M+1,...,OT

(n)Ot = {s,l,v,a,k}for t = T – M + 1,...,Tfor n = 1,..., N

Predictedtrajectory

Background knowledgedigital map

traffic regulationsdriving experience

Real-time perceptionStatic/dynamic obs

GPS/IMU

Offline phase Online phaseOntology-based

scenario description(Abox)

Situationassessment

Rule-basedreasoning V1

PRBPrediction

reference baseline(PRB) produce

Predictionreferencebaseline

PredictionreferencebaselinePredictedtrajectory

Ontology modeling(Tbox)

Figure 4: Overview of the proposed trajectory prediction model.

Background knowledgedigital map

traffic regulationsdriving experience

Real-time perceptionStatic/dynamic obs

GPS/IMU

Offline phase

Online phase

Ontology-basedScenario description Situationassessment

Rule-basedreasoning

Predictionreference

baseline produce

Ontologgy modeling Rule generation(Rules)

Figure 5: (e architecture of the proposed prediction reference baseline determination method.


behavior and includes three behaviors: Change-ToLeft, ChangeToRight, and KeepLane. (eAdvancedBehavior represents advanced drivingbehavior and includes two behaviors: Overtake andMerge.

(3) Obstacle(e obstacle entity represents a collection of obstacleentities encountered by a vehicle during driving.(iswork divides obstacles according to the behaviorcharacteristics of obstacle entities in driving sce-narios into two categories: StaticObstacle andDynamicObstacle.

(4) Road network(e road network entity represents the topologicalconnection of roads by intersecting points and lines.RoadType includes different road types. RoadPartdescribes the components of the road network and isdivided into AreaEntities and PointEntities. Area-Entities refers to road entities that can be abstractedinto lines and areas, such as lane, side walk, junction,and segment, while PointEntities refers to road en-tities that can be abstracted into points, such as roadsigns, traffic signs, and traffic lights.

(5) Driving scenario(e driving scene entity refers to a collection of roadentity elements encountered when a vehicle travels indifferent road areas. In this work, driving scenariosare divided into three categories: InSpecialAr-eascenario (special area driving scenario),OnRoadscenario (road driving scenario), andNearSpecialAreascenario (near special region driv-ing scenario). InSpecialAreascenario category can befurther divided into IntersectionScenario (intersec-tion scene), TunnelScenario (tunnel scene), Bridge-Scenario (elevated scene), and UturnScenario (U-turn scene).

(e object attribute is used to describe the relationshipbetween concept classes. (is attribute restricts the de-scribed relationship regarding the domain and range. (edata attribute restricts the described relationship through thedefinition and value domains. (e definition domain is aclass type.

(e described ontology modeling process of drivingscenario is equivalent to filling the background knowledgeof the TBox that constitutes the ontology knowledge base,but the situational knowledge in the ABox is still lacking.According to the road elements of a real driving scenario,the driving scenario needs to be re-expressed using theconceptual model of the TBox, which is an instantiation ofthe ontology model. A real-traffic scenario is displayed inFigure 6(a); a concrete driving scene that includes instancesof defined classes is presented in Figure 6(b), and its se-mantic description is presented in Figure 6(c). (e in-stances of RoadNetwork are added to the ABox as priorknowledge, and instances of the obstacle are asserted in realtime.

2.2.2. Situation Assessment. After obtaining a semanticdescription of a driving scene, it is necessary to determineand evaluate condition attributes that affect the drivingintention in the driving scene, so as to estimate the drivingintention of a target vehicle. In order to solve the problem ofcombinatorial explosion due to numerous condition attri-butes [28], a deterministic scenario assessment method isadopted to classify and analyze the key attributes that affectdriving intentions from the perspectives of safety, legiti-macy, and reasonableness.

Deterministic scenario assessment methods use thethreat assessment indicators: TTC (time to collision),THW (time headway), TTB (time to brake), DST (de-celeration to safety time), and MSM (minimal safetymargin) in rule-based systems, and the probability ofcollision is estimated as a binary value. For instance,Glaser et al. [29] used the TTC and TIV (time inter-vehicles) indicators to evaluate the possibility of colli-sion. Noh et al. [30] proposed a distributed reasoningmethod by dividing the current and adjacent lanes intothe front and rear areas, and the TTB and MSM indi-cators were used to evaluate the possibility of collision inthe front area, while the TTC and MSM indicators wereused to evaluate the collision of rear area collisionpossibility.

(e proposed deterministic scenario assessment methodconsists of two parts. First, the driving scenario is deter-mined by querying the knowledge base with the currentvehicle position.(en, a reasoning structure of an obstacle isconstructed in eight regions of interest to make safety as-sessment, and a binary result (safe or dangerous) is calcu-lated for each region using critical indicators TTC and TIV.Finally, legitimacy and reasonableness assessments are madeto predict the maneuver of the target vehicle.

(1) Safety Assessment. Safety primarily refers to whether thesurrounding obstacles pose a threat to a vehicle, especially inthe area ahead, but it also refers to whether the left or rightlane can provide a safe lane change.(is paper constructs theeight-direction obstacle inference model. For each area, theTTC and TIV indicators are used for safety assessment. (eTTC indicator is defined as a time when two vehiclescontinue to collide on the same trajectory at the currentspeed, and it is defined by

TTC �Di

V − Vi, (6)

where Di denotes the relative distance between the followingvehicle and followed vehicle, V denotes the speed of thefollowing vehicle, and Vi is the speed of followed vehicle.

(e threshold value TTCth is used to judge whether avehicle is dangerous in high-speed scenarios. (e risk as-sessment formula is as follows:

RTTC(t) �0, t≥TTCth,

1, t

the current scene is considered to be safe; otherwise, it isconsidered to be dangerous.

(e TIV indicator is used to detect low-speed differencescenarios. When the speeds of two vehicles are similar invalue, the TIV indicator is used to judge the degree ofdanger, and it is calculated by

TIV �DiV

. (8)

(reshold TIVth is used to distinguish between safe anddangerous scenes in low workshop distance scenes. (e TIVrisk assessment formula is as follows:

RTIV(t) �0, t≥TIVth,

1, t

rRegion(t) �1, RRegionTTC (t) + R

RegionTIV (t)≥ 1,

0, otherwise,

⎧⎨

⎩ (10)

where Region represents one of the eight regions, as shownin Figure 7; when rRegion(t) has a value of zero, the area isconsidered to be safe, and when rRegion(t) has a value of one,the area is considered to be dangerous.

(e degree of danger in the area ahead is calculated by

rF(t) �1, RFTTC(t) + R

FTIV(t)≥ 1,

0, otherwise,

⎧⎨

⎩ (11)

and when the calculated values of the TTC and TIV indi-cators in the current area are greater than the predefinedthreshold, the following result message is obtained: safe-ToGo (targetVehicle, keep); when the calculated values ofTTC and TIV indicators in the current area are both lessthan the predefined deceleration threshold, but greater thanthe corresponding parking threshold, the result message issafeToGo (targetVehicle, dec); in this case, a lane change canbe performed to improve driving efficiency; otherwise, theresult message is safeToGo (targetVehicle, stop).

(e safety assessment of adjacent lanes is conductedusing the same assessment formula as that of the area ahead.If the left lane is taken as an example, then the degree ofdanger is expressed as

rL(t) �1, has left vehicle at time t,

0, otherwise, (12)

and when there are vehicles in the left area, the left lane canbe considered to be dangerous; otherwise, the safety of theleft front and left rear areas are, respectively, evaluated by

rFL(t) �1, RFLTTC(t) + R

FLTIV(t)≥ 1,

0, otherwise,

⎧⎨

⎩

rBL(t) �1, RBLTTC(t) + R

BLTIV(t)≥ 1,

0, otherwise.

⎧⎨

⎩

(13)

(erefore, the safety assessment of the left lane is asfollows:

RiskL(t) �1, rL(t) � 1 or rFL(t) + rBL(t)≥ 1,

0, otherwise, (14)

If the left lane is safe, safe To Left(target Vehicle, true)will be generated; otherwise, the scene evaluation parameterswill be instantiated as safe To Left(target Vehicle, false).

(2) Legitimacy Assessment. (e legality assessment includesthree assumptions. First, when a vehicle is driving on theroad, it cannot exceed the maximum speed limit of the road;second, when the vehicle is driving to the preintersection, itis necessary to pay attention to the change in traffic lightsand obey the traffic rules; third, when a vehicle is about tochange the lane, the adjacent lane should allow lane changes.

(3) Reasonableness Assessment. Reasonableness assessmentgenerally refers to whether lane changing and other driving

behaviors affect the current goal of a target vehicle. Basedon the current lane of the target vehicle, the specific roadsection or lane to be driven can be known. For instance, onthe one hand, if the next area to be driven by the targetvehicle is an intersection, and the distance between thetarget vehicle and the stop line is less than δ, then lanechange is not recommended. On the other hand, if thedistance between the target vehicle and the stop line isgreater than δ, lane change can be performed. Reason-ableness assessment introduces a situation parameter set(reasonable To Left, reasonable To Right), which is definedas data properties in the ontology model.

2.2.3. Rule-Based Reasoning. (e driving intention is de-termined based on traffic rules and driving experience,where the traffic rules are mainly used to limit the drivingbehavior while the driving experience is utilized to sum-marize the understanding and cognition of human drivers indifferent scenes and obtain some rules that are not specifictraffic rules but are conducive to the reasonable driving.According to the different driving scenarios defined in thedriving knowledge base, the rules stored in the drivingknowledge base are divided into several categories. Differentscenarios have different key road entities. For instance,unlike OnRoad Scenario, in Near Intersection Scenario,traffic lights are considered. Also, the classification of trafficrules can reduce the rule search space and reasoning time.

In order to save computing resources and reduce rea-soning time, SWI-Prolog language is used to write ruleknowledge, which is represented as a set of driving scene-driving behavior mapping pairs, where driving behavior isdescribed as a rule head, and the driving scene is described asa rule body. On the basis of [28], this paper adds the sceneassessment as an intermediate link of mapping driving sceneto the driving behavior and reorganizes 57 rules. Some of theprediction rules are presented in Table 1.

(e online reasoning process can be described as follows.First, the real-time facts related to the scene are used asinput, and each rule statement is matched. If all the facts ofthe corresponding rule are matched, the matched predictionresult will be obtained, and the next rule statement will bematched until each rule is matched. When all matchedresults are obtained, the final result denotes the predicteddriving intention.

2.2.4. Prediction Reference Baseline Fitting. After obtainingthe driving intention of the target vehicle, the driving in-tention is converted into the prediction reference baselineusing the cubic Bezier curves. As shown in Figure 8, first, atarget lane is selected based on the driving intention androad network, where p0 represents the current position ofthe target vehicle, and p3 is selected from the centerline ofthe target lane with distance Ld from p0, and Ld is obtainedbased on the driving experience. (e prediction referencebaseline is divided into three parts by p0 and p3: the pre-dicted extension, the historical extension, and the intentionsegment. In addition, Lp and Lh are determined by the inputand output steps of the LSTM neural network.


Table 1: Prediction rules used in this study.ID SWRL rules

Rule #1

TargetVehicle (target), currentRoadState (target, “ApprJunction”), isOnSegment (target, Seg), connectToJunction (Seg, Junc),intersection (Junc), hasTrafficLight (Junc, TL), (hasLightColor (TL, “red”), hasLightColor (TL, “yellow”)), connectToStopLine

(Seg, SL), distToStopLine (SL, DL), DL< 10.legalToGo (target, SS)

Rule #2

TargetVehicle (target), currentRoadState (target, “ApprJunction”), isOnSegment (target, Seg), connectToJunction (Seg, Junc),intersection (Junc), hasTrafficLight (Junc, TL), hasLightColor (TL, “green”), connectToStopLine (Seg, SL), distToStopLine (SL,

DL), DL �< 20.LegalToGo (target, acc)

Rule #3TargetVehicle (target), currentVelocity (target, V), hasFrontObstacle (target, FO), distToObstacle (FO, DF), (DF/V)< 3,

(DF/V)≥ 2.SafeToGo (target, dec)

Lp

P3

P2

P1

P0

Ld

Lh

Extention

Extention

Intention

(a)

LpP3

P2

P1

P0

Ld

Lh

Extention

Extention

Extention

Extention

Intention

(b)

P2

P3

P1d

d

ωd′

X″

Y″

P0(0, 0) (Lx, LY)

(c)

Figure 8: Schematic diagram of prediction reference baseline fitting: (a) lane-changing scene; (b) intersection turning scene; (c) the cubicBezier curves.

Le� front Front

Target vehicleLe�

Le� back Right back

Right

Right front

Back

Figure 7: Example individuals in traffic scenario.


As shown in Figure 8(c), the cubic Bezier curve con-structed by four control points is used to generate the in-tention segment, which is expressed as

C(t) � B0,3(t)P0 + B1,3(t)P1 + B2,3(t)P2 + B3,3(t)P3,

(15)

where Bi,3 is the Bernstein polynomial and it is given by

Bi,3(t) �

3

i

⎛⎝ ⎞⎠t1 − t

t1 − t0

3−it − t0

t1 − t0

i

, t ∈ 0, 1, 2, 3.

(16)

(e coordinate system (X″, Y″) is built with the originP0 at the vehicle center. (e x-axis direction is the vehicle’sinitial heading, the terminal state will be the end point P3,and P1 and P2 are obtained bymoving forward for distance dalong the vehicle’s initial heading direction from the startpoint and backward for distance d along the terminalheading from the end point P3, respectively. (e position ofthe control points in the above coordinate system isexpressed as

P0 �0

0 ,

P1 �d

0 ,

P2 �Lx − d cosω

Ly − d sinω⎡⎣ ⎤⎦,

P3 �Lx

Ly

⎡⎣ ⎤⎦,

(17)

where Lx and Ly are lateral and longitudinal offsets of theterminal state P3 to P0, respectively; ω is the angle betweenthe terminal heading and the direction of the x-axis. (eterminal heading is defined as the tangential direction of theclosest point on the PRB to P3.

Equations (15) and (16) can be rewritten by applying(17), so the Bezier curve can be represented as

x(t) � 3d + 3d cosω − 2Lx( t3

− 3 2d + d cosω − Lx( t2

+ 3dt,

(18)

y(t) � 3d sinω − 2Ly t3

− 3 d sinω − Ly t2. (19)

Besides, the curvature of the generated path can bederived by applying (19) and (20), which leads to

κ(t) �x′(t)y″(t) − y′(t)x″(t)

x′(t)2 + y′(t)2 3/2 . (20)

(e maximum of the curvature should satisfy the con-dition given by equation (21) to meet the vehicle’s non-holonomic constraint:

κ tm( ≤tan φmax(

L. (21)

In equation (21), L denotes the vehicle wheelbase andφmax denotes the maximum steering angle of the vehicle.

(e maximum of the curvature κ(tm) is a function of d.(e suitable value of d that satisfies the vehicle’s non-holonomic constraint can be found by brutal searching fromd′/6 to d′/2, where d′ denotes the distance between P0 andP3. (e processing time can be reduced by building a look-up table that matches a given set with the correspondingmaximum curvature of the Bezier curve.

2.3. LSTM Network Driven by Knowledge. Since differentdrivers have different driving styles, in order to accuratelypredict the future trajectory of a vehicle, in this work, anLSTM neural network is employed to learn the continuousfeatures of the historical trajectory. (e LSTM is an RNNtype that can effectively overcome the problem of gradientdisappearance [31]. (e LSTM is composed of a unitmemory that stores the previous input sequence informationand a gating mechanism that controls the information flowbetween input, output, and unit memory. (ere are threegates in the core design of the LSTM network, namely, theinput gate, the forget gate, and the output gate. (e specificnetwork structure is shown in Figure 9. (e forget gate isused to control how much information is retained in ct. (einput gate determines how much information of xt remainsin ct, and finally, the output gate determines how muchinformation in the output ot is output to ht by the controlunit ct. (e work of the LSTM is described by the followingrecursive equations:

ft � σ Wf · ht−1, xt + bf ,

it � σ Wi · ht−1, xt + bi( ct � tan h Wc · ht−1, xt + bc( ,

ct � ft ⊙ ct−1 + it ⊙ ctot � σ Wo · ht−1, xt + bo( ,

ht � ot ⊙ tan h ct( ,

(22)

where xt denotes the input vector,σ(x) denotes the acti-vation function, W denotes the linear transformation ma-trix, b denotes the offset vector, it, ft, and ct are gate vectors,ct represents the amount of cell memory, and lastly, htdenotes the output.

In this work, the network presented in Figure 10 is usedas a reference structure. (is network has two layers con-sisting of 256 LSTM cells, followed by one time-distributedlayer consisting of 128 neurons, and the final dense outputlayer containing as many cells as the number of outputs. (enetwork input is a tensor of track histories of a vehicle. (enetwork output consists of the future coordinates and ve-locity of the vehicle. Since the prior knowledge about thedriving scene is expressed by prediction reference baseline,the network can learn the posterior knowledge about the


driving scene only from the relative relationship between thehistorical trajectory and the prediction reference baseline.Compared with the existing prediction models based on theLSTM network, the proposed prediction model reduces thenetwork training difficulty and decreases demand for thecomputing performance of the vehicle platform.

3. Results and Discussion

3.1. Data Preparation and Model Training

3.1.1. ;e Training Dataset. (e next generation simulation(NGSIM) dataset in I-80 and US101 sections is used formodel training and testing [32], and this dataset is derivedfrom the US Federal Highway Administration, which iscurrently the largest public natural driving public datasource, and thus has been widely used in the literature[33, 34]. (e layouts and top-down views of the US101 andI-80 sections are shown in Figure 11. Each data frame in-cludes many vehicle’s parameters, including the position,velocity, yaw rate, size, and others. (e sampling frequencyof the dataset is 10Hz; therefore, in this work, Δt is set to0.1 s.

3.1.2. Data Preparation. (e vehicle positioning data in theNGSIM dataset are obtained by video analysis, so therecorded trajectory contains a lot of noise [26]. (erefore,

the vehicle kinematics model and the road geometric areused to filter the original data, which is expresses as

0< κi < κmax,

θmin < θ< θmax,

θri > θrate,

θri �θi+1 − θit2 − t1

.

(23)

(e vehicle position is transformed to the Frenet co-ordinates based on the centerline. As shown in Figure 12(b),the centerline of each lane is extracted and fitted using theshapefile, and the centerline that the target vehicle wasinitially driven is selected as a reference baseline. For eachoriginal coordinate point (xv, yv), the correspondingmapping point (xp, yp) on the reference baseline is deter-mined, and the Frenet coordinates (sv, lv) are obtained by

lv �

��

xp − xv 2

+ yp − yv 2

,

sv � S xp, yp ,(24)

where lv denotes the Euclidean distance between (xv, yv)and (xp, yp), and S(xp, yp) denotes the length from themapping point (xp, yp) to the starting point of the referencetrajectory.

X X

XX X

XXX

tanh

ht–1 ht+1

ht–1

xt–1xt–1

ct–1

ht

ht

xt

wf wi wc wo

ft it ot

ct

tanh tanh

tanh tanh

σ σ σ σ tanhσ σ σσσ

+ + X +

C~t

X

XX

tanh

tanh σσσ

+

XX

tanh

tanhσ σ σ

X +

AA

Figure 9: (e internal structure of an LSTM cell.

(1)

Vehicle statetransformation

Vehicle V1(1)

OT–M+1,...,OT

(n)Ot = {s,l,v,a,k}for t = T – M + 1,...,Tfor n = 1,..., N

Predictedvehicle stateVehicle V1

(1)(1)PT+1,...,PT+K

(n)Pt = {s,l,v}for t = T + 1,...,T + Kfor n = 1,..., N

Repeat

Relu Relu ReluLSTM[256]

LSTM[256]

Dense[128]

Output[3]

Figure 10: Network structure used as a reference design.


In addition, four other features, curvature κ, velocity V,acceleration a, and heading θ, are also selected so as tocompose the observation vector o with the Frenet coordi-nates (s, l).

3.1.3. Training Details. (ere were 8311 filtered trajectories;80% of the trajectories were selected as the training set, 10%as the test set, and the remaining 10% was used as theverification set to observe if the model is overfitted.

(e network was trained using minibatches with a size of64. Due to the limitation on a sensormeasurement range andnoise in practical application scenarios, it was difficult totrack dynamic vehicles stably for a long time, so the networkwas trained using windows that consisted of 30 inputs,representing a total of 3 s past observations. (e Adam

optimizer was used; the learning rate was 0.0005, and ReLUactivation with α� 0.1. (e loss function adopted the MSE(mean square error) between the predicted sequence and theground truth sequence; the code used to generate the modelwas written in Keras, and the training was performed onNVIDIA GTX 2080 s GPU using the TensorFlow backend.(e model training contained 16 epochs, and the averagetraining time for each epoch of the full training set is around2300 seconds.

3.2. Testing Results and Discussion

3.2.1. ;e Impact of the Prediction Reference Baseline. Toinvestigate the impact of considering prediction referencebaseline on the accuracy of the proposed method, we test the

Venture blvd.onramp

Cahuenga blvd.offramp

640 meters (2100 feet)

Vehicle trajectoryStudy area

10 UCP

Camera1

Camera2

Camera4

Camera3

Camera5

Camera6

Camera7

Camera8

(a)

EB/NB I-80

Powellstreet

onramp420°

140°

2

34

5

6

7

≈170°

Lane 6width > 16’

1280°

Study area

Ashby aveonramp

13456

2

1650°

(b)

Figure 11: Layouts and top-down views of the sites used for the collection of the NGSIM: (a) US101; (b) I-80.

(a)

+

+

+

+

+

+

+

+

+

+

+

+

(b)

Figure 12: Extraction of the lane’s centerline: (a) extraction results obtained by Google Earth; (b) shapefile of the US101 highways.


RMSE performance of the proposed modelwith threemodifications using the NGSIM dataset.

In one experiment, the system is trained and tested withthe absolute coordinate, in the second, the centerline that thetarget vehicle was initially driven is selected as a PRB, andfinally, in the third experiment, the PRB of the target vehicleis determined by the method in Section 2, while the otherattributes of the three models are unchanged. Figure 13shows the accuracy of the trajectory prediction for differenttime horizons, and the RMSE value of the model is decreasedby adding the PRB for both lateral and longitudinaltrajectories.

3.2.2. Comparative Study. To evaluate the proposed ap-proach, we purse a direct comparison with state-of-the-artvehicle trajectory prediction using the same dataset (i.e.,NGSIM). (e results show that the proposed method out-performs the state-of-the-art model and decreases theoverall RMSE value of the system by 10 percent on average.Table 2 summarizes the RMSE values comparing the pro-posed methods with the baseline trajectory predictionmodels in the literature [20, 22–24, 35].

(e comparison results show that the proposedknowledge-driven LSTM network has better performance inRMSE for NGSIM. Note that as compared to the baseline[24], the prediction accuracy gets a slight improvement, butthe proposed method enhances the real-time performanceand much reduces the computational complexity due to thereduction of the feature space dimension.

3.3. Simulation Results and Discussion

3.3.1. Simulation Experimental Platform. (e simulationexperiments were based on the JAC’s automatic drivinghardware-in-the-loop test platform. (e experimentalsimulation platform is presented in Figure 14, where it canbe seen that the experimental platform included the realvehicle braking system, steering system, sensor system, andnetwork communication system, which had dSPACE(Matlab/Simulink) as a core. (e controller rapid proto-typing platform was built, virtual reality interfaces andenvironment-aware sensor modules were provided using thePreScan software, and the CarSim software was used to runthe vehicle dynamic model and provide a platform thatcould be quickly verified for automatic driving algorithmtesting.

3.3.2. Simulation Results. (e simulation scenario shown inFigure 15 was established according to the real urban trafficscenario. Two typical traffic scenarios were selected to verifythe adjustment effect of the PRB on the predicted trajectory.Figure 16(a) shows the driving scene on the road, andFigure 16(b) shows the driving scene at the intersection. Inthe first scene, three PRB intent segments were fitted to lanekeeping (LK), lane change left (LCL), and lane change right(LCR), as shown by the blue curve in Figure 16(a). (ehistorical observation vector of the target vehicle denoted

the network input, and it was unchanged; the output net-work vector was converted according to three PRB to obtainthree predicted trajectories, as shown by the green curve inFigure 16(c). In Figure 16(b), the intersection driving sce-nario is presented, where two PRB intent segments of gostraight (GS) and turn right (TR) are fitted, respectively; theconverted network output results are shown by the greencurve in Figure 16(d). Since the network learns the relativerelationship between the historical and PRB, even at thesame network input, the predicted trajectory will be affectedby the prediction reference baseline. (e experimental re-sults prove that the priori knowledge about the driving scenecan be used to adjust the predicted trajectory effectivelybased on the prediction reference baseline.

3.4. Real-World Urban Traffic Scenarios

3.4.1. Experimental Platform Construction. In order to verifyif the simulation results obtained in coincide well with thereal-world scenario results, an instrumented vehicle wasused to collect data, as shown in Figure 17(a). (e vehicleloading sensors included an IBEO four-layer laser scaninstrument, a Velodyne HDL-64E lidar, two high-resolutioncameras, and a differential GPS/INS (SPAN-CPT) system.(e sensor configuration of the vehicle and its sensing rangeare shown in Figure 17(b). (e differential GPS moduleprovided the information on the position, speed, andheading of the ego vehicle. Based on our previous work [36],moving obstacles were detected and tracked by a four-layerlaser scanner, which was located at the front of the vehicle.According to the space-time relationship between themoving obstacles, such as pedestrians and vehicles, andexperimental vehicles, the position, speed, size, and type ofthe sports vehicles can be measured. We conducted a realvehicle experiment in Hefei, Anhui Province, China.(e testroad is shown in Figure 18(a). (e prediction model wasexemplarily implemented onNVIDIA Xavier platform usingthe C++ programming language.

Before conducting the actual vehicle experiment, thehigh-resolution maps were collected to establish based onour experimental vehicle. (ere were more than 1,140 roadentities on the map, including the stop signs, lane markings,and lane lines, covering approximately 8 km of the roadways(as Figure 18 shows).

3.4.2. Field Test and Discussion. (e experimental drivingroute was located on a typical urban roadway, with a totallength of about 4.7 km. It includes multiple intersections,Y-shaped intersections, T-shaped intersections, and othercommon urban road scenarios. Due to the long experi-mental route and a large number of scenes encountered, itwas inconvenient to conduct the prediction process for eachscene. (erefore, two typical scenes were selected for de-tailed trajectory prediction process analysis.

In Scenario 1, vehicle054 was on the road. (e inputconditions of the scene evaluation module are shown inFigure 19(b). In front of the target vehicle, there was a largetruck denoted as vehicle055 that was moving with a speed


of 5 km/h. At that time, the speed of vehicle054 was 24 km/h. (e calculated value of the TIV was less than the de-celeration threshold.(erefore, it was judged that the targetvehicle would have an intension to change lane. Vehi-che054 was driving on lane00055, which was a straight lane.(rough an associated search in the conceptual ontology

model of the driving scene, it was learned that the right lanewas also a straight lane, and the lane line was a white dottedline; the prolog rule of legality is expressed as follows:

legalToRight (target, true): targetVehicle (target),isOnLane (target, Lane), hasRightLine (Lane, Line),and hasLineType (Line, “dotted_white”).

Table 2: RMSE comparison of the proposed method with the baseline models and state-of-the-art model.Prediction horizon (s) M-LSTM [20] CS-LSTM [22] NLS-LSTM [35] ST-LSTM [23] DLM [24] Proposed method1 0.58 0.61 0.56 0.58 0.41 0.412 1.26 1.27 1.22 1.21 0.95 0.893 2.12 2.09 2.02 1.97 1.72 1.644 3.24 3.10 3.03 2.85 2.64 2.475 4.66 4.37 4.30 3.89 3.87 3.68

220mm

4445mm

3000mm

(a) (b)

Figure 14: Experimental simulation platform: (a) design of the simulation platform; (b) photo of the simulation platform.

Absolute coordinatePRB based on centerlinePRB based on knowledge reasoning

0.5 1 1.5 2 2.5 3 3.5 4 4.5 50Prediction horizon (s)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

RMSE

(m)

(a)

Absolute coordinatePRB based on centerlinePRB based on knowledge reasoning

0.5 1 1.5 2 2.5 3 3.5 4 4.5 50Prediction horizon (s)

0

1

2

3

4

5

6

RMSE

(m)

(b)

Figure 13: Effectiveness of considering prediction reference baseline in the vehicle trajectory prediction: (a) lateral position error; (b)longitudinal position error.


Extention

Extention

Intention

P1

P2

P3

P0

(a)

Extention

Extention

Extention

Intention

P1

P2

P3

P2 P3

P0

(b)

Predictedtrajactory

Historicaltrajactory

(c)

Predictedtrajactory

Historicaltrajactory

(d)

Figure 16: Simulation results: (a) on the road scenario; (b) intersection scenario; (c) simulation results of the on the road scenario; (d)simulation results of the intersection scenario.

Figure 15: Simulation scenario layout.


(e next section of the target vehicle to travel was theintersection, and the distance to the intersection was greaterthan 30m, so the effectiveness of changing lanes was sat-isfied; thus, the lane change did not affect the current targetof the target vehicle; the prolog rule of reasonableness is asfollows:

reasonableToRight (target, true): targetVehicle (target),currentRoadState (target. “ApprJunction”), isOnSeg-ment (target, Seg), connectToJunction (Seg, Junc),intersection (Junc), connectToStopLine (Seg, SL), anddistToStopLine (SL, DL), DL≥ 30.

(e autonomous vehicle was driving in the right backregion of the target vehicle, and the TTC and TIV valueswere both less than the corresponding acceleration thresholdbut greater than the corresponding parking threshold; theprolog rule of safety is as follows:

safeToRight (target, true): targetVehicle (target), has-RightObstacle (target, null), hasRightFrontObstacle(target, null), and hasRightBackObstacle (target,egovehicle).

(e final prolog rule is as follows:

canChangeToRight (target, true): safeToRight (target,true), reasonableToRight (target, true), and legalTo-Right (target, true).

After obtaining the driving intention of the target vehicle,the prediction reference baseline (the blue curve inFigure 19(a)) was fitted, and the historical trajectory wastransformed into Frenet coordinates based on the predictionreference baseline and then fed to the LSTMnetwork input; thepredicted trajectory is shown by the green dotted line inFigure 19(a). (e trajectory prediction results of vehicle054could effectively reduce the reaction time of autonomous ve-hicle while avoiding collisions caused by vehicle054 cutting in.

In Scenario 2, vehicle121 was in the preintersectionscenario. (e input conditions of the scenario evaluationmodule are shown in Figure 20(b). (e current speed ofvehicle121 was 24 km/h, and the lane it traveled waslane000103, which was a right-turn lane. In this scenario, themain factor affecting the target vehicle’s driving intentionwas the traffic light. (e prolog rule of legality is as follows:

legalToTurnRight (ego, acc): targetVehicle (target),currentRoadState (target, “ApprJunction”), isOnSeg-ment (target, Seg), connectToJunction (Seg, Junc),intersection (Junc), hasTrafficLight (Junc, TR),

(a)

Lane line

Side walk

Boundary

Junction

Stop line

(b)

Figure 18: Testing roadway layout: (a) the high-resolution map of the experimental roadway; (b) an enlarged view of the map data.

(a)

2

2

1

3

3

44

SICK

HDL-64E

IBEO

VLP-16

(b)

Figure 17: Experimental platform: (a) the pioneer IV autonomous vehicle; (b) perception range of the autonomous vehicle.


hasLightColor (TR, “green”), connectToStopLine (Seg,SL), and distToStopLine (SL, DL), DL�< 0

(e target vehicle was on the right turn, and the prologrule of reasonableness is as follows:

reasonableToLeft (ego, true): egoVehicle (ego), cur-rentRoadState (ego, “ApprJunction”), isOnSegment(ego, Seg), connectToJunction (Seg, Junc), intersection(Junc), connectToStopLine (Seg, SL), and dis-tToStopLine (SL, DL), DL≥ 30.

(e final rule of safety is as follows:

safeToTurnRight (target, true): targetVehicle (target),hasFrontObstacle (target, null), hasRightFrontOb-stacle (target, null), and hasRightBackObstacle (target,null).

(e final rule is expressed as follows:

canTurnRight (target, true): safeToTurnRight (target,true), reasonableToTurnRight (target, true), andlegalToTurnRight (target, true).

(e blue curve in Figure 20(a) represents the predictionreference baseline fitted based on the turn right drivingintention. (e predicted trajectory is shown by the greendotted line in Figure 20(a).

(e experimental results of Scenarios 1 and 2 verify thatthe proposed LSTM network can effectively combine theprior and posterior knowledge in the driving scene, and thelane change behavior can be predicted before the variation ofvehicle kinemics. (e proposed LSTM network can be it-eratively adapted to a driving scenario without manualannotation during the network training, which significantlyreduces the training complexity and solves the sparse dataproblem. Due to the unique combination of knowledgereasoning, the proposed prediction model can make a moreaccurate and reasonable estimation of future trajectories of

Junction014

Stopline0011

Lane markerApprjunction113

vehicle055

Lane line

Predicted trajectory

Reference lane

Static obstacle

Ego vehicle

vehicle053

vehicle054

Road elementsRoute ID: 33Seg ID: 0053Lane ID: 55Dis to Jun: 50mPath num: 4

PaP2

P1

P0

(a)

Junction014

connectToJunction

hasStopline

isPartOf

isOnLane hasRightObstacle

isOnLane

Lane00054

Lane00055

Lane00056

hasLaneMarker

hasLaneMarker

hasLaneMarker

hasLe�Lane

hasRightLane

hasRightObstacle

hasFrontObstacle

vehicle055distanceTostopline (9)hasVelocity (5)hasObDirection (273)



gostraightLM00032

gostraightLM00031

turnle� LM00021 nearIntersection

scenario0001

EgoVehicledistancetoObstacle (2,2)hasVehicle (26)hasObDirection (270)

Stop line0011

(b)

(c)

Figure 19: (e results of Scenario 1: (a) trajectory prediction interface; (b) scenario reasoning; (c) experimental scenarios.


surrounding vehicles than the existing models in differentenvironments.

4. Conclusions

(is paper combines a maneuver-based and learning-basedtrajectory prediction models and proposes an improved tra-jectory prediction model based on the LSTM neural networkdriven by driving knowledge. In order to achieve better use of aprior driving knowledge in driving scenarios and solve theproblem of the combinatorial explosion caused by a largenumber of conditional attributes, the multisource and het-erogeneous information of the driving scenario is modeledbased on ontology, and a driving knowledge base, including thedriving experience and traffic rules, is constructed. (en, theconditional attributes that affect driving intentions are clas-sified and analyzed from the perspectives of safety, legitimacy,

and reasonableness, and situation parameters in the horizontaland vertical directions are generated by the deterministic sceneevaluation method. Finally, using the obtained situation pa-rameters and the driving knowledge base, the driving intentionis inferred based on the prolog online reasoning system. Inorder to make the prediction results effectively combine theposterior knowledge and solve the problem of insufficientadaptability of the existing learning-based prediction models,this paper converts the driving intention of the target vehicleinto a prediction reference baseline, and the Frenet coordinatesbased on prediction reference baseline are used as a coordinateframe for the LSTM neural network. (e prior drivingknowledge existing in the driving scene can be used to adjustthe predicted trajectory in the form of the prediction referencebaseline without increasing the network complexity but en-suring efficient operation of the proposed model on an em-bedded platform.

Junction022

Lane Marker

Predicted trajectory

Reference lane

Static obstacle

Vehicle117

Vehicle118

vehicle119

vehicle120

vehicle121

Road elementsRoute ID: 41Seg ID: 0061Lane ID: 103Dis to Jun: 20mPath num: 3

(a)

Junction022 connectToJunction

Stopline0015

hasStopline

isPartOf

isOnLane

hasRerObstacle

isOnLane isOnLane

isOnLane

lane000102lane000101

Lane000103

hasLaneMarkerhasLaneMarker

hasRearObstacle

hasLe�RearObstacle

hasLe�FrontObstacle

hasLe�Obstacle

hasLe�Lane

hasLaneMarker

hasRightLane

hasRightObstacle

Vehicle119distanceTostopline (9)distanceToObstacle (3.9)hasVelocity (5)hasObDirection (273)

Vehicle118distanceTostopline (4.2)distanceToObstacle (2.3)hasVelocity (0)hasObDirection (270)

Vehicle120distanceTostopline (5.5)distanceToObstacle (4.9)hasVelocity (5.4)hasObDirection (273)

Vehicle117distanceTostopline (14)hasVelocity (4)hasObDirection (273)

Vehicle121distanceToObstacle (6.9)distanceTostopline (0)hasVelocity (13)hasObDirection (276)

gostraightLM00076

turnRightLM00056

turnle�LM00047

nearIntersectionScenario0001

EgoVehicledistanceToObstacle (2.2)hasVehicle (0)hasObDirection (270)

(b)

(c)

Figure 20: (e results for Scenario 2: (a) trajectory prediction interface; (b) scenario reasoning; (c) experimental scenarios.


(e proposed prediction model was verified by simu-lations and experiments. (e simulation results showed thatthe prediction reference baseline could effectively adjust theoutput of the LSTM neural network, making the predictedtrajectory meet the constraints of the prior knowledge in adriving scenario. (e real-world-experiment results showthat the proposed prediction model can significantly reducethe computing performance requirements while ensuringreal-time performance on the embedded platform. Also dueto the full combination of prior and posterior knowledge inthe driving scene, the target vehicle’s lane-changing behaviorcan be predicted on average 2.05 s (for LCL) or 2.71 s (forLCR) in advance, and the precision can be improved by12.5% for long-term predictions and is more robust, flexible,and adaptive in complex traffic scenarios.

Even though the proposed model has advantages intrajectory prediction, there are still some limitations, such asthat indexes of scenario assessment are not comprehensiveenough. In order to overcome this limitation, in future work,the evaluation index will be considered from the perspectiveof human-vehicle interaction and multivehicle interaction.Furthermore, data from a more complex scenario will becollected and used to verify the proposed prediction model.

Data Availability

(e data used to support the findings of this study areavailable from the corresponding author upon request.

Conflicts of Interest

(e authors declare no conflicts of interest.

Acknowledgments

(e authors gratefully acknowledge the help of the teammembers, whose contributions were essential for the de-velopment of the intelligent vehicle. (e authors would alsolike to thank the Institute of Applied Technology, HefeiInstitute of Physical Science, and the Academy of Sciences ofChina for supporting this study.(is work was supported bythe National Key Research and Development Program ofChina (nos.2016YFD0701401, 2017YFD0700303, and2018YFD0700602), Youth Innovation Promotion Associa-tion of the Chinese Academy of Sciences (grant no.2017488), Key Supported Project in the(irteenth Five-YearPlan of Hefei Institutes of Physical Science, ChineseAcademy of Sciences (grant no. KP 2019 16), EquipmentPre-Research Program (grant no. 301060603), Natural Sci-ence Foundation of Anhui Province (grant no.1508085MF133), and Technological Innovation Project forNew Energy and Intelligent Networked Automobile In-dustry of Anhui Province.

References

[1] S. Lefèvre, D. Vasquez, and C. Laugier, “A survey on motionprediction and risk assessment for intelligent vehicles””Robomech Journal, vol. 1, pp. 1–14, 2014.

[2] Y. Jeong and K. Yi, “Target vehicle motion prediction-basedmotion planning framework for autonomous driving inuncontrolled intersections,” IEEE Transactions on IntelligentTransportation Systems, vol. 99, pp. 1–10, 2019.

[3] N. Kaempchen, K. Weiss, M. Schaefer, and K. C. J. Dietmayer,“IMM object tracking for high dynamic driving maneuvers,”in Proceedings of the IEEE Intelligent Vehicles Symposium,pp. 825–830, Parma, Italy, June 2004.

[4] H. Veeraraghavan, N. Papanikolopoulos, and P. Schrater,“Deterministic sampling-based switching Kalman filtering forvehicle tracking,” in Proceedings of the IEEE IntelligentTransportation Systems Conference, pp. 1340–1345, Toronto,Ontario, September 2006.

[5] A. Broadhurst, S. Baker, and T. Kanade, “Monte Carlo roadsafety reasoning,” in Proceedings of the IEEE Proceedings.Intelligent Vehicles Symposium, IEEE, Las Vegas, NV, USA,June 2005.

[6] M. Bahram, C. Hubmann, A. Lawitzky, M. Aeberhard, andD. Wollherr, “A combined model- and learning-basedframework for interaction-aware maneuver prediction,” IEEETransactions on Intelligent Transportation Systems, vol. 17,no. 6, pp. 1538–1550, 2016.

[7] M. G. Ortiz, “Behavior prediction at multiple time-scales ininner-city scenarios,” in Proceedings of the IEEE IntelligentVehicles Symposium (IV), June 2011.

[8] S. Klingelschmitt, “Combining behavior and situation in-formation for reliably estimating multiple intentions,” inProceedings of the Intelligent Vehicles Symposium IEEE,Dearborn, MI, USA, June 2014.

[9] B. Morris, A. Doshi, and M. Trivedi, “Lane change intentprediction for driver assistance: on-road design and evalua-tion,” in Proceedings of the IEEE Intelligent Vehicles Sympo-sium (IV), June 2011.

[10] P. Kumar, M. Perrollaz, S. Lefevre, and C. Laugier, “Learning-based approach for online lane change intention prediction,”in Proceedings of the IEEE Intelligent Vehicles Symposium,pp. 797–802, Gold Coast, Australia, June 2013.

[11] A. Lawitzky, D. Althoff, C. F. Passenberg et al., “Interactivescene prediction for automotive applications,” in Proceedingsof the Intelligent Vehicles Symposium, June 2013.

[12] A. Zyner, “Long short term memory for driver intent pre-diction,” in Proceedings of the IEEE Intelligent VehiclesSymposium (IV), June 2017.

[13] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei,and S. Savarese, “Social lstm: human trajectory prediction incrowded spaces,” in Proceedings of the IEEE Conference OnComputer Vision And Pattern Recognition (CVPR), pp. 961–971, IEEE, Las Vegas, NV, USA, June 2016.

[14] D. J. Phillips, T. A. Wheeler, and M. J. Kochenderfer,“Generalizable intention prediction of human drivers at in-tersections,” in Proceedings of the IEEE Intelligent VehiclesSymposium (IV), pp. 1665–1670, Los Angeles, CA, USA, June2017.

[15] F. Altché and A. De La Fortelle, “An lstm network for highwaytrajectory prediction,” in Proceedings of the IEEE 20th In-ternational Conference on Intelligent Transportation Systems(ITSC), pp. 353–359, Yokohama, Japan, October 2017.

[16] C. Ding, W. Wang, X. Wang, and M. Baumann, “A neuralnetwork model for driver’s lane-changing trajectory predic-tion in urban traffic flow,” Mathematical Problems in Engi-neering, vol. 2013, Article ID 967358, 8 pages, 2013.

[17] J. Zheng, K. Suzuki, and M. Fujita, “Predicting driver’s lane-changing decisions using a neural network model,” Simula-tion Modelling Practice and ;eory, vol. 42, pp. 73–83, 2014.


[18] S. H. Park, B. Kim, C. M. Kang, C. C. Chung, and J. W. Choi,“Sequence-to-sequence prediction of vehicle trajectory vialstm encoder-decoder architecture,” in Proceedings of theIEEE Intelligent Vehicles Symposium (IV), pp. 1672–1678,Changshu, China, June 2018.

[19] L. Xin, “Intention-aware long horizon trajectory prediction ofsurrounding vehicles using dual lstm networks,” in Pro-ceedings of the IEEE International Conference on IntelligentTransportation Systems (ITSC), November 2018.

[20] N. Deo andM.M. Trivedi, “Multi-modal trajectory predictionof surrounding vehicles with maneuver based LSTMs,” inProceedings of the IEEE Intelligent Vehicles Symposium (IV),pp. 1179–1184, Changshu, China, June 2018.

[21] W. Ding and S. Shen, “Online vehicle trajectory predictionusing policy anticipation network and optimization-basedcontext reasoning,” in Proceedings of the International Con-ference on Robotics and Automation (ICRA), May 2019.

[22] N. Deo and M. M. Trivedi, “Convolutional social pooling forvehicle trajectory prediction,” 2018, https://arxiv.org/abs/1805.06771.

[23] S. Dai, L. Li, and Z. Li, “Modeling vehicle interactions viamodified LSTM models for trajectory prediction,” IEEE Ac-cess, vol. 7, pp. 38287–38296, 2019.

[24] M. Khakzar, A. A. Rakotonirainy, and S. G. Dehkordi, “A duallearning model for vehicle trajectory prediction,” IEEE Access,vol. 8, no. 99, pp. 21897–21908, 2020.

[25] B. Hummel, T. Werner, and I. Lulcheva, Scene Understandingof Urban Road Intersections with Description Logic: DagstuhlSeminar Proceedings, Schloss Dagstuhl-Leibniz-Zentrum frInformatik, Wadern, Germany, 2008.

[26] A. Armand, D. Filliat, and J. Ibañez-Guzman, “Ontology-based context awareness for driving assistance systems,” inProceedings of the IEEE Intelligent Vehicles SymposiumProceedings, June 2014.

[27] R. Kohlhaas, T. Bittner, T. Schamm, and J. M. Zollner, “Se-mantic state space for high-level maneuver planning instructured traffic scenes,” in Proceedings of the IntelligentTransportation Systems (ITSC), pp. 1060–1065, IEEE, Qing-dao, China, October 2014.

[28] G. Xinli, “A scenario-adaptive driving behavior predictionapproach to urban autonomous driving,” Applied Sciences,vol. 7, p. 426, 2017.

[29] S. Glaser, B. Vanholme, S. Mammar, D. Gruyer, andL. Nouveliere, “Maneuver-based trajectory planning forhighly autonomous vehicles on real road with traffic anddriver interaction,” IEEE Transactions on Intelligent Trans-portation Systems, vol. 11, no. 3, pp. 589–606, 2010.

[30] S. Noh, K. An, and W. Han, “Situation assessment and be-havior decision for vehicle/driver cooperative driving inhighway environments,” in Proceedings of the IEEE Interna-tional Conference On Automation Science And Engineering(CASE), January 2015.

[31] S. Hochreiter and J. Schmidhuber, “Long short-term mem-ory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.

[32] US Department of Transportation, “Next generation simu-lation,” Washington, DC, USA, 2008, http://www.ngsim.fhwa.dot.gov.

[33] A. Bender, J. R. Ward, S. Worrall, and E. M. Nebot, “Pre-dicting driver intent from models of naturalistic driving,” inProceedings of the IEEE 18th International Conference onIntelligent Transportation Systems, pp. 1609–1615, IEEE,September 2015.

[34] S. Yoon and D. Kum, “(e multilayer perceptron approach tolateral motion prediction of surrounding vehicles for

autonomous vehicles,” in Proceedings of the IEEE IntelligentVehicles Symposium (IV), June 2016.

[35] K. Messaoud, I. Yahiaoui, A. Verroust-Blondet, andF. Nashashibi, “Non-local social pooling for vehicle trajectoryprediction,” in Proceedings of the IEEE Intelligent VehiclesSymposium (IV), pp. 975–980, Paris, France, June 2019.

[36] R. Huang, J. Chen, J. Liu, L. Liu, B. Yu, and Y.Wu, “A practicalpoint cloud based road curb detection method for autono-mous vehicle,” Information, vol. 8, no. 3, p. 93, 2017.

https://arxiv.org/abs/1805.06771https://arxiv.org/abs/1805.06771http://www.ngsim.fhwa.dot.govhttp://www.ngsim.fhwa.dot.gov

vehicletrajectorypredictionbyknowledge-drivenlstm ... · researcharticle...

Documents