The simulation environment of this project is built based on the open-source simulation software webots. The background of the simulation environment simulates the target tracking problem in outdoor scenes in real life. For this purpose, six maps have been set up, including city streets, city center, lakeside, countryside, desert, and farmland, each of which evolves into various conditions such as day, night, foggy, and snowy, making a total of 24 environments. To facilitate model training from simple environments, several simple environment maps are also available for use.
Schematic diagrams of 24 simulation environments in the project:
This mode generates a tracker in the map. The tracker can fly through the map by keyboard operation, allowing you to browse the map. Meanwhile, the tracker by default tracks a BmwX5 vehicle near its initial position and outputs the reward parameters in real time on the Console status bar. This object can be used to test whether the built-in reward parameters are reasonable for debugging purposes.
up: Drone forward
down: Move the drone backward.
left: Drone rotates vertically counterclockwise.
right: Drone rotates vertically clockwise.
wsad: Controls the rotation of the camera head up, down, left, right, and center.
shift + up: Raise the drone.
shift + down: Drop the drone.
shift + left: Drone pans left.
shift + right: Drone pans right.
Note: At this point, rewards include both continuous and discrete rewards, see built-in Reward for details.
This mode is mainly used to record the behavior of the tracked object. The environment will automatically schedule the tracker error tracking object and export video images according to the simulation time and configuration parameters, while generating labels corresponding to each image.
The data is stored in: ./Webots_Simulation/traffic_project/droneVideos/DRONEX/episodeX, with the image format as jpeg and the labels in the mark.json file.
At the same time, you can generate multiple sets of video image file structure descriptions through the script by directly executing the ./Webots_Simulation/traffic_project/fileStruct.py script, which will generate the description file dir_info.json in the same directory.
This mode is used for algorithm training. You can choose to manually open the Webots simulation first and then run the training script, or you can choose to modify the algorithm configuration file to achieve one-click full startup of the algorithm.
Simulation_Mode : There are three simulation modes: “demo”, “video”, and “train”. The demo and video modes are self-running modes of the simulation (default is demo).
The demo will generate a tracker in the map, which supports direct control via the keyboard. Meanwhile, it will track a stationary vehicle, Bmx, by default. This mode can be used to browse the map or debug the reward parameters.
The video mode is the video shooting mode. In this mode, the tracker will automatically track the target object, exporting video images and labels.
train is the training mode, where the simulation end interacts with the algorithm end to train the tracking algorithm of the tracker.
For the complete system operation flowchart, see: Operation Flow
Tracker_Def : Select the tracker, the current options are drone (“DRONE”) and car (“CAR”). Among them, the drone is a simulated drone tracker with physical property detection and constraints, while the car is considered an ideal tracker that ignores its own physical properties and can perform any action under any circumstances.
Drone_Supervisor_Ctrl : Whether to use the supervisor to control the tracker. The supervisor is Webots’ super administrator and can directly control any node. If true, the supervisor controls the tracker, and all its actions are real-time and responsive, which simplifies algorithm training. If false, the tracker is controlled using a traditional PID controller. This parameter is only applicable to the drone tracker; the car tracker always uses the supervisor for control.
Socket_Ip : The IP address for socket communication, defaults to “127.0.0.1” if communicating locally.
Config_Agen_Num_Port : Configures the socket communication port, which is used to communicate with the algorithm end to determine the number of trackers (agents) to be set.
Train_Total_Steps : The maximum number of training steps. If the steps exceed this number, the simulation returns done.
Init_No_Done_Steps : The number of simulation steps without done status during initialization. This prevents the unstable physical properties of the tracker during initialization from triggering the done status. This parameter should be set to a reasonable value.
No_Reward_Done_Steps : The number of simulation steps that result in a done status when the reward is continuously zero, which helps improve training efficiency.
Control_Frequence : The control frequency for the control algorithm. During training, the simulation end executes at a fixed control frequency of 500Hz based on the simulation time. This parameter is used to adjust the average control frequency of the algorithm end.
Customized_Rewards: Whether to enable the customized reward parameter. Users can calculate rewards based on the customized reward parameter.
Tracking_Object: The tracked object setting (default is SUMO_VEHICLE). Example tracking objects include “SUMO_VEHICLE”, “Pedestrian”, “Shrimp”, “Create”, “Sojourner”, “Mantis”, etc. Custom tracking objects are also supported. See the project manual.
For detailed descriptions of tracking object types, see: Tracked Objects
Verbose: Enable log recording. If set to true, some important parameters will be printed in the Webots terminal window for users to observe. Additionally, program run logs will be generated in the Webots_Simulation/traffic_project/logs folder, mainly for debugging the simulation environment.
id : The name of the tracker.
done : Whether to return done.
reward : The current reward size.
count : The step count of steps.
time : Simulation time.
carDir : The movement direction of the tracked target.
pitchAngle : The pitch axis angle of the camera.
doneState : The reason for generating done, including “height” (unreasonable tracker height), “distance” (the error between the tracker’s target position and the actual position is too large), “velocity” (unreasonable tracker movement speed), “yaw velocity” (unreasonable tracker rotational speed around the vertical axis), “pitch or yaw angle” (unreasonable tracker attitude).
Tracker (only effective for drones) generates a physical range parameter configuration of done
reward_type : The type of reward (default is view). “View” detects based on the quadrilateral projected onto the ground from the camera’s perspective, while “distance” detects based on the distance from the midpoint projection in the camera’s view to the target.
reward_mode : The default mode of reward. “Continuous” produces a continuous floating-point number between [0, 1], while “discrete” produces an integer of 0 or 1.
reward_cut_off : Reward cut-off. When the obtained reward is less than this value, No_Reward_Done_Steps starts counting.
distance_scale : Distance scale (default is continuous). This parameter is only effective in continuous mode and controls the decay rate of the reward due to distance error.
distance_range : The distance error range for generating rewards.
view_scale : View scale. This parameter is only effective in continuous mode and view type, controlling the decay rate of the reward due to deviation from the view center.
view_range : The view range for generating rewards. This parameter is used to scale the quadrilateral used for reward calculation, i.e., the size of the quadrilateral box in the camera view. Generally, the range is set to [0,1].
rou_update : Whether the traffic simulation route file is updated. If you want to obtain a specific route, set it to false. Note: Any change in the parameters of this module will cause the route to refresh automatically.
max_sumo_car : Maximum number of vehicles in the map.
fixed_car_group_num : Whether to customize the number of vehicles added to the environment in each group.
car_group_num : Custom number of vehicles added to the environment in each group. Note: In training and video mode, if this parameter is set unreasonably (< Agent number * 3), the system will automatically set a reasonable parameter (= Agent number * 3), otherwise the environment cannot run stably.
car_import_interval : The interval time (s) between each group of vehicles added to the map.
car_type : The type of vehicle (default is passenger), including “passenger”, “bus”, “truck”, “trailer”, “motorcycle”.
All supported vehicle types can be found at: SUMO_VEHICLE
fixed_color : Whether to fix the vehicle color.
normalize_color : Normalized RGB color vector.
max_car_speed : Maximum vehicle speed (\(m/s\)).
max_car_accel : Maximum vehicle acceleration (\(m/s^2\)).
max_car_decel : Maximum vehicle deceleration (\(m/s^2\)).
max_rou_distance : Maximum route length (\(m\)).
min_rou_distance : Minimum route length (\(m\)).
route_num : Maximum number of routes (note that this is the maximum, obj_edge_distribution_multilateral may actually have fewer routes than the set parameter due to the randomization process).
fixed_seed : Whether to use a fixed random seed. If you want to obtain stable routes and vehicle styles, set it to true.
obj_import_interval ：Time interval (s) for adding each group of objects to the map.
import_group_num ：Number of objects added per group.
obj_edge_distribution_random ：Whether to enable random distribution of objects along the sides of the map roads. If true, objects will be randomly placed in a random direction at random positions on a random path. If false, objects will be placed at a fixed distance in the vertical direction at random positions on a random path.
obj_edge_distribution_multilateral ：Whether to enable bilateral distribution. If true, the distribution range will be [min,max]∪[-max,-min] . If false, the distribution range will be [min,max] .
obj_edge_distribution_fixed ：Fixed distance (\(m\)) from the center of the road.
obj_edge_distribution_max ：Maximum distance (\(m\)) from the center of the road.
obj_edge_distribution_min ：Minimum distance (\(m\)) from the center of the road.
channels ：Number of video channels, which corresponds to the number of trackers in the imported map.
fps ：Frames per second calculated based on simulation time (video frames are exported at this frame rate).
start_time ：Simulation time (\(s\)) at which video export begins.
total_time ：Duration of the video in simulation time (\(s\)).
warm_up_steps ：The number of steps after reset during which no data is output. Note that each step has a cycle of 2ms.
output_car_dir ：Whether to output the vehicle direction. If enabled, only the vehicle movement direction is output; otherwise, reward and action are output (Vehicle directions: 0 stop, 1 forward, 2 left turn, 3 right turn).
random_action ：Whether the tracker uses random actions. If enabled, random actions are output based on Control_Frequency. Otherwise, the tracker will precisely follow the vehicle, and the action output will be -1 (meaningless).
forward_speed ：Forward speed of the tracker’s random action (\(m/s\)).
backward_speed ：Backward speed of the tracker’s random action (\(m/s\)).
left_speed ：Leftward speed of the tracker’s random action (\(m/s\)).
right_speed ：Rightward speed of the tracker’s random action (\(m/s\)).
cw_omega ：Clockwise rotation speed of the tracker’s random action (\(rad/s\)).
ccw_omega ：Counterclockwise rotation speed of the tracker’s random action (\(rad/s\)).
start_time_bias_ms : Due to the time required for environment initialization, theoretically, the more agents are set, the longer the initialization time. This parameter can adjust the overall initialization time.
Note: The total initialization time will increase with the number of agents set, and it will be longer than the time set by this parameter.
view_pitch_random : Randomization of the tracker camera pitch angle.
view_pitch_fixed : If the above parameter is false, the angle defined by this parameter will be used.
If the angle is greater than 0, it is a downward pitch; otherwise, it is an upward pitch.
In practical usage, a downward pitch is common. Therefore, the parameter range in the simulation environment is limited to \([0.5, 1.57](rad)\).
view_pitch_random_max : Maximum camera downward pitch.
view_pitch_random_min : Minimum camera downward pitch.
height_random : Randomization of tracker flight height. Other parameter meanings are the same as above.
direction_random : Randomization of the tracker’s initial view direction following the object, with a range of \([-pi,pi](rad)\).
direction_random_multilateral : Whether to enable bilateral randomization configuration.
horizon_bias_random : Horizontal offset of the initial view tracker tracking object. If the subsequent parameter is positive, the object shifts to the left; if negative, the object shifts to the right. Other parameters are the same as above.
verticle_bias_random : Vertical offset of the initial view tracker tracking object. If the subsequent parameter is positive, the object shifts upward; if negative, the object shifts downward. Other parameters are the same as above.
This project provides 18 types of 24 predefined tracking targets, and offers a plug-and-play interface that supports users in utilizing self-designed models and controllers.
SUMO_VEHICLE is controlled by the SUMO traffic management system to manage the traffic conditions on the map, allowing vehicles to follow planned routes on the map. The type of vehicle can be specified.
passenger
Passenger cars include common sedans, SUVs, and business vehicles, as shown below:
This project supports custom tracked objects, and the steps are as follows:
Create a proto file for Webots following webots wiki . The attributes that must be exposed to the scene tree include “translation”, “rotation”, and “name”. The description file should be stored in UAV_Follow_Env/Webots_Simulation/traffic_project/protos
Due to the existence of the super intelligent agent in Webots, we can obtain the coordinates of the aircraft and the tracked car at each moment, (x_f, y_f, z_f) and (x_v, y_v, z_v), and directly determine whether the tracking is successful based on the actual pitch angle of the camera and the set camera FOV.
For the tracker (drone), there are two options for the reward function. The first option is sparse rewards, where rewards are given when the tracked object enters the tracking range, and no rewards are given when it leaves the range (the reward is 0). The mathematical expression is as follows:
\[\begin{split}r_{d} = \begin{cases}
1 & t \in I, \\
0 & \text{otherwise}.
\end{cases}\end{split}\]
Where \(I\) represents the tracking range (the area captured by the camera lens), and \(t\) represents the position of the tracked object within the tracking range (the area captured by the camera lens).
The second type is dense reward, which gives a reward that decreases as the distance increases by evaluating the position of the tracking object relative to the center of the tracking range. The higher the reward, the closer the target is to the center point of the range, and vice versa.
The closer the target is to the center of the range, the closer the reward is to 1; conversely, the farther away, the closer the reward is to 0. \(\alpha\) is a constant, and a recommended value is between 3 and 5.
In addition, \(I_{clip}\) is the effective reward determination range (as shown in the red box in the figure above Continuous Reward Diagram for Object Tracking), and the ratio of its side length \(W_{I_{clip}}\) to the original image side length \(W_{I}\) can be expressed by \(\lambda_{clip}\) as follows.
\[\lambda_{clip} = \frac{W_{I_clip}}{W_{I}}\]
According to the above definition, the reasonable range of \(\lambda_{clip}\) is \([0,1]\), and in actual training, it is recommended to set the parameter as \(\lambda_{clip}=0.7\).
The acquisition of the above reward function is based on the judgment of the tracked object’s position. In practice, the position of the tracked object in the camera image can be provided through machine vision recognition or manual annotation. However, in simulation training, in order to simplify the calculation and speed up the algorithm, we directly use the coordinates of the tracked object to determine whether it has entered the tracking range.
Where \(W\) represents the width of the camera image (in pixels), and \(FoV\) refers to the field of view of the tracker camera, as shown in Figure b of the Schematic diagram of reward acquisition (a).
Then, as shown in the Schematic diagram of reward acquisition (a), we need to project the camera’s corner points onto the spatial plane where the tracked object is located. Next, as shown in the Schematic diagram of reward acquisition (b), determine the coordinates of the tracked object. Finally, by establishing the camera coordinate system \(\{c\}\) at the optical center of the camera, the coordinates of the four corners of the camera image in the coordinate system \(\{c\}\) can be obtained:
Then get the equation for the ground (horizontal):
\[G_{w}: z = h\]
\(\{w\}\) represents the world coordinate system, and \(h\) is the ground height.
Since the pose of the tracker gimbal is known, it is easy to obtain the transformation relationship from the camera coordinate system \(\{c\}\) to the world coordinate system \(\{w\}\), which is the homogeneous transformation matrix \(T_{ct}\).
By transforming the plane equation \(G_{w}\) to the coordinate system \(\{c\}\), and solving it together with the four line equations, the four ground intersection points \(LU_{G}, LD_{G}, RU_{G}, RD_{G}\) can be obtained. These four points form the four corner points of the tracking range \(I\), thus determining the tracking area \(I\).
Finally, read the tracking object’s coordinate point \(p_{vG}\) from the simulation environment, and convert it along with the image center point I_{OG} to the coordinate system \(\{c\}\), obtaining \(p_{vc}\). Using the car’s coordinates \(p_{vc}\), the four intersection points (LU_G, LD_G, RU_G, and RD_G), and the image center I_{OG}, by utilizing the Continuous Reward Diagram for Object Tracking , the ratio \(x\) can be calculated. When \(0 \leq x \leq 1\), the object is within the image, indicating successful tracking.
The calculation method is similar when the ground is not horizontal.
After implementing the above principle, the effect is as shown in the following video. By placing the block at the intersection point of the ground and the calculated line in the simulation, the camera’s view range can be seen in real time:
Note: The upper limit of the field of view cannot exceed the horizon, otherwise the calculation results will be meaningless.
The input actions are used to control the movement of the tracker. Currently, the tracker has four actions: move forward, move backward, move left, move right, plus the reset data, totaling 5 floating-point numbers.
The custom reward data packet consists of 59 floating point numbers + 1 string, including the following parts
typedefstruct_rewardData{doublecameraWidth;// Image width (pixels)doublecameraHeight;// Image height (pixels)doublecameraFov;// Camera field of view (rad) [0,pi]doublecameraF;// Camera focal length estimate (pixels)doublecameraPitch;// Camera pitch angle (rad)doubletrackerHeight;// Tracker height (m)doubleT_ct[16];// Homogeneous transformation matrix from camera to world coordinate systemdoubleT_tw[16];// Homogeneous transformation matrix from vehicle to world coordinate systemvector3cameraMidGlobalPos;// 3D coordinates of the camera center mapped to the ground in the world coordinate systemvector3carMidGlobalPos;// 3D coordinates of the vehicle center in the world coordinate systemvector3cameraMidPos;// Camera center coordinates in the world coordinate systemvector4carDronePosOri;// 1D pose + 3D coordinates of the vehicle center in the tracker coordinate system (the first element is the pose)vector3carDroneVel;// 3D velocity of the vehicle center in the drone coordinate systemvector3carDroneAcc;// 3D acceleration of the vehicle center in the drone coordinate systemdoublecrash;// Whether the tracker has collided with a buildingdoublecarDir;// Vehicle running direction (0 stop, 1 straight, 2 left turn, 3 right turn)std::stringcarTypename;// Object type}rewardData;
Among them, the parameter carCameraPosOri defines the tracker coordinate system and pose angles as follows:
Radar point cloud data. The tracker in this environment is equipped with RpLidarA2 single-line radar, and its point cloud data will output an array with a length of 2000.
