The invention relates to a method and to a system for proactively recognizing an action of a road user in road traffic. The invention further relates to an accordingly equipped motor vehicle.
Driver assistance systems, which warn the driver of collisions and/or in part automatically control the motor vehicle, are used with ever greater frequency in motor vehicles to avoid such collisions between motor vehicles and other road users, such as pedestrians or bicyclists who do not have corresponding safety systems to avoid collisions, or even animals. In general, appropriate sensors are provided, which are used to detect the road user by way of measurement. The driver assistance system then becomes active based on the resulting measured data.
Various sensors have already been proposed for detecting road users by way of measurement, for example optical cameras, including video cameras, radar sensors, laser beam sensors or infrared (IR) cameras.
It is particularly important to recognize the action of road users at pedestrian crosswalks, such as zebra crossings. Such areas are frequently subject to situations where an approaching motor vehicle driver is not aware that a pedestrian is planning to cross the road because the pedestrian, for example, is standing with his back to the zebra crossing, but then suddenly turns around and, without further mindfulness, crosses the road because he feels safe on the zebra crossing. Collisions may then occur between the motor vehicle and the pedestrian, even when driving carefully.
A driver assistance system for avoiding collisions of a vehicle with pedestrians is known from WO 2009/141092 A1, in which the speed of an object moving toward a pedestrian crossing is detected by way of a camera sensor and/or a beam sensor, such as a millimeter wave radar, and the probability with which the object will cross the pedestrian crossing is determined. A warning is issued to the driver when a predefined probability threshold is exceeded.
A system for avoiding collisions between a vehicle and pedestrians is also known from EP 1 095 832 A1, in which pedestrian data is detected by way of measurement, as is additional data, such as regarding the roadway surface. A warning is optionally issued to the driver and/or control interventions in the vehicle control are carried out, such as automatic braking, based on processing of the data.
The content of the above-mentioned publications is hereby incorporated as non-essential matter herein.
It is the object of the invention to make a contribution toward the ability of avoiding collisions between a motor vehicle and other road users.
This and other objects are achieved according to the invention by providing a method and system for proactive recognition of an action of a road user in road traffic. For this purpose, an image of the road user, which is structured in a pixel-wise manner, is captured by way of at least one camera, and corresponding image data is generated; image data of multiple pixels is grouped in each case by cells, wherein the image comprises multiple cells; a respective centroid is determined based on the image data within a cell; for each of the pixels, the distance from the centroids of a plurality of cells is ascertained, wherein a feature vector that is associated with the pixel is formed based on coordinates of the respective pixel and the centroids; and the feature vector is compared to at least one reference vector cluster, and a pose is associated with the road user based on the comparison, the pose being representative of the road user planning to carry out the action.
The feature vector in particular has as many dimensions as cells of the image are provided.
A road user can, in particular, be a pedestrian, an animal, a bicyclist, a wheelchair user or another motor vehicle driver, including a motorcyclist. An action of the road user can, in particular, be a crossing of the roadway. By way of the pose recognized according to the invention, it can advantageously be achieved that the corresponding intention of the road user is already recognized before the action begins, for example a crossing intention. The invention is based on the finding that the intention of a road user can be recognized based on poses even before the actual action is carried out, which is to say that it is possible, based on the correct association of a pose of the road user, to proactively recognize that the user will carry out a certain action. Based on this pose recognition, it is then likewise possible to proactively take measures even before the action occurs, for example an acoustic and/or visual warning can be issued to the driver and/or to the road user detected by way of measurement, and/or a controlling intervention in a vehicle system can take place, for example in the brakes or in the steering.
Within the scope of the pose recognition, it is also possible to recognize or associate a sequence of poses, movements and/or partial movements of the road user. For example, the poses or partial movements of the individual extremities of the pedestrian can be determined by the camera of the vehicle and associated already in the region of the curb, even before the actual crossing of a pedestrian crossing by a pedestrian. In particular, the orientation of the pedestrian with respect to the camera and/or the direction of movement of the pedestrian can be detected. It is also possible to detect a group of road users and carry out the corresponding associations for the road users with respect to their potential intention to carry out a corresponding action. For example, it is possible to detect parameters such as poses and directions and speeds of partial movements of the individual extremities of potential pedestrians who intend to cross the road within a future time period. If the system ascertains that a pedestrian plans to cross the road, a warning can be issued to the driver of the motor vehicle, for example, the braking action during braking can be boosted and/or a direct intervention in the braking system can be carried out. Time data, such as a respective start, a duration and/or an end of a time period, at which it is to be expected that the road user detected by way of measurement will carry out the action, can be associated with a pose or multiple consecutive poses. Based on the corresponding time data, further processes, in particular control processes, can be initiated and/or controlled in the motor vehicle for collision avoidance.
A system according to the invention can in particular be a driver assistance system or be integrated therein. It can in particular be provided in a motor vehicle.
It is also advantageously possible to achieve by way of the invention that a correct association of a pose of the road user is already possible using a camera having a relatively low resolution of just a few 10,000 or 100,000 pixels, for example.
On the other hand, it is possible to achieve in this way that the image processing step can be carried out at high speed, and in particular in real time. In this way, effective collision protection can advantageously be achieved for the road users. The image signals can additionally have a high noise component without significantly impairing the detection quality. In this way, the action or the corresponding intention of the road user can be reliably proactively recognized even under unfavorable lighting and/or weather conditions.
In one preferred exemplary embodiment of the invention, the image data captured by the camera is filtered for the determination of the respective cell centroid to the effect that data of a contiguous image area, referred to as foreground image data, is created, wherein the road user is depicted in the image area and at least some of the other image components captured by the camera are filtered out. The cells comprise the contiguous image area of the road user, wherein in particular each of the cells or only a small portion of less than 10% of the cells, for example, comprises at least one pixel of the contiguous image area.
In a further preferred exemplary embodiment of the invention, a respective pixel-wise orientation of the road user relative to the camera is associated in the course of the comparison of the respective feature vectors of the respective pixels to the respective at least one reference vector cluster. Based on the pixel-wise associations, an orientation of the road user is associated. Overall, the ultimate association of the orientation of the road user can, in particular, take place based on a multitude of pixel-wise associations.
In a further preferred exemplary embodiment of the invention, the feature vectors are compared pixel-wise to the respective reference vector clusters and/or further reference vector clusters, which are classified according to possible poses of the road user, and in particular according to the associated orientation based on the associated orientation. The pose is then associated by way of the result of the comparison. The orientation and pose can also be associated in one step, for example by combining the two described steps for associating the orientation and the pose into one step. The two respective reference vector clusters can accordingly be summarized or combined for this purpose in a combined reference vector cluster.
The invention can advantageously be used to provide a two-stage or two-part process in the association of the pose of the road user in such a way that, on the one hand, or first, an association of the orientation of the road user relative to the camera is carried out, in particular from a predefined number of orientations and in particular in two directions as to whether the road user faces away from the camera or faces the same. On the other hand, or thereafter, the association of the pose of the road user is carried out, in particular based on the first association of the orientation, and in particular from a predefined number of poses.
Such a two-part or two-stage process in the association of the pose allows a high association accuracy of the pose to be achieved since the image data for the association of a pose can be evaluated significantly more easily when the orientation of the road user has already been determined.
According to a further preferred exemplary embodiment of the invention, the comparison of the feature vectors to the reference vector clusters is carried out by way of a random forest method. In such a classification method, decision trees that have been constructed in a certain kind of randomized process and comprise the reference vector clusters are used in particular in a learning process. The reference vector clusters can each be associated with an orientation of the road user relative to the camera and/or a pose of the road user.
So as to proactively recognize the action of the road user, advantageously further data can be captured and used, in particular surroundings data, such as data regarding a traffic light, a traffic sign or a zebra crossing; geographic data, such as position data determined by a global positioning system (GPS); map data from a navigation system and/or road images from a two- or three-dimensional map. It is also possible to use measured data regarding the road user, such as data from a LIDAR measuring device, an ultrasonic measuring device or an IR measuring device, and/or data of the motor vehicle, such as the speed thereof.
The invention can be employed for a plurality of traffic situations, for example for crossing a zebra crossing by pedestrians or for the detection of children playing between cars who suddenly cross the road.
A stereo camera system having two cameras can advantageously be used for the invention. In this way, three-dimensional images of the road user can be recorded and used, and the recognition reliability can be increased yet again. The pixels and the respective feature vectors thereof are then likewise configured according to these three spatial dimensions.
Other objects, advantages and novel features of the present invention will become apparent from the following detailed description of one or more preferred embodiments when considered in conjunction with the accompanying drawings.