Modern video management software packages provide a range of tools for the situational control of objects of various kinds. In most video systems placed in public places (safe cities, banks, shopping centers, markets, railway stations, airports, subways), the abandoned object detector is a very popular tool for anti-terrorist security tasks. The detector allows you to recognize objects which were in motion and then stopped moving within a certain time period.
Today, two classes of algorithms can be defined for solving abandoned objects detection issues:
- Algorithms based on search of foreground objects;
- Algorithms based on tracking (tracking moving objects).
Let’s consider these classes in detail.
Detecting abandoned objects by searching foreground objects
The foreground refers to all moving objects on a screen. All other fixed pixels form a background.
This method uses two background models: short-term (rapidly changing, fast) and long-term (slowly changing, long). Each has its own speed and update time.
The update time of a short-term model is very short. That is why all changes of a current frame are compared to a preceding one captured quickly in the changing short-term background. If a new object in motion appears on a screen, it will not become part of the background. But if an object has stopped moving (which is quite often what an abandoned object does), it is united with a fixed background. Thus, short-term background determines that there was movement on a screen or there was a moving object, but then this movement stopped. An object has became static and merged with the background.
A single use of a short-term model is impossible, since a static object, presumably left behind, "disappears" very quickly - merge with the background. It will be impossible to track the time period when it does not move. That is why the algorithm uses both short-term and long-term background models. A long-term background model changes much more slowly. New objects which come into it remain in the foreground for some time and become part of the background after a while. A long background model is used for abandoned object detection. It allows one to capture the fact that it appeared on a screen, but was not there from the very beginning as part of the original background.
A system’s operator can set-up a time after which an object will be considered abandoned. After receiving this information, the system calculates a ratio which adjusts an update time of the long-term background.
The logic of this method is shown in picture 1. Each pixel of a current frame is compared with corresponding pixels of the long-term and short-term backgrounds. After that, a hypothesis is made, based on a comparison of results.
Picture 1. Abandoned object detector’s logic based on a foreground objects search
Detecting abandoned objects using tracking
By using tracking algorithms for the detection of abandoned objects, a system not only detects objects left by an object (person) but a person who left an object behind as well. To be exact, the system detects a moving object split into two parts: static and moving. It begins monitoring a static object after that.
In the approach of using tracking, the first step is to separate the background. The next step is to build pathways of all moving objects on a screen. Then it captures the "split" of a moving object into two, one of which continues to move, while the other remains static. The program analyses speed, the direction of movement, and colour characteristics of objects. This data is used to compare objects to people and their "split" into two properties.
Рiс. 2. Detecting an abandoned object and its owner
Comparison of methods
Each method has its advantages and weaknesses. Let’s consider them in detail:
1. Working with a crowd. Algorithms based on a foreground objects search have shown good results in this situation, provided that the background is correctly determined. It is possible if the background appears on a screen for some time. In other words, it is not blocked by the crowd completely for all of the time. For the method that employs tracking algorithms while working with a crowd, major difficulties lie in the capture of a separate moving object.
2. The use of resources. The complexity of data processing and, consequently, the use of resources for tracking is greater. The system needs to determine the background, detect moving objects, highlight characteristics of moving objects (colour characteristics, speed, and direction of movement), and calculate the relativity of an object’s transition from one point in the preceding frame to another point in the current one. All of these tasks should be fulfilled in order to the build pathways of a few moving objects on a screen. More computing power is required in order to carry out all these calculations.
3. Difficult situations in which the possible omission of abandoned objects may occur. Major difficulties occur for the method based on the foreground objects search if it is impossible to determine a background correctly (working with a crowd, partial lighting changes). It is very challenging for the tracking method to deal with scenes of high traffic, moving objects overlap, and temporary object’s disappearance from a camera’s sight.
4. False starts. False starts are possible with both methods in situations where a person stays on a screen for a period of time equal to the time which has been set-up as a trigger for abandoned objects. The reason for this is that the system is unable to distinguish between a person and an object, especially if this person takes a pose in which it is difficult for the program to identify him as a human being (e.g., sitting). This problem might be addressed by combining a detector with a classifier, which can identify categories of objects and distinguish between human beings from other objects, in any position.
5. Tracking method human beings detection. This method also allows you to obtain data on all movements of a person within a camera's range. The system operator can find a person who brought an abandoned object in and the use other intellectual functions such as inter-camera tracking or "interception" in order to capture this afterwards.
|
Algorithms based on foreground objects search |
Algorithms based on object tracking |
|
|
Working with crowds |
A high level of detection is achieved |
There are difficulties with a single moving |
|
Complexity of processing |
Average |
High |
|
Consumption of resources |
Average |
High |
|
Complex situations |
Impossible to determine a background correctly |
High-intensity movement of objects on a screen, |
|
False starts |
When a person’s stay on screen, |
When a person’s stay on screen, |
|
Detection of humans |
Impossible |
Possible |
Method’s based on foreground objects search modifications
The comparative analysis of these methods shows that an algorithm which uses highlighting of foreground objects is the most stable and consumes fewer resources. But it is not stable in high traffic density or under changing conditions (e.g., lighting changes). Software developers have implemented the following improvement in order to minimize the impact of environmental condition changes.
The improved method of background updating. Motion detection
The use of motion detection improves the stability of the algorithm under changing shooting conditions (e.g. lighting changes). If motion detection is in place, the software identifies the framework of a moving object. Then it creates masks from the frameworks of all moving objects. Background update occurs only in areas of the screen where there are no moving objects. In areas where there are masks of moving objects, an update does not occur.
Static check
Additionally, a static check is used to reduce the number of false starts and increase the accuracy of abandoned object detection. This modification eliminates temporarily static objects which are of no interest to a system’s operator. When performing this check, the program allocates a frame of a potentially abandoned object and memorizes its position. When a new frame is received the frame is allocated to a static object, its coordinates are compared with the value of the stored coordinates. If, after comparison, a positive result occurs (i.e., the frames are approximately in line), the test is repeated several times. The number of inspections depends on the required detection time of the left object. If the comparison yields a negative result (i.e., the frame did not match), then the checks counter is reset, and the stored frame is deleted.
Selection criteria
The existing detectors of abandoned objects are primarily based on two methods. One of them uses tracking algorithms (building movement pathways of moving objects). Another one is based on an algorithms foreground objects search. Each method has its own characteristics and applicability. The main advantage of the tracking detector is its ability to detect a person who has left an object behind. However, this method requires high system performance. It fails to provide reliable results in heavy traffic scenes. The foreground objects search method provides a high degree of accuracy in detecting abandoned objects, provided that the original background was correctly determined. It means that if a video system "sees" a clean background scene for at least a few seconds, its detector will show good results detecting abandoned objects. In addition, the detector has low resource consumption. Recently introduced modifications of this method increased detection accuracy and reduced false starts.