Frame Filters

With frame filters, a versatile and powerful way to manipulate image data is introduced. Besides other features, frame filters can be used to change or convert the video format of the image stream, implement image processing and control which frames are processed.

Introduction

In contrast to other DirectShow libraries, the frame filters of IC Imaging Control are not DirectShow filters. But they are executed in the context of a DirectShow thread that processes the image stream. This saves the programmer the hassle of creating a DirectShow filter module, registering it and defining interfaces for the parameters. Frame filters are implemented by deriving from the class FrameFilterImpl or FrameUpdateFilterImpl. Easy access to the parameters of a frame filter is provided by the class IFrameFilter. On the one hand, frame filters may be implemented as part of a filter module. On the other hand, a frame filter can be created directly in an application, if its class is known.

Filter Categories

Frame filters open the door to a vast number of new IC Imaging Control based applications. In order to show you what is possible with this new concept, we discuss categories of tasks that can be implemented with frames filters. For some of these tasks, frame filters are shipped with IC Imaging Control.

Gate:: Gate filters drop unwanted frames based on image analysis or other information like the frame count or time stamps.

Color Space Conversion:: All kinds of transformations from one color format to another fall in to this category. In addition, the reduction of the pixel depth, e.g. extracting the 8 most significant bits out of a 10 or 12 bit monochrome format is regarded as a color space conversion. The reduction of the pixel depth is used to display a 10 or 12 bit monochrome image. A good example for a color space conversion is the DeBayer Filter contained in stdfilters.ftf. It converts color raw data in Bayer format to RGB.

ROI:: Filters in this category allow a Region Of Interest to be cropped out of an image stream. The ROI Filter that is contained in stdfilters.ftf implements this functionality.

Rotation:: This category includes rotation and flipping. The Rotate Flip Filter that is contained in stdfilters.ftf provides rotation by 90°, 180° and 270°, as well as flipping on the horizontal and vertical axis. There is a filter in the DemoFilter.ftf that provides rotation by any angle, but this filter is a demo implementation and very slow.

Scaling:: Scaling up and down may be necessary to match a predefined display size or to save disk space for surveillance applications.

Sink:: Sink filters are consumers. They process the image stream in their transform method completely and do not send frames to the IC Imaging Control sink. If you want to use professional high performance codecs, you would implement a sink filter in order to interface the codec API. This approach is necessary because most professional codecs come as a DLL instead of a DirectShow codec. These codecs provides functionality to compress image data and write it to a file at the same time, which makes a sink obsolete. Of course, you could use the MembufferCollection of a FrameHandlerSink to access the image data and send it to the codec API. In this case, the image data would have been copied by the sink to the MembufferCollection which is not the case, if the codec is interfaced using a frame filter. In other words, the sink filter approach saves one copy operation per frame, which can increase the performance significantly, especially if large frames are processed.

Extraction:: Extraction filters take specific frames out of the image stream and apply processing on them, e.g. saving them to disk as image files. The image stream itself remains unchanged and is sent further to the sink. Such a filter allows specific images to be saved uncompressed, while a MediaStreamSink generates a compressed video file.

Image Processing:: All image processing functions that produce one image as output can be implemented easily by a frame filter. For functions that produce more than one image, special concepts have to be developed. One possible concept would be to write the output images one after the other into the resulting image. If data other than images is generated by the function (e.g. data generated by a blob analysis), the application could read out this data by using filter parameters.

Buffer Format Conversion:: Most image processing libraries have a special internal image buffer format. Frame filters can be used to convert image buffers of the image stream to such a special image buffer format. Doing so, the elements in the ring buffer of the FrameHandlerSink contain data in the special format. Therefore, the data of these buffers may be passed directly to the according image processing library. This saves one copy step and, therefore, increases the performance of the system.

Filter Types

There are 2 basic types of frame filters which will be discussed in the following. Both types provide the same way of parameter access and allow frames to be dropped explicitly.

Transform Filter: This type of frame filter has to be derived from FrameFilterImpl and allows to implement a conversion from one video format to another. This includes changing the color format (e.g. eY800 to RGB24), as well as changing the width and height.

Update Filter: This type of filter has to be derived from FrameUpdateFilterImpl. In contrast to a transform filter, an update filter may not change the color format nor the width and height. In order to speed up the processing of the image stream, an update filter can tell the framework that it will only read image data. This allows the framework to make certain optimizations. As a rule of thumb, update filters should only be used, if the image data is not changed at all or only a few pixels are modified. This is because the framework has to copy image buffers before it calls the FrameUpdateFilterImpl::updateInPlace method, if the filter has indicated that it will modify image data by returning true in the method FrameUpdateFilterImpl::modifiesData.

Using Existing Frame Filters

As described earlier, frame filters can be created directly in an application or loaded from a filter module. IC Imaging Control provides some filters in the module stdfilters.ftf. In order to obtain a list of the frame filters in all currently available filter modules, call the method FilterLoader::getAvailableFrameFilters. With the method FilterLoader::setLoadPath, an additional path for filter modules may be specified. A call to FilterLoader::createFilter creates an instance of a filter that was selected from the list of available filters. Now that we have a frame filter instance, we can insert it at three different locations in the image stream:

Behind the device: Grabber::setDeviceFrameFilters
In front of the display: Grabber::setDisplayFrameFilters
In front of the sink: MediaStreamSink::create, FrameHandlerSink::create

Please note that the same instance of a frame filter should not be used at more than one place in the image stream because its internal state may get inconsistent, especially if FrameFilterImpl::notifyStart and FrameFilterImpl::notifyStop are used. If it is necessary to insert the same filter at more than one location in the image stream, different instances of the frame filter should be used.

The internal data of a frame filter can be accessed through the parameter interface. With the method IFrameFilter::getParameter and IFrameFilter::setParameter, the following basic data types can be read from or written to the filter: bool, int, long float and std::string. A block of binary data can be exchanged with a frame filter by the methods IFrameFilter::getData and IFrameFilter::setData. Since the transform method of a filter is executed within a DirectShow thread, it is very important to synchronize external and internal access to the filter's data. Therefore, it is very important that a sequence of calls to the four methods mentioned above is preceded by a call to IFrameFilter::beginParamTransfer and is followed by a call to IFrameFilter::endParamTransfer:

pFrameFilter->beginParamTransfer();
// All calls to setParameter and getParameter here...
pFrameFilter->endParamTransfer();

If a frame filter is used by generic applications, such as the Filter Inspector, a dialog should be implemented that allows the parameters to be altered interactively. A frame filter indicates that it provides a parameter dialog by returning true in the method IFrameFilter::hasDialog. The dialog should be displayed when the method IFrameFilter::callDialog is called.

Every frame filter should provide the serialization of its internal data by implementing the methods IFrameFilter::getSettings and IFrameFilter::setSettings. The representation of the data has to be string compatible.

<< Technical Articles