Video product differentiation
This article refers to the address: http://
The standardization of high-definition digital video codec algorithms ensures interoperability between different products, and the output of the standard encoder must be able to be decoded by all standard decoders. Therefore, there is not much room for differentiation in the codec algorithm. This is good for end product designers because it helps chip designers make very efficient designs for algorithms; it is a disadvantage for designers who want their products to be different. Fortunately, video pre-processing and post-processing modules for improving image quality and color reproduction provide opportunities for product differentiation while also requiring product programmability.
Video preprocessing algorithm
Undoubtedly, the video stream will not be transmitted in the initial state from the sensor, and a series of transforms will be performed on the original stream before encoding. The main pre-processing operations include:
* Pixel Scan/Data Transfer – This step simply extracts the image from the sensor.
* Bayer format de-interlacing – Modern video image technology replaces the monochrome image sensor with a three-color RGB Bayer filter, so the data stream from the sensor includes red, green, and blue color information. After the information is separated, it is converted into luminance and chrominance information of YCbCr to represent the image.
* Noise filtering – there is always noise in the electronic world, and the best stage to reduce or eliminate noise is before encoding.
* Jitter detection and compensation – Jitter detection and compensation can reduce the degradation of image quality caused by camera shake.
* Local dynamic range compensation – In certain modes, the dynamic range of the image may exceed the limits of the sensor. The camera or camera should have the ability to intelligently adjust the exposure to extend the basic dynamic range of the sensor.
* Focus adjustment (sharpening) – The image sensor cannot capture continuous image information. They break the image down into pixels and then interpolate between pixels and recombine into new images based on different resolutions. This operation can result in a decrease in image sharpness, but can be corrected by an appropriate preprocessing algorithm. In addition, the sharpening algorithm can make the image that is slightly out of focus due to the lens look sharper.
* Color Correction - White light at different color temperatures is different, which affects the resulting image. In addition, different display devices handle color differently. Color correction compensation in video systems corrects color shifts, especially skin tone shifts
* Face Recognition - In images with many people, the photographer usually wants the image to focus on the person's face. An important feature of cameras and camcorders is the ability to recognize faces and autofocus.
* Stereoscopic image – Constructs a stereo image based on the density information of the planar image. This preprocessing is commonly used in mixed reality systems that combine virtual and real images.
Figure 1 is an example image of four preprocessing algorithms: noise filtering, jitter detection and compensation, dynamic range compensation, and color correction.
Figure 1: Examples of four preprocessing algorithms: noise filtering, jitter detection and compensation, dynamic range compensation, and color correction.
Video post-processing algorithm
The images obtained from the standard video decoder are not perfect, and the video post-processing algorithm can greatly improve the image quality and increase the added value of the final product. Video post-processing algorithms include:
* Deblocking/De-looping Filter – In the process of video compression and decompression, the image is divided into small blocks and encoded separately. When decoding, these blocks are combined into a complete image. This operation is lossy, and the deblocking/de-ringing filter can reduce its effect on the image.
* Boundary Detection – Boundary detection algorithms are used for scaling, deinterleaving, and other video processing operations. In addition, it plays an important role in security, video surveillance, traffic management and medical image processing.
* Image scaling – From flat-panel TVs hanging on the wall to mobile phones in the hands, the screen size of video devices varies widely, which makes the encoded video stream size and screen size of the playback device different. The image scaling algorithm can adjust the size of the video stream to fit the screen of the playback device.
* Deinterlace – Video interleaving is designed to be displayed on a TV screen. Each interlaced video image is displayed as two consecutive frames, each of which contains odd or even strips of video lines, so that each of the alternate images contains Half of the video line corresponding to the image. However, when displaying on a computer screen or a liquid crystal television, each picture needs to display all the video lines of the corresponding image, and the interlaced video must be deinterleaved before being displayed.
* Frame rate conversion - The method of interpolating between decoded frames to increase the video frame rate enhances the smoothness of moving images. This ability is very important for the conversion of high-end 120fps video and 24fps movie video.
* Noise Filtering - Noise can degrade the quality of video images. Various noise filtering algorithms can improve image quality.
* Video overlay / transparency effects - Many video systems overwrite the display user interface while playing video.
* Color Space Conversion / Brightness / Contrast / Gamma Correction - Different display devices have different dynamic ranges and different color processing (different color transfer functions). If you know the corresponding parameters, you can choose the appropriate video post-processing algorithm. Gamma correction is very important for liquid crystal displays because each LCD panel has a unique transfer function.
2 is an example image of four post-processing algorithms: deblocking and de-ringing, boundary detection, image scaling, and de-interlacing.
Figure 2: Examples of four post-processing algorithms: deblocking and de-ringing, boundary detection, image scaling, and de-interlacing.
Implementation of video preprocessing and post processing algorithms
Unlike video codec algorithms, there is no uniform standard for pre- and post-processing algorithms, and the algorithms used in each product are different. In addition, the original algorithm will be continuously upgraded, and new algorithms will continue to emerge. These require the hardware of the pre/post processing algorithm to have higher programmability, so the configurable processor optimized for these algorithms is a good one. s solution.
You may not be familiar with the concept of configurable processors, but you must be familiar with fixed instruction set architecture processors, and use them since the day the system was first introduced. The fixed instruction set architecture processor continues the tradition of using standard processor architectures since 1971, when Intel invented the industry's first commercial single-chip processor, the 4004. The emergence of SOC (in fact, the ASIC with built-in processor) changed the traditional architecture of the microprocessor in the mid-1990s. Since the user can customize the silicon according to the specific application, the on-chip processor can also be customized. But in the past, custom processors have not been realized due to the lack of expensive cost for specialized processor designers and development and maintenance software tool chains.
Today, automation tools can help logic designers and software developers customize processors for specific tasks without the need for specialized processor designers. With the addition of new instructions and registers and the merging of multiple instructions into a single instruction, the optimized custom processor achieves a much higher performance than the general purpose processor and DSP. Therefore, it is logical to develop a video-specific processor to accelerate various video pre/post processing algorithms. There are many ways to implement video algorithms. The approach mentioned here is simple: use C to describe the algorithm, compile and run on the processor and perform performance analysis. Then you can see the performance difference on different processor architectures. .
Table 1 lists the results of the above-described methods for five different video algorithms running on RISC processors, DSP processors, processors customized for algorithmic automation tools, and processors manually tailored by the designer. The RISC processor here is a RISC core based on the Tensilica Xtensa architecture. The DSP processor is also based on the Tensilica Xtensa architecture, but adds a series of DSP extension instructions including 4 SIMD MAC and DSP-specific ALU to execute 4 or 8 The SIMD vector operation, which also supports the very long instruction word function, can emit 3 independent instructions per clock cycle. This set of extended instructions is called the Vectra LX instruction set.
The video processing algorithms listed in Table 1 include:
* 5x5 pixel 2D filter: 2-D filter can be used in operations including image sharpening, blurring and increased jitter.
* 5x5 pixel threshold filter: Threshold filtering uses black and white binary values ​​to represent image pixels, so that each pixel occupies only 1 bit. The grayscale information is not included in the filtering result and is used in algorithms such as noise detection.
* Median filtering: Median filtering is often used to reduce the effects of salt and pepper noise and Gaussian noise.
* Frame Sampling: Simple Image Reduction
* Frame summation: This algorithm is used to measure the light intensity of different parts of the video image. It is used in both noise reduction and exposure compensation, and is also used for auxiliary video compression.
The Xtensa processor with SIMD/DSP extension is much faster than the un-expanded version, and the performance of most video processing algorithms has been greatly improved, but the increased area is relatively small (about 0.4mm 2 in the TSMC 65LP process) .
The "XPRES" column in Table 1 shows the performance of the five algorithms performed by the processor customized with the Tensilica automation tool XPRES.
XPRES performs performance analysis on C source code and automatically generates processor extension instructions optimized for the corresponding code. The automatically generated processor is slightly smaller than the processor that extends the Vectra LX instruction set, but the performance of most algorithms is slightly better.
The last column of Table 1 shows the manual custom extension instructions and IO interfaces to speed up the processor performance of a particular algorithm. We can see that the performance of the 5x5 2D filtering algorithm is 170 times higher than that of the uncustomized processor; the performance of the 5x5 threshold filtering algorithm is improved by 225 times; the performance of the frame summation algorithm is improved by 44 times.
Table 1: Video Algorithm Performance Results: Performance comparison between RISC processor, DSP processor and Tensilica automation tool XPRES custom processor.
Cat7 Network Cable,Cat 7 Flat Lan Cable,Cat 7a Ethernet Cable
Kuyia Technology Co., Ltd. , http://www.gdhdmicable.com