Suppose we have two consecutive image frames f(n1,n2,t-1) and f(n1,n2,t0), as shown in Figure 3.46. We wish to create a new frame f(n1,n2,t) where t-1< t <t0. A simple approach is to choose the original frame that is closer in time to the desired frame. One problem with this approach is the jerkiness that results if the sequence has a large global motion.
An alternative is motion-compensated interpolation using the motion estimation algorithms discussed in the previous section, in motion-compensated interpolation, we assume the uniform translational motion model within a local spatiotemporal region. From f(n1,n2,t-1) and f(n1,n2,t0) we compute the velocities at f(n1,n2,t). We then project the velocities to the frame at t-1 or t0 closer in time to the desired time t, as shown in Figure 3.46. Since the projected spatial point generally does not lie on the original sampling grid, spatial interpolation is needed to obtain the interpolated frame. If the velocity estimated at a particular pixel in f(n1,n2,t-1) is not considered accurate enough, the velocity is assumed to be zero, in this case, the interpolated pixel value is identical to that at the same pixel location in f(n1,n2,t-1) or f(n1,n2,t0), whichever is closer in time to the desired time t.
It is not possible to illustrate the motion rendition characteristics of motion-compensated frame interpolation by using only still pictures. However, we can get a rough idea by looking at a still frame created from two image frames by this method. Figure 3.47 shows a set of four frames: two original frames, shown in Figures 3.47(a) and (d), and the two interpolated frames, shown in Figures 3.47(b) and (c). The interpolated frame in (b) is by motion compensation. The frame in (c) is obtained by simply averaging the two key frames. This frame shows the amount of motion that occured between the two key frames. The four frames have a spatial resolution of 512 x 512 pixels. The interpolated frame corresponds to the time instant midway between the two key original frames. Note that the interpolated image’s quality in this example is essentially the same as that of the two key frames when motion compensation is used. The motion estimation method used here is the spatio-tetftporal constraint method with polynomial interpolation, discussed in Section 3.4.2.
Motion-compensated interpolation has been used in modifying the frame rate. Frame rate modification can be combined with time-scale modification of audio to modify the length of a motion picture or a TV program. Experience with typical scenes indicates that frame rate modification of video through motion-compensated interpolation can produce video of quality comparable to the original, except for somewhat unnatural motion rates for such actions as walking and talking which occur when the rate modification factor is sufficiently high.