The MEMAD Model
The motion enhancement by moving average differencing (MEMAD) model used here is described as follows. The camera stream is described by RGB images of size M × N at sample time t. The motion amplification filter maps each pixel intensity value Ik (k = R, G, B) at position (x = 1...M, y = 1…N) via
\({I}_{k}\left(x,y,t\right)\to {\beta }{I}_{k}\left(x,y,t\right)+{\alpha }\left({I}_{k}\left(x,y,t\right)-\frac{1}{m}{\sum }_{i=1}^{m}{I}_{k}\left(x,y,t-i\right)\right) .\) | (1) |
This filter compares the present intensity values to the moving average of immediate prior values, and is thus a high-pass filter sensitive to changes in intensity values. The high-pass values are amplified by a factor α and then added to the present intensity, which can be dimmed by a factor β in order to make the amplified differences more stand out against a darker background image. This filter has to be understood modulo the number representation of pixel intensities in specific hardware applications; in the present application, RGB values are represented by 8-bit unsigned integers and thus have a range of 0 to 255. In all applications described here, mapped intensities exceeding this range are set to 0 or 255, for the case of underflow and overflow, respectively.
The motivation for the model Eq. (1) is computational simplicity, allowing for real-time computation on contemporary smartphones. More advanced motion detection algorithms have been proven difficult to implement on smartphones in real-time due to their higher computational demand.
Implementation Details
Figure 1 shows the SIMULINK flow diagram of the MEMAD model for Android mobile device implementations. The file itself is provided as a data file (see section Description of data) but the here provided description should contain all information needed to reproduce the application, too. The camera stream from the Android device consists of three separate channels for RGB encoding. These RGB streams are each delayed by up to m = 5 sampling time steps and then fed into the MATLAB function MEMAD, which performs the actual filtering given by Eq. (1). The filter result is then streamed to the device’s display in real time.
In all applications shown, the parameters of Eq. (1) were m = 5, α = 16 or 32, β = 0.5 or 1.0, as specified.
Camera block
The resolution of the Android camera block was set to 240 × 320 pixels, and its sampling rate to 30 fps. Depending on the chosen camera resolution, the MEMAD function might not be fast enough to process the images in real time within this sampling time. Therefore, it was taken advantage of the SIMULINK feature that sampling rates can be backwards adjusted. In other words, the output rate on the device’s display can appear to be lower than 30 fps. This was observed in the present applications with the given image matrix size.
Delay blocks
At the beginning of the stream, the initial values of the delay blocks are set to zero, and the input processing of the delay blocks is “Elements as channels” to define sample-based processing.
MATLAB function block
The MEMAD block content is
function [Rout,Gout,Bout] = MEMAD(R,G,B,Rbuff1,Gbuff1,Bbuff1,Rbuff2,Gbuff2,Bbuff2, Rbuff3,Gbuff3,Bbuff3,
Rbuff4,Gbuff4,Bbuff4, Rbuff5,Gbuff5,Bbuff5)
alpha = 16; beta = 1;
Rout = beta*R + alpha*( R-Rbuff1/5-Rbuff2/5-Rbuff3/5-Rbuff4/5-Rbuff5/5 );
Gout = beta*G + alpha*( G-Gbuff1/5-Gbuff2/5-Gbuff3/5-Gbuff4/5-Gbuff5/5 );
Bout = beta*B + alpha*( B-Bbuff1/5-Bbuff2/5-Bbuff3/5-Bbuff4/5-Bbuff5/5 );
Hardware and software versions
The SIMULINK version was 10.2 (R2020b). SIMULINK was used together with Android Studio Chipmunk 2021.2. with Android SDK Build-Tools 33-rc4 and Android SDK Platform 30 for Android 11.0 (R).
The device used for testing the filter was a Motorola Moto G Stylus, hardware version pvt, RAM size 4 GB, ROM size 128 GB, display size 2300 × 1080, front camera 16 MP, and rear camera 48 MP. The software version is Android 11, kernel version 4.14.180-perf+ #1 Sun Mar 27 09:24:05 EDT 2022, build number RPRS31.Q1-56-9-15.
Recording
For recording MEMAD videos the smartphone was mounted on a tripod. The phone’s screen was captured in real time with the XRecorder app (InShot Inc., update 4/28/2022). Unfiltered videos for comparison were recorded with the phone’s in-built camera app. Recorded files were transferred to a PC for inspection and selection of sections for the video files.
Example – Falling Rain
The MEMAD algorithm was tested in real time on an Android smartphone. The purpose was to amplify small moving objects. The amplification of subtle motion in larger objects will be shown in the Applications section below. Video file rain_a.mp4 (see data files) is an unfiltered capture of a light rainfall. The motion amplified MEMAD results are shown in video files rain_b.mp4 and rain_c.mp4. Video file rain_b.mp4 has amplification α = 32 and dimming factor β = 1, and was recorded right after rain_a.mp4. Video file rain_c.mp4 has the same amplification but β = 0.5. As expected, falling raindrops are amplified in real time by the MEMAD model Eq. (1), and the case of β < 1 relatively amplifies movement stronger at the cost of still image component quality. Snapshots of the videos are shown in Fig. 2.