Issue
I need to put overlay images to a video. It is working on Android without problem.
But on iOS platform, if I try 23-24 overlay images, it is working correctly.
If I try it with 30+ images, it gives memory allocation error.
Error while filtering: Cannot allocate memory
Failed to inject frame into filter network: Cannot allocate memory
Every overlay image is around 50 kb
Video is around 250 MB
I tried with smaller images, so I can use 40+ images without problem, so it is not related with counts, it is related with file size.
I think there is a limit like 1MB for complex filter streams.
I tried lots of thinks but no luck.. I have two questions:
- Is my ffmpeg command correct?
- Can you suggest me any improvements, alternatives?
Update: What am I trying to do?
I’m trying to make burned subtitled video. But I also need to support emoji too. So I figured out it like these steps:
- Create all subtitle items as .png images.
- Overlay these images to video with correct timing.
FFmpeg Command:
-i video.mov -i image1.png -i image1.png -i ... image43.png
-filter_complex "[0][1]overlay=x=0:y=0:enable='between(t,10.9,16.4)'[out];[out][2]overlay=x=0:y=0:enable='between(t,16.8,20.0)'[out];[out][3]overlay=x=0:y=0:enable='between(t,20.0,22.4)'[out];[out][4]overlay=x=0:y=0:enable='between(t,22.4,26.5)'[out];[out][5]overlay=x=0:y=0:enable='between(t,26.5,29.8)'[out];[out][6]overlay=x=0:y=0:enable='between(t,30.0,33.7)'[out];[out][7]overlay=x=0:y=0:enable='between(t,33.7,38.4)'[out];[out][8]overlay=x=0:y=0:enable='between(t,38.6,41.5)'[out];[out][9]overlay=x=0:y=0:enable='between(t,41.5,44.2)'[out];[out][10]overlay=x=0:y=0:enable='between(t,44.3,46.5)'[out];[out][11]overlay=x=0:y=0:enable='between(t,47.2,48.4)'[out];[out][12]overlay=x=0:y=0:enable='between(t,49.9,52.6)'[out];[out][13]overlay=x=0:y=0:enable='between(t,52.6,54.2)'[out];[out][14]overlay=x=0:y=0:enable='between(t,54.2,57.0)'[out];[out][15]overlay=x=0:y=0:enable='between(t,57.0,57.6)'[out];[out][16]overlay=x=0:y=0:enable='between(t,63.0,65.6)'[out];[out][17]overlay=x=0:y=0:enable='between(t,66.5,71.5)'[out];[out][18]overlay=x=0:y=0:enable='between(t,72.2,73.7)'[out];[out][19]overlay=x=0:y=0:enable='between(t,74.4,75.7)'[out];[out][20]overlay=x=0:y=0:enable='between(t,77.9,78.7)'[out];[out][21]overlay=x=0:y=0:enable='between(t,83.5,85.4)'[out];[out][22]overlay=x=0:y=0:enable='between(t,87.3,91.0)'[out];[out][23]overlay=x=0:y=0:enable='between(t,91.0,96.1)'[out];[out][24]overlay=x=0:y=0:enable='between(t,96.1,100.8)'[out];[out][25]overlay=x=0:y=0:enable='between(t,100.8,103.5)'[out];[out][26]overlay=x=0:y=0:enable='between(t,103.5,106.9)'[out];[out][27]overlay=x=0:y=0:enable='between(t,106.9,109.1)'[out];[out][28]overlay=x=0:y=0:enable='between(t,109.2,110.6)'[out];[out][29]overlay=x=0:y=0:enable='between(t,110.6,113.7)'[out];[out][30]overlay=x=0:y=0:enable='between(t,114.8,117.6)'[out];[out][31]overlay=x=0:y=0:enable='between(t,117.6,119.8)'[out];[out][32]overlay=x=0:y=0:enable='between(t,120.5,122.8)'[out];[out][33]overlay=x=0:y=0:enable='between(t,123.5,125.0)'[out];[out][34]overlay=x=0:y=0:enable='between(t,128.3,129.1)'[out];[out][35]overlay=x=0:y=0:enable='between(t,129.7,130.5)'[out];[out][36]overlay=x=0:y=0:enable='between(t,131.4,136.3)'[out];[out][37]overlay=x=0:y=0:enable='between(t,137.1,138.6)'[out];[out][38]overlay=x=0:y=0:enable='between(t,139.3,139.9)'[out];[out][39]overlay=x=0:y=0:enable='between(t,143.2,143.6)'[out];[out][40]overlay=x=0:y=0:enable='between(t,148.8,152.2)'[out];[out][41]overlay=x=0:y=0:enable='between(t,152.2,155.3)'[out];[out][42]overlay=x=0:y=0:enable='between(t,155.3,158.2)'[out];[out][43]overlay=x=0:y=0:enable='between(t,158.2,159.2)'[out]"
-map [out] -map 0:a? -vsync vfr -vcodec libx264 -x264-params crf=23 -preset superfast -acodec aac output.mp4
Logs:
ffmpeg version v4.5-dev-3393-g30322ebe3c Copyright (c) 2000-2021 the FFmpeg developers
built with Apple clang version 13.0.0 (clang-1300.0.29.30)
configuration: --cross-prefix=arm64-ios-darwin- --sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.2.sdk --prefix=/Users/taner/Projects/ffmpeg-kit/prebuilt/apple-ios-arm64/ffmpeg --pkg-config=/opt/homebrew/bin/pkg-config --enable-version3 --arch=aarch64 --cpu=armv8 --target-os=darwin --enable-neon --enable-asm --ar=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar --cc=clang --cxx=clang++ --as='/Users/taner/Projects/ffmpeg-kit/.tmp/gas-preprocessor.pl -arch aarch64 -- clang -arch arm64 -target arm64-apple-ios12.1 -march=armv8-a+crc+crypto -mcpu=generic -DFFMPEG_KIT_ARM64 -Wno-unused-function -Wno-deprecated-declarations -fstrict-aliasing -fembed-bitcode -DIOS -DFFMPEG_KIT_BUILD_DATE=20220114 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.2.sdk -Oz -miphoneos-version-min=12.1 -I/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS15.2.sdk/usr/include' --ranlib=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib --strip=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip --nm=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/nm --extra-ldflags='-miphoneos-version-min=12.1' --disable-autodetect --enable-cross-compile --enable-pic --enable-inline-asm --enable-optimizations --enable-swscale --enable-shared --disable-static --install-name-dir='@rpath' --enable-pthreads --disable-v4l2-m2m --disable-outdev=v4l2 --disable-outdev=fbdev --disable-indev=v4l2 --disable-indev=fbdev --enable-small --disable-xmm-clobber-test --disable-debug --disable-neon-clobber-test --disable-programs --disable-postproc --disable-doc --disable-htmlpages --disable-manpages --disable-podpages --disable-txtpages --disable-sndio --disable-schannel --disable-securetransport --disable-xlib --disable-cuda --disable-cuvid --disable-nvenc --disable-vaapi --disable-vdpau --disable-alsa --disable-cuda --disable-cuvid --disable-nvenc --disable-vaapi --disable-vdpau --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-gmp --enable-gnutls --enable-libmp3lame --enable-libass --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxml2 --enable-libopencore-amrnb --enable-libshine --enable-libspeex --enable-libdav1d --enable-libkvazaar --enable-libx264 --enable-libxvid --enable-libx265 --enable-libvidstab --enable-libilbc --enable-libopus --enable-libsnappy --enable-libsoxr --enable-libtwolame --disable-sdl2 --enable-libvo-amrwbenc --enable-libzimg --disable-openssl --enable-zlib --enable-audiotoolbox --disable-outdev=audiotoolbox --enable-bzlib --enable-videotoolbox --enable-avfoundation --enable-iconv --disable-coreimage --disable-appkit --disable-opencl --disable-opengl --enable-gpl
libavutil 57. 13.100 / 57. 13.100
libavcodec 59. 15.102 / 59. 15.102
libavformat 59. 10.100 / 59. 10.100
libavdevice 59. 1.100 / 59. 1.100
libavfilter 8. 21.100 / 8. 21.100
libswscale 6. 1.102 / 6. 1.102
libswresample 4. 0.100 / 4. 0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'video.mov':
Metadata:
major_brand : qt
minor_version : 0
compatible_brands: qt
creation_time : 2022-03-19T16:24:25.000000Z
encoder : Lavf58.20.100
Duration: 00:02:47.77, start: 0.000000, bitrate: 12431 kb/s
Stream #0:00x1: Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 248 kb/s (default)
Metadata:
creation_time : 2022-03-19T16:24:25.000000Z
handler_name : Core Media Audio
vendor_id : [0][0][0][0]
Stream #0:10x2: Video: h264 (avc1 / 0x31637661), yuv420p(progressive), 1080x1920 [SAR 1:1 DAR 9:16], 12174 kb/s, 29.99 fps, 30 tbr, 90k tbn (default)
Metadata:
creation_time : 2022-03-19T16:24:25.000000Z
handler_name : Core Media Video
vendor_id : FFMP
Input #1, png_pipe, from 'image1.png':
Duration: N/A, bitrate: N/A
Stream #1:0: Video: png, rgba(pc), 1080x1920, 25 fps, 25 tbr, 25 tbn
Input #2, png_pipe, from 'image2.png':
Duration: N/A, bitrate: N/A
Stream #2:0: Video: png, rgba(pc), 1080x1920, 25 fps, 25 tbr, 25 tbn
.
.
Input #43, png_pipe, from 'image43.png':
Duration: N/A, bitrate: N/A
Stream #43:0: Video: png, rgba(pc), 1080x1920, 25 fps, 25 tbr, 25 tbn
[aac @ 0x1047f4480] The "sub_text_format" option is deprecated: Deprecated, does nothing
[h264 @ 0x15f2a70f0] The "sub_text_format" option is deprecated: Deprecated, does nothing
[png @ 0x15f47f710] The "sub_text_format" option is deprecated: Deprecated, does nothing
[png @ 0x1059da740] The "sub_text_format" option is deprecated: Deprecated, does nothing
.
.
[png @ 0x15f2a8780] The "sub_text_format" option is deprecated: Deprecated, does nothing
Stream mapping:
Stream #0:1 (h264) -> scale (graph 0)
Stream #1:0 (png) -> overlay:overlay (graph 0)
Stream #2:0 (png) -> overlay:overlay (graph 0)
Stream #3:0 (png) -> overlay:overlay (graph 0)
Stream #4:0 (png) -> overlay:overlay (graph 0)
Stream #5:0 (png) -> overlay:overlay (graph 0)
Stream #6:0 (png) -> overlay:overlay (graph 0)
Stream #7:0 (png) -> overlay:overlay (graph 0)
Stream #8:0 (png) -> overlay:overlay (graph 0)
Stream #9:0 (png) -> overlay:overlay (graph 0)
Stream #10:0 (png) -> overlay:overlay (graph 0)
Stream #11:0 (png) -> overlay:overlay (graph 0)
Stream #12:0 (png) -> overlay:overlay (graph 0)
Stream #13:0 (png) -> overlay:overlay (graph 0)
Stream #14:0 (png) -> overlay:overlay (graph 0)
Stream #15:0 (png) -> overlay:overlay (graph 0)
Stream #16:0 (png) -> overlay:overlay (graph 0)
Stream #17:0 (png) -> overlay:overlay (graph 0)
Stream #18:0 (png) -> overlay:overlay (graph 0)
Stream #19:0 (png) -> overlay:overlay (graph 0)
Stream #20:0 (png) -> overlay:overlay (graph 0)
Stream #21:0 (png) -> overlay:overlay (graph 0)
Stream #22:0 (png) -> overlay:overlay (graph 0)
Stream #23:0 (png) -> overlay:overlay (graph 0)
Stream #24:0 (png) -> overlay:overlay (graph 0)
Stream #25:0 (png) -> overlay:overlay (graph 0)
Stream #26:0 (png) -> overlay:overlay (graph 0)
Stream #27:0 (png) -> overlay:overlay (graph 0)
Stream #28:0 (png) -> overlay:overlay (graph 0)
Stream #29:0 (png) -> overlay:overlay (graph 0)
Stream #30:0 (png) -> overlay:overlay (graph 0)
Stream #31:0 (png) -> overlay:overlay (graph 0)
Stream #32:0 (png) -> overlay:overlay (graph 0)
Stream #33:0 (png) -> overlay:overlay (graph 0)
Stream #34:0 (png) -> overlay:overlay (graph 0)
Stream #35:0 (png) -> overlay:overlay (graph 0)
Stream #36:0 (png) -> overlay:overlay (graph 0)
Stream #37:0 (png) -> overlay:overlay (graph 0)
Stream #38:0 (png) -> overlay:overlay (graph 0)
Stream #39:0 (png) -> overlay:overlay (graph 0)
Stream #40:0 (png) -> overlay:overlay (graph 0)
Stream #41:0 (png) -> overlay:overlay (graph 0)
Stream #42:0 (png) -> overlay:overlay (graph 0)
Stream #43:0 (png) -> overlay:overlay (graph 0)
overlay (graph 0) -> Stream #0:0 (libx264)
Stream #0:0 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
Error while filtering: Cannot allocate memory
Failed to inject frame into filter network: Cannot allocate memory
Error while processing the decoded data for stream #43:0
[aac @ 0x15f23b480] Qavg: 12511.496
[aac @ 0x15f23b480] 2 frames left in the queue on closing
Conversion failed!
Solution
What you are experiencing is the nature of large filtergraphs. Every link between filters requires a frame buffer (at least 6 MB) and filtering operation itself may require additional memory space. So, it must use up your iDevice’s memory (which must be smaller than the Androids).
So, the solution must be the one which minimizes the number of filters, and you can do that by using the concat demuxer so all your images originates from one (virtual) source, and use overlay with more complex enable option.
png_list.txt
ffconcat version 1.0
# starts at 0 and ends sometime between 16.4 and 16.8 seconds (picked 16.5)
file image1.png
duration 16.5
# starts at 16.5 and ends at 20
file image2.png
duration 3.5
# starts at 20.0 and ends at 22.4
file image3.png
duration 2.4
...
- Images will be concatenated w/out any pause, so the sum of durations must be least as long as the end time of the last overlay image, which is 159.2 s
- Make sure to pick the durations correctly so each images’ actual start and end times are contained within the total display duration
FFmpeg call
ffmpeg -i video.mov -f concat -i png_list.txt \
-filter_complex "[0][1]overlay=enable=\
'between(t,10.9,16.4) +
between(t,10.9,16.4) +
between(t,16.8,20.0) + ... +
between(t,158.2,159.2)'[out]"
-map [out] -map 0:a? ... output.mp4
I cannot guarantee you this will evade the problem, but this should be a lot easier on memory.
Burning subtitles
You probably know this already and there must be a reason for bitmap-base subtitles that you are using, but there are subtitles and ass filters to burn subtitle texts onto the video if you have subtitles in a text format.
Answered By – kesh
Answer Checked By – Candace Johnson (FlutterFixes Volunteer)