Issue
I am trying to make a video player plugin work for the video_player package on flutter. This is all on an embedded target (iMX8M Mini). This plugin uses gstreamer to function. However, while gstreamer pipelines on the terminal work as intended, through the plugin it hangs at the first frame for video playback.
You can find the video player code for the plugin here: https://github.com/mmerah/flutter-elinux-plugins-imx8/blob/main/packages/video_player/elinux/gst_video_player.cc
(This is just a fork of https://github.com/sony/flutter-elinux-plugins)
Some snippets from the code:
GstVideoPlayer::GstVideoPlayer(
const std::string& uri, std::unique_ptr<VideoPlayerStreamHandler> handler)
: stream_handler_(std::move(handler)) {
gst_.pipeline = nullptr;
gst_.playbin = nullptr;
gst_.video_convert = nullptr;
gst_.video_sink = nullptr;
gst_.output = nullptr;
gst_.bus = nullptr;
gst_.buffer = nullptr;
uri_ = ParseUri(uri);
if (!CreatePipeline()) {
std::cerr << "Failed to create a pipeline" << std::endl;
DestroyPipeline();
return;
}
// Prerolls before getting information from the pipeline.
Preroll();
// Sets internal video size and buffier.
GetVideoSize(width_, height_);
pixels_.reset(new uint32_t[width_ * height_]);
stream_handler_->OnNotifyInitialized();
}
bool GstVideoPlayer::Play() {
if (gst_element_set_state(gst_.pipeline, GST_STATE_PLAYING) ==
GST_STATE_CHANGE_FAILURE) {
std::cerr << "Failed to change the state to PLAYING" << std::endl;
return false;
}
return true;
}
bool GstVideoPlayer::Pause() {
if (gst_element_set_state(gst_.pipeline, GST_STATE_PAUSED) ==
GST_STATE_CHANGE_FAILURE) {
std::cerr << "Failed to change the state to PAUSED" << std::endl;
return false;
}
return true;
}
bool GstVideoPlayer::Stop() {
if (gst_element_set_state(gst_.pipeline, GST_STATE_READY) ==
GST_STATE_CHANGE_FAILURE) {
std::cerr << "Failed to change the state to READY" << std::endl;
return false;
}
return true;
}
// (...)
bool GstVideoPlayer::CreatePipeline() {
gst_.pipeline = gst_pipeline_new("pipeline");
if (!gst_.pipeline) {
std::cerr << "Failed to create a pipeline" << std::endl;
return false;
}
gst_.playbin = gst_element_factory_make("playbin", "playbin");
if (!gst_.playbin) {
std::cerr << "Failed to create a source" << std::endl;
return false;
}
gst_.video_convert = gst_element_factory_make("videoconvert", "videoconvert");
if (!gst_.video_convert) {
std::cerr << "Failed to create a videoconvert" << std::endl;
return false;
}
gst_.video_sink = gst_element_factory_make("fakesink", "videosink");
if (!gst_.video_sink) {
std::cerr << "Failed to create a videosink" << std::endl;
return false;
}
gst_.output = gst_bin_new("output");
if (!gst_.output) {
std::cerr << "Failed to create an output" << std::endl;
return false;
}
gst_.bus = gst_pipeline_get_bus(GST_PIPELINE(gst_.pipeline));
if (!gst_.bus) {
std::cerr << "Failed to create a bus" << std::endl;
return false;
}
gst_bus_set_sync_handler(gst_.bus, HandleGstMessage, this, NULL);
// Sets properties to fakesink to get the callback of a decoded frame.
g_object_set(G_OBJECT(gst_.video_sink), "sync", FALSE, "qos", FALSE, NULL);
g_object_set(G_OBJECT(gst_.video_sink), "signal-handoffs", TRUE, NULL);
g_signal_connect(G_OBJECT(gst_.video_sink), "handoff",
G_CALLBACK(HandoffHandler), this);
gst_bin_add_many(GST_BIN(gst_.output), gst_.video_convert, gst_.video_sink,
NULL);
// Adds caps to the converter to convert the color format to RGBA.
auto* caps = gst_caps_from_string("video/x-raw");
auto link_ok =
gst_element_link_filtered(gst_.video_convert, gst_.video_sink, caps);
gst_caps_unref(caps);
if (!link_ok) {
std::cerr << "Failed to link elements" << std::endl;
return false;
}
auto* sinkpad = gst_element_get_static_pad(gst_.video_convert, "sink");
auto* ghost_sinkpad = gst_ghost_pad_new("sink", sinkpad);
gst_pad_set_active(ghost_sinkpad, TRUE);
gst_element_add_pad(gst_.output, ghost_sinkpad);
// Sets properties to playbin.
g_object_set(gst_.playbin, "uri", uri_.c_str(), NULL);
g_object_set(gst_.playbin, "video-sink", gst_.output, NULL);
gst_bin_add_many(GST_BIN(gst_.pipeline), gst_.playbin, NULL);
return true;
}
void GstVideoPlayer::Preroll() {
if (!gst_.playbin) {
return;
}
auto result = gst_element_set_state(gst_.pipeline, GST_STATE_PAUSED);
if (result == GST_STATE_CHANGE_FAILURE) {
std::cerr << "Failed to change the state to PAUSED" << std::endl;
return;
}
// Waits until the state becomes GST_STATE_PAUSED.
if (result == GST_STATE_CHANGE_ASYNC) {
GstState state;
result =
gst_element_get_state(gst_.pipeline, &state, NULL, GST_CLOCK_TIME_NONE);
if (result == GST_STATE_CHANGE_FAILURE) {
std::cerr << "Failed to get the current state" << std::endl;
}
}
}
What I believe is the equivalent pipeline: gst-launch-1.0 playbin uri=https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_cropped_multilingual.webm video-sink="videoconvert ! video/x-raw ! fakesink sync=FALSE. This works when called from the terminal. Same when using autovideosink instead of fakesink with the terminal, the video is correctly streamed on the display on target.
As an aside:
- with sync=TRUE, the video hangs on the first frame when the pipeline is called from the terminal.
- with format=RGBA, same thing: the video hangs on the first frame when the pipeline is called from the terminal.
When using flutter, the video seems correctly buffered (first frame appears) but when I click play nothing other than that first frame appears. It seems to hang there (but I did put sync=FALSE and removed format=RGBA in my fork of the plugin):
root@verdin-imx8mm-06898530:/# GST_DEBUG=3 flutter-client -f -n -b /usr/share/flutter/demo_flutter_app
====== AIUR: 4.7.1 build on Jul 19 2022 06:15:03. ======
Core: MKVPARSER_01.08.17 build on Apr 26 2022 07:37:32
file: /usr/lib/imx-mm/parser/lib_mkv_parser_arm_elinux.so.3.1
0:00:03.282695211 868 0xffff600029e0 WARN aiurdemux aiurdemux.c:752:gst_aiurdemux_handle_sink_event: need to drop sink event GST_EVENT_CUSTOM_DOWNSTREAM_STICKY
0:00:03.830029344 868 0xffff600029e0 WARN aiurdemux aiurdemux.c:752:gst_aiurdemux_handle_sink_event: need to drop sink event GST_EVENT_CUSTOM_DOWNSTREAM_STICKY
0:00:04.340769125 868 0xffff600029e0 WARN aiurdemux aiurdemux.c:752:gst_aiurdemux_handle_sink_event: need to drop sink event GST_EVENT_CUSTOM_DOWNSTREAM_STICKY
------------------------
Track 00 [audio_0] Enabled
Duration: 0:00:47.791668000
Language: por
Mime:
audio/x-vorbis, channels=(int)2, rate=(int)44100, bitrate=(int)0, framed=(boolean)true
------------------------
0:00:06.614267007 868 0xffff54298760 WARN pulse pulsesink.c:617:gst_pulseringbuffer_open_device:<pulsesink0> error: Failed to connect: Connection refused
0:00:06.614774627 868 0xffff54298760 WARN playbin gstplaybin2.c:4775:autoplug_select_cb:<playbin> Could not activate sink pulsesink
0:00:06.623663411 868 0xffff54298760 WARN alsa pcm_hw.c:1716:snd_pcm_hw_open: alsalib error: open '/dev/snd/pcmC0D0p' failed (-77): File descriptor in bad state
====== BEEP: 4.7.1 build on Jul 19 2022 06:15:03. ======
Core: OggVorbis decoder Wrapper build on Dec 7 2017 18:15:03
file: /usr/lib/imx-mm/audio-codec/wrap/lib_vorbisd_wrap_arm_elinux.so.3
CODEC: OGGVORBISD_ARM_02.04.00_ARMV8 build on Mar 11 2019 19:56:02.
------------------------
Track 01 [audio_1] Enabled
Duration: 0:00:47.791668000
Language: eng
Mime:
audio/x-vorbis, channels=(int)2, rate=(int)48000, bitrate=(int)0, framed=(boolean)true
------------------------
====== BEEP: 4.7.1 build on Jul 19 2022 06:15:03. ======
Core: OggVorbis decoder Wrapper build on Dec 7 2017 18:15:03
file: /usr/lib/imx-mm/audio-codec/wrap/lib_vorbisd_wrap_arm_elinux.so.3
CODEC: OGGVORBISD_ARM_02.04.00_ARMV8 build on Mar 11 2019 19:56:02.
------------------------
Track 02 [video_0] Enabled
Duration: 0:00:47.791668000
Language: eng
Mime:
video/x-vp8, width=(int)1024, height=(int)436, framerate=(fraction)30/1
------------------------
====== V4L2DEC: 1.20.0 build on Jul 28 2022 06:57:16. ======
0:00:06.719066809 868 0xffff54298760 WARN v4l2 gstv4l2object.c:4638:gst_v4l2_object_probe_caps:<v4l2vp8dec0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
------------------------
Track 03 [audio_2] Enabled
Duration: 0:00:47.791668000
Language: spa
Mime:
audio/x-vorbis, channels=(int)2, rate=(int)44100, bitrate=(int)0, framed=(boolean)true
------------------------
====== BEEP: 4.7.1 build on Jul 19 2022 06:15:03. ======
Core: OggVorbis decoder Wrapper build on Dec 7 2017 18:15:03
file: /usr/lib/imx-mm/audio-codec/wrap/lib_vorbisd_wrap_arm_elinux.so.3
CODEC: OGGVORBISD_ARM_02.04.00_ARMV8 build on Mar 11 2019 19:56:02.
0:00:07.069225601 868 0xffff600029e0 WARN aiurdemux aiurdemux.c:752:gst_aiurdemux_handle_sink_event: need to drop sink event GST_EVENT_CUSTOM_DOWNSTREAM_STICKY
0:00:07.070779210 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3411:aiurdemux_send_stream_newsegment: Pad audio: Send newseg 0:00:00.000000000 first buffer 0:00:00.000000000
0:00:07.071953948 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3411:aiurdemux_send_stream_newsegment: Pad audio: Send newseg 0:00:00.000000000 first buffer 0:00:00.000000000
0:00:07.073762929 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3411:aiurdemux_send_stream_newsegment: Pad video: Send newseg 0:00:00.000000000 first buffer 0:00:00.000000000
0:00:07.079756118 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3411:aiurdemux_send_stream_newsegment: Pad audio: Send newseg 0:00:00.000000000 first buffer 0:00:00.000000000
0:00:07.080847982 868 0xffff48057700 WARN v4l2 gstv4l2object.c:2352:gst_v4l2_object_get_streamparm:<v4l2vp8dec0:src> VIDIOC_G_PARM failed
0:00:07.082493465 868 0xffff48057700 WARN v4l2 gstv4l2object.c:4638:gst_v4l2_object_probe_caps:<v4l2vp8dec0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
0:00:07.084351571 868 0xffff48057700 WARN v4l2 gstv4l2object.c:3382:gst_v4l2_object_save_format:<v4l2vp8dec0:src> Driver bug detected, stride (512) is too small for the width (1024)
0:00:07.093393103 868 0xffff48057700 FIXME playbin gstplaybin2.c:3308:_uridecodebin_event_probe:<uridecodebin0:src_0> Consider implementing group-id handling on stream-start event
0:00:07.093672601 868 0xffff48057c60 FIXME playbin gstplaybin2.c:3308:_uridecodebin_event_probe:<uridecodebin0:src_3> Consider implementing group-id handling on stream-start event
0:00:07.094126221 868 0xffff480576a0 FIXME playbin gstplaybin2.c:3308:_uridecodebin_event_probe:<uridecodebin0:src_2> Consider implementing group-id handling on stream-start event
0:00:07.094386343 868 0xffff60019640 FIXME playbin gstplaybin2.c:3308:_uridecodebin_event_probe:<uridecodebin0:src_1> Consider implementing group-id handling on stream-start event
0:00:07.125438650 868 0xffff5001b400 WARN audio-resampler audio-resampler.c:274:convert_taps_gint16_c: can't find exact taps
0:00:07.126845636 868 0xffff5001b400 WARN alsa pcm_hw.c:1360:snd_pcm_hw_get_chmap: alsalib error: Cannot read Channel Map ctl
: No such file or directory
0:00:07.130274101 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
(...)
0:00:07.156427583 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
0:00:07.156507082 868 0xffff48057700 WARN videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.156604956 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
0:00:07.156630956 868 0xffff48057700 WARN videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.156701455 868 0xffff48057700 WARN videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.156774830 868 0xffff48057700 WARN videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.156879954 868 0xffff48057700 WARN videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.158322064 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
(...)
0:00:07.165199243 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
0:00:07.168740582 868 0xffff30010ea0 WARN v4l2videodec gstv4l2videodec.c:785:gst_v4l2_video_dec_loop:<v4l2vp8dec0> Received resolution change
0:00:07.169758322 868 0xffff30010ea0 WARN v4l2 gstv4l2object.c:2352:gst_v4l2_object_get_streamparm:<v4l2vp8dec0:src> VIDIOC_G_PARM failed
0:00:07.170134818 868 0xffff30010ea0 WARN v4l2 gstv4l2object.c:4638:gst_v4l2_object_probe_caps:<v4l2vp8dec0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
0:00:07.170767437 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
(...)
0:00:09.395853468 868 0xffff54298760 WARN aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
The flutter app is here: https://github.com/mmerah/flutter-elinux-plugins-imx8/blob/main/packages/video_player/example/lib/main.dart
With GST_DEBUG=4, the last few log after clicking play show this:
0:00:23.661699891 912 0xaaab07b22000 INFO GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline> child 'playbin' changed state to 4(PLAYING) successfully
0:00:23.661749015 912 0xaaab07b22000 INFO GST_STATES gstelement.c:2806:gst_element_continue_state:<pipeline> completed state change to PLAYING
0:00:23.661793639 912 0xaaab07b22000 INFO GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<pipeline> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:23.664287226 912 0xffff5c298b60 WARN aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
(...)
0:00:23.668285040 912 0xffff5c298b60 WARN aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
0:00:23.669715393 912 0xaaab07b22000 INFO GST_EVENT gstevent.c:1363:gst_event_new_seek: creating seek rate 1.000000, format TIME, flags 1, start_type 1, start 0:00:00.000000000, stop_type 1, stop 99:99:99.999999999
0:00:36.603878098 912 0xaaab07db2000 INFO basesrc gstbasesrc.c:2913:gst_base_src_loop:<source> pausing after gst_base_src_get_range() = eos
0:00:36.604063346 912 0xaaab07db2000 INFO task gsttask.c:368:gst_task_func:<source:src> Task going to paused
0:00:36.604394091 912 0xffff680029e0 INFO task gsttask.c:368:gst_task_func:<queue2-0:src> Task going to paused
pastebin for the complete log: https://pastebin.com/wzsbeC0e
What is happening here?
Thank you in advance for your help.
Solution
A couple things were wrong:
0:00:06.614267007 868 0xffff54298760 WARN pulse pulsesink.c:617:gst_pulseringbuffer_open_device:<pulsesink0> error: Failed to connect: Connection refusedthis blocked the pipeline when SYNC=TRUE. Withpulseaudio --startthis is solved.- The
flutter-clientembedder used this repohttps://github.com/sony/flutter-embedded-linuxto be built along with a video player plugin. I was trying to add the plugin code to the flutter app and thought it would be bundled together but that is wrong. The flutter app uses no plugin, it isflutter-clientthat uses it. This did not impact anything but when I thought I had set SYNC=FALSE in the plugin I actually did not (once again because that’s flutter-client that has it).
Answered By – mmnano50
Answer Checked By – Senaida (FlutterFixes Volunteer)