Gstreamer plugin for embedded flutter freeze at first frame

Issue

I am trying to make a video player plugin work for the video_player package on flutter. This is all on an embedded target (iMX8M Mini). This plugin uses gstreamer to function. However, while gstreamer pipelines on the terminal work as intended, through the plugin it hangs at the first frame for video playback.

You can find the video player code for the plugin here: https://github.com/mmerah/flutter-elinux-plugins-imx8/blob/main/packages/video_player/elinux/gst_video_player.cc
(This is just a fork of https://github.com/sony/flutter-elinux-plugins)

Some snippets from the code:

GstVideoPlayer::GstVideoPlayer(
    const std::string& uri, std::unique_ptr<VideoPlayerStreamHandler> handler)
    : stream_handler_(std::move(handler)) {
  gst_.pipeline = nullptr;
  gst_.playbin = nullptr;
  gst_.video_convert = nullptr;
  gst_.video_sink = nullptr;
  gst_.output = nullptr;
  gst_.bus = nullptr;
  gst_.buffer = nullptr;

  uri_ = ParseUri(uri);
  if (!CreatePipeline()) {
    std::cerr << "Failed to create a pipeline" << std::endl;
    DestroyPipeline();
    return;
  }

  // Prerolls before getting information from the pipeline.
  Preroll();

  // Sets internal video size and buffier.
  GetVideoSize(width_, height_);
  pixels_.reset(new uint32_t[width_ * height_]);

  stream_handler_->OnNotifyInitialized();
}

bool GstVideoPlayer::Play() {
  if (gst_element_set_state(gst_.pipeline, GST_STATE_PLAYING) ==
      GST_STATE_CHANGE_FAILURE) {
    std::cerr << "Failed to change the state to PLAYING" << std::endl;
    return false;
  }
  return true;
}

bool GstVideoPlayer::Pause() {
  if (gst_element_set_state(gst_.pipeline, GST_STATE_PAUSED) ==
      GST_STATE_CHANGE_FAILURE) {
    std::cerr << "Failed to change the state to PAUSED" << std::endl;
    return false;
  }
  return true;
}

bool GstVideoPlayer::Stop() {
  if (gst_element_set_state(gst_.pipeline, GST_STATE_READY) ==
      GST_STATE_CHANGE_FAILURE) {
    std::cerr << "Failed to change the state to READY" << std::endl;
    return false;
  }
  return true;
}

// (...)

bool GstVideoPlayer::CreatePipeline() {
  gst_.pipeline = gst_pipeline_new("pipeline");
  if (!gst_.pipeline) {
    std::cerr << "Failed to create a pipeline" << std::endl;
    return false;
  }
  gst_.playbin = gst_element_factory_make("playbin", "playbin");
  if (!gst_.playbin) {
    std::cerr << "Failed to create a source" << std::endl;
    return false;
  }
  gst_.video_convert = gst_element_factory_make("videoconvert", "videoconvert");
  if (!gst_.video_convert) {
    std::cerr << "Failed to create a videoconvert" << std::endl;
    return false;
  }
  gst_.video_sink = gst_element_factory_make("fakesink", "videosink");
  if (!gst_.video_sink) {
    std::cerr << "Failed to create a videosink" << std::endl;
    return false;
  }
  gst_.output = gst_bin_new("output");
  if (!gst_.output) {
    std::cerr << "Failed to create an output" << std::endl;
    return false;
  }
  gst_.bus = gst_pipeline_get_bus(GST_PIPELINE(gst_.pipeline));
  if (!gst_.bus) {
    std::cerr << "Failed to create a bus" << std::endl;
    return false;
  }
  gst_bus_set_sync_handler(gst_.bus, HandleGstMessage, this, NULL);

  // Sets properties to fakesink to get the callback of a decoded frame.
  g_object_set(G_OBJECT(gst_.video_sink), "sync", FALSE, "qos", FALSE, NULL);
  g_object_set(G_OBJECT(gst_.video_sink), "signal-handoffs", TRUE, NULL);
  g_signal_connect(G_OBJECT(gst_.video_sink), "handoff",
                   G_CALLBACK(HandoffHandler), this);
  gst_bin_add_many(GST_BIN(gst_.output), gst_.video_convert, gst_.video_sink,
                   NULL);

  // Adds caps to the converter to convert the color format to RGBA.
  auto* caps = gst_caps_from_string("video/x-raw");
  auto link_ok =
      gst_element_link_filtered(gst_.video_convert, gst_.video_sink, caps);
  gst_caps_unref(caps);
  if (!link_ok) {
    std::cerr << "Failed to link elements" << std::endl;
    return false;
  }

  auto* sinkpad = gst_element_get_static_pad(gst_.video_convert, "sink");
  auto* ghost_sinkpad = gst_ghost_pad_new("sink", sinkpad);
  gst_pad_set_active(ghost_sinkpad, TRUE);
  gst_element_add_pad(gst_.output, ghost_sinkpad);

  // Sets properties to playbin.
  g_object_set(gst_.playbin, "uri", uri_.c_str(), NULL);
  g_object_set(gst_.playbin, "video-sink", gst_.output, NULL);
  gst_bin_add_many(GST_BIN(gst_.pipeline), gst_.playbin, NULL);

  return true;
}

void GstVideoPlayer::Preroll() {
  if (!gst_.playbin) {
    return;
  }

  auto result = gst_element_set_state(gst_.pipeline, GST_STATE_PAUSED);
  if (result == GST_STATE_CHANGE_FAILURE) {
    std::cerr << "Failed to change the state to PAUSED" << std::endl;
    return;
  }

  // Waits until the state becomes GST_STATE_PAUSED.
  if (result == GST_STATE_CHANGE_ASYNC) {
    GstState state;
    result =
        gst_element_get_state(gst_.pipeline, &state, NULL, GST_CLOCK_TIME_NONE);
    if (result == GST_STATE_CHANGE_FAILURE) {
      std::cerr << "Failed to get the current state" << std::endl;
    }
  }
}

What I believe is the equivalent pipeline: gst-launch-1.0 playbin uri=https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_cropped_multilingual.webm video-sink="videoconvert ! video/x-raw ! fakesink sync=FALSE. This works when called from the terminal. Same when using autovideosink instead of fakesink with the terminal, the video is correctly streamed on the display on target.

As an aside:

  • with sync=TRUE, the video hangs on the first frame when the pipeline is called from the terminal.
  • with format=RGBA, same thing: the video hangs on the first frame when the pipeline is called from the terminal.

When using flutter, the video seems correctly buffered (first frame appears) but when I click play nothing other than that first frame appears. It seems to hang there (but I did put sync=FALSE and removed format=RGBA in my fork of the plugin):

root@verdin-imx8mm-06898530:/# GST_DEBUG=3 flutter-client -f -n -b /usr/share/flutter/demo_flutter_app

====== AIUR: 4.7.1 build on Jul 19 2022 06:15:03. ======
        Core: MKVPARSER_01.08.17  build on Apr 26 2022 07:37:32
 file: /usr/lib/imx-mm/parser/lib_mkv_parser_arm_elinux.so.3.1
0:00:03.282695211   868 0xffff600029e0 WARN               aiurdemux aiurdemux.c:752:gst_aiurdemux_handle_sink_event: need to drop sink event GST_EVENT_CUSTOM_DOWNSTREAM_STICKY
0:00:03.830029344   868 0xffff600029e0 WARN               aiurdemux aiurdemux.c:752:gst_aiurdemux_handle_sink_event: need to drop sink event GST_EVENT_CUSTOM_DOWNSTREAM_STICKY
0:00:04.340769125   868 0xffff600029e0 WARN               aiurdemux aiurdemux.c:752:gst_aiurdemux_handle_sink_event: need to drop sink event GST_EVENT_CUSTOM_DOWNSTREAM_STICKY
------------------------
    Track 00 [audio_0] Enabled
        Duration: 0:00:47.791668000
        Language: por
    Mime:
        audio/x-vorbis, channels=(int)2, rate=(int)44100, bitrate=(int)0, framed=(boolean)true
------------------------
0:00:06.614267007   868 0xffff54298760 WARN                   pulse pulsesink.c:617:gst_pulseringbuffer_open_device:<pulsesink0> error: Failed to connect: Connection refused
0:00:06.614774627   868 0xffff54298760 WARN                 playbin gstplaybin2.c:4775:autoplug_select_cb:<playbin> Could not activate sink pulsesink
0:00:06.623663411   868 0xffff54298760 WARN                    alsa pcm_hw.c:1716:snd_pcm_hw_open: alsalib error: open '/dev/snd/pcmC0D0p' failed (-77): File descriptor in bad state

====== BEEP: 4.7.1 build on Jul 19 2022 06:15:03. ======
        Core: OggVorbis decoder Wrapper  build on Dec  7 2017 18:15:03
 file: /usr/lib/imx-mm/audio-codec/wrap/lib_vorbisd_wrap_arm_elinux.so.3
CODEC: OGGVORBISD_ARM_02.04.00_ARMV8  build on Mar 11 2019 19:56:02.
------------------------
    Track 01 [audio_1] Enabled
        Duration: 0:00:47.791668000
        Language: eng
    Mime:
        audio/x-vorbis, channels=(int)2, rate=(int)48000, bitrate=(int)0, framed=(boolean)true
------------------------

====== BEEP: 4.7.1 build on Jul 19 2022 06:15:03. ======
        Core: OggVorbis decoder Wrapper  build on Dec  7 2017 18:15:03
 file: /usr/lib/imx-mm/audio-codec/wrap/lib_vorbisd_wrap_arm_elinux.so.3
CODEC: OGGVORBISD_ARM_02.04.00_ARMV8  build on Mar 11 2019 19:56:02.
------------------------
    Track 02 [video_0] Enabled
        Duration: 0:00:47.791668000
        Language: eng
    Mime:
        video/x-vp8, width=(int)1024, height=(int)436, framerate=(fraction)30/1
------------------------

====== V4L2DEC: 1.20.0 build on Jul 28 2022 06:57:16. ======
0:00:06.719066809   868 0xffff54298760 WARN                    v4l2 gstv4l2object.c:4638:gst_v4l2_object_probe_caps:<v4l2vp8dec0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
------------------------
    Track 03 [audio_2] Enabled
        Duration: 0:00:47.791668000
        Language: spa
    Mime:
        audio/x-vorbis, channels=(int)2, rate=(int)44100, bitrate=(int)0, framed=(boolean)true
------------------------

====== BEEP: 4.7.1 build on Jul 19 2022 06:15:03. ======
        Core: OggVorbis decoder Wrapper  build on Dec  7 2017 18:15:03
 file: /usr/lib/imx-mm/audio-codec/wrap/lib_vorbisd_wrap_arm_elinux.so.3
CODEC: OGGVORBISD_ARM_02.04.00_ARMV8  build on Mar 11 2019 19:56:02.
0:00:07.069225601   868 0xffff600029e0 WARN               aiurdemux aiurdemux.c:752:gst_aiurdemux_handle_sink_event: need to drop sink event GST_EVENT_CUSTOM_DOWNSTREAM_STICKY
0:00:07.070779210   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3411:aiurdemux_send_stream_newsegment: Pad audio: Send newseg 0:00:00.000000000 first buffer 0:00:00.000000000
0:00:07.071953948   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3411:aiurdemux_send_stream_newsegment: Pad audio: Send newseg 0:00:00.000000000 first buffer 0:00:00.000000000
0:00:07.073762929   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3411:aiurdemux_send_stream_newsegment: Pad video: Send newseg 0:00:00.000000000 first buffer 0:00:00.000000000
0:00:07.079756118   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3411:aiurdemux_send_stream_newsegment: Pad audio: Send newseg 0:00:00.000000000 first buffer 0:00:00.000000000
0:00:07.080847982   868 0xffff48057700 WARN                    v4l2 gstv4l2object.c:2352:gst_v4l2_object_get_streamparm:<v4l2vp8dec0:src> VIDIOC_G_PARM failed
0:00:07.082493465   868 0xffff48057700 WARN                    v4l2 gstv4l2object.c:4638:gst_v4l2_object_probe_caps:<v4l2vp8dec0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
0:00:07.084351571   868 0xffff48057700 WARN                    v4l2 gstv4l2object.c:3382:gst_v4l2_object_save_format:<v4l2vp8dec0:src> Driver bug detected, stride (512) is too small for the width (1024)
0:00:07.093393103   868 0xffff48057700 FIXME                playbin gstplaybin2.c:3308:_uridecodebin_event_probe:<uridecodebin0:src_0> Consider implementing group-id handling on stream-start event
0:00:07.093672601   868 0xffff48057c60 FIXME                playbin gstplaybin2.c:3308:_uridecodebin_event_probe:<uridecodebin0:src_3> Consider implementing group-id handling on stream-start event
0:00:07.094126221   868 0xffff480576a0 FIXME                playbin gstplaybin2.c:3308:_uridecodebin_event_probe:<uridecodebin0:src_2> Consider implementing group-id handling on stream-start event
0:00:07.094386343   868 0xffff60019640 FIXME                playbin gstplaybin2.c:3308:_uridecodebin_event_probe:<uridecodebin0:src_1> Consider implementing group-id handling on stream-start event
0:00:07.125438650   868 0xffff5001b400 WARN         audio-resampler audio-resampler.c:274:convert_taps_gint16_c: can't find exact taps
0:00:07.126845636   868 0xffff5001b400 WARN                    alsa pcm_hw.c:1360:snd_pcm_hw_get_chmap: alsalib error: Cannot read Channel Map ctl
: No such file or directory
0:00:07.130274101   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
(...)
0:00:07.156427583   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
0:00:07.156507082   868 0xffff48057700 WARN               videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.156604956   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
0:00:07.156630956   868 0xffff48057700 WARN               videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.156701455   868 0xffff48057700 WARN               videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.156774830   868 0xffff48057700 WARN               videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.156879954   868 0xffff48057700 WARN               videometa gstvideometa.c:414:gst_video_meta_validate_alignment: Stride of plane 0 defined in meta (1024) is different from the one computed from the alignment (640)
0:00:07.158322064   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
(...)
0:00:07.165199243   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
0:00:07.168740582   868 0xffff30010ea0 WARN            v4l2videodec gstv4l2videodec.c:785:gst_v4l2_video_dec_loop:<v4l2vp8dec0> Received resolution change
0:00:07.169758322   868 0xffff30010ea0 WARN                    v4l2 gstv4l2object.c:2352:gst_v4l2_object_get_streamparm:<v4l2vp8dec0:src> VIDIOC_G_PARM failed
0:00:07.170134818   868 0xffff30010ea0 WARN                    v4l2 gstv4l2object.c:4638:gst_v4l2_object_probe_caps:<v4l2vp8dec0:src> Failed to probe pixel aspect ratio with VIDIOC_CROPCAP: Invalid argument
0:00:07.170767437   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
(...)
0:00:09.395853468   868 0xffff54298760 WARN               aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1

The flutter app is here: https://github.com/mmerah/flutter-elinux-plugins-imx8/blob/main/packages/video_player/example/lib/main.dart
With GST_DEBUG=4, the last few log after clicking play show this:

0:00:23.661699891   912 0xaaab07b22000 INFO              GST_STATES gstbin.c:2928:gst_bin_change_state_func:<pipeline> child 'playbin' changed state to 4(PLAYING) successfully
0:00:23.661749015   912 0xaaab07b22000 INFO              GST_STATES gstelement.c:2806:gst_element_continue_state:<pipeline> completed state change to PLAYING
0:00:23.661793639   912 0xaaab07b22000 INFO              GST_STATES gstelement.c:2706:_priv_gst_element_state_changed:<pipeline> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)
0:00:23.664287226   912 0xffff5c298b60 WARN               aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
(...)
0:00:23.668285040   912 0xffff5c298b60 WARN               aiurdemux aiurdemux.c:3549:aiurdemux_push_pad_buffer: Pad audio push error type -1
0:00:23.669715393   912 0xaaab07b22000 INFO               GST_EVENT gstevent.c:1363:gst_event_new_seek: creating seek rate 1.000000, format TIME, flags 1, start_type 1, start 0:00:00.000000000, stop_type 1, stop 99:99:99.999999999
0:00:36.603878098   912 0xaaab07db2000 INFO                 basesrc gstbasesrc.c:2913:gst_base_src_loop:<source> pausing after gst_base_src_get_range() = eos
0:00:36.604063346   912 0xaaab07db2000 INFO                    task gsttask.c:368:gst_task_func:<source:src> Task going to paused
0:00:36.604394091   912 0xffff680029e0 INFO                    task gsttask.c:368:gst_task_func:<queue2-0:src> Task going to paused

pastebin for the complete log: https://pastebin.com/wzsbeC0e

What is happening here?

Thank you in advance for your help.

Solution

A couple things were wrong:

  1. 0:00:06.614267007 868 0xffff54298760 WARN pulse pulsesink.c:617:gst_pulseringbuffer_open_device:<pulsesink0> error: Failed to connect: Connection refused this blocked the pipeline when SYNC=TRUE. With pulseaudio --start this is solved.
  2. The flutter-client embedder used this repo https://github.com/sony/flutter-embedded-linux to be built along with a video player plugin. I was trying to add the plugin code to the flutter app and thought it would be bundled together but that is wrong. The flutter app uses no plugin, it is flutter-client that uses it. This did not impact anything but when I thought I had set SYNC=FALSE in the plugin I actually did not (once again because that’s flutter-client that has it).

Answered By – mmnano50

Answer Checked By – Senaida (FlutterFixes Volunteer)

Leave a Reply

Your email address will not be published.