New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
No ATSC CEA-608/EIA-708 Closed Caption subtitles with NVDEC #364
Comments
The problem is that closed-captioned subtitles are incorporated into the mpeg stream. NVDEC and other hardware decoders do not extract them and we do not have code to examine the stream directly for this. Originally I saw this problem on Raspberry Pi with the hardware decoder. I have thought about this before, and the solution I thought of was to take the source code of ccextractor and incorporate it into the player logic in a way that could extract subtitles while playing. |
I have not yet looked at the code but as I understand it VDPAU and VAAPI are also hardware decoders. Yet ATSC CC subtitling does actually work with these decoders. |
Correction: test clip "frida.ts" (re-used from #326) does show the subtitle menu but only the first choice of both the ATSC CC and the VBI CC entries. This means only the Spanish subtitle can be selected; the English subtitle, which is the second entry for each, cannot be selected. |
That sounds like an ffmpeg bug I've seen before where it only extracts the first subtitles. To work around it on my transcoding tasks, I had to separately extract both subtitles with ccextractor and remux them later as subtitle streams. |
This ffmpeg ticket https://trac.ffmpeg.org/ticket/8078 tells that you need to add the command line option "-hwaccel nvdec" in the ffmpeg transcoding command to get the closed captions transferred when transcoding. |
After looking at this for a while I do agree with Peter's comment that it would be best to make something separate that collects the closed caption information from the video stream. The way it currently works for VDPAU is that there is a lot of MythTV-specific code in file mpeg12dec.c to capture the closed caption information. To make this work for NVDEC a similar set of changes would be needed in file cuviddec.c where the NVDEC decoding is located. |
Would it be possible to duplicate the stream and pass it to both the hardware decoder and ffmpeg for demuxing the subtitles? |
FWIW, the Android frontend has this problem too when using MediaCodec. |
@ulmus-scott @kmdewaal Rather than adding new closed caption code to cuviddec.c, wouldn't it be easier to pull the closed caption code out of mpeg12dec.c and put it in a separate C program so that any decoder, including cuviddec.c, could use it? |
The FFmpeg change can't really be separated out. Also, FFmpeg should support A/53 subtitles (if our customization didn't override it), see |
@ulmus-scott That's too bad. What is the import of FFmpeg supporting A/53 subtitles to NVDEC? Could |
On 25/11/2022 2:48 pm, bshanteau wrote:
@ulmus-scott <https://github.com/ulmus-scott> That's too bad.
What is the import of FFmpeg supporting A/53 subtitles to NVDEC?
Could |mpeg_decode_a53_cc()| also serve cuviddec.c?
Check that its not a forces vs unforced issue too.
Ive seen problems with this in the past displaying subtitles when every
odd one was forced and erasing the unforced ones.
Mark
|
On Nov 11, 2021 @ulmus-scott wrote:
Are you saying that while a GPU is decoding the video and audio streams that it is incapable of decoding the subtitles? According to FFmpeg closed defect #8078, |
That was mostly a reminder that our FFmpeg customization may be unnecessary and that we should look into removing that FFmpeg customization.
Looking at my suggestion again, it is basically the same as Peter's: add code to detect and parse the various A/53 subtitle flavors to our video player. The problem is that the subtitles are not a separate stream but encoded as part of each frame. This means the hardware acceleration needs to support separating the subtitles or you need to parse the stream in parallel to extract them. I am not familiar enough with either the FFmpeg hardware acceleration code or our player code to say exactly what is needed. |
@ulmus-scott wrote
Would it be sufficient to just have MythTV's video player call |
I've been out of town and haven't been following this (nor much of any other thread) lately. However, if this is the problem I think it is, Channels DVR has a solution they use. They setup a BitReader to capture the subtitles encoded in the video and decode them themselves. They then keep track of the timecode and reattach the decoded subtitlte back to the video frame when it comes back from the video decoder. The last time I talked to him, Aman Karmani aman@tmm1.net was willing to share some (maybe all) of their code that does that. The big catch is that at the time, it used an older ffmpeg and probably needed serious porting. |
@gigem If I'm not mistaken, the |
@kmdewaal Would you mind explaining the necessity of creating |
It is not really related to this ticket but a bit of background never hurts. |
@bshanteau, that would be excellent! Aman told me he tried to get it accepted before but it was rejected as being too much of a hack. And he's an ffmpeg committer! The last time I checked myself, it didn't look like anything had been done. I'll check again soon but if you have a reference handy, please send it my way. |
I just checked and the nvdec subtitle bug has been fixed in v33.0 Thanks to all!!! |
@gigem Also see the |
@gigem And the |
@gigem Finally, the Wikibook: FFMPEG An Intermediate Guide/subtitle options |
@bshanteau, thanks for the links. I don't think this problem has been fixed for the mediacodec case, which is the one that matters most to me. I checked the ffmpeg, git logs and don't see anything having been done regarding closed captions. I've asked Aman for confirmation but he hasn't replied yet. FWIW, I did see some other changes I'd been loosly tracking that have made it into master. Those changes use the native API for mediacodec so Java is no longer needed. It's possible that change fixes the closed captions issue as a side effect but I doubt it. Regardless, support for that should be pulled into MythTV eventually. |
Nothing has changed wrt to the mediacodec decoders. If someone wants to work on that I can point them in the right direction. |
@tmm1 I'm willing to help, although I suspect the resolution might be as simple as grabbing the latest binaries of fmpeg and ffplay that are generated when, as @kmdewaal said above:
How is the latest MythTV version of FFmpeg compiled? |
That may help with nvdec, but not with mediacodec (android). To extract closed caption data from mpeg2 streams without decoding, the BSF patchset that adds A53_CC support needs to be merged. There is an old version that no longer applies cleanly and must be forward-ported: https://ffmpeg.org/pipermail/ffmpeg-devel/2018-March/227321.html |
@tmm1 wrote:
What is the command line option that calls mediacodec? This doesn't work: |
@bshanteau, mediacodec is the Android, hardware decoder and only runs on Android. I doubt there is a true, command-line way to run it. Instead, you will need to use an Android app which uses it, and probably one that uses the NDK at that. I don't know if Android, Java apps have this problem. BTW, @tmm1 is an ffmpeg developer and a Channels DVR developer. He knows what he's talking about. |
@gigem I have no doubt that @tmm1 knows what he's talking about. Nevertheless, I'm struggling to understand what he means when he says:
On first glance it sounds like he's saying that So what is @tmm1 asking for? Is it that he wants to be able to transfer a recording file from an Android device to a Linux machine and decode it with Unless the current versions of |
Since this thread is about NVDEV, maybe a new issue about mediacodec makes more sense. I don't want to distract or conflate two issues. My understanding is that there is some mythtv android client. In that client, when you playback it uses mediacodec for hardware decoding. In this decoding setup, caption data is not available because the mediacodec decoders cannot parse and return it. Thus you must extract the caption data before decoding, then reattach it to the decoded output. For decoding/playback, ffmpeg cli is not used but rather libavcodec is used directly. However, since last week it is now possible to also run ffmpeg cli on android by using NDK directly instead of java. You can see the patchset here: https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_5F3D69DA7D4AADBE3995C32DB6EC685A010A@qq.com/ |
@bshanteau, I think your confusion stems from not knowing that MythFrontend runs on Android and not just Linux and Mac. For some MythTV users and developers, like me, the Android frontend running on Nvidia Shields is the primary frontend. Because of the mediacodec/subtitle issue being discussed here, we can't see subtitles when mediacodec is enabled. The change @tmm1 is referring to is a patch he wrote some time ago for Channels DVR which fixes the issue. It's similar to the idea propesed by @ulmus-scott where the video stream is split and decoded by hardware for the video and separately by software for the subtitles. For those of us using the Android frontend, the inclusion of @tmm's patch or something similar would be greatly apprecieated. The catch is that the ffmpeg API has changed and the patch requires non-trivial porting before it can be used. |
@tmm1 It appears you resubmitted the FFmpeg patch set https://patchwork.ffmpeg.org/project/ffmpeg/patch/20190911185610.87081-2-ffmpeg@tmm1.net/ and it was rejected for the same reason: anonymous unions were not supported by all supported compilers. I assume this is still the case since anonymous unions are a C11 feature and FFmpeg is C99. Is that the newest version of the patch set?
|
Yes that's the newest version. I think I started to port it forward so I will find my WIP and share that. Looks like I removed the unions already: |
Here is my WIP tree. Basically the way the BSF filters work changed pretty dramatically so a lot of the code has to be restructured and rewritten. |
@tmm1 @gigem Thanks for the clarification. Getting back to the subject issue, I'm trying to find out the details of what @kmdewaal describes:
Where can I find these efforts? Is it MythTV/FFmpeg? |
@kmdewaal @tmm1 @gigem Do I understand correctly that mythffmpeg is compiled from a different version and with a different configuration than the ffmpeg in the Ubuntu repository (or whatever)? On my system (LInux Mint 21 XFCE based on Ubuntu 22.04) ffmpeg is version 4.4.2 while mythffmpeg is version 5.1. Also, the configurations are different (for instance, mythffmpeg is built with --disable-nvenc).
|
Yes the mythff* executables are compiled using our copy of the ffmpeg source and the configuration can be tweaked by passing options to our configure script. It should accept all the same configure options that the ffmpeg configure accepts unless something has changed. |
@tmm1 I am open to the idea of rebasing the FFmpeg patch series, but I have a few questions first:
Note: A/341 also provides for the CC data to be embedded in HEVC streams, identical to AVC via ANSI/SCTE 128-1. @bshanteau https://github.com/MythTV/FFmpeg is our downstream fork of FFmpeg that we have customized over the years. It is copied into https://github.com/MythTV/mythtv/tree/master/mythtv/external/FFmpeg . What @kmdewaal is referencing is probably my efforts to reduce our downstream changes on the way to the long term goal of using the system supplied FFmpeg, see #428 . |
#if CONFIG_H264_MEDIACODEC_DECODER
-DECLARE_MEDIACODEC_VDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb")
+DECLARE_MEDIACODEC_VDEC(h264, "H.264", AV_CODEC_ID_H264, "h264_mp4toannexb,h264_metadata=a53_cc=extract")
#endif |
That would be a good thing. I hope my questions help. I know I am a novice in this stuff, so please let me know if my questions are helping or getting in the way. Lately I've been wondering: What happens if I have both And what does it mean if |
@bshanteau, the system and MythTV versions of mmpeg can coexist just fine as the MythTV version uses different file and library names. The case that mythfrontend doesn't show subtitles with nvdec is a bug in how MythTV is using nvdec. I expect it will get fixed by those knowledgeable in that area soonish. |
Would this change be accepted by FFmpeg? How would we attach the filter to the NVDEC decoders or the the mediacodec decoders without that FFmpeg change? |
I just checked and the nvdec subtitle bug has been fixed in Thanks to all!!! |
@gigem Nope. Bug reappeared after reboot. |
Platform: Ubuntu 18
MythTV version: master
Package version: compiled from source
Component: mythfrontend
What steps will reproduce the bug?
Use a frontend with an Nvidia card and the Nvidia driver. Then:
NOW the subtitle is displayed. This is correct.
The problem appears when:
NOW: cannot select the subtitle menu, hence cannot select the subtitle and no subtitle is shown.
How often does it reproduce? Is there a required condition?
The issue is 100% reproducible on my ATSC test streams. I do play them back as a Video.
The issue has also been reproduced by pgbennett.
What is the expected behaviour?
The expected behavior is that, when a subtitle stream is present, this can be selected and shown on the screen, for NVDEC, for VDPAU and for any other video driver.
What do you see instead?
Additional information
This problem is reported on the mailing list, item https://forum.mythtv.org/viewtopic.php?f=36&t=4458
Reported for MythTV version 31.
Note that this is specific for ATSC CC subtitles; DVB subtitles can be selected and are shown as expected.
The text was updated successfully, but these errors were encountered: