[Matroska-devel] Re: Dirac Video Codec

[Paul Bryson]

"Steve Lhomme" wrote...
Tim Borer a écrit :
> It's all on the website, but if you need more precision you can use our 
> mailing list or our IRC channel (#matroska on irc.corecodec.com).

The mailing list is best for writing out points that require lengthy 
explanations.  IRC on the other hand is very effective for a quick 
discussion about some point.  You can usually get quick comments from 
multiple developers at once.

> > I guess one thing we need to know is what we need to do to integrate 
> > Dirac with Matroska. We already have a C interface for the decoder and 
> > are working on one for the encoder. These are only alpha and so the 
> > details will change but the principle should be about right. We have 
> > modeled them on C interfaces used for codecs in other libraries (e.g. 
> > ffmpeg).

There are currently several independently written Matroska writing libs that 
are free for use through GPL/LGPL licenses.  You could certainly write 
directly to a file using one of these.  Although in the long run I would 
imagine that it would probably be better to find a way to integrate into an 
existing media framework.  Steve mentioned GStreamer and it probably has the 
best chances to be a real cross platform media encoding/playback system. 
But as also mentioned, you will likely need another tool to add some of the 
more advanced features.

> > >I couldn't tell about MXF. All I know is that it's a professional class 
> > >container, but maybe not >good for everyday/basic use.
> >
> > Its a professional container. From our point of view it has problems in 
> > getting additions agreed with the standards body (SMPTE). There are also

I had a look at MXF around a year and a half ago and even exchanged some 
emails with some of the people in the group.  For some reason though they 
stopped responding and I became busy with other things.  I had looked 
through some of their specification drafts and it looked like a good thing 
for large scale data imports.  Basically just a good way to transfer data 
from one system to another. (Much like XML might be used to transfer data 
between different types of databases)  When I went back to look at some 
things around 8 months ago I couldn't find any of their documents that I 
could look at for free.  Maybe this has changed or maybe I just wasn't 
looking in the right place, but this made the format seem much less 'open'. 
(To me 'open' means anyone can look at the specs,  not just people with 
money.)  Fortunately I still had a copy of the drafts and was able to refer 
to that for what I was interested in. Regardless, MXF seems like overkill 
for what you want as it looks like you need a playback/authoring format.

In regards to Ogg, it is a good format for basic streaming, but not much 
more.  Also, any format that is put into Ogg basically has to be more or 
less approved by Xiph as there is not a 'standard' way of adding formats. 
In many ways it is the polar opposite of Matroska in philosophy in that 
almost no information is stored at the container level whereas Matroska 
stores as much information at the container level.  For instance, you don't 
know the timecode of the data packet you are looking at without the codec in 
Ogg.

> > issues about the availability of open source libraries for it and their 
> > limitations.
> >
> > Open source, freely available, cross platform is where we want to be.
> >
> > One question is can Matroska be used commercially as well as personally, 
> > or are there intellectual property issues that would cause problems?
>
> Yes, all the code we (the core team) write is either LGPL. The code of 
> other people (like the C implementation in GStreamer) is usually LGPL too. 
> Mkvtoolnix and AVI-Mux GUI are GPL.

There are several independently written reading libraries that are all open. 
The official library libmatroska (C++), AVI-Mux GUI (Delphi), GStreamer (C), 
Mplayer which is somehow based on GStreamer (C), as well as some ones in 
development such as Javatroska (Java), RbMatroska (Ruby).  There are 
probably others that I'm forgetting.

> > Sounds like this might be useful, although I though we needed to handle 
> > this ourselves in an elementary stream to ensure that the stream could 
> > be stand alone if required. Do you have a URL that would point to
>
> Yes. But it is very important to have this information available to the 
> container (too). It's needed for precise seeking. It's also needed for 
> "lossless" or "blind" editing (not modifying the codec data when 
> inserting/cutting frames in a file).

As he says, it is very useful to have as much data as possible available at 
the container level for authoring purposes.  This is part of the philosophy 
of Matroska.

> > I am wary about using time code. It works well for video at 24, 25 and 
> > 30 frames per second. For NTSC at 30000/1001 (aprox 29.97) frames it is 
> > bodged using drop frame. There is also confusion about streams which say 
> > they are 29.97 when they may mean 30000/1001. The next result is that my 
> > experience is that timecode is not as reliable as I would like. 
> > Therefore for Dirac we are planning to use sequential frame numbers 
> > rather than time code. then you can keep accurate track of frames and 
> > convert to a time code if required. What is you opinion on this?

This is what MPEG currently does.  It can only display a constant framerate 
and does not allow for dropped frames.  Things get very interesting when you 
mix framerates.  For instance, Star Trek TNG was filmed using standard NTSC 
speeds of around 30fps.  But all of the scenes in space that were rendered 
on computers were rendered at film's 24fps.  So when storing on a DVD, they 
use 30fps and have the 24fps images interpolated to 30fps.  (Maybe you 
already know how they do this, if it's important to you I could explain it.) 
In our view this seems a little silly.

During playback, all systems convert to a timecode anyway.  DirectShow uses 
a 100ns accuracy timecode.  Matroska can handle accuracy up to 1ns 
internally, but typical Matroska video files are stored with 1ms accuracy. 
(It is fully adjustable for what is needed.)  Granted this will produce 
timecodes that have error of <1ms from what was originally used (Ex: 5370 * 
30000/1001 for the timecode of frame 5370).  But keep in mind that the 
effect is not cumulative.  Each frames timecode is calculated individually. 
So no timecode would have an error that is more than <1ms for typical 1ms 
accuracy files.

While the idea of introducing a level of 'error' into the timecode can seem 
like a bad thing, having a frame off by 0.5ms is less than the total time 
that it would be off due to typical display processes.  There are also some 
advantages.

VFR (Variable Frame Rate).  Vorbis is Xiph's free general audio codec.  Ogg 
was designed for it, but it is also usable in Matroska.  The number of 
samples in a Vorbis packet are variable.  One might have 1000 samples and 
another might have 300. (I can't recall real numbers you would see.)  So the 
length of time that one packet covers is not the same as another.  This 
introduces issues in systems that demand a constant rate of packets, or 
CFR(Constant Frame Rate).  Because Matroska simply attaches a timecode to 
the packet, this is not an issue.

Now to extend this idea a little further.  This may require a bit of thought 
as the world of video has been a CFR place for so long that it is difficult 
for people to grasp the idea that it could be done differently.  Because of 
timecodes, we don't have to be dependant on a specific framerate.  Instead 
we can record video frames as they come in.  If one video frame lasts 37ms, 
and the next lasts 283ms, its not an issue.  This also gives you the option 
that if you have a very low motion scene in video, you could dither the 
frames together and instead of storing them at 30fps, you could store them 
at 23fps.  Completely optional depending on your codec and the level of 
compression you are seeking.

This also gives many options for authoring as well.  If you want to speed up 
a sequence of frames or slow them down, there is no reason to reencode them 
and lose efficiency.  Simply adjust the timecodes to put the frames closer 
together or further apart.  I have a tool that lets me edit Matroska files 
by cutting, offsetting audio/video, and appending streams.  Because all of 
the data is at the container level, the tool is able to do this no matter 
what codec was used on the audio or video.

This is also a huge boon in the ability to maintain audio and video 
synchronization in the event of an error.  If there is data corruption in a 
Matroska file, the next audio and video packets to be found will have 
timecodes.  Instead of the application having to guess at the timing of the 
packets relative to each other, it can instead know for certain because it 
knows exactly when a packet should be displayed or played.  While I have 
seen many AVI and MPEG files lose A/V sync for various reasons, I have never 
seen this happen to a Matroska file.

Here is an example video sequence that shows the exact same sequence of 
frames, but with the timecodes adjusted so that the video moves at different 
speeds.
http://commo.de/3blocks-vfr.mkv


Paul Bryson