MKVToolNix v75.0 released

Heyo!

Time for another release. Again nothing major, just some bug fixing & adding a simple converter from timed-text to Matroska’s simple format (SRT-like).

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are stil being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: MP4 reader: mkvmerge will now read Timed Text (FourCC tx3g) subtitle tracks & convert them on the fly to Matroska’s simple text subtitle format (S_TEXT/UTF8; text only, no styles). Implements #2208, #2242, #2613, #3000, #3243, #3418 and possibly others.

Bug fixes

  • all: switched back to using boost::filesystem functions for creating directories instead of the ones introduced to work around bugs in std::filesystem. The latter didn’t work correctly with UNC paths after the switch to boost::filesystem::path in v74. Fixes #3483.
  • mkvmerge: VobSub reader: mkvmerge will now probe the .idx file during VobSub identification even if the .sub file is passed as the source. Avoids mis-detection of the .sub as MPEG program streams. Fixes #3489.
  • MKVToolNix GUI: the GUI could abort with an exception on startup while looking for the mkvmerge or mediainfo executables due to inaccessible folders. Fixes #3481.
  • MKVToolNix GUI: preferences: when opening the preferences the first time the UI might pre-select the first entry in the list of interface languages if the operating system’s language is not available for MKVToolNix. This might also happen on Linux if e.g. en_GB is set, even though en_US is available. Now English (en_US) will be selected instead. Fixes #3486.
  • MKVToolNix GUI: multiplexer: when adding files the GUI has special handling for chapter/tags/segment info files. This is done by comparing their content to certain patterns. This recognition could wrongfully be triggered if any such file was embedded in another file verbatim, e.g. with a chapter XML file attachment in a Matroska file. When trying to add that Matroska file, the GUI would treat it as a chapter file instead of a regular one. This content-based detection was fixed. Fixes #3487.

Other changes

  • mkvpropedit, GUI’s header editor: removed support for the deprecated “minimum cache” & “maximum cache” track header elements.

Have fun! 😁

MKVToolNix v74 released

Hey y’all!

MKVToolNix v74 is out. This is another one in a longer line of small-ish bug fix releases. Nothing too earth shattering. Anyway, have fun with it.

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are still being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: SRT handling: added a hack called --engage keep_whitespaces_in_text_subtitles which disables stripping whitespaces from the start & end of each line of SRT entries during muxing. Part of the implementation of #3470.

Bug fixes

  • all: Windows: UNC paths of type \\?\C:\… are supported again by switching back to Boost’s file system library instead of using C++17’s file system library. Fixes #3058.
  • mkvmerge: AVC/H.264 & HEVC/H.265 packetizers: when appending tracks the pixel dimensions will be checked & muxing will be aborted if they don’t match. Fixes #3480.
  • mkvmerge: file type detection: file types that can be detected unambiguously by their content (e.g. Matroska, MP4, WAV…) will now preferred in the detection order over file types based on their extension. Prevents certain cases of mis-detection, e.g. DTS in WAV but with a file name extension of .dts being detected as the wrong type of DTS. Fixes #3462.
  • mkvmerge: SRT handling: whitespaces will now be stripped from the start & the end of each line of each entry, not just from the end of the last line. Part of the fix of #3470.
  • mkvextract: SRT extraction: whitespaces will not be stripped from the start & end of the whole entry anymore. Part of the fix of #3470.
  • MKVToolNix GUI: multiplexer: when adding multiple files that include certain file types (chapters, segment info and tag files) and when the choice where to add them is “all files to a single new tab”, these certain file types will now be added in the newly added tab as well instead of the current tab that had already been open. Fixes #3469.
  • build system: fixed compatibility with Ruby 3.2.0 by using FileTest.exist? instead of File.exists?, Dir.exists? & FileTest.exists? which were removed in that release.

Build system changes

  • MKVToolNix is now using & requiring Boost’s “file system” and “system” libraries again instead of C++17’s file system library. This is due to gcc’s library not supporting UNC paths of types \\?\C:\dir\file.ext.

Have fun 😁

MKVToolNix v73.0 released

Hey y’all!

Even though not much has happened with MKVToolNix over the last six or seven weeks, I’ve decided to finalize the first release of the year, v73. Nothing overly amazing in it.

One note for macOS users that isn’t stated in the NEWS below: this release’s DMG was built with FLAC support again. Sorry for the confusing in the previous version; that was due to build system setup SNAFUs on my end.

You can download the source code or one of the binaries. The Windows and macOS binaries as well as the Linux AppImage are available already. The other Linux binaries are stil being built and will be available over the course of the next couple of hours.

Here are the NEWS since the previous release:

New features and enhancements

  • mkvmerge: greatly enhanced the speed of Base64 decoding which is used used e.g. when reading pictures embedded in Ogg or FLAC files or for binary data inside XML chapter & tag files. Even small blocks of data could take unreasonably long.
  • mkvmerge: added support for reading HEVC/H.265 video from FLV containers. Implements #3466.

Bug fixes

  • mkvmerge: splitting: fixed a typo in the string output in GUI mode regarding the timestamps before which splitting was done. See #3421.
  • MKVToolNix GUI: job output: after having run multiple jobs the “Clear output” functionality only reset the visible parts. However, using the “save output” function would also save the output generated before the user used “clear output”. Now only the output after the last use of “clear output” is saved. Fixes #3438.
  • build system: when probing for the FLAC library the Ogg libraries will be linked with the compiled test program as well. This fixes detection if the pkg-config settings for FLAC don’t list Ogg & the linker doesn’t find the library automatically (e.g. for my macOS builds). Fixes #3439.

Have fun 😁

Running Buildbot on IPv6

As I’ve struggled a bit here I thought I’d make this short write-up for others to find. The following writeup was done against Buildbot 3.6.1 on Ubuntu 22.04 & Arch Linux.

There are two well-known components to Buildbot: one master process (purposefully ignoring multi-master setup) & one or more workers. The master is the one providing the web interface, scheduling the work, keeping track of results & sending notifications. The workers do the work (well, duh!).

This means that we’re basically dealing with two different scenarios here:

  1. The master process must listen on IPv6 addresses for connections, both for the web interface & the workers which connect on a dedicated port.
  2. The workers must connect via IPv6 to the master.

Using the default configuration or the examples from the website neither of the two components use IPv6, for different reasons.

Both use the twistd Python framework for connection handling.

Making the master listen on IPv6

The configuration for the master is done solely via master.cfg, not in buildbot.tac.

Basically for every connection the master listens on a ConnectionString is used which contains both the port number & the address to listen on, e.g. "tcp:8010:interface=127.0.0.1" One might attempt to simply replace the IPv4 address 127.0.0.1 with an IPv6 address, but that doesn’t work as colons are used to separate key/value pairs within the ConnectionString. Therefore the colons must be escaped. Note that the syntax often used in other programs such as web browser to escape IPv6 addresses, placing them in square brackets such as [::], doesn’t work here either.

How are those colons escaped, though? With backslashes. However, these are Python strings, and Python uses backslashes for escaping in strings as well. Therefore in the source code we have to use two backslashes per colon. The resulting connection string might look like this: "tcp:8010:interface=\\:\\:" (which means “listen on all IPv6 & IPv4 addresses, regardless of the interface, on port 8010”).

Web interface handler

For the web interface listener, which is usually referred to as c["www"] (with c being the an instance of BuildmasterConfig), a dict is used. The port key can be a port number, or a full-blown connection string. Example:

c['www'] = {
    'port': "tcp:8010:interface=\\:\\:",
    'plugins': { 'waterfall_view': {}, 'console_view': {}, 'grid_view': {} },
    'auth': util.UserPasswordAuth({'user': 'supersecret'}),
    'authz': authz,
    'change_hook_auth': [strcred.makeChecker("file:changehook.passwd")],
    'change_hook_dialects': { 'gitlab': { 'secret': 'reallysecret'  } }
}

As hinted in the example the web interface handles both requests from us humans as well as change hooks from version control systems.

Worker handler

The worker handler is configured via c["protocols"]["pb"], and again a dict with a key called port is used which can be set to a connection string. For example:

c['protocols'] = {'pb': {'port': "tcp:interface=\\:\\::port=9989"}}

Making the workers connect via IPv6

The configuration is done in buildbot.tac.

The worker creates an instance of the Worker class. The first two parameters are the host address & the port number of the worker handler on the master’s side. Unlike the master, the parameters cannot seem to be ConnectionStrings.

Furthermore, the twistd client framework used here cannot resolve DNS AAAA records at all. If you provide a host name instead of an IPv6 or IPv4 address as the first argument, the library will only try to resolve it to an A record. For AAAA-only DNS records this means that name resolution fails, and the worker will not start. For DNS entries with both AAAA and A records, only the A record will be used, and the connection will be attempted via IPv4.

Luckily you can use an IPv6 address directly as the first argument. As there are no ConnectionStrings involved, colons can be left as-is. For example (all following arguments left out as they aren’t important):

s = Worker('2001:db8:1234::2', 9989, …)

Making the master connect via IPv6

We have to go back to the master once more as it can also make certain outbound connections that you might want to run via IPv6. These include but aren’t limited to:

  • email notifications
  • version control system status updates (Gitlab, Github etc.)

Email notifications

The email notifier also uses a pair of parameters specifying the SMTP server’s hostname & its port. Unfortunately it uses the same twistd client components that the worker uses, too, with the same DNS resolution issues. Sure, you can specify an IPv6 address directly instead of a host name, but that’ll be problematic if you want to enable TLS as certificate validation will now fail. I don’t have a good solution for IPv6-only setups in this case.

An example without authentication & without TLS:

mailer = reporters.MailNotifier(
    fromaddr="buildbot-master@example.com",
    sendToInterestedUsers=False,
    extraRecipients=["buildbot-failure-recipient@example.com"],
    relayhost='2001:db8:fefe::9876',
    generators=[generator_failing])
c["services"].append(mailer)

Version control status handlers

Handlers that signal the build status to version control systems such as Gitlab & Github don’t use the twistd framework for communication, it seems. They probably use something like urllib2. For their configuration they actually expect a base URL such as https://gitlab.com, and luckily for us, if that address resolves to an AAAA record then the status is indeed sent via IPv6. Nothing for us to do here.

Other considerations

Last but not least: the build instructions you create must also take IPv6 connectivity into account. If you configure git repositories on IPv4-only websites such as Github (shame!), you won’t be able to use them in an IPv6-only environment. Dual-stack setups will be fine, though.

On the other hand: external tools such as git that do not use the same Python libraries will resolve DNS entries to AAAA & A records & prefer AAAA = IPv6 over A = IPv4 — meaning configuring a git repo on sites such as Gitlab will indeed be retrieved via IPv6.