Google has recently announced their intention to remove the H.264 video codec from its Chrome browser. This decision has been smeared as an evil campaign for controlling video on the web, akin to not-invented-here syndrome. It’s also been lauded as the push that the web needs to remain open and free. Mostly, it’s been marked as inconsistent, due to the bundling of Adobe’s proprietary Flash player.
Richard Stallman doesn’t like the term Open Source because it fails to embody the true meaning of “free software”, and the one thing that’s worse is the word “open”. This debate can’t simply be labeled as one for or against openness (even ignoring the technical details).
H.264 is an open standard. It was developed by a committee, standardized, reviewed by many engineers and developers for multiple companies and has been standardized for use with a multitude of containers and devices. However, H.264 is not royalty free. Software patents in many countries restrict the distribution of software that utilizes the codec to those who pay the MPEG-LA.
VP8 is not a standard. It was developed secretly by a single company, and until recently, had only a single working implementation. The public wasn’t open to collaboration on the specification until the bitstream spec was frozen, including the bugs that existed within. Now, the source code and reference implementation are available under liberal licenses, and all the related patents are irrevocably royalty-free.
Adobe Flash, while not synonymous with a video codec (unless you mean .flv flash videos, which are either VP6 or Sorenson Spark), is going here anyway because everyone feels like comparing it. Flash’s SWF format is not standard, but it is open. There are a few implementations (Swfdec, Gnash, GameSWF, Gordon, etc), but none of them are as complete as the official proprietary implementation. There’s a bitstream specification that anyone can read to create an independent implementation of the player. Implementing and using the Flash player is still royalty-free (Since the Flash VM can decode H.264, obviously that part, not controlled by Adobe still has royalties), anyone can make software that can export SWF animations without paying Adobe.
Implementation vs. Distribution
Or: How Bundling Flash doesn’t violate the Novikov self-consistency principle
| Name | Standardization | Implementation | Distribution | Dev History |
|---|---|---|---|---|
| Theora | Standardized? | Open Source | Royalty Free | Mostly Open |
| VP8 | Not Standardized | Open Source | Royalty Free | Mixed |
| H.264 | Standardized | Open Source | Royalties | Open |
| Flash* | Not Standardized | Proprietary | Royalty Free | Proprietary |
* Adobe Flash isn’t a video codec
Each column of the chart can be treated as one part of “open”. The <video> debate now primarily concerns VP8 and H.264, as Theora is inferior for technical reasons and Flash isn’t a video codec. I don’t know if Theora’s actually standardized, but it has multiple implementations and was developed by the Xiph.Org foundation, and is the closest thing to an actual standard, short of being a product of MPEG. Theora was created from On2′s VP3 code, but was changed significantly by the Xiph.org Foundation before release, so the development history can be considered mostly open. VP8 as announced, and the bitstream was frozen shortly afterwards, leaving little community involvement. For the years before its debut as part of WebM, the format was secret, patent encumbered, and proprietary. At least now, the source code is open and people are free to do whatever they please, but the open source video community probably had very little say in the development of the codec. H.264 has an open software reference implementation, as does VP8 and Theora. Flash’s de-facto implementation is Adobe’s, which is proprietary.
The one that concerns Mozilla, Opera and Google is the distribution rights; whether or not royalties have to be paid. Notably, this is where the parallel drawn by most critics of Google’s move with H.264 and Flash comes into question. Many people believe that Flash is the epitome of evil, and proprietary software, and embodies everything wrong with proprietary software. The distribution column is arguably the most important, and it’s the one that fundamentally determines whether or not something acts as a detriment to “open innovation”. Flash animations and applications can be created and distributed without paying royalty fees to Adobe, regardless of the viewership. Innovation is still permitted as long as the distribution is free (except with changing the inner workings of the proprietary implementation).
The Ambiguity of “Plugin”
TODO: Use this subtitle to mention random tangential thoughts.
It seems there are lots of misconceptions about WebM, and new ones as well, because of the use of the word “plugin” by Google in their “More about Chrome HTML Codec Change“. Here’s what the relevant part of the post says (it’s buried into the last paragraph).
This is why we’re joining others in the community to invest in WebM and encouraging every browser vendor to adopt it for the emerging HTML video platform (the WebM Project team will soon release plugins that enable WebM support in Safari and IE9 via the HTML standard <video> tag).
Microsoft and Apple through IE9 and Safari, respectively, rely on the underlying operating system’s multimedia frameworks to handle <video> decoding. Chrome and Chromium bundle a customized version of the ffmpeg framework, while Opera uses gstreamer and Firefox has its own framework built on various open source codec libraries.
| Browser | WebM | H.264 | Theora | Anything Imaginable |
|---|---|---|---|---|
| Firefox | Version 4 | Never | Version 3.5 | Never |
| Chrome | Version 6 | Removed in Version 10 | Version 3 | Never |
| Opera | Version 10.60 | Never | Version 10.50 | Never |
| Safari | QuickTime | Default QuickTime | QuickTime | Default QuickTime* |
| IE9 | Windows Media | Default Windows Media | Windows Media | Default Windows Media* |
* Well, not really *anything* imaginable, but a lot of them
QuickTime and Windows Media are pluggable multimedia frameworks. And since Safari and IE9 use them for <video> support, any codec or container format supported by the respective frameworks, works in the browser. For QuickTime, the list for videos goes along the lines of 3GP, Apple Video, AVI, DV, Cinepak, H.261, H.262, H.263, H.264, Microsoft Video 1, MPEG-1, MPEG-4 Part 2, Motion JPEG, Pixlet, Planar RGB, Sorenson Video, Qtch, QuickTime Movie, and QuickTime VR. But these media frameworks are pluggable, which means they play whatever codecs are installed. It just happens that those codecs listed above are plugins that are installed by default, somewhat analogous to how Google Chrome bundles Flash. The plugin frameworks for multimedia don’t treat plugins any different from “native” codecs. There’s absolutely no way to tell the difference from a user perspective. Right clicking a video will give you the same ordinary context menu.
Codec packs are often installed by users anyway. A codec pack consists of a set of plugins for the operating system’s multimedia framework. Once one of those codecs are installed to the system, support is available to the applications that rely on the OS’s framework: QuickTime Player, Windows Media Player, IE9 and Safari.
Notably absent are the Opera, Chrome and Firefox browsers that ignore whatever is installed, to use their own bundled decoders. There’s a reason for this, and it’s pretty easy to spot. Firefox, Chrome and Opera are also the only ones on the list that aren’t tied to a specific operating system, and bundling your own media framework lets you keep the functionality consistent. IE and Safari aren’t cross platform. Well, Safari works on Windows, but it still requires QuickTime for Windows to be installed to play the video content.
You also don’t want all the other video codecs thrown into the mix. Standards are about standardization, where you have a limited number of codecs instead of an entire forest of them. This brings us to a bit of History, I’ll begin with the olden days. I can’t tell you too much, as much of the first section happened when I was 6 years old- and certainly wasn’t into the negative user experience of platform-specific multimedia plugins.
A Brief History of Stuff Named “A Brief History…”
Uh… I mean, Video on the Web.
Before the popularity of HTML5 Video, or event Flash, video was viewed on the web using the RealVideo, MPlayer, VLC, Windows Media Player, and Quicktime Plugins. Note that pretty little plugins aren’t plugins to the underlying multimedia frameworks: these are the nasty type of plugins that Flash is among. They’re the plugins that take a long time to load, have interfaces that never look like the surrounding page or browser, only work on certain operating systems, at best, inconsistently. This is the epitome of what standardization is meant to prevent, and the distillation of everything that’s wrong with plugin based video.
Flash’s popularity (probably thanks to YouTube), brought a semblance of standardization to the industry. Video on the web was delivered through the flv container, encoded either with Sorenson Spark or, in later versions of Flash, On2′s VP6. A while later, H.264/AVC which was later added to Flash, and as the superior codec, most people switched to that and FLV is slowly fading away.
HTML5 video became popular recently. I’ll say it’s probably because of iOS, since that’s the only way to get web video to play there and the rest of the aggressive marketing that Apple does. This brings us to today (or at least the general time period of early January 2011 in which I’m writing this post in). Google has just announced it’s intention to remove H.264 from an upcoming revision of the Chrome browser (much like how the Chromium browser never supported H.264). And everyone on the internet that cares enough to say a word is going insane. Which brings us back to the last section, about how Safari and IE9 use the operating system’s underlying multimedia frameworks and how all the other browsers (Firefox/Chrome/Opera) that work on my beloved operating system (Linux) bundle their own.
If Firefox, Chrome and Opera used the OS’s media frameworks, we would be set back into the dark ages when people used every video codec imaginable and nobody would be happy. Standards exist for consistency, and it would be terrible if people started to go on a cost saving measure to never again transcode video, and serve it straight from the server’s filesystem as AVI files containing DV or god forbid MJPEG (the same format the users’s uploaded!) because all the Safaris and IE9s could play it fine. The fact Firefox, Chrome and Opera bundle their own multimedia frameworks forbids the possibility of this, because those browsers (in the forseeable future) will never support anything other than WebM, Theora and potentially H.264.
There’s more to H.264 than just H.264
This part doesn’t make much sense.
Often, the argument against VP8 is that it’s inferior to the H.264 codec, and to me, this seems like the most ideologically valid concern. But in a lot of cases, it stems from a misunderstanding of how H.264 works. H.264 is not a single video codec (not even to mention multimedia container formats), but rather has several Profiles that work on different devices and implementations. Rarely are videos encoded only in MPEG-4/AVC H.264 Extended Profile at 1080p and 60fps. Go on the Apple store and you’ll see that every iPod device you can find (including the iPhone), includes what video codecs are supported. And it doesn’t just say H.264, but rather, something much more wordy like
Video formats supported: H.264 video up to 720p, 30 frames per second, Main Profile level 3.1 with AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats
For the insanely great iPhone 4. Google is meaner and doesn’t make the information about profile support on the Nexus S quite as accessible (though it’s not the only reason I’m an iPhone user), but it should be safe to assume it’s something along the lines of what the iPhone 4 supports. The iPod Classic has a nice, even longer string, that represents even less support for H.264:
H.264 video, up to 1.5 Mbps, 640 by 480 pixels, 30 frames per second, Low-Complexity version of the H.264 Baseline Profile with AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats; H.264 video, up to 2.5 Mbps, 640 by 480 pixels, 30 frames per second, Baseline Profile up to Level 3.0 with AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats;
This is because very few devices can actually utilize all the great features that H.264 defines. Wikipedia has a nice pretty chart. So the point of all of this is to say, even though VP8 is inferior to H.264 from a purely technical standpoint, you probably can’t just use the Main or Extended profiles to support all the devices that “support H.264″. Does this invalidate the inferiority argument? Nope. Dark Shikari said “I expect VP8 to be more comparable to VC-1 or H.264 Baseline Profile than with H.264″. But the large number of devices that support H.264 might actually only support the baseline profiles.
My Hopeless Ideals
Better be blunt with what will probably never happen.
VP8 is a bit worse than H.264, and had it been a patent encumbered video format, there would be almost no reason to prefer it over AVC. The <video> part of the HTML5 specification states:
It would be helpful for interoperability if all browsers could support the same codecs. However, there are no known codecs that satisfy all the current players: we need a codec that is known to not require per-unit or per-distributor licensing, that is compatible with the open source development model, that is of sufficient quality as to be usable, and that is not an additional submarine patent risk for large companies. This is an ongoing issue and this section will be updated once more information is available
The only major codecs that are royalty free are Theora and VP8, and the former probably isn’t of sufficient quality. Both of them come with patent risks (or at least that’s what MPEG-LA wants people to believe), leaving a set of zero acceptable codecs. For this to work at all, something needs to be compromised.
H.264 defines a baseline profile that all decoders could be reasonably expected to handle. Such a profile exists so that the video can be viewed, albeit with inferior compression on a variety of devices and platforms with limited computational ability. The internet was built on interoperability, and HTML5 needs an equivalent “baseline codec” for the web. Something that compresses video at sufficient, though not bleeding-edge quality. Something that can be implemented and distributed openly on all platforms.
For HTML5 <audio>, nearly all browsers implement the basic WAV codec. It serves as an acceptable baseline for small snippets of audio to be played in a cross-brower manner. It’s usually uncompressed, and for most applications, inferior to MP3 and Vorbis. <video> needs similar treatment. That niche is, at this time, best filled by WebM/VP8. If Apple and Microsoft were to add support for the WebM format, it would only improve the environment for open innovation. H.264 can be considered the “bleeding-edge” codec, heck, they could even add HEVC/H.265 to set off an environment of useful competition (or not). But a “baseline codec” should be established first. Something that publishers can encode their videos in, so that any modern browser can view.
Right now, the discussion has been polarized between free software advocates who often seek to eradicate proprietary or patent-encumbered ideas from the face of reality and those who hold a disregard for the open source values. This way of thinking, and of treating innovation is profoundly dangerous for both the free software community and the patent-creators. The free software community can not afford to be constantly twenty years behind the times in terms of innovation (especially if you subscribe to the law of accelerating returns), and the patent-creators can’t afford for the free software community to involved in actively foiling their innovations. It’s truly great what MPEG has done, and it’s terrible that such a controversy exists around its adoption. The best scenario would be for MPEG or the ISO standards bodies to eliminate royalties. The fact multiple industries can agree upon a single, high quality, interoperable codec should be enough of a market and innovation advantage to waive the relatively nominal money from licensing.
Will this actually happen? I doubt it. Apple seems firmly invested in the success of H.264. Microsoft may be more willing to add VP8 support if and when the format gains popularity (and especially if Google adds patent indemnification just so they can see a lawsuit made). Mozilla will likely never compromise on principle and Opera might add AVC one day. Given the huge backlash, Google’s the most likely to revert its opinion and add AVC back into Chrome in a future release. Yay for hopeless idealism! And since I’m espousing hypothetical and impossible ideas anyway, why not vouch for reform of the U.S. patent system as well? If you think of it, the fact it’s such a controversy and that people need to use an inferior product in order to innovate, that smacks in the face of the very Article 1, Section 8, Clause 8 of the United States Constitution:
To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.
And this ends what may be the longest blog post I’ve ever written, revised three times. If you’ve actually read this through, please consider commenting or sending me a tweet at @antimatter15 (or following me, that would be great!). Also, if I’m misinformed, please inform me and I might revise this yet again.
Good explanation. I’m really quite curious how this will turn out. Something tells me we won’t see resolution for a couple years yet.
Mozilla Firefox, Chrome/Chromium and Opera bundle a copy of a customized version of the ffmpeg framework.
This is true only of Chrome. Firefox implements its own media framework on top of the various open source codec libraries. Opera incorporates a copy of the Gstreamer media framework, at least on some platforms.
Thanks! I’ll correct it
If WAV is the default baseline codec for audio, then.. why not something similar for video? Uncompressed RGB? HuffYUV? MJPEG would also suit well – low complexity and all browsers already have jpeg decoders.
Obviously, they all require much, much greater filesizes to achieve comparable quality to VP8 and H.264, but then again, so does WAV.
Well, uncompressed audio is pretty inefficient, but audio is much less complex than video. Generating a WAV file from JavaScript is pretty easy to do. It’s easy enough to parse a vorbis waveform with JS and some binary XHR hacks and generate a wav audio file, and not totally impractical. But uncompressed video is *huge*, and the recent port of libvpx to Flash with Alchemy only got something like 1.5fps. If uncompressed video were an option, there would be no advantage of using video over using canvas to paint individual pixels and synchronizing audio.
I don’t think my baseline codec argument was really great, but I’ll try to stretch the analogy. Image’s baseline would still be compressed: something like PNG or GIF. The compression isn’t as great as JPEG or WebP (again, this is stretching it as PNG is lossless). And video’s baseline needs to be even more compressed.
What annoys me most about this whole thing is that we’ll be stuck with flash for much longer than I hoped for. I’m also a Linux user and I can’t go a day without flash crashing at least 5 times. With h.264 support in chrome I can watch a lot of online video without needing that unbelievable unstable POS plugin. And I actually hoped that some day it will run as nicely as it does on my iPod touch. Without flash.
But google came and completely shattered that dream. They do offer WebM as an open/free alternative, but there is no content available!! Do no we’re back to square one. As if all the “progress” of html5 never happened.
Dammit!!
I totally agree. I end up using Firefox to load flash content and use Chrome to browse the web because half the time I go to a new tab or scroll a page, all the flash embeds suddenly die and become black boxes. Fortunately most of the web video I watch is on YouTube and it has most videos transcoded to WebM already.
I really don’t know of that much content that uses html5 video. Since the dropping of H.264 will basically reduce mainline chrome to the state that Chromium has been for a while (For a long time, I used Chromium instead of chrome and was bothered once in a while by the lack of h.264 video). There’s this tutorial for copying the ffmpeg library from a chrome installation (or in this case, possibly a pre-removal ffmpeg library). http://chromestory.com/2010/11/how-to-make-chromium-os-play-mp3-files-on-windows-linux-and-mac/
The double-edged side of the sword of open innovation is that the community can still do things if the members disagree, and getting h.264 to work should be more or less trivial anyone who is willing to copy a file.
YouTube has most videos transcoded to WebM already? That’s something I haven’t heard in the dozen or two articles I’ve read about this debate.
I don’t understand the argument against using system-based media codecs. The argument that this would send us back to using all known codecs in the universe is nonsensical. Just because a codec exists and can be plugged into a system does not mean that it necessarily has to be used. Standards don’t just form on the web; they form in the real world too. In fact, in the real world H.264 is already the standard. People aren’t going to start using oddball codecs because it’s feasible. Consistency is arguable, but it’s not as if the inconsistency would reach incomprehensible levels to the point where it would outweigh the value of having options. Also, I was under the impression that Opera does allow fallback to system codecs — am I wrong? And Mozilla preventing it from being possible for purely ideological cheerleading of free software isn’t an argument against the possible solution: they could very well allow it — it is a viable solution that they refuse to support, but it is viable. The best solution to me is each browser platform can have a preference, its own media playback system if necessary, and provide the use of plug-ins and fallback to system codecs.
The img tag didn’t specify a format; I don’t see the reason we need to dictate a video format.
Would love to hear you expand your argument on this area of the post.
That’s an interesting link. Thanks.
Perhaps I’m missing something but…
H.264 is an Open standard encumbered by royalty fees (they may disappear possibly) and is excellent and simple to use
I know & unfortunately dislike Flash
WebM is “owned” and not a standard in any recognizable way
Why is google trying to roil the waters now? There isn’t any $$ in it for them and the enmity of developers doesn’t sound like a Yellow Brick Road.
Also, I wanted to applaud you for recognizing that we are talking about a baseline for accessibility (or at least I think this is what you are arguing). My concern is that the free software advocates are derailing the whole debate and demanding that the entire video industry (they will claim offline video can be re-encoded for web use, but this is impractical for a number of reasons) must only use a free and open standard. But I am perfectly happy with a baseline format competing with whatever else the commercial world and consumers would prefer — in which case, everyone would provide a baseline free codec AND have the option of using h.264, and if h.264 is preferred the vast majority of the time, that’s okay because the free software users and those without financial resources could fall back on whatever free standard (just as .wav is an available option but everyone uses .mp3 and uses javascript, Flash, or some other mechanism to provide playback since that is just as trivial as using .wav)… which is where I thought we were headed (and I think it’s very clear that this was the likely conclusion and STILL probably inevitable) until Google decided to be ideological.
So, congratulations for that point — if, that is, I am correctly interpreting your point. I think we need a much saner conversation regarding the true purpose of the w3c specifications and the reality of a web that, while based on open, free standards, is also propelled forward by commercial competition.
@Rem, Most of the new videos and the ones in HD have been transcoded to WebM. It’s not scientific, but instead purely anecdotal. The videos I tend to watch have mostly been encoded to WebM.
@Tim F. I don’t use Opera, so I really have no clue and I haven’t tested many of the claims in the post. In the real world, there’s lots of codecs. Cameras, camcorders, and feature phones often record in non-h.264 formats, and it not transcoding is always easier than transcoding. So as you said, that it’s unreasonable to ask an entire industry to re-encode all their videos, people may take that to the extreme and never transcode videos to h.264 in the first place.
There are a lot of ways in which video is different from images. The time (and memory) it takes to transcode and store video is much greater than that of imagery, and so the cost of inconsistency is that much greater. I’m not quite old enough to have experienced the development of the web back then, but I’ll guess it was in the era before the advent of open source browsers (Mozilla, Webkit/Safari, Chromium didn’t exist until later, Chrome happened really recently, so recently in fact, that I can actually remember it). And back then, Netscape and Microsoft, had the ability and resources to license all the codecs (as Apple, Microsoft, and Google can today). There wasn’t a Mozilla Foundation to, as you say, to do the “purely ideological cheerleading of free software”. It seems that the patent issue with GIF did cause issues, leading to the creation of PNG.
Standards in a document by the w3c don’t really matter. Much of HTML5 developed with WHATWG while the w3c ignored it, and the unwritten standard for jpeg, gif and png still exists. What the web needs isn’t a reference within some standard, but rather, unity (this is starting to sound like a civil rights speech, happy martin luther king day on monday everyone!).
@Tim the system framework implementation makes sense for Apple and Microsoft since they have their one OS’s to work with. The media frameworks in windows and os x are quite mature and the browser just builds upon them. There no sense in doing the same work twice. The other browsers have it much harder. They support not only windows and os x, but many flavour of unix and Linux which might or might not have decent media frameworks build in or even available. For these companies it makes sense to implement their own (or an open source) framework to play media. That ensures them consistency across platforms. The fallback is a good idea, but it can get pretty complex if you take into account all the frameworks it could fallback to.
@HESweeney You may have missed the last sentence of the fourth paragraph “Now, the source code and reference implementation are available under liberal licenses, and all the related patents are irrevocably royalty-free.” There’s no monetary incentive for Google because of the irrevocable patent license. Google couldn’t possibly make money out of it and already lost a hundred million from it. It /was/ owned, and now it belongs to the community more or less.
@Tim F. The purpose of patents are to promote innovation, and the best way to do promote innovation is through competition. H.264 has slight advantages almost always, but people may want to stick to the baseline for the sole purpose of being a baseline, to reach the greatest potential audience. If a publisher is concerned with squeezing every last byte of bandwidth and quality, a case could be made for H.264, just like the rest of the bleeding edge HTML5 functionality comes at some cost of potential audience.
Tim F. – Using system codecs only makes sense if you only care about one operating system, and you have control over that operating system.
On Windows, an H.264 codec is only available with Windows 7. Last I heard, that was still less than 20% of Windows users. The other 80% get nothing. If they wanted an H.264 codec (legally, without violating any patent licenses), they would have to buy one.
Microsoft only cares about Windows 7, and they do control the operating system. They’ll probably just ship an H.264 codec for Vista when they release IE 9, but nobody else can do that.
Same deal with Safari. Apple can get away with using QuickTime, because they control it. If they want to ship QuickTime on Windows bundled with Safari, they can do that. Nobody else could.
On Linux, there’s only one legal H.264 codec. You would have to buy that if you wanted to use it. On other operating systems, there aren’t any codecs at all. You can rely on Theora and WebM being present though.
And what about embedded systems, like phones, tablets, games consoles… They usually don’t have any codecs at all. And yes, Opera, Mozilla, and Google care about these. Opera makes it’s living from selling web browsers for embedded systems, Mozilla wants Firefox to be available on everything with a CPU, and Google uses Chrome on Android and ChromeOS, neither of which have native video codecs.
So, you end up with inconsistent support across different platforms. Apple’s H.264 decoder supports a different subset of H.264 than Microsoft’s decoder does, and changes depending on which version of Mac OS X you’re using. Microsoft’s H.264 decoder is only available on Windows 7. Theora and WebM are normally only available on Linux. WMV is always available on Windows, sometimes on MacOS X, and rarely on Linux.
It’s also a maintainence headache. Since neither MIcrosoft nor Apple care about supporting lots of operating systems, they each have only one codec interface. Opera, Chrome, and Safari would need at least three, and then a fallback in case your platform doesn’t have any codecs at all. That’s just way too much work to maintain, and way too many chances for it to all blow up in your face.
Even worse – it’s a security problem. The native codecs are often very old, and were never designed to be accessible from the ‘net. By exposing them to the web, you expose any security vulnerabilities they have. That’s not just codecs installed by default either – that includes some random set of codecs that might have been installed by the user. You can’t just only allow a predetermined set of codecs either, because that defeats the whole point of using system codecs in the first place.
We seem to be treating licensing fees as a very black and white issue.
Does anyone know the fee structure for 264? It may help put this discussion into perspective
Thanks for the response.
I would say that the argument that allowing system fallback would permit or even encourage laziness or the proliferation of codecs simply doesn’t hold water nor match up to what we see in reality. People aren’t going to do straight dumps from their device (even if they do, the majority is likely to be some profile of AVC) — anything of considerable size is understandably going to be compressed and edited: there is going to be an intermediary phase of editing, processing, or storage of some kind, and the software is without a doubt going to support h.264 (re-encoding isn’t, per say, bad; it’s actually quite necessary — I am just opposed to it strictly to meet an ideological goal).
Of course, images are different. The main one being video is far more complex and expensive (figuratively and literally) — for this reason, I fear that a free, open, web video codec will never be adequate. H.264 is 13 years in the making and in its optimization. We are soon (in MPEG year; it’s likely to be a “while”) to see its successor. This is based on the world’s expertise collaborating openly to produce the best solution for transmission, encoding, decoding, storage, devices, applications, etc, etc, etc… With respect to video, the goal is not to fracture the offline world from the real world — the two should be seamless when appropriate, but the simple case is that the real world will likely always be ruled (in this case, either a significant majority, or a de facto standardization) by a proprietary format. However, that being said, I think there are basic similarities: at the time, the w3c realized that it didn’t have the expertise or influence in the offline world to dictate a web format… and they have been compromising ever since.
“Standards in a document by the w3c don’t really matter. Much of HTML5 developed with WHATWG while the w3c ignored it, and the unwritten standard for jpeg, gif and png still exists. What the web needs isn’t a reference within some standard, but rather, unity (this is starting to sound like a civil rights speech, happy martin luther king day on monday everyone!).”
And I would prefer, and think its best, if the system works itself out and self-regulates. My concern is when those who formally were allowing the options to play themselves out artificially close off options and/or are currently making similar compromises of one type or another but are using this case as an ideological line in the sand, hypocritically. This is becoming a FS issue that isn’t about a basic baseline standard, but rather trying to destroy the value (yes, value) of the MPEG organization (not MPEGLA the licensing body). Also, when I hear those who are supporting Google’s current stance say things like, “there is absolutely no place for proprietary technologies on the web”, etc… This concerns me. There is a place for it; much of the innovation is proprietary. Above openness, I think one of the greatest virtues of the web is its practicality.
@BlueDuck: Agreed with regard to Linux. But it is my understanding that we are really talking about GStreamer or ffmpeg the vast majority of the time. And again, the idea would be as a fallback; I’m not suggesting that they forgo having their own included media frameworks. It doesn’t seem to me that adding an optional fallback to QT on the Mac, the appropriate framework on Windows, or a check for GStreamer or ffmpeg (or any other Linux frameworks used to a reasonable % worthy of support) is really adding that much complexity. Certainly, this seems far less burdensome than the potential mess we are headed for at the consumer/producer level. Of course, this isn’t bulletproof, but even in the cases of Mac and Windows where they do do this on purpose with reasonable guarantee, you aren’t ensuring that a particular codec is actually installed. Again, I am looking for practical solutions for the vast majority of all situations.
And, again, Mozilla has purposefully foreclosed this option for ideological reasons. I thought Opera permitted this — I hope another reader can confirm.
“H.264 has slight advantages almost always, but people may want to stick to the baseline for the sole purpose of being a baseline, to reach the greatest potential audience”
Absolutely, and I am for the marketplace to decide rather than an ideology. I would argue that h.264 is far superior for a large number of reasons besides a “slight advantage” technologically (de facto support; portability across media and applications; a long history and mature understanding; a clear path forward to future improvements: whether or not FS supporters fear submarine patents, commercial businesses are actually greatly reassured by the consortium’s patent position and legal standing, etc…), but even if someone wants to support a baseline — I would wager that, a producer rationally considering the situation, would fine that h.264 virtually serves as a baseline (the cost issue is being overblown, the lack of support for Mozilla and others is being overblown, etc…) AND as the leading edge choice worth paying for. But I digress, I would be fine with the competition or the choice being available — what concerns me is seeing FS supporters making this a wedge issue in an effort to eradicate MPEG from the web, or even from the world, or even to destroy patents. This is not an ideology I can support, while I am very much happy to allow it to exist and thrive on its own.
I am concerned about production of WebM content
No popular video editing software — Sony’s Vegas Pro, Apple’s Final Cut Express/Pro, Avid Xpress, Adobe After Effects & Premiere supports WebM.
So if I want to produce H 264 video for my HTML5 site I can do it easily with any video editing package I use,
but I can’t do it for WebM
Besides big software package like Avid Xpress, Apple Final Cut there are plenty of small packages use to convert/process video, none of them supports WebM
Many camcorder/camera model produce H 264 video, none produce WebM encoded video. AVCHD is based H 264. How much quality of AVCHD content will be lost if I convert it from H 264 to WebM?
If I put H 264 video on my HTML 5 site, I know that my clients will watch it well since most of them have H 264 GPU at their laptops and mobile phone
But it is not true in case of WebM video.
So who and how is going to produce WebM content since it is not supported by video editing packages, converters, camcorders
@BlackAura: “Using system codecs only makes sense if you only care about one operating system, and you have control over that operating system.”
I disagree. I am speaking of it only as a fallback method and as an attempt to cover as many systems as is reasonable. I concede I hadn’t considered embedded systems, but I’m unfamiliar with what types of support they provide for advanced video in the first place. (This is an area where I may be ignorant: what types of embedded systems would support video playback but have no system codecs at all?)
I may have neglected the security angle, but I don’t find that Apple, Microsoft, or the OSS codec frameworks are avoiding maintenance. It seems just as risky relying on Mozilla, Opera, and Google maintaining native frameworks, and certainly better than relying on Flash which is a security nightmare.
Again, I am not looking for a 100% coverage scenario — just a pragmatic, lowest common denominator, broadest coverage scenario that provides fallbacks and options. No browser perfectly reflects the existing spec, and every browser contains non-spec technology — but on this issue, a firestorm of absolutism has erupted — even though the web has always reflected pragmatic compromise. This is what I am seeking to avoid.
(I apologize for a large number of “dyslexic typos” and other errors, that I now see as I look back. I hope my comments are still reasonably intelligible.)
antimatter15, I also wanted to note: regarding the fact that there are many “real world” video codecs still… This is what is truly advantageous, remarkable, and possibly inevitable about h.264 — its broad adoption across all disciplines, applications, and needs. It certainly is the first time in history where virtually every mode of motion video at almost every stage (but certainly at the distribution stage) can be met, not merely adequately, but quite well by a single format (even if through a number of profiles). I do not foresee WebM (not merely by name alone) ever accomplishing that, even if MPEG remained frozen in time and Google had two decades to catch up. And erecting a wall between this reality and the web does not seem logically tenable, even if ideal to some.
Using the codecs in pluggable multimedia frameworks, to me does seem like a valid and extremely dangerous security threat. I’ve seen many times, scary sites that encourage you to download a new video codec that could very well be malware in and of itself, or could be used as an attack vector. Opera, Mozilla and Google and those who actually write the codecs, like the Xiph.org foundation (and Google, yet again) are certainly *far* more trustworthy in creating and maintaining a codec than a three-person group of hackers who wants to try out a new idea because of not-invented-here syndrome and security-through-obscurity DRM. If codecs are pluggable, people will begin to invent their own codecs.
The cost isn’t really an issue because the Mozilla Corporation does make money, and could pay for it, and if they were open to the idea, some company (who supports AVC) is going to fork over the cash. Many FS supporters do seem to be trying to eradicate MPEG from the web, or to destroy patents. Patents do expire (thankfully at a much shorter lifetime than copyrights, but I digress), and in a decade, AVC will be a perfectly logical choice for FS supporters. But this way of thinking, and of treating innovation is profoundly dangerous for both the FS community and the patent-creators. The FS community can not afford to be constantly twenty years behind the times in terms of innovation (especially if you subscribe to the law of accelerating returns), and the patent-creators can’t afford for the FS community to involved in actively foiling their innovations. It’s truly great what MPEG has done, and it’s terrible that such a controversy exists around it.
So if I were to revise my “hopeless ideals”, I think the best would be for MPEG or the ISO standards bodies to eliminate royalties. The fact an industry can agree upon a single codec should be enough of a market and innovation advantage to waive the relatively nominal money from licensing.
antimatter15, I concede there are potential security risks by crossing from the app to the system for media playback that more intelligent people can speak to than I, but I think the security threat you are calling out is not a real risk AS A RESULT OF system codec fallback. Systems and frameworks are already pluggable and there are already malicious trojans embedded in codecs. This doesn’t become a reality by allowing fallback. And it does not spread or become worse as a result either: providing fallback doesn’t encourage codec proliferation, it merely providers optional support when there is a disagreement as to what the 1 de facto standard is while allowing 2 or 3 alternatives. The marketplace still moves toward standardization. We are far past the point where we have not only many competitors but many more new ones entering the fray. We are down to 3 options where the disagreement hinges on purely philosophical differences.
” The FS community can not afford to be constantly twenty years behind the times in terms of innovation (especially if you subscribe to the law of accelerating returns), and the patent-creators can’t afford for the FS community to involved in actively foiling their innovations.”
On the former, I have less sympathy. The FS community argues they are ideologically more innovative, they recognize they are foreclosing themselves to accessing much proprietary software, they argue that their development methods are more principled and beneficent to the world community, etc… If their ideology places them at a disadvantage from an innovation perspective, it is not the commercial philosophy that must compromise to an unyielding perspective. It is up to them to actually deliver and compete on merit. As we have discussed, Mozilla is foreclosing reasonable compromises that would be practical solutions for their customers on ideological grounds — what room does that leave me sympathy for? I can only hope that there is some lowest common denominator for them, an intermediary enabler, or some unthought of solution — I cannot hope that a more successful strategy compromises its advantages and patronizes its oppositional philosophy.
As further minor notes:
1. There are innumerable ways in which programs fall back or call system frameworks — I think it simply disingenuous to say it is not practical to do so in the case of codecs; however, I concede my level of knowledge doesn’t adequately cover all of the potential issues which I am aware are real. I am very happy to hear intelligent conversation on the matter.
2. I realize that the last paragraph of my last comment may seem inconsiderate, and I do not want that to be the case. I respect and use a great deal of OSS and recognize that FS has its own advantages. But I don’t think it’s disadvantages should be a crutch either. So let me see if I can state my point more simply: If free and uncompromising, a decision made by conscious choice, is unable to achieve parity with commercial and very much compromising; why is it the commercial and compromising philosophy’s responsibility and even NEED to compromise? Again, I am rather impressed and sympathetic to OSS, but from a purely logical perspective, that does not compute.
3. I think it is clear that it has become a Free Software issue. As a practical matter, access to the poor and technologically disadvantaged is almost erased. The edge cases where small commercial enterprises or individuals would be denied access to video are virtually incomprehensible. The very act of creating video media requires investments which far outweigh license royalties. The royalties truly are a small portion of device/software costs or an even smaller portion of potential revenues. There are exceptions for non-profit use that cover most cases and little incentive to prosecute non-profit entities as damages have to be calculated based on real losses or real profits. So when we are discussing baseline access on this matter we are really discussing access to those who have consciously chosen Free Software and have foreclosed themselves to proprietary software. This represents a small minority. And I’m not even speaking to general open source advocates who often live in a world of compromise going back and forth between open and closed. (Admittedly, there are concerns about reliance on questionably legal implementations of MPEG standards.) I am speaking of the minority within the open source minority of Free Software advocacy. Even if I had all the sympathy in the world for this group, it truly disgusts me that they are hijacking the debate by invoking the noble goal of access to the poor and disadvantaged for their own motivations rather than being honest about their intentions. (There are many who are honest about it, but they also know their philosophy largely falls on deaf ears.)
Oh (sorry if I am hijacking discussion on your blog), I forgot to speak to this:
“and the patent-creators can’t afford for the FS community to involved in actively foiling their innovations.”
I disagree with this. I think it’s the FS community’s responsibility (even their very purpose) to try to foil the commercial world’s advances. This is competition. However, it should never be done to the detriment of consumers and their community. Which is what I think we are seeing now: they are shooting themselves in the foot thinking they can blindside the commercial world and gain advantage, not realizing they are hurting themselves, their community, and consumers in the proprietary world who would just stare at them, eyes glazed over, if they tried to discuss the nuances of the GPL3.0 or tried to insist that none of the video they own should be permitted on the web until converted to something called Ogg or WebM.
VP8 is not Royalty Free, It is “yet to be patent enforced” there are whole sections of VP8 that use code from the H.264 Spec,
http://www.xyhd.tv/2011/01/industry-news/google-will-have-to-pony-up-to-mpeg-la-why-webm-is-not-going-to-be-royalty-free/
The post on the whole is pretty comprehensive and to the point. I especially agree with your point that both the open source and closed source supporters have to find a middle ground. Each of them should be willing to bend their rules a little in favor of the other.
OR
We could leave the video tag somewhat open by not specifying a particular codec (a la ). But i guess video is somewhat more complicated. (I am not educated enough to take sides). However, given that Microsoft and Apple both have patents in the MPEG-LA patent pool means that they may have some say and maybe able to persuade it to get rid of the royalties
@Srinivasan R, Apple has 4 patents (all related to the container format, which could probably be easily replaced). Microsoft has a considerable number more: 12 (I think, the same patent is filed in several nations so it’s difficult to tell). However, Samsung has at least 32 (again, it’s difficult to determine if you are counting multiple filings of the same patent or not). There are just under 30 licensors. MPEG contributions are based on the essentially and number of contributions. Apple and Microsoft aren’t swaying anyone by pleading or payment. And it’s worth pointing out that Samsung, a leading manufacturer of Android devices, will never license WebM since they’ve given over administration of their mpeg patents to MPEGLA, and the WebM has a patent litigation poison pill. How Google plans to swing that one will be one of the most interesting bits of gamesmanship we see in this battle. H.264 is going nowhere.
Google is playing a game of chicken hoping that MPEGLA will go royalty free, and it thinks the w3c specification and their open source supporters are enough of a wedge. But MPEGLA doesn’t have to blink because they know the market much better than Google does; they’ve been through this before with Microsoft and VC-1 (a paid alternative but at the time WMV9 posed the same threat — the commercial threat, far more dangerous than an open source one, quickly evaporated under patent review). In the meantime, the consumer loses.
Hi
I think you need to correct the statement “H.264 is an open standard”. This is not necessarily true.
For example, in the EU H.264 cannot be called as such due to the pre-requisite of and Open Standard being “The intellectual property – i.e. patents possibly present – of (parts of) the standard is made irrevocably available on a royalty-free basis.” and “It must be permissible to all to copy, distribute and use it for no fee or at a nominal fee.”
H.264 does not meed these qualifications.
ref: http://en.wikipedia.org/wiki/Open_standard#European_Union_definition
Umm, when you say “Firefox browsers that ignore whatever is installed, to use their own bundled decoders.” you should look at my FireFox, which happily uses VLC to play ‘anything imaginable’… but then you’ve ignored ‘nix the whole article, so …
@Tom Chiverton, The VLC plugin is supported by just about every browser to load videos hosted through the embed tag, but this isn’t HTML5 behavior. I’m almost certain Firefox doesn’t offload handling of <video> content to VLC, only <embed>.
zmmz, This is semantics: the same page shows numerous other bodies not requiring free to be called an “open standard.” If saying “open and free standard” suffices for a logical discourse when talking about w3c, and “open standard” suffices to show that WebM is neither open or standard, I prefer the precision of noting “open and free” when appropriate and “open” when appropriate.
A couple of things you missed in your article: flv is a wrapper format, just like avi, or mp4. For many years VP6 inside an flv wrapper was the preferred video format on the web, not sorenson spark. Also, Adobes proprietry h.264 decode support is actually code licensed from Main Concept, and supports the *full* range of h.264 profiles, which was not an omission as such in relation to your article, but still worth mentioning since QuickTime/safari didn’t until QuickTime X
Thanks! Updated article.
I totally agree with Tim F on this:
“But I am perfectly happy with a baseline format competing with whatever else the commercial world and consumers would prefer — in which case, everyone would provide a baseline free codec AND have the option of using h.264, and if h.264 is preferred the vast majority of the time, that’s okay because the free software users and those without financial resources could fall back on whatever free standard (just as .wav is an available option but everyone uses .mp3 and uses javascript, Flash, or some other mechanism to provide playback since that is just as trivial as using .wav)”.
This is the perfect solution that would make both worlds happy.
Liked the article, well written. First sensible charts i’ve seen listed pro/con.
It’s a shame people like Tim F have been posting (or should I say trolling) their opinion on pretty much every Chrome & h264 article over the weekend. Yes, you want browsers to use system codecs, and you dont see a flaw in that… but there are 10s of OS, and only two major ones support h264 out of the box, and in Windows case, only ~30% of Windows users have it. Before DivX/MPEG4 became common around 2000, video shared on the Internet could be found in various codecs and containers, and it was a horrid experience. I’m glad we’ve narrowed it down to soo few, and i’ll be very glad when WebM & Ogg are the common *web* format.
Just a note, I came across your article on *another* site after googling, it was only when I noticed ‘via antimatter15’ at the bottom that I twigged it was … borrowed. → http://buzzgamesnews.blogspot.com/2011/01/best-explanation-i-have-read-so-far.html
That’s cool that others are mirroring the content, I don’t mind, at least it makes the content more available
Though I’ve made some changes to the article after they’ve mirrored it and those versions still include some minor factual inaccuracies.
Great work! A detailed and sensible analysis.
nice write…lets hope people actually read it.
“Mozilla Firefox, Chrome/Chromium and Opera bundle a copy of a customized version of the ffmpeg framework.”
Opera uses gstreamer, as stated on the opera:about page.
@Remco, odd. I thought I corrected that.
You have AVI listed along with a whole lot of codecs that QuickTime supports. AVI is an ancient container format and not a codec. It’s a little confusing.
That part was pretty much copied from wikipedia, so I’ll change the wording to add container formats too.
The summary is fine, but there is a constant assumption that just because video has become easier today (i.e., because Google releases their implementation, an extremely recent development), that in the past it wasn’t extremely difficult and time-consuming to make decent playback possible. Perhaps we should all consider that and actually appreciate the fact that without folks like Adobe, you couldn’t even have watched your precious Youtube lolcat videos.
It’s OK to want a standardized way of processing video, but it is ridiculous to demonize companies that spent time and money on making technology possible when no one else could or would.
Great article cleared up some questions I had on the issue, I also agree that we do need some sort of decent baseline that everyone can use for free.
This is probably the best post I have read on the subject, the only argument I felt you skipped over was that of hardware decoding. A lot of hardware, be it iOS based, Androids or netbooks simply _have_ to use hardware acceleration in order to be able to play video smoothly at the resolution of their screens. The only option for existing devices is H.264 which all of them has hardware decoding support for and which has already been paid royalties for by the manufacturer of the device. With Google’s move they’re just screwing everyone, including people who have bought Android devices capable of H.264 hardware decoding (most, if not all, of them). Also on desktop systems almost everyone has video cards capable of decoding H.264 which the manufacturers have already paid royalties for. In face of this I think the only sane resolution is to use the system frameworks for playing back video. This would probably lead to WebM becoming the standard for free low-res video, since codecs will be available that work with IE and Safari – and to H.264 being used by most commerical players. It will also probably push forward hardware accelerated video for Linux which I understand is not very far along at the moment. I don’t believe in you ‘tower of Babel’ scenario when it comes to video since in the real world there is already a de facto standard and we’re just trying to find an unecumbered baseline.
Things of note…..
Apple and Microsoft are both members ot the mpeg-la and make money from royalties
Theora was on its way to becoming the defualt for but Apple came along and crushed that!
WebM can be freely and easily implemented by microsoft and apple
Webm is not a standard YET but it can be.
It seems to me that from a USER perspective, the best solution would be for the H.264 owners to make it available royalty free for resolutions up to and including 640 x 480 x 30fps, while reserving the right to enforce royalties for resolutions beyond that. That would give us a practical baseline for everybody to fall back to, while leaving a commercial market for high definition video technology.
Nice analysis! I do think you forgot to address one major point though: the fact that there are millions of devices on the market that only play H264 video and not WebM, including many devices with YouTube functions (media players, higher-end television sets, tablets, phones, etc). The overwhelming majority of these will never be upgraded to support hardware WebM decoding, for the simple fact that they are either not upgradable, not capable of accelerating WebM (apparently, according to the x264 developers, WebM is ‘a nightmare for hardware designers’ because of some of the algorithms it uses), or just simple End-Of-Life for the manufacturer. Among these manufacturers are some of Googles biggest industry partners, such as Samsung, LG, Toshiba, Sony, which all put eggs in Google’s basket by selling Android phones and tablets or products using YouTube, Google Search, Google Maps, etc, and which are all also MPEG-LA members.
What does all this mean? That Google can only do one of two things if it wants to stay credible on the H264 issue: a) either they practice what they preach and completely remove all H264 encoded videos from YouTube, effectively crippling devices their partners sold to their customers, or b) they keep making up inconsistent justifications why they ‘want to promote open and free web video codecs’, yet they keep serving H264 through YouTube and Flash.
If they Google chooses option a) their industry partners will be royally pissed off: to them, ditching H264 didn’t bring any advantages in the first place, but now they have to explain their customers why the YouTube functions in their devices don’t work anymore, or spend millions trying to get WebM working on them. I don’t think Google is in a position to do this, as it would probably inderectly kill Android, I can’t imagine any of the Samsungs or LG’s will want to work with Google on that after they pulled a trick like that. So it’s not going to happen. Which leaves option b) Google removes perfectly fine functionality from Chrome, tells everyone to switch to WebM, yet they still serve H264 video through YouTube because they have a business interest to do so. Effectively this tells web developers that they are the ones getting screwed over this, because they will now have to either encode in 2 formats: H264 and WebM, or encode in H264 and serve that through Flash. In any case, they would still have to pay H264 royalties, but now they are also forced to either use Flash, or encode in 2 formats for no good reason.
So either way, Googles move doesn’t make any sense at all. If Google would really care about the open and free web, and about saving commercial parties (note that non-profit H264 use is and will always be free), they should have invested the $100 million they paid for VP8 into buying out MPEG-LA so they could lift the royalties for H264 baseline profile completely (which they already partially did, encoding, decoding and serving web video is already free, only distribution of an encoder/decoder requires royalties). No matter how many arguments I read in favor of Googles move, none of them make sense, most of them are based on hypothetical or ideological and emotional arguments. I have yet to come across the first rational justification for ditching H264 for WebM that doesn’t involve some uninformed rant on how ‘the web is supposed to be open’ or ‘h264 is stifling innovation’. None of these arguments make any practical sense, if anything, H264 is promoting open standards on the web, and has boosted innovation in video codecs. Just because it isn’t ‘free as in beer’ doesn’t mean it is evil.