Google has recently announced their intention to remove the H.264 video codec from its Chrome browser. This decision has been smeared as an evil campaign for controlling video on the web, akin to not-invented-here syndrome. It’s also been lauded as the push that the web needs to remain open and free. Mostly, it’s been marked as inconsistent, due to the bundling of Adobe’s proprietary Flash player.
Richard Stallman doesn’t like the term Open Source because it fails to embody the true meaning of “free software”, and the one thing that’s worse is the word “open”. This debate can’t simply be labeled as one for or against openness (even ignoring the technical details).
H.264 is an open standard. It was developed by a committee, standardized, reviewed by many engineers and developers for multiple companies and has been standardized for use with a multitude of containers and devices. However, H.264 is not royalty free. Software patents in many countries restrict the distribution of software that utilizes the codec to those who pay the MPEG-LA.
VP8 is not a standard. It was developed secretly by a single company, and until recently, had only a single working implementation. The public wasn’t open to collaboration on the specification until the bitstream spec was frozen, including the bugs that existed within. Now, the source code and reference implementation are available under liberal licenses, and all the related patents are irrevocably royalty-free.
Adobe Flash, while not synonymous with a video codec (unless you mean .flv flash videos, which are either VP6 or Sorenson Spark), is going here anyway because everyone feels like comparing it. Flash’s SWF format is not standard, but it is open. There are a few implementations (Swfdec, Gnash, GameSWF, Gordon, etc), but none of them are as complete as the official proprietary implementation. There’s a bitstream specification that anyone can read to create an independent implementation of the player. Implementing and using the Flash player is still royalty-free (Since the Flash VM can decode H.264, obviously that part, not controlled by Adobe still has royalties), anyone can make software that can export SWF animations without paying Adobe.
Implementation vs. Distribution
Or: How Bundling Flash doesn’t violate the Novikov self-consistency principle
| Name | Standardization | Implementation | Distribution | Dev History |
|---|---|---|---|---|
| Theora | Standardized? | Open Source | Royalty Free | Mostly Open |
| VP8 | Not Standardized | Open Source | Royalty Free | Mixed |
| H.264 | Standardized | Open Source | Royalties | Open |
| Flash* | Not Standardized | Proprietary | Royalty Free | Proprietary |
* Adobe Flash isn’t a video codec
Each column of the chart can be treated as one part of “open”. The <video> debate now primarily concerns VP8 and H.264, as Theora is inferior for technical reasons and Flash isn’t a video codec. I don’t know if Theora’s actually standardized, but it has multiple implementations and was developed by the Xiph.Org foundation, and is the closest thing to an actual standard, short of being a product of MPEG. Theora was created from On2′s VP3 code, but was changed significantly by the Xiph.org Foundation before release, so the development history can be considered mostly open. VP8 as announced, and the bitstream was frozen shortly afterwards, leaving little community involvement. For the years before its debut as part of WebM, the format was secret, patent encumbered, and proprietary. At least now, the source code is open and people are free to do whatever they please, but the open source video community probably had very little say in the development of the codec. H.264 has an open software reference implementation, as does VP8 and Theora. Flash’s de-facto implementation is Adobe’s, which is proprietary.
The one that concerns Mozilla, Opera and Google is the distribution rights; whether or not royalties have to be paid. Notably, this is where the parallel drawn by most critics of Google’s move with H.264 and Flash comes into question. Many people believe that Flash is the epitome of evil, and proprietary software, and embodies everything wrong with proprietary software. The distribution column is arguably the most important, and it’s the one that fundamentally determines whether or not something acts as a detriment to “open innovation”. Flash animations and applications can be created and distributed without paying royalty fees to Adobe, regardless of the viewership. Innovation is still permitted as long as the distribution is free (except with changing the inner workings of the proprietary implementation).
The Ambiguity of “Plugin”
TODO: Use this subtitle to mention random tangential thoughts.
It seems there are lots of misconceptions about WebM, and new ones as well, because of the use of the word “plugin” by Google in their “More about Chrome HTML Codec Change“. Here’s what the relevant part of the post says (it’s buried into the last paragraph).
This is why we’re joining others in the community to invest in WebM and encouraging every browser vendor to adopt it for the emerging HTML video platform (the WebM Project team will soon release plugins that enable WebM support in Safari and IE9 via the HTML standard <video> tag).
Microsoft and Apple through IE9 and Safari, respectively, rely on the underlying operating system’s multimedia frameworks to handle <video> decoding. Chrome and Chromium bundle a customized version of the ffmpeg framework, while Opera uses gstreamer and Firefox has its own framework built on various open source codec libraries.
| Browser | WebM | H.264 | Theora | Anything Imaginable |
|---|---|---|---|---|
| Firefox | Version 4 | Never | Version 3.5 | Never |
| Chrome | Version 6 | Removed in Version 10 | Version 3 | Never |
| Opera | Version 10.60 | Never | Version 10.50 | Never |
| Safari | QuickTime | Default QuickTime | QuickTime | Default QuickTime* |
| IE9 | Windows Media | Default Windows Media | Windows Media | Default Windows Media* |
* Well, not really *anything* imaginable, but a lot of them
QuickTime and Windows Media are pluggable multimedia frameworks. And since Safari and IE9 use them for <video> support, any codec or container format supported by the respective frameworks, works in the browser. For QuickTime, the list for videos goes along the lines of 3GP, Apple Video, AVI, DV, Cinepak, H.261, H.262, H.263, H.264, Microsoft Video 1, MPEG-1, MPEG-4 Part 2, Motion JPEG, Pixlet, Planar RGB, Sorenson Video, Qtch, QuickTime Movie, and QuickTime VR. But these media frameworks are pluggable, which means they play whatever codecs are installed. It just happens that those codecs listed above are plugins that are installed by default, somewhat analogous to how Google Chrome bundles Flash. The plugin frameworks for multimedia don’t treat plugins any different from “native” codecs. There’s absolutely no way to tell the difference from a user perspective. Right clicking a video will give you the same ordinary context menu.
Codec packs are often installed by users anyway. A codec pack consists of a set of plugins for the operating system’s multimedia framework. Once one of those codecs are installed to the system, support is available to the applications that rely on the OS’s framework: QuickTime Player, Windows Media Player, IE9 and Safari.
Notably absent are the Opera, Chrome and Firefox browsers that ignore whatever is installed, to use their own bundled decoders. There’s a reason for this, and it’s pretty easy to spot. Firefox, Chrome and Opera are also the only ones on the list that aren’t tied to a specific operating system, and bundling your own media framework lets you keep the functionality consistent. IE and Safari aren’t cross platform. Well, Safari works on Windows, but it still requires QuickTime for Windows to be installed to play the video content.
You also don’t want all the other video codecs thrown into the mix. Standards are about standardization, where you have a limited number of codecs instead of an entire forest of them. This brings us to a bit of History, I’ll begin with the olden days. I can’t tell you too much, as much of the first section happened when I was 6 years old- and certainly wasn’t into the negative user experience of platform-specific multimedia plugins.
A Brief History of Stuff Named “A Brief History…”
Uh… I mean, Video on the Web.
Before the popularity of HTML5 Video, or event Flash, video was viewed on the web using the RealVideo, MPlayer, VLC, Windows Media Player, and Quicktime Plugins. Note that pretty little plugins aren’t plugins to the underlying multimedia frameworks: these are the nasty type of plugins that Flash is among. They’re the plugins that take a long time to load, have interfaces that never look like the surrounding page or browser, only work on certain operating systems, at best, inconsistently. This is the epitome of what standardization is meant to prevent, and the distillation of everything that’s wrong with plugin based video.
Flash’s popularity (probably thanks to YouTube), brought a semblance of standardization to the industry. Video on the web was delivered through the flv container, encoded either with Sorenson Spark or, in later versions of Flash, On2′s VP6. A while later, H.264/AVC which was later added to Flash, and as the superior codec, most people switched to that and FLV is slowly fading away.
HTML5 video became popular recently. I’ll say it’s probably because of iOS, since that’s the only way to get web video to play there and the rest of the aggressive marketing that Apple does. This brings us to today (or at least the general time period of early January 2011 in which I’m writing this post in). Google has just announced it’s intention to remove H.264 from an upcoming revision of the Chrome browser (much like how the Chromium browser never supported H.264). And everyone on the internet that cares enough to say a word is going insane. Which brings us back to the last section, about how Safari and IE9 use the operating system’s underlying multimedia frameworks and how all the other browsers (Firefox/Chrome/Opera) that work on my beloved operating system (Linux) bundle their own.
If Firefox, Chrome and Opera used the OS’s media frameworks, we would be set back into the dark ages when people used every video codec imaginable and nobody would be happy. Standards exist for consistency, and it would be terrible if people started to go on a cost saving measure to never again transcode video, and serve it straight from the server’s filesystem as AVI files containing DV or god forbid MJPEG (the same format the users’s uploaded!) because all the Safaris and IE9s could play it fine. The fact Firefox, Chrome and Opera bundle their own multimedia frameworks forbids the possibility of this, because those browsers (in the forseeable future) will never support anything other than WebM, Theora and potentially H.264.
There’s more to H.264 than just H.264
This part doesn’t make much sense.
Often, the argument against VP8 is that it’s inferior to the H.264 codec, and to me, this seems like the most ideologically valid concern. But in a lot of cases, it stems from a misunderstanding of how H.264 works. H.264 is not a single video codec (not even to mention multimedia container formats), but rather has several Profiles that work on different devices and implementations. Rarely are videos encoded only in MPEG-4/AVC H.264 Extended Profile at 1080p and 60fps. Go on the Apple store and you’ll see that every iPod device you can find (including the iPhone), includes what video codecs are supported. And it doesn’t just say H.264, but rather, something much more wordy like
Video formats supported: H.264 video up to 720p, 30 frames per second, Main Profile level 3.1 with AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats
For the insanely great iPhone 4. Google is meaner and doesn’t make the information about profile support on the Nexus S quite as accessible (though it’s not the only reason I’m an iPhone user), but it should be safe to assume it’s something along the lines of what the iPhone 4 supports. The iPod Classic has a nice, even longer string, that represents even less support for H.264:
H.264 video, up to 1.5 Mbps, 640 by 480 pixels, 30 frames per second, Low-Complexity version of the H.264 Baseline Profile with AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats; H.264 video, up to 2.5 Mbps, 640 by 480 pixels, 30 frames per second, Baseline Profile up to Level 3.0 with AAC-LC audio up to 160 Kbps, 48kHz, stereo audio in .m4v, .mp4, and .mov file formats;
This is because very few devices can actually utilize all the great features that H.264 defines. Wikipedia has a nice pretty chart. So the point of all of this is to say, even though VP8 is inferior to H.264 from a purely technical standpoint, you probably can’t just use the Main or Extended profiles to support all the devices that “support H.264″. Does this invalidate the inferiority argument? Nope. Dark Shikari said “I expect VP8 to be more comparable to VC-1 or H.264 Baseline Profile than with H.264″. But the large number of devices that support H.264 might actually only support the baseline profiles.
My Hopeless Ideals
Better be blunt with what will probably never happen.
VP8 is a bit worse than H.264, and had it been a patent encumbered video format, there would be almost no reason to prefer it over AVC. The <video> part of the HTML5 specification states:
It would be helpful for interoperability if all browsers could support the same codecs. However, there are no known codecs that satisfy all the current players: we need a codec that is known to not require per-unit or per-distributor licensing, that is compatible with the open source development model, that is of sufficient quality as to be usable, and that is not an additional submarine patent risk for large companies. This is an ongoing issue and this section will be updated once more information is available
The only major codecs that are royalty free are Theora and VP8, and the former probably isn’t of sufficient quality. Both of them come with patent risks (or at least that’s what MPEG-LA wants people to believe), leaving a set of zero acceptable codecs. For this to work at all, something needs to be compromised.
H.264 defines a baseline profile that all decoders could be reasonably expected to handle. Such a profile exists so that the video can be viewed, albeit with inferior compression on a variety of devices and platforms with limited computational ability. The internet was built on interoperability, and HTML5 needs an equivalent “baseline codec” for the web. Something that compresses video at sufficient, though not bleeding-edge quality. Something that can be implemented and distributed openly on all platforms.
For HTML5 <audio>, nearly all browsers implement the basic WAV codec. It serves as an acceptable baseline for small snippets of audio to be played in a cross-brower manner. It’s usually uncompressed, and for most applications, inferior to MP3 and Vorbis. <video> needs similar treatment. That niche is, at this time, best filled by WebM/VP8. If Apple and Microsoft were to add support for the WebM format, it would only improve the environment for open innovation. H.264 can be considered the “bleeding-edge” codec, heck, they could even add HEVC/H.265 to set off an environment of useful competition (or not). But a “baseline codec” should be established first. Something that publishers can encode their videos in, so that any modern browser can view.
Right now, the discussion has been polarized between free software advocates who often seek to eradicate proprietary or patent-encumbered ideas from the face of reality and those who hold a disregard for the open source values. This way of thinking, and of treating innovation is profoundly dangerous for both the free software community and the patent-creators. The free software community can not afford to be constantly twenty years behind the times in terms of innovation (especially if you subscribe to the law of accelerating returns), and the patent-creators can’t afford for the free software community to involved in actively foiling their innovations. It’s truly great what MPEG has done, and it’s terrible that such a controversy exists around its adoption. The best scenario would be for MPEG or the ISO standards bodies to eliminate royalties. The fact multiple industries can agree upon a single, high quality, interoperable codec should be enough of a market and innovation advantage to waive the relatively nominal money from licensing.
Will this actually happen? I doubt it. Apple seems firmly invested in the success of H.264. Microsoft may be more willing to add VP8 support if and when the format gains popularity (and especially if Google adds patent indemnification just so they can see a lawsuit made). Mozilla will likely never compromise on principle and Opera might add AVC one day. Given the huge backlash, Google’s the most likely to revert its opinion and add AVC back into Chrome in a future release. Yay for hopeless idealism! And since I’m espousing hypothetical and impossible ideas anyway, why not vouch for reform of the U.S. patent system as well? If you think of it, the fact it’s such a controversy and that people need to use an inferior product in order to innovate, that smacks in the face of the very Article 1, Section 8, Clause 8 of the United States Constitution:
To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.
And this ends what may be the longest blog post I’ve ever written, revised three times. If you’ve actually read this through, please consider commenting or sending me a tweet at @antimatter15 (or following me, that would be great!). Also, if I’m misinformed, please inform me and I might revise this yet again.
I’m sure Google would have tried to buy out all the H.264 licencors but I have huge doubts that’s possible even if they used billions of dollars toward that goal. Removing AVC doesn’t make any practical sense, but there’s no alternative. They already spent a hundred thirty million on buying On2, and might as well try for the greatest possible impact. I don’t think you can call freedom evil by any stretch. There are costs and casualties (though I doubt anyone’s died over the issue of free video codecs) to any shift. I can’t say that Google’s move is the right one, but it can be justified.
This is a very well written article, and I enjoyed reading it. I look forward to your thoughts as the debate continues, although I don’t envy your reading of the miles of comments. I am supportive of Google’s position, and your conclusion on WebM being the most appropriate baseline choice for HTML5.
I am not sure if it a typo but you wrote regarding the WAV audio file format:
“It’s usually uncompressed, and for most applications, inferior to MP3 and Vorbis.”
As an uncompressed non-lossy format, it is most certainly not inferior to either of them in terms of audio quality. If you meant inferior in terms of file size, that didn’t come through.
Thanks for an informative article!
Headphonomenon,
I took “inferior” as referring nearly entirely to file size. Obviously, for tings like sound effects less than 5 seconds, .WAV will work fine. (For things like short sound effects, 8-bit, 22 KHz or even 11 KHz mono WAV is fine, and sufficiently small as to be not worth worry about the size.) But for larger things, like music samples of 30 or 90 seconds, or voice tracks accompanying a presentation, WAV is horrendous because of the size requirement.
But, when it comes down to it, WAV is a known baseline.
http://x264dev.multimedia.cx/archives/377
from an x264 insider so maybe a little prejudiced but they believe chunks of the WebM VP8 code are direct copies from patented H.264 code: “VP8′s intra prediction is basically ripped off wholesale from H.264… Chroma prediction modes are practically identical as well”
This implies a possible patent breach. Maybe they’re just trying to stall it but WebM may be neither free nor open.
Maybe Google has had, or will have, the whole thing rebuilt?
I strongly doubt Jason Garrett-Glaser, the author of that blog post has anything against VP8. He’s one of the people who developed and optimized ffmpeg’s VP8 implementations. As for the ripped off code, he notes “Update: spatial intra prediction apparently dates back to Nokia’s MVC H.26L proposal, from around ~2000. It’s possible that Google believes that this is sufficient prior art to invalidate existing patents — which is not at all unreasonable!”. Ideas that are mathematically and theoretically equivalent things, implemented slightly differently such as arithmetic coding versus range coding can apply to only the former.
Video compression is very complex, and builds on decades of work before. The lineage of AVC starts off something like JPEG, which became MJPEG, and after basic interframe prediction became MPEG-1, with a few changes became MPEG-2, then MPEG-4 and new MPEG-4/AVC H.264.
The best scenario would be for MPEG or the ISO standards bodies to eliminate royalties
The best scenario would be for the legislatures of the world to eliminate the scourge of patents forever (perhaps with a handful of exceptions). Patents are evil. The next best scenario would be for the MPEG / ISO “standards” bodies to refuse to putatively “standardize” patent encumbered “standards”. “Patented standard” is an oxymoron. Third would be for the MPEG-LA to eliminate royalties indefinitely for all open source software implementations, and to reject, repudiate, and forswear the idea of collecting royalties on content forever. That might actually work.
@Headphonomenon
I’m not sure if the WAV file format is limited to 16 bit sample resolution but if that is the case, WAV actually IS inferior in terms of quality as well. Since formats as MP3 or Vorbis are not limited in bit resolution, as far as I know, one could always create a MP3 file that is a better approximation to the original wave spectrum than the WAV file. Anyway, I don’t think that is of any practical relevance.
@mattviator
The amount of money MS and Apple get from royalties doesn’t cover the costs to them of being a member of the MPEG-LA group.
“Theora was on its way to becoming the defualt for but Apple came along and crushed that!”
No one but the free software folks had ever heard of Theora. Quicktime and Windows Media were the two prominent formats.
“WebM can be freely and easily implemented by microsoft and apple”
I’m going to assume you have never worked on software, especially a codec. Free? Hardly, they have to pay someone to do it. Easy? To do it right? Hardly.
A great post.
A great load of tripe is written about the morality of open- vs closed-source. The assumption is that open-source is always better than closed. What I like about this post is that it shows a few of the cracks that that moral stance papers over.
I am not convinced that Google have made the most moral decision in dropping support for H.264 from Chrome. They are trying to own the standard web video format in the way the MS have done historically. That they have ‘opened’ it does not mean that they do not own it and would not have the greatest influence on future directions.
JPEG and MPEG are great formats. They were developed in an open way with industry support. Participants who spent time and effort on creating the formats want some financial reward for that (in terms of royalties). I expect to be paid for the work I do and have no problems with others being paid for their work. It seems that Google think differently.
The thing about Google is the way they make their money. They aren’t selling products – but their revenue would not be secured unless they released products. They are a great example of how the brand has become as important as what they do. They can appear to be ‘doing no evil’ by releasing software to the ‘community’ (because, obviously, I’m in a community with other Chrome users – we’re all great pals and share the washing up, child care and car cleaning duties); but that software secures their revenue by giving them more control of web formats.
All companies try to maximise profits for the sake of investors. If we’re lucky, we benefit from the creation of those profits. But don’t try and kid yourselves that morality is high up on their agenda.
What about Dirac? This codec was designed with the same goals as H.264 (same MPEG-2 quality at half the bitrate), scales from web content to HD, is patent free, royalty free, open source, and has been tested in real world scenarios by the BBC for several years. Even more, the Dirac codec is even mentioned in the W3C Working Draft for HTML5 tag. Sounds like a good candidate to me.
Instead, Google chooses an obscure codec from a private company, buys it, freezes the non-existent written spec before opening the source, keeps all the patents, and just provides a royalty free license (and good luck with the code if you don’t even have a spec). Open development? Innovation? Yeah, for sure.
From what I’ve read, Dirac isn’t nearly as good as VP8 or H.264 for low bitrates (It may be a good candidate in the future when everyone has gigabit internet speeds). Dirac is based on Wavelet compression technology, unlike H.264, VP8, Theora, JPEG, MPEG which use the discrete cosine transform. Wavelet transforms are apparently much more computationally intense and end up making the picture blurrier (and codec quality metrics like PSNR tend to favor blurriness).
Thay have been arguing about this since 2007, and still no conclusion is made
http://lists.whatwg.org/htdig.cgi/whatwg-whatwg.org/2009-June/020620.html
One thing I didn’t see mentioned is any talk of the quality of the current WebM player.
On the Mac, Chrome has been updated such that it uses WebM playback for YouTube. It turns out if you measure the CPU load, WebM video playback uses MORE CPU than even the Flash player, and much more than h.264 playback!
How is that supposed to go onto mobile devices? Why would I prefer this chunky player over even Flash video (which I previously hated for CPU load) when performance is even worse?
As far as I can tell WebM is really being pushed not by Google, but by a secret cabal of fan manufactures that want to sell bigger fans for laptops and break into the until now, previously quiet and fanless mobile market.
I wouldn’t say Apple is “firmly invested in the success of H.264″ so much as that H.264 decoding is nearly ubiquitous in both mobile and desktop GPUs these days, and hardware decoding is the only way to get practical video playback on mobile devices (otherwise you’re running the CPU flat-out and killing the battery). iPhone plays H.264 well because that’s a feature of it’s GPU. Most Android phones also have hardware H.264 decoding, and all Nvidia and ATI desktop GPUs have it as well (as does Nvidia’s much-touted Tegra 2 chip for mobile devices).
So what’s actually happening is that Google is shooting its phone partners in the foot. Widespread non-Flash H.264 video on the Web is just as good for Android (and Windows Phone 7, for that matter) as it is for iPhone. Google does have the ace up it’s sleeve of real licensed Flash Player on Android, but it’s using Adobe’s god-awful Linux version and the end user experience of H.264-in-Flash on Android is distinctly inferior. Amusingly, the Linux Flash Player is now actually getting some hardware acceleration support because of the necessity of making it tolerable on Android, but they’re a long way from having it be as fluent as the Windows version.
Sory if I missed something but there seems to be no mention of a very important point in that debate: hardware decoding. I mean… Is it event technically POSSIBLE to play webm or theora on an iPhone without frames skipping ? (or any android phone – through flash or not)
There’s one important thing missing from your piece : hardware acceleration. It is critical for devices like the iPhone – or for that matter, Google’s Nexus phones – as a software decoder not only kills the battery of such devices, but their processors might not even be up the task for smooth playback. There is simply no other option for these than H.264. It’s not in Google’s interest to see H.264 dropped in favor of WebM, in the way that it’d make online video unusable on Android phones (and Flash only seems to make it worse on these devices). Apple certainly doesn’t want the iPhone to become crippled when it comes to online video, hence their unabashed support for H.264.
Second, I don’t think Mozilla would include H.264 if the MPEG-LA were to drop irrevocably their royalties on it. It’s still be a proprietary software, and I don’t think that’s something the people there want to promote in any way, as per what you said about dogmatism.
Lets get straight to the point. This is a war about opposing business models and is not about open or closed or whatever everyone pretends. The fact is most web companies ( google / Mozilla et al being the top of this group) make money harvesting user information and reselling it. Their business model works on having a huge amount of users in order to pimp that user data. That process takes a while so they need the lowest costs possible in operating their businesses. Thats why everything has to be FUCKING FREE!
Everyone else in the world (the Sonys, Apples, Panasonics etc..) actually sell things and just needed to get together to create a codec that helps them sell more things. They make money in the normal, less opaque way as opposed to the new web guys so they have no problem with actually paying small royalties. They’ve been doing It for years (cd, DVD etc..) and just see it as a normal cost of doing business.
At the heart of this “web economy” isnt some “freedom for all” bullshit. They all want to make money but they don’t want to pay to do it. In fact google and Mozilla don’t sell anything but your user search habits so they will never fit into the normal model of business that everyone else is ok with. Android , chrome OS and everything else in between is all based on the same principal. And the more web firms they can encourage to run with them on this “pimp the data” crusade the better it is for google because they are the gatekeepers of the “pimp that data crew”.
WebM is a hustle.. A means to an end, just like everything else google does, they want your data, nothing more nothing less.
The reason the post doesn’t have anything on hardware decoding is that whatever people have already reported on seems mostly accurate to me and I don’t know much about that aspect of it. H.264 is widely deployed throughout the industry, and one of Android’s closest partners, Samsung owns many of the patents related to AVC. I’ve read in some places that the hardware acceleration comes in the form of accelerated functions for handling discrete cosine transforms, that may allow for drivers that allow accelerated VP8 decoding. But I have no idea.
However, it seems to me that it should be expected for a publisher to re-encode videos on mobile platforms since you probably have to optimize the profiles and resolution for limited bitrate anyway. But again, I don’t have much knowledge about this area and I won’t pretend to know everything about the debate.
The companies opposed to standardizing Theora were primarily Nokia and Apple (Google also opposed it, but only because it was not of sufficient quality for YouTube). Incidentally, they’re also the hardware manufacturers. The billions of devices Apple has produced, each with hardware H.264 support, serve as a sort of investment.
Another x264 developer, Mike Melanson, happens to work for Adobe has some interesting insights. There’s some pretty interesting information on his blog as well as the Penguin.SWF blog. One post that I found really fascinating was “Solving Different Problems” which discussed why flash video is more computationally intense.
This is a great overview of the issue from the perspective of software developers and other computer nerds. Sorry to break it to you, but you’re not the most important people in this discussion.
MPEG-4 is royalty free for both content consumers and content producers. It is already built-into the hardware of all of the devices that both content consumers and content producers use. And both content consumers and content producers are indemnified against all patent liability by MPEG-LA. When you also add in that MPEG-4 offers the highest quality, there is just no good reason for a content consumer or content producer to use anything other than MPEG-4. Therefore, the conversation is happening in MPEG-4. People are producing video in H.264 with pro cameras and smartphones, they are editing in iMovie and Final Cut or Avid, they are burning to Blu-Ray, they are playing that video on the Web. They are tweeting H.264. There aren’t enough people who care to transcode, let alone enough CPU cycles.
So you’re telling me I should take the time to transcode to some non-standard format and expose myself to patent liability just so the quality of the video and the size of my potential audience can go DOWN? Even if I wanted to, why would my lawyer let me do that? Because it makes some software developers feel better about their religion?
The few people who do pay MPEG-4 royalties all make more money because they are using MPEG-4 than if they used another codec. For example, a video retailer such as iTunes sells so much more video because of MPEG-4 than if they were publishing in Ogg that the 2 cent royalty to MPEG-LA pays for itself. Would you rather sell 1,000,000 copies and pay a 2 cents royalty or would you rather sell 1,000 Ogg copies with no royalty? (I am being generous to Ogg with the 100:1 ratio by the way.)
As for Chrome, Firefox, and Opera, they should send open standard H.264 to the operating system and thereby to the GPU so that it can be decoded with maximum quality, efficiency, and minimal impact on battery life. End of story. The user didn’t buy the H.264 decoder in their device so that software could ignore it and take 4x the battery life to decode a video in software. Software developers need to realize that they are running inside a next-generation DVD player. Do not circumvent the DVD player to playback some funky format on the CPU and cause the user’s device to run out of battery at 6 PM instead of 10 PM.
Further: Chrome, Firefox, and Opera are alternative browsers that only run on a minority of desktop computers. How they think the world should be matters less and less with each passing minute. Today, every mobile device and PC comes out-of-the-box not only with H.264 decoders in their hardware, but the browser they ship with supports the playback of H.264 video via the HTML5 video tag. EVERY SINGLE DEVICE. Mobile and PC has HTML/H.264 open standard video.
If you want to sabotage your ability to view standardized video by installing Chrome, Firefox, or Opera on your PC, then be my guest, by all means, please do that. Since those browsers all have the FlashPlayer plug-in with H.264 playback, in many cases you will get a fallback FlashPlayer instance to view the same H.264 as everyone else. Going forward, though, this will become more and more rare because Flash is expensive and tricky and the number of users who require it goes down exponentially every year. Especially as mobiles replace desktop systems.
In short: this debate is already over. H.264 players and content are already universal, even in alternative browsers, which are only going to get less important anyway.
Finally, I submit to you that what Google is doing with Chrome is not pro-VP8 or pro-Web but actually pro-YouTube. There are only 2 kinds of universal video: H.264 because it is an open standard codec, and YouTube, because it is uniquely multi-codec. The less universal H.264 becomes, the more there is a need for YouTube to abstract those problems away. So if Google is not willing to remove H.264 from YouTube then they are hypocrites playing anti-competitive games.
God this shit is fucking complicated. I work in IT which makes me a tech-savy god to most of my friends and family and this gives ME a headache. All someone like my Mum cares about is watching the same video on any and every single device without every seeing any form of plugin or codec or having to worry about the behind the scenes technical wizardry to make it work Farrrrrrrkkkkkk!!!
“MPEG-4 is royalty free for both content consumers and content producers.”
It is only royalty free for end users of Internet video that is viewed free of charge. Any other product that encodes or decodes h.264 is required to pay licensing fees.
Bill, I’m unsure why this is such a surprising gotcha (the goal is free access to those who need it — is someone going to charge for their videos but less than licensing fees so they actually lose money rather than breaking even or profiting? so they are making money using the IP of others in a video format that they presumably chose purposefully because it presented their own commercial video product better than other formats), but what’s interesting is that this is not even wholly true:
http://www.zdnet.com/blog/bott/a-closer-look-at-the-costs-and-fine-print-of-h264-licenses/2884?pg=2&tag=mantle_skin;content
If the video is under 12 minutes, you can actually charge for it without paying licensing fees. (Which dovetails interestingly with the “short snippets” of .wav scenario we’ve thrown around here and discussion that, for all intents and purpose, when needing free access it is free (even if licensing requirements which could require payment could be applicable under some scenarios).
I agree with the posts describing Googles business model (basically they are the biggest vertically integrated advertising company in the world) and how this is all about business and not about ‘openness’, standards or even ideology.
In broadening the debate from one of ‘openness’ and ‘standards’, which this post covers well enough )even though like others I think it’s a furphy*) it’s worth remembering that HTML5 and other technologies associated with it exist for the exact reason the WC3 couldn’t get their open-standards-free-everybody-in-nobody-out shit together on the next iteration of web standards.
Apple and Google sore a need to make progress where WC3 couldn’t and put significant resources (Apple at any rate) into webfonts, animation, video, client database etc etc (not all strictly HTML5 but often referred to in that way). Then they gave this IP to the world, free of licensing.
I think the real conspiracy atm is that Adobe, Google and MS have decided Apple is getting to big, to far ahead and too influentual and so in CEO Art of War speak are the “common enemy”. One implied effect and possible motivation for this whole BS is trying to prolong the life of Flash (as the default video play amongst other uses) which is at serious risk with 50 million devices (and growing) devices out their that can’t play Flash. I’ve seen recruitment ads for Web developers saying our client says we can’t use Flash because of iPhone. That simple. It’s a war out there. (I think Apple’s position is totally justifiable in the same way as them dropping the floppy drive 5-10 years before PC box makers did).
*[[http://en.wikipedia.org/wiki/Furphy | Furphy]]
Actually, the discussion also involves open web advocates (Opera, Mozilla, etc.). As we know, H264 can never become an open web standard. On the other hand, the W3C could easily and freely take WebM and standardize it, even without Google’s permission.
I agree, but many of the members of the H264 patent pool have shown that they would love to close down the web.
I doubt it. The “backlash” is, to a large degree, orchestrated by commercial interests, and run as astroturfing campaigns.
It seems like you’re a bit too dismissive of the concerns over potential lawsuits regarding WebM patent infringement. Unfortunately, there are very few people who have comprehensive understanding of the compression technologies involved in both codecs as well as what patents that are behind those technologies, so most of us can’t do much more than take the word of those that claim to have that understanding. But to some extent, it doesn’t matter so much whether WebM infringes said patents as whether it can be construed as doing so convincingly enough to inflict expensive legal costs on those who use the format.
Telling Apple, “Hey, use this codec. You might get sued, but you’ll *probably* win the lawsuit.” is not a very good sell.
A couple of things to realize here…
First.. H.264 is not a “bleeding edge” CODEC in the real world, it’s the very definition of mainstream. It’s the video standard used by most satellite broadcast, some cable, even OTA HD in Europe. It’s the most common CODEC used for video recording on camcorders. It’s one of the three primary video standards on Blu-Ray. And as mentioned, on the iPod and most other PMPs (even the Zune). Every smart phone and tablet includes specialized hardware for H.264 decoding (some are general enough to assist on any DCT-based algorithm, which includes VP6, VP8, MPEG1/2, and WMV/VC-1). So, patents or no patents, it’s important to consider just what H.264 actually is these days.
Dirac was invented by the BBC for digital video archival, and the state of the CODEC is still way behind most others, in terms of usability by regular content creators (eg, there are slow, experimental versions that run on Linux, and that’s about it). I’ve played with it a little… not sure it’s quite comparable to H.264, but it’s good… and no way would I wait the insane encoding times to use it on a regular basis. That can be fixed. What can’t, at least in the short term… Dirac is a wavelet CODEC, not DCT-based. So every portable device would have to decode it on the CPU (or maybe a spare DSP, if you have a TI chip in your phone)… not the dedicated, low power hardware used for H.264. So this would be slow and power hungry — not a win.
Standards are great, but oddly enough, Apple and Microsoft are doing the video thing right, not Google, Firefox, and Opera. The video decoder doesn’t belong built-in to the web browser any more than the window manager or the file system… this was long ago a defined part of most operating systems. Your built-in video CODEC will be using far more power, and delivering poor performance, compared to the hardware optimized CODEC you’ll get though the operating system.
Here’s an example. Run VLC (which is using much the same FOSS components for video playback, self-contained, as Chrome) on a 1080/60p video stream. On my AMDx6 system, this is going to take 40-50% of all CPU, and streaming a 28Mb/s stream from a 7200RPM SATA drive, it can’t manage to keep the full framerate… it’s jerky as hell. Now play that same video from that same drive (or for that matter, a USB 2.0 drive) in Splash Lite, or even Windows Media Player (on a WIndows 7 machine, at least), and I get 5-7% CPU use.
Why? These other players are using system level video acceleration though my GPU. Splash is hitting a nVidia GPU API directly, while WMP is just calling Microsoft’s own H.264 CODEC (same as any other program would, using the OS-provided media frameworks), which in turn uses DXVA 2.0 for video acceleration.
Same situation on portable hardware, but the need is even greater. Sure, no one’s likely to be streaming 1080/60p, today…. though even YouTube supports 1080/24p streaming these days. On a PC, using Chrome just means more heat and less free CPU for other things, versus IE9. On a tablet or smartphone, skipping the OS-defined CODEC means, if the video even plays in realtime, you’re using 10x-50x more battery. That’s unacceptable.
In fairness to VP8, it can probably be accelerated on many, if not all, such portable devices just as well as H.264. But this usually requires some proprietary involvement — the acceleration engines in many if not most of these devices are not public documents. And you’d need root access to install your own. Maybe Google can get “everyone who’s not Apple or Microsoft” supporting this — make VP8 a standard thing to support in Android, etc. Maybe that’s a good long-term strategy. But today, both Flash and H.264 support is required to see most of the video on the web.
The question boils down to, “what was this move by Google really about?” Here’s a clue, it has nothing to do with being “open” or using standards much less “do no evil”. It’s about providing Google with a competitive edge. Why? Let’s see…
There is no technical advantage to WebM. Yes, unlike Ogg Theora, it’s at least in the same ball park as H.264, but still inferior none the less. If Google were interested in WebM alone, then they would support WebM along side their support for H.264. Up until recently, Google had the best approach to the codec war by supporting all of them. This time, they decided to back the wrong horse though. As others have mentioned, tons of content is being created in H.264 these days. Nothing is being created in WebM. Most video cameras, cell phones, etc. generate H.264 video. Blu-Ray uses H.264, etc. Why? Because it’s the best. Mobile devices depend on hardware support. Just as there is no content for WebM now, there is no hardware support for it either. Further, in the unlikely event WebM ever did get traction, we all know there will be patent infringement cases brought on very quickly.
Google knows this. I don’t believe Google actually expects WebM to take off. This is clearly a shot against Apple’s iOS based devices. Google knows that people will continue to create H.264 content. Chrome (like Android) will still be able to play H.264 content, only through Adobe’s Flash plug-in. Google knows Apple will not put Flash on their iOS devices (for obvious reasons). At least that’s Google’s plan.
The reality will be a bit different. IE9 will support H.264 video natively as will Safari. This means the default browser for both Windows and Mac will handle H.264 from the tag natively. Combine that with the huge success that Apple is seeing with their iOS based devices (iPod Touch, iPhone, iPad, etc.) and you see a bright future for HTML5′s tag with H.264 video.
Look what happened to Firefox. They were on a tear, stealing away market share at a very good pace for a few years. Then something happened and they hit a brick wall. Sure, some of Firefox’s loss of market share came from the popularity of Chrome. However, I don’t think it’s much of a coincidence that Mozilla’s loss of favor isn’t connected to their position against H.264 video. They’ve handicapped themselves. Google is now heading down the same path. I expect Microsoft’s launch of IE 9 to put an end to Chrome’s growth, especially as more sites shift away from Flash in favor of standards such as HTML 5′s tag using H.264 video content.
@Steve
No, that is not the question at all. In fact, that question is irrelevant. What’s relevant is the actual end-result. The end-result is that the web remains open rather than patent-encumbered.
Two problems:
1. Safari is irrelevant on the desktop. It’s too small
2. IE users are extremely slow at upgrading, and 50% of all users still use Windows XP, which will not get IE9
In other words: Desktop browsers with H264 support will be a minority for a long time. On the other hand, Chrome is growing fast, and when Firefox 4 is released, most Firefox users will have upgraded within a couple of months, meaning that H264 will cover maybe 5-10% of the desktop browser market, while WebM will cover 35-50% (since Chrome will keep growing and stealing market share from IE).
Firefox stalling has got nothing to do with H264. It has everything to do with Chrome taking over as the “alternative browser” of choice. It is, in fact, rather insane to suggest that H264 is the least bit relevant here, since everyone is using Flash to serve video on the web.
Firefox stalling and the H264 situation did not even happen at the same time.
But the bottom line is: Your speculations and FUD about Google’s intentions are irrelevant. Google’s intentions are irrelevant. What matters is the actual end-result.
@JohnTheBastard
This is FUD. Show everyone the infringements in a court of law, or quit spreading FUD.
Apple can be sued over H264 too. There’s no guarantee.
In fact, they are more likely to be sued over H264, since while On2′s business relied specifically on avoiding patents, the MPEG-LA did no such thing. They just threw together a bunch of patents in a pool and hoped for the best.
@Dave Haynie
This is the strangest statement yet! Are you also suggesting, then, that image decoders don’t belong built into browsers either? What about JavaScript engines?
Where have you been? Were you on Mars when browsers started adding hardware acceleration?
Hardware support for WebM/VP8 is already coming. All future Android devices will have it, for example. And Android will be the biggest non-PC platform.
In the second table, it’s inaccurate to show (in yellow) WebM as available in QuickTime & Windows media. Those Google plugins are vapor at this point. Things will be different on desktops/laptops when they are publicly available, but not iOS/Windows Phone 7, where MobileSafari & Pocket IE (or whatever MS calls it) don’t run plugins. This second bit applies to Theora and your “Anything Imaginable” columns as well.
In the abstract, I think Google pushing for a more-open format is laudable, but WebM isn’t unambiguously “more open” than h.264. More importantly, at the level concrete it looks like an attempt to use YouTube/Chrome to force everybody else to change how the Internet & web work.
From Firefox/Mozilla this is pro-WebM/anti-h.264 attitude credible and laudable. But so long as Google is shipping the Flash plugin and sees Apple as their real competition, this all looks like pure corporate maneuvering, using ‘open’ as ammunition, rather than an honest idealogical decision.
This isn’t the case for IE9. IE9 will only play back H.264 or VP8 video in the video tag. No other video codec is supported even if it is installed. See their blog posts for details:
http://blogs.msdn.com/b/ie/archive/2010/05/19/another-follow-up-on-html5-video-in-ie9.aspx
http://blogs.msdn.com/b/ie/archive/2010/05/03/follow-up-on-html5-video-in-ie9.aspx
makvie, a lot of your comments are of the passionate but completely unhelpful sort that very much hurt the open argument:
1. “Safari is irrelevant on the desktop. It’s too small” No, it’s not irrelevant: even if it represents a small share of the total browser market, it is the majority browser on the Mac OS, a significant and important segment of the total market that cannot be ignored. But most importantly, your statement goes against the entire “open” philosophy. If your statement held any weight, it could be said that Opera is irrelevant, or more importantly desktop OS systems are irrelevant.
2. “It is, in fact, rather insane to suggest that H264 is the least bit relevant here, since everyone is using Flash to serve video on the web.” Simply wrong. Mobile only represents 5% of total web browsing, but by users it already represents 60% of all desktop users and is likely to surpass it this year. As someone who uses Android, the iPhone, and the iPad daily, about 50% of my browsing delivers video via the video tag using h.264, and about 70% of all desirable video is available by this method.
3. “Your speculations and FUD about Google’s intentions are irrelevant. Google’s intentions are irrelevant. What matters is the actual end-result.” If all you care about is end results, that’s all that matters. Far many, both large and small, Google’s intentions are very much important. Many are not believing Google’s stated intentions, and many are very much worried about their true intent. You can dismiss this personally, but that doesn’t make it irrelevant for all decision makers.
4. “Show everyone the infringements in a court of law, or quit spreading FUD.” Everyone knows that someone stating the patent risk cannot do this because they do not have cause so it is pointless to state.
5. “In fact, they are more likely to be sued over H264, since while On2′s business relied specifically on avoiding patents, the MPEG-LA did no such thing. They just threw together a bunch of patents in a pool and hoped for the best.” This argument is heard often from those opposed to patents, and it even sounds logical upon hearing, but it simply does not hold. Large businesses have a far greater and genuine fear of being sued for using open codecs with no patents and no patent protection versus using those which are backed by patents administered by a licensing body.
Your final statement that they threw some patents into a pile and hoped for the best is simply ludicrous. It is the result of RFPs from international standards bodies, developed by leading developers in the field over 25 year in the open with strong incentive for patent holders to participate, continually working to include the best technology and uses the group to cull the least desirable features, with tons of legal assistance, input, and review throughout. Moreover, an organization with the financial resources and simply usage backing of the bulk of the industry is very reassuring. Additional, patent litigation between established companies, each with patents and financial resources, generally results in compromise or even a financial settlement but the ability to use the patents in question. A patent holder knows a successful battle against the MPEGLA is much less likely and undesirable than it is to simply cooperate and try to be a part of the organization. However, major industries are fearful of being sued by someone with little to lose and everything to gain. Whichever way the argument is stated, it is very much true that large businesses are more confident in the patent situation with MPEGLA than they are with OSS projects that a handful of devs claim they created together outside of patents they are likely not even aware of.
Here’s a Reddit discussion from about a year ago of the technical merits of Theora. It features several people who actually implement codecs for a living, so it’s pretty in depth, as opposed to “Theora is inferior for technical reasons” and “probably isn’t of sufficient quality”. The Executive Summary: Theora *was* designed with hardware in mind (albeit not “modern” hardware) and is roughly equivalent to the way H.264 is normally used. http://www.reddit.com/comments/ayb23/theora_is_not_advanced_enough_to_be_the_next/
@Chris Pepper
“In the abstract, I think Google pushing for a more-open format is laudable, but WebM isn’t unambiguously “more open” than h.264.”
Actually, it is. Open web advocates have all applauded this move by Google. WebM is more open than h264 because it it’s royalty-free.
“More importantly, at the level concrete it looks like an attempt to use YouTube/Chrome to force everybody else to change how the Internet & web work.”
Speculation about Google’s reasons doesn’t determine whether it’s a move that makes the web more open or not.
“From Firefox/Mozilla this is pro-WebM/anti-h.264 attitude credible and laudable. But so long as Google is shipping the Flash plugin and sees Apple as their real competition, this all looks like pure corporate maneuvering, using ‘open’ as ammunition, rather than an honest idealogical decision.”
This is irrelevant FUD. Google is obviously shipping Flash to maintain market share (it would be suicide to no longer be able to play video) while pushing for an open codec for native html5 video.
But Google’s reasons for doing this are irrelevant. What’s relevant is that the likes of Mozilla are applauding it, and that means it’s moving everyone towards a more open web.
@Tim F.
FUD, FUD, FUD, and denial. And red herrings. Your comments in a nutshell. Keep changing the subject!
” No, it’s not irrelevant: even if it represents a small share of the total browser market, it is the majority browser on the Mac OS, a significant and important segment of the total market that cannot be ignored.”
Safari is not relevant. Mac OS is not relevant. It is an insignificant share of the market compared to Firefox+Chrome. That’s the bottom line here.
“Simply wrong. Mobile only represents 5% of total web browsing, but by users it already represents 60% of all desktop users and is likely to surpass it this year.”
Huh? The market share of mobile browsers is tiny compared to desktop browsers. It will remain this way for quite some time. And in that time, WebM will have plenty of time to rid the web of the scourge that is h264.
“If all you care about is end results, that’s all that matters. Far many, both large and small, Google’s intentions are very much important. Many are not believing Google’s stated intentions, and many are very much worried about their true intent. You can dismiss this personally, but that doesn’t make it irrelevant for all decision makers.”
You have failed to come up with an argument that shows why Google’s intentions are relevant. Google’s intentions are only known to Google, and the only reason to speculate about them is to spread FUD about WebM, and throw around red herrings to take away focus from the real issue: The open web just got a huge boost.
“This argument is heard often from those opposed to patents, and it even sounds logical upon hearing, but it simply does not hold. Large businesses have a far greater and genuine fear of being sued for using open codecs with no patents and no patent protection versus using those which are backed by patents administered by a licensing body.”
Once again you have failed to address my actual point. I pointed out that On2′s business relied on avoiding patents. Your response? “Herp, derp, open is bad m’kay!”
“Your final statement that they threw some patents into a pile and hoped for the best is simply ludicrous.”
No it isn’t. They have pooled their patents, and are relying on the strength of their patent pool to fend of lawsuits.
Simple rule: the browser don’t rule the OS. That’s my opinion as a professional. Adobe was like that once and Apple did cut his wings.
If Chrome don’t open my videos next time, surely I’ll delete it you know… That’s my opinion as a user.
Great article !
@Leonardo Kasperavicius
The simple rule is that the browser is becoming the OS.
Another simple rule is that the web needs to be open, and h264 threatens to close the web.
Go on msn I need you
Can anyone tell me how 34 intra prediction modes in h.265 help in achieving coding gain?
Hey! Do you know if they make any plugins to help with SEO?
I’m trying to get my blog to rank for some targeted keywords but I’m not seeing very good gains.
If you know of any please share. Thank you!
Having read this I believed it was rather informative. I appreciate you taking the time and energy to put this content together. I once again find myself personally spending a lot of time both reading and posting comments. But so what, it was still worthwhile!
これらは本当に印象的なアイデアについてのブログです。ここにいくつかの快適なポイントに触れています。どのような方法は、ライティングを維持します。
hey there and thank you for your info – I’ve definitely picked up something new from right here. I did however expertise several technical points using this site, since I experienced to reload the website a lot of times previous to I could get it to load correctly. I had been wondering if your web host is OK? Not that I am complaining, but slow loading instances times will very frequently affect your placement in google and could damage your high quality score if advertising and marketing with Adwords. Well I’m adding this RSS to my e-mail and can look out for a lot more of your respective fascinating content. Make sure you update this again soon..
I’ve read a few excellent stuff here. Certainly worth bookmarking for revisiting.
I surprise how much effort you put to create this
sort of wonderful informative website.
Howdy! I could have sworn I’ve been to this site before but after browsing through some of the post I realized it’s new to me. Nonetheless, I’m definitely glad I found it and I’ll be bookmarking and checking back frequently!
My Partner And I really wish to show you the fact that I’m just raw to blogging and incredibly admired your post. Probably I am likely to store your webpage post . You really have fantastic article posts. Appreciate it for sharing with us your website report.
I simply desire to share it with you the fact that I am new to putting up a blog and totally valued your write-up. Likely I am going to store your site post . You definitely have outstanding article reports. Admire it for swapping with us your internet report.