Tuesday, 26 January 2010

ActiveX All Over Again

Many responses to my post about video codec issues have expressed the opinion that "Mozilla should do whatever is most useful for users today, regardless of 'politics' or 'ideology'" and also that "if a user already has codecs on their system, it's wrong for Mozilla to refuse to use them". These opinions fail due to various legal and technical issues that I've already discussed, but I also want to point out that taking such positions is nothing new for Mozilla and history has proved us right for doing so, in particular regarding ActiveX and Web standards in general.

Perhaps it's not widely known, but Gecko has had code to support hosting ActiveX controls, dating back as far as 1999. ActiveX controls are very much like system video codecs. ActiveX support would have been very useful to users ever since 1999, and still would be now --- certainly in corporate intranets, and everywhere in China and South Korea. Enabling ActiveX support would probably boost our market share significantly. Most users have useful ActiveX controls on their machines. But for the last ten years, even during Mozilla's most desperate days, we have consistently refused to turn this feature on, because we believe that ActiveX is not good for the Web.

I think history has proved that this decision was completely right. Our market share rose anyway. The ActiveX ecosystem was a big vector for security attacks. Most importantly, if we'd caved, the Web everywhere would look like it does in China and South Korea, but more so --- dependent on ActiveX, and tied to Windows. No resurgent Apple, no Linux netbooks, precious few Linux users, no ChromeOS, no iPhone, no usable browsers on phones at all, and Microsoft's grip on the industry stronger than one dare imagine. We would have sacrificed huge long-term wins for users --- ALL users, not just Firefox users --- for the sake of a temporary filip.

During the ascendancy of IE, we had similar pressures over the issue of whether to follow Web standards or focus on IE compatibility. The "pragmatists" wanted us to focus on IE compatibility for the very sensible reason that authors had to develop for IE anyway, so it would be easier and cheaper for them if we just fell in line. Obviously pursuing IE compatibility would also have been good for our market share --- and our users --- at the time. We chose standards. Again, I think history shows we made the right decision.

I'm not suggesting that the consequences of exposing system codecs to the Web would be identical to exposing ActiveX. That's unlikely, and unknowable. But favouring our principles over short-term gains for users is nothing new for Mozilla, and when we've done it in the past, history shows it was the right thing to do.


  1. Thank you for staying hard on this.

  2. Two questions:
    1) Would you use/implement H.264, if MPEG-LA wouldn't require licensing for decoding (no fees, free implementation)?
    2) What will you (Mozilla) do to make Theora/Ogg better. How do you want to improve quality/bitrate ratio and the issues with hardware en-/decoders?
    "Free" won't impress much people, but "better" will! And Firefox was and is better than Internet Explorer. But will Theora/Ogg be better?

  3. I think the main reason for video codec issues is Youtube's decision to use 264 video for their HTML5 video implementation. As soon as I found out I exchanged back to flash video...I use firefox as my main browser and I am not switching to Chrome.
    If I have to stick to the demonized proprietary formats I'd better do it in the browser I choose at least.
    Eventually Youtube will give up on it (since most requesters for HTML5 video were linux/OSS advocates/hippies/flash haters, it can happen, but not warranted. The Youtube support page for the topic is full of "WTF" comments) or a way to add the codecs to firefox will arise, even as a hack or fork.
    Unlike DirectX, however, this has a solution (sticking to flash) so I don't think this case can become such a part of history, personally...

  4. Robert, I agree with the decision Mozilla has made in regards to codecs. Keep up the good fight!
    This is totally off topic, but it has bugged me for a long time. The 3px wide black column dividers are very heavy and distract from your content.
    Please consider something like
    .colums {
    -moz-column-rule: 1px solid #ccc;

  5. Robert O'Callahan26 January 2010 at 02:54

    steve: 1) if both H.264 encoding and decoding were unencumbered by patents, I'm sure we'd add support right away. If decoding was unencumbered but encoding still required licensing, I think we'd still try to resist it, because mandatory licensing would still be very bad for Web authors. 2) We are working to make Ogg Theora better, by improving seeking using an index, by funding improvements to the encoder, by funding implementation of the decoder on a popular mobile DSP, and by improving Firefox's own implementation.
    Jim: done!

  6. Why not just support the built in encoder on the respective OS.
    DirectShow on Windows
    Quicktime on Mac
    and GStreamer on Linux
    Wouldn't that allow you guys to get around having to pay license fees?
    and thanks to Firefox 3.6 you guys could push this without a large update, plus this would make a nice temporary fix until a more HTML5 worthy codec springs up.

  7. Thinking some more, I suspect a common response to this will be "that might have worked for ActiveX, but h.264/YouTube/Google has too much of a head start, Theora isn't good enough, so this time you should compromise".
    To which I guess the counter-argument is "there's no point having principles if you abandon them when the going gets tough".

  8. Robert O'Callahan26 January 2010 at 03:47

    In 2001-2004 the going WAS tough, incredibly tough. For a while IE had >90% share and Mozilla was some volunteers plus a handful of people in a small office in Mountain View with a small and dwindling pile of cash. Personally, I had no hope. But we got some breaks that I did not anticipate. We survived by putting ourselves in a position to take unexpected opportunities, and that means not giving up early.
    This is especially relevant for Christians like me, because God is a potential source of unexpected blessings.
    However, as I said in comments in my other post, there is point short of total annihilation where it would make sense for us to compromise our principles. We're nowhere near that yet.

  9. 1. Based on your arguments, how do you justify Flash?
    2. H.264 _is_ a standard. (unlike Flash)
    3. You already use any number of parts of the operating system you run upon, why not treat video as just another part (you'll use the OS's own text rendering or TCP stack, but not its licensed support for a standard?)
    I'm still having a hard time understanding the problem.

  10. Robert O'Callahan26 January 2010 at 04:28

    1. I answered this in the other post.
    2. Yes, H.264 is a standard, but being a standard isn't enough to make it good.
    3. Free, unencumbered implementations of text rendering and TCP exist, so our principles are not at risk.

  11. While I agree refusing activex was totally a good thing i cant see how this analogy is valid.
    don't get me wrong. i agree that h264 is a bad idea. but not for exactly the same reason.
    the only reason h264 is bad, is that it enforce a vendor lock-in due to licensing. it's simple, if you want to make a HTML5 youtube copy, to stream in the USA (and you want that), you'd have to pay billions of dollars JUST for the "fucking video codec".
    likewise, if you want to make your own player, except it will be thousand of dollars instead of billions (which you won't be able to pay in both cases = lock in, everyones forced to use an iphone and youtube for HTML5 stuff yay.)

  12. I don't understand why everyone is so surprised that YouTube is only offering h.264 right now. Only their ENTIRE LIBRARY was already encoded that way.
    It would be nice if they'd make clear whether they have some Theora-related plans for the future, but whether they intend to go that direction or not, offering HTML5 for the millions of video files they already have is hardly unexpected.

  13. With regards to the DirectShow/Gstreamer/etc. backends, I think that in at least the case of Gstreamer, they should support it, but as a secondary option. Mozilla should always have the option of shipping without any of these backends, for the very reasons mentioned here, but by supporting the backends as well, it can help distributors, like the various Linux distributions, with not shipping multiple copies of the same library.

  14. Not to nitpick, but compat with an IE feature was part of what drove the resurgence of webapps in general: XmlHttpRequest.

  15. Robert O'Callahan26 January 2010 at 05:28

    True. In this context, by "IE compatibility" I meant where IE diverged from an existing standard, not areas where IE innovated in useful ways.

  16. At the moment, one of the bigger things that bothers me with Theora is the lack of discussion on the format's improvement. Don't get me wrong, Thusnelda is quite the improvement... but I would like to hear about what is happening to make Theora better (particularly in the rendering and frame-rate department), and Xiph has been poor at communicating that kind of discussion. Another review (and/or declaration of improvements to be implemented) about the format currently like what happened with to form the Thusnelda branch would be nice to see among the good work.
    As it stands, I have heard and seen the discussion on the indexing issue and others areas of the format. Many have not, and assume that the Theroa project is dead due to a perceived lack of communication (As much as you and I read the mailing lists, many others don't) and site activity.
    I would really like to see Xiph start talking more... if possible.

  17. Being idealist is why I love you so much :)
    It is also my enthusiasm for the free web why I contribute, which I should do more often again.

  18. If you really want to make Ogg the web's default video/audio codec, convince Adobe to support it in Flash

  19. I support Mozilla's decision not to support H.264. Patent-encumbered video would be disastrous for the web.

  20. Robert, first, thanks for this blog it is very useful.
    I saw you answered "If decoding was unencumbered but encoding still required licensing, I think we'd still try to resist it,".
    I have a similar, but slightly different question: what if encoding and decoding in software didn't require licensing, but encoding (ev. decoding) in hardware did. In that case, would it be ok for Mozilla to implement it?
    I'm thinking about it because that way the MPEG-LA could still earn money with it (via hardware manufacturer like now) but all software would be free of licensing.

  21. Robert O'Callahan26 January 2010 at 10:37

    Teoli: That's a hard question and I don't know the answer. It still sounds bad, but less bad.

  22. How is this situation different from Mozilla's support of the then-patented GIF file format?
    Why didn't Mozilla turn off GIF display?

  23. Robert O'Callahan26 January 2010 at 12:43

    Probably because we thought we could leave GIFs enabled and not be sued. We were right. The situation with H.264 doesn't look so good.
    Plus, it doesn't make sense to take a stand that completely destroys the usefulness of the browser.

  24. I'm sorry for such a naive dumb question, but those h264 patents can't last forever? Am I completely wrong in thinking there's no need to pay any licensing fees after the patents expire provided there will be alternate free implementations which if they quick become defacto (encoder) due to being free will remove the risk of "embrace, enhance".
    If this is the case and this will be a non-issue in few years, then IMO it seems this is being blown out of proportion and after all it's only fair for the Moving Picture Experts to get paid for their expert work. This slow development of everything surrounding ogg/theora just seems to say "free doesn't pay for the plumber/food". I bet the development would accelerate if you could buy more experts to develop the free alternative. I suggest starting with getting ogg/theora radio/video streaming work from WMP if they're not already (they're not last I checked) because as a user I want the freedom to use my preferred player that comes with the OS and not install some one more 3rd party player for each codec to work around some specific bugs (eg. mkv vfr bugs in VLC).

  25. Robert O'Callahan26 January 2010 at 13:34

    I addressed the "wait for H.264 to expire" strategy in my previous post.

  26. I couldn't find anything in the other post that explained why Flash still is ok while ActiveX and native video codecs aren't. As far as I can tell, it's pretty much the same thing in principle.

  27. So, a downloadable codec (in Flash on Youtube) for H.264 is good in Firefox, a codec from disk is bad?
    Or what do I get wrong?

  28. Is there anything stopping someone (other than time and talent) from writing an add-on that uses QuickTime/WMP to play h264 video?

  29. The browser is not the only place to play videos, these videos can also be downloaded and played with the standalone player. Why is there one video decoder in my standalone player and one in firefox ? There should be only one.
    In one of your previous posts, you mention that this approach is a problem in the case of windows : malware can be embedded in video codecs. Fair enogh.
    But i.e. on Ubuntu, at first attempt of playing a patent encumbered video, the system will download "ugly" packages automatically after a proper warning to the user.
    If the user said OK to that warning, he should be able to play those videos in Firefox as well. Mozilla would be protected against legal issues because the download is performed by the OS, not the browser.
    You mention "idealism" as a reason for the current status. But in my opinion, idealism is more about the client design : video decoding is an OS thing, it should be treated at OS level.

  30. 我是中文用户，我痛恨ActiveX，我才使用Firefox的，由于某些原因，比如电子银行业务，我可能会换用IE，但是那只是极少的时间。

  31. I'm tired of waiting to publish video with a standard format. And if necessary, I'll wait until I can.
    Our organization is about to put a lot of content on the Web. I'm going to argue that it MUST be in a standard format. If anyone wants to view it, they can view it with Firefox. And if they don't want to view it, that's their problem.

  32. On the topic of patents and your last post, the layer system described at:
    Directly references Core Animation, including it's organization of layer trees and transactions, you should be aware of Apple's patent application in this area:
    Which is basically a patent on Core Animation and a system organized like this.
    have fun.

  33. The comparision is absurd! The analogy is very poor, like using bananas to explain oranges! It's easy to argument using false analogies.

  34. The analogy is not good at all. You sure can compare support to display GIF files and support to play H264 files. Both are media formats, both aren't free.
    But, definitely doesn't make any sense comparing support to ActiveX with support to play H264. This doesn't make any sense to me.

  35. Robert O'Callahan26 January 2010 at 21:57

    Håvard, Stephan: See my comment at January 24, 2010 9:09 AM. But I can enlarge on that for you here:
    We've never been in a situation where we could drop support for Flash without driving away most of our users and becoming irrelevant. Doing that would not help our mission. That's probably the primary reason why we've never even considered it AFAIK.
    Reducing use of Flash on the Web is a much, much larger project than dealing with the problem of encumbered codecs accessed via the tag. The first step is to ensure we can offer Web authors standards-based functionality as an alternative to Flash. We are working very hard on that!
    webuser: that would be very difficult. However, producing your own Firefox derivative browser which supported those codecs would be doable. That's what free software is all about.
    nicoO: I've talked in several places about how moving the problem down to the OS is not a good idea. I do agree that if we were to support system codecs, GStreamer on Linux would be the least problematic platform.
    cjwl: just the fact that a patent exists isn't necessarily a problem. A patent is a problem when someone is going to threaten us with a lawsuit and demand licensing. If they aren't, we can ignore it. The problem with H.264 is that everyone is expected to pay license fees to the MPEG-LA or get sued (unless they're too small to bother with; we aren't). Patents aren't a problem if we don't need to license them, which is why the GIF patents weren't a problem for us.
    In general I'm not particularly concerned that Apple would demand licensing from free software projects. That would trigger a firestorm of negative PR, freak out a lot of their own employees, and cripple their relationship with the free software community, which they depend on in many ways. As far as I know, they haven't done it.
    I didn't look at the Apple patent, but in general it requires detailed comparison of the claims with our code to determine if our code would actually infringe, so it would be unwise to jump to a conclusion that that patent bears on what we're doing. For what it's worth, there are other frameworks that work in a similar way, for example Clutter, and Microsoft's WPF (which probably predates Core Animation).

  36. Uhh... What?
    I agree with what you say, but how exactly is NPAPI different from ActiveX? It's the same thing. You visit a page that needs a proprietary, closed-source plugin. You download and install it. What makes these plugins better than ActiveX ones? The fact that they're available for a different plugin architecture?
    Let's not pretend the web today is plugin-free. Unfortunately you are in for a world of hurt without Flash, and chances are you'll need to use Java some day. The other ones are easier to replace as they're just a way to open documents in the browser (what a stupid idea) and compromise security (if I had a dime every time a shady ad network triggered a PDF download... I don't want to think what would happen if I had that plugin enabled).
    Same for the video thing: instead of installing a codec, they'll just install a plugin (which is worse, as it implicitly has more code, which means more attack surface, worse integration and worse stability).
    To reiterate I agree with the message but I find a bit hypocritical calling on ActiveX or other such mechanism to execute opaque binary closed-source stuff inside the browser, when every browser starting with Firefox happily supports it.

  37. Robert, don't get too emotional about those users saying "Mozilla should do whatever is most useful for users today, regardless of 'politics' or 'ideology'". I agree that Mozilla's very reason to exist is to be different, and a big part of that difference is the different ideology.
    I first want to correct that. I don't think it's true. IIRC, there was a time (several years) where we shipped with activex enabled and maybe even whitelisted a few ActiveX components, but never allowed all by default. I might be mis-collecting this, though, or confusing it with the Netscape browser.
    More to the point, I don't think that the situation between enabling ActiveX and allowing MPEG 2/4 specifically (which I asked for) is anyhow comparable.
    1) MPEG is an ISO standard, not a vendor API.
    2) MPEG is not limited to a certain platform nor implementation.
    3) Enabling MPEG codecs would be selective. It's not the same as enabling any and all system codecs or system APIs.
    4) The security risk is opening a file (MPEG) vs. allowing hostile code to call your API (ActiveX).

  38. Robert O'Callahan26 January 2010 at 23:25

    One important difference between ActiveX and NPAPI is that ActiveX was solely controlled by Microsoft but the NPAPI interfaces are managed by browser vendors working together.
    Another important difference is NPAPI is platform-independent. You can build NPAPI plugins and browsers for any platform with no dependencies on any particular platform functionality or framework. ActiveX, however, depends very heavily on COM, which is a huge Windows-only framework.
    Another important difference is that ActiveX was designed based on VBX Visual Basic controls, so from the beginning there was a large ecosystem of Windows-only ActiveX controls, and of course since Microsoft was the driving force behind ActiveX, they kept building more Windows-only ActiveX controls. ActiveX was never pitched as anything but a Windows technology. On the other hand, in the NPAPI ecosystem people were building plugins for Mac and X from the beginning.
    Having said all that, dependence on closed-source NPAPI plugins isn't a good situation either. Just a lot better than ActiveX.

  39. Robert O'Callahan26 January 2010 at 23:30

    Ben, I think Netscape shipped with Windows Media Player whitelisted for a while. That's not Mozilla though --- I don't Mozilla builds did. Also, the whitelist minimized any damage to the Web. (The fact that hardly anyone used Netscape 6 and 7 also limited the damage to the Web...)
    As I said in the final paragraph of my post, I am not trying to draw a direct analogy between ActiveX and supporting system codecs. I am merely pointing out that denying Web access to code on your system is not unprecedented for us, and history shows that at least in some cases it's been good for the Web.

  40. > Unfortunately you are in for a world of hurt without Flash
    Actually, I think you're in for a world of hurt, if you do install Flash. I've tried it for a moment, and was disgusted by the Flash ads, I couldn't stand it anymore.
    I am doing just fine without Flash. Apart from videos: YouTube and others on news sites and similar. Thus, I'd be happy if I could play videos right in my browser, on the 2-3 websites where I run into videos.
    (And to avoid misunderstanding: I was also always strongly against ActiveX. I am also very much against closed formats.
    The "only" problem with MPEG are patents - and whether software patents are legal at all is still an open debate.)

  41. > the whitelist minimized any damage to the Web
    That's why I filed bug 541286 ;-).
    > history shows that at least in some cases it's been good for the Web.
    Unquestionably. I agree with the general stance, too.

  42. I'm glad my Android phone (Moto Droid) can play H.264, but I wish it supported FLAC and OGG.
    Too bad the W3C pulled back from their Theora recommendations at the pressure of big companies.
    Why does Google use H.264 (Youtube, Android)? You'd think they'd want to save money, at least...

  43. I mostly agree, I think, that doing h.264 specifically would be bad from a strategy point of view, and bad for distributors (including Mozilla), among other things. Besides, my laptop won't be able to do anything with it anyway - it already struggles with h.264 content in Flash and standalone players, so personally it is irrelevant.
    What I do not like, and believe to be contrary to Mozilla's goal of being open and making the Internet a better place, however, is the fact that there's no way to extend the decoders available. With image decoding, this is theoretically possible - I had a JavaScript based imgIDecoder a few years back (too slow to be practical, but it was possible), as well as a C++ based one that actually functioned for an obscure image format. There's no possibility of that here; nsMediaDecoder is completely internal, and the list of possible decoders is statically coded in nsHTMLMediaElement::CreateDecoder.
    Extending Mozilla in ways that Mozilla doesn't have energy for is what makes the platform - both Gecko and Firefox - awesome. Please make <video> awesome by opening the gates to external, non-libxul, programmers?
    P.S. Somehow your comment preview manages to clear the text entry box, so once you preview you lose the post!

  44. @Ben:
    As noted before, h.264 is what youtube was already encoded in before Google bought Youtube. Takes time/money/space to convert, so it's something that they'll have to work on over time.

  45. I think comparing ActiveX to video codec's is a very bad analogy, Ben Bucksch covered the reasons why quite nicely around 8 posts up...
    And the biggest issue right now is that the big content sites (aka youtube) have gone with the "commercial" codec. And as long as they're targeting devices that require hardware offload chips to play the content, they're going to stay with h.264. No-one is making a Theora decode chip to put in cell phones, netbooks, tablets, set-top streaming media players, etc. and they aren't putting in enough CPU power to handle HD content either. The sheer amount of h.264 stuff out there already means you are going to have to co-exist with it (not just YouTube/Vimeo, but most BluRay disks, iTunes video stuff, AVCHD camcorders, ...)

  46. My view:
    I don't care.
    Sure, I WANT Web developers to use open standards. However, they don't.
    If I see a site that I can't view in Firefox, I'll open up Safari or Chrome, not write a 300-page manifesto about Theora.
    Denying users access to their own codecs is biting off the nose to spite the face.
    Plus, if you're really that into not supporting proprietary standards by any means, disable Flash and THEN we'll talk.
    Couldn't you just turn it on, but bury it deep in about:config so only people who really don't care about Theora turn it on?

  47. I admire your courage. Thank you for having what it takes.

  48. @As noted before, h.264 is what youtube was already encoded in before Google bought Youtube
    Are you sure about that? AFAIK Youtube content was VP6. They converted to h.264 sometime later either when it was first supported by Flash (perhaps when they started offering higher 480?) or perhaps to offer it to mobile device like the iPhone. It's possible Youtube always had h.264 in the background but I'm not convinced. I do believe they had to reencode most of their videos when they switched to h.264. (I presume they stored the original videos and used these as sources hence the reason they were able to add higher res to long existing videos, makes the most sense to me.)
    Incidentally I wasn't aware until researching stuff for this post but Google now owns On2. If they wanted to, they could contribute whatever patents On2 has to the open source community. Since Theora is based on VP3, even if On2 granted related patents I wonder if there's some new stuff from their later developments that may be useful. Of course they could also open source all the other codecs (along with the patents as I already mentioned) and perhapds something there could be a new Xiph codec. I'm personally still unconvinced Theora is ever going to come close to VC-1 let alone h.264 which makes problems for it trying to gain a foothold given the widespread usage of these codecs including with hardware support. If they had something closer, they may fare better. I've had some hope for Dirac, but perhaps VP8 would be something to look at too.
    Of course I'm not convinced Google is actually going to do something like that. And even then, lacking the vast patent portfolio that's behind h.264 I wonder whether they'll really get close.
    P.S. It seems I'm not the only one to have this idea, I came across a FSF campaign.