

Re: [Public WebGL] double-buffering and back buffer preservation

On Mon, Nov 15, 2010 at 4:53 PM, Chris Marrin <cmarrin@apple.com> wrote:

On Nov 15, 2010, at 2:34 PM, Gregg Tavares (wrk) wrote:

> On Mon, Nov 15, 2010 at 1:56 PM, Vladimir Vukicevic <vladimir@mozilla.com> wrote:
> One of the major issues that came up during the last f2f was that the current canvas model is not the most efficient for implementing on hardware, especially on mobile hardware.  The current model was based on how the 2D canvas works: drawing happens during content JS script, and as soon as control is returned to the browser, it's supposed to be presented for display.  Content can also read back the current contents of the displayed image (in 2D canvas, via getImageData; in WebGL, via readPixels; and in both via toDataURL).
> However, most 3D hardware doesn't really want to work like that -- they're optimized for double-buffering, where you draw a scene to the back buffer, and then swap buffers; after a swap, the new back buffer contains garbage.  Implementing the current canvas semantics is not a big issue on the desktop, because there is cpu/gpu/bandwidth to spare, but it's a pretty big deal on mobile.
> We identified a few different options:
> 1) Do nothing.  Leave things as it is.  There would be fairly significant overhead that all apps would pay, even if they never want to call readPixels or toDataURL.
> 2) Add explicit double-buffering to WebGL canvases, and add an explicit present() call.  This complicates things for developers, because they need to then actually make this present() call, and can potentially result in higher memory usage due to some implementations needing to keep around both a front and back buffer, where before they could be effectively single-buffered.
> 3) Add implicit double-buffering to canvas.  Follow the same semantics as canvas does currently -- swap happens whenever control is returned to the browser -- but always enforce an uninitialized/cleared back buffer after each swap.
> 4) Like #2 and #3, by adding a context attribute that allows the author to choose which they would want.
> 5) Like #1 (do nothing), but add a context attribute for the author to indicate whether readback from the window/canvas buffer will ever be done.  If it's set to true, then dimply disallow readPixels/toDataURL when framebuffer 0 is bound (or have them always return all black).
> We identified a few different use cases that should be considered.  Printing and screenshots were two, along with using a WebGL canvas as a texImage2D source, and using a WebGL canvas as a 2D canvas drawImage source.
> Note that even with implicit double-buffering, it's possible to use a FBO to build up a scene across many events and only draw it to the drawbuffer once (just like you can do today, with #1).  Adding context attributes complicates the implementations, potentially significantly, so I'd like to avoid 4 and 5.
> I'd lean towards #3, but what are the list's thoughts on this?
> I'm fine with 3 with the caveats that I mentioned before. Namely that toDataURL, readPixels, drawImage, texImage2D(canvas3d, ...) all work as expected. Namely that reading the canvas through any of those calls gives you the contents what you see displayed until first draw call after a swap. I mentioned 2 ways to implement that. Either don't issue the clear until the first draw call after a swap or, read from the display buffer after a swap, read from the draw buffer after a draw,

But that is exactly the case that is expensive or impossible on some embedded hardware (at least). You can never get access to the contents you see displayed. Once the drawing buffer is displayed, its contents are not longer available to be read. Essentially the problem is this:

1) All OpenGL-like implementations (including D3D) do an explicit Present() (a). WebGL hides this by saying that this Present() is implicitly done on return from any JS function in which WebGL drawing is done. But when Present() is actually done is not well defined.

2) In some implementations (mostly on the Desktop), Present() is simply a message to the compositor that the current drawing buffer should be composited with the rest of the page. The drawing buffer image doesn't change so it can be used until new WebGL calls are made which overwrite it.

3) Some hardware doesn't work that way. In the hardware used in the iPhone, for instance, Present() actually takes the drawing buffer away from the WebGL context and uses it in a compositing operation that might not even be in sync with the WebGL rendering process. So there is no access to the drawing buffer possible after Present(). If you need access to it, you'd have to make a (costly) copy before issuing the Present() call. Given that a lot of mobile hardware is similar to that in iPhone, I would assume similar issues exist.

4) It would be a shame if the default WebGL operation required that copy. It would mean that platforms that can least afford it (mobile) would be the most heavily penalized. All for the possibility that some fairly rare operations might be called.

Of those operations, I'm not sure readPixels() should be included. It seems reasonable that readPixels() would have to be called before a Present() (implicit or explicit) is performed. The real question is what are the rules for calling toDataURL(), or calling drawImage or texImage2D with a WebGL context.

An illustrative example was raised at the last F2F that involved taking a snapshot of a running WebGL game. It is interesting because you don't want to penalize the performance of your game just on the odd chance that someone might hit control-S to capture a screen snapshot. Assuming such a penalty were not the default, then what would the author have to do when control-S is pressed? The problem is that the control-S handler would come in as a browser event so you won't know if an implicit Present() had been called and whether or not the data in the current drawing buffer is valid.

I'm not sure this is a real issue though. For an app that is continuously drawn, you can just as easily set a flag when control-S comes in and then do a toDataURL after the next render if that flag is set. Calling toDataURL after rendering the current scene and before returning from that rendering operation ensures that the data in the drawing buffer is valid.

I think the real problem is in calls to drawImage() or texImage2D() with a WebGL context. Since these are rendering to different 2D or WebGL Canvas contexts, you can't be sure of the state of the WebGL context you're trying to use as the image source. How do you solve that problem? I think adding context attributes and the like are overkill. There's no need to mark a WebGL context as "preserving the drawing buffer" for it's entire life. I think it's also a mistake to have a WebGL context that requires a Present() call.

I think it's better to just acknowledge that using a WebGL context as an image source is not "free". Doing it requires cooperation on the part of the context wanting to use a WebGL image source and the WebGL context being used as the source. We can have a Present() call, but make it optional and pass a parameter to it. Passing true says that the current drawing buffer should still be available after the Present() call. Also, passing false should say that the buffer will not be available after Present(), rather than making it undefined whether or not it is available, for consistency. If Present() is not called on return from a JS call that did some rendering, then it would be implicitly called as before, with a param of false. Using a WebGL context as an image source when Present(true) had been previously called on that context causes the contents of the drawing buffer when Present(true) was called to be used as that image source. Otherwise using that context will result in an image of transparent black (0x00000000) being used.

Adding a Present function seems like it certainly has advantages. Why not require it instead of having the current behavior? Like others have mentioned, this makes it more like real OpenGL and makes it easy to spread out rendering across callbacks, possibly making it better for web worker stuff in the future as well.

I'm not sure printing is an issue. Printing is a system function and so the contents of all buffers should be available to the printing functionality. I think this is true both on desktop and mobile.


(a) It's sometimes possible to do single buffer rendering with OpenGL and D3D, where what you are rendering is actually what is being shown on the display. When doing this, the author of the app has to make sure to render in the "vertical blanking" interval and take no longer than a fraction of the display's frame-rate, or tearing of the image will occur. So this is very specialized and not applicable to WebGL. In fact, because the WebGL drawing buffer has to be "composited" with the other HTML content on the page, single buffering is never possible. So for all practical purposes, all WebGL implementation need an explicit Present().