dx11 TU_RENDERTARGET texture leak.

[creadmefford]

overview

this is a "prototype" fix for the dx11 render system and should not be checked into ogre main as it will cause leaks with non dx11 render systems. it does however fix my issues with directx 11 so it's illustrative of the issue.

problem:

there are two memory leaks with the dx11 render system that cause crashes in my tools and app on shutdown. these leaked resources appear to hold strong references to each other so basically if either of the following leaks occur some 80+ directx objects get leaked (including the top level DX11 device, etc etc). the two issues are:

1. class D3D11RenderTexture holds a reference to a class D3D11DepthBuffer which is not always freed.
2. class D3D11DepthBuffer holds a reference to ID3D11DepthStencilView which is not always freed regardless of whether or not the owning D3D11DepthBuffer is freed.

investigation:

there are (at least) 3 scenarios where render textures are created, used, and freed. the issue is each of these 3 scenarios is slightly different so the real fix needs to take this into account somehow. the 3 scenarios are: 1. the render window, 2. compositors / pooled depth buffers, 3. manual textures of type Ogre::TU_RENDERTARGET.

1. the render window creates and frees it's depth buffer and depth stencil view manually. see D3D11RenderWindowBase::_createSizeDependedD3DResources(void) which calls D3D11RenderSystem::_addManualDepthBuffer() to create and D3D11RenderSystem_removeManualDepthBuffer() and mDepthStencilView.Reset(); to cleanup.

2. next are the compositor / pooled depth buffers. they use RenderSystem::_cleanupDepthBuffers () to iterate the mDepthBufferPool and free any depth buffers here. the problem here is _cleanupDepthBuffers () expects the compositor buffers to still exist so if you free them in ~D3D11RenderTexture() you will get a double free crash when _cleanupDepthBuffers() is called. if you don't free the buffers in ~D3D11RenderTexture() then scenario 3 will occur, however. damned if you do, damned if you don't. this scenario correctly frees the ID3D11DepthStencilView by specifying manual=false

3. finally, the Ogre::TU_RENDERTARGET scenario. this was the first scenario i found because it occurs on the 3 shadow textures created by Ogre in Ogre::SceneManager::ShadowRenderer::ensureShadowTexturesCreated() via ShadowTextureManager::getSingleton().getShadowTextures(mShadowTextureConfigList, mShadowTextures);. it turns out though, this isn't shadow specific and you can repro it easily but just creating a manual Ogre::TU_RENDERTARGET like this:

// repro
auto buggyTex = Ogre::TextureManager::getSingleton ().createManual ("buggytex",
        YOUR_RESOURCE_GROUP,
        Ogre::TEX_TYPE_2D, 256, 256, 0, Ogre::PF_DEPTH16,
        Ogre::TU_RENDERTARGET, NULL, false, 0);
    // Ensure texture loaded
    buggyTex->load ();

// and then shutdown your app:
OgreBites::ApplicationContext::shutdown ();

the Ogre::TU_RENDERTARGET scenario does not get deleted by RenderSystem::_cleanupDepthBuffers () and since the D3D11DepthBuffer allocated in void D3D11RenderTexture::rebind() is marked manual=true the ID3D11DepthStencilView is also leaked.

current solution and conclusion:

a. change D3D11RenderTexture::rebind () to pass false in for "is manual". this allows the ID3D11DepthStencilView to be released properly for scenario #3 and it's what #2 does already. (see change in OgreD3D11Texture.cpp)

b. the remaining issue is how do you free the mDepthBuffer from D3D11RenderTexture() for scenario #3 without breaking scenario #2 ???

the current solution stops double deleting memory in the render system _cleanupDepthBuffers() but this may add complexity to other render systems who may not know to do this. i'm hoping the memory contract is clearer for paroj. perhaps Ogre::RenderTarget class could own and free the attached depth buffers instead of having the Ogre::RenderSystem pool manage it. how do other render systems delete Ogre::DepthBuffer instances that are not in the pool?

[paroj]

#include "Ogre.h"
#include "OgreApplicationContext.h"

int main(int argc, char *argv[])
{
    OgreBites::ApplicationContext app;
    app.initApp();
    auto buggyTex = Ogre::TextureManager::getSingleton().createManual("buggytex",
                                                                      Ogre::RGN_DEFAULT,
                                                                      Ogre::TEX_TYPE_2D, 256, 256, 0, Ogre::PF_DEPTH16,
                                                                      Ogre::TU_RENDERTARGET);
    // Ensure texture loaded
    buggyTex->load();
    app.closeApp();
    return 0;
}

shuts down cleanly without crashes - not saying there are no leaks..

[creadmefford]

[creadmefford]

fyi - i noticed today that i only see the crash on my nvidia based desktop. there is no crash (but still leaks) on my AMD based machine.

[creadmefford]

thanks so much!