Advanced NSView Setup with OpenGL and Metal on macOS

I have been tweaking the UI code for the macOS variant of Shapeflow over the last couple of days. It’s surprisingly difficult to get rid of all the kinks and quirks. Two interesting aspects were how to get on-demand rendering working (e.g. only draw a new frame if something really has changed) and drawing only the active document tab.

So the general setup is as follows: We have an NSDocument-based application that within each window has a render view (OpenGL or Metal). So if the app uses the default-tabbing behaviour, opening the second document will not create a second window but show up as a tap in the same window. The desired behaviour is to only be updating the render view of the active document.

Also we want to be synchronised with the displays VSync when it comes to updating the render view. However, we do not want to have to re-render a frame if nothing changed as this would only cost unnecessary CPU/GPU performance and power. It may not seem important if you own a big iMac that is always plugged into power, but when working on a MacBook this can save valuable battery time and improve responsiveness.

NSOpenGLView vs NSOpenGLLayer

Let’s start with OpenGL: Using NSOpenGLView worked great until I tried to use it in multiple tabs. Initial I was using a CVDisplayLink and NSOpenGLContext per view. This resulted in a freeze in [NSOpenGLContext flushBuffer] as soon as the second tab opened. Up to this point, I still haven’t found any conclusive information on why this would happen. It seems to be related to the view becoming layer-backed when the tab-bar appears and this causing problems. However, there didn’t seem to be any way to fix this issue.

The solution was to completely ditch NSOpenGLView and switch to a layer-backed NSView with an NSOpenGLLayer. The extra bonus is that NSOpenGLLayer has the method canDrawInOpenGLContext which can be used to only render a new frame when something actually changes. If not, the layer content is simply re-used. Think of it as Core Animation taking a snapshot of your render and using that image instead of rendering the full 3D scene.

RenderView.h

@interface RenderView : NSView

@end

RenderView.m

@implementation RenderView

- (id)initWithCoder:(NSCoder *)coder
{
    self = [super initWithCoder:coder];
    if ( self != nil )
    {
        // Enable retina-support
        self.wantsBestResolutionOpenGLSurface = YES;
        
        // Enable layer-backed drawing of view
        [self setWantsLayer:YES];

        ...
    }
    return self;
}

- (CALayer *)makeBackingLayer
{
    return [[OpenGLLayer alloc] init];
}

- (void)viewDidChangeBackingProperties
{
    [super viewDidChangeBackingProperties];
    
    // Need to propagate information about retina resolution
    self.layer.contentsScale = self.window.backingScaleFactor;
}

...

@end

OpenGLLayer.h

@interface OpenGLLayer : NSOpenGLLayer

@end

OpenGLLayer.m

@implementation OpenGLLayer

- (id)init
{
    self = [super init];
    if ( self != nil )
    {
        // Layer should render when size changes.
        self.needsDisplayOnBoundsChange = YES;
        
        // The layer should continuously call canDrawInOpenGLContext
        self.asynchronous = YES;
    }
    
    return self;
}

- (NSOpenGLPixelFormat *)openGLPixelFormatForDisplayMask:(uint32_t)mask
{
    NSOpenGLPixelFormatAttribute attr[] = {
        NSOpenGLPFAOpenGLProfile, NSOpenGLProfileVersion3_2Core,
        NSOpenGLPFANoRecovery,
        NSOpenGLPFAAccelerated,
        NSOpenGLPFADoubleBuffer,
        NSOpenGLPFAColorSize, 24,
        0
    };
    return [[NSOpenGLPixelFormat alloc] initWithAttributes:attr];
}

- (NSOpenGLContext*)openGLContextForPixelFormat:(NSOpenGLPixelFormat *)pixelFormat
{
    return [super openGLContextForPixelFormat:pixelFormat];
}

- (BOOL)canDrawInOpenGLContext:(NSOpenGLContext *)context pixelFormat:(NSOpenGLPixelFormat *)pixelFormat forLayerTime:(CFTimeInterval)timeInterval displayTime:(const CVTimeStamp *)timeStamp
{

    // TODO: return YES here if a new frame needs to be rendered (e.g. because of some animation
    //       or result of a user interaction). Return NO if the last frame can be re-used.

}

- (void)drawInOpenGLContext:(NSOpenGLContext *)context pixelFormat:(NSOpenGLPixelFormat *)pixelFormat forLayerTime:(CFTimeInterval)timeInterval displayTime:(const CVTimeStamp *)timeStamp
{
    [context makeCurrentContext];
    
    CGLLockContext(context.CGLContextObj);

    // TODO: Perform rendering here.

    [context flushBuffer];
    CGLUnlockContext(context.CGLContextObj);
}

@end

Metal

After having so much success with NSOpenGLLayer, I thought I use a similar approach for Metal and use NSView plus CAMetalLayer. However, this proved to be difficult for various reasons. In the end, I ended up using MTKView instead but that cause one major problem: Even if a tab was in-active, it’s MTKView was still rendering all the time!

To achieve a similar “on demand rendering”, I ended up using a CVDisplayLink (for continues triggering in alignment with the VSync) in addition to the usage of MTKView.enableSetNeedsDisplay. MTKView by default has its own loop continuously calling the delegate’s drawInMTKView method. By using enableSetNeedsDisplay (and MTKView.paused), we can disable that loop and rather have the view render only when needsDisplay is set. The trick is to use CVDisplayLink to continuously figure out if we need to set it to YES for a particular time frame.

RenderView.h

@interface RenderView : MTKView

@end

RenderView.m

@interface RenderView()
{
    CVDisplayLinkRef displayLink;
}
- (CVReturn)getFrameForTime:(const CVTimeStamp*)outputTime;
@end

static CVReturn MyDisplayLinkCallback(CVDisplayLinkRef displayLink, const CVTimeStamp* now, const CVTimeStamp* outputTime, CVOptionFlags flagsIn, CVOptionFlags* flagsOut, void* displayLinkContext)
{
    return [(__bridge RenderView*)displayLinkContext getFrameForTime:outputTime];
}

@implementation RenderView

- (id)initWithCoder:(NSCoder *)coder
{
    self = [super initWithCoder:coder];
    if ( self != nil )
    {
        self.device = MTLCreateSystemDefaultDevice();
        self.colorPixelFormat = MTLPixelFormatBGRA8Unorm;
        self.clearColor = MTLClearColorMake(0.0, 0.0, 0.0, 0.0);
        self.depthStencilPixelFormat = MTLPixelFormatDepth32Float_Stencil8;
        self.sampleCount = self.renderer.multisamples;
        self.delegate = self;
        
        // Set paused and only trigger redraw when needs display is set.
        self.paused = YES;
        self.enableSetNeedsDisplay = YES;

        // Setup display link.
        CVDisplayLinkCreateWithActiveCGDisplays(&displayLink);
        CVDisplayLinkSetOutputCallback(displayLink, &MyDisplayLinkCallback, (__bridge void*)self);
        CVDisplayLinkStart(displayLink);
    }
    return self;
}

- (void)dealloc
{
    CVDisplayLinkStop(displayLink);
}

- (CVReturn)getFrameForTime:(const CVTimeStamp*)outputTime
{
    if ( /* decide if scene has changed and needs render */ )
    {
        // Need to dispatch to main thread as CVDisplayLink uses it's own thread.
        dispatch_async(dispatch_get_main_queue(), ^{
            [self setNeedsDisplay:YES];
        });
    }
    return kCVReturnSuccess;
}

- (void)drawInMTKView:(nonnull MTKView *)view
{    
    id commandBuffer = ...

    // Render frame

    [commandBuffer presentDrawable:self.currentDrawable];
    [commandBuffer commit];
}

@end

Summary

It’s surprising how little code it takes but this has cost me a lot of nerves. None of the examples I found properly handled on-demand rendering and none addressed the multi-tab issue mentioned above. Hopefully this post will save you the hassle of going through all the trouble figuring out what the correct solution is.