AVPortraitEffectsMatte

Capture

AVCapturePhoto

Loading and Saving

Sample

• Capturing Photos with Depth
• Enhancing Live Video by Leveraging TrueDepth Camera Data
• Streaming Depth Data from the TrueDepth Camera

Demo 1

Working with Depth

Demo 2

Working with the Portrait Effects Matte

Real-Time Video Effects with TrueDepth

• TrueDepth stream
• Point clouds
• Backdrop

Depth Map

How do you add a TrueDepth stream?

Disparity or Depth?

• Disparity usually yields better results
• Depth values are intuitive, but the error goes with Z²

Resolution

Test System Pressure

High resolution + fast frame rate + processing + duration

• AVCaptureDevice.SystemPressureState.Level
• Nominal
• Fair
• Serious
• Critical
• Shutdown

Monitor System Pressure and Adapt

Prevent AVCaptureDevice shutdown

• Reduce frame rate when level is increasing:
  - Nominal (30fps) → Serious (15 fps)
  - Nominal (30fps) → Fair (24 fps) → Serious (20 fps) → Critical (15 fps)

Filtered Depth Map

• Smoothed, spatially and temporally
• Holes filled, using RGB
• For photography/ segmentation applications
• iOS 12: spatial resolution of filtering improved from iOS 11

Using Raw Depth Data

• Use for point clouds, or real world measurements
• No depth value == 0 ⚠️
• Watch out for averaging / downsampling

Depth Map Holes ⚠️

• The TrueDepth camera detects most objects up to about 5 meters
• Low reflectivity materials absorb light
• Reflectivity
• Specular materials reflect light only from certain directions

• Outdoor conditions are more challenging

• Shadow from parallax between projector and camera

Point clouds

RGB-D

Vertex Shader

Point location

• Z = func( U, V )
• Tranform to ( X, Y, Z )
• Reproject with view matrix

Fragment Shader

Point color

• Get vertex info
• Discard holes ( Z == 0 )
• Apply color from RGB frame at vertex coordinates

Backdrop

Per-Frame, Real-Time Processing

• Detect a face
• Create binary foreground mask, smooth, and upscale
• Upscale foreground to background and blend
• Resize background to video resolution once:
  - Not upscaling foreground
  - Blending at lower resolution

Getting Face Metadata

• Until a face is detected, use default face depth of 0.5 meters
• Transform center of face to depth map coordinates, and get its depth

Binary Foreground Mask

• Threshold = face depth + 0.25 meters
• Binary mask:
  - Foreground = 1
  - Background = 0

Gaussian Blur

Gamma Adjust

Binary Mask

Alpha matte

Clamp Before Filtering

Avoiding edge softening

Blending Foreground with Background

TrueDepth Stream

640 x 480 depth map registered to video at 30 fps