Creating Photo and Video Effects Using Depth
Creating Photo and Video Effects Using Depth
WWDC 2018
![](https://cdn-images-1.medium.com/max/800/1*6ow27jyBdNVLyg-M0o5YTg.jpeg)
Portrait Matte
- iOS 12
- Front and rear
- Portrait
- People
- Linear
- No guarantee
![](https://cdn-images-1.medium.com/max/800/1*kMwBTSfDyiKdBO_rkrzpAA.png)
![](https://cdn-images-1.medium.com/max/800/1*VklDG6ntzBs8TUKOD3gsZg.png)
Loading
ImageIO
![](https://cdn-images-1.medium.com/max/800/1*RJuoirGNvfEb24PYfmSKqQ.png)
AVPortraitEffectsMatte
![](https://cdn-images-1.medium.com/max/800/1*xLtbl0nkKwWnxJayvnaT6g.png)
Capture
AVCapturePhoto
![](https://cdn-images-1.medium.com/max/800/1*ajtsLGtg6asThV1NE9xlEA.png)
Loading and Saving
![](https://cdn-images-1.medium.com/max/800/1*v3l7SynhHjMTXYxOGeoTFQ.png)
![](https://cdn-images-1.medium.com/max/800/1*SxZ_1vK6vTKaxPjcLmBHew.png)
Sample
- Capturing Photos with Depth
- Enhancing Live Video by Leveraging TrueDepth Camera Data
- Streaming Depth Data from the TrueDepth Camera
Demo 1
Working with Depth
![](https://cdn-images-1.medium.com/max/800/1*xUq-k35K2hasvE8SAbFbeg.jpeg)
Demo 2
Working with the Portrait Effects Matte
![](https://cdn-images-1.medium.com/max/800/1*CG4t06Sd7kpeuoAk62HccA.jpeg)
Real-Time Video Effects with TrueDepth
- TrueDepth stream
- Point clouds
- Backdrop
![](https://cdn-images-1.medium.com/max/800/1*86WMiy2GVcT9k3ZbfUIoEg.gif)
Depth Map
![](https://cdn-images-1.medium.com/max/800/1*Nbxcw5bLgV5VOYVZJtsGXA.png)
![](https://cdn-images-1.medium.com/max/800/1*2UTvWFyUxo4abCLsJOJZAg.gif)
![](https://cdn-images-1.medium.com/max/800/1*skkK7U8qZFhSOe3fLIhPPA.gif)
How do you add a TrueDepth stream?
![](https://cdn-images-1.medium.com/max/800/1*Fbq1rniX_jz8rw36yeSnTQ.png)
Disparity or Depth?
- Disparity usually yields better results
- Depth values are intuitive, but the error goes with Z²
![](https://cdn-images-1.medium.com/max/800/1*SH7hgk2sKVRAuUON-09WRQ.png)
Resolution
![](https://cdn-images-1.medium.com/max/800/1*ie7T4KSvD-db9fHPiY2B5g.png)
Test System Pressure
High resolution + fast frame rate + processing + duration
AVCaptureDevice.SystemPressureState.Level
- Nominal
- Fair
- Serious
- Critical
- Shutdown
Monitor System Pressure and Adapt
Prevent AVCaptureDevice shutdown
- Reduce frame rate when level is increasing:
- Nominal (30fps) → Serious (15 fps)
- Nominal (30fps) → Fair (24 fps) → Serious (20 fps) → Critical (15 fps)
![](https://cdn-images-1.medium.com/max/800/1*NpGVJTO1zflK2jCmoGO6Dg.png)
Filtered Depth Map
- Smoothed, spatially and temporally
- Holes filled, using RGB
- For photography/ segmentation applications
- iOS 12: spatial resolution of filtering improved from iOS 11
![](https://cdn-images-1.medium.com/max/800/1*HvVK-Z2z6rmkR-TYBwVe1Q.png)
![](https://cdn-images-1.medium.com/max/800/1*nGrjUp3_I3fr8ha_PoHlEA.gif)
Using Raw Depth Data
- Use for point clouds, or real world measurements
- No depth value == 0 ⚠️
- Watch out for averaging / downsampling
Depth Map Holes ⚠️
- The TrueDepth camera detects most objects up to about 5 meters
- Low reflectivity materials absorb light
- Reflectivity
- Specular materials reflect light only from certain directions
![](https://cdn-images-1.medium.com/max/800/1*L06UV7xPjKN6igaU-jjRXA.gif)
- Outdoor conditions are more challenging
![](https://cdn-images-1.medium.com/max/800/1*hxSVTznZx8i_1K_TiCUpdw.gif)
- Shadow from parallax between projector and camera
![](https://cdn-images-1.medium.com/max/800/1*e05aCCzbzwYRHowfUgQOow.gif)
Point clouds
![](https://cdn-images-1.medium.com/max/800/1*yz3Ujil6xmzYJmpe8KNItQ.png)
![](https://cdn-images-1.medium.com/max/800/1*LGkbWgF2GyT8U8hWy1OeVw.gif)
RGB-D
![](https://cdn-images-1.medium.com/max/800/1*S2pDQriAqkrocD19cWGcfQ.gif)
Vertex Shader
Point location
- Z = func( U, V )
- Tranform to ( X, Y, Z )
- Reproject with view matrix
![](https://cdn-images-1.medium.com/max/800/1*49XpXHUh-rb5qd_et_Ja0Q.gif)
Fragment Shader
Point color
- Get vertex info
- Discard holes ( Z == 0 )
- Apply color from RGB frame at vertex coordinates
![](https://cdn-images-1.medium.com/max/800/1*9x_m6rvys97dTT2vwgZUGw.gif)
Backdrop
![](https://cdn-images-1.medium.com/max/800/1*lcBMlh2K37JGsWx0tT-HMA.gif)
Per-Frame, Real-Time Processing
- Detect a face
- Create binary foreground mask, smooth, and upscale
- Upscale foreground to background and blend
- Resize background to video resolution once:
- Not upscaling foreground
- Blending at lower resolution
Getting Face Metadata
- Until a face is detected, use default face depth of 0.5 meters
- Transform center of face to depth map coordinates, and get its depth
![](https://cdn-images-1.medium.com/max/800/1*dkc3mczabUc8lVz97BB-0w.png)
Binary Foreground Mask
- Threshold = face depth + 0.25 meters
- Binary mask:
- Foreground = 1
- Background = 0
![](https://cdn-images-1.medium.com/max/800/1*v4JQGmplp3Nsm-WicDgSAw.gif)
Gaussian Blur
![](https://cdn-images-1.medium.com/max/800/1*rRA91lSdGTWCo48MDx08Rw.png)
![](https://cdn-images-1.medium.com/max/800/1*uYvKPBeQG29UrptPn0aoBA.gif)
Gamma Adjust
![](https://cdn-images-1.medium.com/max/800/1*v88G9rxc-a4QAYe_2XDA7A.png)
![](https://cdn-images-1.medium.com/max/800/1*lQSZc12sp9h8vUH0ElCkbA.gif)
Binary Mask
Alpha matte
![](https://cdn-images-1.medium.com/max/800/1*hecMwIF_4g8pEeQDQmaSBQ.png)
Clamp Before Filtering
Avoiding edge softening
![](https://cdn-images-1.medium.com/max/800/1*6Ti15mK-QVr-IZu4VuLl6w.png)
Blending Foreground with Background
![](https://cdn-images-1.medium.com/max/800/1*sI-Cip_a8ydh12VEXYH3aQ.png)
TrueDepth Stream
640 x 480 depth map registered to video at 30 fps
![](https://cdn-images-1.medium.com/max/800/1*Gk2Mj9ZBt2FAef99s9e1VQ.png)