One of the modules I teach at University is dynamic web development. The COVID-19 pandemic means I will be using a blended learning approach in the upcoming semester, and I have been investigating some tools which may help with this. Specifically, I have been experimenting with drawing on live videos (heavily inspired by Norm Chan’s videos). It would allow me to sketch out coding solutions to common issues students encounter whilst using web technologies.

I thought I would share my set-up in the hope it helps my fellow academics.

Software Required

Initial set-up on the MacBook
First of all, you will need to install Open Broadcaster Software (OBS). OBS is a piece of free, open-source software for recording videos (or live streaming). It is available on Windows and Linux too.

You will also need some way of mirroring the iPad to the MacBook. There are various pieces of software available which will do this. However, I settled on AirServer as it was quick and easy to use. I installed the trial version for my demo to see how well it worked. AirServer is available Windows and the Xbox in addition to macOS (sorry Linux users, you will need to find an alternative).

Initial set-up on the iPad Pro
Please note, I have specifically mentioned the iPad Pro because that is what I am using, but I assume this would work for the newer version of the regular iPad since it supports the Apple Pencil.

The only app you will need to install on the iPad for this demo is Adobe Fresco. It is a free digital painting app (there is a premium version available). I imagine this could also work with similar digital painting apps such as Procreate.

Annotating videos live
On the MacBook, start by running AirServer. You may not see much happening on-screen at this stage. Move over to the iPad and enable screen mirroring (swipe down from the top-right of the iPad, hit screen mirroring, and wait for the name of your MacBook to appear- tap on it when it shows). The iPad will then mirror to your MacBook.

Next, open up OBS. At the bottom part of the screen, click on the + in the Sources section. You will want to add a Video Capture Device (your webcam), and a Window Capture (AirServer iPad mirror). Drag/drop the ordering so the iPad (Window Capture) sits on top of the MacBook webcam (Video Capture Device) (Figure 1). You may need to tweak the iPad size so it fits on top of the MacBook webcam.

sources
Figure 1 – Source options.

We’re part of the way there, though currently the webcam feed is obscured by the iPad. To fix this, move over to the iPad and open up Adobe Fresco. Create two layers. Make the bottom layer green (RGB 0, 255, 0). Add a transparent layer on top (Figure 2).

Adobe Fresco Set-up
Figure 2 – Adobe Fresco Set-up

Move back over to OBS on the MacBook- a filter is needed to handle the green screen. Right-click on Window Capture under Sources and click on Filter (Figure 3).

filters
Figure 3 – Filter select.

At the bottom half of the screen, you will see Effect Filters. Click on the + and select Chroma Key. Select the Key Colour Type of Green (you can have a custom colour if you so wish). You will notice you no longer see the green from the iPad. Instead, it looks grey (Figure 4).

green screen iPad
Figure 4 – iPad mirrored in OBS.

That’s everything set up. You can now move over to the iPad and draw on the transparent layer with a brush, using the Apple Pencil (e.g. I wrote “hello” in the YouTube video below). There is a small amount of latency, but on the whole, it functions well.

Hopefully this post is helpful!

(very) brief demo.