Big

lecture 06
object detection with Create ML
+
live capture APP

after today's lecture:
-- object detection: how to build data, how to train 🤖
-- a live capture app that recognises sea creatures 🦈

What's the difference between a static image, a static video, a live capture video?

on digital devices, video is nothing more than a sequence of images(frames)
=>
most video processings eventually boil down to good old image processings
=>
all the image-based AIs in our toolbox are ready to go for video processing

One key difference between static video and live capture video is that static video is pre-recorded, while live capture video is captured in real-time

pre-recorded video: a fixed set of frames
vs.
live capture video: a dynamic set of frames that keeps being updated

real-time processing is an interesting topic (think about digital music instrument)
where processing speed imposes bottleneck for how "real time" the output is

in apple framework, we have AVFundation doing live capture for us
and we are counting on CoreML and optimised chips to have fast AI computation that brings smooth real time experience rather than laggging

next: train our own object detection model using CreateML (no python this time) and integrate it into the app

the re-occuring machine learning workflow throughout our unit

part 1: data collection
CreateML has its own data format for training, check it out here
10 mins read, take a note of keywords and alien words!

to save us some headaches from finding data, annotating and formatting
introduce... roboflow

These datasets that are annotated, split and you can choose to download from a range of formats including CreateML format 🥰

15 mins browsing this dataset library, find one dataset that you like
take a note of:
-- how many images are there?
-- what are the classes available?
-- have you seen the annotation window?
-- does each image have the same dimension?

also ask chatgpt why dataset is split into training/validation/testing sets for training an AI?

validation, evaluation and testing are very similar notions (almost interchangeable)
they all seek to answer this question "how does my model perform?"

the keyword to model performance is *generalizability* which can only be evaluated on unseen data
using a set for training or selecting will "pollute" its unseenness

let's try the aquarium dataset!
after clicking the "Download" button
don't forget to select the life-saving "CreateML" format

your turn:
--1. add train/val/test data folder into the data sources
--2. select transfer learning
--3. enter a smaller number of iteration (e.g. 1k)
this is just for lecture demonstration purpose, in practice you can try a larger number of iterations
--4. fire off the training!!!

what does I/U mean?
it is a metric for measuring the similarity of two bounding boxes (the groudtruth box and the box predicted by the newbie AI)
15 mins read here
-- what does I/U 50% mean?

import the model to our live capture APP and change this line:

change the model name in line 24 to your model class name

question 1: from the entire pipeline
-- data collection -- training -- integration
where do we specify/input the target classes information?

this is equivalent to ask, if i don't want a sea creature live detector but a car model live detector insteads, what are the steps to go?

question 2: can you think of possible extension/modification of this app to do some cool AR stuff?

20-30 mins lil exercise:
-- select another dataset on Roboflow and download
-- train an abject detector using CreateML (enter small interation number)
-- update your app with the new model

today we talked about:
-- static vs. live capture content
-- real-time processing, computation speed as bottleneck
-- object detection datasets on Roboflow
-- object detection training using CreateML
-- object detection evaluation metric
-- integrate new model into live capture app