Introduction to Computer Vision

CS150-09

This course is an introduction to low and intermediate level Computer Vision. We will learn how to design algorithms that process visual scenes to automatically extract information. The course will cover fundamental principles and important applications of computer vision, including image formation, processing, detection and matching features, image segmentation, and multiple views.

Cover photo: “symbolism painting of computer vision” by Linaqruf/anything-v3.0 model

Hours: Monday - Wednesday, 4:30PM – 5:45PM

Location: Joyce Cummings Center, Room 265

Instructor: Dr. Roy Shilkrot roy.shilkrot@tufts.edu roy@cs.tufts.edu LinkedIn GitHub Twitter

Office hours: Schedule online with Prof. Roy: https://calendly.com/royshilkrot/15min 

Prerequisites: Algorithms and data structures, linear algebra (I or II), statistics and probability theory (I), calculus (I), working knowledge of Python

Teaching assistant: Li Yukun Yukun.Li@tufts.edu 

TA Office hours: Wednesday 6:30 pm-7:30 pm: https://tufts.zoom.us/j/3088684222 

Piazza: https://piazza.com/tufts/spring2023/cs15009 

Course Goals

By the end of the semester, students should be able to:

  1. Identify and understand the fundamental aspects and techniques of Computer Vision
  2. Interpret and analyze a given problem using these techniques
  3. Adopt and Implement the appropriate techniques to solve a given problem

Textbooks

We will follow chapters from the books below:

  1. Computer Vision: Algorithms and applications, 2nd Ed (2022). Richard Szeliski. The book PDF is freely available online (http://szeliski.org/Book)
  2. Simon J.D Prince. Computer Vision Models (2012)
  3. Aeorlien Geron. Hands on Machine Learning 3rd ed. (2022)
  4. Francois Chollet. Deep Learning with Python 2nd ed. (2021)
  5. Excerpts from Hartley & Zisserman (2004), and Marr (2010)

Syllabus

Class

Date

Content

Assignment

1

1/18

Introduction: Class, Computer Vision, Math

Slides

Video

2

1/23

Image Formation: Human Vision, Optics, Physics, Cameras

Slides

Video

3

1/25

Image Formation cont.: Perspective projection, Pinhole camera model, Digital Imaging, Colorspaces, Sampling

Slides

Video

HW1: Hello Vision World

4

1/30

Image Processing: Filters, Edges: Convolutions, Separability, Local operators, Frequency domain and Fourier Transform, Histograms, non-linear (median, bilateral), Pyramids

Slides

Video

5

2/1

Features: Interests points, Corners, Harris, Blobs, FAST, Multiscale

Slides

Video

HW2: Pyramid blending, Corners & Blobs, Key points

6

2/6

Features: Descriptors, Gradient Histograms, Edges & Contours, Snakes & Scissors

Slides

Video

7

2/8

Geometric / Planar Transforms Parameterization, Feature Matching, Least Squares optimization

Slides

Video

HW3: Feature Extraction & Matching, Snakes

8

2/13

Image Alignment, RANSAC, Stitching, Panoramas, Cylindrical coordinates, Global optimization / Bundle Adjustment, Blending

Slides

Video

9

2/15

Tracking: Motion estimation parameterizations, CONDENSATION, Particle and Kalman Filtering

Slides

Video

HW4: Panorama Stitching

2/20

No class - President’s day

10

2/22

Tracking: Correlation Filters, Optical flow, Lucas-Kanade, Shi-Tomasi

Slides

Video

HW5: Tracking

11

2/27

Multiview 1: Intro, Camera Calibration, Calibration and Projection Matrices

Slides

Video

12

3/1

Multiview 2: Parallel Motion Stereo, Depth and Disparity, Epipolar Geometry, Essential and Fundamental Matrices

Slides

Video

Mid-term prep

13

3/6

Mid-term prep

Slides

Video

Mid-term prep

14

3/8

Mid-term (in class)

15

3/13

Multiview 3: Triangulation, Camera Pose Estimation, Structured Light

Slides

Video

16

3/15

Bundle Adjustment, Intro to Visual Machine Learning

Slides

Video

HW6: MVG, Bundle Adjustment

3/20

No class - spring break

3/22

17

3/27

Linear Models, Linear Regression and Classification, Logistic Regression

Slides

Video

18

3/29

Linear SVMs, Neural Networks

Online: Gradient descent, Backpropagation and Learning Practices

Slides

Video

19

4/3

Convolutional Neural Networks, Deep Learning

Slides

Video

HW7: Hello CNNs, Classification

20

4/5

Object detection 1: V-J, Region Proposals Nets, R-CNN, SPP, Fast-RCNN

Slides

Video

21

4/10

Object detection 2: Faster-RCNN, YOLOs, SSD

Segmentation 1: Gestalt theory, Clustering, Meanshift, GMM, watershed, superpixels, Graph-Cut

Slides

Video

22

4/12

Segmentation 2: Types of Segmentation, FCNs, U-net, Upconvolutions, SegNet

Slides

Video

HW8: Object Detection

4/17

No class - Patriot’s day

23

4/21

Vision and Language: Introduction, VQA, Captioning, RNNs and Language Models, LSTMs

Slides

Video

24

4/24

Vision and Language II: Attention and Visual Attention, Word Embeddings, VQA, Transformers, ViT

Slides

Video

25

4/26

Vision and Language III: CLIP, Generative modeling, AE, VAE, GANs, Vision Applications, CLIP-guided Diffusion and Text-to-Image

Slides

Video

26

5/1

Final class. What have we missed? Further reading, final exam prep

Final exam prep

Problem Sets, Tests

Problem sets: Nine (9) problem sets will be given during the class. Any material regarding the set solution will be submitted electronically. Homework will be due up to one or two weeks from the assigned date at midnight. Late assignments are penalized at 20% for each 24 hours’ delay. No homework will be accepted one week after the deadline. If a serious illness or another major life event prevents you from completing a homework assignment on time, you should report the event to your instructions team, after which alternate arrangements can be made. Reporting must be done before the assignment in question is due.

Tests: There will be two (2) tests over the course of the semester. The tests will be a combination of multiple choices and open-ended answers. Books must be closed and no electronic devices are allowed. If, for any reason you must miss a test, you must inform me before the day of the exam and schedule a make-up session. Please Note: If you wish to dispute a grade, it is mandatory that you do so within one week of receiving the grade. After such a term, the grade will be considered final. Note: Grading is generous, re-grading is strict.

Final grades: You must show proficiency in all grading areas to pass the class. A failing average (below 50%) in any of the grading areas (problem sets, tests, attendance) will result in a failing grade in the class.

Your final grade will be determined using the following percentage breakdown:

Problem Sets

50%

Tests

40% (mid 13%, final 27%)

Attendance, Behavior and Attitude

10%

Communications

Piazza will be our primary means of communication. All course announcements will also be made through Piazza.

Rather than emailing questions to the teaching staff, we encourage you to post your questions as a public discussion. You are also encouraged to help each other, as long as the question does not contain any code or portion of a problem set answer. In such a case, the question must be made private (please, refer to the section Academic Honesty below).

To schedule office hours outside the posted times, please email any of the teaching staff or post a private message on Piazza.

Academic Honesty

Science is, to its core, a collaborative effort. The advantages of coming together for examination or comparison, sometimes even just to explain the problem, are well known. I strongly encourage students to discuss course material, problems, and applications outside the classroom with the teaching staff and other students. You are also encouraged to form study groups for the tests.

Having said that, integrity and honesty are equally important qualities of any future academic, scientist or engineer. We take plagiarism very seriously. You must do homework, the problem sets and the final projects on your own. If you need help, the teaching staff will be more than happy to help you!

Academic Integrity Policy: Tufts holds its students strictly accountable for adherence to academic integrity. The consequences for violations can be severe. It is critical that you understand the requirements of ethical behavior and academic work as described in Tufts’ Academic Integrity handbook. If you ever have a question about the expectations concerning a particular assignment or project in this course, be sure to ask me for clarification. The Faculty of the School of Arts and Sciences and the School of Engineering are required to report suspected cases of academic integrity violations to the Dean of Student Affairs Office. If I suspect that you have cheated or plagiarized in this class, I must report the situation to the dean.