Computer Science
2002 - COMPSCI. 773 ST Robotics and Realtime Control
Introduction
This course introduces computational methods and techniques used in vision-based robotics and real-time control. Many topics are only overviewed, but a number of interesting theoretical and practical problems are analyzed in detail. You should not expect exciting things which may be found in sci-fi books or movies like "Terminator" as you will soon find out that even a seemingly simple robotics action may be a real challenge.Design of modern robotics and industrial control systems involves different mathematical tools, especially, optimization techniques, matrix analysis, and analytic 2D / 3D geometry. Some tools will be explained in brief in the lectures. Still, you are expected to learn these methods in details and use them to complete assignments.
Assessment
Assessment is based on 60% course work (20 % group work, 40 % individual work assessment) and 40% open-book final examination. Course work includes assignments that exploit the hardware (digital cameras, remotely controlled toy cars, and manipulators) available in the CITR Robotics Lab at Tamaki (room 731.234). For each assignment, Each group will have to write a report which should be organized as follow:- Each member of the group will have to work on a distinct part of the assignment and write an individual report.
- Each group will have to provide a report presenting the group solution and achievements for each assignment. Basically it should consists on introducing the problem and different solutions proposed.
Course work
A particular feature of the course work is the emphasis on complete system design. Therefore, instead of picking a small part of the material covered in lectures as assignment tasks, the project in this paper has the aim of developing a complete system to perform a specified task. The individual assignments present intermediate stepping stones toward achieving this goal. At the end of the paper, there will be a competition to evaluate your project.The equipment in the CITR Robotics Lab consists of a number of PCs running Linux. There are a number of cameras and two pan-tilt cameras forming a stereo system. We also use two medium sized remotely controlled Hummer cars. This year a DV camera will be added to perform hand-gesture recognition.
Nowadays, Human Computer Interaction
is a hot research topic. It consists mainly on extracting information (from
audio-visual speech, visual expression, hand signs, body expression) to
intercact efficiently with a computer. Potential applications range from
automatic speech recognition (ASR), videoconference, virtual reality, communication
for disabled people, user verification
and recognition (audiovisual biometrics features), remote control of
robots and devices.
This year course project will encompass
topics such as stereo vision and 3D positioning, feature extraction and
classification, motion planning and reinforcement learning with a focus
on real-time processes for efficient interaction.
Basically, you will have to remotely
control a robot via hand gesture recognition. Whenever a robot is
in adverse environment (such as a Nuclear power plant or deep under the
sea), the operator is not physically present. Then, the operator receive
information from the robot environment via sensors (radar, camera, etc..)
and guide the robot in regard of the planned task and potential threats
(obstacles, traps, doors, walls, bridge).
Your task will be to drive a car-like robot along a path with mandatory
check points in a 3D field. You will have to pass a bridge and safely move
the car from start point to endpoint. Requirements are:
- The car will be controlled by hand gesture recognition.
- The "driver" will not see the car but will know its 3D position.
The work is subdivided into three
assignments covering the following parts of the project:
- Calibration of stereo cameras for computation of 3D positions of an item in the cameras field-of-view by intersecting optical rays and visualization of 3D movements of the item (you will use in this assignment the existing Tsai calibration software but also do several programming tasks in networking, client-server camera access). You will build a GUI which will display the 3D position of the car in the field and control basic movements of the car (stop, forward, backward, turn left, turn right).
- Hand gesture recognition: You will have to perform hand localisation, hand mask extraction and hand signs recognition at real-time.
- Whole system testing: Integrate previous work to effectively control the car by hand signs. You will have to complete (by reaching the check points) a randomly chosen path by controlling the car through hand signs in a limited time. Motion planning should help you to follow the shortest way while clearing obstacles, reaching checkpoints and passing the bridge.
| Contents | Deadline | |
| Assignment 1 | Tsai's camera calibration and stereo calibration | 25.08.2002 |
| Assignment 2 | Hand Gesture Recognition | 04.10.2002 |
|---|---|---|
| Assignment 3 | Whole System testing | 25.10.2002 |
Groups
| Group | 1 | 2 | 3 | 4 | 5 | 6 |
| Students | amak008
ccha196 jhon019 mcha166 |
asri003
dand001 pgau003 |
dwil143
epur008 jlin054 jwan035 |
dzha021
hzhu008 xlin013 yche158 |
jyag001
nsye002 shil048 szha018 |
xye004
ywu034 yzou005 |
Marks
| Student | Asst 1 | Asst 2 | Asst 3 | Exam | Overall mark (grade) |
| 2390858
2379106 2387827 2490090 9878637 9931741 2275171 9913682 2361192 2284326 2135013 2333781 2157722 2376837 2404973 2481186 9873967 9782934 9772736 3006782 9889408 9772250 |
A
A- A A A- B- B+ A A- B+ B A- A- B+ A- B A+ B A- A+ A B |
People
The following people are involved in COMPSCI.773ST this year.- A/Prof. Georgy Gimelfarb (Lecturer and course supervisor).
- Dr. Patrice Delmas (Lecturer).
- BSc (Hon). Benn Vosseteig (Lab Supervisor and Teaching Assistant).
Topics
- Basics of digital signal processing
- Random processes and linear systems
- Wiener filtering
- Adaptive signal processing
- Kalman filtering
- Active Vision
- Single and stereo cameras calibration
- 2D and 3D projective geometry
- Color detection and discrimination
- Binary machine vision
- Feature extraction and classification
- Basics of applied AI
- Reinforcement learning
- Motion planning
Wed: 12.30 pm - 2.30 pm, room:
723.203
Fri: 12.30 pm - 1.30 pm,
room: 723.203
Preliminary schedule
| 1 | Introductory lecture | P. Delmas, B.Vosseteig | 24.07.2002 |
| 2 | Robotics vision: an overview - Pt.1 | P. Delmas | 24.07.2002 |
| 3 | Robotics vision: an overview - Pt.2 | P. Delmas | 26.07.2002 |
| 4 | 2D and 3D vision geometry | P. Delmas | 31.07.2002 |
| 5 | Camera calibration | P. Delmas | 31.07.2002 |
| 6 | Stereo cameras calibration | P. Delmas | 02.08.2002 |
| 7 | Color Imaging - Pt.1 | P. Delmas | 07.08.2002 |
| 8 | Color Imaging - Pt.2 | P. Delmas | 07.08.2002 |
| 9 | Binary image segmentation - Pt.1 | P. Delmas | 09.08.2002 |
| 10 | Binary image segmentation - Pt.2 | P. Delmas | 14.08.2002 |
| 11 | Networking, client-server camera access, device drivers | B. Vosseteig | 14.08.2002 |
| 12 | Feature extraction | P. Delmas | 16.08.2002 |
| 13 | Feature classification: PCA and other methods Pt.1 | P. Delmas | 21.08.2002 |
| 14 | Feature classification: PCA and other methods Pt.2 | P. Delmas | 21.08.2002 |
| 15 | 3D scene description/understanding | G. Gimel'farb | 23.08.2002 |
| 16 | Motion planning - Pt.1 | P. Delmas | 28.08.2002 |
| 17 | Motion planning - Pt.2 | P. Delmas | 28.08.2002 |
| 18 | Real-time image processing | G. Gimel'farb | 30.08.2002 |
| 19 | Discrete Random Processes - Pt.1 | G. Gimel'farb | 18.09.2002 |
| 20 | Discrete Random Processes - Pt.2 | G. Gimel'farb | 18.09.2002 |
| 21 | Discrete Linear Systems - Pt. 1 | G. Gimel'farb | 20.09.2002 |
| 22 | Discrete Linear Systems - Pt. 2 | G. Gimel'farb | 25.09.2002 |
| 23 | Discrete Wiener filtering | G. Gimel'farb | 25.09.2002 |
| 24 | Adaptive filters - Pt.1 | G. Gimel'farb | 27.09.2002 |
| 25 | Adaptive filters - Pt.2 | G. Gimel'farb | 02.10.2002 |
| 26 | Stochastic approximation | G. Gimel'farb | 02.10.2002 |
| 27 | Kalman filtering - Pt.1 | G. Gimel'farb | 04.10.2002 |
| 28 | Kalman filtering - Pt.2 | G. Gimel'farb | 09.10.2002 |
| 29 | Kalman filtering - Pt.3 | G. Gimel'farb | 09.10.2002 |
| 30 | Basics of AI: reinforcement learning - Pt.1 | P. Delmas | 11.10.2002 |
| 31 | Basics of AI: reinforcement learning - Pt.2 | P. Delmas | 16.10.2002 |
| 32 | Basics of AI: reinforcement learning - Pt.3 | P. Delmas | 16.10.2002 |
| 33 | Binocular and Trinocular Stereo - Pt.1 | G. Gimel'farb | 18.10.2002 |
| 34 | Binocular and Trinocular Stereo - Pt.2 | G. Gimel'farb | 23.10.2002 |
| 35 | Binocular and Trinocular Stereo - Pt.3 | G.Gimel'farb | 23.10.2002 |
| 36 | Course overview and final demo | P. Delmas, G. Gimel'farb | 25.10.2002 |
Recommended texts:
- S. B. Niku, Introduction to Robotics: Analysis, Systems, Applications. Prentice Hall, 2001.
- R. M. Haralick, L. S. Shapiro : Computer and robot vision, Vol II, Addison Wesley, 1993.
-
Related Programmes
