Assignment: Image Processing Coursework
A System for the Recognition of Hand Gestures
Section A:
Example outputs for Section A are provided in Appendix A. For each question in Section A, provide a source code script (properly formatted, see Appendix B), any numerical measurements requested, and a brief evaluation of the success of the operation. In total, the text for section A should not exceed 500 words. Note: these tasks do not require loops if implemented efficiently.
A1. First we will prepare a test image of your hand. You might like to remove rings, wristwatches, and other coloured objects that may interfere with detection. Take one set of coloured fingertips (supplied), and keep them safe. If you need a replacement set, you will need to purchase them.
Take a colour photograph of your left hand, palm flat down against a sheet of white A4 paper on which you have written your SID number written in black ink, attempting to capture only the paper (rather than the surrounding table), like the example shown in Fig. 1. A template sheet (for printing) with a box in which to write your SID number is provided in the file template.
Note the order with which the coloured fingertips are to be placed on the fingers. The fingertips (properly called the distal phalanx) should be worn in the following order: orange for pollex (thumb), yellow for digitus secundus manus (index finger), blue for digitus me'dius (middle finger), green for digitus annula'ris (ring finger), and red for digitus mi'namus ma'nus (little finger).
Once you have your image, load it into MATLAB, produce a greyscale version, and then display both images side by side (see Fig. 1). State the equation and name of the greyscale conversion.
A2. Next we collect and display some statistics relating to the images from question A1. We will plot normalised histograms of the RGB and greyscale images, and report the extrema (max and min) and arithmetic mean for each colour channel (R, G and B) and the greyscale image. Examine your histogram and collect a new test image if it seems to be under- or overexposed. Note that you can plot lines for the red green and blue channels on the same axis for the RGB histogram if you use lines rather than bars (see Fig. 2). Axes should be fully labelled and appropriately scaled. Plot the arithmetic mean of each colour channel, and the greyscale mean, as vertical dotted bars on the respective histograms.
A3. Next, use high, low, band pass, and band reject thresholding on the RGB channels of the colour image (as required), to produce five binary images (thumb, index, middle, ring and little) that isolate, as well as possible, the five coloured fingertips into five separate images wherein 1 = in fingertip and 0 = not in fingertip. Present these in a figure, along with isolated RGB pixels from the original colour image (see Fig. 3, which shows an imperfect outcome). Report the thresholds and type of thresholding used. Note that you will need to experiment to find red, green and blue threshold values and threshold types to achieve the desired effect.
A4. Next, implement code that places the coloured fingertips on top of the greyscale version of the hand image, to emphasise their locations (see Fig. 4). Also, in this question, report the size of the five fingertip pixels' clusters identified (i.e., the cardinality of each set)
A5. Next, calculate the average coordinate of each of the five pixel clusters, and display the image created in A4 annotating with 5 circles (for the five average pixel cluster coordinates), and four green lines connecting adjacent fingers (see Fig. 5). Report the coordinates of the five cluster averages and the lengths of the 5 lines (i.e., the Euclidean distance between each adjacent pair of fingertips). To accomplish this, you may need to investigate the find function.
A6. Next, using the RGB image from question A1, devise thresholds that isolate the skin from the background. You may wish to conduct some research to determine appropriate values to use, and should consider constructing versions of the colour image in alternative colourspaces, such as HSV (hue, saturation and value). Once you have achieved satisfactory segmentation (i.e., a binary image in which the skin is 1 and the background and fingertips are 0), apply the following edge detection methods to see which yields the most accurate edge contour along the outside boundary of the hand: 1. Robert's, 3. Prewitt, 4. Sobel, 4. Canny. Take the contours produced by these methods and trace a red line around the original RGB image (see Fig. 6, which shows an imperfect outcome).
Section B:
For each question in Section B, provide code, test output figures, and text describing your approach and how well the solution operated. In total, the text for Section B should not exceed 1000 words (e.g., 250 words per question).
B1. Collect another set of images with your left hand palm upwards this time in which you make the following five hand gestures: (a) flat hand, (b) thumb touching index finger curved down into palm, (c) thumb touching middle finger curved down into palm, (d) thumb touching ring finger curved down into palm, (e) thumb touching little finger curved down into palm. Using the code you developed above, and some additional logic (e.g., based on Euclidean distance between fingertip clusters), be able to determine which one of these five gestures is represented in the image loaded.
B2. Evaluate whether the use of morphological operators improves the identification of the binary clusters belonging to each fingertip (i.e., remove false positives and closes small holes) from Section A, and whether this improves the estimation of the central coordinate of each. Describe the approach taken and provide test output images. Next, evaluate whether morphological operators improve the classification of the gestures described in question B1.
B3. Evaluate whether the use of hysteresis thresholding improves the identification of the binary clusters belonging to each fingertip (i.e., remove false positives and closes small holes) from Section A, and whether this improves the estimation of the central coordinate of each. Describe the approach taken and provide test output images. Next, evaluate whether hysteresis thresholding improve the classification of the gestures described in question B1.
B4. Collect test images and develop code for classifying a range of additional gestures based upon the relative positions of the five fingertips - for example, can gestures corresponding to the 10 decimal digits (0..9) be interpreted? Codes should also be operated in real-time using a webcam input, perhaps with a mouse click or button press to indicate that the gesture is ready for interpretation. You may wish to start by looking at the snapshot function, which acquires single frames from the computer's webcam.
Format your assignment according to the following formatting requirements:
1. The answer should be typed, double spaced, using Times New Roman font (size 12), with one-inch margins on all sides.
2. The response also include a cover page containing the title of the assignment, the student's name, the course title, and the date. The cover page is not included in the required page length.
3. Also Include a reference page. The Citations and references should follow APA format. The reference page is not included in the required page length.
Attachment:- Image-Processing-Assignment.rar