Software Security

CompSci 725 FC 03
Clark Thomborson, James Goodman
Handout 16: Samples of our Feedback on Oral Presentations and Synopses

Draft (1 point):

• 0/1 You did not give a draft presentation in a tutorial session, although you did send me draft slides on dd mmm. Your presentation was on the following day.

Slides (7 points):

• Full bibliographic information should include important data about the publication (i.e. LNCS XXXX, “Title of Publication”, year of publication), publisher (i.e. Springer-Verlag), and page range, in addition to the name of the conference, author and article (which you have provided).
• Your slides had no “focus” — you covered the entire article. You should have shown your critical and appreciative understanding by emphasizing the most important point or points.
• Your appreciative comment was really a summary or paraphrase of a large segment of your article: it wasn’t clearly focused on anything that you believe to be of particular interest to your classmates in COMPSCI 725, and you didn’t give any indication of why you thought it was particularly interesting.
• Overall your slideshow showed some understanding of your topic, however I see little indication of a strong understanding because your appreciative and critical comments did not give evidence of any in-depth analysis.
• Your criticism was very similar to the previous presenters’ criticism (see http://www.cs.auckland.ac.nz/compsci725s1c/assignments/presentations/xxx.ppt) -- a similar criticism would be fine if you had developed it in a novel way that showed your own understanding.
• I would have been happier with your second criticism if you had phrased it in standard terminology, for example that of Pfleeger.
• Overall your slideshow didn't “cut below the surface” of this article: you didn't develop any of your comments in any detail.
• You introduced a change to the authors’ example on your slide XX, for no apparent reason.
• You did an excellent job of correcting a mistake made by the author.
• Your slides do not show a strong understanding of the paper.
• You misspelled some technical words, possibly causing confusion.
• Good effort, especially in redrawing the figures for clarity in the slideshow.
• The slideshow was generally unsuccessful in carrying the meaning of any important point in your article, because you omitted the definitions of key concepts.
• You had too many slides, with insufficient “focus” on the most important elements. Your oral presentation was rushed as a result.
• You spoke at great length on several topics, which were only mentioned by name on your slide. (Your slides should contain the most important information, so that your audience can read it, as well as hear it. If a topic is important enough to talk about for a long time, you should revise your slide to contain more information about it.)
• Good length & amount of detail.
• Your slideshow was successful at explaining some difficult and important concepts.
• Your analysis on slides XX was an excellent linkage to material presented earlier in Compsci 725.
• You introduced some excellent diagrams, not present in the original text.
• Your slides were carefully prepared in general, although some of your points were not entirely clear or even incorrect. We don't believe you are paraphrasing XX accurately when you write (on slide X) “XXX”. What XX actually wrote was “YYY”.
• Overall your slides showed an adequate but not complete understanding of your topic.
• Your slides include several very insightful comments on the principles underlying the attack by YY, leading me to believe that you have an excellent understanding of this portion of the paper -- stronger than my own. [However] your slides do not exhibit a strong understanding of the security implications of this attack.
• I'm unable to find textual support [in your article] for your assertion (criticism #X) that the authors have made an incorrect claim to XX. … The validity of your criticism #X seems thus to rest on [your] restatement or interpretation of the authors' claims to generality.
• Your characterization of XX, on slide X, as being a "XXX" is not accurate.
• Because you have not made any explicit critical or appreciative comments [in your slideshow] I will accept your diagrams on slides X and Y as non-explicit criticisms, and give you partial credit for these.

Timing (2 points):

• 2/2 You finished your slideshow in 6 to 10 minutes.
• 1/2: You completed in 5 or 6 minutes.

Question (2 points):

• 2/2 You posed a stimulating question.
• 1/2 Your question was rather open-ended, although it did stimulate discussion.
• 1/2 Your prepared question was not thought provoking; instead it could be answered by “pattern-matching” on words and phrases in your slideshow.
• 1/2 Your question was very difficult: are you able to answer it? (If not, then there’s little chance that anyone in the audience will be able to do so.)
• 1/2: Your question about ethics was rather open-ended and vaguely stated, making no explicit use of the ethical terminology and frameworks presented earlier this term.
• 0.5/2 Your question doesn't show much understanding of your topic, nor are you using the standard terminology of a security analyst.
• 0.5/2 Your question stands somewhat in contradiction to your appreciative comment…
• 0/2 You had not prepared a question in your slideshow.

Discussion (3 points):

• Your limited participation in the discussion did not demonstrate much understanding of the paper.
• You gave an incorrect answer to my question.
• Your answer to my question did not show any depth of understanding of material in your slideshow.
• You handled the discussion very well, showing excellent understanding of the material you presented.
• You defended your critical appreciation ably in the Q&A session.
• Overall in your discussion you gave me the impression that you were much more interested in defending your point of view, rather than in encouraging an open discussion that might reach a better understanding after taking other points of view into full consideration.
• Overall you seemed to be struggling to keep up with the discussion; most of your questioners seemed to know more about your chosen topic than you did.
• 3/3: excellent discussion, you (and your audience) raised a number of interesting points. (In the future you probably should allow your questioners a little more time to complete their points, before you step in with a response.)
• 3/3: You held your own in the discussion, without dominating it. Several interesting points were made, to which you had appropriate responses.
• 2/3: your topic generated a spirited discussion, however we didn't hear you making any particularly insightful contributions.
• 2/3: Unfortunately your good question [for which you received 2/2 marks] did not lead to a stimulating discussion.
• 0/3: The people in the audience who contributed to your discussion showed more understanding of this paper than you did in your response to their comments.

Oral Presentation Totals (15 points maximum):

• A histogram of total marks is available at http://www.cs.auckland.ac.nz/compsci725s1c/assignments/oralhist17.htm.

Assignment 1 (ungraded): Draft Title, Synopsis, Reference List

Title:

• This does not match your synopsis.
• This is a good description of your synopsis.
• Too wordy.
• Too vague.
• This title is somewhat broader than your synopsis, which only promises to examine three of the many available methods for [accomplishing] XX. Perhaps you should add the qualifier "Three" to the front of your title.
• Your title is quite short, so you might add some words to more clearly convey your topic. Your current synopsis describes a survey paper -- so the word “Survey” could be added to your title (unless you refine your topic -- and I strongly recommend you narrow your topic, see below).
• Before you finalise your title for submission on your term paper, I strongly encourage you to ask advice from a native English speaker. Because many people will read your title, it is important that it be as clear as possible -- and thus you should make special effort (well beyond what you put into the rest of your paper) to detect & correct any grammatical errors.
• Your title contains two uncommon acronyms, so it will not be helpful to most potential readers.

Synopsis:

• You must find a much narrower focus for your paper, otherwise you'll be writing in such vague generalities that you cannot possibly show your critical and appreciative understanding of your sources. … You should also pick one audience and stick with it: you won't be able to write a paper that is appropriate for both “common people” and “specialists”. As I mentioned recently in class, I expect your term paper to be written for an audience that knows stage-3 computer science.
• This sounds interesting but is far too ambitious. … I think you should try to focus your topic, perhaps by considering only two of your three alternatives, and/or perhaps by focussing on a very short list of security objectives (perhaps with a single entry), and/or perhaps by focusing on a very narrow range (perhaps a single one) of applications for [security technology XX].
• You should probably focus on either XX or YY, and not try to cover both.
• I suggest you revise both your title and your synopsis, to focus on the security issues in just one of the many applications of XX. If you try to cover all issues and all applications, even briefly, then I doubt you will have much chance to show your critical and appreciative understanding.
• I am pleased to see that the three of you have independently submitted your project plans. However I am not responding to your plan independently of the others, for I don't see any indication that you have developed independent goals for each of your term projects. I am happy for you to cooperate closely -- especially on setting up [your experimental framework] -- but I will insist that each of you develop and pursue your own line of … practical work, for writing up in your term project report
• Your synopsis does not clearly convey a goal. Vaguely planned experimentation that might lead to “practical experience” is not an acceptable goal for a term project.
• I have learned to distrust any technical argumentation that begins with the phrase “It is plain to see that...” or any similar phrase -- generally such argumentation tells me that the writer is unable (or unwilling) to supply a good reason why anyone should believe the assertion that follows.
• I don't know what you mean by the phrase "the reliability of the results received". Please be careful to distinguish between false positives and false negatives.
• … I strongly suspect you're right to think that there's very little published information on XX, other than website promos and other unrefereed ephemera. I think you'll find at least a few substantive white-papers, i.e. I found http://www.xxx.pdf by a Google search on “XX YY”. One way to handle this shortage of refereed publications on your topic is to treat the ephemera as "primary sources" in much the same way as a historian might treat personal letters, diaries etc. Such sources are authoritative on some topics, i.e. documenting the author's point of view at a particular point in time. If you pursue this line of enquiry, don't try to verify any of the claimed properties of the XXs on offer; but instead you can analyze the product literature for its claims -- what are the primary “selling points” of the current batch of XXs? How are these points described, i.e. do the authors invent their own terminology or do they adopt someone else's terminology?
• Your synopsis does not clearly define a topic. Are you proposing to take a critical and appreciative look at the proposal by XX, with particular reference to earlier proposals by YY and others?
• All of your references are at least 4 years old. I suggest you do a "forward literature search" (ask a librarian for assistance on this, or use Citeseer to discover more recent publications that reference either the XX paper or one of YY’s attack papers. I think you'll find that ZZ is a very large and well-researched area. So an appropriate challenge for you to meet in your term paper is to define a small piece of this area, where you'll have a chance of developing your own critical and appreciative understanding.
• You have proposed to write a survey paper of the sort that would be appropriate only if you can't find any previously-published survey paper on XX. If you can find a survey paper, then I'd expect you to work within the structure (i.e. taxonomy, terminology, methodology) set up by the previous writer(s) of survey papers -- you could add new items to their taxonomy (if it omits some promising techniques in XX), you could critically & appreciatively examine the methodology (or methodologies) used in previous surveys to compare/contrast different approaches to XX, etc. If you can't find a survey paper, then you have a more open field -- your goal, in writing the paper, is to develop a consistent terminology and perhaps a reasonable taxonomy for classifying some of the existing methods. …I don't think it's realistic to expect a term paper to be an exhaustive survey of any field. You'll be much better off reading (and writing on) a few papers carefully, rather than spending a lot of time trying to discover and report on all XX systems.
• You have found six very solid sources, congratulations! However -- the concept of XX is quite subtle, and has many applications in software security, so I strongly recommend that you focus your research on one way in which XX can be applied to achieve a specific security goal. Otherwise I think you will almost certainly write an incoherent paper that will not show much critical or appreciative understanding of the many topics you might try to cover.
• Your synopsis gives me no indication of what you're expecting to do with the information in XXl's article, other than that you might be intending to paraphrase it. This would not be acceptable: I expect all COMPSCI 725 students to write a term paper showing critical and appreciative understanding of at least three articles.
• In any technical paper you should treat your subject fairly -- you must disclose all relevant data, especially if this data would tend to weaken your argument.

Reference List:

• No matter what you choose as a specific focus for your paper, I will expect you to draw on your knowledge of (and make relevant references to) article XX [on the required reading list of COMPSCI 725].
• When you read your articles, you should note the page range and include this information in your bibliography.
• Congratulations on your literature review, you have made an excellent start on your term paper. I think once you focus your topic, you will quickly develop a “point of view” that allows you to write a coherent term paper that draws on the sources you list.
• Your reference list is rather short for such a broad topic. I think you should search the literature again after you focus on a more specific topic. However: the references you have found seem to be substantial and authoritative -- although I haven't read any of them except the first!
• You have supplied far too little bibliographic detail. Any article posted to CNet news.com would not be an appropriate primary reference for a term paper (i.e. do not use it as one of the three required sources on which you will make critical and appreciative comment). However it is a reputable news bureau so it could well be appropriate for you to cite one or more news articles to support a factual assertion (i.e. about current-year sales of [some product], or about [someone’s] testimony before the US Congress).
• The phrase “fair use” has only a vague moral/ethical meaning in jurisdictions (such as NZ) where it doesn't have a legal meaning. If you want to use this phrase in your term paper or project report, you must be careful to define what you mean by it!
• Your references are somewhat carelessly constructed (with typos, inappropriate abbreviations etc) and they lack some important bibliographic information, i.e. date of publication.
• I suspect you have access to a draft version of an article which you obtained somehow from XX or one of the other authors. This would be fine as a primary source for your COMPSCI 725 term paper, as XX (http://www.YY.edu.ZZ/~XX) is a respected academic researcher and this work seems (from its title) to be quite interesting and relevant to COMPSCI 725. However if you use any such non-archival source, you *must* cite it very carefully, to reveal its exact provenance to any reader who might want to refer to your actual source.
• After reading your synopsis … I don't see any reason why the XX paper on YY should appear in your reference list.
• You have done an excellent job of tracking down, and citing, what (to my untutored eye appear very likely to be) very appropriate references in your topic area. Congratulations.
• Authors' institutional affiliations are not relevant to a citation, unless the cited material is an unpublished document which you obtained directly from the author -- and if you cite such unpublished documents you must give full information on its provenance (including the date you received it and how you received it).
• I don't think you should start work on this term paper by reading books. Instead I recommend you search for, and read, technical articles that will help you build on what you have learned in some previous computer science coursework.
• The trouble with using books as sources for a COMPSCI 725 paper is that a good book contains far too much information for anyone to read from cover-to-cover in the time available, if you are thinking carefully enough about what you are reading that you'll be able to show your critical and appreciative understanding in your term paper. So I strongly suggest you pick a topic that is quite narrow, so that only a chapter of each book is directly relevant to your topic. Then you can have a chance of reading this chapter carefully enough to develop a strong understanding of how it relates to other authors' treatments of your chosen topic.
• Your references [1] and [2] will, I think, have very similar information. You should look for a third reference, ideally one that clearly describes an XX implementation that is focussed on a specific application (perhaps YY) with specific security goals. You'll also need an authoritative reference that will support whatever definition you choose, in your term paper, for the phrase "XX". You'll have to define this term very briefly in your abstract, and more carefully in your introduction. (One way of determining authority is by looking at the reference list of the article in question -- if it refers [appropriately] to other authoritative sources, then you should gain confidence in its authority.)
• I expect COMPSCI 725 students to provide a reference list that provides complete and accurate bibliographic information in a consistent style. Your style is somewhat inconsistent in its ordering of information (especially about page numbers), and you have suppressed important information. …It is generally considered inappropriate style to abbreviate titles of articles in a reference list.
• … you are proposing to write on a topic that has already attracted a lot of attention. Much of what is written is polemic in nature, rather than academic; I'm pleased to see that you have picked academic sources.