Keep in mind that your very smart friend (VSF) has not read the paper, which is why you should: - explain the task before you explain their approach/method. - summarize the evaluation paradigm - cite the most important number to support the claim that their method improves on the baselines
If you include terminology in your review, e.g., “ODP bias”, you have to explain it, because your friend would not know what that means.
Because you don’t have much space in your review, you have to separate which details of the approach are central, which ones are unimportant. (I think this is one of the most difficult parts of writing a good reading note. Often it takes a second read through the paper to determine this.)
You can avoid repeating phrases such as “The author states”, by describing the approach an idea that exists independent of the author (who was merely the person who wrote it up). I usually use language like:
“The paper is about a method that… . The idea is to X, then Y, because … This approach is evaluated in comparison to … , by using data from … and evaluating it with xyz measure. The results suggest that”.
It is usually easy to criticize a paper with respect to the benchmark collection, because there is never enough data, reviewer, assessments, etc. While you should voice your criticism, I would appreciate if you could also say something more substantial.
You should voice flaws in the evaluation paradigm in general. However to be would be fair you should also include a comment, where you assume that the evaluation paradigm is sound about whether the results show improvement over the state-of-the-art. (and hence justify the claim that their method is better than the baseline systems).
In general I encourage you to differentiate between fatal flaws (i.e., no significance testing ;) from areas with potential for improvement.
In future, can you also comment on how the paper could contribute to a solution to our shared task?
Hint: the task was chosen so that all papers that we will read can be used to construct a solution to the task. This does not mean that it will lead to significant improvements. You also don’t have to implement it. All I want is for you to think out of the box.