- no comparison against baselines. The functionality similarity comparison study reports only accuracy across optimization levels of binaries, but no baselines are considered. This is a widely-understood binary analysis application and many papers have developed architecture-agnostic similarity comparison (or often reported as codesearch, which is a similar task).
- rebuttal promises to add this evaluation
- in addition, the functionality similarity comparison methodology is questionable. The authors use cosine similarity with respect to embeddings, which to me makes the experiment rather circular. In contrast, I might have expected some type of dynamic analysis, testing, or some other reasoning to establish semantic similarity between code snippets.
- rebuttal addresses this point.
- vulnerability discovery methodology is also questionable. The authors consider a single vulnerability at a time, and while they acknowledge and address the data imbalance issue, I am not sure about the ecological validity of such a study. Previous work has considered multiple CVEs or CWEs at a time, and report whether or not the code contains any such vulnerability. Are the authors arguing that identifying one vulnerability at a time is an intended use case? In any case, the results are difficult to interpret (or are marginal improvements at best).
- addressed in rebuttal
- This paper is very similar to another accepted at Usenix 2023: Can a Deep Learning Model for One Architecture Be Used for Others?
Retargeted-Architecture Binary Code Analysis. In comparison to that paper, I do not quite understand the novelty here, except perhaps for a slightly different evaluation/application domain. I certainly acknowledge that this submission was made slightly before the Usenix 2023 proceedings were made available, but I would still want to understand how this differs given the overall similarity in idea (building embeddings that help a model target a new ISA).
- addressed in rebuttal
- relatedly, only x86 and ARM appear to be considered in the evaluation (the authors discuss building datasets for these ISAs). There are other ISAs to consider (e.g., PPC), and validating the approach against other ISAs would be important if claiming to build models that generalize to across architectures.
- author rebuttal promises a followup evaluation
- vulnerability discovery methodology is also questionable. The authors consider a single vulnerability at a time, and while they acknowledge and address the data imbalance issue, I am not sure about the ecological validity of such a study. Previous work has considered multiple CVEs or CWEs at a time, and report whether or not the code contains any such vulnerability. Are the authors arguing that identifying one vulnerability at a time is an intended use case? In any case, the results are difficult to interpret (or are marginal improvements at best).