Why Crypto-detectors Fail: A Systematic Evaluation of Cryptographic Misuse Detection Techniques

Written on May 4, 2022

How well do tools that detect Cryptographic misuse (i.e., Crypto-detectors) work in practice?

We answer this question in our IEEE Symposium on Security and Privacy (A*/top-tier) 2022 paper using a systematic, data-driven mutation framework for evaluating detectors, that reveals significant flaws in popular tools.

Key ideas are 👇🏽

  • Crypto is a vast and complex domain, and developers can be expected to (mis)use Crypto-APIs in inordinate ways. We capture this complexity in a data-driven taxonomy of 105 unique misuse cases, obtained from over 20 years of discourse in both academia and industry.
  • We expressively instantiate/mutate the cases from the taxonomy using (1) a set of generalizable mutation operators that express variations inspired by typical Java API-use conventions, and (2) scopes that place the mutant in software based on a “threat model” for Crypto-detectors.
  • Our framework, MASC, implements these primitives in a manner that allows it to generate thousands of compilable, non-trivial mutants in seconds. Using MASC, we evaluated 9 major detectors from industry, academia and open source community, and found 19 designs and implementation flaws that result in non-detection!
  • Are these flaws realistic/practical? We studied publicly available source code in both GitHub and Stack Overflow and confirmed that the misuse instances are reflected in real, popular, applications, and often in apps that have been previously analyzed by the detectors.
  • To improve this status-quo, we must understand why these flaws occur, and reach a consensus in terms of what is expected from Crypto-detectors and how we will design and evaluate them to satisfy the expectations.
  • This paper initiates this discourse within the security community by concluding with a discussion that integrates several views on the design decisions behind Crypto-detectors, informed by our results and conversations with tool designers (included with consent).

Our poster on this same topic was also published at the Network and Distributed Systems Security Symposium (NDSS) 2022, a top-tier security venue!

  author = {Ami, {Amit Seal} and Cooper, Nathan and Kafle, Kaushal and Moran, Kevin and Poshyvanyk, Denys and Nadkarni, Adwait},
  booktitle = {2022 IEEE Symposium on Security and Privacy (S&P)},
  title = {\{Why Crypto-detectors Fail: A Systematic Evaluation of Cryptographic Misuse Detection Techniques\}},
  year = {2022},
  address = {San Francisco, CA, USA},
  month = may,
  pages = {397--414},
  publisher = {IEEE Computer Society},
  issn = {2375-1207},
  pdf = {https://arxiv.org/pdf/2107.07065.pdf},
  sourcecode = {https://github.com/Secure-Platforms-Lab-W-M/masc-artifact},
  url = {https://doi.ieeecomputersociety.org/10.1109/SP46214.2022.00024}

[articlePDF] [sourceArtifact]

First, click on "Comments" below to view/post comments.
To comment as guest, click on the field "Name". The option to do so will become visible.
লগইন ছাড়াই কমেন্ট করতে নাম এ ক্লিক করুন, দেখবেন তার নিচেই আছে অতিথি হিসাবে কমেন্ট করার অপশন।