Why Crypto-detectors Fail: A Systematic Evaluation of Cryptographic Misuse Detection Techniques
Amit Seal Ami, Nathan Cooper, Kaushal Kafle, Kevin Moran, Denys Poshyvanyk; Adwait Nadkarni
Table of Content
Overview
The correct use of security techniques, in particular, cryptography, is central to ensuring data security in modern software systems, whereas the incorrect use results in compromising security.
Several academic and commercial security analysis tools exist that help identify the incorrect use of cryptography related techniques.
Stakeholders, such as researchers, code-hosting services, app markets and developers rely on these security analysis tools to avoid introducing vulnerabilities through improper use of those techniques.
However, we know very little regarding their actual effectiveness at finding cryptography-specific techniques, and whether these techniques contain flaw(s). Unfortunately, beyond manually created benchmarks, there is no approach for systematically evaluating crypto-detectors. Instead, what we need is a systematic, data-driven, evolving approach that can be used reliably to evaluate the security techniques. Furthermore, we need to understand why flaws in cryptography-specific security techniques occur.
We address these issues in our IEEE Symposium on Security and Privacy (Oakland’22) (A*/Top-Tier) paper [1] [2] using a systematic, data-driven mutation framework (MASC) for evaluating detectors, that reveals significant flaws in popular tools.
1 Minute Teaser for the IEEE SP’22
Please turn on sound!
Talk at the IEEE SP’22
Key ideas of the paper are 👇🏽
- Crypto is a vast and complex domain, and developers can be expected to (mis)use Crypto-APIs in inordinate ways. We capture this complexity in a data-driven taxonomy of 105 unique misuse cases, obtained from over 20 years of discourse in both academia and industry.
- We expressively instantiate/mutate the cases from the taxonomy using (1) a set of generalizable mutation operators that express variations inspired by typical Java API-use conventions, and (2) scopes that place the mutant in software based on a “threat model” for Crypto-detectors.
- Our framework, MASC, implements these primitives in a manner that allows it to generate thousands of compilable, non-trivial mutants in seconds. Using MASC, we evaluated 9 major detectors from industry, academia and open source community, and found 19 designs and implementation flaws that result in non-detection!
- Are these flaws realistic/practical? We studied publicly available source code in both GitHub and Stack Overflow and confirmed that the misuse instances are reflected in real, popular, applications, and often in apps that have been previously analyzed by the detectors.
- To improve this status-quo, we must understand why these flaws occur, and reach a consensus in terms of what is expected from Crypto-detectors and how we will design and evaluate them to satisfy the expectations.
- This paper initiates this discourse within the security community by concluding with a discussion that integrates several views on the design decisions behind Crypto-detectors, informed by our results and conversations with tool designers (included with consent).
MASC - ESEC/FSE’23
With numerous improvements to facilitate evaluation of crypto-detectors, such as plugin-architecture, multiple user-friendly interfaces, and automated evaluation support, MASC’s next iteration became accepted to the Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’23) Demonstrations track!
Poster
Our poster on this same topic was also published at the Network and Distributed Systems Security Symposium (NDSS) 2022, a top-tier security venue!
Acknowledgements
This work is supported in part by NSF-1815336, NSF1815186 and NSF-1955853 grants. Additionally, the poster is supported in part by the Coastal Virginia Center for Cyber Innovation and the Commonwealth Cyber Initiative, an investment in the advancement of cyber R&D, innovation, and workforce development. For more information about COVA CCI and CCI, visit www.covacci.org and www.cyberinitiative.org.
Bibliography
BibTeX
@inproceedings{asr-masc-demo,
author = {Ami, Amit Seal and Ahmed, Syed Yusuf and Redoy, Radowan Mahmud and Cooper, Nathan and Kafle, Kaushal and Moran, Kevin and Poshyvanyk, Denys and Nadkarni, Adwait},
title = {{MASC: A Tool for Mutation-Based Evaluation of Static Crypto-API Misuse Detectors}},
booktitle = {Proceedings of the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE'23), Demonstration Track},
year = {2023},
doi = {10.1145/3611643.3613099},
address = {San Francisco},
month = dec,
month_numeric = {12}
}BibTeX
@inproceedings{ami-masc-oakland22,
author = {Ami, {Amit Seal} and Cooper, Nathan and Kafle, Kaushal and Moran, Kevin and Poshyvanyk, Denys and Nadkarni, Adwait},
booktitle = {2022 IEEE Symposium on Security and Privacy (S\&P)},
title = {{Why Crypto-detectors Fail: A Systematic Evaluation of Cryptographic Misuse Detection Techniques}},
year = {2022},
address = {San Francisco, CA, USA},
month = may,
pages = {397--414},
publisher = {IEEE Computer Society},
issn = {2375-1207},
url = {https://ieeexplore.ieee.org/document/9833582},
doi = {10.1109/SP46214.2022.9833582},
month_numeric = {5}
}