A Semantic Approach Towards Automatic Protocol Reverse Engineering

Abstract

PhD Defense:

Reverse engineering message formats from static network traces is a difficult and time-consuming security task. This work is critical for a variety of purposes, including bug-finding via fuzz testing, automatic exploit generation, understanding the communications of adversary systems, and recovering specifications that are proprietary or have been lost.

Unlike protocols which use human-readable data such as ASCII, binary protocols are difficult for human experts to reverse engineer, as binary data itself is ambiguous without a specification describing the layout and semantics.

In this talk I introduce the insight that binary protocols tend to re-use common data-types, serialization patterns, and organizational structures such as tagged unions. These commonalities are due to the need for binary messages to be quickly and unambiguously deserialized using as few computing resources as possible, and the structure of the deserializing programs themselves. As a result, by tailoring automatic reverse engineering methods to these common features, underlying message formats and semantics can be uncovered automatically.

First I introduce the security motivations behind protocol reverse engineering, and the intuition behind this line of research. Next , I present two complementary tools for automatic protocol reverse engineering: BinaryInferno and BinaryRoulette. BinaryInferno uses a novel ensemble approach to identify individual fields in binary messages, while BinaryRoulette uses a novel information-theoretic approach to group messages by format. I then discuss the evaluation of BinaryInferno and BinaryRoulette on real-world binary protocols from a variety of contexts. Finally, I conclude with an examination of future directions for automatic protocol reverse engineering.

(Please email jared.chandler@tufts.edu if you'd like the Zoom link)