Known/chosen key attacks against software instruction set randomization

Instruction set randomization (ISR) has been proposed as a form of defense against binary code injection into an executing program. One proof-of-concept implementation is randomized instruction set emulator (RISE), based on the open-source Valgrind IA-32 to IA-32 binary translator. Although RISE is effective against attacks that are not RISE-aware, it is vulnerable to pure data and hybrid data-code attacks that target its data, as well to some classes of brute-force guessing. In order to enable the design of a production version, we describe implementation-specific and generic vulnerabilities that can be used to overcome RISE in its current form. We present and discuss attacks and solutions in three categories: known-key attacks that rely on the key being leaked and then used to pre-scramble the attacking code; chosen-key attacks that use implementation weaknesses to allow the attacker to define its own key, or otherwise affect key generation; and key-guessing ("brute-force") attacks, about which we explore the design of mini-malistic loaders which can be used to minimize the number of mask bytes required for a successful key-guessing attack. All the described attacks were tested in real-world scenarios