How Hackers Exploit Protocols

When two layers on the same node communicate, or when peer layers
communicate, they rely on a well-defined protocol and precisely stated
message formats. The protocol states who initiates the connection, how
the session is terminated, the order of messages, what to do if an
error occurs, and other characteristics of the session that are
necessary for the exchange to succeed. A protocol also can be thought
of as an algorithm because most protocols are defined as state
machines.

As with other algorithms, order is important. If you want to bake a
cake, getting the right ingredients is just one part of the process.
Mixing and cooking the ingredients in the right order are required to
achieve an edible result. The same is true for network communications
protocols. If one half of the session decides to get creative with the
protocol, the results will not be guaranteed.

Getting the order of messages right is important, but you also need to
format the messages properly. A baking recipe is useless if the order
of the steps is clear, but the steps are not accurately defined. For
example, if the recipe is incorrectly copied from a friend and asks
for one unit of butter instead of one unit of flour, you will end up
with a different dessert. Similarly, in network message exchanges, if
message integrity cannot be guaranteed, any dialogue between the peers
will not succeed.

Designing a secure distributed protocol is more of an art than a
science. When someone at your site invents a new distributed security
protocol, alarm bells should sound. Unless this person is
knowledgeable and has studied a number of references, a better-than-
average chance exists for the protocol to have weaknesses.