I'm not about to start speaking for the signature team here at Cisco, but you can "sometimes" do something with encrypted data. Cisco IPS has had (technically still does) signatures that are based on traditional cryptographic traffic analysis. A packet capture is the place to start...preferrably under controlled conditions so that you can positively eliminate "other" traffic. In fact, a bunch of packet captures are usually better. Comparing all the captures, you have to look for patterns and trends. Things like predictable packet contents at certain offsets and packet exchange series of intermediate length, say 5 or 6 in each direction, for which there is some repeatable characteristic across all of the packet captures.
Lets say you find something, then what? Depending on what you find, you can write a series of Atomic IP or Multistring, or String TCP (to name the top 3) signatures and combine them using one or two layers of Meta. You might also be able to write a Service Generic signature (I wouldn't try this without Cisco's signature team's help). Ultimately, Cisco could hard code a signature into P2P (thats what that engine is for) that would directly process packets.
I realize this doesn't provide an answer to the topic's thread, but I thought I'd let you know that encryption doesn't automatically equate to impossible. It might, it might not...depends on how clever each side is ;-)
Scott Cothrell
Cisco IPS Dev.
