it's difficult to answer.
There will be an initial delay required to establish the ssl session, then for each packet there will be additional delay to encrypt/decrypt the packet.
It also depends on load.
An CSM can handle lot of traffic but not an SSLM.
So, if you see something wrong in a sniffer trace like packet drops, retransmits, huge delays per packet [like 300ms], then we need to investigate what is going on.
Otherwise, this would be normal.
Regards,
Gilles.