It's kind of sensitive if you swap "ground" and "signal". Basically headset port has pinout "Spkr-MicGnd-Mic-SpkrGnd". If you do other direction, speaker will work, but microphone won't.
You can also try cell phone headset adapter for Cisco IP phone.
http://www.ciscoheadsetadapter.com