When you ping the IP address of vlan interface, the ping packet need to be punted to CPU to be processed and then send the response back. Therefore, the delay might be caused by a busy CPU.
When you ping a host in a different vlan, the packet (passing through traffic) will be hardware switched and won't need be processed by CPU.
In one word, the packet go through the different data path within the switch, that's why you see this difference.