An SVI (Switched Virtual Interface) is a L3 interface for a vlan on a L3 switch. iIt is a virtual interface because it is not a physical interface. Lets say you create 2 vlans on a L3 switch -
vlan 10 = 192.168.5.0/24
vlan 11 = 192.168.6.0/24
you allocate ports on the switch into those vlans and attach PCs to ports. PCs in vlan 10 get 192.168.5.x addresses and PCs in vlan 11 get 192.168.6.x addresses.
As it is PCs in vlan 10 will be able to talk to other PCs in vlan 10 and PCs in vlan 11 will be able to talk to PCs in vlan 11. But they will not be able to talk to each other. Remember a vlan is only a L2 broadcast domain.
If you want the 2 vlans to be able to talk to each other you need to be able to route between the vlans and that is where SVIs come in. So on the switch you would do -
int vlan 10 <--- this is the SVI
ip address 192.168.5.1 255.255.255.0
int vlan 11
ip address 192.168.6.1 255.255.255.0
A PC in vlan 10 would have it's default gateway set to 192.168.5.1 and a PC in vlan 11 would have it's default gateway set to 192.168.6.1. Then the PCs can talk to each in vlan 10 and vlan 11.
In addition to Jon's awesome reply, simply think of the switch as also being a host (i.e. a computer having its own CPU). Just like any PC connected to the switch can be assigned to a VLAN, also the switch as a host can be assigned into a VLAN. This allows the switch to have an appropriate IP address in that VLAN and to be managed remotely via SSH, Telnet, SNMP, HTTP - you name it.
You can assign the internal CPU of the switch into a particular VLAN by creating an interface Vlan for the particular VLAN, assigning an IP address and activating the SVI. This will make the switch - as a host - to become connected to that VLAN and to have the IP address as assigned on the SVI.
Just like a normal host, it is not common for a usual PC to have multiple network cards. Same with a Layer2 switch, it is not common to have multiple SVIs. For remote management purposes, it is completely sufficient to have a single VLAN for which you have created the SVI and assigned an address. This VLAN is also called a management VLAN. In other words, you do not want to create an SVI for each defined VLAN just because they exist - you do not want nor need to have multiple management VLANs.
There is, however, a very significant exemption from this rule. Imagine that a PC had multiple network cards because it was capable of routing IP packets, in other words, because it was operating as a router. If you have a Layer3 switch that is also capable of routing IP packets, you want to connect its CPU into multiple VLANs because you want to allow it to route packets between VLANs. That is the reason you can see switches that have multiple SVIs configured and active - because they provide inter-VLAN routing function to the VLANs they create. In that case, you would configure multiple SVIs, assign unique IP addresses to each of them - and exactly as Jon has shown, you would configure PCs in individual VLANs to use the SVI addresses as their own gateway addresses.
To be very picky, true multilayer switches do not really route packets through CPU because that would be very inefficient. Instead, they have specialized circuitry that is capable of routing IP packets without ever bothering the CPU, but the main idea to visualize the concept remains - there is a route processor inside a multilayer switch which has to be connected to all relevant VLANs to provide routing between them, and its interfaces to these VLANs are the respective SVIs.
So finally i just want to repeat one thing..
So, It is not neccessary to assign an IP address for each and every VLAN... Since it is only used for remote management.
The answer is that it depends. On a L2 switch you would need only one SVI and that would be for managing the switch. You would give it an IP and just like an end host, it would need a default gateway too. This is what Peter was explaining. The IP address on this SVI is never used as default gateway for end devices such as PCs, it is only so you can connect to the switch itself. The default gateway for the switch would be a L3 device (usually a L3 switch) with an SVI in the same vlan. That L3 switch can then route packets between vlans.
On a L3 switch you probably would assign an IP to each vlan interface because usually you want to route between those vlans and to do that you need to have IPs on the vlan interfaces and use that IP as the default gateway for clients as i explained in my first reply.
Note also, i keep saying route to distinguish it from L2 switching but as Peter has pointed out L3 switches actually route the packets in hardware and we generally call this L3 switching.
Just wanted to add some points to the information Jon and Peter have already provided.
If you're just setting up a SVI for management, Peter noted you normally define a default-gateway. This is true when the L3 switch is running in L2 mode, when L3 mode is enabled, you generally use a static route to point to the default gateway instead (unless the device is operating as a router with a dynamic routing protocol.)
Jon mentions defining multiple SVI interfaces may allow routing between them on a L3 switch. This is true, but generally the L3 switch needs its routing mode enabled, otherwise it will function as a L2 switch and won't route between them. (BTW, not all Cisco L3 switches start in L3 mode.)
Also, SVIs provide an (IP'ed) interface to the device. When a L3 switch is operating in L2 mode, the SVI basically allows the switch to "appear" on the network as a host for management purposes. When a L3 switch is operating in L3 mode, additionally the SVIs allow transit traffic to/from the SVI interface. You can still use any SVI interface for management, but often when a dynamic routing protocol has been enabled, you define a loopback interface, make it known to the dynamic routing topology, and use if for management. The two major advantages of doing the latter, you can manage the device with a single IP if there's any good routable interface that are active, and it makes device protection a bit "clearer" as traffic to a SVI should only then be transit traffic, not directed to the device itself.
BTW, L3 switches also generally support "routed" or non-switchport interfaces too.