Hello Adil,
the preferred solution would be to add L2 LAN switches on each site.
If this is not possible you should deploy /30 IP subnets one for each server/ASR 1004 GE interface and you can configure each port as a router port.
Example:
192.168.124.0/24
192.168.124.0/30 associated to gi0/0/1 192.168.124.1 on router port 192.168.124.2 server side
192.168.124.4/30 associated to gi0/0/2 192.168.124.5 on router port 192.168.124.6 server side
...
and so on
Edit:
according to feature navigator IRB (integrated routing and bridging 9 is supported on ASR 9000 but not on ASR 1000 This could be a cool feature for your application.
http://www.cisco.com/go/fn
search by feature type IRB select filter
Hope to help
Giuseppe