Layer 1 Switching - How to configure server network interfaces

[Reha Diri]

Hello ,

I didn't receive any comments in the "Other Networking" subtopic so i moved the discussion in "Switching" subtopic. I hope there can be a specialist to answer the below issue;

We have the layer1 setup shown in the picture.  We have Cisco 3550-F Layer 1 switch for the switch and EXANIC Exablaze as the server interfaces.

If we use only one interface in the server 1 and server 2,  and make these two interface ip and mac addresses exactly the same it looks that the setup is working. Ip packet returning from the Financial Organization  copied to all the downstream fastmux ports and related server takes care. 

The other party organization forwards all the packets for our subnet by using a single L3 ip address  (10.164.148.126/30). Using a second or third interface in any server seems that this won't work because we can't set the ( 10.164.148.126/30 ) again. Is there a solution for using all the other Exanic interfaces in this setup.

Thanks

Regards

[pieterh]

this is new technology for me. 
but I understand from your post that the 3550F is a multiplexer and I understand you use it as one-to-many distribution
-> you can use a second incoming interface 
    configure this also as one-to-many to distribute to the second Exanic port
   you need a different secondary IP (with corresponding routes) for this interface

[Reha Diri]

Incoming traffic is not a problem , layer 1 switch distributes the downstream packets to fastmux ports . The problem is on upstream traffic.

For each Trader , Financial Trading Center expects a specific /24 subnet ( as source ip  ) through a single /30 subnet ( communicating interface )   . /30 subnet is defined per Trader and can not be modified. So we should put the source ip as requested and use a single ip and mac address for neighbor interface. Otherwise Financial Trading Center neighbor switch arp infrastructure will get confused . 

If the OS Network infrastructure of the servers let us using same ip addresses in a single server, it can be a solution . But unfortunately we can't do that. 

[pieterh]

this is a curious setup !
so basically you are missing a router on your end to terminate the /30 interconnect network and route to the /24 local network
and you are trying to solve this by giving multiple  hosts the IP-address of the missing router ?

of course the router will add to latency and that is what you want to avoid

I'll do more research on this subject before i post any suggestions

[Reha Diri]

Yes, this is exactly what we are trying to do Pieterh,

Actually i realized that i didnt express the issue well . Some information that i gave here may lead some misunderstandings. So i redrawed the picture below. 

We don't need to set primary IP's of the servers the same , primary IP's can be in the subnet range. We only need the set the /30 IP address ( 10.164.148.126 ) same . That's why we created the second ip interface and set this same IP address.

If we dont set any secondary IP for the other interfaces of Server 2, these other interfaces will be able to communicate through the first interface. This is something that we dont want.  As you may notice this time first interface will be a bottleneck.

 

[pieterh]

>>>  these other interfaces will be able to communicate through the first interface. This is something that we dont want <<<

this will allways happen!
you set a static route to the remote gateway,
OS'es will use only a single interface for the destination of a static route, only when this interface has issues reaching this destination, it will switch to the other interface

what you need to look for is interface-bonding/etherchannel/LACP/LAG

this manual  reports : The exasock driver has been extended to support bonding, but only in the active-backup mode.
Cisco Nexus SmartNIC Adapters User Guide - Cisco Nexus SmartNIC Bonding Extensions (exasock-bonding) [Cisco Nexus SmartNIC] - Cisco
=>  this will not improve throughput

[Reha Diri]

Yes, bonding can be solution but as far as i know when you do bonding in Linux another type of network latency arises.

There is allways a tradeoff

We shouldn't exceed 1usec when sending a single ethernet frame.