What should be the behaviour of TcpIp stack if a TCP segment (SYN) with MSS 0 is received?
1. Connection request be dropped?
2. Connection request accepted with a default value of MSS as 536 (for IPv4)?
3. Accept the connection with MSS = 0, do only one way communication
It depends on the tcp stack implementation on the server.
If you look at the TCP standard (https://tools.ietf.org/html/rfc793), the MSS field is optional. If not present, any segment is allowed.
If present and set to 0 it is probably an invalid packet. The RFC879 (https://tools.ietf.org/html/rfc879) specifies,
The rule relating the maximum IP datagram size and the maximum TCP segment size is: TCP Maximum Segment Size = IP Maximum Datagram Size - 40
In this case the server can either drop the connection request or set the MSS according to the previous rule. Seems it depends on server implementation as not specified on RFC.
On the safe side you could force setting MSS on the gateway router with interface command:
ip tcp adjust-mss
I agree with o.melendres that the indicated MSS of 0 is an invalid value because it effectively prevents the other party from sending any data to us whatsoever. Different TCP implementations in various operating systems may behave differently.
However, I have to point out that the ip tcp adjust-mss can not be used to raise the MSS. The value specified in ip tcp adjust-mss is the top-allowed MSS value, and if either side of a TCP session advertises a higher MSS, it will be clamped down to the specified value. However, if any TCP party advertises an MSS that is smaller, the ip tcp adjust-mss will not affect that particular value.