". . . MSS is used in the transport layer to avoid fragmentation in the hosts that participate in the communication . . ."
Yes and no. Generally, when we say "fragmentation", we mean splitting an IP packet. From the sending host, MSS may be larger than what an IP packet can contain, but the host IP packets sent won't be fragmented. The TCP segment, though, would span multiple IP packets, and so you then have the risk of losing an IP packet will lose part of the TCP segment, which, if I remember correctly, will require the retransmission of the segment, i.e. multiple packets. For this reason, TCP generally sets MSS to not exceed what one IP packet can contain, so if a packet is lost, only that IP packet needs to be retransmitted.
(BTW, the impact of losing part of a transmission's lower stack unit isn't limited to just IP and TCP, similar issue with ATM and IP. I.e. lose an ATM cell, and the whole IP packet needs to be resent.)
That said, if you know a certain path supports a smaller MTU, you can set hosts to use a smaller MSS, to avoid the need to fragment IP packets downstream. (This is what the command "IP TCP mss-adjust" is used for.)
". . . PMTUD is used to discover the smallest MTU between two hosts that have established some TCP or UDP communication . . ."
It's not limited to just TCP or UDP, it's an IP packet parameter.
". . . but my question is, at what point is the PMTUD actually used? Are the machines enabled by default?"
It's generally off by default. All PMTUD does is set the DF bit in the IP packet. At any IP hop, when the IP packet cannot be forwarded without it needing fragmentation (at that hop), and DF bit is set, the IP packet is dropped and a message is sent to the sender saying fragmentation is required. (If DF bit is not set, the IP packet is fragmented and forwarded. Oh, and if the fragmentation needed message is blocked, sometimes done by firewalls, that causes the host to keep sending the too large packets, which all get dropped.)
BTW, since networks are dynamic, a path might change to a link whose MTU is smaller than what was okay earlier. Conversely, a path might change such that larger MTUs are supported than earlier. Once a host is notified that packets need fragmentation (NB: older messages only say fragmentation is needed, it's up to the sending host to try different sizes to find what's good. Newer messages specific the max MTU that's supported.), it generally starts a clock after after some time (generally multi minutes) it will try its max MTU again to determine whether a larger or max MTU might be used now. This cycle, of PMTUD, is repeated. As noted above, for TCP, if MSS has been adjusted, the need for PMTUD, for that TCP traffic, shouldn't be needed although, again, remember network paths can change. I.e. even with an adjusted MSS, to avoid a known small path MTU, PMTUD is often still advisable.)
To avoid the PMTUD fragmentation cycle for other than TCP traffic (as it's not limited by MSS), you need to modify the sending host's MTU down. (This may need to be done to avoid fragmentation of UDP video streams, for example.)
