Latency would be the thing that affects round-trip-time (or round trip delay).
Latency occurs in both directions, and can be different in each direction, depending on the media and path involved.
Another factor of round-trip delay is the thing at the other end: reception is usually a high-priority event, processing of the received information may have a variety of priorities, depending on the port and path through the stack.
Responding to the received info could also have a variety of priorities.
The bottom line is that you might have the fastest possible path bi-directionally, but the thing you are accessing / pinging might be bogged down or consider the traffic low priority and not respond as fast as you hope.
Latency is caused by a lot of things in the path, but it can also be a processing delay at the thing you are accessing.
Ping sends an "echo request," the thing being pinged responds with an "echo reply" both are ICMP types.
There is also "traceroute" (Microsoft uses "tracert") that sends a series of packets with an incrementing Time-To-Live (TTL) starting at "1."
When the TTL expires, an ICMP notification is sent back to the originator, and the originator keep track of how long it takes for the ICMP notification to return.
Most of this is covered, along with other basics in the Cisco Internetworking book. The physical book is very large and expensive.
To summarise what Scott has said, the different types of delays involved in the path of a apkt are:
Delays introduced by end devices (routers) like packetisation, serialisation, processing, queuing delays etc.
Delay due to link problems viz crc etc
Delay due to network issues viz introduced by devices in ISP for FR networks.
Now coming to your point, if RTD is 130ms then 1way is ideally 65ms.
Assuming that your ISP & link doesnt introduce any delay, then one of the router may have burdened cpu or long queue. Considering these factors, 1way delay practically is somewhat very near to the ideal value mentioned by you.