as far as I can tell, the solid lines represent the shortest paths to the neighbors. The dotted lines represent possible paths, but not shortest paths.
So, as an example, the shortest path from x to G is x --> D --> G, while x --> A --> D --> G is a possible path, but not the shortest.
You can have multiple path to reach a destination but the OSPF router will select the path with the lowest total cost to the destination because it represents the best bandwidth through the path. On the diagram the dotted lines are the possible paths to the destination.
Regarding your original question, as already noted by Georg, it appears the dotted lines represent connections that have been found not to be on the shortest path.
BTW, the second diagram is lacking link costs, so we cannot confirm what are the actual shortest paths actually are, based on the sum of link costs.
Regarding your path X->C->D->G and X->C->E->G question, that costs appear to be equal, again, the diagram doesn't show costs. Hops are equal for those two paths, but that alone doesn't really tell us what OSPF would consider the shortest path. So the diagram is a poor example. From the diagram, we can only assume the sum of the path costs for X->C->D->G and X->C->E->G is higher than for X->D->G. The latter has a lower number of hops, but it could have a higher sum of links costs that could make X->C->D->G and/or X->C->E->G the shorter OSPF path. As also noted by Julio, depending on the sum of the path's link costs, muliple paths could be considered equal.
Hi OSPF uses the shortest path first term based on the Dijkstra algorithm. OSPF's metric is the cost, so where I can find this cost? each interface on a router that is part of the OSPF process will be assigned with a cost (it could be seen on the link below) based on their interface bandwidth. Once received the routes through OSPF neighbors the router will select the best path with the lowest total cost.
The cost can be changed to manipulate the traffic but by default the interface have a cost based on Interface bandwidth.
Also it is the formula used by OSPF to calculate the cost:
Cost = Reference bandwidth / Interface bandwidth in bps.
The reference bandwidth by default is 100Mb but it can be changed using auto-cost reference bandwidth <new value> under the ospf process. And in order to see the cost you can execute this comman line: show ip ospf interface
Hope it is useful