The first scenario you describe is a GRE tunnel between routers which is sent through IPSec VPN on firewall devices. And obviously it works.
The second scenario is a GRE tunel which is sent through IPSec VPN and processed on the same device. This is very possible. I have configured this many times and it works quite well. Since you did not provide any details of what you configured it is difficult to make suggestions about what was wrong.
The principles of configuring IPSec VPN are the same on firewall or on router. But the details of how to configure them are quite different depending on the platform. The configuration of the GRE tunnel is pretty straightforward. The configuration of IPSec is a bit more complex. My suggestion is to focus first on the configuration of the GRE tunnel. Make sure that you have the tunnel configured and have traffic flowing through the tunnel successfully before you begin the IPSec part of the configuration.
The essentials of what you do to configure IPSec VPN on a router include:
- configure a transform set.
- configure ISAKMP key for the peer.
- configure a crypto map to identify the VPN peer, and the access list to identify traffic to be protected by the VPN.
- configure an access list to identify traffic to be protected by the VPN.
- assign the crypto map to the outbound interface.