From your description of the symptoms I would guess that the layer 2 switch has both ASA ports in the same VLAN (which leads me to wonder if the layer 2 switch is doing any VLANs or are all ports in the native VLAN?). Can you provide details of how the layer 2 switch is configured?
It also would help to know how the ISP is set up. You tell us that it is an Ethernet handoff. But you do not tell us whether the ISP is handling that as a trunk with multiple VLANs or whether the ISP is treating it as an access port with a single VLAN. Can you clarify this?
Knowing how the ISP is set up and how the layer 2 switch is configured will be critical in finding a solution that works for your ASA.
HTH
Rick
HTH
Rick