The negative impacts are 1. your PIX now has to proccess access lists against non existent machines, and 2. if a machine gets added later with that IP address, and you forget to remove the conduit, you have a potential security hole.
I would remove any access list that is not being used any more. And speaking of access lists, you should convert the conduits to access lists. Cisco is moving away from conduits. You can go to the following link, select PIX from the drop down list, and post your config for suggestions not only on converting conduits to access lists (this is how I did it, and it worked flawlessly), but also other potential errors or security risks.
https://www.cisco.com/cgi-bin/Support/OutputInterpreter/home.pl
