If you do a debug/packet capture on the EIGRP process you'll see that the messages are only sourced from the primary address. Which means you can only form neighborships with the primary address.
Please remember to rate useful posts, by clicking on the star below.
-Troy J.