This is a VERY big topic and I suggest you read large documents on the internet. However since you asked in simple terms :
Everyone should have 2 keys a public and private. Everyone keep his private key and NEVER give it to someone else. The public you actually give it to everyone.
To send you messages, everyone uses your public key to encrypt and you use your private key to decode. To reply back you need to use the public key of the other person to encrypt and he will use his private to decode.
A comparison (while a bit strange) would be that the public key is your phone number, you give it to everyone and people use it to call you. However that doesn't mean that someone can "steal your calls".
Same thing for bank account, you give your number to everyone but they can just send you money, not steal from you.
The private key is like your SIM card in your phone that is the only one having your phone number, so you can receive calls targeted to your number.
This system ensures that the communication is secure, why ?
-People contacting you have a proof that it's you. They send encrypted with your public key and you are able to decode, so you have to be the only one on earth having the private decoding key : you.
-You know that anyone could have written the message but for sure it's targeted to you since it's with your public key.
-To verify the other person identity you simply reply back using his public key. If he's really the guy he will be able to understand what you say since he has his private key to decode.
