What did his friend mean by this?

He's 20 years old and boys will talk just like women will. Of course they know he's not having sex with you. Be proud of him for being able to say that to his buddies and not lying about it. He obviously cares about you and respects you. Don't start a fight over something stupid. He was probably commenting more on their lack of discretion on whom they sleep with.