“With people of color, we’re often seen as dangerous even when we’re not,” says John Legend of discrimination he faced back in college

John Legend is one of the most celebrated stars around, but he’s faced his share of adversity.

PEOPLE’s Sexiest Man Alive opens up in this week’s cover story, sharing details of times where, as a black man in America, he’s experienced racial discrimination first hand.

“We faced a lot of challenges but I think my mother and father prepared me well for the challenges that I faced,” says Legend, 40, of his humble beginnings growing up with his mother Phyllis, a seamstress, and father Ronald, a factory worker, along with his three siblings in Springfield, Ohio.

John Legend with his family. COURTESY PATRICIA TYREE

Thinking back to college, Legend says, “I loved going to Penn, but occasionally you’ll see things that happen in social media right now and it reminds me of things that happened to me when I was in college. People treating you like you don’t belong there or asking for your ID when they wouldn’t ask for a white kid’s ID.”

He continues, “Asking me to prove that it was my car that I was getting into in West Philadelphia. Things like that, just occasional encounters with police and with people who questioned your belonging.”

John Legend Rails Against Mass Incarceration and Rikers Island: ‘It’s Radically Terrible’

They are experiences “I think a lot of black students deal with in college,” says Legend. “The scariest thing is when police are called and you never know how that can escalate. Particularly with people of color, we’re often seen as dangerous even when we’re not. When that suspicion is added to any circumstance, it can be dangerous for us.”

John Legend graduating college. COURTESY PATRICIA TYREE

Legend, an impassioned criminal justice reform activist, says that these days the biggest misconception about him is simply that he’s “super serious and kind of hyper-romantic all the time,” he says. “And that is not me. I let loose a lot more than people would expect.”

The one person who best gets him out of his shell would be wife of six years, Chrissy Teigen. Legend says that though they come from very different backgrounds, they’re better together and have common ground.

“I grew up in a black family in Ohio. Chrissy grew up in a half Thai, half Norwegian-American family all over the west coast,” says Legend of his model-turned-cookbook author wife. “One major difference is the kinds of foods.” But, he says, “I think we meet in the middle because I think black people and Thai people love spicier, flavorful kind of hearty foods. We connect on a lot of food things.”

Luna, John Legend, Miles and Chrissy Teigen. DOUG INGLISH

Other things they have in common: “We have a very similar taste in comedy, the way we want our home to look and what we like fashion-wise,” he says, to name a few things. And the two are excited to see how their respective cultures enrich the lives of kids Luna, 3½, and Miles, 18 months.

“It’s interesting because our kids see all their grandparents quite a bit. They have a white grandparent who’s 80-years-old, a Thai grandmother who’s a lot younger and plays with them all the time. They have my mom and my dad, and they get a little bit of different vibes from all of them,” says Legend.

He adds, “I think their lives are going to be interesting in that way because they’re going to get different cultures. A little bit of this, a little bit of that. I think it’ll be a cool thing for them growing up to get exposed to so many different things.”

John Legend. DOUG INGLISH

For more on the Sexiest Man Alive – and all the hottest guys in Hollywood – click here and pick up this week’s issue, on stands Friday, Nov. 15!