Saturday Night Live, a bad rental house, a mean dog and East Los Angeles: That’s the simplest way to start the story of Shaun White’s relationship with rocker Sarah Barthel.

The Olympic snowboarder and two-time gold medalist, who is in Pyeongchang, South Korea, for his fourth Winter Games, sat down with PEOPLE this week ahead of his competition. During the chat, he opened up about his connection with Barthel, 34, who is one half of the electro-rock band Phantogram.

“It’s funny: We actually met backstage at SNL,” says White, 31. He was in New York City several years ago apartment-hunting and Barthel, a native of upstate New York, was living in Brooklyn.

White says the two had mutual friends on the sketch series and “and we just bumped backstage and became friends, and then over time it kind of turned into something.”

Later, as White was training before the 2014 Winter Olympics in Russia, their relationship deepened thanks to a “really terrible” home Barthel and her bandmate, Josh Carter, had rented “way out in East L.A.” while they were recording.

Keep Following PEOPLE’s Complete Coverage of the 2018 Winter Olympics

The house was listed as having a dog, and Barthel “loves dogs” — but this one turned out to be “like that old angry dog, the don’t-touch-me dog,” White says.

“They didn’t really know the landscape and I was like, ‘I got a house, there’s a house, you got a car, just take it, I’m training for the Olympics, I won’t be home,’ ” White says. “So her and her bandmate and some others came and stayed at my place and that’s kind of how things got going.”

Speaking to Bleacher Report last month, Barthel recalled White at the time as “this wild child, just all over the place — in a good way.”

About five years later, and though they’ve been through the “normal relationship things over the years,” they’re in a good place, White says.

The two share some key interests (like Barthel, White is in a band and like White, Barthel snowboards; the two co-parent dog Leroy) but the differences between them are also valuable.

“We live in two different worlds, so that’s why it’s nice to come together and share stories from the road,” White says. They aren’t cozying up and analyzing his tricks.

“When we’re both home, it’s just like we’re home,” he says.

RELATED VIDEO: Kids Interview the 2018 Winter Olympic Athletes

“You get somebody that understands what it’s like to travel and to have to be on the road. And I thought I traveled a lot — man, musicians, it’s rough,” White says. “And I didn’t really get it until l was in a band myself and we’re on the road and it’s like it’s Groundhog Day.”

“So it gave me a real nice look at the other side of life and how great I got it,” he continues. “I’m like, ‘Yeah I was kicking it in Tokyo for like the whole week and then we decided to go here and ride,’ but it’s nice to not have somebody like, ‘When are you coming home?’ They get it, they understand being on the road.”

Barthel, who lives in the L.A. area with White, will join him in Korea before he competes on the halfpipe next week. Should he advance to the finals in that event, he’ll be snowboarding for gold on Valentine’s Day.

Asked about a possible wedding in the future, White tells PEOPLE they “haven’t gotten that far.”

“It’s tough,” he says. “It’s like you put your life on hold for the Olympics. … This is the goal at hand and we’re here, finally, it’s all happening.”

Still, White is “very excited to actually get back to living that aspect” — the domestic side — “of my life.”

“I’m the guy that will like, I’ll just be stoked to help you trim the bushes,” he says. “I’ll do yard work, I like to paint the houses, any sort of physical work, I’m down, I’m your guy. It’s ridiculous.”

And he means it. He tells this story with a laugh:

“I was at mom’s house, I was like, ‘Yeah, sure, what do you need?’ And then the neighbors — it was a beautiful East Coast day in the summertime — and you saw all the other neighbors kind of like eyeballing. They came out, they started doing their yards like, ‘You’re not going to have the nicest yard on the block.’ And then I’m out there all — it got very competitive.”

No surprise for an Olympian.

“I find competition in life when it’s not there,” White says. “I just look for it.”