Sister Wives' Kody Brown Says He's 'Never Claimed to Be a Ladies' Man' Despite Previously Having 4 Wives

Kody Brown is setting the record straight about his skills with the ladies.

On this week's episode of Sister Wives airing on Sunday, May 11, Kody shared with the cameras that his family is focusing on "doing happy things" in the aftermath of his son Garrison's sudden death.

"We're moving through our grief because you need that time of mourning," Kody, 56, explained before tearing up. "We're doing mourning things to respect and love the dead, to mourn to own that pain. We're doing happy things. We're doing happy things so that we are moving on from mourning, so we can have faith in the hereafter and in our own futures and our own lives."

Even though the episode was still focused on how the family continued to navigate the tragedy, it also featured some lighter moments from the brood. Kody traveled with his wife, Robyn Brown, and their children to his former stomping ground in his native Wyoming. He not only showed them around town, which included showing them the various signs he crafted for local businesses early in his career, but he also got them to engage in local activities.

At one point, the family gathered at a local restaurant for a meal and engaged in light-hearted conversation. Kody shared with Robyn, 46, and his children the origins of his name, which was inspired by his father's admiration for Buffalo Bill, a.k.a. William Frederick Cody.

The PEOPLE Puzzler crossword is here! How quickly can you solve it? Play now!

"He was a scout for the US Army. And later on in his life, he became friends, or business partners with Sitting Bull, and they travelled with the Wild Wild West show," Kody explained of Buffalo Bill. "Basically, he was a showman. He was an entertainer."

To that, Robyn asked, "So, do you think that's like you?"

"I think I'm sort of entertaining," Kody said in response. "I try to entertain you!"

As Robyn noted how Buffalo Bill was "kind of crazy and wild, though," Kody said: "I think he was probably very charming, very entertaining, very friendly and maybe a bit of a ladies' man, I'm assuming."

"I don't think I'm a ladies' man at all," Kody added.

The PEOPLE App is now available in the Apple App Store! Download it now for the most binge-worthy celeb content, exclusive video clips, astrology updates and more!

The "ladies' man" comment led to laughter amongst the table, and Kody questioned why his remark was "so funny" to them. "I'm not saying I'm — I never claimed to be a ladies' man! I was a polygamist because of religion," the father of 18 then said. "OK? I was never a ladies' man."

Doubling down, Kody then told the cameras, "I had four wives, that's all. I wasn't a ladies' man, and having four wives didn't make me a ladies' man."

But Robyn disagreed with her longtime husband's take. "He did have four wives! I mean, like, most people would consider that a ladies' man, but whatever," she said.

Since 2010, the TLC series has shown how they've navigated life as a plural family amid Kody's marriages to Robyn, Meri Brown, Christine Brown and Janelle Brown. His divorces from Meri, 54, Christine, 53, and Janelle, 56, imploded within a 14-month period. He's now only married to Robyn, whom he legally wed in 2014 after their spiritual marriage occurred in 2010.

After going through three divorces, Kody declared that he would never endure polygamy — a practice that is a part of their religion, which is a sect of fundamentalist Mormonism — again.