Letterman began by poking fun at his retirement.
"It's so good to see famous people again. Just to be out of the house!" Letterman said. "For a year, I have been looking, high and low, I am determined to find a shirt that looks good untucked. I can't find one!"
The former host of Late Show with David Letterman then told the story of how giving Conan O'Brien a horse as a gag gift backfired.
Kimmel asked Letterman if he misses his life in late-night.
"Some parts of it," he said. "I miss wearing make-up."
The appearance, which was to promote Letterman's new Netflix show out next year, was his first interview during a late-night show since his retirement in 2015, according to ABC News.
Jimmy Kimmel Live is in Brooklyn this week. Kimmel spoke during his monologue about what an honor it was to have Letterman on the show.
"David Letterman is to me what Beyonce is to everyone else," he said.
Letterman is being honored with the Mark Twain Prize for American Humor on Sunday.