Who are some well-known authors who would be considered "literary" if they only wrote in the Science Fiction/Fantasy genres instead of their usual genres?

An Unanswerable question. Sorry. Almost all “literary” authors have never tackled “Science Fiction” and Sci-fi is not considered a “literary” genre. Sure, some sci-fi authors of great worth, Vonnegut, Dick, Attwood, have crossed that boundary but normally Science fiction is abjured by the literary world.

Those rules apply, similarly, to works of fantasy, few fantasy writers have crossed that unknown boundary. Naomi Mitcheson, Karen Blixen and Mary Shelley have perhaps darkened its doorstep.

Also, similar rules apply to the “Horror” genre, despite some of the wondrous works by Blatty, King, Simmons and others, none are ever recognized as Literary. So to finally make an attempt at answering your question. To write “science fiction” is mainly to exclude yourself from Literary writing.

Science fiction is considered among the elite as a mongrel trait and unworthy of note.