These five novels originally tied for first place, with each winning the support of 9 of the editorial board's 10 members. Scott Fitzgerald's magical tale of romance, mystery and violence among rich Long Island socialites in the 1920's another work by Joyce, ''A Portrait of the Artist as a Young Man,'' his autobiographical account of a young man's intellectual awakening'' ''Lolita,'' Vladimir Nabokov's tale of the aging Humbert Humbert's doomed passion for the nymphet Dolores Haze, and ''Brave New World,'' Aldous Huxley's satirical horror tale of a civilization where humans are literally made to order. ''Ulysses'' is followed in descending order by ''The Great Gatsby,'' F. Woolsey, who called the book ''a sincere and serious attempt to devise a new literary method for the observation and description of mankind.''

''Ulysses'' was banned in the United States as obscene from 1920 to 1933, when the ban was lifted by a Federal judge, John M. Byatt, Edmund Morris, John Richardson, Arthur M. Boorstin, Shelby Foote, Vartan Gregorian, A. The board members are Christopher Cerf, Gore Vidal, Daniel J. The list is to be released on Friday at a workshop for young publishers known as the Radcliffe Publishing Course at Radcliffe College of Harvard University. The book - in which an immensely long account of a single day in the lives of a group of Dubliners becomes a metaphor for the human condition and the author experiments with language almost to the point of unintelligibility - heads the list of 100 novels drawn up by the editorial board of Modern Library, which has been publishing classic English-language literature at affordable prices since 1917 and is now a division of Random House. ''Ulysses,'' that sprawling, difficult, but uniquely original masterpiece by James Joyce, has been voted the finest English-language novel published this century by a jury of scholars and writers.

Ulysses novel