Art dealing and collection
From 1963, Brown operated as an art consultant from a small space at the Flinders Lane showroom of J. Brown Mantles, his fashion business. In 1967, after selling J. Brown Mantles, he opened the Joseph Brown Gallery at 5 Collins Street, Melbourne, with the encouragement of fellow artists Russell Drysdale and William Dobell, who participated in the gallery's inaugural two-man exhibition. That same year, he purchased Caroline House, a large nineteenth-century residence in South Yarra, to accommodate both his family and his growing art collection.
The gallery held solo exhibitions for artists including Sidney Nolan, Russell Drysdale, John Brack, and Rick Amor, and presented large-scale historical surveys accompanied by fully illustrated, documented catalogues that became sought-after collectors' items. His exhibitions featured both historical and contemporary Australian artists such as John Glover, Eugène von Guérard, Tom Roberts, Frederick McCubbin, Arthur Streeton, John Peter Russell, Rupert Bunny, Margaret Preston, Albert Tucker, and John Perceval.
Brown was instrumental in reclaiming the work of forgotten artists, including Danila Vassilieff, for whom he staged two major exhibitions at the Joseph Brown Gallery in 1973 and 1974. The gallery relocated from Collins Street to Caroline House in 1976 and held its last exhibition in 1982, though Brown continued to work as an art consultant.
During his travels to Paris and New York for his fashion business, Brown visited the Louvre, Metropolitan Museum of Art, and Guggenheim Museum, which sparked his serious interest in collecting. He came to recognise that greatly undervalued Australian colonial art was comparable in quality to American nineteenth-century art. In the 1950s, recognising that European masterpieces would be financially prohibitive, he began systematically collecting Australian art, aiming to create a survey collection spanning two centuries of European settlement in Australia.