103 Church Street lies at the virtual center of the early city of Charleston, as determined by the walls of 1704. Pronounced "notable" in the landmark first edition of This is Charleston, 103 Church Street is especially significant in its representation of the early Charleston house type that combined commercial and residential functions. The three-and-one-half story, Flemish-bond brick building was constructed circa 1816-1820 by George Hoffman on a site purchased from Joel R. Poinsett, United States ambassador to Mexico.

African American entrepreneur, broker of wild game, and chef Nat Fuller owned 103 Church Street in the mid-1800s, and in the building Fuller opened the city's finest restaurant, The Bachelor's Retreat. In the spring of 1865, Mr. Fuller hosted an infamous dinner to commemorate the end of the Civil War. At this dinner black and white Charlestonians shared a table to celebrate the war's conclusion and reconcile differences. Due to the significance of the famous dinner, 103 Church Street was honored in 2015 as one of "The Porgy Houses", a Spoleto Festival USA special series honoring African American history in Charleston.

The property was purchased in 1996 by Mr. and Mrs. Robert M. Hicklin, Jr. and converted for use as an art gallery on the first floor with office and residential space occupying the upper floors. Restored to period perfection in 1997-1999, 103 Church Street was awarded the 1999 Carolopolis Award by the Preservation Society of Charleston in recognition of its outstanding exterior rehabilitation.

In the nineteenth century, the John William Hill View of Charleston in 1851 shows only the third story and roof of 103 Church Street from the south. The Plan/View of Charleston drawn by Camille Drie in 1872 depicts 103 Church as a three-story building with a two-story addition on the north side. The earliest photograph to detail the property, captioned "View from St. Michael's Steeple, Looking East" in Arthur Mazyck's volume titled Charleston, South Carolina in 1883, confirms the accuracy of the Drie rendering and reveals the rear piazza as only two stories high.

The 1997-1999 exterior rehabilitation work program included the restoration of the central doorway and sidelights on the front façade, infilling three later first-floor windows along St. Michael's Alley with brick excavated from the basement. This renovation also included restoring the rear piazzas by replacing the twentieth century rails and balusters with those of appropriate design for the late Federal period; removing approximately seventy pipes, wires, electrical lines, and cables from the building's exterior; stripping the brick of modern paint; and re-pointing the masonry using a mortar recipe appropriate for historic brick. At the rear of the property, a brick storage "privy" was constructed using a design and materials comparable with other structures in the neighborhood.

The Adamesque woodwork on the second floor is original to the building and to the period of construction. It was stripped of numerous layers of paint and returned to its historic prominence. The ground level was taken back to construction timbers and built out using the second floor above it as reference, but with careful attention paid to the demands of a commercial enterprise.