The name Clermont derives from the French "clair montagne" for "clear mountain" inspired by the view of the Catskill Mountains across the Hudson River from the estate.

The estate was established by Robert Livingston following the death of his father, the first Lord of Livingston Manor, in 1728; while most of the manor was inherited by the eldest son Philip Livingston, 13,000 acres (5,300 ha) in the southwest corner, later named Clermont, was willed to Robert.

In October 1777, British Major General John Vaughan, who led a raiding party up the Hudson River, came to Clermont and burned Livingston's home because of his prominent role in the American Revolution. Margaret Beekman Livingston rebuilt the family home between 1779 and 1782. Her son Robert R. Livingston became the estate's most prominent resident, serving as a member of the committee that drafted the Declaration of Independence, Secretary of Foreign Affairs, and negotiator of the Louisiana Purchase. He was also a partner with Robert Fulton, creating the first commercially successful steamboat, the North River of Clermont, commonly called the Clermont, which stopped at the house on its inaugural trip.