"When writing about the first stone used in the White House and Capitol, some historians assume that slaves did all the quarrying."
Quote from Slave Labor in the Capital, page 65
I copied two payrolls in the National Archives that list some of the quarry workers in June and July 1794. Eleven of the 12 men listed are free workers. Note that the payroll says it is a receipt for wages paid to "each of us for ourselves respectively," which is to say the money was not paid to masters. One man was evidently an indentured servant. The payroll lists "Luke Huer's Donald Hadmary." Searching genealogy websites I tried to see if any of the men listed were free blacks. The ones I saw listed in any sites appeared to be free whites. Of course, much work needs to be done to learn more about these workers.