Irishmen did the bulk of the digging, not slaves

"His major work for the commissioners was digging a canal or at least 27,000 cubic yards of it that widened and deepened James Creek, a small stream south of the Capitol. That was enough digging to make a hole largest enough to swallow the entire White House. He only used 15 hands. The commissioners and the President, whose pet idea it was, lost interest in the canal when they realized how much it would cost. In the summer of 1793, the commissioners paid Whelan at least $860 for digging the foundation for the Capitol."

Quote from Slave Labor in the Capital, pages 60 & 61

The document below is a photocopy of a page of the commissioners' proceedings in September 1792 that discusses digging a canal to connect St. James Creek south of the Capitol with Tiber or Goose Creek that flowed west of the Capitol. The commissioners began using slave labor that summer, but, as the document shows they made a contract with Patrick Whalen or Whelan.

There is rarely any list of the workers used by a contractor but Irish laborers were commonly used to make the canals around the falls of the Potomac. There is evidence that Whalen himself owned a slave named Bill Scotland, and certainly many Irish workers bought slaves once they had the money to do so. However, I think it best to assume that Whalen used his countrymen in 1792. There is no evidence that he ever hired out slaves to the commissioners which other contractors who had slaves did do.