I'm not sure which part you think is impossible, but it's absolutely possible to chisel a stone out of a quarry with stone tools. There were subsequent steps to work that rough shape down into the final product, of which abrasion was one of the last steps. None of them are especially sensitive to the size of the stone. We've found the rocks they used to do it littering known quarries, on obelisks and the like.