The advice here is great, let it go.
BUt, if you do want to pursue it, know that the contractor is BS'ing you. I'd egg him on to take me to court. That way, you have him admitting that there is a verbal contract. If both people agree, then it exists. When he's under oath, in front of a judge, it's going to be a lot harder for him to lie.
Then present all your evidence and the fact that he hired other painters to finish your work. Let it be known that other trades kept you from finishing etc, etc.
Take pictures, anything you can. Then ask for damages in addition to what he owes you.
It's only small claims anyway.