Why did Scott Patterson leave “Sullivan's Crossing”? Inside the conflicting stories of his exit

Harry Sullivan promised to return to the Crossing at the end of season 3, but Patterson is nowhere to be found in season 4.