The owner of property destroyed pursuant to § 27-20 shall be entitled to recover from the county, city, or town the amount of the actual damage that he may have sustained by reason of the same having been pulled down or destroyed under such direction.