When attacking the AI (I have not tested against human players) if there is a fortified tile with walls or fence a siege unit cannot attack it if there is no enemy unit on the tile. It will display a message saying no path to the tile, even if you click the attack button first. 

I had a game where I got the fence of the tile down to 36 health then the unit behind the wall died and I could not attack the tile with siege units to remove the fortification and my chariots could not occupy the tile. I could only move a melee unit in to the tile. The same applies if there is a garrison tile that has no unit, the walls will stay at 100% as you cannot attack the tile with a siege unit.