I would understand if you couldn't connect a territory to a city because there was an enemy in THAT territory.

But the fact that if there's an enemy ANYWHERE in ANY territory belonging to that city it stops you is idiotic. It means you need to go searching though every forest and hidden patch to find whatever useless unit is magically slamming your empire to a halt. It should definitely only apply to enemies within the territory you're trying to attach. Especially when you have some massive 80+ territory city that's spanning the continent.

I once had a non-aggression pact AND open borders with a neighbor.

He decided to park his units over a district in one of my outposts.

I couldn't attach it.

I had to cancel the non-aggression pact. He instantly started ransaking my district. I then attacked the unit and was able to attach.

In this scenario it seems to work as intended.

But, yes, I agree it should only apply to enemies within the territory you're trying to attach/detach.

Yes this is a poorly thought out mechanic seeing as the AI will almost always park units right outside the target it wants to take if it were at war in some gamey attempt to declare war and rush for the city in a single turn. That target is always the player. Considering the gamey-ness of this mechanic and how the AI (or even wildlife....) can block the development of a territory without war or conflict I propose allowing territories to be attached and cities to be built NO MATTER WHAT. This means even if a mammoth is standing on your outpost you can make it into a city.

It is outrageous to be forced to make another outpost into your first city because a Mammoth decided to camp out on your preferred one during the start of YOUR turn. 

Yeah, it just seems entirely designed to hinder the player. Having someone you have non-aggression with being a blocker to your expansion seems wrong.

And as said, I wouldn't mind as much if they were in the area you were trying to expand into (then they could argue that it's something like guerilla attacks stopping expansion). Or if it's in the district connected to where you're trying to expand into (if that's the only one, if you could expand into the same area from a different district, it shouldn't be blocked).

I don't mind the whole "armies not being booted out" when you declare war. I know most games do that for game balance, to prevent one side just stacking armies ready to attack. I mean, it makes sense that if one side is aggressively placing units, that they might be trying to provoke conflict. And that sort of thing should be resolved in negotiation (basically just demand they pull their forces out).