Those all go away.

To convert back (very hard) you can

1. cut trade routes (they spread religion)

2. build faith buildings (Wonders, holy sites, EQs) in locations that are still under your religion

The other option is to give up and just take the new religion as your state religion (so you can get its bonuses and no Grievances)

Religion tends to be a snowball because for every territory a religion controls, you get more pressure from every territory it controls (ie because London flipped to Christianity, Moscow puts out more Christian pressure)

Elteras wrote:

I cannot seem to find a way to remove foreign religions from my empire. Do your territories still produce faith for your own religion if it has converted to another religion? Do the faith from my wonders ,holy sites and shamanism + 1 faith per population not work once the territory has been converted?

This seems to be correct.  I documented this in another thread, "WHY FAITH SNOWBALLING OCCURS AND CAN'T BE STOPPED: CONVERTED TERRITORIES STOP SPREADING ORIG. FAITH", and a dev reached out to ask for saved games to analyse, so I'm hopeful this means that it's unintended that a converted territory effectively "shuts off" producing state faith, and they'll hopefully change things so and a faithful territory will be harder to convert, and an "extremely faithful" territory can convert back.

As it stands, territory and tenets overpower everything else in the game -- an empty outpost believing in a Tier 4 religion generates far more faith than a settled territory with a lower-tier religion, several faith EQs, and multiple holy sites, and will eventually overcome (and disable) that territory's faith.  Hopefully this gets sorted out.  In the meantime, I've put my "religious defense" game on pause and I'm playing another one.

Apparently the new beta patch fixes the big problem (the tenets having a bonus for total territory/pop under that religion)