The Behemoths are going to stay in the game, if you don't wish to play with/against them just deactivate the Supremacy DLC in the Contents page on the main menu 

Two options really: turn off the dlc, or play with humans and agree to gang up and wipe out anyone who uses them.

I'm going to copy-paste something I wrote on the Steam forums about Core-Crackers here, it mostly applies to Behemoths as well.

-------------

Core Crackers are a somewhat necessary thing for the late game, especially if you are going for a conquest or elimination victory.

Late in the game when the faction core systems are super high pop and loaded with defenses, you need a way to cut the system population down quickly so they cannot stall an invasion by repeatedly using 'Draft' over and over. (Lumeris and Cravers seem especially prone to this, doesn't matter if you brought 1500 troops if they have 40 pop on the system to draft from)


But they are fairly easy to catch if you make a super speed fleet, since the AI doesn't seem to prioritize move speed at all.

Make "tugboat" ships, Small or Medium coordinator ships with as many 'Fleet Mover' upgrades as you can afford, and flesh out the rest of the anti-deathstar fleets with whatever else you can afford. I usually make railgun heavy Medium attackers for this purpose, since Railguns don't cost Strat resources (all rails + 1 zero-strat slug gun for flak), saving those for the fleet movers / coordinator ships.

Usually 3+ rail-boats & 2+ tugboats to a fleet, and position them near various "entry points" to my constellations near opposing factions. If they retreat from combat (and they will), then you should have the move speed to either catch them immediately, or overtake them and kill them at the next system.

For Supremacy, they added an extra way to stop Core Crackers and Obliterators, there is a support module for bigger ships that is a tractor beam, which prevents retreating. Slap one of those in your anti-deathstar fleet, and kill all the planetary / system destruction ships you want.

Added : For Obliterator chasing, you need to be more proactive rather than camping near constellation choke points, it can be worthwhile to have a few scout ships that throw probes towards enemy systems to see if / when they make a Behemoth.

---

Aside from all that, they don't come out to play until mid-late / late game, when you should be worrying less about individual systems anyway. Shield your systems, Citadel your home system, and don't worry too much about end game planet destruction. 

Does it suck to have a good system wiped out by an Obliterator? Yes, but that just means you need to be way more proactive on what your opponents are doing, especially if you are at war with them.