The difference between Infrastructure buildings in both cities is the main reason why the price lowers or raises. The more expensive the building is, the more impact on the price it will have. For example, building a granary in the second city will barely drop the price, while building a Grain Silo will drop it by a few thousand units. Districts and probably population to a very low extent also impact the price.

You seem to have built some very expensive buildings in the first city and this raised the price. The way to deal with this is to build/buy more infrastructure buildings in the second city - I activated Inherited lands and had a merge price of 163 000 gold between my two main cities. Once I managed to build up the second one over the course of 10 turns, the price was lowered to 68 000. I then saved some money for 3 turns and merged them. Make a point of not building any new infrastructure in the first city while you are trying to lower the price of the merge. You are still safe to build districts of any kind. :)

If both cities have the exact same infrastructure in terms of buildings the price will be negligible. This is I think a good balance mechanic to stop players from building temporary cities to build only districts and save time and resources overall. Also it makes sense from a narrative point of view - you have to integrate the new city into the other one and thus have to harmonize infrastructure and services.