I have a city beside a rocky forest.

The game won't allow me to build a lumber mill.

Another city besides a regular forest can build it.

Why would the dirt matter?

A forest is a forest.