It would not make sense for them to go down there. Most of the citizens don't even know about the underground port and everybody also witnessed Pica's strength. If they would go down there they would be sitting ducks. Doflamingo would be able to throw attacks at them and use Parasite until they all die, or Pica would be able to just bury them all alive.
Keep in mind that we know more than the people in One Piece know. Most people don't know about the port and a lot of people don't know about the sharpness of the cage. Normal citizens in general don't know about the port, because it was always kept secret, while those that were turned into toys, have a big grudge against Doflamingo and would be going straight for him to take revenge. There was also a short time period where the citizens believed Doflamingo and wanted to end the game by capturing the criminals.
Dressrosa is a pretty big country and is in a state of confusion. Aside from the fact that they just wouldn't all fit in there, most citizens take the most logical decision and that is running away from the most imminent treat. Those at the suburbs would notice the cage coming closer and would run towards the city center, while those at the city center, would see the Donquixote family being close and would try to run towards the cage as they don't know it is closing in and is as sharp as it is. The citizens are powerless and would just run until this is all over.
Additionally, the tunnel under Dressrosa is filled with water, so they either have to swim out of there, or take a boat. Since there are many people that cannot swim (elderly, kids, even adults, Devil Fruit users), they should all be carried or taken on a ship. Yet most citizens don't even know how to navigate a ship and there just are not enough ships available to carry everybody. So in any case many would have to be left behind.
When they would eventually reach the outskirts of the country, they should go somewhere. Yet they are in open sea, so there is nowhere to go. The only thing they could do is, build a tunnel upwards or make a net to climb the huge boulders and wait until the fight is over. While this on its own sounds very unlikely to be done, when they get out, Doflamingo would just redo the birdcage over the entire island, destroy the tunnel/net and start over from the beginning.