It is a lot more complicated than the first glance. I have been testing out different fleets fighting against each other, and it really seems like once fleets get big, destroyers are a major liability.
If the enemy has a good number of subs, along with some planes or high value ships, then the you would be better off without any destroyers.
So when is it good to have a destroyer? If the enemy has few or no high value ships and few or no planes, then you will want the destroyer to negate the subs first strike ability. (or if the engagement is very small, like 1-4 units total, you may want it to make sure you are just outright killed).
So in other words, avoid building destroyers until you see what your enemy is building for a couple turns. And most likely you should never build them in the pacific.