One way i found useful was using creep as the shallow water areas.
There is a way to make creep completely transparent by renaming its default texture.
I did it on my one map but i cant remember how and i have stopped working on it right now it needs an overhaul.
Then to allow ports to be built you just specify only built on creep.
Then make all your ships flying units that follow the water height.
There is a sightOp you can add that makes anything follow whatever is higher water or ground.
Make the deep areas a lower cliff level to prevent units from walking into it
And then add No ground pathing in the lower level
If no flying units exist in your map then you could outline your shore areas with no fly zone pathing.
(I think it exists maybe im wrong or go the burrow root.)
Just some thoughts!