So, the key problem I faced is how sc2 graphical engine handles the normal on the sides parallel to Z axis.
First image show how the TOP side reflects the light. It reflects it right, no matter what orientation the models has.
Ok, let's check how the same normal will control reflections on the SIDE side. Second image show that if a side faces, say, south-east, and its south edge is bright (case 1 on image 2), then if we rotate the model, so the same side will now face south-west, then it will still have the same edge bright, but this edge faces north now (case 2 on image 2). And this is what I don't like, it has to have southern side bright now, just like we saw on the first image.
Why doesn it work for top side, but doesn't work for side? Can it be fixed one day? <- these questions are for blizz guys in case they check this post.
Ok so try this:
Instead of producing normal map in max, render to texture a height map.
Then open that map in photoshop and:
a) copy the image to two seperate layers
b) on the first one apply embos filter with direction 0deg and 2px
c) on the second layer apply the same embos but with direction 90deg
d) take the first modified layer to green channel
e) take the second modified layer to alpha channel
f) fill red with white and blue with black.
g) save as .dds and check how it behaves in the editor.
Why embos? Well if you look at say Zealot normal map or any protoss or terran normal map you can see that they don't look generated but rather drawn.
If you draw a white line on a grey (149,149,149) background and apply the embos it will generate something that looks EXACTLY like one of the channels of any original StarCraft II normal map. The directions are obvious and the mapping layout as well as the orientation of faces of the mapping seem to be irrelevant.
If you think about it, you can generate so much information just from the data provided by the two channels (green and alpha). You can combine those two and tell how high the element should be, what's the height of a bump. And actually that should be enough to recalculate it to give the proper normal.
Of course, that kind of height map requires some editing to add it a bit of contrast so that the high elements are white actually. Proper level editing does the trick.
And this mean that my normals have always been right. So, no matter, how I create them, by generating in max, using filter on height map or painting it by hands.
Now I see that there is a problem which doesn't let normals modify light reflections correctly. They perfectly work on top face but messed up on side faces.
So, it's a kind of model orientation <-> mormals interaction issue. And its reason may be either 1) sc2 graphical engine bug or 2) I do something wrong in max or 3) m3 exporter saves some object info the wrong way. Or something else, but probably one of these reasons.
If we'd have art tools, it would instantly become obvious, what is the source of the problem. But for now I can assume that blizz guy's image look like there is no problem with normals. So, sc2 engine is ok. And there are only 2 reasons remain: me doing somerthing wrong in max, and exporter writes some values wrong.
Have you tried to create a simple 12 triangles box with smooth normals? I wish you try and and analyse its look and clarify the possibility of "m3 exporter's fault" hypothesis.
This is a picture of where I think the problem is.
It looks like all shaders except the normal map ones use correct orientation info. But normal map shader using a wrong scheme of orientation. I've tried to play with euler parameters in 3d max, but it didn't help.
So, it looks like when we rotate a model, its normal map shader's vector doesn't turn with the model. So, if a faces orientation accidently parallel to the constant normal shader vector, then we can see it lit properly. If we rotate the model by 180 degrees, the face will look inverted. If we rotate the model by 90 degrees, light will look fucked.
That's why the top vector always lit right. Because we rotate model around Z, and its normal vector is always parallel to the normal map shader's vector.
For now I think it's happening because m3 exporter exports one of model's flags wrong. But still it may be me wrong in problem's interpretation.
I have a slightly different theory about what is happening.
While I don't think that this is the full cause of the problem, I am fairly certain that it is part of it.
As stated by Zolden, the model doesn't quite work right when rotated. While true, it is also true that it doesn't quite work without being rotated.
My theory is that upon exporting vertex normals, the normals are only exported with a single vector upon conversion to SC2's normalized vectors(as stated in libm3's documentation).
Below is an illustration of my theory:
The image assumes that there are smoothing groups(vertex splits) at each side of the cube.
While I'm not entirely sure that this is what is happening, there seems to be, in most cases, 2 directions or axis(only positive or negative) in which the model renders properly.
i.e. Top and left would work properly, while the others would not.
I am, at time of writing, not sure if we have actually tested if the 2 correct sides stay the same after rotation, as this would reinforce the vertex normal theory.
However, this is the only theory which, at this point, seems logical.
So to sum up:
It is my theory that the m3 exporter only exports 2 axis properly, and when exporting the rest, forgets to modify the normal vector as to reflect the facing of the vertex.**