SC2Mapster

Found some compiler bugs:

1. Galaxy does not support identifiers starting with _ (but yours accept it)
2. Your compiler messes up inlining sometimes, FixedToInt(InlinedFunction()), doesn't inline the function... however, it doesn't include the InlinedFunction() either... so the generated code references InlinedFunction(), but it doesn't exist.

A few other things I've noticed:

1. Auto-completion does not work inside brackets... when you're indexing an array.
2. Ctrl-F does not select the content of the input-field... meaning, you often end up writing what you want to search for, behind what you previously searched for.

Awesome!

Quote from SBeier: Go

@syranide2: Go

It doesn't complain if you define them, and it will rename them to have unique names. But if you try to invoke one of them, you get an error about ambiguous invokes.

Ah, that might explain it, I always messed up when copying my trigger functions, which of course aren't "invoked" as usual.

Long time no see I guess ;)

Anyway, found a few more issues:

1. Auto-indentation messes up when {} or }{ exist on a single line... most troublesome is "} else {"
2. Integers cannot be typecast to bytes, or converted at all right now it seems, bytes can't be typecast either it seems. This is quite an issue right now.
3. Page down, goes to the end of the document, not a page down.
4. Something is off with "replace all", was going to replace 2 items, it said 2 items where replaced I think, but only 1 item was actually replaced it seemed, pressing again said 0 items was replaced ... but yet the last one also got replaced.
5. Doesn't seem to complain when there are several identical function signatures.
6. "cond ? iftrue : iffalse", would be much appreciated ... although the expansion might be less than trivial I assume... would simplify certain code, but far from important really.

@Vestras: Go

Uh... let me ask a stupid question then, exactly what does it compile to... or what is the purpose of compiling it? External error checking?

Since it seems a lot of people still haven't found the proper solution to this, and rely on a lot of crappy out-dated guides and information (including me until I stumbled by it yesterday), just use this simple function/trigger instead:

http://wiki.sc2mapster.com/galaxy/triggers/players-on-team/

Quote from Vestras: Go

@syranide2: Go

Direction as in the order of functions? I haven't implemented validation of that yet.

What other compilers are there?

There's andromeda (heavy OOP), galaxy plusplus (fixing galaxy flaws), and a bunch of others I'm not too read up about. So my question was basically, do you have any special focus with your compiler (like above)... simplicity, flexibility, performance, OOP, etc, etc?

Or am I mistaken about what you are trying to do?

Interesting, I'm curious if you've thought about direction (purpose)?

There are already a bunch of compilers out there, in various stages and with various intentions (I'm not implying that one more is bad or futile, it isn't).

Quote from SBeier: Go

Well.. in any case, in order for the galaxy compiler to accept it, the fields must be defined before they are used. And they were marked as const (before the function implementation). Still, some of them appeared to not be set when using them in TriggerCreate outside a method. I don't understand why the errors are there, but they are.. so I'll try rewriting stuff until it works.

I think I'll keep the obfuscation the way it is. While an adversary could just use the algorithm to decrypt strings, it does take quite a bit of work on his part.. Assuming short naming is also on in the map, he would have a hard time finding out which string is used in what context, and he would need to take time to write a map/program to decrypt all the strings. Yes, it's still possible, but that's the point of obfuscation. I don't want obfuscation to just keep the strings in cleartext, out of context.

Well, not really. All a user would have to do is to just insert a TriggerDebugOutput in your deobfuscation-function, and voila, all strings that are encountered can be seen as "from->to" in the debugger, and can be easily replaced in the script afterwards. This way he'd basically get access to all the vital strings... and even just extracting all the strings from the script-file and running them through your deobfuscator the same way would require minimal effort.

Meaning, I see little point in using an advanced encryption function, when a very basic one would work just as well (read: non-trivial). Hiding the actual strings is an obstacle, but it doesn't matter how high the obstacle is if you can just go around it.

I'm not saying that obfuscation doesn't work, it does, but the way yours is implemented severely restricts the usefulness, it doesn't matter how advanced of an encryption you use, I would still just use the above method, and it would work every time, minimal effort. If you were to inline the function-calls and parts of the deobfuscation-function and provide a separate seed for every string, then yeah, it would be a whole lot harder (however, care still needs to taken when caching the strings).

I'm guessing one might even be able to play with arrays as well for the decryption, splitting the encrypted strings (compilation), store them in arrays (runtime) and then assemble them back together with tricky indexing math... as long as there is no single deobfuscation function after this, it mighty be really rather annoying. However, the issue is always, caching the results or even just storing it in intermediate strings would make it easy to intercept.

The idea is to remove any single vulnerable point, so that it can't be trivially automated or intercepted. Lots of manual labor is probably the biggest turn-off.

Quote from SBeier: Go

@syranide2: Go

I had problems where fields on the form

trigger t = TriggerCreate("foobar");

would not be deobfuscated correctly. Don't know why, but using a function call for getting the value seemed to work better. It's not perfect though.. There are still some errors like the above which persists. I'm thinking I could try putting fields like that into the start of the MapInit function, but this is really a frustrating and illogical issue :(

Ah ok, anyway, I'd say that you could probably simplify your deobfuscation function, it doesn't matter what kind of advanced stuff it does, it is all visible in the debugger and can easily be deobfuscated using the function itself. Implementing something minimal would be good enough I'd say (just some form generated scrambled look-up table and doing the lookup with some variable offset, or such should be more than good enough or not make a difference anyway).

But then again, since it's only done once, perhaps it doesn't really matter, I'm not sure how much of an impact it actually has right now (probably not significant).

Although I'm not exactly sure I follow what's going wrong in your example, as long as have "string x = deobfuscate("blablabla")" BEFORE anything that uses it, it should be fine right? Or is the order of initialization not guaranteed for variables perhaps?

I'm curious why string obfuscation uses nested-methods instead of just using global strings that are decoded on init? Are there any obfuscation benefit to doing that?

EDIT: Also, perhaps you should strip out all indentation and empty lines when obfuscating.

Nevermind

Quote from SBeier: Go

I assume you mean something like this

int[42] MyStruct_a;
bool[42] MyStruct_b;
byte[42] MyStruct_FreeIndexes;
int[2] MyStruct_InUse;//I assume that a bool takes up 1 byte. Want to save memory here.
int MyStruct_FreeSlotsLeft = 42;//Array indexes must be integers

//Run this as first thing during map init
void MyStruct_PreProcess()
{
    int i = 0;
    while (i < 42)
    {
        MyStruct_FreeIndexes[i] = i;
        i = i + 1;
    }
}

int Power2(int i)
{//Did try doing this with >>, but 1>>1 returned 0 :S
    int ret = 1;
    while (i > 0)
    {
        ret *= 2;
        i -= 1;
    }
    return ret;
}

byte MyStruct_Create()
{
    byte i;
    if (MyStruct_FreeSlotsLeft == 0)
    {
        UIDisplayMessage(PlayerGroupAll(), c_messageAreaDebug, StringToText("Attempeted to allocate MyStruct, but the defined maximum had already been reached"));
        IntToString(1/0);
    }
    i = MyStruct_FreeIndexes[0];
    MyStruct_FreeSlotsLeft -= 1;
    MyStruct_FreeIndexes[0] = MyStruct_FreeIndexes[MyStruct_FreeSlotsLeft];
    MyStruct_InUse[i/31] = MyStruct_InUse[i/31] | Power2(i%31);
    return i;
}



int MyStruct_Delete(int i)
{
	//Check if the specified index was already deleted
    if (MyStruct_InUse[i/31]  Power2(i%31) == 0)
    {
        return;
    }
    MyStruct_InUse[i/31] -= Power2(i%31);
    MyStruct_FreeIndexes[MyStruct_FreeSlotsLeft] = i;
    MyStruct_FreeSlotsLeft += 1;
}

The previous algorithm would take O(n) time for creating, O(1) for deleting, and 4*log31(n) extra bytes of memory for keeping track of free indexes (where n is the size of the arrays, and I assume I make the same optimization, for saving bool arrays, to the previous algorithm as done above). This one would take O(1) time for creating and deleting, but 4*(n + 1) + 4*log31(n) extra bytes of memory (for n>255). n + 4 + 4*log31(n) if n<256. Actually, since the Power2 function isn't exactly O(1), but O(i), and i is bounded by n, the algorithm will still run O(n). So, the second algorithm would allocate a lot more memory, and run O(n) too, because I now need to check that the user doesn't delete the same object twice. Also, I really don't think that the first one would hit a running time of n very often in practice. Especially not if we add the optimization you mentioned about decrementing MyStruct_Index when deleting (if the deleted object was the last allocated object). Can you see an optimization that would improve this algorithm, or did you have something else in mind?

I made the assumption that if the user deletes an item twice, it's his fault (meaning, create O(1), delete O(1), create pops, delete pushes). But yeah, that would definately mess things up badly if it happened, in very weird ways. However, one could just add the queue as an optimization on-top of your algorithm... it would make create, delete O(1) and really fast, and safe, however, it would add the 1-bit overhead PLUS 1-16 bits for the entries in the queue.

And I do agree that for the kind of algorithms and structures we see in SC2, I find it hard to imagine that it would spend O(n) looking for an available instance very often like you say (and increasing the pool size only even slightly would make it a non-issue). And I doubt it would be an issue even if it frequently did do a full scan (create/delete shouldn't be happening in such huge amounts). I doubt the performance gains of my proposed algorithm above is worthwhile, however, what may make it worthwhile is that it is predictable, create and delete will always be strictly O(1), which may be important for really large pools. Again though, I'm doubtful that it is an issue to begin with, but that's the only reason I see for considering it.

I would argue that you can do away storing "used" as bits in an int, as the overhead would be minimal for any reasonable uses... although at the same time, I doubt the mathematical overhead is even noticeable (again, construct/destruct should be somewhat rare) so the memory savings may be worth it. However, Power2() shouldn't be O(i) unless it's badly implemented, it really should be a simple lookup-function (oh hey, perhaps it's faster just to have your own lookup), or at the very least, be recursive O(log(i)).

-

Btw, I have no code to show you, but it seems that #inline messes up at times, I had a very simple function that just returned a value from an array. However, when inlined, would cause script errors on launch. Also, inlining produced really horrible code ;) ... it even seems to be assigning the variable to itself (inlinedvar = inlinedvar).

Found a compile bug (or so I argue anyway), your compiler tends to re-use variables when it can, which is ok. But it doesn't reset their values when it reuses them. Which is rather dangerous in my opinion, since variables always start with a default value, I would say that reused variables should be reset.

Quote from SBeier: Go

@syranide2: Go

Making something run every frame is bound to be inefficient tbh. Only way to do something every frame that I can think of is doing a periodic event every 0 seconds. That actually gets run more often than the frame redraw.. In my test, i made an increment of a global variable every 0 seconds. The trigger ran 23 times pr second and caused the fps (ctrl, alt, f) to drop to 4.

Anyway, it's still a problem that the data table might cause lag spikes.

About the implementation. I would like to keep things as simple as possible for the user. So I would probably rather have some way of setting a struct dynamic types to be global variables, and then keep everything else the way it is.

Such as

#array[42] struct MyStruct
{
    int a;
    bool b;
    
    void SetA(int i)
    {
        a = i;
    }
}

void foo()
{
    MyStruct str;
    str.SetA(2);
}

Becomes
struct MyStruct
{
    int a;
    bool b;
    
    
}
int[42] MyStruct_a;
bool[42] MyStruct_b;
bool[42] MyStruct_Used;
int MyStruct_Index;

int MyStruct_Create()
{
    int i = MyStruct_Index + 1;
    while (true)
    {
        if (i >= 42)
        {
            i = 0;
        }
        if (!MyStruct_Used[i])
        {
            break;
        }
        if (i == MyStruct_Index)
        {
            UIDisplayMessage(PlayerGroupAll(), c_messageAreaDebug, StringToText("Attempeted to allocate MyStruct, but the defined maximum had already been reached"));
            IntToString(1/0);
        }
        i = i + 1;
    }
    MyStruct_Index = i;
    MyStruct_Used[i] = true;
    return i;
}

//Could be inlined
int MyStruct_Delete(int i)
{
    MyStruct_Used[i] = false;    
}

void MyStruct_SetA(int i, int structIndex)
{
    MyStruct currentStruct;
    currentStruct.a = MyStruct_a[structIndex];
    currentStruct.a = i;
    MyStruct_a[structIndex] = currentStruct.a;
    //Yes, yes.. In this case, I might just do MyStruct_a[structIndex] = i; but that is an unrelated optimization.
}

void foo()
{
    MyStruct str;
    //Create a dynamic version of the struct, and pass that to the method
    int bulkCopyVar = MyStruct_Create();
    MyStruct_a[bulkCopyVar] = str.a;
    MyStruct_SetA(2, bulkCopyVar);
    str.a = MyStruct_a[bulkCopyVar];
    MyStruct_Delete(bulkCopyVar);
}

A solution like that would not require the user to worry about anything but the #array[42] notation.

@whimsyduke: Go

It apears to be working for me.. Unless I misunderstand your problem. Could you be more specific about what actually prevents you from making changes in the bottom of the file?

-

I fixed some bugs and such.. v2.1.7 Fixed a bug that made compile and save act as compile and save as Fixed a bug where a project's properties could not be changed if the project directory had changed. The editor now accepts + and - for points, like galaxy does. Added typedefs. (typedef int* pInt;) Fixed an error that made trigger definitions conflict incorrectly with some methods.

Yeah definately, stuff shouldn't be run every frame, but exceeding the frame time in execution time is bound to have bad results, as you say. And galaxy doesn't appear to be all that fast.

Looks good to me, although I can think of two things right now...

MyStruct_Delete() should probably decrement MyStruct_Index until it finds a used index, but only if the index being deleted is the same as MyStruct_Index. This would make short-lived objects not impact the overall performance as much for larger well-populated pools (although note, it would have to count the number of allocated objects too). But it depends on what pool-type you decide for, and how it will be used.

Speaking of which, one could probably implement a strictly O(1) approach to this, by keeping a list of bytes/integers instead of a bool, and having that be a queue of available object indexes (freed indexes are appended to the end). A byte would accomodate a pool up to 256 objects, which should be good enough for most circumstances. One could then fall back to integers or bools (integers could be cheaply split into 2 byte-integers instead for a maximum of 65k objects, if memory is an issue). Anyway, just a though, I'm not really sure about all the different uses of pools for SC2 right now, so it might really be a non-issue to start with.

As for the transformation of foo(), I would advice against being able to instantiate the struct locally, as it could then not be passed around without bulkcopying, like you do in foo(), which interestingly, is rather pointless it seems because you still have to allocate an object from the pool. And if you intend to support constructors/destructors (which should trivial), then you always have to allocate it from the pool.

-

Regarding whimsyduke, I do believe he's specifically referring to chinese characters, not regular ascii.