Well, first of all you could prompt the user to provide a password - it doesn't have to be static or literal.
The problem is that maps aren't protected that well. As soon as cracking tools appear, every noob will be able to open maps, even if they're protected (just like what happened in Warcraft3).
But since it's in Galaxy it will probably at least scare away the total noobs lol.
The reals in SC2 are floats (just like vjeux said). Basically they're an integer divided by 4096. So if you want to save a real, then just multiply the number by 4096 and store it. When you retrieve it you divide by 4096 and you get the exact same number you saved.
I'm making a library for making the bank easily accessible.
I've got base 92 compression on my integer arrays and separating them with reserved characters.
I want to use starcode to compress the integer arrays even further, but I'm not sure I understand how to use it, or what the limitations are.
I read a little bit into arithmatic and huffman encoding, and it just gives me a headache. xD
1) can it store negative values?
2) can it store an arbitrary amount of values, or does it need to know specifically how many values are stored within?
3) does the 'maximum' value of an integer effect the efficiency of the compression? For an arbitrary amount of values stored within, I'd have to assume the maximum and minimum possible values of every integer.
4) how is the performance of the compression and decompression? I've read in here that there are concerns of timing out during decompression on values over 100 characters? Approximately how many integers is that?
Any help is much appreciated! You guys accomplishing arithmatic encoding in such a shitty language is quite amazing, much props to you guys!
1) Can't store negative values. That'd break the algorithm and my bigNum library. If you really want a negative values then you can just double the integer size (the maximum value) and count the first half as negative. The result (in terms of compression efficiency) would be the same.
2) I can store any amount of integers, however when extracting the data from the compressed string you need to extract them in the exact reverse order. So indirectly you do need to know the number of integers saved.
3) The maximum values doesn't effect the efficiency, but obviously the system requires more room for higher maximum values (even if the actual saved value is low).
4) At some point the system hits the execution limit. I can't really give you an exact count of integers as they take a dynamic amount of space. The larger the integers, the fewer there are.
However, I'm a little bit drunk right now and this seems to shut down my brain enough to actually find a solution to the execution limit that's so mind bloggingly easy that I didn't think of it when I was sober:
Right now the execution limit is hit when too many integers are saved at once.. just add a wait of 0.01 seconds between every 30-or-so integer storages and the execution limit should never be hit.
tordecybombo has brought to me attention that there are some problems with saving generated string to banks.
The problem: Some characters get formatted by the XML banks and cannot be loaded correctly afterwards.
The affected characters:
& ' >
I updated STARCODE to version 1.3 with these characters removed. There shouldn't be any problems now. (The encryption alphabet is now 89 characters long).
html formatted characters have worked fine for me so far, even the " and < characters despite what i've read here.
they save to the bank html encoded (and don't take up additional space, more on that later), and they load back up just fine.
I've tested the all of the 95 printable ascii characters and they all save and load back up just fine.
When I tested the bounds of the bank and found that I could save 849 characters if you name the bank, section, and key as single characters, I did so with numeric characters.
When I discovered that some characters save in an html encoded format, I tried seeing if the extra characters used for encoding took up additional characters and they DO NOT. After that I tried making the bank file itself really large by adding a metric shitton of comments to it and it did not effect whether or not the map would load. So the whole "4k divided among all players" appears to be a bogus assumption.
So far what I've read from others about the limitations of the bank have not been very accurate. I'd doublecheck their claims before acting on them.
When we found out about the 4kb limit on wc3c it only applied to online games. So if you tested your map through the editor you can load more than 2MB (that's how far I tested) without problems.
But you're right that another check on the limits would be good. I'll see if I can get anyone to help me testing that online.
I've been testing online as well, publishing maps to bnet and playing it through multiplayer custom map.
Take any bank you have right now and load it up a huge comment <! 20Mb string goes here > and you'll see the map still loads even through b.net.
What brought this to my attention was when adding an extra key to the bank adds at least 37 characters of xml, which you would assume would reduce the amount of savable user data by 37 characters, but it only reduced the amount of usable user data by 9 characters.
My own theory is that the file is loading to memory, ofc ignoring comments, and forming a response in a format thats more efficient than xml, perhaps json, and then measuring the size of that response in memory.
Id love to help test the limitations with multiple players but I'm in usa region. I'll probably use one of my free trials and find out the impact of 2 players.
If that is true then we might, just might, be able to store almost the full range of SC2 string characters (255), which would give a great boost to compression.
Hi s3rius. I'm been using your code for the map I'm developing, and it's been a great tool and it was instrumental for the popularity. However, there's a small portion of the players who reported that their saved variables aren't quite correct.
So I went about using your example map (1.3) to create a test case, where I looped over 1000 times of it. Basically, I stored 21 randomly generated numbers between 0 to 500000, compressed, encrypted and added a 3-level security hash. The reverse process was then done, the results compared with the original random values.
Results of the test: roughly a min. of 30 mismatches for 1000 tries. Just in case I've made a mistake, here's the test map, not 100% concluding anything yet.
As the game is initialized, the trigger Example Usage is repeated over and over. You'll see error debug messages for each mismatch.
Thanks to you and tordecybombo for pointing this out.
This is a sort of rare bug that happens while encryption and decryption when the code string is one character longer than the encryption key is.
Updated to 1.4 with the bugfix (It'll probably break older codes).
Once I know whether the XML encoding really doesn't increase bank size more than usual I'll throw out another update.
PS:
Updating the library is rather tedious, as you have to manually update all links to functions when you delete the old version of the library.
If you don't want to go through that you can open the Example map and copy the content of the "Source" trigger file, then paste that content over the Source file in your map (the pasting might take up to 10 seconds because of Blizzard's slow syntax checker).
This won't update the version number, so you shouldn't confuse the different versions.
I don't know what I said that made you think that we could use more than the 95 printable ascii character, but I'm interested! Hows that coming along?
I wouldn't know how you could get the extended ascii alphabet into galaxy editor. It wouldn't let me paste them in as a string and I don't know of a way to caste them to a string, other than perhaps having to load them from a preset bank, if thats even possible.
You can't write the other ASCII symbols in these strings, but you can use their codings: "/xZZZ" where ZZZ is a number between 0 and 255.
If the aforementioned symbols really don't take up additional space, then it's probable that the other signs don't either.
I haven't looked into all of that yet, because I spend my time of Blizzard's map contest thingy. Once I'm done with that I'll try it out.
Quote from s3rius:
@RodrigoAlves: Go
Nothing can convert text to string. Not even God.
Erm..bad joke..
Anyway, How powerful is the encryption system?. Is it possible to be cracked/deciphered?
Everything can be cracked. Starcode uses a rather simple encryption. If it was stand-alone someone with enough dedication and a lot of time could brute-force it. But the mathematical compression that turns the information into a string of seemingly random characters is impossible to crack unless you want to wait 10 years for a supercomputer to crack it.
The real weak point is that you can open the map, look at the script and modify the triggers. No encryption would help there as they basically circumvent it completely.
The only thing that the encryption should prevent is that someone could notice a pattern in the saved string and be able to manipulate it by trial-and-error.
The Encryption will change the string so you can't find a patter that easily while the hashing prevents 99% of all manipulation attempts.
PS: Initially the encryption was also thought to lock the string for other players. Encrypting using the player's name would make it inaccessible by anyone else. Since we still can't return a player's name as string this option doesn't work.
I still hope we get that native somewhen soon.
We take a bunch of Integer values that correlate with various saves or upgrades or whatever and then run those integers through the Starcode using Store Integer ([integer number], [maximum value]). The maximum value is an actual maximum value but also an identifier on how you retrieve the integer you are looking for. Anyways, then you save the string (TempString in the example map) to a Bank so it's an encrypted save.
When you load the map again, you load the string from the Bank, run it through the Starcode and load the Integers by assigning various variables by doing Variable - Set [whatever] = (<Starcode> Get Integer Value([maximum value from above Store command]). Then you can just have your game load the appropriate upgrades based off of those integers.
@vjeux:
what I meant was that as long as the user can't open up the triggers and see what the encoding string is.
@YiffMaster: Go
Well, first of all you could prompt the user to provide a password - it doesn't have to be static or literal.
The problem is that maps aren't protected that well. As soon as cracking tools appear, every noob will be able to open maps, even if they're protected (just like what happened in Warcraft3).
But since it's in Galaxy it will probably at least scare away the total noobs lol.
@Anthius: Go
The reals in SC2 are floats (just like vjeux said). Basically they're an integer divided by 4096. So if you want to save a real, then just multiply the number by 4096 and store it. When you retrieve it you divide by 4096 and you get the exact same number you saved.
I'm making a library for making the bank easily accessible. I've got base 92 compression on my integer arrays and separating them with reserved characters.
I want to use starcode to compress the integer arrays even further, but I'm not sure I understand how to use it, or what the limitations are. I read a little bit into arithmatic and huffman encoding, and it just gives me a headache. xD
1) can it store negative values? 2) can it store an arbitrary amount of values, or does it need to know specifically how many values are stored within? 3) does the 'maximum' value of an integer effect the efficiency of the compression? For an arbitrary amount of values stored within, I'd have to assume the maximum and minimum possible values of every integer. 4) how is the performance of the compression and decompression? I've read in here that there are concerns of timing out during decompression on values over 100 characters? Approximately how many integers is that?
Any help is much appreciated! You guys accomplishing arithmatic encoding in such a shitty language is quite amazing, much props to you guys!
1) Can't store negative values. That'd break the algorithm and my bigNum library. If you really want a negative values then you can just double the integer size (the maximum value) and count the first half as negative. The result (in terms of compression efficiency) would be the same.
2) I can store any amount of integers, however when extracting the data from the compressed string you need to extract them in the exact reverse order. So indirectly you do need to know the number of integers saved.
3) The maximum values doesn't effect the efficiency, but obviously the system requires more room for higher maximum values (even if the actual saved value is low).
4) At some point the system hits the execution limit. I can't really give you an exact count of integers as they take a dynamic amount of space. The larger the integers, the fewer there are.
However, I'm a little bit drunk right now and this seems to shut down my brain enough to actually find a solution to the execution limit that's so mind bloggingly easy that I didn't think of it when I was sober:
Right now the execution limit is hit when too many integers are saved at once.. just add a wait of 0.01 seconds between every 30-or-so integer storages and the execution limit should never be hit.
tordecybombo has brought to me attention that there are some problems with saving generated string to banks.
The problem: Some characters get formatted by the XML banks and cannot be loaded correctly afterwards.
The affected characters: & ' >
I updated STARCODE to version 1.3 with these characters removed. There shouldn't be any problems now. (The encryption alphabet is now 89 characters long).
@s3rius: Go
html formatted characters have worked fine for me so far, even the " and < characters despite what i've read here. they save to the bank html encoded (and don't take up additional space, more on that later), and they load back up just fine.
I've tested the all of the 95 printable ascii characters and they all save and load back up just fine.
When I tested the bounds of the bank and found that I could save 849 characters if you name the bank, section, and key as single characters, I did so with numeric characters.
When I discovered that some characters save in an html encoded format, I tried seeing if the extra characters used for encoding took up additional characters and they DO NOT. After that I tried making the bank file itself really large by adding a metric shitton of comments to it and it did not effect whether or not the map would load. So the whole "4k divided among all players" appears to be a bogus assumption.
So far what I've read from others about the limitations of the bank have not been very accurate. I'd doublecheck their claims before acting on them.
@Hokibukisa: Go
When we found out about the 4kb limit on wc3c it only applied to online games. So if you tested your map through the editor you can load more than 2MB (that's how far I tested) without problems.
But you're right that another check on the limits would be good. I'll see if I can get anyone to help me testing that online.
@s3rius: Go
I've been testing online as well, publishing maps to bnet and playing it through multiplayer custom map. Take any bank you have right now and load it up a huge comment <!
20Mb string goes here> and you'll see the map still loads even through b.net. What brought this to my attention was when adding an extra key to the bank adds at least 37 characters of xml, which you would assume would reduce the amount of savable user data by 37 characters, but it only reduced the amount of usable user data by 9 characters.My own theory is that the file is loading to memory, ofc ignoring comments, and forming a response in a format thats more efficient than xml, perhaps json, and then measuring the size of that response in memory.
Id love to help test the limitations with multiple players but I'm in usa region. I'll probably use one of my free trials and find out the impact of 2 players.
@Hokibukisa: Go
If that is true then we might, just might, be able to store almost the full range of SC2 string characters (255), which would give a great boost to compression.
Hi s3rius. I'm been using your code for the map I'm developing, and it's been a great tool and it was instrumental for the popularity. However, there's a small portion of the players who reported that their saved variables aren't quite correct.
So I went about using your example map (1.3) to create a test case, where I looped over 1000 times of it. Basically, I stored 21 randomly generated numbers between 0 to 500000, compressed, encrypted and added a 3-level security hash. The reverse process was then done, the results compared with the original random values.
Results of the test: roughly a min. of 30 mismatches for 1000 tries. Just in case I've made a mistake, here's the test map, not 100% concluding anything yet.
As the game is initialized, the trigger Example Usage is repeated over and over. You'll see error debug messages for each mismatch.
@EarendilSphere: Go
Thanks to you and tordecybombo for pointing this out. This is a sort of rare bug that happens while encryption and decryption when the code string is one character longer than the encryption key is.
Updated to 1.4 with the bugfix (It'll probably break older codes).
Once I know whether the XML encoding really doesn't increase bank size more than usual I'll throw out another update.
PS:
Updating the library is rather tedious, as you have to manually update all links to functions when you delete the old version of the library. If you don't want to go through that you can open the Example map and copy the content of the "Source" trigger file, then paste that content over the Source file in your map (the pasting might take up to 10 seconds because of Blizzard's slow syntax checker). This won't update the version number, so you shouldn't confuse the different versions.
@s3rius: Go
Cool s3rius. It worked smoothly w/o errors from the brute force testing.
Thanks a lot.
@s3rius: Go
I don't know what I said that made you think that we could use more than the 95 printable ascii character, but I'm interested! Hows that coming along?
I wouldn't know how you could get the extended ascii alphabet into galaxy editor. It wouldn't let me paste them in as a string and I don't know of a way to caste them to a string, other than perhaps having to load them from a preset bank, if thats even possible.
You can't write the other ASCII symbols in these strings, but you can use their codings: "/xZZZ" where ZZZ is a number between 0 and 255.
If the aforementioned symbols really don't take up additional space, then it's probable that the other signs don't either.
I haven't looked into all of that yet, because I spend my time of Blizzard's map contest thingy. Once I'm done with that I'll try it out.
Everything can be cracked. Starcode uses a rather simple encryption. If it was stand-alone someone with enough dedication and a lot of time could brute-force it. But the mathematical compression that turns the information into a string of seemingly random characters is impossible to crack unless you want to wait 10 years for a supercomputer to crack it.
The real weak point is that you can open the map, look at the script and modify the triggers. No encryption would help there as they basically circumvent it completely.
The only thing that the encryption should prevent is that someone could notice a pattern in the saved string and be able to manipulate it by trial-and-error.
The Encryption will change the string so you can't find a patter that easily while the hashing prevents 99% of all manipulation attempts.
PS: Initially the encryption was also thought to lock the string for other players. Encrypting using the player's name would make it inaccessible by anyone else. Since we still can't return a player's name as string this option doesn't work.
I still hope we get that native somewhen soon.
Just wondering, why don't you use the \ character?
I've been able to save it perfectly fine with banks/strings.
@fernsauce: Go
Because \ is used in string formatting in SC2.
Just to verify what I'm doing is correct:
We take a bunch of Integer values that correlate with various saves or upgrades or whatever and then run those integers through the Starcode using Store Integer ([integer number], [maximum value]). The maximum value is an actual maximum value but also an identifier on how you retrieve the integer you are looking for. Anyways, then you save the string (TempString in the example map) to a Bank so it's an encrypted save.
When you load the map again, you load the string from the Bank, run it through the Starcode and load the Integers by assigning various variables by doing Variable - Set [whatever] = (<Starcode> Get Integer Value([maximum value from above Store command]). Then you can just have your game load the appropriate upgrades based off of those integers.
Is that about correct?
@siege911: Go
Yep.