The checksum part of starcode is better security than any obfuscation or encryption. Plus the compression makes it so that you're having to convert bases on every cleartext value.
The checksum part may be readable, but if its a good 1-way hash then thats all they'll be able to do, read it and NOT change it.
350 lines of code doesnt sound bad at all and I like the check sum modification check. I think Ill make use of this just need some time to figure it out I guess. Thanks for the answers.
I too am interested in this checksum. It is definitely a part of starcode thats highly valuable even if you don't intend to use the integer encoding.
With it you could save compressed data without encryption and still have a good level of security.
I use base 92 integers and currently sum the entire save string with 3 characters at the front, but its actually easier to break than a random hash sampling.
Now I'm thinking...
Max base size times the max character's store-able minus 3 characters for checksum. (95 * 846 = 80370)
3 characters in base 64. (64^3 = 262144)
So if you summed the entire save string, serialized it in base 64 as a bool[18], you could apply 64 bitwise encryption to just the sum.
I had a 64bit xor encryption lib that I deleted because it increased the size of the save string by 17%~ , but 80370 has to be stored with 3 characters regardless of base 44 or base 284. Sacrificing 3 characters for 64bit encryption doesn't seem like too bad an idea.
Now, where'd i put that library...
Oh well the 5 characters that I'm talking about are Html encoded. & ' " < and >
I can save 849 html encoded characters and load them back up.
What I meant was that no matter if the character is saved as "a" or "'" in the xml file, they both only count as 1 toward the 849 character limit in this example.
Tested by the below bank named "a.SC2Bank".
The name of the bank has to be 1 character long in order for this bank to load.
I've got a couple of people in the states that can help me test my boring ass bank-testing maps now, shouldn't be too long before I figure out the true science behind the limitations of the bank.
I don't know what I said that made you think that we could use more than the 95 printable ascii character, but I'm interested! Hows that coming along?
I wouldn't know how you could get the extended ascii alphabet into galaxy editor. It wouldn't let me paste them in as a string and I don't know of a way to caste them to a string, other than perhaps having to load them from a preset bank, if thats even possible.
I've been testing online as well, publishing maps to bnet and playing it through multiplayer custom map.
Take any bank you have right now and load it up a huge comment <! 20Mb string goes here > and you'll see the map still loads even through b.net.
What brought this to my attention was when adding an extra key to the bank adds at least 37 characters of xml, which you would assume would reduce the amount of savable user data by 37 characters, but it only reduced the amount of usable user data by 9 characters.
My own theory is that the file is loading to memory, ofc ignoring comments, and forming a response in a format thats more efficient than xml, perhaps json, and then measuring the size of that response in memory.
Id love to help test the limitations with multiple players but I'm in usa region. I'll probably use one of my free trials and find out the impact of 2 players.
html formatted characters have worked fine for me so far, even the " and < characters despite what i've read here.
they save to the bank html encoded (and don't take up additional space, more on that later), and they load back up just fine.
I've tested the all of the 95 printable ascii characters and they all save and load back up just fine.
When I tested the bounds of the bank and found that I could save 849 characters if you name the bank, section, and key as single characters, I did so with numeric characters.
When I discovered that some characters save in an html encoded format, I tried seeing if the extra characters used for encoding took up additional characters and they DO NOT. After that I tried making the bank file itself really large by adding a metric shitton of comments to it and it did not effect whether or not the map would load. So the whole "4k divided among all players" appears to be a bogus assumption.
So far what I've read from others about the limitations of the bank have not been very accurate. I'd doublecheck their claims before acting on them.
I'm making a library for making the bank easily accessible.
I've got base 92 compression on my integer arrays and separating them with reserved characters.
I want to use starcode to compress the integer arrays even further, but I'm not sure I understand how to use it, or what the limitations are.
I read a little bit into arithmatic and huffman encoding, and it just gives me a headache. xD
1) can it store negative values?
2) can it store an arbitrary amount of values, or does it need to know specifically how many values are stored within?
3) does the 'maximum' value of an integer effect the efficiency of the compression? For an arbitrary amount of values stored within, I'd have to assume the maximum and minimum possible values of every integer.
4) how is the performance of the compression and decompression? I've read in here that there are concerns of timing out during decompression on values over 100 characters? Approximately how many integers is that?
Any help is much appreciated! You guys accomplishing arithmatic encoding in such a shitty language is quite amazing, much props to you guys!
The checksum part of starcode is better security than any obfuscation or encryption. Plus the compression makes it so that you're having to convert bases on every cleartext value. The checksum part may be readable, but if its a good 1-way hash then thats all they'll be able to do, read it and NOT change it.
@s3rius: Go
Ok thanks, appreciate it.
Heres the test I'm running:
@s3rius: Go
Thats strange indeed, could the EU version of the game actually be different than the US version?
I too am interested in this checksum. It is definitely a part of starcode thats highly valuable even if you don't intend to use the integer encoding. With it you could save compressed data without encryption and still have a good level of security.
I use base 92 integers and currently sum the entire save string with 3 characters at the front, but its actually easier to break than a random hash sampling.
Now I'm thinking... Max base size times the max character's store-able minus 3 characters for checksum. (95 * 846 = 80370)
3 characters in base 64. (64^3 = 262144)
So if you summed the entire save string, serialized it in base 64 as a bool[18], you could apply 64 bitwise encryption to just the sum.
I had a 64bit xor encryption lib that I deleted because it increased the size of the save string by 17%~ , but 80370 has to be stored with 3 characters regardless of base 44 or base 284. Sacrificing 3 characters for 64bit encryption doesn't seem like too bad an idea. Now, where'd i put that library...
@s3rius: Go
Oh well the 5 characters that I'm talking about are Html encoded. & ' " < and >
I can save 849 html encoded characters and load them back up. What I meant was that no matter if the character is saved as "a" or "'" in the xml file, they both only count as 1 toward the 849 character limit in this example. Tested by the below bank named "a.SC2Bank". The name of the bank has to be 1 character long in order for this bank to load.
I've got a couple of people in the states that can help me test my boring ass bank-testing maps now, shouldn't be too long before I figure out the true science behind the limitations of the bank.
@s3rius: Go
I don't know what I said that made you think that we could use more than the 95 printable ascii character, but I'm interested! Hows that coming along?
I wouldn't know how you could get the extended ascii alphabet into galaxy editor. It wouldn't let me paste them in as a string and I don't know of a way to caste them to a string, other than perhaps having to load them from a preset bank, if thats even possible.
@s3rius: Go
I've been testing online as well, publishing maps to bnet and playing it through multiplayer custom map. Take any bank you have right now and load it up a huge comment <!
20Mb string goes here> and you'll see the map still loads even through b.net. What brought this to my attention was when adding an extra key to the bank adds at least 37 characters of xml, which you would assume would reduce the amount of savable user data by 37 characters, but it only reduced the amount of usable user data by 9 characters.My own theory is that the file is loading to memory, ofc ignoring comments, and forming a response in a format thats more efficient than xml, perhaps json, and then measuring the size of that response in memory.
Id love to help test the limitations with multiple players but I'm in usa region. I'll probably use one of my free trials and find out the impact of 2 players.
@s3rius: Go
html formatted characters have worked fine for me so far, even the " and < characters despite what i've read here. they save to the bank html encoded (and don't take up additional space, more on that later), and they load back up just fine.
I've tested the all of the 95 printable ascii characters and they all save and load back up just fine.
When I tested the bounds of the bank and found that I could save 849 characters if you name the bank, section, and key as single characters, I did so with numeric characters.
When I discovered that some characters save in an html encoded format, I tried seeing if the extra characters used for encoding took up additional characters and they DO NOT. After that I tried making the bank file itself really large by adding a metric shitton of comments to it and it did not effect whether or not the map would load. So the whole "4k divided among all players" appears to be a bogus assumption.
So far what I've read from others about the limitations of the bank have not been very accurate. I'd doublecheck their claims before acting on them.
I'm making a library for making the bank easily accessible. I've got base 92 compression on my integer arrays and separating them with reserved characters.
I want to use starcode to compress the integer arrays even further, but I'm not sure I understand how to use it, or what the limitations are. I read a little bit into arithmatic and huffman encoding, and it just gives me a headache. xD
1) can it store negative values? 2) can it store an arbitrary amount of values, or does it need to know specifically how many values are stored within? 3) does the 'maximum' value of an integer effect the efficiency of the compression? For an arbitrary amount of values stored within, I'd have to assume the maximum and minimum possible values of every integer. 4) how is the performance of the compression and decompression? I've read in here that there are concerns of timing out during decompression on values over 100 characters? Approximately how many integers is that?
Any help is much appreciated! You guys accomplishing arithmatic encoding in such a shitty language is quite amazing, much props to you guys!
I was veeery dissappointed when I learned you can only store the limited and undocumented 800ish~ characters in a player's bank. :(