Anyone else ever noticed that catalog functions are ridiculously slow? I was experiencing BIG lag spikes in the mod I'm making and managed to track it down to catalog operations. That's only in early game, if I ignored the problem and continued it would probably ramp up exponentially.
For example I'm looking to find all the units that a certain train ability produces. Using the function "catalog field value get" the field for what units are made by this ability is an array so I give it the ability and a command index.
For example the first 10 entries for units in the factories train ability =
The only way to figure out which units this ability produces is to go through the array and look at each entry. If I ask it which unit is in slot 4 (thor) that operation takes about 0.1 seconds. However if I ask it about a slot with no unit in it then it takes about 0.3 seconds.
These figures are from running the operation 20 times and dividing the total time by 20.
0.1 and 0.3 seconds might not sound like much but added up these operations are HUGE! The factory has 30 entries mostly blank. (Widow mine is number 25). So that means to figure out what units a factory produces takes a whole 5 seconds! That's a completely unnacceptable amount of time for such a simple operation!
That's only one of the catalog lookups I am doing. I was also trying to look up other stats for units on the fly using catalogs as well but how long it takes renders that option impossible.
I thought of maybe creating arrays for all the information about unit types I'm going to need and populating it at map creation. The amount of time it takes to retrieve a variable from memory is negligable. But that leaves a MASSIVE lag spike at the start of the game.
An alternative being maybe having the arrays but only populating them as I go. So each time I look up a stat for a unit it will check the variables in memory first, otherwise do a catalog lookup and put the info in the array for future use.
(third option that I see is to manually add each of the entries)
Anyone got any ideas how I can find this kind of data without the massive wait times?
User Types to store static data. Also, unless I'm terribly mistaken, there are natives that will determine the requirements to do anything. AI uses them as to know which buildings to build in order to produce a given unit.
Also, I would test how fast a catalog lookup is the first time, and then subsequent times for the exact same query. If the later ones are faster, then the slow speed the first time is the result of loading the data into memory (on demand). This can be negated by preloading that unit (which will preload all data associated with it, ie, if you Preload a unit, it will preload its unit data, its weapon data, its actor, its model, abilities, etc.)
User Types to store static data. Also, unless I'm terribly mistaken, there are natives that will determine the requirements to do anything. AI uses them as to know which buildings to build in order to produce a given unit.
Also, I would test how fast a catalog lookup is the first time, and then subsequent times for the exact same query. If the later ones are faster, then the slow speed the first time is the result of loading the data into memory (on demand). This can be negated by preloading that unit (which will preload all data associated with it, ie, if you Preload a unit, it will preload its unit data, its weapon data, its actor, its model, abilities, etc.)
Yeah. I can manually recreate the information that already exists in the data editor creating a second copy. But that is wasteful, error prone and a lot of work. I would just like to be able to reference the data which is already there.
Makes no difference if it's the first or tenth time I look up the data in catalogues. In fact it often gets WORSE with time. I think that is because when you access an empty element in the array it adds it to the error log and the longer it gets the slower it goes.
AI functions exist to tell you what unit is a prerequisite. For example you give it marine as an argument and it returns barracks. But not the reverse.
I also need to look up details about unit weapons like damage types, attribute bonuses, etc and there are no functions for that either.
All the info I want is in the data files but it seems as if the only way to access it would take 5 minutes worth of lag
The only way to figure out which units this ability produces is to go through the array and look at each entry. If I ask it which unit is in slot 4 (thor) that operation takes about 0.1 seconds. However if I ask it about a slot with no unit in it then it takes about 0.3 seconds.
Something looks wrong with those times... How did you measure them?
I created a trigger such that when I ype "cats" (for catalogues) it does this:
int i;
ability abil;
string s;
set abil = factory train ability
debug "starting"
wait 1 second
for i = 1 to 20
{
s = catalog field value get ability abil infoarray unit[4]
debug s;
}
debug "finished"
wait 10 seconds
debug "starting"
wait 1 second
for i = 1 to 20
{
s = catalog field value get ability abil infoarray unit[3]
debug s;
}
debug "finished"
That would be the only trigger running. I used a wait 1 after declairing that it was about to start because the messages to screen do not happen instantly but are buffered. I would then select a worker and move him around the screen and count for how long the game freezes each time the loops are entered. unit 4 in the ability is a thor and unit 3 is empty so that is how I know it takes longer when the entry is empty. I then divide the total time spent when the game freezes by 20 to get the amount of time it takes for one call.
I would run this trigger many times in a row to make sure the result was consistent and would not improve due to memory being loaded in.
I ran this in "test document" which has a list of run time errors and warnings you can view. When doing a catalog lookup on an empty field it generates a warning so the discrepancy between empty and non empty fields might be due to generating that warning message and adding it to the list
Prints out Siege Tank, Thor and Hellion as expected (I am using WoL dependencies for this test, would print out the rest with other dependencies I am sure since the code is entirely dynamic). It also does all this producing no errors at all as it only fetches unit entries that exist.
How long does it take to execute you ask? I cannot tell exactly but what I can tell you is that it is completely trivial as the game does not even drop a single frame! As this is 300 executions of catalog get calls one can deduce that the execution time of a single catalog get is well below 55.6 micro seconds (0.0000556 seconds). This means that catalog natives are as good as trivial to execute.
Your poor performance must be coming from two places.
The Galaxy Virtual Machine error caused by an invalid catalog get call. My examples produce no such errors.
Logging/producing the debug messages used. My examples kept these calls to a minimum.
If you need any help implementing a specific system I could possibly help.
Thank you for taking the time to try and replicate my problem, I realy appreciate it!
I won't be able to test this for a week or two but I'll try your code and I'll be interested to see if it experiences the same amount of lag or none at all. If none, I'll be curious about what the cause of lag is.
Once I can test it I'll get back to you about where the discrepancy is coming from.
Do note that if you use the debug/performance tools that they add an execution overhead due to the instrumentation. Something may execute slowly in debug mode but not at all noticeable in normal mode as a result.
For that reason the above results I posted were for just plain old testing the map.
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
Anyone else ever noticed that catalog functions are ridiculously slow? I was experiencing BIG lag spikes in the mod I'm making and managed to track it down to catalog operations. That's only in early game, if I ignored the problem and continued it would probably ramp up exponentially.
For example I'm looking to find all the units that a certain train ability produces. Using the function "catalog field value get" the field for what units are made by this ability is an array so I give it the ability and a command index.
For example the first 10 entries for units in the factories train ability =
0 -
1 - siege tank
2 -
3 -
4 - thor
5 - hellion
6 -
7 -
8 -
9 -
10 -
The only way to figure out which units this ability produces is to go through the array and look at each entry. If I ask it which unit is in slot 4 (thor) that operation takes about 0.1 seconds. However if I ask it about a slot with no unit in it then it takes about 0.3 seconds.
These figures are from running the operation 20 times and dividing the total time by 20.
0.1 and 0.3 seconds might not sound like much but added up these operations are HUGE! The factory has 30 entries mostly blank. (Widow mine is number 25). So that means to figure out what units a factory produces takes a whole 5 seconds! That's a completely unnacceptable amount of time for such a simple operation!
That's only one of the catalog lookups I am doing. I was also trying to look up other stats for units on the fly using catalogs as well but how long it takes renders that option impossible.
I thought of maybe creating arrays for all the information about unit types I'm going to need and populating it at map creation. The amount of time it takes to retrieve a variable from memory is negligable. But that leaves a MASSIVE lag spike at the start of the game.
An alternative being maybe having the arrays but only populating them as I go. So each time I look up a stat for a unit it will check the variables in memory first, otherwise do a catalog lookup and put the info in the array for future use.
(third option that I see is to manually add each of the entries)
Anyone got any ideas how I can find this kind of data without the massive wait times?
User Types to store static data. Also, unless I'm terribly mistaken, there are natives that will determine the requirements to do anything. AI uses them as to know which buildings to build in order to produce a given unit.
Also, I would test how fast a catalog lookup is the first time, and then subsequent times for the exact same query. If the later ones are faster, then the slow speed the first time is the result of loading the data into memory (on demand). This can be negated by preloading that unit (which will preload all data associated with it, ie, if you Preload a unit, it will preload its unit data, its weapon data, its actor, its model, abilities, etc.)
Yeah. I can manually recreate the information that already exists in the data editor creating a second copy. But that is wasteful, error prone and a lot of work. I would just like to be able to reference the data which is already there.
Makes no difference if it's the first or tenth time I look up the data in catalogues. In fact it often gets WORSE with time. I think that is because when you access an empty element in the array it adds it to the error log and the longer it gets the slower it goes.
AI functions exist to tell you what unit is a prerequisite. For example you give it marine as an argument and it returns barracks. But not the reverse.
I also need to look up details about unit weapons like damage types, attribute bonuses, etc and there are no functions for that either.
All the info I want is in the data files but it seems as if the only way to access it would take 5 minutes worth of lag
i never had lag-issues using catalog actions.
what would be a typical call that results in an error?
Something looks wrong with those times... How did you measure them?
Methodology:
I created a trigger such that when I ype "cats" (for catalogues) it does this:
int i;
ability abil;
string s;
set abil = factory train ability
debug "starting"
wait 1 second
for i = 1 to 20
{
s = catalog field value get ability abil infoarray unit[4]
debug s;
}
debug "finished"
wait 10 seconds
debug "starting"
wait 1 second
for i = 1 to 20
{
s = catalog field value get ability abil infoarray unit[3]
debug s;
}
debug "finished"
That would be the only trigger running. I used a wait 1 after declairing that it was about to start because the messages to screen do not happen instantly but are buffered. I would then select a worker and move him around the screen and count for how long the game freezes each time the loops are entered. unit 4 in the ability is a thor and unit 3 is empty so that is how I know it takes longer when the entry is empty. I then divide the total time spent when the game freezes by 20 to get the amount of time it takes for one call.
I would run this trigger many times in a row to make sure the result was consistent and would not improve due to memory being loaded in.
I ran this in "test document" which has a list of run time errors and warnings you can view. When doing a catalog lookup on an empty field it generates a warning so the discrepancy between empty and non empty fields might be due to generating that warning message and adding it to the list
Tested the following.
Does not drop a single frame when executed and does correctly display "Thor".
However this is simple, lets try doing what you want but with correct validation.
Prints out Siege Tank, Thor and Hellion as expected (I am using WoL dependencies for this test, would print out the rest with other dependencies I am sure since the code is entirely dynamic). It also does all this producing no errors at all as it only fetches unit entries that exist.
How long does it take to execute you ask? I cannot tell exactly but what I can tell you is that it is completely trivial as the game does not even drop a single frame! As this is 300 executions of catalog get calls one can deduce that the execution time of a single catalog get is well below 55.6 micro seconds (0.0000556 seconds). This means that catalog natives are as good as trivial to execute.
Your poor performance must be coming from two places.
If you need any help implementing a specific system I could possibly help.
Thank you for taking the time to try and replicate my problem, I realy appreciate it!
I won't be able to test this for a week or two but I'll try your code and I'll be interested to see if it experiences the same amount of lag or none at all. If none, I'll be curious about what the cause of lag is.
Once I can test it I'll get back to you about where the discrepancy is coming from.
Do note that if you use the debug/performance tools that they add an execution overhead due to the instrumentation. Something may execute slowly in debug mode but not at all noticeable in normal mode as a result.
For that reason the above results I posted were for just plain old testing the map.