Ugh.. I really need help. My mod causes games with high player count and long game durations to have long exit times.
It wasn't originally there - I want to blame the Nova Patch but I feel I might have done something to mess it up although I have no idea what. Are there any general scripting rules for preventing long exit times? I haven't found ANY answers relevant to this issue.
The game doesn't run slower towards the end, my frames are consistently high, but F10 -> Quit Game causes players to be stuck for about 5-20 minutes before they are back at the b.net screen.
I already have, it's prettty much the same thing with other games.
Only one, it does not cause it. Because I have tried turning it off and on.
I have tried turning off some other triggers and it did prevent the long exit time - but I have no idea why and how to adjust it since they are critical to the game.
Before exiting the session, what is the virtual address size of StarCraft II?
Such long unload times would be the result of excessively complex game state being cleaned up. For example if a lot of data leaked during the session (was not destroyed when it should) and is then flushed when exiting it will cause a massive stall. Warcraft III suffered something similar if you leaked a lot of objects however its severity was limited due to degrading in game performance and ultimately a crash.
The 64bit build of StarCraft II is pretty much limited to as much virtual memory your system has. If the game state complexity exceeds the available memory then paging will occur. If this complexity is not used, eg leaked data, then the paging will not result in a noticeable performance decrease. However on exiting a session when the complexity is cleaned up then all that data has to be paged in again which can take a long time on mechanical drives. Worse is page thrashing could occur as the cleanup might not be memory sequential.
I recommend checking out actors in case there are some logical leaks there. Although situations like orphaned and duplicate actors should throw an error they might not under some cases. Also actor leaks might not cause performance degradation as invisible actors with no assets could persist in the actor system adding overall complexity while not requiring any computational maintenance.
Although it's a little hard to understand, I really appreciate the response. How can I check the virtual address size? Is that how much memory it's taking up in the task manager?
I will check for any lingering actors.
Is there any thing else that is likely to leak besides actors, such as data effects for example?
Yes those can leak. However the amount that can leak should be limited to only a few hundred or thousand before some limit is reached and errors start to be printed.
As for memory, you will be after the "Commit size" from task manager. That column for the processes view might not be visible by default and requires manual adding to the view (right click on the header and choose to add columns). Commit size represents the amount of allocated virtual memory address space for a process and hence the amount of data needed to be allocated somewhere in the system.
Sorry Doc, but I think your info - although maybe correct, is not relevant to my issue.
I have done lot's of data related tests and have failed to reproduce similar results from my game. For the past few months, I've been testing lots of different things to see what could be happening.
I don't have any serious data leaks after some numerous checks, but if I did have any it would be marginal.
The real issue I have is using CatalogReferenceSet or CatalogFieldValueSet.
I have an extremely dynamic game where tooltips and numbers are based around field values of Damage Effects like AttributeBonus, AttributeFactor.
They are updated very often, like every time a hero gets a status update or behavior update. I am starting to realize this issue will worsen, as the map gets even more sophistication.
After about a few million uses of CatalogReferenceSet or CatalogFieldValueSet, I get this long game exit hang. I've done a small test to prove this case (I've attached a map below).
So how is it possible to get a few million uses of the function?
Each player has a hero, gets about a status update every second (Theoretically). In each status update, there is about 150 (100 - 200) Catalog changes. With an expected game time of 20 - 60 minutes or 180,000 - 540,000+ Catalog changes per person, per game. My map supports up to a max of 12 players. But keep in mind, when certain effects and condition are passed - there are more than 1 status update per second - so it can easily do more than the amounts I just listed. The numbers might be wrong, but the point is that catalog setting has some kind of limit unknown to me and perhaps to most of the community as well.
Anyways the cap is easily broken, especially as new items and spells get featured into the game and thus adding even more injury.
Of course, knowing this now - I guess I can work around it and use behaviors to modify certain things like armor, movespeed, attack speed. And write more efficiently - checking if catalog setting is necessary before doing so. But who knew something like catalog setting would have such a hidden tax?
The map below has a custom script that has the following written in it. Just be sure to have window mode on, so you can end task without issues.
It writes 384,000 catalog reference every second. Your comp will probably lag, but I recommend just letting the map run for about 20 - 40 seconds and then leaving the game afterwards. Your game will be frame stuck for a really long time, if not forever.
Actually it is exactly as I said. Leaks cause long exit times. The game has problem destructing the game state because it is so complex due to leaks.
However what I did not know was that CatalogFieldValueSet used like this causes a leak. If you look at the memory used by SC2 under certain test conditions.
10,000 -> 950 MB -> 3 seconds to exit
100,000 -> 1000 MB -> 9 seconds to exit
1,000,000 -> 1500 MB -> exit too long to measure (>>5 minutes)
The way it degrades like this points towards some cleanup algorithm with O(n^2) or worse complexity. I am guessing the way catalog natives are implemented is that they make a new entry for the player which takes precedence over the currently used entry. However although it replaces the existing entry, it does not remove it resulting in a leak. When you exit a session it has to clean up all these entries.
Why the cleanup has a complexity of O(n^2) or worse I do not know. I am guessing it might be trying to destroy the entries in the inverse order of creation using a linear search, possibly to maintain base data integrity for caching purposes. However this really is a wild guess.
I would recommend reporting this as a bug to Blizzard. It really should not be leaking like this. I can understand the creation of data structures for a per player data entry however if you change that entry the old data structures should be discarded rather than retained.
Ugh.. I really need help. My mod causes games with high player count and long game durations to have long exit times.
It wasn't originally there - I want to blame the Nova Patch but I feel I might have done something to mess it up although I have no idea what. Are there any general scripting rules for preventing long exit times? I haven't found ANY answers relevant to this issue.
The game doesn't run slower towards the end, my frames are consistently high, but F10 -> Quit Game causes players to be stuck for about 5-20 minutes before they are back at the b.net screen.
Thanks so much for anyone who knows.
check the sc2 error logs on your computer. also do you have a trigger that fires on player leaves game?
@FunkyUserName: Go
I already have, it's prettty much the same thing with other games.
Only one, it does not cause it. Because I have tried turning it off and on.
I have tried turning off some other triggers and it did prevent the long exit time - but I have no idea why and how to adjust it since they are critical to the game.
Before exiting the session, what is the virtual address size of StarCraft II?
Such long unload times would be the result of excessively complex game state being cleaned up. For example if a lot of data leaked during the session (was not destroyed when it should) and is then flushed when exiting it will cause a massive stall. Warcraft III suffered something similar if you leaked a lot of objects however its severity was limited due to degrading in game performance and ultimately a crash.
The 64bit build of StarCraft II is pretty much limited to as much virtual memory your system has. If the game state complexity exceeds the available memory then paging will occur. If this complexity is not used, eg leaked data, then the paging will not result in a noticeable performance decrease. However on exiting a session when the complexity is cleaned up then all that data has to be paged in again which can take a long time on mechanical drives. Worse is page thrashing could occur as the cleanup might not be memory sequential.
I recommend checking out actors in case there are some logical leaks there. Although situations like orphaned and duplicate actors should throw an error they might not under some cases. Also actor leaks might not cause performance degradation as invisible actors with no assets could persist in the actor system adding overall complexity while not requiring any computational maintenance.
Although it's a little hard to understand, I really appreciate the response. How can I check the virtual address size? Is that how much memory it's taking up in the task manager?
I will check for any lingering actors.
Is there any thing else that is likely to leak besides actors, such as data effects for example?
Yes those can leak. However the amount that can leak should be limited to only a few hundred or thousand before some limit is reached and errors start to be printed.
As for memory, you will be after the "Commit size" from task manager. That column for the processes view might not be visible by default and requires manual adding to the view (right click on the header and choose to add columns). Commit size represents the amount of allocated virtual memory address space for a process and hence the amount of data needed to be allocated somewhere in the system.
Sorry Doc, but I think your info - although maybe correct, is not relevant to my issue. I have done lot's of data related tests and have failed to reproduce similar results from my game. For the past few months, I've been testing lots of different things to see what could be happening. I don't have any serious data leaks after some numerous checks, but if I did have any it would be marginal.
The real issue I have is using CatalogReferenceSet or CatalogFieldValueSet. I have an extremely dynamic game where tooltips and numbers are based around field values of Damage Effects like AttributeBonus, AttributeFactor. They are updated very often, like every time a hero gets a status update or behavior update. I am starting to realize this issue will worsen, as the map gets even more sophistication.
After about a few million uses of CatalogReferenceSet or CatalogFieldValueSet, I get this long game exit hang. I've done a small test to prove this case (I've attached a map below). So how is it possible to get a few million uses of the function? Each player has a hero, gets about a status update every second (Theoretically). In each status update, there is about 150 (100 - 200) Catalog changes. With an expected game time of 20 - 60 minutes or 180,000 - 540,000+ Catalog changes per person, per game. My map supports up to a max of 12 players. But keep in mind, when certain effects and condition are passed - there are more than 1 status update per second - so it can easily do more than the amounts I just listed. The numbers might be wrong, but the point is that catalog setting has some kind of limit unknown to me and perhaps to most of the community as well. Anyways the cap is easily broken, especially as new items and spells get featured into the game and thus adding even more injury.
Of course, knowing this now - I guess I can work around it and use behaviors to modify certain things like armor, movespeed, attack speed. And write more efficiently - checking if catalog setting is necessary before doing so. But who knew something like catalog setting would have such a hidden tax?
The map below has a custom script that has the following written in it. Just be sure to have window mode on, so you can end task without issues. It writes 384,000 catalog reference every second. Your comp will probably lag, but I recommend just letting the map run for about 20 - 40 seconds and then leaving the game afterwards. Your game will be frame stuck for a really long time, if not forever.
Actually it is exactly as I said. Leaks cause long exit times. The game has problem destructing the game state because it is so complex due to leaks.
However what I did not know was that CatalogFieldValueSet used like this causes a leak. If you look at the memory used by SC2 under certain test conditions.
10,000 -> 950 MB -> 3 seconds to exit
100,000 -> 1000 MB -> 9 seconds to exit
1,000,000 -> 1500 MB -> exit too long to measure (>>5 minutes)
The way it degrades like this points towards some cleanup algorithm with O(n^2) or worse complexity. I am guessing the way catalog natives are implemented is that they make a new entry for the player which takes precedence over the currently used entry. However although it replaces the existing entry, it does not remove it resulting in a leak. When you exit a session it has to clean up all these entries.
Why the cleanup has a complexity of O(n^2) or worse I do not know. I am guessing it might be trying to destroy the entries in the inverse order of creation using a linear search, possibly to maintain base data integrity for caching purposes. However this really is a wild guess.
I would recommend reporting this as a bug to Blizzard. It really should not be leaking like this. I can understand the creation of data structures for a per player data entry however if you change that entry the old data structures should be discarded rather than retained.
That actually makes a lot of sense, although I don't get the algorithms part since I don't know anything about them - the rest of your post does.
I will write something on their forums soon.