Avoiding Stack Overflow

If you create long and elaborated macros or use lots of lots of text you probably will run into a stack overflow error.

At first only two things to do come to mind:

1. clean up you html as much as possible (e.g. remove all comment)
2. raise the stack. (start high and work your way down) This is actually a bad advice, below is explained why.

This happens because the way the macro parsing works. It uses a reggex parser that starts by looking for the largest possible string that matches the regex, then backtracking to a shorter string (using recursion) if the first one ends up not working. This ends up requiring a huge amount of stack space in certain pathological strings when combined with the particular regexes being used. Regular expression require stack space based on the amount of text entered and how that text might match the regex. That's the whole point of backtracking -- the regex might match up to a certain point, so it saves its state on the stack and then checks the next piece.

The errors are more likely when the text contains a lot of braces, brackets, and/or single/double quotes. But none of those are REQUIREMENTS for having a stack overflow.

(A search on user "Azrhei" and the word "backtrack" or "backtracking" will probably find the threads that explain this in great detail.)

MapTool creates and destroys many threads on the fly and all the time. Set your stack as low as possible as it's part of the overall address space available to Java. And the larger you make the heap limit, the more of the address space that can be consumed by heap and thus be unavailable for stack space. This is normally only a problem for the 32-bit Java. You'll know you've hit this problem when you get an exception that says that a new Thread couldn't be created and the description will say "Native code". (In other words, it's the C language code that implements the JVM that is having the problem.)

There wont be done anything about this for MT 1.3. We'll be moving to JS for 1.4. This should eliminate the regex stack overflow entirely as all macros will be JS. And that language uses a lexical parser and not a bastardized (but easier to implement) regex parser. The MTscript parser is great at doing what it was designed to do, but it has been pushed waaaaay beyond it's original vision!

Ways to avoid running out of stack.

1. Break up your macros into smaller parts (especcially where you have brackets or quotes, but not just in these cases)
2. If you have long static strings store them in a token property instead of the macro body. Lib tokens work well for this. A table might work as well.
3. Because of the way the regex is structured, you might be able to get more HTML text into a macro by using this construct: [r: "html text here"]
4. Use strformat() to create your html, prefereably with the format string stored in a token property.
5. If you get stack issues its better to refactor your code than bump up the stack.

This issue has come up many times in the forum. This text was stitched together from this latest thread about it.