Avoiding Stack Overflow

Stack Overflow Exceptions

If you create long and elaborate macros or use a lot of text in a macro, you will probably run into a Stack Overflow Exception.

Overview of the Problem

At first only two things come to mind (don't miss more advice at the bottom of this page):

1. Clean up your HTML as much as possible (e.g. remove all comments).
2. Raise the stack size. (Start high and work your way down) This is actually bad advice; why is explained below.

Description of Why the Error Occurs

The error happens because of the way the macro parsing works. It uses a Regular Expression parser that starts by looking for the largest possible string that matches the regex, then backtracking to a shorter string (using recursion) if the first one ends up not working. This ends up requiring a huge amount of stack space in certain pathological strings when combined with the particular regexes being used by MapTool. Regular expressions require stack space based on the amount of text entered and how that text might match the regex. That's the whole point of backtracking -- the regex might match up to a certain point, so it saves its state on the stack and then checks the next piece.

The errors are more likely when the text contains a lot of braces, brackets, and/or single/double quotes. But none of those are REQUIREMENTS for having a stack overflow occur.

(A forum search on user "Azhrei" and the word "backtrack" or "backtracking" will probably find the threads that explain this in greater detail.)

How to Determine an Appropriate Stack Size

MapTool creates and destroys many threads on the fly and all the time. So the best advice would be, Set your stack as low as possible.

The stack size allocated to a particular thread is part of the overall address space available to Java. So the larger you make the heap limit, the more of the address space that can be consumed by heap and thus will be unavailable for stack space. And the same thing works in reverse: the more stack space you allocate per-thread, the less heap space there will be. This is normally only a problem for the 32-bit Java since the maximum heap size is roughly 1.5GB (depends on the platform; Unix systems will be slightly larger than Windows). You'll know you've hit this problem when you get an exception that says that a new Thread couldn't be created and the description will say "Native code". (In other words, it's the C language code that implements the Java Runtime Environment that is having the problem.)

In the 64-bit world, the amount of address space available to the Java Runtime Environment is tremendous (on most hardware it's about 2^40 or roughly 1TB; some hardware will support up to 256TB). That means the stack size and heap space could be set very large. This will still have an effect on the machine's performance, since using more virtual memory than the machine has in physical memory will require paging. And paging is I/O intensive and thus will slow down the entire machine.

How to Change the Amount of Stack Space Available to MapTool

On all systems there is a command line parameter to the Java interpreter that tells it how much stack space to allocate for each thread and how much heap space to allocate overall. The question becomes, "How do I tell my Java installation how much to use?" And that depends on how you start MapTool.

Memory Size Settings When Using the Batch Files (All platforms)

If you have downloaded the application and installed it on your system, you'll find various batch files that can be used to start the application. On the Windows platform, you'd use the files whose names end in .BAT. On Mac OS X, you'd use the .command file. And on other Unix systems, you'd use the .sh file.

When using the .BAT files, simply edit the file and change the values that appear after the -Xss (stack size) and -Xmx (max heap memory).

On Unix systems (including OS X), edit the script and change the variable assignments that appear towards the top of the file.

Memory Size Settings When Using the Mac OS X Installer

If you downloaded the .DMG file for OS X and installed MapTool, you got an application that ended in .app. That's what Apple calls an "application bundle". The memory size information is stored inside that bundle. To change it, you'll need to Ctrl-click that application file and choose Show Package Contents. Inside the Contents directory you'll see a text file named Info.plist. That's where the information is stored.

Double-click the Info.plist file to open the Property List Editor.

<insert image here>

Towards the bottom, click the arrow next to Java and you'll see an entry named VMOptions. As of March 2011, the value shown there will be -Xmx768m -Xss4m. As described above, the number following -Xmx sets the maximum heap space, and the number following -Xss sets the stack size.

Memory Size Settings When Using MapToolLauncher.exe (Windows-only)

This program displays a GUI dialog where you can change the values. When you're done, it saves those changes into MT.CFG and then executes MapTool. If you ever need to reset the values, the easiest way is to simply delete the file! When you run the launcher again it will prompt you for new values.

Memory Size Settings When Using Java Web Start

This is the approach used when you visited the Launch page on the main web site. That page has a hidden configuration panel that you can access by clicking the CUSTOMIZATION link. In that panel you can specify the various memory settings. When you then click the link for the tool you want to run (MapTool, in this case) the configuration file sent to your computer will include all the settings you chose.

Outlook for Future Versions of MapTool

There wont be done anything about this for MT 1.3. We'll be moving to JavaScript for 1.4. This should eliminate the regex stack overflow entirely as all macros will be JavaScript. That language uses a lexical parser and not a bastardized (but easier to implement) regex parser. The MTscript parser is great at doing what it was designed to do, but it has been pushed waaaaay beyond it's original vision!

Ways to Avoid Running Out of Stack Space

Here's a list of things you can do to help avoid stack overflow errors. They are in the order of Most Likely to Least Likely in terms of the level of benefit.

1. Break up your macros into smaller parts (especially where you have brackets or single/double quotes, but not just in these cases)
2. If you have long static strings store them in a token property instead of the macro body (since the content of properties are not subject to the parser). Lib tokens work well for this. A table might work as well if your data is read-only.
3. Use strformat() to create your HTML, preferably with the format string stored in a token property.
4. If you get stack issues its better to refactor your code than bump up the stack.
5. Because of the way the regex is structured, you might be able to get more HTML text into a macro by using this construct: [r: "html text here"] This causes a different path to be taken in the MapTool source code, so different regexes are used.
6. If you get a stack overflow while creating a dynamic form, move the HTML part of the form into a new macro (e.g. you could create a user-defined function named htmlBuilder(arguments) and pass it any variable arguments). Then when you need the form, assign the HTML code first to a variable (e.g. tmpCache) and then run the form using the variable:

 [h: tmpCache = htmlBuilder(arguments)]
 [h: dialog("Dialog Test"): {[r:tmpCache]}]

This issue has come up many times in the forum. This text was stitched together from this latest thread about it.