Two JVM options are often used to tune JVM heap size: -Xmx for maximum heap size, and -Xms for initial heap size. Here are some common mistakes I have seen when using them:

• Missing m, M, g or G at the end (they are case insensitive). For example,
  

  java -Xmx128 BigApp
  java.lang.OutOfMemoryError: Java heap space

  The correct command should be: java -Xmx128m BigApp. To be precise, -Xmx128 is a valid setting for very small apps, like HelloWorld. But in real life, I guess you really mean -Xmx128m
  


• Extra space in JVM options, or incorrectly use =. For example,
  

  java -Xmx 128m BigApp
  Invalid maximum heap size: -Xmx
  Could not create the Java virtual machine.
  
  java -Xmx=512m HelloWorld
  Invalid maximum heap size: -Xmx=512m
  Could not create the Java virtual machine.

  The correct command should be java -Xmx128m BigApp, with no whitespace nor =. -X options are different than -Dkey=value system properties, where = is used.
  


• Only setting -Xms JVM option and its value is greater than the default maximum heap size, which is 64m. The default minimum heap size seems to be 0. For example,
  

  java -Xms128m BigApp
  Error occurred during initialization of VM
  Incompatible initial and maximum heap sizes specified

  The correct command should be java -Xms128m -Xmx128m BigApp. It's a good idea to set the minimum and maximum heap size to the same value. In any case, don't let the minimum heap size exceed the maximum heap size.
  


• Heap size is larger than your computer's physical memory. For example,
  

  java -Xmx2g BigApp
  Error occurred during initialization of VM
  Could not reserve enough space for object heap
  Could not create the Java virtual machine.

  The fix is to make it lower than the physical memory: java -Xmx1g BigApp
  


• Incorrectly use mb as the unit, where m or M should be used instead.
  

  java -Xms256mb -Xmx256mb BigApp
  Invalid initial heap size: -Xms256mb
  Could not create the Java virtual machine.

• The heap size is larger than JVM thinks you would ever need. For example,
  

  java -Xmx256g BigApp
  Invalid maximum heap size: -Xmx256g
  The specified size exceeds the maximum representable size.
  Could not create the Java virtual machine.

  The fix is to lower it to a reasonable value: java -Xmx256m BigApp


• The value is not expressed in whole number. For example,
  

  java -Xmx0.9g BigApp
  Invalid maximum heap size: -Xmx0.9g
  Could not create the Java virtual machine.

  The correct command should be java -Xmx928m BigApp

PS:

How to set java heap size in Tomcat?
Stop Tomcat server, set environment variable CATALINA_OPTS, and then restart Tomcat. Look at the file tomcat-install/bin/catalina.sh or catalina.bat for how this variable is used. For example,

set CATALINA_OPTS=-Xms512m -Xmx512m  (Windows, no "" around the value)
export CATALINA_OPTS="-Xms512m -Xmx512m"  (ksh/bash, "" around the value)
setenv CATALINA_OPTS "-Xms512m -Xmx512m"  (tcsh/csh, "" around the value)

In catalina.bat or catallina.sh, you may have noticed CATALINA_OPTS, JAVA_OPTS, or both can be used to specify Tomcat JVM options. What is the difference between CATALINA_OPTS and JAVA_OPTS? The name CATALINA_OPTS is specific for Tomcat servlet container, whereas JAVA_OPTS may be used by other java applications (e.g., JBoss). Since environment variables are shared by all applications, we don't want Tomcat to inadvertently pick up the JVM options intended for other apps. I prefer to use CATALINA_OPTS.

How to set java heap size in JBoss?
Stop JBoss server, edit $JBOSS_HOME/bin/run.conf, and then restart JBoss server. You can change the line with JAVA_OPTS to something like:

JAVA_OPTS="-server -Xms128m -Xmx128m"

How to set java heap size in Eclipse?
You have 2 options:
1. Edit eclipse-home/eclipse.ini to be something like the following and restart Eclipse.

-vmargs
-Xms64m
-Xmx256m

2. Or, you can just run eclipse command with additional options at the very end. Anything after -vmargs will be treated as JVM options and passed directly to the JVM. JVM options specified in the command line this way will always override those in eclipse.ini. For example,

eclipse -vmargs -Xms64m -Xmx256m

How to set java heap size in NetBeans?
Exit NetBeans, edit the file netbeans-install/etc/netbeans.conf. For example,

netbeans_default_options="-J-Xms512m -J-Xmx512m -J-XX:PermSize=32m -J-XX:MaxPermSize=128m -J-Xverify:none

How to set java heap size in Apache Ant?
Set environment variable ANT_OPTS. Look at the file $ANT_HOME/bin/ant or %ANT_HOME%\bin\ant.bat, for how this variable is used by Ant runtime.

set ANT_OPTS=-Xms512m -Xmx512m  (Windows)
export ANT_OPTS="-Xms512m -Xmx512m"  (ksh/bash)
setenv ANT_OPTS "-Xms512m -Xmx512m"  (tcsh/csh)

How to set java heap size in jEdit?
jEdit is a java application, and basically you need to set minimum/maximum heap size JVM options when you run java command. jEdit by default runs with a default maximum heap size 64m. When you work on large files, you are likely to get these errors:

java.lang.OutOfMemoryError: Java heap space
at java.lang.String.concat(String.java:2001)
at org.gjt.sp.jedit.buffer.UndoManager.contentInserted(UndoManager.java:160)
at org.gjt.sp.jedit.Buffer.insert(Buffer.java:1139)
at org.gjt.sp.jedit.textarea.JEditTextArea.setSelectedText(JEditTextArea.java:2052)
at org.gjt.sp.jedit.textarea.JEditTextArea.setSelectedText(JEditTextArea.java:2028)
at org.gjt.sp.jedit.Registers.paste(Registers.java:263)

How to fix it? If you click a desktop icon, or Start menu item to start jEdit: right-click the icon or menu item, view its property, and you can see its target is something like:

C:\jdk6\bin\javaw.exe -jar "C:\jedit\jedit.jar"

You can change that line to:

C:\jdk6\bin\javaw.exe -Xmx128m -Xms128m -jar "C:\jedit\jedit.jar"

If you run a script to start jEdit: just add these JVM options to the java line inside the script file:

java -Xmx128m -Xms128m -jar jedit.jar

If you start jEdit by running java command: just add these JVM options to your java command:

java -Xmx128m -Xms128m -jar jedit.jar

Note that when you run java with -jar option, anything after -jar jar-file will be treated as application arguments. So you should always put JVM options before -jar. Otherwise, you will get error:

C:\jedit>java -jar jedit.jar -Xmx128m
Unknown option: -Xmx128m
Usage: jedit [options] [files]

How to set java heap size in JavaEE SDK/J2EE SDK/Glassfish/Sun Java System Application Server?
Stop the application server, edit
$GLASSFISH_HOME/domains/domain1/config/domain.xml, search for XML element name java-config and jvm-options. For example,

<java-config suffix="...">
<jvm-options>-Xmx512m</jvm-options>
<jvm-options>-XX:NewRatio=2</jvm-options>
<jvm-options>-XX:MaxPermSize=128m</jvm-options>
...</java-config>

You can also change these settings in the web-based admin console, typically at http://localhost:4848/, or https://localhost:4848/. Go to Application Server near the top of the left panel, and then on the right panel, click JVM Settings | JVM Options, and you will see a list of existing JVM options. You can add new ones and modify existing ones there.

Yet another option is to use its Command Line Interface (CLI) tool command, such as:

./asadmin help create-jvm-options
./asadmin help delete-jvm-options

They may be a bit hard to use manually, but are well suited for automated scripts.